Why does AI agent observability need a different model?

Standard application monitoring answers: "Did this request succeed? How long did the database query take?" These questions work because traditional software is deterministic: same input, same output, errors have error codes.

AI agents break that model. A single user request to a Foundry hosted agent might trigger ten LLM calls, five tool executions, two file searches, and an MCP server call, each with its own latency and potential failure mode. The agent can return HTTP 200 and still produce a wrong answer. It can silently loop on the same tool call until it hits a token limit. It can hand off a task to a specialist agent and lose context in transit, and none of that shows up in your error rate or P99 latency.

The questions you actually need to answer in production are different:

• Why did this run consume 40,000 tokens when the average is 3,000?
• Which tool call is responsible for the long tail?
• When the orchestrator handed off to the search agent, did the trace context survive? For those, you need hierarchical trace data that maps the agent's decision tree, not just its I/O boundary.

Foundry Agent Service has strong built-in observability. For Prompt agents, server-side tracing is zero-config: connect an Application Insights resource to your project and Foundry automatically captures inputs, outputs, tool calls, token usage, and latency with no code changes required. For Hosted agents, you add client-side instrumentation to your container code, which is what this post focuses on.

What Foundry's built-in tracing doesn't give you is the ability to run arbitrary queries over the raw OTel data or correlate agent traces with infrastructure telemetry from the rest of your stack. That's where routing those same traces to Elastic adds value. The Instrument → Debug → Evaluate → Optimize loop that Foundry supports in its portal is even more powerful when you can drive it from ES|QL queries against the full trace corpus.

A note on Application Insights: it receives OTel traces but converts them into its own schema on ingestion. Elastic stores the data as-is (resource attributes, semantic convention fields, everything preserved) so you can query exactly what was emitted, without a translation layer between you and the data.

LLM tracing with the OpenTelemetry GenAI conventions

Foundry uses OpenTelemetry's GenAI semantic conventions to structure its traces. Three core span types cover most agent workloads.

Stability note: All gen_ai.* span names and attributes currently carry a Development stability badge in the OpenTelemetry registry; they are pre-1.0 and have already changed once (for example, gen_ai.system was renamed to gen_ai.provider.name). Pin your SDK versions and expect attribute strings to shift before these conventions reach stable status.

invoke_agent wraps the entire agent execution. Every run gets one of these as the root span.

chat is a single LLM API call. It carries gen_ai.provider.name (the provider: openai, anthropic, aws.bedrock), gen_ai.request.model, and gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on every single call, which is what makes per-call token cost attribution possible.

execute_tool is a tool or function invocation triggered by the model, nested under the chat span that requested it.

For multi-agent systems, Microsoft (in collaboration with Cisco Outshift) extended these conventions with additional span types now integrated into Foundry, Agent Framework, LangChain, LangGraph, and the OpenAI Agents SDK:

• execute_task captures task planning and how work is decomposed and distributed across agents.
• agent_to_agent_interaction (child of invoke_agent) traces direct communication between agents.
• agent_planning logs an agent's internal planning steps
• agent.state.management covers context and memory operations

The resulting trace for a multi-step Foundry agent looks like this:

[invoke_agent: research-agent]             ← root: the whole task
  [agent_planning]                         ← agent decides its approach
  [chat: azure]                            ← first LLM call
  [execute_tool: file_search]              ← Foundry built-in tool call
  [chat: azure]                            ← LLM call to reason over results
  [agent_to_agent_interaction: summarizer] ← handoff to specialist agent
    [invoke_agent: summarizer]             ← nested agent execution
      [chat: azure]
  [chat: azure]                            ← final synthesis call

In Elastic, the agent trace renders as a waterfall. You see how long each step took, which one errored, and where the tokens went, including across the agent handoff boundary. That's the data model we're getting into Elastic.

How to instrument a Foundry Hosted agent to emit OTel traces

Foundry Hosted agents run your code in a container managed by Foundry. Because you own the container, you control the instrumentation. Which path you take depends on the framework you're using.

| Framework | Language | Key package | Instrumentation method | |---|---|---|---| | Agent Framework | Python | azure-ai-projects | AIProjectInstrumentor().instrument() | | LangGraph | Python | langchain-azure-ai | AzureAIOpenTelemetryTracer callback | | Custom container | TypeScript / Node.js | @opentelemetry/sdk-node | Manual startActiveSpan |

Trace Agent Framework agents with the Azure AI Projects SDK (Python)

Agent Framework is Microsoft's framework for building Hosted agents on Foundry. It calls the Foundry Responses API for model inference and tool orchestration. To route traces to Elastic, configure the OTel SDK at container startup and add the OTLP exporter pointing at your Elastic endpoint.

Install the packages:

pip install azure-ai-projects azure-identity opentelemetry-sdk azure-core-tracing-opentelemetry opentelemetry-exporter-otlp-proto-http

Configure at startup in your agent container:

import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.telemetry import AIProjectInstrumentor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# OTLPSpanExporter reads OTEL_EXPORTER_OTLP_ENDPOINT and
# OTEL_EXPORTER_OTLP_HEADERS automatically, no need to pass them explicitly
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

# Tracing is off by default — opt in explicitly (experimental preview as of 2026)
# Set AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING=true in your container environment
AIProjectInstrumentor().instrument()

client = AIProjectClient(
    endpoint=os.getenv("AZURE_AI_FOUNDRY_PROJECT_ENDPOINT"),
    credential=DefaultAzureCredential(),
)

Every model call, tool invocation, and agent handoff made through the Responses API now produces a structured OTel span with token usage, model identity, and tool metadata. Note that GenAI tracing in the Azure AI Projects SDK is an experimental preview — you must set AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING=true in your container environment and call AIProjectInstrumentor().instrument() before any agent runs, or no spans are produced.

Trace LangGraph agents with AzureAIOpenTelemetryTracer (Python)

LangGraph is a supported framework for Foundry Hosted agents. Microsoft's langchain-azure-ai package provides an OTel-compliant tracer for LangGraph that emits spans for graph steps, tool invocations, and model calls. Configure the OTLP exporter the same way, then attach the tracer as a callback on each invocation:

pip install langchain-azure-ai langgraph langchain langchain-openai opentelemetry-sdk opentelemetry-exporter-otlp-proto-http azure-identity

from langchain_azure_ai.callbacks.tracers import AzureAIOpenTelemetryTracer
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# OTLPSpanExporter reads OTEL_EXPORTER_OTLP_ENDPOINT and
# OTEL_EXPORTER_OTLP_HEADERS automatically, no need to pass them explicitly
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

# AzureAIOpenTelemetryTracer is Microsoft's supported tracer for LangChain and LangGraph.
# Pass it as a callback when invoking your graph — it uses the active TracerProvider above.
azure_tracer = AzureAIOpenTelemetryTracer(name="my-langgraph-agent")

# app is your compiled LangGraph workflow (e.g. workflow.compile())
config = {"callbacks": [azure_tracer]}
result = app.invoke({"messages": [...]}, config=config)

Trace custom Node.js agents with the OpenTelemetry SDK (TypeScript)

For custom agent architectures in Node.js, wrap your orchestration logic directly with the OTel SDK. The startActiveSpan API handles parent-child nesting automatically. Any span created inside the callback is a child of the current span, giving you the decision tree hierarchy without explicit parent references.

import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { trace, SpanKind, SpanStatusCode } from "@opentelemetry/api";

// OTLPTraceExporter reads OTEL_EXPORTER_OTLP_ENDPOINT and
// OTEL_EXPORTER_OTLP_HEADERS automatically, no need to pass them explicitly
const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter(),
  serviceName: "my-foundry-agent",
});
sdk.start();

const tracer = trace.getTracer("foundry-agent", "1.0.0");

// Wrap your agent's entry point: everything inside becomes a child span
async function runAgent(userMessage: string) {
  return tracer.startActiveSpan(
    "invoke_agent my-agent",
    {
      kind: SpanKind.INTERNAL,
      attributes: {
        "gen_ai.operation.name": "invoke_agent",
        "gen_ai.agent.name": "my-agent",
        "gen_ai.provider.name": "azure",
      },
    },
    async (span) => {
      try {
        const result = await agentLoop(userMessage);
        span.setStatus({ code: SpanStatusCode.OK });
        return result;
      } catch (err) {
        span.recordException(err as Error);
        span.setStatus({ code: SpanStatusCode.ERROR });
        throw err;
      } finally {
        span.end();
      }
    }
  );
}

// Record token usage on every LLM call
async function callAzureOpenAI(messages: Message[]) {
  return tracer.startActiveSpan(
    "chat azure",
    {
      kind: SpanKind.CLIENT,
      attributes: {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": "azure",
        "gen_ai.request.model": "gpt-4o",
      },
    },
    async (span) => {
      const response = await client.chat.completions.create({ model: "gpt-4o", messages });
      span.setAttributes({
        "gen_ai.usage.input_tokens": response.usage.prompt_tokens,
        "gen_ai.usage.output_tokens": response.usage.completion_tokens,
        "gen_ai.response.model": response.model,
      });
      span.end();
      return response;
    }
  );
}

If your Node.js agent calls the OpenAI SDK directly, OpenTelemetry JS also ships an instrumentation-openai auto-instrumentation package as an alternative to the manual spans above (it works with Azure OpenAI clients too). Check its semantic-convention version against what's shown here before relying on it; third-party instrumentations for LangChain and others also exist but vary in how current they are.

How to get your Elastic managed OTLP endpoint and API key

The managed OTLP endpoint (mOTLP) is generally available on both Elastic Cloud Serverless and Elastic Cloud Hosted. It is not available for self-managed, ECE, or ECK deployments. For those, use the EDOT Collector as a gateway instead.

Serverless: Log in to Elastic Cloud → find your project → Manage → Application endpoints, cluster and component IDs → Ingest. Copy the endpoint value. Alternatively, go to Add data → Applications → OpenTelemetry inside your project, which also generates a pre-configured API key.

Elastic Cloud Hosted: Log in → Hosted deployments → Manage → Application endpoints → Managed OTLP. Copy the public endpoint value.

The API key must have event:write privilege on the apm application. The Elastic Cloud quickstart wizard generates this for you. Set two environment variables in your Foundry Hosted agent container:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://<your-motlp-endpoint>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey <your-api-key>"

Note the header format: ApiKey \<key>, not Bearer. And the env var uses = as the separator between header name and value, not :.

Traces sent to the endpoint land in the traces-generic.otel-default data stream by default. In Kibana, find them under Observability → APM → Traces or query them directly with ES|QL against traces-generic.otel-*. No OTel Collector required. No schema translation.

Auto-instrument Foundry agents on AKS with the OpenTelemetry Operator

Foundry Hosted agents support bring-your-own VNet and can run container workloads on AKS. If you're in that configuration, you can skip the SDK-level OTLP configuration entirely. Add this annotation to your pod spec and the OpenTelemetry Operator injects the SDK automatically:

annotations:
  instrumentation.opentelemetry.io/inject-python: "true"

You still set OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS to point at Elastic. The SDK wiring is handled for you.

For agents running in Azure Container Apps, the platform's built-in managed OTel collector routes to Application Insights by default. To also get traces into Elastic, use the SDK-level configuration from the examples above alongside Foundry's built-in observability, or run a sidecar OTel Collector with an OTLP exporter configured for your Elastic endpoint.

How to keep a multi-agent handoff in one trace

If your architecture involves an orchestrator delegating to specialist agents, you want all of that to appear as one connected trace, not N disconnected fragments.

OTel handles this via the W3C TraceContext standard (traceparent and tracestate headers). For HTTP-based inter-agent calls, the SDK propagates these automatically. For queue-based handoffs (Service Bus, Event Hubs), you carry the context in the message itself:

from opentelemetry import propagate, context

# Sending agent: inject the active trace context into the message
carrier = {}
propagate.inject(carrier)

await queue.send_message({
    "payload": task_data,
    "trace_context": carrier,  # {"traceparent": "00-abc123...", "tracestate": "..."}
})

# Receiving agent: restore the trace context before doing any work
incoming_ctx = propagate.extract(message["trace_context"])
with context.use_context(incoming_ctx):
    await process_task(message["payload"])

With this in place, the entire work item appears as one trace in Elastic, from orchestrator through every worker agent, across process boundaries. The waterfall shows exactly where time was spent at each tier.

What Foundry agent traces look like in Elastic APM

Traces arrive as a hierarchical waterfall in the APM UI. For each agent invocation, you get the full span tree: every LLM call, tool execution, and sub-agent call nested under the root invoke_agent span. Every chat span carries gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, so you can see at a glance which model invocations are expensive and which are routine. Errors surface as specific failed spans with full stack traces, not a red line on a latency graph.

You can also query the trace data directly with ES|QL. To find every agent run in the past hour that consumed more than 20,000 input tokens:

FROM traces-generic.otel-default
| WHERE attributes.gen_ai.operation.name == "invoke_agent"
  AND @timestamp > NOW() - 1 hour
| STATS total_input_tokens = SUM(attributes.gen_ai.usage.input_tokens)
    BY trace.id, attributes.gen_ai.agent.name
| WHERE total_input_tokens > 20000
| SORT total_input_tokens DESC

Combining trace structure with token accounting in a single query is what OTel native storage makes possible.

How to sample agent traces without losing errors

For tail-based sampling, add a local OTel Collector as an intermediate hop before the Elastic endpoint. A policy that keeps all error traces, all slow traces, and samples down routine successes is a reasonable starting point:

# Run this Collector between your Foundry Hosted agent and the Elastic endpoint
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 5000 }
      - name: sample-routine
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  otlp/elastic:
    endpoint: "${OTEL_EXPORTER_OTLP_ENDPOINT}"
    headers:
      Authorization: "ApiKey ${ELASTIC_API_KEY}"
    sending_queue:
      enabled: true
      sizer: bytes
      queue_size: 50_000_000
      block_on_overflow: true

If you're sending directly from the SDK without a Collector, start at 100% sampling. Agent trace volume is usually smaller than expected. Evaluate storage costs after a week and tune from there. For most Foundry Hosted agent workloads, the direct SDK path is the right starting point.

How to use agent traces for evaluation and optimization

The Build session demo framed tracing as a four-step production loop: instrument, debug, evaluate, optimize. Debugging is the obvious first payoff: when an agent run fails or produces a wrong answer, the trace shows exactly which span introduced the problem. But the evaluate and optimize steps are where the loop compounds.

With traces flowing into Elastic, you can identify which agent runs produced incorrect or low-quality outputs, then pull those traces directly into an evaluation workflow in Foundry. The trace gives you the full context (the prompt, the tool calls, the model's reasoning path) that an eval needs to score quality and catch regressions. From there, optimization targets are concrete: this tool call is slow, this model is expensive for its output quality, this planning step runs unnecessarily on every request.

Foundry's agent optimizer can use trace data to improve agent instructions automatically. Elastic gives you the query layer to find the traces worth optimizing in the first place.

What you need to start tracing Foundry agents

You need four things: an Elastic Cloud Serverless project (the managed OTLP endpoint is included), your endpoint URL and API key from Project Management → Edit alias, instrumentation that matches your stack (AIProjectInstrumentor for Agent Framework, AzureAIOpenTelemetryTracer from langchain-azure-ai for LangGraph, or the OTel SDK directly for custom container code), and one agent run to verify the trace appears in Kibana's APM view.

The first trace that shows you exactly which tool call caused that 45-second timeout makes the setup worth it.

More resources: Microsoft Foundry Agent Service overview | Set up tracing in Foundry | Tracing integrations by framework | Elastic managed OTLP endpoint docs | OpenTelemetry GenAI semantic conventions | Build 2026 DEM341: Any agent, any cloud

Why manual incident response can't keep up at scale

Modern platforms run at a scale the human brain was never meant to triage. A single service can handle tens of thousands of transactions a second, and each one leaves a trail of metrics, logs, and traces. When something breaks, the answer is somewhere in that flood of data, but finding it by hand is slow, and slow is expensive. The AI era is making it worse, not better: data volumes are compounding, and the more tools a team adds, the more scattered the answer becomes.

Three things go wrong in the manual model:

• Exploding cost and volume. The sheer amount of telemetry is compounding observability spend, and teams often drop data to control the bill, which means the one signal that explains the outage may not even be there when they look.
• Lost and fragmented context. The real story usually lives across container logs, infrastructure events, and application traces at once. When those sit in different tools, no single platform correlates them at the moment an alert fires, and stitching them together under pressure is slow and easy to get wrong.
• Slow investigations and rushed calls. Every minute spent searching for the cause is a minute the outage continues, and a 3am restart made on a hunch can make the incident worse instead of better.

The result is long outages, stressed teams, and a Mean Time to Resolution (MTTR) that stays stubbornly high no matter how many dashboards you build. Dashboards show you the problem. They do not fix it.

What automated incident response does end to end

Autonomous SRE means the system handles the full incident loop on its own: it detects the problem, investigates the likely cause, resolves it with a corrective action, and verifies the action worked. The same loop a skilled on-call engineer runs in their head, running continuously and at machine speed.

It follows a simple cycle: detect, investigate, resolve, verify.

1. Detect. Elastic Observability continuously collects the metrics, logs, and traces from across the environment and maintains a live system model: an always-current map of your services, hosts, and the dependencies between them. It surfaces the events that actually matter instead of flooding the team with raw alerts.
2. Investigate. When something crosses a threshold, the platform pulls together the related evidence and asks an AI model, one that reads the evidence and explains it in plain language, what is happening, how confident it is, and which other services the problem will affect.
3. Resolve. Based on that diagnosis, a remediation step is carried out, for example, scaling a service, restarting it, or rolling back a recent change, either automatically or with a person's approval.
4. Verify. The system then checks whether the fix worked and the service returned to healthy, and records the whole sequence so a human can review exactly what happened and why.

The important word is loop. The system does not stop at an alert or a recommendation. It closes the gap between knowing and doing, which is precisely the gap where outages live.

How Elastic Observability correlates telemetry for AI root cause analysis

Autonomous action is only as good as the context behind it, and context is where Elastic Observability is strong. It brings the metrics, logs, and traces from across your environment into one place, then builds a live model of how those pieces connect, so the system reasons over the full story rather than a single noisy signal.

That unified view matters for three reasons:

• Better diagnosis. Grounding the AI model in real, correlated telemetry rather than a single metric means the diagnosis reflects what is actually happening, not a guess.
• Fewer false moves. When the evidence is complete, the system is far less likely to act on a symptom and miss the cause.
• A record you can trust. Every observation, decision, and action is captured, so the incident comes with a built-in audit trail instead of a gap in the story.

This is the same foundation Elastic already provides for search and security, applied to keeping services healthy.

Why automated incident response runs on Red Hat OpenShift

The reason this approach works for regulated, sovereign, and on-premises environments is that the entire loop runs inside your own Red Hat OpenShift cluster. Nothing about an incident, not the telemetry, not the diagnosis, not the action, has to leave your walls.

Four things make Red Hat OpenShift the right home for it:

• The full stack is in-cluster. Elastic Observability is deployed and managed natively on Red Hat OpenShift through the Elastic Cloud on Kubernetes (ECK) operator, so the data foundation lives next to the workloads it watches. That keeps analysis fast and keeps your data under your control. It runs on managed OpenShift (such as Red Hat OpenShift Service on AWS or Azure Red Hat OpenShift) or self-managed Red Hat OpenShift.
• The reasoning model runs locally too. Red Hat OpenShift AI serves the language model, for example, IBM Granite, inside the same cluster. The AI that diagnoses your incidents never sends your telemetry to an outside service, which is what makes the approach viable for air-gapped and sovereign deployments.
• Remediation speaks to Red Hat OpenShift natively. When the system acts, it is performing ordinary Red Hat Kubernetes operations: scaling a deployment, restarting a workload, rolling back to the last good version. These are the same actions your platform team already trusts, now triggered automatically and verified.
• It is a packaged, validated starting point. The whole pattern ships as a quickstart in the Red Hat catalog, built jointly so a team can stand it up on an existing Red Hat OpenShift environment and see the loop work without assembling the pieces from scratch.

The payoff is sovereignty without a tradeoff: you get machine-speed, AI-driven incident response and you keep every byte of telemetry and every decision inside infrastructure you already run.

From alert to verified fix

Here is the routine incident, told the way the team experiences it once the loop is in place.

A service starts to slow down. Response times climb past their healthy range and an alert fires, the same trigger that would normally start a human's night. Instead, the platform immediately gathers the relevant evidence: which service, what changed recently, the related errors and events. It hands that package to the model running on Red Hat OpenShift AI, which returns a plain-language diagnosis, a confidence level, and the blast radius, which is the set of other services this incident will affect if it is left alone.

The recommended action, if it is one you have allowed to run automatically, is then carried out as a native Red Hat Kubernetes operation; the service is scaled to absorb the load, and the platform watches the response times settle back to normal. The entire sequence, from the first alert to the confirmed recovery, is written up as a case: what happened, why, what was done, and the proof it worked. In the morning, the team reviews a finished incident report instead of reconstructing a fire drill.

The engineer's job shifts from finding and fixing to reviewing and improving. That is the real change. The work moves from reactive firefighting to oversight.

Which Kubernetes incidents can be remediated automatically

Autonomous response is most valuable on the common, well-understood incidents, the ones that are tedious rather than novel. Each maps to a native Red Hat Kubernetes action the system can take and then verify:

| When this happens | The Red Hat Kubernetes action | And confirms it by | | ---- | ---- | ---- | | A service slows down under load | Scales the deployment to add capacity | Watching response times return to normal | | A service runs out of memory and crashes | Restarts the workload cleanly | Checking it comes back healthy and stays up | | A recent change breaks something | Rolls back to the last good version | Confirming the service passes its health checks again |

These are the incidents that make up most of the pages a team gets, and they are exactly the ones a closed loop is best suited to take off their plate. Novel or high-stakes incidents still rise to a human, which is by design.

How you stay in control of automated remediation

Autonomous does not mean unaccountable. The principle is simple: the agent recommends, you decide. The system removes the toil, not the oversight, and a few rules keep humans firmly in charge.

• Every action is recorded. The full sequence, from diagnosis to action to verification, is captured as a reviewable case. Nothing happens off the record.
• Confidence is part of the decision. Remediation options are ranked by confidence, so high-confidence fixes can run automatically while low-confidence situations are routed to a person instead of acted on.
• Humans set the boundaries. You decide which actions the system is allowed to take on its own and which require a human to approve. You can keep a person in the loop wherever it matters.
• Your model, in your cluster. The reasoning runs on Red Hat OpenShift AI inside your own environment, so sensitive telemetry never has to leave your walls. For regulated and sovereign environments, that keeps the whole loop, data and decisions alike, under your control.

The goal is a system that earns trust the way a good junior engineer does: it shows its work, it knows when to ask, and it never hides what it did.

How automated incident response reduces MTTR

The headline outcome is a lower MTTR, because the slow part of an incident, the time before anyone understands it, is largely removed. Teams using Elastic Observability have put real numbers on that shift:

• WePay, a Chase company, cut the time to find customer impact during incidents by 90%, improving app performance and releasing product faster.
• DISH Media reached 100% visibility, a 10x increase in coverage, and reduced problem-resolution time for its developers.
• Accolade monitors about 400 services on Elastic Observability and doubled developer productivity.

Beyond the numbers, the change is structural:

• Consistency. The loop responds the same correct way at 3am as it does at 3pm. No fatigue, no improvised fixes.
• Capacity. Engineers stop spending nights on routine incidents and get that time back for the work only humans can do.
• Resilience. Faster, more consistent recovery means outages stay small, and small outages are the ones customers never notice.

In other words, autonomous SRE does not just make incident response faster. It changes what your best people spend their time on. That direction is also where the market is heading: Elastic was named a Leader in the 2025 Gartner Magic Quadrant for Observability Platforms.

How to get started on your existing OpenShift cluster

You do not need to rebuild your stack to begin. Because the pattern ships as a quickstart in the Red Hat catalog, the natural first step is to deploy it on an existing Red Hat OpenShift environment, with Elastic Observability as the unified view across your services, so the context for good decisions already exists. From there you can let the loop run in an advisory mode, where it diagnoses and recommends while a human approves each action, and widen its autonomy as it earns trust on the routine incidents.

Standing it up on Red Hat OpenShift means the observability stack, the reasoning model on Red Hat OpenShift AI, and the remediation agent all come up together inside one cluster, so a team can see the full detect-investigate-resolve-verify loop work end to end before committing to it broadly.

Start by letting the system watch and explain. Let it recommend. Then, incident by incident, let it act. The path to zero-touch operations is incremental, and every step along the way buys back time your team is spending on the 3am page today.

Frequently asked questions

What is autonomous SRE? Autonomous SRE is an approach where the observability platform handles the full incident loop on its own: detecting the problem, investigating the likely cause, resolving it with a corrective action, and verifying the service recovered. It automates the routine incident so engineers can focus on the novel and the high-stakes.

Why run autonomous SRE on Red Hat OpenShift? Running the loop on Red Hat OpenShift keeps the entire stack in-cluster: Elastic Observability (deployed through the ECK operator), the reasoning model on Red Hat OpenShift AI, and the remediation agent all run inside your own environment. No telemetry or decisions leave the cluster, which is what makes the approach a fit for regulated, sovereign, air-gapped, and on-premises deployments. It runs on Red Hat OpenShift Service on AWS, Azure Red Hat OpenShift, or self-managed Red Hat OpenShift.

Does autonomous SRE replace my engineers? No. It removes the repetitive toil, the routine 3am pages, and shifts engineers from finding-and-fixing to reviewing-and-improving. Humans set which actions are allowed to run automatically, approve anything sensitive, and review every action after the fact.

How does Elastic Observability decide what to do? It grounds an AI model in real, correlated telemetry, the metrics, logs, and traces from across your environment, plus a live model of how your services depend on each other. The model returns a cause, a confidence level, and the blast radius, and the platform acts only within the boundaries you set.

Is it safe to let a system take action automatically? Every action is recorded as a reviewable case, remediation options are ranked by confidence, low-confidence situations are routed to a human, and you choose which actions run automatically versus require approval. The remediation steps are ordinary Red Hat OpenShift operations, the same ones your platform team already trusts.

Can this run without sending my data to an outside service? Yes. With the reasoning model served by Red Hat OpenShift AI inside your cluster, sensitive telemetry never leaves your infrastructure. That makes the approach a fit for regulated, sovereign, and on-premises settings.

To go deeper on the architecture behind this, see the companion technical walkthrough of the in-cluster autonomous SRE stack on Red Hat OpenShift (coming soon). For the broader picture of how Elastic supports agentic and AI-driven workflows, see Elastic's Search AI Platform.

Gartner, Magic Quadrant for Observability Platforms, 7 July 2025. Gartner does not endorse any vendor, product, or service depicted in its research publications. GARTNER and Magic Quadrant are registered trademarks of Gartner, Inc. and/or its affiliates and are used herein with permission. All rights reserved.

Forking the agent would fix this, but then you own every upstream merge. You could also use manual instrumentation for complete control, but that requires code changes and ongoing upkeep. OpenTelemetry Java extensions give you a cleaner path: a separate JAR the agent loads at startup, giving you precise control over what gets captured and exported, without touching agent or application code.

For example, the following challenges are very common:

• Health check probes are flooding your trace view.
• Span names reflect generic framework patterns rather than your domain operations.
• Some span names or attributes have high cardinality creating noise in your traces.
• Spans are missing attributes relevant to your business logic.
• Baggage headers are propagating to downstream services when they shouldn't.
• Resource attributes that describe your deployment are not automatically captured because they rely on custom environment variables.

Some of those can be solved through configuration, or by using an intermediate OpenTelemetry Collector for processing. However, this also might add complexity to the telemetry pipeline, and you might prefer to solve this at the source, where the data is captured.

What are OpenTelemetry Java extensions

An extension is a JAR file the agent loads at startup. It hooks into the agent's extension points through Java's Service Provider Interface (SPI) mechanism, the same mechanism the agent uses internally.

The extension mechanism works identically with the upstream OpenTelemetry Java agent and with Elastic's OpenTelemetry distribution. You write the extension once and it works with either.

For reference, the upstream extension documentation provides an exhaustive overview of extension points and a few examples.

This post does not aim to provide a complete reference, but focuses on simple use cases you're likely to reach for in production: renaming spans, filtering noisy traces, or propagating context that the agent doesn't cover in your environment.

Extensions also let you modify and extend the agent instrumentation itself. That goes beyond what this post covers. Here are two starting points:

• Modify instrumentation using instrumenter customizers.
• Modify or add instrumentation using the instrumentation module.

Setting up an OpenTelemetry Java extension project

An extension is a standard Java Gradle project with two requirements: the output must be a shadow JAR (a fat JAR with all extension dependencies bundled), and OpenTelemetry dependencies must be declared compileOnly so you don't bundle the SDK itself.

The shadow JAR requirement exists because the agent loads the extension in its own classloader. If you declare a dependency as implementation, it gets bundled and may conflict with the version already in the agent. Using compileOnly keeps those JARs out of the extension JAR entirely.

Here is a minimal build.gradle.kts for a simple extension that does not customize instrumentation and thus relies only on the OpenTelemetry SDK/API.

plugins {
  id("java")
  id("com.gradleup.shadow")
}

repositories {
  mavenCentral()
}

java {
  toolchain {
    languageVersion.set(JavaLanguageVersion.of(8))
  }
}

dependencies {
  // Use BOM to manage OpenTelemetry dependency versions
  compileOnly(platform("io.opentelemetry:opentelemetry-bom:1.64.0"))
  // OpenTelemetry SDK autoconfiguration SPI (provided by agent)
  compileOnly("io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi")
  // OpenTelemetry SDK
  compileOnly("io.opentelemetry:opentelemetry-sdk")
  // Annotation processor for automatic SPI registration
  compileOnly("com.google.auto.service:auto-service:1.1.1")
  annotationProcessor("com.google.auto.service:auto-service:1.1.1")
}

tasks.assemble {
  dependsOn(tasks.shadowJar)
}

Check Maven Central for the latest version of the BOM before starting.

Extensions only depend at compile-time on the OpenTelemetry SDK and the autoconfiguration SPI. The agent provides the rest of the SDK and instrumentation implementation at runtime.

Loading OpenTelemetry Java extensions at runtime

To load an OpenTelemetry Java extension at runtime, you can use the otel.javaagent.extensions system property or OTEL_JAVAAGENT_EXTENSIONS environment variable. The value is a comma-separated list of paths to extension JARs:

java -Dotel.javaagent.extensions=/path/to/my-extension.jar -javaagent:/path/to/opentelemetry-javaagent.jar -jar myapp.jar

The upstream OpenTelemetry Java agent also lets you embed extensions directly into the agent JAR to simplify deployment.

Filtering and renaming spans with OpenTelemetry Java extensions

You can modify spans in two ways:

• Using a SpanProcessor that is called synchronously when the span starts or ends.
• Using a SpanExporter that is called asynchronously when the span is exported.

Rename spans with a SpanProcessor

SpanProcessor.onStart receives a ReadWriteSpan, which means you can call span.updateName() before the span is exported. This is the right hook for renaming based on attributes that are available at span start.

public class OperationRenamingSpanProcessor implements SpanProcessor {

  @Override
  public void onStart(Context parentContext, ReadWriteSpan span) {
    String operation = span.getAttribute(AttributeKey.stringKey("app.operation"));
    if (operation != null) {
      span.updateName(operation);
    }
  }

  @Override
  public boolean isStartRequired() { return true; }

  @Override
  public void onEnd(ReadableSpan span) {}

  @Override
  public boolean isEndRequired() { return false; }

  @Override
  public CompletableResultCode shutdown() { return CompletableResultCode.ofSuccess(); }

  @Override
  public CompletableResultCode forceFlush() { return CompletableResultCode.ofSuccess(); }
}

Register the SpanProcessor via AutoConfigurationCustomizerProvider, composing it with whatever processor you have already configured:

@AutoService(AutoConfigurationCustomizerProvider.class)
public class RenamingCustomizerProvider implements AutoConfigurationCustomizerProvider {

  @Override
  public void customize(AutoConfigurationCustomizer customizer) {
    customizer.addTracerProviderCustomizer(this::configureSdkTracerProvider);
  }

  private SdkTracerProviderBuilder configureSdkTracerProvider(
      SdkTracerProviderBuilder tracerProvider, ConfigProperties config) {
    return tracerProvider.addSpanProcessor(new OperationRenamingSpanProcessor());
  }

}

The modify-span EDOT Java extension example provides a complete implementation.

Filter spans with a SpanExporter

A SpanExporter wrapper lets you modify or drop spans before they leave the process. This works well for known noisy endpoints like health checks.

public class FilteringSpanExporter implements SpanExporter {

  private final SpanExporter delegate;

  public FilteringSpanExporter(SpanExporter delegate) {
    this.delegate = delegate;
  }

  @Override
  public CompletableResultCode export(Collection<SpanData> spans) {
    List<SpanData> filtered = new ArrayList<>();
    for (SpanData span : spans) {
      if (!"GET /health".equals(span.getName())) {
        filtered.add(span);
      }
    }
    return delegate.export(filtered);
  }

  @Override
  public CompletableResultCode flush() { return delegate.flush(); }

  @Override
  public CompletableResultCode shutdown() { return delegate.shutdown(); }
}

Register the FilteringSpanExporter via addSpanExporterCustomizer:

customizer.addSpanExporterCustomizer((existing, config) -> new FilteringSpanExporter(existing));

The modify-span EDOT Java extension example provides a complete implementation.

The approach has two limitations:

• This won't discard any child span that may have been created, for example, if the healthcheck calls the database.
• Spans filtered at the exporter have already passed through the full processor pipeline and occupied buffer space in the batch processor.

If you're dropping a large fraction of your traffic at this stage, a custom Sampler (shown below) is more efficient because it drops spans before any processing happens and also filters out child spans. Also, when using declarative configuration, the rule-based sampler lets you implement filtering on rules using only configuration.

Adding custom resource attributes with a ResourceProvider

Resource attributes describe what's running: the service name, its version, the host. A ResourceProvider lets you attach additional attributes that the agent doesn't know about, such as deployment metadata your platform injects through environment variables.

The example below uses environment variables, but it could also be a configuration file, a cloud metadata service, or any other source available to the agent at startup.

Because the SDK initialization is synchronous, when querying an external service like a metadata endpoint, this can make the agent (and thus the application) startup slower. If possible, prefer checking environment variables and local config first before calling an external service.

@AutoService(ResourceProvider.class)
public class DeploymentResourceProvider implements ResourceProvider {

  @Override
  public Resource createResource(ConfigProperties config) {
    AttributesBuilder attributes = Attributes.builder();

    String region = System.getenv("DEPLOY_REGION");
    if (region != null) {
      attributes.put(AttributeKey.stringKey("deployment.region"), region);
    }

    String buildVersion = System.getenv("BUILD_VERSION");
    if (buildVersion != null) {
      attributes.put(AttributeKey.stringKey("build.version"), buildVersion);
    }

    return Resource.create(attributes.build());
  }
}

Attributes from a ResourceProvider merge with the agent's own resource. When two providers supply the same key, the one with the higher order() value wins. The agent's built-in providers use order 0, so overriding order() to return a positive integer gives your provider priority.

The resource-attribute EDOT Java extension example provides a complete implementation.

Custom sampling in OpenTelemetry Java

When filtering at the exporter is too late or too expensive, implement a Sampler directly. The sampler runs before any span processing, so dropped spans never touch the batch buffer.

However, the sampling decision can only rely on attributes that are provided when the span starts. For example, the status code of an HTTP response can't be used as it is only available when the span ends.

The key detail: wrap the existing sampler rather than replacing it. That way, your logic composes with whatever you configured, and parent-based decisions from an upstream service are still respected.

public class HealthCheckSampler implements Sampler {

  private final Sampler delegate;

  public HealthCheckSampler(Sampler delegate) {
    this.delegate = delegate;
  }

  @Override
  public SamplingResult shouldSample(
      Context parentContext,
      String traceId,
      String name,
      SpanKind spanKind,
      Attributes attributes,
      List<LinkData> parentLinks) {
    if (spanKind == SpanKind.SERVER && name.contains("health")) {
      return SamplingResult.create(SamplingDecision.DROP);
    }
    return delegate.shouldSample(parentContext, traceId, name, spanKind, attributes, parentLinks);
  }

  @Override
  public String getDescription() {
    return "HealthCheckSampler{" + delegate.getDescription() + "}";
  }
}

Register the HealthCheckSampler via addSamplerCustomizer, which gives you both the existing sampler and the resolved config:

customizer.addSamplerCustomizer((existing, config) -> new HealthCheckSampler(existing));

Community extensions in opentelemetry-java-contrib

The opentelemetry-java-contrib repository contains several community-maintained extensions.

Some of them are already included in the OpenTelemetry Java agent (and inherited in the Elastic distribution), but are opt-in:

• azure-resources
• aws-resources
• gcp-resources
• cloudfoundry-resources
• baggage-processor

Most Elastic distribution features exist as extensions in the contrib repository, so you can use them with the upstream agent in a vendor-neutral way.

• inferred-spans
• span-stacktrace

Further reading and extension examples

The upstream extension examples cover additional extension points not shown here, including custom propagators, ID generators, and ignored-type configurers.

The Elastic baggage example shows the filtering propagator for baggage running end-to-end with a two-service application, it also demonstrates custom instrumentation to add baggage without modifying the application code.

This post covered the project setup and the patterns most likely to come up in production. Both links above go deeper: the upstream examples add extension points not covered here, and the baggage example shows a complete two-service implementation you can run locally.

That team still wants a model. So the request lands on an SRE's desk, and it sounds deceptively small: "Can you stand up an open-weight model for the claims team? Sixty people. It has to run on our hardware." They aren't even allowed to use a neocloud. There is a cost associated with this, but we won't explore that part. Just the part that covers running the model and observing the configuration.

Standing it up is the easy half. Four manifests and an afternoon, and you have a model answering questions. The hard part arrives a week later, when someone says "it feels slow" and you realize you have no idea whether the deployment is configured well, badly, or catastrophically — and no obvious way to find out.

This guide shows you how Elastic Observability can help you analyze the metrics from the configuration. It walks through tuning a real vLLM deployment using the metrics vLLM already emits. vLLM exposes these on a /metrics endpoint in Prometheus exposition format — no instrumentation, no sidecar, no code change — which is why every query in this guide starts from a Prometheus scrape. The goal: turn "it feels slow" into a specific, defensible decision.

Test environment: vLLM on a Kubernetes cluster using NVIDIA A10G with dcgm-exporter and Prometheus metrics

Every figure in this guide was measured on the following stack — one replica, one GPU, no autoscaling.

• Workload — Kubernetes-native load generator, scaled from 8 to 32 concurrent requests.
• Model — Qwen/Qwen2.5-3B-Instruct, bf16, --max-model-len 4096
• Engine — vLLM v0.23.0, OpenAI-compatible server, Prometheus /metrics on :8000
• GPU — NVIDIA A10G, 24 GB — an AWS g5.xlarge
• Cluster — Amazon EKS 1.30, tainted GPU node pool with minSize: 0
• Telemetry — Prometheus scraping every 15s, plus dcgm-exporter on :9400, shipped via remote_write
• Analysis — Elastic Observability, queried with ES|QL and PromQL
• Measured — 2026-07-27

Why is it hard for an SRE to self-host and tune an open-weight LLM?

The difficulty is not the deployment, it's the tuning which has no feedback loop. vLLM starts, serves, and reports success whether it's configured brilliantly or wastefully. Nothing tells you which.

When loading up the model, your manifest would have this configuration:

      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.23.0  # pin an exact release — metric names shift between versions
          args:
            - "--model=Qwen/Qwen2.5-3B-Instruct"
            - "--max-model-len=4096"         # cap context → predictable KV-cache size
            # A10G has native bf16 — do NOT add --dtype=half (T4-only).
          ports:
            - name: http
              containerPort: 8000            # OpenAI API + /metrics

But you can run into specific issues, such as: Hugging Face downloads take minutes. If your cluster expects a server to start in 30 seconds, it will assume the app is dead and kill it mid-download, putting you in an infinite crash loop.

Or you could have a hardware mismatch, and potentially degrade your model’s speed or precision because hardware architectures vary

or a bevy of other issues.

Once the model is finally running, optimizing performance is complete guesswork because default metrics don't tell you if you're being efficient.

You could use nvidia-smi, but this only understands raw hardware state, not application software logic.

Now that you have it running, a few hours to maybe even a day in, the team says "it feels slow." You are, functionally, tuning blind.

How do you tune a self-hosted vLLM deployment?

You're not an inference engineer, you own forty other services besides this one, and you don't have a forward-deployed engineer from a model vendor on call. But tuning an LLM server turns out to need exactly one skill you already have: reading telemetry and reasoning about saturation. The only missing piece is telemetry that exists and means something.

It does. vLLM emits a rich Prometheus endpoint out of the box — latency decomposed by inference phase, cache hit rates, batch occupancy, token accounting, completion outcomes. Almost nobody looks at it. The rest of this guide is how to read it.

Defining the workload: sixty users, short prompts, streaming responses

With the slowness detected and reported, you gather the usage profile of the users. Their usage pattern is as follows:

• ~60 users, but not concurrent. Realistic peak is 8–12 simultaneous in-flight requests; sustained is lower.
• Short prompts, long answers. The user pastes a paragraph and asks for a structured summary. Prompts run ~50 tokens; useful answers run 500–1,000.
• Interactive, streaming UI. Perceived speed is dominated by time to first token (TTFT), not total time — the same psychology as a chat interface.
• Heavy prompt reuse. Every request carries the same system prompt and the same policy-language boilerplate.

From that, you write down actual service objectives — the step most self-hosted LLM projects skip:

• TTFT p95 < 300 ms
• inter-token latency < 50 ms (≥ 20 tokens/sec, faster than reading speed)
• zero queueing at 12 concurrent
• error + abort rate < 0.5%

Those four numbers are the point of everything that follows. Without them, "it feels slow" has no answer. With them, every metric below passes or fails a stated bar.

How do you get Prometheus metrics out of vLLM and the GPU?

There are two sources, and you need both.

• vLLM reports on itself — latency by phase, cache hit rates, batch occupancy, token counts — on /metrics at its serving port, with no adapter and no instrumentation work.
• The GPU reports separately, through NVIDIA's dcgm-exporter on :9400 (NVIDIA Data Center GPU Manager (DCGM) is a suite of tools and libraries designed to comprehensively manage, monitor, and diagnose enterprise-grade NVIDIA GPUs in clusters and data centers). vLLM tells you what the engine thinks is happening; DCGM tells you what the card is actually doing. Step 6 is built entirely on the gap between those two answers.

On this EKS cluster that means three things running side by side:

• vLLM as a plain Deployment on the tainted g5.xlarge GPU node pool. For one model on one card, a Deployment and a Service is the whole architecture.
• dcgm-exporter as a DaemonSet, pinned to the same GPU nodes.
• A Prometheus server on a CPU node, scraping both endpoints every 15 seconds.

Nothing here is AWS-specific. The production version is the same manifests on an on-prem cluster with L40S or H100 nodes — which is the point of doing this on Kubernetes rather than on a vendor's platform.

What about KServe and llm-d?

Neither was run for this guide, and neither changes where the metrics come from. KServe and llm-d sit on top of vLLM rather than replacing it — vLLM is still the engine, so /metrics is still the source of every number here. Each adds its own layer on top (KServe: autoscaler and revision metrics; llm-d: router and cache-routing metrics), but the inference telemetry underneath is identical.

What they change is when you need them — and each promotion is triggered by a metric you're already collecting:

How do you ship the vLLM metrics and DCGM metrics to Observability

A Prometheus scraping inside the cluster only holds hours of data — the metrics have to reach a store you can still query next week. There are two paths for that, and they are not equivalent.

Path A — OpenTelemetry Collector. Puts inference metrics into the same pipeline as your traces and logs. One collector, one auth path, one mental model. The cost is that Prometheus metrics sent through OTLP get normalized: the stored schema ends up neither purely Prometheus nor purely OTel, and metric names shift.

Path B — native Prometheus remote_write. Stands up a small Prometheus that scrapes both endpoints and pushes to a backend speaking the remote-write protocol. Names and labels land untouched, _sum / _count / _bucket histogram parts stay intact, and existing queries keep working.

For a tuning exercise, choose Path B. That's what produced every number in this guide. The reason is narrow but decisive: tuning means comparing against the vLLM documentation and the vLLM community, and both speak in exact metric names. When your chart says vllm:kv_cache_usage_perc, you can search for it.

The deployment is two YAML files — a Prometheus Deployment with two scrape jobs and a remote_write block, plus a Secret holding the backend credential. In this build the destination was an Elastic Serverless project, which exposes a Prometheus remote-write endpoint and lands data in a time-series data stream, metrics-vllm.prometheus-inference.

Two things cost me real time. If your backend has a separate ingest host for OTLP versus its main API, remote-write usually lives on the main API host, not the ingest one — pointing at the wrong one returns a 404 that looks like a path error. And the credential needs index-write privileges, not just ingest authentication; a key that works fine for OTLP can authenticate successfully and then 403 on every sample. Check prometheus_remote_storage_samples_failed_total on the Prometheus itself before looking anywhere else.

How do you confirm vLLM metrics landed in Elastic?

Once the pipeline is up, look at the field list. Roughly 127 metric series arrive from a single vLLM pod plus DCGM:

This screen is more useful than it looks. Scanning the field list is how you confirm your vLLM version's exact metric names — they do shift between major vLLM releases, and a dashboard built against the wrong names fails silently by returning nothing rather than erroring.

Two gotchas when querying vLLM metrics

Both produce results that look like "the metrics aren't working" when the pipeline is perfectly healthy.

vLLM metric names contain a colon (vllm:num_requests_running), so they need escaping in most query languages. More insidiously, if you filter by metric name across several metrics and then aggregate only one of them, you get rows back — full of nulls, with no error. Each Prometheus metric lands in its own field, so naming the field is the filter; you don't need the name predicate at all.

Counters need rate functions, gauges don't. vllm:generation_tokens_total is cumulative and monotonic — taking a max of it gives the pod's lifetime total, not its throughput. Gauges like vllm:num_requests_running, vllm:num_requests_waiting and vllm:kv_cache_usage_perc are instantaneous and want max or average. Mixing these up produces charts that are wrong but plausible, which is considerably worse than charts that are empty.

Which vLLM Prometheus metrics actually matter?

A reference for the metrics used in this guide, what each tells you, and the condition worth watching. Names are as vLLM emits them; the DCGM_FI_* series come from dcgm-exporter.

| Metric | Type | What it tells you | Watch for | |---|---|---|---| | vllm:time_to_first_token_seconds | Histogram | TTFT — how long before the first token streams | p95 above your interactive bar (300 ms here) | | vllm:inter_token_latency_seconds | Histogram | Streaming speed after the first token | Above ~50 ms is slower than reading speed | | vllm:e2e_request_latency_seconds | Histogram | Total request time | Rising while TTFT is flat = decode or workload change | | vllm:request_queue_time_seconds | Histogram | Time waiting for admission | Earliest saturation signal — any sustained rise | | vllm:request_prefill_time_seconds | Histogram | Time processing the prompt | Dominant share = prefill-bound workload | | vllm:request_decode_time_seconds | Histogram | Time generating tokens | Dominant share = memory-bandwidth-bound | | vllm:num_requests_running | Gauge | Requests currently being decoded | Batch occupancy | | vllm:num_requests_waiting | Gauge | Requests queued for admission | Sustained non-zero = add a replica | | vllm:kv_cache_usage_perc | Gauge | Occupancy of the KV block pool — not VRAM | Autoscaling trigger (~60%) | | vllm:prompt_tokens_total + vllm:prompt_tokens_cached_total | Counters | Prefix-cache hit rate | A drop means routing scattered your prefixes | | vllm:generation_tokens_total | Counter | Output throughput in tokens/sec | Headline throughput number | | vllm:request_prompt_tokens + vllm:request_generation_tokens | Histograms | Per-request token sizes; their ratio is the workload's shape | A moving ratio means the workload changed character | | vllm:iteration_tokens_total | Histogram | Tokens advanced per forward pass | Near 1.0 with concurrency = batching broken | | vllm:request_success_total{finished_reason} | Counter | Completion outcomes | error/abort = SLO; length share = truncation | | http_requests_total{status} | Counter | Server-level requests | Catches 4xx and malformed requests vllm:* never sees | | DCGM_FI_DEV_FB_USED / _FB_FREE | Gauge | Physical VRAM | Capacity planning only — never alert on it | | DCGM_FI_DEV_GPU_UTIL | Gauge | "A kernel is resident" | Not a measure of useful work | | DCGM_FI_PROF_PIPE_TENSOR_ACTIVE | Gauge | Tensor-core activity | Low here + high DRAM = memory-bound | | DCGM_FI_PROF_DRAM_ACTIVE | Gauge | Memory-bandwidth activity | High = the bottleneck is bandwidth | | DCGM_FI_DEV_POWER_USAGE | Gauge | Watts drawn | Pairs with throughput for tokens-per-watt |

Step 1: Where does vLLM latency go? TTFT, prefill, and decode decomposed

Decompose total latency into queue, prefill, and decode before optimizing anything. vLLM reports all three separately, and they have completely different fixes. This is the single most valuable chart in the setup.

The query averages each phase's cumulative time by request count in the same window — in Prometheus terms, rate(vllm:request_prefill_time_seconds_sum) / rate(vllm:e2e_request_latency_seconds_count), and the same for queue and decode:

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS reqs = SUM(RATE(`metrics.vllm:e2e_request_latency_seconds_count`)),
        q_s  = SUM(RATE(`metrics.vllm:request_queue_time_seconds_sum`)),
        pf_s = SUM(RATE(`metrics.vllm:request_prefill_time_seconds_sum`)),
        dc_s = SUM(RATE(`metrics.vllm:request_decode_time_seconds_sum`))
    BY minute = BUCKET(@timestamp, 1 minute)
| EVAL queue_ms   = ROUND(q_s  / reqs * 1000, 2),
       prefill_ms = ROUND(pf_s / reqs * 1000, 1),
       decode_ms  = ROUND(dc_s / reqs * 1000, 1)
| KEEP minute, queue_ms, prefill_ms, decode_ms
| SORT minute ASC

At 8 concurrent requests on the A10G:

minute    | queue_ms | prefill_ms | decode_ms | e2e_ms  | ttft_ms | inter_token_ms
22:55:00  | 0.01     | 37.98      | 1664.69   | 1715.18 | 50.81   | 16.23
22:56:00  | 0.01     | 40.99      | 1670.44   | 1723.78 | 53.50   | 16.20

Read it against the objectives:

• TTFT is 51 ms against a 300 ms target. Passing, with almost 6× headroom. Perceived responsiveness is not the problem, whatever was said in the meeting.
• Inter-token latency is 16 ms — about 62 tokens/sec against a 50 ms / 20 tok-s bar. Text arrives roughly three times faster than a person reads it.
• Queue time is 0.01 ms. Nothing is waiting for admission; the engine has capacity to spare at this concurrency.
• Decode is 1,665 ms against 38 ms of prefill — 97% of the time is decode.

That last line is the finding. Every optimization aimed at prefill is worthless for this workload. Chunked prefill, prompt compression, a faster attention kernel for long contexts — all real techniques, all irrelevant when prefill is 2% of the time. Decode is memory-bandwidth-bound, so the levers that would actually move it are quantization, tensor parallelism across two cards, or a smaller model. A single chart eliminated the wrong shopping list.

Watch vllm:request_queue_time_seconds specifically. It is the earliest saturation signal in the entire vLLM metric set — queue time climbs before vllm:num_requests_waiting becomes visibly non-zero, because a request can wait milliseconds for admission without ever registering as queued at scrape time. If you alert on one thing from this section, alert on queue time crossing a small absolute threshold.

Step 2: Is vLLM using the GPU efficiently? KV cache, prefix caching, and batch occupancy

Four metrics answer this: prefix-cache hit rate, tokens per iteration, KV-cache occupancy, and running-vs-waiting requests. Latency tells you the experience is good; these tell you whether you're overpaying for it.

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS ptok    = SUM(RATE(`metrics.vllm:prompt_tokens_total`)),
        cached  = SUM(RATE(`metrics.vllm:prompt_tokens_cached_total`)),
        gen     = SUM(RATE(`metrics.vllm:generation_tokens_total`)),
        it_s    = SUM(RATE(`metrics.vllm:iteration_tokens_total_sum`)),
        it_c    = SUM(RATE(`metrics.vllm:iteration_tokens_total_count`)),
        running = MAX(`metrics.vllm:num_requests_running`),
        waiting = MAX(`metrics.vllm:num_requests_waiting`),
        kv      = MAX(`metrics.vllm:kv_cache_usage_perc`)
    BY minute = BUCKET(@timestamp, 1 minute)
| EVAL prefix_cache_hit_pct = ROUND(cached / ptok * 100, 1),
       tokens_per_iteration = ROUND(it_s / it_c, 2),
       gen_tokens_per_sec   = ROUND(gen, 1),
       kv_cache_pct         = ROUND(kv * 100, 3)
| KEEP minute, prefix_cache_hit_pct, tokens_per_iteration,
       gen_tokens_per_sec, running, waiting, kv_cache_pct
| SORT minute ASC

minute   | prefix_cache_hit_pct | tokens_per_iteration | gen_tok/s | running | waiting | kv_cache_pct
22:55:00 | 32.5                 | 10.36                | 479.1     | 8.0     | 0.0     | 0.255
22:59:00 | 32.3                 | 10.34                | 480.0     | 8.0     | 0.0     | 0.121

A 32% prefix-cache hit rate is a third of all prefill work simply not done. The claims team's requests share a system prompt and policy boilerplate, and vLLM's automatic prefix caching recognizes that. This is a direct argument for raising prompt standardization: the more the application puts shared context in a consistent leading position, the higher this climbs and the cheaper every request gets. It is also the number that will crater the day a naive round-robin load balancer sits in front of two replicas — precisely the condition that justifies llm-d's cache-aware routing.

tokens_per_iteration ≈ 10.3 with 8 concurrent requests is continuous batching working correctly. Each forward pass through the model advances about ten sequences at once. If this sat near 1.0 with multiple requests in flight, batching would be broken and you'd be paying full model-forward cost per token per user. This metric proves you're getting vLLM's core value.

What does vllm:kv_cache_usage_perc actually measure?

vllm:kv_cache_usage_perc reports occupancy of vLLM's pre-allocated KV block pool — not physical GPU memory. At startup, vLLM reserves a fraction of VRAM (governed by --gpu-memory-utilization, default 0.9) and carves a KV block pool out of that reservation. This gauge reports how full that pool is.

That's why it read 0.25% here. Eight concurrent requests holding ~150 tokens each barely touch an A10G's block budget. The pool is large, and correctly so. Push the same server to 32 concurrent requests with 512–1,024 token generations and it moves — to about 2.8%. Still small.

The instinct is to read a number that low as "the cache is broken" or "I've massively over-provisioned." Both are wrong. Treating this gauge as a VRAM proxy is the most common self-hosted vLLM configuration error I see. Step 6 shows exactly how far apart the two are.

Step 3: What shape is your vLLM inference workload?

Confirm the workload is what you think it is before tuning anything. This is the panel that explains a latency "regression" that isn't your fault.

minute   | requests_per_min | avg_prompt_tokens | avg_generated_tokens | avg_max_tokens | gen_to_prompt_ratio
22:55:00 | 282.2            | 49.6              | 103.6                | 103.6          | 2.09
23:01:00 | 276.0            | 50.6              | 103.0                | 103.0          | 2.04

The generation-to-prompt ratio is 2.04 — this workload writes twice as much as it reads. That single ratio is the explanation for Step 1's 97%-decode finding, and it holds for any summarize-and-draft use case. If the team later adds a long-document RAG feature, prompts jump to thousands of tokens, the ratio inverts, the workload becomes prefill-bound, and the correct tuning changes completely. Watching this ratio is how you learn your workload changed character before someone files a ticket.

Now the column that should bother you: avg_generated_tokens equals avg_max_tokens exactly. Every request is stopping because it hit its token ceiling, not because the model finished its thought. The screenshot shows the same pattern holding as generation lengths scale to ~685 tokens against a ~777 ceiling.

In a load test that's an artifact of the generator. In production, that number is users getting cut off mid-sentence — and it is invisible in every latency metric you have. Which brings us to the metric that catches it.

Step 4: Are vLLM requests actually succeeding? Checking finished_reason

Break vllm:request_success_total down by its finished_reason label. This is the closest thing self-hosted inference has to an application-level SLI, and it catches a failure mode no latency chart can.

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS completions_per_min = ROUND(SUM(RATE(`metrics.vllm:request_success_total`)) * 60, 2)
    BY minute = BUCKET(@timestamp, 1 minute), finish_reason = labels.finished_reason
| SORT minute ASC, finish_reason

Five outcomes, each meaning something different operationally:

| finished_reason | What it means | What to do about it | |---|---|---| | stop | The model finished naturally | This is the number you want large | | length | Truncated at max_tokens | High share means users are cut off — raise the ceiling, or shorten the ask | | abort | The client disconnected first | Users giving up, or a proxy timeout shorter than your generations | | error | The engine failed | Your hard SLO signal. Should be flat zero | | repetition | Degenerate looping output | A sampling-parameter problem, not an infrastructure one |

Under the small load generator the split was 100% length — expected, since it requested a fixed ceiling. In the screenshot, at a mixed load, stop and length run side by side at roughly 51 and 32 completions/min. That mix is the healthy shape: most requests finishing on their own, a minority hitting the ceiling.

The lesson generalizes. error and abort are what you page on. But the stop-to-length ratio is what you review weekly, because drift toward length means answers are being truncated and no latency dashboard on earth will tell you.

One blind spot to close: vllm:* metrics only count requests the engine accepted. Malformed JSON, 4xx, auth failures and dropped connections never reach it. Those live in http_requests_total with status and handler labels — worth a panel beside this one, because "the model is broken" reports frequently turn out to be the gateway in front of it.

Step 5: What NVIDIA DCGM metrics say the GPU is actually doing

Use NVIDIA DCGM as an independent witness to vLLM's own account. Everything so far is the engine describing itself; DCGM describes the silicon.

FROM metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS gpu_util_pct    = MAX(`metrics.DCGM_FI_DEV_GPU_UTIL`),
        mem_bw_util_pct = MAX(`metrics.DCGM_FI_DEV_MEM_COPY_UTIL`),
        vram_used_mib   = MAX(`metrics.DCGM_FI_DEV_FB_USED`),
        vram_free_mib   = MIN(`metrics.DCGM_FI_DEV_FB_FREE`),
        power_w         = ROUND(MAX(`metrics.DCGM_FI_DEV_POWER_USAGE`), 1),
        temp_c          = MAX(`metrics.DCGM_FI_DEV_GPU_TEMP`)
    BY minute = BUCKET(@timestamp, 1 minute)
| SORT minute ASC

→ 100% GPU util · 21,483 MiB used / 1,352 MiB free · 240 W · 73 °C

100% utilization and 94% VRAM. Under the conventional reading, this card is maxed out and it's time to ask for more hardware. That reading is wrong.

Why is GPU utilization a misleading metric for LLM inference?

DCGM_FI_DEV_GPU_UTIL means "a kernel is resident on the device," not "the device is doing useful work." It reads 100% for a perfectly-tuned server and 100% for a badly-tuned one, so it cannot distinguish them. The DCGM profiling counters can:

gr_engine_active 99.8%  ·  tensor_active 16.6%  ·  dram_active 80.4%

Read those three together. The GPU's compute engine is busy essentially all the time — but its tensor cores, the units that do the actual matrix math, are active only 16.6%, while DRAM is active 80.4%. The card is not computing. It is waiting on memory.

This is independent, hardware-level confirmation of what Step 1 inferred purely from timings: decode is memory-bandwidth-bound. Two entirely different instruments, two different layers of the stack, one conclusion — the difference between a hypothesis and a finding.

It also permanently retires GPU utilization as a capacity metric for LLM inference. If your GPU capacity planning rests on DCGM_FI_DEV_GPU_UTIL — and most does — it rests on nothing.

Step 6: vLLM KV cache vs GPU VRAM, and why they disagree

Put vllm:kv_cache_usage_perc and physical VRAM usage on one 0–100% axis. They describe the same GPU memory, they sit at opposite ends of the chart, and both are correct.

minute   | running | gen_tok_s | kv_cache_pct | vram_used_pct | gpu_util | dram_active_pct | tensor_active_pct | tokens_per_watt
00:05:00 | 31      | 1627.5    | 2.82         | 94.1          | 100.0    | 80.4            | 16.6              | 6.80

KV cache at 2.8%. VRAM at 94.1%.

vLLM pre-allocates a large fraction of VRAM at startup — governed by --gpu-memory-utilization, default 0.9 — and carves its KV block pool out of that reservation. vllm:kv_cache_usage_perc reports occupancy of the pool. DCGM reports what the driver sees, which is the whole reservation, whether or not it's holding anything.

The operational consequences are precise, and they're the practical payoff of the entire exercise:

• Autoscale on vllm:kv_cache_usage_perc and vllm:num_requests_waiting. These describe admission capacity — whether the engine can take another request right now.
• Capacity-plan on VRAM. This describes physical space — whether a second model could ever fit on this card. (It can't. 1.3 GB free.)
• Never alert on VRAM. It will page you at 3 a.m. for a healthy, mostly-idle server, every single night, forever.

And tokens_per_watt — generated tokens divided by power draw, 6.8 here — is a genuine cost-efficiency metric. It's comparable across GPU models, batch settings and quantization levels in a way that neither latency nor utilization is. When you go back to Finance for card number two, this is the number that makes the argument: at 32 concurrent we sustain 1,627 tokens/sec at 240 watts, and here's what that becomes on an L40S.

vLLM tuning decisions: what the SRE does with these Prometheus metrics

Six steps, thirty minutes, one server. The verdict against the stated objectives:

| Objective | Measured | Verdict | |---|---|---| | TTFT p95 < 300 ms | 51 ms | Pass, 6× headroom | | Inter-token < 50 ms | 16 ms (≈62 tok/s) | Pass | | Zero queueing at 12 concurrent | queue_ms 0.01, waiting 0 at 8; still 0 at 32 | Pass, large margin | | Error + abort < 0.5% | 0% | Pass |

The configuration is correct for this department, and the department is over-provisioned rather than under-provisioned. That's a defensible, evidence-backed answer to "it feels slow" — and it redirects the investigation to the app, the gateway, or the prompt, which is where the problem actually is.

The concrete follow-ups, each tied to a metric rather than a hunch:

1. Stop optimizing prefill. — vllm:request_decode_time_seconds vs vllm:request_prefill_time_seconds. Decode is 97% of the time against prefill's 2%, confirmed twice. Chunked prefill and prompt compression are off the table for this workload.
2. If more throughput is needed, quantize before buying hardware. — DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (16.6%) vs DCGM_FI_PROF_DRAM_ACTIVE (80.4%). The bottleneck is memory bandwidth, not compute, so an FP8 or AWQ build of the same model is the highest-leverage single change: it moves fewer bytes per token, which is exactly the constrained resource.
3. Raise the client-side max_tokens ceiling. — vllm:request_success_total{finished_reason}. Every request finishing on length rather than stop is a user getting cut off mid-answer. This is the one finding that's a live user-experience defect. You need to increase the prompt max_token limit.
4. Standardize the prompt prefix. — vllm:prompt_tokens_cached_total over vllm:prompt_tokens_total, 32% today. But it should be better (more like 70%) More shared boilerplate in a consistent leading position raises it, and it's free.
5. Set the autoscaling trigger now, before it's needed. — vllm:kv_cache_usage_perc and vllm:num_requests_waiting. Scale when the first crosses ~60% or the second stays above zero. Do not scale on DCGM_FI_DEV_GPU_UTIL — it's pinned at 100% regardless.
6. Alert on queue time, not on VRAM. — vllm:request_queue_time_seconds is the earliest true saturation signal; DCGM_FI_DEV_FB_USED is a constant that looks like an emergency.
7. Revisit when the workload changes shape. — vllm:request_generation_tokens over vllm:request_prompt_tokens, 2.04 today. When the RAG feature ships that ratio inverts, the workload becomes prefill-bound, and half of this analysis needs redoing. The chart tells you the day it happens.

Notice that most of the metrics are looking at the vLLM metrics not the GPU metrics in helping optimize. These are still within the limit of a single service, but when you get vllm:num_requests_waiting to persistently non-zero, then you need to run KServe or you can use KEDA and HPA autoscaling. But you get the metrics to help you determine or allow KServe to scale. So you can see that understanding these metrics are crucial to tuning the inference service.

Elastic Observability can provide this to you.

Why self-hosted LLM tuning is an SRE problem, not an ML problem

Self-hosting an open-weight model is not primarily an ML problem. It is a capacity and saturation problem — something SREs have been extremely good at for twenty years. The blocker was never skill. It was that the telemetry sat unexamined on a /metrics endpoint nobody scraped, in a schema nobody had mapped to the questions they actually had.

Once it's collected, the reasoning is familiar work in unfamiliar clothes:

• Decompose latency by phase before optimizing anything (queue / prefill / decode).
• Distinguish the logical resource from the physical one (KV block pool ≠ VRAM), and know which each metric describes.
• Never trust a single-source utilization number — corroborate the engine's account with the hardware's.
• Tie every knob to a metric and every metric to a stated objective, so tuning converges instead of wandering.

For a team that isn't allowed to send its data anywhere, that difference — between running a model and operating one — is the whole ballgame. The department gets a capability it's otherwise locked out of, and the SRE gets to answer questions about it with numbers.

Frequently asked questions

What does vllm:kv_cache_usage_perc measure? It measures occupancy of vLLM's pre-allocated KV block pool, not physical GPU memory. vLLM reserves a fraction of VRAM at startup (--gpu-memory-utilization, default 0.9) and carves the KV pool from that reservation. In this deployment it read 2.8% while DCGM reported 94.1% VRAM used on the same card at the same moment. Use it as an autoscaling signal; use VRAM for capacity planning.

Why is my vLLM deployment decode-bound? Because the workload generates more tokens than it reads. Compare vllm:request_decode_time_seconds against vllm:request_prefill_time_seconds, and check the generation-to-prompt token ratio. In this deployment the ratio was 2.04 — twice as many output tokens as input — which produced 1,665 ms of decode against 38 ms of prefill. Decode is memory-bandwidth-bound, so quantization, tensor parallelism, or a smaller model help; prefill optimizations do not.

Should I autoscale vLLM on GPU utilization? No. DCGM_FI_DEV_GPU_UTIL means a kernel is resident on the device, not that the device is doing useful work — it reads 100% for both a well-tuned and a badly-tuned server. Autoscale on vllm:kv_cache_usage_perc (around 60%) or on vllm:num_requests_waiting staying above zero, since those describe whether the engine can admit another request.

Why does my GPU show 100% utilization when it isn't fully used? Because GPU utilization only reports kernel residency. Check the DCGM profiling counters instead: in this deployment DCGM_FI_PROF_GR_ENGINE_ACTIVE was 99.8% while DCGM_FI_PROF_PIPE_TENSOR_ACTIVE was only 16.6% and DCGM_FI_PROF_DRAM_ACTIVE was 80.4%. That combination means the GPU is waiting on memory bandwidth rather than computing.

How do I get vLLM metrics into Prometheus? vLLM already exposes Prometheus exposition format on /metrics at its serving port — no adapter or instrumentation needed. Point a Prometheus scrape job at the vLLM Service, add a second job for dcgm-exporter on :9400, and use remote_write to ship to long-term storage. Sending through an OpenTelemetry Collector also works but normalizes the metric names, which makes them harder to match against vLLM documentation.

What TTFT should I target for an interactive LLM application? For a streaming chat-style interface, a p95 time-to-first-token under 300 ms feels immediate, and inter-token latency under 50 ms (about 20 tokens/sec) outpaces reading speed. This deployment measured 51 ms TTFT and 16 ms inter-token latency on a 3B model on a single NVIDIA A10G, leaving roughly 6× headroom.

Why are all my vLLM requests finishing with length? Because they're hitting the max_tokens ceiling instead of the model choosing to stop. Break vllm:request_success_total down by its finished_reason label: a high length share means answers are being truncated mid-sentence. This is invisible in every latency metric, so review the stop-to-length ratio regularly and raise the client-side ceiling if it drifts.

When should I move from a plain vLLM Deployment to KServe or llm-d? Move to KServe when vllm:num_requests_waiting is persistently non-zero at peak and you need replicas to appear without human intervention. Move to llm-d when your prefix-cache hit rate collapses across replicas — a sign that load balancing scattered conversations that shared a prefix — or when prefill time starts stealing measurably from decode.

What you need

• An Observability serverless project, or an Elastic Cloud Hosted or self-managed stack at version 9.4 or later. PromQL is generally available in Elastic Cloud Serverless and Elastic Stack 9.5, and available as a technical preview in Elastic Stack 9.4.
• Kubernetes state and container memory metrics in Elasticsearch.

What is the alert telling us?

This is the alert that opened the investigation:

It comes from this waiting-reason query:

PROMQL
  max by (namespace, pod, container) (
    max_over_time(
      kube_pod_container_status_waiting_reason{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api",
        reason="CrashLoopBackOff"
      }[2m]
    )
  ) == 1

A result of 1 means Kubernetes is delaying another start because the container has failed repeatedly. That is the correct paging signal here because the checkout path has a single replica: when that replica restarts, requests fail.

The max_over_time(...[2m]) range keeps the alert tied to recent samples. Without it, the last observed value of 1 can outlive the pod, and the rule keeps matching after that pod is gone.

That PromQL query ran every minute over a two-minute window and created an alert after one matching run.

Why did the last container stop?

The alert shows what Kubernetes is doing now. It does not show how the previous container ended:

PROMQL
  max by (namespace, pod, container) (
    kube_pod_container_status_last_terminated_reason{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      reason="OOMKilled"
    }
  ) == 1

A result of 1 for the same namespace, pod, and container means the last recorded exit was out of memory. Kube-state-metrics keeps that last reason as a gauge, so the value can stay visible after recovery. It points the investigation at memory; it does not prove that every restart in the window was an OOM kill.

Is the failure repeating?

A single restart can still be transient. The restart counter shows whether the failure keeps happening:

PROMQL
  max by (namespace, pod, container) (
    increase(
      kube_pod_container_status_restarts_total{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api"
      }[10m]
    )
  )

increase() shows how much the restart counter rose over the selected range. Repeated increases during the incident window explain why Kubernetes entered backoff.

How close is memory to the limit?

We need to know how close the container is to its memory limit, and whether that gap collapses right before each restart. This deployment allows only 128MiB, so the next query divides working-set memory by that configured limit:

PROMQL
  max by (namespace, pod, container) (
    container_memory_working_set_bytes{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      image!=""
    }
  )
  /
  max by (namespace, pod, container) (
    container_spec_memory_limit_bytes{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      image!=""
    }
  )

The chart shows a repeating sawtooth: memory approaches 90% of the limit, drops when the process stops, and climbs again after each restart.

Working set is the better signal here than total usage. Total usage includes reclaimable file cache, so it can sit near the limit without a kill. Working set is closer to the memory that triggers OOMKilled for this workload.

Is the node under memory pressure?

OOMKilled can mean the container hit its own limit, or the node ran low on memory and Kubernetes started reclaiming. To separate those cases, first find which node runs the pod, then check whether that node (or any peer) reported MemoryPressure.

PROMQL
  max by (namespace, pod, node) (
    kube_pod_info{
      namespace="checkout",
      pod=~"checkout-api-.*"
    }
  ) == 1

The pod sits on ip-10-0-2-18.ec2.internal. That is the node whose MemoryPressure result matters most for this incident:

PROMQL
  max by (node) (
    max_over_time(
      kube_node_status_condition{
        condition="MemoryPressure",
        status="true"
      }[30m]
    )
  )

Every node returns 0, including ip-10-0-2-18.ec2.internal. So the host was not under node-wide memory pressure. The kill came from the container limit itself.

Does raising the limit clear the alert?

checkout-api was healthy, then began building an in-memory cache that grows to 200MiB in 10MiB steps. The container only allows 128MiB, so the process is killed with OOMKilled before that cache is fully allocated.

We will raise the memory limit to 512MiB so the 200MiB cache fits with room for the runtime, then check whether CrashLoopBackOff clears:

kubectl set resources deployment/checkout-api -n checkout --limits=memory=512Mi

The same waiting-reason query then stops matching. CrashLoopBackOff drops off:

PROMQL
  max by (namespace, pod, container) (
    max_over_time(
      kube_pod_container_status_waiting_reason{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api",
        reason="CrashLoopBackOff"
      }[2m]
    )
  ) == 1

And the alert that started this investigation? Gone.

That is how you detect and investigate a Kubernetes CrashLoopBackOff with PromQL: from the firing alert, through OOMKilled and the limit mismatch, to a recovered alert. Elasticsearch holds the metrics; Kibana runs the same PromQL queries you already know from Prometheus.

Try it

1. Open an Observability project on Elastic Cloud Serverless, or use Elastic Stack 9.4 or later.
2. In the ES|QL editor in Kibana, run the waiting-reason query against a workload you care about.
3. Follow the same path from that alert to termination reason, restarts, memory versus the limit, and recovery.

For more PromQL in Elastic, see Elasticsearch supports PromQL, PromQL queries in Kibana, and Investigate Kubernetes infrastructure with PromQL.

If you've been an SRE for any length of time, you know what usually comes next. You acknowledge the page, open your observability tool, and start the process: pull up the cluster dashboard, find the namespace, find the pod, check restart counts, pivot to logs, check whether an upstream dependency is degraded, compare against last week, and slowly assemble a picture from a dozen tabs. It works, but it's a process, and the process is where the minutes go.

Elastic's new Kubernetes Experience changes the starting point. When the CrashLoopBackOff alert fires, an Investigation Workflow runs automatically alongside it. By the time you open the alert, the evidence has already been gathered and a root-cause hypothesis is waiting for you. Instead of a blank dashboard, you open the alert to an answer. Or at minimum, a strong starting point that tells you exactly where to look next.

This post walks through a typical CrashLoopBackOff scenario end to end. The sections that follow break down what Elastic's UI shows at each step and why it saves you time.

The manual CrashLoopBackOff investigation: six steps every SRE runs

Here's the shape of a normal CrashLoopBackOff investigation, minus Elastic's workflow:

1. You get paged. Something restarted too many times.
2. You orient. Which pod? Which namespace? Which deployment owns it?
3. You characterize. How many restarts? What was the last termination reason — OOMKilled, a failed liveness probe, a bad exit code?
4. You classify. Is this a memory problem, a config problem, a dependency problem, a scheduling problem?
5. You corroborate. Pull Kubernetes events, read the pod logs, check whether an upstream service started erroring first, compare current behavior against a healthy baseline.
6. You conclude. Only now can you form a hypothesis and act.

Every one of those steps is a query, a click, or a context switch. None of them is hard on its own. Together, on a bad night, they're twenty minutes you don't have and they're twenty minutes of the same steps you ran during the last incident, and the one before that.

The insight behind Elastic's Kubernetes Experience is simple: that sequence is deterministic enough to automate. So Elastic automated it.

Alert to Root cause in minutes

Elastic's Kubernetes integration ships pre-built alert rule templates for states that are wrong by definition; no baseline or warmup required. A pod in CrashLoopBackOff is always a problem, so the rule fires the moment the restart count crosses your configured threshold within a rolling window.

Here's the alert firing in this scenario, both the [Kubernetes OTel] Pod CrashLoopBackOff and OOMKilled containers rules light up for the recommendation group in the otel-demo namespace:

The alert itself is defined by an ES|QL query, so it's transparent and tunable; you can read exactly what triggers it and adjust the threshold to match your environment. And its action is what makes the rest of this post possible: the rule runs the K8s CrashLoopBackOff Investigation (OTel) workflow per alert, the instant a new alert fires. (For a refresher on the alert library and how these templates work, see Part 1.)

In this scenario the rule fires on the recommendation pod (recommendation-788dc88c6c-56w74) in the otel-demo namespace. When you open the alert, Elastic's AI Agent is already summarizing what happened; no rule-details spelunking required:

But the alert firing is only half the story. Attached to it is a Kubernetes Investigation Workflow (technical preview), a directed graph of steps that triggers the instant the alert does. While your phone is still buzzing, the workflow is already querying your cluster, branching on what it finds, and synthesizing an answer.

Inside the investigation: what the workflow actually does

The workflow mirrors the exact sequence an experienced SRE would run by hand, except it runs in seconds and writes nothing down that it can't back up with evidence. For our CrashLoopBackOff scenario, here's the path it takes.

Step 1: What does the workflow check first on the crashing pod?

The workflow queries Kubernetes metrics for the restart count, the last termination reason, and utilization against the pod's declared limits.

Result: last termination reason OOMKilled, restart count 7. Memory utilization data happened to be unavailable at query time, but the OOMKilled reason is definitive: the container is being killed by the kernel for exceeding its memory limit on each startup, then immediately restarting.

The OOMKilled termination reason determines everything that follows. The workflow branches down the memory-investigation path rather than the log-investigation path it would take for a non-memory crash.

Step 2: How does the workflow tell a memory leak from a load spike?

The ML anomaly check is the step that separates a good investigation from a fast-but-wrong one. OOMKilled does not automatically mean "memory leak." Rather than recompute memory trends from scratch, the workflow queries the ML anomaly index for an active k8s_pod_memory_growth anomaly on this pod.

Result: no anomaly. The memory spike is flagged as load-driven, not a suspected leak. The ML baseline — established over the preceding days — didn't see the slow, creeping growth trajectory that characterizes a leak. It saw a jump consistent with real traffic.

Distinguishing a load-driven memory spike from a genuine leak would take a human several minutes and a good deal of judgment. The workflow reaches it because Part 1's anomaly detection jobs were already learning the workload's baseline in the background.

Step 3: Is the failure spreading to other Kubernetes services?

A crashing pod is often a symptom, not a cause, and it can also cause problems downstream. So the workflow enumerates the pod's dependencies from APM service_destination aggregates and compares the current error rate and latency against baseline. An AI classification step decides whether the failure is spreading.

Result: the sole direct caller is the frontend service, which is absorbing the impact with just a 0.14% error rate against the recommendation service and no other service exceeds its degradation threshold relative to baseline. The blast radius is isolated to the recommendation service; there's no significant downstream cascade. The problem is local to this pod.

Step 4: Did a recent change in the namespace cause the crash loop?

Finally, the workflow scans the namespace event log. It finds a continuous Pulled → Created → Started → Killing → BackOff cycle running from roughly 18:51 to 18:54 UTC, the textbook signature of an active crash loop at the time the alert fired. Nothing changed operationally; this is a steady-state resource problem.

What the CrashLoopBackOff root-cause hypothesis looks like

When you open the alert, this is what greets you:

ROOT CAUSE HYPOTHESIS (confidence: high)

The recommendation service pod (recommendation-788dc88c6c-56w74) is in a
crash-loop caused by repeated OOMKilled terminations. The pod has restarted
7 times and Kubernetes events confirm a continuous BackOff/restart cycle
since at least 18:51 UTC. Memory utilization data was unavailable at query
time, but the OOMKilled termination reason is definitive: the container is
exceeding its configured memory limit on each startup, being killed by the
kernel, and immediately restarting. No memory leak was detected by ML
anomaly analysis, indicating the memory pressure is load-driven — the
container's memory limit is simply insufficient for the current request
volume. The frontend service (the sole direct caller) is absorbing the
impact with a 0.14% error rate on the recommendation service itself, but
no significant downstream cascade is observed.

EVIDENCE
- Pod restarted 7 times; last termination reason: OOMKilled — container is
  consistently exceeding its memory limit
- ML memory anomaly check: no anomaly found; memory spike assessed as
  load-driven, not a leak
- Blast radius is isolated to the recommendation service; no other service
  exceeds degradation thresholds relative to baseline
- Continuous BackOff events from 18:51–18:54 UTC confirm active crash-loop
  at alert time

PROBABLE CAUSE: The recommendation container's memory limit is too low for
current traffic load, causing repeated OOMKilled terminations and a
crash-loop backoff.

RECOMMENDED NEXT STEPS
1. Immediately increase the memory limit (and request) for the
   recommendation container in its Deployment spec to provide headroom
   above the observed peak usage, then redeploy to break the crash-loop.
2. Profile the recommendation service under representative load to
   determine the actual memory working set and set a right-sized limit
   with a safe buffer (e.g., 20–30% above peak observed).
3. Add a Kubernetes HorizontalPodAutoscaler or VPA policy for the
   recommendation service so memory resources scale with traffic rather
   than requiring manual intervention.

Read that again from the perspective of the on-call engineer. You were paged. You opened the alert. And the alert didn't hand you a pile of logs and a dashboard; it handed you a calibrated root-cause hypothesis with the evidence attached and the next actions spelled out. The six manual steps from the "old way" are done. Your job is now to decide, not to dig.

That's the time savings: not shaving a few seconds off each query, but removing the entire investigative scavenger hunt from the critical path.

How does Elastic Observability avoid misdiagnosing OOMKilled as a memory leak?

Speed is worthless if the answer is wrong, so it's worth noting how the workflow avoids the classic misdiagnoses. It encodes the reasoning an experienced SRE applies instinctively:

• OOMKilled is not automatically a leak. It compares against a 7-day baseline before ever claiming one. Here, that check is what turned "the app is leaking memory" into the correct "the limit is undersized for real load."
• Co-symptoms are not causes. It explicitly checks whether the upstream degraded first before blaming or clearing it.
• Absence of evidence is not evidence. If a query returns zero rows, it reports "no data available" rather than inventing a failure mode.
• It's honest about confidence. The hypothesis is labeled high, medium, or low. When two failure modes fit the evidence, the workflow names both and says which it believes is causal and why. Manufacturing false confidence is treated as a failure of the investigation itself.

This is the same diagnostic protocol encoded in Elastic's observability-k8s-investigation Skill, a failure-mode taxonomy covering 16 distinct Kubernetes failure patterns, from OOMKilled and CPU throttling through scheduling and networking issues. (More on that in Part 2.)

Still want to double-check? Dashboards, Discover, and APM

The workflow gives you the answer. But a good root-cause tool should also make it trivial to verify that answer because sometimes you want to see it with your own eyes, and sometimes the workflow returns medium confidence and you need to close the gap yourself. Everything the workflow reasoned over is available to you directly.

Dashboards — confirm the restart cascade visually. The Kubernetes dashboards are built for drill-down. Start at the cluster Overview, where "top namespaces by container restarts" surfaces the problem at a glance. Click into the flagged namespace, then the pod driving the restarts. The Pods view flags container restarts on the recommendation pod and plots memory against requests and limits over time — you'll see the working set pressing against the limit exactly as the workflow described. It's roughly four clicks from cluster to container.

Discover — read the raw evidence. The pod detail dashboard links directly to correlated pod logs and events in Discover. Here you can confirm the OOMKilled events and the restart cadence in the raw log and event stream, and run your own ES|QL queries if you want to slice the data differently.

APM — verify the blast radius is really contained. The workflow found the impact isolated to the recommendation service, with frontend (the sole caller) absorbing it at a 0.14% error rate. You can confirm that independently in the APM UI: open the service map, check the caller's latency and error rate over the incident window, and compare against the weekly baseline yourself.

The point isn't that you have to do any of this; it's that the workflow's conclusion is fully auditable. Fast when you trust it, transparent when you want to check.

The same investigation from your IDE: the MCP App

Not every investigation starts from a Kibana alert. Sometimes a developer just asks, "why is this service crashing?" from their editor. Elastic's Observability MCP App (technical preview) exposes the same telemetry (and the same investigation workflow) as AI-callable tools that render interactive views inline in your chat or IDE, no context switch to Kibana required.

For our CrashLoopBackOff scenario, the flow looks like this from an MCP-compatible client such as Claude Desktop or VS Code:

"What's broken?" → the cluster health rollup returns an overall health badge, degraded services, top memory consumers, and a Kubernetes breakdown (CPU, memory, restarts, nodes) in one inline view. Here it flags a critical cluster with payment, cart, frontend, and frontend-proxy degraded.

"Is anything anomalous in the recommendation pod?" → the memory analysis view confirms the spike is load-driven, not a leak — the crashing 788dc88c6c pods report null memory (they die too fast to emit a sample) while the healthy pods sit flat at 45MB, exactly the ML result the workflow used.

"Why is the recommendation service crashing?" → the agent returns the same structured root-cause reasoning you'd see on the alert, rendered inline: memory limit set below what the container needs to boot, the ReplicaSet has been intermittently OOMing for weeks, and a concrete mitigation (roll back, then fix the limit), all without leaving the editor.

Same evidence, same root cause, delivered wherever you happen to be working. (For the full set of MCP App views and architecture, see Part 2.)

From paged to decided: skipping the Kubernetes investigation entirely

The change here is not "a better dashboard." It's a shift in what you do when you get paged.

Before, the alert was the start of the investigation. You were notified, and then you went and found the answer. Now, the alert arrives with the investigation already run: evidence gathered, dead ends eliminated, a calibrated hypothesis, and next steps in hand. You go straight from "notified" to "deciding," and you keep the full trail of dashboards, Discover, and APM for whenever you want to verify or dig deeper.

For a single incident that's a few minutes saved. Across a quarter of on-call rotations, across every engineer who no longer re-runs the same six steps at 3 a.m., it's real time back and a lot less alert fatigue.

Try it yourself

You don't need a production incident to see this. The OpenTelemetry Astronomy Shop demo environment ships with a feature-flag service that lets you trigger failure scenarios on demand. Enable a cart/checkout failure, watch the restart cascade unfold, and the CrashLoopBackOff alert rule fires with the investigation workflow running right behind it.

To get set up:

1. Install the Kubernetes integration — dashboards are available immediately. (Part 1: Getting started)
2. Deploy data collection via the EDOT Collector (OpenTelemetry) or standalone Elastic Agent, both Helm-based.
3. Enable the alert rule templates in Observability > Alerts, including CrashLoopBackOff, and connect your notification channel.
4. Let the ML modules warm up for 24–48 hours so anomaly baselines are ready when you need them.
5. Enable the Investigation Workflow (technical preview) — import the Kubernetes Crashloop Investigation Workflow from the Workflows page and configure it to trigger on the alert. (Part 2: Getting started)
6. Install the MCP App (technical preview) on your favorite agentic client to bring investigations into your IDE.

Running Kubernetes on Elastic today? Tell us which investigation steps you still repeat by hand on every incident, and which remediations you'd trust a workflow to propose. Join the Elastic Community Discussion.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Availability

This is available in Elastic Observability serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

Prerequisites for linking APM services to Kubernetes infrastructure

You need application traces and Kubernetes infrastructure metrics in the same Elastic Observability project.

• Application instrumentation: EDOT-instrumented services sending traces via the EDOT Collector or an upstream OpenTelemetry Collector with both the elasticapm connector and the elasticapm processor. The EDOT Collector includes both by default; for a custom upstream pipeline, see the upstream collector setup.
• Kubernetes observation: the cluster observed via OpenTelemetry with host and Kubernetes metrics from the EDOT Collector. See the Kubernetes quickstarts and Kubernetes observability with EDOT for setup.
• Backend: Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

APM alert triage: from notification to problematic pod

Step 1: Confirm the service degradation on the APM alert detail page

The redesigned alert detail page in Elastic Observability shows the impacted service, environment, endpoint and RED metrics in one view. Open it from the alert notification.

You can clearly see which service is impacted, which environment it runs in, what endpoint is being affected and easily look for correlations in their RED metrics. In this case, we can immediately rule out a spike in traffic as the throughput is clearly stable.

Step 2: Rule out service dependencies with the embedded service map

The newly embedded service map preview on the alert detail page shows the health and RED metrics of every dependent service, so you can rule out upstream causes without navigating away. In this case, we have been able to quickly rule out problems with other services causing the symptom with the symptomatic service:

Step 3: Review Kubernetes infrastructure metrics per pod, container and host

After ruling out service dependencies, open the service's updated Infrastructure tab in Elastic Observability to check for infrastructure-level patterns.

Step 4: Compare per-instance metric trends to find the root cause

The Infrastructure tab shows the average metric values over the specified time period. To really understand whether there is a problem with the infrastructure, we need to compare the pod, container and host metrics over time. This allows us to easily spot differences between different entities that may correlate with when the service started showing symptoms. In our example, we can clearly see a difference between some of the metrics between the pods that correlates with when the service symptoms began. So we know there is something going on with the infrastructure that needs investigating:

Summary: from APM alert to root cause in four clicks

In just a few clicks from an alert in Elastic Observability, you can rule out healthy dependent services without leaving the alert detail page, then compare per-pod infrastructure metrics to see which instance correlates with when the symptoms started.

The Datadog Kubernetes dashboard used in this migration

The walkthrough uses Kubernetes - Overview from infra/datadog/dashboards/integrations/kubernetes.json in the migration repository. It is a cluster-wide board with the signals operators check during an incident: pod counts, CPU and memory by host or pod, non-running pods, and containers stuck in CrashLoopBackOff.

Below are representative queries from the source dashboard:

# Pod CPU by host
sum:kubernetes.cpu.usage.total{$scope,$cluster,$label,$node} by {host}

# Pod memory by pod
sum:kubernetes.memory.usage{$scope,$deployment,$statefulset,$replicaset,$daemonset,$cluster,$namespace,!pod_name:no_pod,$label,$service,$node} by {pod_name}

# Pods not running (pressure / scheduling signal)
sum:kubernetes_state.pod.status_phase{$scope,$cluster,$namespace,$deployment,$statefulset,$replicaset,$daemonset,!pod_phase:running,!pod_phase:succeeded,$label,$node,$service} by {kube_cluster_name,kube_namespace,pod_phase}

# CrashLoopBackOff
sum:kubernetes_state.container.status_report.count.waiting{$cluster,$namespace,$deployment,$statefulset,$replicaset,$daemonset,reason:crashloopbackoff,$scope,$daemonset,$label,$node,$service} by {pod_name}

If this board translates cleanly, most production Datadog Kubernetes folders are worth testing with the same workflow.

Why Datadog-to-Elastic migration is faster now

The migration platform automates the query translation and panel rebuilds that used to dominate Datadog moves. Elasticsearch stores Kubernetes metrics efficiently and runs the ES|QL queries those panels use. See Elasticsearch as a metrics backend for benchmark context and storage comparisons.

The platform maps Datadog queries to Kibana panels, validates ES|QL against live data, and writes artifacts you can inspect before anything goes to production.

Prerequisites

You need an Elastic Observability Serverless project, a project API key, and the migration CLI installed from the Observability Migration Platform repository.

Export your endpoints and API key:

export ELASTICSEARCH_ENDPOINT="https://YOUR_ES_ENDPOINT"
export KIBANA_ENDPOINT="https://YOUR_KIBANA_ENDPOINT"
export KEY="YOUR_API_KEY"

Install the CLI and confirm the toolchain:

python3 -m venv .venv
.venv/bin/pip install ".[all]"
.venv/bin/obs-migrate doctor

The doctor command checks compile and lint dependencies. Resolve any errors before you migrate production dashboards. Pin a release tag if you plan to run this in CI.

To pull dashboards from the Datadog API instead of JSON files, copy datadog_creds.env.example to datadog_creds.env and set DD_API_KEY, DD_APP_KEY, and DD_SITE.

Ingest Kubernetes metrics first

Empty panels after upload usually mean Elasticsearch does not yet have the series the Datadog queries reference. Make sure to confirm ingest before you run the migration.

There are two common paths to do so:

1. OpenTelemetry into managed OTLP with Kubernetes receivers (kubeletstats, k8s_cluster), then explore in Discover
2. Existing Prometheus or agent pipelines that already write pod and node metrics to metrics-*

The migration CLI accepts --field-profile otel to map Datadog tags such as pod_name, kube_namespace, and kube_cluster_name to OpenTelemetry fields like kubernetes.pod.name and kubernetes.namespace. If panels are empty after migration, verify field mapping and the selected time range before you change translator settings.

Run the Datadog dashboard migration CLI

Export the Datadog dashboard JSON from the UI, or copy the sample kubernetes.json from infra/datadog/dashboards/integrations/ in the migration repo. Place files in a directory such as ./datadog_k8s_exports/.

Run the migration from that directory:

datadog-migrate \
  --source files \
  --input-dir ./datadog_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --field-profile otel \
  --data-view "metrics-*" \
  --logs-index "logs-*" \
  --upload \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --ensure-data-views \
  --create-alert-rules \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY"

These flags matter for Kubernetes boards:

• --field-profile otel maps Datadog Kubernetes fields to OpenTelemetry field names in Elasticsearch
• --assets all includes dashboards and Datadog monitor definitions when present
• --validate runs emitted ES|QL against your cluster before upload
• --create-alert-rules creates Kibana rules in a disabled state

The unified CLI performs the same work:

obs-migrate migrate \
  --source datadog \
  --input-mode files \
  --input-dir ./datadog_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --field-profile otel \
  --data-view "metrics-*" \
  --logs-index "logs-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload \
  --create-alert-rules

To fetch a dashboard from Datadog directly:

datadog-migrate \
  --source api \
  --env-file datadog_creds.env \
  --dashboard-ids YOUR_DASHBOARD_ID \
  --output-dir ./migration_output \
  --assets all \
  --field-profile otel \
  --data-view "metrics-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload

Validate the migrated Datadog dashboard in Kibana

Open Kibana → Dashboards and locate the migrated Kubernetes - Overview board. Confirm that cluster and namespace pod counts, CPU and memory series, pod phase panels, CrashLoopBackOff widgets, and deployment replica charts return data for your selected time range.

If you migrated monitors, open Observability → Rules. Imported rules remain disabled until you enable them after reviewing thresholds.

The CLI also writes local artifacts under ./migration_output/:

• dashboards/yaml/ contains the translated dashboard definition.
• dashboards/migration_report.json lists panels that translated automatically and panels flagged for manual review.
• alerts/ contains monitor translations when monitors were included in the export.

Handle manual-review panels

Some Datadog widget types do not translate on the first pass. Exotic formulas, log-only panels, and unsupported widgets appear as manual-review entries in the migration report rather than as silently broken charts.

| Result | Recommended action | | --- | --- | | Panel returns data | Accept the translation and continue | | Panel is empty | Confirm metric names and data_stream.dataset values in metrics-*, then widen or shift the time range | | Manual-review marker | Open the original Datadog query and simplify or redesign the panel | | Monitor never fires | Confirm the rule is enabled and thresholds match your environment |

Datadog coverage is narrower than Grafana in some areas. Read the migration report before you commit to full parity with leadership. The platform prefers conservative failures over uploading panels that look correct but query the wrong fields.

Related Datadog and Grafana migration guides

For the Grafana PromQL version of this workflow, see Migrate your Grafana Kubernetes dashboard to Elastic Observability. For platform-level context, see Migrating Datadog and Grafana dashboards and alerts to Kibana. Review known limitations before you migrate every production folder.

The migration tool keeps panel queries in PromQL instead of rewriting them into a new dialect, validates them when you pass --validate, and uploads compiled dashboards to Kibana. You still review the migration report and enable alerts on your schedule.

Sample Grafana Kubernetes dashboard used in this migration

The walkthrough uses Kubernetes / Views / Global, a community-style Grafana dashboard in the migration repository. It includes the signals operators check during an incident: namespace CPU and memory, throttling pressure, and restart counts.

Below are representative PromQL queries from the source dashboard:

# Pod / container CPU by namespace
sum(rate(container_cpu_usage_seconds_total{image!="", cluster="$cluster"}[$__rate_interval])) by (namespace)

# Memory working set by namespace
sum(container_memory_working_set_bytes{image!="", cluster="$cluster"}) by (namespace)

# Node / CPU pressure style signal: throttled seconds
sum(rate(container_cpu_cfs_throttled_seconds_total{image!="", cluster="$cluster"}[$__rate_interval])) by (namespace) > 0

# Container restart counts
sum(increase(kube_pod_container_status_restarts_total{cluster="$cluster"}[$__rate_interval])) by (namespace) > 0

If this board translates cleanly, most production Grafana Kubernetes folders are worth testing with the same workflow.

Why Grafana-to-Elastic migration is faster now

The migration platform automates the query translation and panel rebuilds that used to dominate Grafana moves. Native Prometheus Remote Write and PromQL in Kibana let you keep the dialect your on-call team already uses. The --native-promql flag passes PromQL through unchanged. See Elasticsearch as a metrics engine for storage and query context.

Prerequisites

You need an Elastic Observability Serverless project, a project API key, and the migration CLI installed from the observability-migration-platform repository.

Export your endpoints and API key:

export ELASTICSEARCH_ENDPOINT="https://YOUR_ES_ENDPOINT"
export KIBANA_ENDPOINT="https://YOUR_KIBANA_ENDPOINT"
export KEY="YOUR_API_KEY"

Install the CLI and confirm the toolchain:

python3 -m venv .venv
.venv/bin/pip install ".[all]"
.venv/bin/obs-migrate doctor

The doctor command checks compile and lint dependencies. Resolve any errors before you migrate production dashboards. Pin a release tag if you plan to run this in CI.

How do you get Kubernetes metrics into Elasticsearch?

Empty panels after upload usually mean Elasticsearch does not yet have the series the PromQL references. Make sure to confirm ingest before you run the migration.

There are two common paths to do so:

1. Prometheus Remote Write into Elasticsearch from kube-state-metrics, cAdvisor or kubelet metrics, and node exporters.
2. OpenTelemetry into managed OTLP for Kubernetes receivers, then explore in Discover.

Test with a query such as sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace) against Elastic. If that returns data, continue. If it does not, fix ingest first.

Run the Grafana dashboard migration CLI

Export your Grafana dashboard JSON, or copy the sample k8s-views-global.json from infra/grafana/dashboards/ in the migration repo. Place files in a directory such as ./grafana_k8s_exports/.

Run the migration from that directory:

grafana-migrate \
  --source files \
  --input-dir ./grafana_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --native-promql \
  --data-view "metrics-*" \
  --esql-index "metrics-*" \
  --upload \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --ensure-data-views \
  --create-alert-rules \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY"

These flags matter for Kubernetes boards:

• --native-promql keeps pod CPU, memory, throttling, and restart queries in PromQL
• --assets all includes dashboards and Grafana PromQL alert definitions when present
• --validate runs emitted queries against Elasticsearch before upload
• --create-alert-rules creates Kibana rules in a disabled state

The unified CLI performs the same work:

obs-migrate migrate \
  --source grafana \
  --input-mode files \
  --input-dir ./grafana_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --native-promql \
  --data-view "metrics-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload \
  --create-alert-rules

Validate the migrated Grafana dashboard in Kibana

Open Kibana → Dashboards and locate Kubernetes / Views / Global. Confirm that namespace or pod CPU utilization, memory working set panels, throttling or pressure widgets, and restart charts return data for your selected time range.

If you migrated alerts, open Observability → Rules. Imported rules remain disabled until you enable them after reviewing thresholds.

The CLI also writes local artifacts under ./migration_output/:

• dashboards/yaml/ contains the translated dashboard definition.
• dashboards/migration_report.json lists panels that translated automatically and panels flagged for manual review.
• alerts/ contains alert translations when alert definitions were included in the export.

What do you do when Grafana panels don't migrate automatically?

The Observability Migration Platform flags PromQL expressions that do not translate automatically. Hard joins, unusual arithmetic, and a few Alertmanager-era edge cases appear as manual-review entries in the migration report rather than as silently broken charts.

| Result | Recommended action | | --- | --- | | Panel returns data | Accept the translation and continue | | Panel is empty | Confirm metric names exist in metrics-*, then widen or shift the time range | | Manual-review marker | Open the original PromQL and simplify or redesign the panel | | Alert never fires | Confirm the rule is enabled and thresholds match your environment |

This path migrates Grafana PromQL dashboards and Grafana unified PromQL alert definitions into Kibana. It does not ingest a raw alertmanager.yml. The goal is to keep the PromQL your pager already trusts instead of rebuilding the Kubernetes board from zero.

Related guides

For platform-level context, see Migrating Datadog and Grafana dashboards and alerts to Kibana. Review known limitations before you migrate every production folder.

Elasticsearch now stores Prometheus metrics in the same columnar backend as your logs and traces, at 3.75 bytes per datapoint, with no custom-metric penalty. The graph, the logs, and the traces answer to one query language.

Using an existing Kubernetes cluster (AWS EKS in this example): point Prometheus at Elastic with one config block, see every metric render in Discover with no dashboard to build, run most of your existing PromQL unchanged, find where ES|QL takes you past what PromQL can express, and finish by reading your logs with the same query language.

Nothing about your collection changes. Your scrape configs, relabeling rules, and service discovery carry over as-is. Most of your PromQL comes with you too — see the coverage note in Step 5 for the gaps.

Step 1: get a Prometheus endpoint and an API key from Elastic Observability

You need Elastic Cloud Serverless or Elastic Cloud Hosted. Both expose the Prometheus endpoints with no configuration.

Serverless is the fastest start. Sign in at cloud.elastic.co and create an Observability project. There is nothing to size and nothing to provision.

Elastic Cloud Hosted works the same way for everything in this post, and is the right choice when you need a specific stack version, a specific region topology, or the deployment-level controls that come with a managed cluster.

Find the Prometheus endpoint and create an API key in the UI

For Prometheus endpoint and the API key go to Kibana, click Add data in the left nav.

For Prometheus, scroll to Connect directly to the endpoint at the bottom and select the Prometheus tab.

Two things to copy:

The endpoint. It looks like https://my-observability-project-xxx.ingest.us-west-2.aws.elastic.cloud. Note the .ingest. host. This is not the same host as your Elasticsearch search endpoint or your Kibana URL.

The API key. Click Create key. Copy the value before you close the dialog, because it is not retrievable afterward.

If you want to scope it by hand instead, Open in API keys takes you to the full editor, and the minimum privilege for metrics ingest is:

{
  "ingest": {
    "indices": [
      {
        "names": ["metrics-*"],
        "privileges": ["auto_configure", "create_doc"]
      }
    ]
  }
}

Keep both values. Every remaining step uses them.

Step 2: make sure Prometheus is scraping your Kubernetes cluster

Prometheus should already be scraping your cluster.

If it is not, the shortest path is the kube-prometheus-stack Helm chart, which installs the Prometheus Operator, Prometheus itself, kube-state-metrics, and node-exporter in one command:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

That gives you the metrics you see for the rest of this post:

• cAdvisor container metrics (container_cpu_usage_seconds_total, container_memory_working_set_bytes)
• kube-state-metrics cluster objects (kube_deployment_spec_replicas, kube_daemonset_status_number_ready).

Step 3: configure Prometheus remote write to the Prometheus endpoint in Step 1

Elasticsearch implements the Prometheus Remote Write protocol natively. There is no adapter, no sidecar, and no translation layer. You add one block and the data flows on the next scrape interval.

If you run the Prometheus Operator

The Operator does not read a prometheus.yml you write by hand. It generates one from the Prometheus custom resource, and authorization.credentials there is a reference to a Kubernetes Secret, not an inline value. Create the secret first:

kubectl create secret generic elastic-prometheus \
  --namespace monitoring \
  --from-literal=api_key='YOUR_API_KEY'

Then reference it from values.yaml:

prometheus:
  prometheusSpec:
    remoteWrite:
      - url: "https://my-observability-project-xxxx.ingest.us-west-2.aws.elastic.cloud:443/api/v1/write"
        authorization:
          type: ApiKey
          credentials:
            name: elastic-prometheus
            key: YOUR_API_KEY

And apply it:

helm upgrade prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  -f values.yaml

Configure Prometheus remote write with prometheus.yml

Same thing, in prometheus.yml:

remote_write:
  - url: "https://YOUR_ES_ENDPOINT/_prometheus/metrics/node/eks/api/v1/write"
    authorization:
      type: ApiKey
      credentials: YOUR_API_KEY

If you run Grafana Alloy uss the following configuration

prometheus.remote_write "elasticsearch" {
  endpoint {
    url = "https://YOUR_ES_ENDPOINT/_prometheus/metrics/node/eks/api/v1/write"
    headers = {"Authorization" = "ApiKey YOUR_API_KEY"}
  }
}

How the remote write URL maps to Elasticsearch data streams

You do not name an index anywhere. There is no index field in the payload and no data stream in your remote_write config. Elasticsearch derives the target data stream from the write path and creates it on the first sample. The two path segments after /metrics/ are the dataset and the namespace:

| URL | Data stream | |---|---| | /_prometheus/api/v1/write | metrics-generic.prometheus-default | | /_prometheus/metrics/{dataset}/api/v1/write | metrics-{dataset}.prometheus-default | | /_prometheus/metrics/{dataset}/{namespace}/api/v1/write | metrics-{dataset}.prometheus-{namespace} |

The examples above use /metrics/node/eks/, which writes to metrics-node.prometheus-eks. That is the data stream you will see in Discover in the next step. Use dataset and namespace to separate production from staging, or to give each cluster and each team a data stream with its own retention and downsampling policy.

If you would rather keep a bare /api/v1/write URL, you can route per time series instead: attach data_stream_dataset and data_stream_namespace labels to the series, and they take precedence over the URL path. These two are control fields, so they route the document without being stored in its labels object.

Elasticsearch installs the index template for metrics-*.prometheus-* itself. You do not create templates or mappings.

What Prometheus metrics look like in Elastic Observability

Every Prometheus sample becomes a document. Labels become keyword fields that serve as time series dimensions. The value goes under metrics.<metric_name>:

{
  "@timestamp": "2026-07-02T10:30:00.000Z",
  "data_stream": {
    "type": "metrics",
    "dataset": "node.prometheus",
    "namespace": "eks"
  },
  "labels": {
    "__name__": "container_memory_working_set_bytes",
    "pod": "checkout-7d9f6c4b8-x2kqp",
    "namespace": "oteldemo",
    "container": "checkout",
    "node": "ip-10-0-3-14.us-west-2.compute.internal"
  },
  "metrics": {
    "container_memory_working_set_bytes": 36700160
  }
}

The one gotcha to know now. Elasticsearch infers whether a metric is a counter or a gauge from its name. Names ending in _total, _sum, _count, or _bucket are counters. Everything else is a gauge. That inference is correct for container_cpu_usage_seconds_total and correct for container_memory_working_set_bytes. It is wrong for any metric in your estate that does not follow Prometheus naming convention, and a misclassified metric gets rejected by the function that should accept it: RATE(my_metric::counter) works on counters only. Step 6 shows how to override the inference.

Current limits. Remote Write v1 only. Classic histograms and summaries are supported through their _bucket, _sum, and _count series, each mapped to the right metric type. Native (sparse) histograms and exemplars arrive with Remote Write v2, which is on the roadmap. Staleness markers are not stored or respected, and non-finite values (NaN, Infinity) are dropped silently. See the Prometheus remote write ingest docs for the full list.

Step 4: verify Prometheus metrics are flowing into Elasticsearch

Do not go build a dashboard. Confirm the pipeline first, and Elastic makes that a single command.

In Kibana, open Discover, switch the query editor to ES|QL, and type the name of the data stream you just wrote to:

TS metrics-node.prometheus-eks

That is the whole query. Discover reads the data stream, finds every metric in it, and renders each one as its own time series chart. Fifty metrics, fifty charts, no dashboard to build and no query to write per metric. It reads the metric type and charts each one correctly: gauges as averages, counters as rates, histograms as p95 distributions.

You can also get here without typing anything. Observability → Streams lists every data stream in the cluster. A Time series badge means it is a time series data stream. Click View in Discover and the TS query is filled in for you.

This is your ingest health check. Three things to look for.

• Data is flowing. Recent, continuous values. Not gaps, and not a line that stops an hour ago.
• Values are plausible. Memory in the tens of megabytes for a small container. CPU as a fraction of a core. Network bytes tracking real traffic.
• Coverage is what you expected. If kube_pod_container_status_restarts_total is missing, your kube-state-metrics scrape config is wrong, and you want to know that now rather than when you are building an alert on it.

Widen the time picker before you conclude anything is broken. A 15-minute window over a quiet period makes healthy data look flat.

To list what actually has data rather than what the mapping declares:

TS metrics-node.prometheus-eks | METRICS_INFO | SORT metric_name

The No dimensions selected control above the charts lets you break every chart out by a label: select pod and each chart splits into one series per pod.

Step 5: run PromQL queries on Prometheus metrics in Elastic Observability

If your team writes PromQL, keep writing PromQL. PROMQL is a source command in ES|QL, alongside FROM and TS, and it runs anywhere ES|QL runs: Discover, dashboard panels, and alert rules.

It does not run a separate engine. It parses the expression, resolves each function to its ES|QL equivalent, and builds a TS execution plan, so your PromQL gets the same vectorized, parallel execution as native ES|QL.

Current limits. PROMQL is generally available on Elastic Cloud Serverless and a tech preview on Elastic Stack 9.4, benchmarked at over 80% query coverage against popular Grafana OSS dashboards. The gaps worth knowing before you paste a dashboard in: histogram_quantile is not yet implemented, which matters most because it is how nearly every latency dashboard computes p95; group modifiers (on(...) group_left(...)) and the set operators or, and, and unless are unsupported; and predict_linear, label_join, and label_replace are not yet available. Time buckets also align to fixed calendar boundaries rather than the query start time, so short ranges or large steps can differ slightly from Prometheus. See the PROMQL command reference for the current list.

CPU: per-pod CPU rate with PromQL

The per-second CPU rate across containers, grouped by pod. This is the first thing you look at when something is hot:

PROMQL sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

Broken out by namespace instead, to find which team is burning the cluster:

PROMQL sum by (namespace) (rate(container_cpu_usage_seconds_total[5m]))

Note what came back: a pod column, a step column, and the value column named after the expression itself. That is a normal ES|QL table, which is the whole point and the thing Step 6 builds on.

Memory: working set and RSS with PromQL

Working set is the number that matters for OOM risk. It is what the kernel counts against the limit, and it is not the same as total allocated memory:

PROMQL sum by (pod) (container_memory_working_set_bytes)

Resident set, for comparison, when you are trying to tell a real leak from page cache:

PROMQL sum by (pod) (container_memory_rss)

Network: receive rate by pod with PromQL

PROMQL sum by (pod) (rate(container_network_receive_bytes_total[5m]))

Cluster objects: replica counts with PromQL

Deployments that are not running the replica count they declare:

PROMQL kube_deployment_spec_replicas

One nicety worth calling out: in Prometheus every query needs an explicit start, end, and step. In Kibana you drop all three. The date picker supplies the range and Kibana derives the step, which is why every query above is a single line.

Build a Kibana dashboard from PromQL queries

Every query above is a dashboard panel. In Discover, click Save, or go to Dashboards → Create → Add panel → ES|QL and paste the query in. The date picker drives start, end, and step, so a panel written once works at every zoom level.

Four panels, four one-line PromQL queries, no translation step. If you are coming from Grafana, this is the same dashboard you already have, rebuilt in about five minutes. If you would rather not rebuild it at all, keep Grafana and point its existing Prometheus datasource at Elasticsearch. That is covered at the end of the post.

Step 6: query Prometheus metrics with ES|QL, beyond what PromQL can express

The PROMQL command in ES|QL compiles to TS. These two queries are equivalent:

PROMQL sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

TS metrics-node.prometheus-eks
| WHERE TRANGE(1h)
| STATS SUM(RATE(metrics.container_cpu_usage_seconds_total, 5m)) BY labels.pod, TBUCKET(1m)

The second form is where metrics stop being a separate world from logs and traces — and where the real joins happen.

Top-N and filtering

A TS query returns a normal ES|QL table, so SORT, LIMIT, and WHERE all work downstream. Top-N needs no special function, and no topk. The ten pods using the most memory are a SORT and a LIMIT:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL
| STATS mem = SUM(metrics.container_memory_working_set_bytes) BY `labels.pod`
| SORT mem DESC
| LIMIT 10

Filtering is the same move, a WHERE on the aggregated column. Only the pods over 50 MB:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL
| STATS mem = SUM(metrics.container_memory_working_set_bytes) BY `labels.pod`
| WHERE mem > 50000000
| SORT mem DESC

You actually can pipe the results of a PROMQL query and post-process with regular ES|QL.

Memory usage against the request

A useful question during a memory scare: which pods are using more than they requested, and by how much. That combines two metrics, and ES|QL expresses it in a single pass with filtered aggregations, one MAX per metric:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL AND `labels.namespace` IS NOT NULL
| STATS
    used_memory      = MAX(metrics.container_memory_working_set_bytes),
    requested_memory = MAX(metrics.kube_pod_container_resource_requests)
                       WHERE `labels.resource` == "memory"
  BY `labels.pod`, `labels.namespace`, time_bucket = TBUCKET(5 minute)
| EVAL pct_of_request = 100 * used_memory / requested_memory
| WHERE pct_of_request > 80
| SORT pct_of_request DESC
| LIMIT 100

The WHERE attached to requested_memory is a filtered aggregation: kube_pod_container_resource_requests carries both CPU and memory under a labels.resource dimension, and the filter keeps only the memory rows, so the ratio is memory over memory. Each metric lives in its own documents; the shared BY key lands both aggregates on one row.

Step 7: query Prometheus metrics and logs together with ES|QL

Over the last few sections we used ES|QL and PromQL to explore metrics. ES|QL reads logs too. Here is a quick query against the OpenTelemetry demo running on this cluster:

FROM logs-*
| WHERE TRANGE(30m) AND kubernetes.pod.name == "checkout-9656cbd88-fsr9v"
| STATS events = COUNT(*) BY log.level, TBUCKET(1m)
| SORT events DESC

Same editor, same date picker you used for metrics. Only the source command changed, from TS to FROM, and now you are reading log volume by level per minute. One query language across metrics and logs, no tab switch and no second tool.

What's running in Elastic Observability now

Prometheus is writing to Elastic with one config block. Every metric renders in Discover with no dashboard built. Most of your existing PromQL runs unchanged, and ES|QL takes you further: top-N, filtering, and cross-metric ratios you can pipe into the rest of the language. Metrics and logs answer to the same query in the same window.

You did not drop a single label to get here, and you are not being billed for cardinality.

Next steps: Grafana, dashboard migration, and OpenTelemetry

1. Keep Grafana if you want it. Elasticsearch exposes a Prometheus-compatible read API at <endpoint>/_prometheus. Point Grafana's existing Prometheus datasource at it, set httpMethod: GET on the datasource, and your PromQL dashboards keep working. Keep Grafana, replace Mimir.
2. Migrate your Grafana dashboards. When you are ready to move off Grafana rather than point it at Elasticsearch, the Observability Migration Platform translates Grafana dashboards, panels, and alert rules into Kibana-native equivalents. It is a source-agnostic CLI (obs-migrate) that converts what it can and flags what needs a human, with a migration report showing what translated cleanly and where semantic gaps remain, so nothing is silently dropped. The walkthrough covers a Grafana and Datadog migration end to end.
3. Want to add OpenTelemetry? Read Part 2 if you are also running OpenTelemetry, or if you would rather collect with an OTel Collector than with Prometheus. Both land in the same store and the same queries read across both.

Start a free trial: cloud.elastic.co/registration Docs: elastic.co/docs/solutions/observability

Frequently asked questions

How do I send Prometheus metrics to Elasticsearch? Add a remote_write block to your Prometheus configuration pointing to https://<YOUR_ES_ENDPOINT>/_prometheus/metrics/{dataset}/{namespace}/api/v1/write with an Authorization: ApiKey <YOUR_KEY> header. Elasticsearch implements the Prometheus Remote Write v1 protocol natively — no adapter or sidecar required.

Can I run existing PromQL queries in Elasticsearch? Most of them, yes. ES|QL includes a PROMQL source command that accepts standard PromQL expressions, benchmarked at over 80% coverage against popular Grafana OSS dashboards. It is generally available on Elastic Cloud Serverless and a tech preview on Elastic Stack 9.4. histogram_quantile, predict_linear, label_join, and label_replace are not yet implemented, and group modifiers and the set operators or, and, and unless are unsupported.

Does Elasticsearch charge based on Prometheus metric cardinality? No. Elasticsearch stores Prometheus metrics at 3.75 bytes per datapoint with no cardinality-based billing and no custom-metric penalty, regardless of how many unique label combinations your metrics produce.

How do I route Prometheus metrics to different data streams in Elasticsearch? The two path segments after /metrics/ in the Remote Write URL set the dataset and namespace. /metrics/node/eks/api/v1/write writes to metrics-node.prometheus-eks. You can also route per time series using data_stream_dataset and data_stream_namespace labels, which take precedence over the URL path.

What Prometheus metric types does Elasticsearch support? Elasticsearch supports Remote Write v1, including counters, gauges, classic histograms, and summaries. Counter vs. gauge classification is inferred from the metric name: names ending in _total, _sum, _count, or _bucket are counters; everything else is a gauge. Native histograms and exemplars require Remote Write v2, which is on the roadmap.

Can I query Prometheus metrics and logs together in Elasticsearch? Yes. ES|QL reads both time series metrics and logs in the same query editor with the same date picker. Switch from TS metrics-node.prometheus-eks to FROM logs-* — same syntax, same window, no second tool.

Related reading

• Elasticsearch: best-in-class for logs, now best-in-class for metrics
• Ship Prometheus metrics to Elasticsearch with Remote Write
• Query Prometheus metrics in Elasticsearch with native PromQL support
• Don't leave metrics on the table: query them with the ES|QL TS command
• Elasticsearch as a backend for Grafana
• How we rebuilt Elasticsearch as a columnar metrics engine

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

What you need before running these Kubernetes ES|QL queries

To follow the queries in this article, you need:

• Elasticsearch 9.2 or later.
• EDOT Collectors running in your cluster and shipping data to Elasticsearch, with the kubeletstats, k8s_cluster, and filelog receivers enabled.

Why ES|QL for Kubernetes monitoring

Elasticsearch ES|QL is a piped query language that lets you start from a data source and then add one operation per line: filter, compute, aggregate, sort. That structure fits investigation work well because you refine a query step by step as you narrow down a problem.

When to use the ES|QL TS command for Kubernetes metrics

The TS command understands time series. This matters because most Kubernetes metrics are counters or gauges sampled over time. Using TS for metrics avoids the common mistake of summing cumulative counters across pods and getting a meaningless number.

EDOT receivers these Kubernetes queries depend on

The queries below assume EDOT Collectors are running in your cluster and shipping data to Elasticsearch. A typical setup uses:

• The kubeletstats receiver for pod, container, and node resource metrics.
• The k8s_cluster receiver for object state such as pod phase and container restarts.
• The filelog receiver with the k8sattributes processor for container logs.

Resource attributes follow the OpenTelemetry semantic conventions. Pod, namespace, and node identifiers appear as k8s.pod.name, k8s.namespace.name, and k8s.node.name. Metric names are defined by the receiver that emits them, not by the semconv spec itself: k8s.pod.phase comes from the k8s_cluster receiver, while utilization metrics like k8s.container.memory_limit_utilization come from kubeletstats. EDOT preserves all of these names natively in Elasticsearch. The exact fields you have depend on which receivers you enabled, so treat the queries as templates and adjust field names if a metric is missing.

Exploring the cluster

Start broad. Before investigating a specific symptom, it helps to see what the cluster looks like in your data.

This query counts the pods reporting metrics in each namespace over the past hour.

FROM metrics-*
| WHERE @timestamp >= NOW() - 1 hour
| STATS pod_count = COUNT_DISTINCT(k8s.pod.name) BY k8s.namespace.name
| SORT pod_count DESC

COUNT_DISTINCT collapses the many metric samples per pod into a single count per namespace. The result is a quick inventory: which namespaces are busy and whether anything you expected to be running is missing.

Finding pod restarts and crash loops

Restarts are usually the first signal that something is wrong. The k8s.container.restarts metric is a gauge that reports the current restart count for each container.

This query surfaces the containers that have restarted the most in the last 24 hours.

FROM metrics-*
| WHERE @timestamp >= NOW() - 24 hours AND k8s.container.restarts IS NOT NULL
| STATS restarts = MAX(k8s.container.restarts)
    BY k8s.namespace.name, k8s.pod.name, k8s.container.name
| WHERE restarts > 0
| SORT restarts DESC
| LIMIT 20

MAX takes the highest restart count seen in the window, which reflects the latest value of the gauge. A container with a high and climbing restart count is the textbook sign of a crash loop. Once you have the pod name, you can pivot straight to its logs with the queries further down.

Spotting pods that are not running

A restart count tells you a pod recovered. The pod phase tells you whether it is healthy right now. The k8s.pod.phase metric encodes the phase as a number:

| Value | Phase | | :---- | :---- | | 1 | Pending | | 2 | Running | | 3 | Succeeded | | 4 | Failed | | 5 | Unknown |

This query uses TS to read the latest phase per pod and keeps anything that is not Running.

TS metrics-*
| WHERE TRANGE(15m)
| STATS phase = MAX(LAST_OVER_TIME(k8s.pod.phase))
    BY k8s.namespace.name, k8s.pod.name
| WHERE phase != 2
| EVAL phase_name = CASE(
    phase == 1, "Pending",
    phase == 3, "Succeeded",
    phase == 4, "Failed",
    phase == 5, "Unknown",
    "Other")
| SORT phase_name

LAST_OVER_TIME picks the most recent sample for each pod's time series, so you compare the current state rather than an average. Pods stuck in Pending often point to scheduling problems, such as insufficient CPU or memory on the nodes. Pods in Failed or Unknown are worth an immediate look.

Catching memory pressure before the OOM kill

Out-of-memory kills are one of the most common Kubernetes failures, and they are easier to prevent than to debug after the fact. When you enable limit metadata on the kubeletstats receiver, EDOT reports k8s.container.memory_limit_utilization as a fraction between 0 and 1 of the container's memory limit.

This query finds containers that ran close to their limit in the last hour.

TS metrics-*
| WHERE TRANGE(1h)
| STATS peak_mem_pct = MAX(MAX_OVER_TIME(k8s.container.memory_limit_utilization))
    BY k8s.namespace.name, k8s.pod.name, k8s.container.name
| EVAL peak_mem_pct = ROUND(peak_mem_pct * 100, 1)
| WHERE peak_mem_pct > 85
| SORT peak_mem_pct DESC

MAX_OVER_TIME finds the peak within each container's series, and the outer MAX keeps that peak per container. A container that repeatedly touches 95% or higher is a strong candidate for the next OOM kill. Pair this with the restart query above: a container with both a rising restart count and high memory utilization was very likely OOM killed.

Tracking CPU usage and node pressure

CPU problems show up as throttling and slow response times rather than crashes. The k8s.pod.cpu.node.utilization metric reports each pod's CPU use as a fraction of total node capacity.

This query charts the busiest pods over the last hour in five-minute buckets.

TS metrics-*
  | WHERE TRANGE(1h)
  | STATS avg_cpu = AVG(AVG_OVER_TIME(k8s.pod.cpu.node.utilization))
      BY k8s.pod.name, TBUCKET(5m)
  | SORT avg_cpu DESC

AVG_OVER_TIME averages the samples inside each pod's series for the bucket, and the outer AVG combines series that share a pod name. TBUCKET(5m) produces one point every five minutes, which renders cleanly as a time series chart.

To check whether the nodes themselves are saturated, query node utilization directly.

TS metrics-*
  | WHERE TRANGE(1h)
  | STATS cpu = AVG(AVG_OVER_TIME(k8s.node.cpu.usage)),
          mem = AVG(LAST_OVER_TIME(k8s.node.memory.usage))
      BY k8s.node.name, TBUCKET(5m)
  | SORT cpu DESC

A node sitting near full CPU explains throttled pods across many namespaces at once, which is easy to misread as an application bug when you only look at a single pod.

Investigating container logs

Once metrics point you at a pod, logs explain what it was doing. EDOT stores the log message in body.text and the level in severity_text, alongside the same k8s.* fields as the metrics.

This query ranks namespaces and pods by error volume in the last hour.

FROM logs-*
| WHERE @timestamp >= NOW() - 1 hour
  AND severity_text IN ("ERROR", "FATAL")
| STATS errors = COUNT(*)
    BY k8s.namespace.name, k8s.pod.name
| SORT errors DESC
| LIMIT 20

Counting by pod tells you whether errors are concentrated in one workload or spread across the cluster. A single noisy pod and a cluster-wide spike call for very different responses.

To read what a specific pod is logging, filter by pod name and search the message text.

FROM logs-*
| WHERE @timestamp >= NOW() - 1 hour
  AND k8s.pod.name == "checkout-<your-hash>"
  AND body.text LIKE "*timeout*"
| KEEP @timestamp, severity_text, body.text
| SORT @timestamp DESC
| LIMIT 50

LIKE "*timeout*" does a simple wildcard match on the message. For full-text relevance instead of wildcards, swap it for MATCH(body.text, "timeout").

Chaining ES|QL queries to investigate a Kubernetes incident

Real Kubernetes investigations chain multiple ES|QL queries together instead of running one in isolation.

A useful loop looks like this:

1. Count errors by pod to find the noisy workload.
2. Check that pod's restart count and memory utilization to see if it is crashing or starved.
3. Read its recent logs to find the specific failure.

Because every query uses the same k8s.namespace.name and k8s.pod.name fields, you can carry a pod name straight from one query to the next. The same fields let you build a single dashboard where a metrics panel and a logs panel filter together as you click through namespaces.

Turning queries into alerts and dashboards

Any ES|QL query in this post that produces an aggregated value can back an alert, not just support ad hoc investigation.

For example, the restart query becomes an alert when you keep only containers above a threshold and trigger on a non-empty result.

FROM metrics-*
| WHERE @timestamp >= NOW() - 15 minutes AND k8s.container.restarts IS NOT NULL
| STATS restarts = MAX(k8s.container.restarts)
    BY k8s.namespace.name, k8s.pod.name, k8s.container.name
| WHERE restarts >= 5

Wire this into an Elasticsearch query rule and you get notified the moment a container crosses five restarts in fifteen minutes, instead of finding out when a user does. The same pattern applies to memory utilization, node saturation, and error counts.

Building a Kubernetes monitoring toolkit with ES|QL

ES|QL gives you one language for every Kubernetes signal, from object state to resource metrics to container logs. Start with the exploration query to understand your cluster's shape, then keep the restart, phase, memory, CPU, and log queries close for the next incident. Use TS when you need time series functions like MAX_OVER_TIME or TBUCKET to aggregate correctly within each pod or container series. For counting distinct values or taking a simple MAX on a gauge, FROM is enough.

From here, you can adapt the field names to your own receivers, save the most useful queries as dashboard panels, and promote the critical ones to alerts.

To go deeper, see the ES|QL reference, the TS command documentation, and the EDOT Kubernetes guide.

Problem context

Outages often show up as correlated errors on multiple services like checkout, frontend, and recommendation in Elasticsearch. That pattern can mean a shared dependency, or it can mean Kubernetes is misleading callers: wrong targetPort, empty Endpoints, or pods that never become ready. Observability tools like Elasticsearch tell you that callers fail and which edges look wrong. They generally do not fetch the live Service spec or compare containerPort to targetPort for you.

The Elastic AI Agent in Agent Builder is built for APM, logs, metrics, dependencies, and service maps. It is not a full EKS operations console. You could attach all EKS MCP tools to the same agent, but long tool lists increase wrong-tool calls, slow planning, and widen blast radius if a prompt accidentally asks for mutating actions.

Solution overview

Use Elastic AI Agent as the only agent your SRE chats with. It reasons from Elasticsearch first. When evidence points to cluster config, it calls a workflow tool that invokes the K8s Troubleshooter agent over /api/agent_builder/converse with a structured user_prompt. The K8s Troubleshooter agent carries only the EKS MCP tools, and cluster access stays scoped to one specialist identity, IAM, and RBAC. You can audit like any other integration.

Elasticsearch reaches EKS through an in-cluster bridge, exposed to Kibana as an MCP connector with a shared secret.

Before you start

You need:

• An EKS cluster with kubectl configured.
• An Elasticsearch 9.3+ deployment, an OTLP endpoint, an Elasticsearch API key, Agent Builder, and rights to create agents, MCP tools, and Workflows.
• An AI Connector in Elasticsearch for your chosen LLM.
• Budget two to four hours the first time you run these steps.

Implementation walkthrough

Step 1: deploy the Elastic OpenTelemetry Demo and ship telemetry to Elastic Observability

Follow elastic/opentelemetry-demo for Kubernetes and deploy elastic-opentelemetry-demo application to your EKS cluster. Configure your Elasticsearch OTLP endpoint and API key, confirm workloads are running, and note the namespace. In Kibana (APM, Logs, or Service Map), confirm data for checkout, frontend, recommendation, and product-catalog.

If you see healthy traffic to product-catalog, you are ready for the failure drill.

Step 2: run the EKS MCP bridge, register the connector, and bulk import EKS MCP tools

Complete the steps in aws-eks-mcp-setup end to end. The flow you would be following is:

1. Build and push the EKS MCP Bridge image
2. Create IAM policies
3. Create IRSA Service Account
4. Map IRSA role in aws-auth and apply Kubernetes RBAC
5. Deploy the bridge with a strong API_ACCESS_TOKEN to the EKS cluster
6. Connect Elastic Agent Builder with EKS MCP

A green MCP connector proves Kibana can reach the bridge.

For production, restrict LoadBalancer security groups to known Elasticsearch egress, prefer TLS on real paths, store tokens in Kubernetes Secrets, and use read-only MCP modes when you only diagnose.

Step 3: create a K8s Troubleshooter agent with EKS tools only

In Agent Builder, create an agent with agent ID k8s_troubleshooter, display name K8s Troubleshooter, and custom instructions from k8s_troubleshooter_agent. Attach only EKS MCP tools to this agent.

Chat directly with K8s Troubleshooter agent once and confirm a harmless read (for example list pods in the demo namespace).

Step 4 (Elasticsearch 9.3 only): clone the Observability Agent without EKS tools

Clone the bundled Observability Agent (Agent Builder → Manage Agents → Observability Agent → Clone) and name it Elastic AI Agent so it keeps the stock Observability system instructions and tools. Do not attach EKS MCP tools to this copy.

The parent Elastic AI Agent stays an observability-first interface for whoever chats with it.

Step 5: create the workflow and make it a callable tool

Create a new Elastic Workflow by importing k8s_troubleshooter_workflow.yaml and enable it.

Create a new tool in Agent Builder of type Workflow. Select the k8s_troubleshooter workflow, set tool ID custom.k8s_troubleshooter, and set the description to Tool to triage and troubleshoot kubernetes related issues (or equivalent wording your team standardizes on).

On Elastic AI Agent, attach the custom.k8s_troubleshooter workflow tool that you just created.

The parent’s tool list should show the custom.k8s_troubleshooter workflow tool attached, and K8s Troubleshooter agent should still answer when invoked on its own.

Step 6: inject the product-catalog service misconfiguration

Save the original targetPort, then patch to a wrong value (for example 9999).

kubectl get svc -A | grep product-catalog
kubectl get svc product-catalog -n YOUR_NAMESPACE -o yaml

kubectl patch svc product-catalog -n YOUR_NAMESPACE --type='json' \
  -p='[{"op": "replace", "path": "/spec/ports/0/targetPort", "value": 9999}]'

kubectl rollout restart deployment/checkout deployment/recommendation deployment/frontend -n YOUR_NAMESPACE

Callers still resolve Endpoints, but traffic lands on a port the container does not listen on, so Elasticsearch shows downstream errors on checkout, frontend, and recommendation.

You now have symptoms in Elasticsearch and a clear kubernetes cluster-side fault.

Step 7: run two prompts on the parent agent

Use AI Agent chat on Elastic AI Agent, not on the specialist.

Note: If you are using Elasticsearch 9.3, make sure you use the Elastic AI Agent that you created, not the stock agent.

Prompt 1: Why are failure transactions increasing for services like checkout and frontend?

Expect Elastic AI Agent to narrow the issue to the product-catalog service and note possible configuration issues as one of the probable causes, without yet invoking the custom.k8s_troubleshooter tool.

Prompt 2: Why is product-catalog service not servicing any requests in (insert your k8s cluster name) cluster? Is there any misconfiguration in the service?

Expect Elastic AI Agent to call the custom.k8s_troubleshooter tool, which invokes the K8s Troubleshooter agent and reads Service, Endpoints, and pods, compares targetPort to containerPort, and explains the mismatch with evidence. Expect to also see the recommended remediation steps.

Note: depending on the LLM you are using, the response from the agents may vary.

You get agent-led triage in Elastic Observability and cluster-grounded confirmation in the same thread, along with recommended remediation steps.

Step 8: patch the product-catalog service

Prompt 3: Patch the product-catalog service in (your EKS cluster name) cluster to have 8080 as the targetPort

Expect Elastic AI Agent to call the custom.k8s_troubleshooter tool, which invokes the K8s Troubleshooter agent and patches the product-catalog service.

Prompt 4: Rollout restart upstream services of product-catalog service

Expect Elastic AI Agent to identify all upstream services of product-catalog and call the custom.k8s_troubleshooter tool, which invokes the K8s Troubleshooter agent to roll out restarts for upstream services such as checkout, frontend, and recommendation.

Confirm product-catalog and upstream services recover in Elasticsearch.

Validation and trade-offs

You validated that Elastic AI Agent stays the main surface, that ~20 EKS tools live on one specialist K8s Troubleshooter agent, and that the Workflow plus Agent Builder converse API keeps a clear boundary for audits and reviews.

Trade-offs: MCP bridges need ongoing token and network hygiene, and you should keep mutating tools off or tightly RBAC-scoped until you accept the risk.

Frequently asked questions

Why is Elastic AI Agent not triaging the issues as explained in this article?

There could be two primary reasons. (A) If you are on Elasticsearch 9.3, make sure you chat on the Elastic AI Agent that you created, and not on the stock agent. (B) Make sure to use one of the LLM models rated Excellent or Great in Large language model performance matrix for Observability

Why do my services look unhealthy in Elasticsearch when the app code did not change?

Kubernetes can mislead HTTP clients: a bad Service targetPort, empty Endpoints, or pods that never become ready can fan out as errors on multiple edges in traces and service maps. Elastic Observability shows which dependencies fail; confirming the live Service spec usually needs cluster access.

How do I give Kubernetes access to Elastic AI Agent without putting every EKS tool on it?

Run two Agent Builder agents: keep the stock tools on the parent (Elastic AI Agent), and attach only EKS MCP tools to a specialist agent(K8s Troubleshooter agent). Invoke the specialist through a workflow that calls the Agent Builder converse API so the boundary is explicit and auditable.

Why chain agents with Elastic Workflows instead of one long system prompt?

Workflows give a callable, reviewable step between observability reasoning and cluster actions, which helps with governance and keeps the parent agent’s tool list short. Long unified tool lists often increase mistaken tool use and broaden blast radius if a prompt requests a mutating operation.

How does this compare to kubectl or a cloud console for incident response?

Consoles and kubectl stay the source of truth for live object state. This pattern automates the handoff from Elastic Observability signals to those checks through MCP, while still relying on IAM and Kubernetes RBAC on the specialist identity.

What are the main limitations or risks of an EKS MCP bridge with Agent Builder?

MCP bridges need token rotation, network restrictions, and TLS discipline on real paths. Mutating EKS tools should stay off or tightly RBAC-scoped until you accept operational risk.

Why do we need an EKS MCP bridge?

The managed EKS MCP server authenticates via AWS SigV4 through a stdio-based proxy (mcp-proxy-for-aws). Elastic's MCP connector requires an HTTP/SSE endpoint. The bridge pod runs mcp-proxy to expose the stdio proxy as an SSE/HTTP endpoint.

Can I reuse the same layout on GKE, AKS, or self-managed Kubernetes?

Yes. The separation principle is the same: observability data in Elasticsearch plus a specialist agent with cluster-scoped tools and a workflow-mediated handoff. Swap the MCP server or bridge, adjust RBAC, and parameterize cluster name or region in workflow inputs where needed.

When cluster CPU spikes and you need to find which workload is responsible, narrow from fleet to namespace to pod, one step at a time.

What you need

• An Observability serverless project or a self-managed or Elastic Cloud Hosted stack at version 9.4 or later, where PromQL is available as a preview query language for metrics.
• Kubernetes metrics flowing into Elasticsearch. For this exercise we have considered OpenTelemetry data.
• One or more clusters with workloads running so group by queries have something to compare.

The scenario

You manage a fleet of Kubernetes clusters:

| Cluster | Region | Role | |---------|--------|------| | prod-us-east-1 | US East | Production: services, ML training | | prod-eu-west-1 | EU West | Production: regional web tier, cache | | staging-us-east-1 | US East | Staging: QA, integration tests | | dev-sandbox | US East | Developer sandbox |

The production cluster in US East runs a mix of services and ML training jobs across several namespaces.

An alert fires: cluster-wide CPU is elevated, but only one team is complaining about slower response times.

You are triaging which cluster, then which namespace, then which pod.

You are not after a full root-cause proof in one query, but enough to name the suspect and hand off.

Your data

The OpenTelemetry Collector's Kubelet Stats Receiver populates data streams like metrics-kubeletstatsreceiver.otel-default. Metrics follow the k8s.* naming convention (for example k8s.pod.cpu.usage) and labels like k8s.cluster.name or k8s.namespace.name let you slice by cluster, namespace, or pod.

To verify the data is there, open Discover, switch to ES|QL mode, run TS metrics-*, and scope the query with WHERE data_stream.dataset == "kubeletstatsreceiver.otel".

Investigation: find the noisy neighbor

Step 1: Which cluster is hot?

When you manage multiple clusters, start at the fleet level.

PROMQL sum by (k8s.cluster.name) (k8s.pod.cpu.usage)

This groups total pod CPU by cluster.

prod-us-east-1 immediately stands out: total pod CPU is an order of magnitude higher than the other clusters.

The EU production cluster, staging, and dev-sandbox are all quiet.

Now you know where the problem is, time to zoom in.

Step 2: Overall CPU in the hot cluster

Filter to prod-us-east-1 and look at total CPU:

PROMQL sum(k8s.pod.cpu.usage{k8s.cluster.name="prod-us-east-1"})

This gives you the cluster-wide pod CPU footprint over time.

If the total is climbing or spiking, something changed, but you don't yet know what.

Step 3: Break down by namespace

The fastest way to isolate which team is responsible: group by namespace.

PROMQL sum by (k8s.namespace.name) (k8s.pod.cpu.usage{k8s.cluster.name="prod-us-east-1"})

Set the time picker in Kibana to cover your incident window.

ml-training dominates at ~2.0 cores while every other namespace stays well below 0.2 cores.

Step 4: Drill down to the pod

Now that you know the namespace, identify the specific pod:

PROMQL sum by (k8s.pod.name) (k8s.pod.cpu.usage{k8s.cluster.name="prod-us-east-1", k8s.namespace.name="ml-training"})

That ranks pods in the namespace by total CPU.

The chart should make the outlier obvious.

Pod model-train-v2-run-47-d9j67 is consuming the full 2.0 cores. It is a training job saturating its allocation.

Step 5: Check resource utilization ratios

Raw CPU cores tell you how much. Utilization ratios tell you how close to limits.

A pod hitting 100% of its CPU limit is being throttled, and it is both the noisy neighbor and a victim of its own limits.

PROMQL sum by (k8s.namespace.name) (k8s.container.cpu_limit_utilization{k8s.cluster.name="prod-us-east-1"})

ml-training shows ~100% CPU limit utilization (pegged at the 2-core limit), while the other namespaces stay under 20%.

This confirms the training job is saturating its allocation and likely causing scheduling pressure on the shared node.

What happens next

The PromQL query named the suspect: the training job model-train-v2-run-47 in ml-training.

From here:

• Logs: Filter by the pod name in Discover to see what the training job was doing and whether it logged errors or warnings.
• Kube events: Check for OOMKilled, throttling, or eviction events in the same time window.
• Resource policies: Review whether the training job's requests and limits match its actual usage. A large gap between request and limit lets a pod burst past what the scheduler planned for. Consider ResourceQuota or LimitRange on the namespace.

The new Elastic Kubernetes integration is built for that operator. It includes dashboards with drilldowns, alert rule templates, and ML anomaly detection jobs. Additionally Elastic also offers Agentic Investigations, that drives investigations automatically.

This blog will cover the foundational observability components (dashboards, drilldowns, alert templates, etc), while a part 2 covering the agentic investigations will cover workflows, agent skills, and MCP tools and views

The new Kubernetes integration content in this post is generally available across Elastic Cloud Hosted, Serverless, and self-managed deployments.

Dashboards designed for drill-down, not just display

The new Kubernetes dashboards are organized around a three-tier design: a cluster Overview that surfaces what needs attention at a glance, object summary pages for clusters, nodes, namespaces, workloads, and pods, and object detail pages that give you the full picture for any single entity.

Every layer connects to the next: click any entity in a summary table and choose: apply it as a filter on the current view, or open its dedicated detail page.

Here's what that looks like when something's actually wrong:

Following a restart cascade from overview to container

Overview: The Overview surfaces what needs attention across your cluster. You can see top pods by CPU, top namespaces by container restarts, and top nodes by memory utilization in one screen. When the "container restarts" panel starts climbing, you know where to look.

Namespaces Overview: Click into the flagged namespace with 1232 restarts and CPU limit utilization at 116%. The detail view plots CPU and memory against requests and limits over time. This shows both the size and duration of the overage.

Namespace Details: We can get more info on the various pods in this namespace here. Click the pod driving the restarts.

Pod Details: The pod detail dashboard is organized into capacity, metrics, and containers sections. Container restarts are flagged in red at the top of the page. Most panels are metric-driven, and the dashboard also links to correlated pod logs in Discover.

It takes four clicks to move from the Cluster Overview to container logs that explain the failure. These dashboards are starting points for your team. You can copy and customize them with ESQL visualizations.

Alert rules that fire on day one

The integration ships with pre-built alerting rule templates for states that are wrong by definition. No historical baseline or warmup period is required. Enable them during setup and they work immediately.

These rules do not ask, "Is this abnormal for this service?" They ask, "Is this a known bad state in Kubernetes?" A pod in CrashLoopBackOff is always a problem. A container killed by the kernel for exceeding its memory limit is always a problem.

Like the Kubernetes dashboards, these alerts are built on ES|QL queries. You can see that in the CrashLoopBackOff definition below. If you are new to ES|QL, you can start with the ES|QL docs.

The alert templates cover:

• CrashLoopBackOff detection - Fires when a pod's restart count exceeds a configurable threshold within a rolling window. The default catches a real restart cycle without triggering on routine restarts during a rolling deployment.
• Container OOMKilled - Surfaces kernel-level container terminations due to memory limits. These events are easy to miss in dashboards and often precede wider failures. This rule fires on any occurrence.
• Deployment below desired replicas - Fires when a deployment runs fewer replicas than declared for longer than a grace period. This catches scaling failures and partially failed rollouts.
• Pod stuck in Pending - Fires when a pod cannot be scheduled past a configurable time threshold. This surfaces node capacity problems, missing resources, and affinity failures before availability drops.
• Node disk pressure - Fires immediately when the Kubernetes DiskPressure node condition is True. A node condition is a direct state signal, not a statistical threshold.
• Persistent volume near capacity - Alerts when storage utilization crosses a configurable threshold before writes start failing.

Each template is parameterized. Adjust thresholds in the ES|QL query to match your environment. Connect notifications to PagerDuty, Slack, or another destination in your runbook.

Anomaly detection jobs with ML baselines

Alert rules catch what is definitively wrong. ML anomaly detection catches patterns that often precede failures. If you are new to this area, see the Elastic anomaly detection overview.

A pod that always runs at 85% memory utilization might be healthy. A pod that grew from 40% to 85% over twelve hours is usually not healthy. A static threshold often catches this only after an OOM kill. The ML module should catch the trajectory earlier.

The integration ships with ML module configurations that learn workload baselines and flag meaningful deviations. These jobs need 24 to 48 hours of data before results become useful. Results become more reliable as jobs continue to run.

The included modules

1. Pod memory growth anomalies

• What it learns: per-pod memory consumption pattern over time
• What it flags: Growth trajectories that are inconsistent with baseline behavior, such as a slow leak that never crosses the hard limit.
• Why ML (not alert rule): The alert rule catches the OOMKill after the fact. The ML job catches the trajectory that leads there.

2. Network I/O anomalies

• What it learns: per-pod network transmit/receive byte rate patterns
• What it flags: Unusual spikes or drops relative to the pod baseline. A spike can indicate a runaway process or unexpected load. A drop can indicate a network partition that causes the pod to go idle.
• Why ML (not alert rule): Normal network traffic varies by time of day and workload type. A batch job pod at high throughput during its normal window is expected. The same throughput outside that window can be anomalous.

3. Pod restart frequency

• What it learns: Per-workload restart rate patterns during deployments, scaling events, and routine operations.
• What it flags: Restart patterns that are anomalous relative to each workload's own history. This is distinct from the CrashLoopBackOff alert rule, which fires on a fixed threshold regardless of context.
• Why ML (not alert rule): A deployment that restarts twice during every rollout can be healthy. The same deployment restarting twice on a Tuesday afternoon may be unhealthy. The alert rule cannot distinguish these cases without workload history.

Here's our Single Metric Viewer showing anomalies triggered against a specific pod, for the memory growth job:

And here's the multi-series Anomaly Explorer view of the same job, showing detections firing across a variety of pods:

Try it yourself: the OTel Astronomy Shop

If you do not have a Kubernetes cluster ready, you can use the OpenTelemetry Astronomy Shop demo environment. It uses the same commands as Getting Started Step 2, Path A, but points to demo services. Create the namespace and secret, then run the Helm install. All 16 services, Kafka, and PostgreSQL start flowing into Elastic without instrumentation changes.

The demo ships with a built-in feature flag service, flagd, that lets you activate failure scenarios. Enable cartServiceFailure and watch the checkout-service restart cascade unfold in real time. The CrashLoopBackOff alert rule fires. The ML modules begin establishing baselines. If you have the investigation workflow enabled, it runs automatically when the alert fires.

Getting started

Step 1 - Install the Kubernetes integration. Dashboards are available immediately. No additional configuration is required.

Step 2 - Deploy data collection. There are two supported paths, both based on Helm. Choose the one that fits your deployment model.

Path A - OpenTelemetry (EDOT collector): This path uses the opentelemetry-kube-stack Helm chart with the Elastic Distribution of OpenTelemetry (EDOT) collector. Create a namespace and a secret with your endpoint and API key, then install:

kubectl create namespace opentelemetry-operator-system

kubectl create secret generic elastic-secret-otel \
  --namespace opentelemetry-operator-system \
  --from-literal=elastic_otlp_endpoint='https://<your-endpoint>.elastic.cloud:443' \
  --from-literal=elastic_api_key='<your-api-key>'

helm upgrade --install opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack \
  --namespace opentelemetry-operator-system \
  --values 'https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v9.3.2/deploy/helm/edot-collector/kube-stack/managed_otlp/values.yaml' \
  --version '0.12.4'

Path B - Elastic Agent (standalone): This path uses the elastic/elastic-agent Helm chart. The default manifest includes resource limits that may not be appropriate for production. Review the Scaling Elastic Agent on Kubernetes guide before deploying.

helm repo add elastic https://helm.elastic.co/ && \
helm install elastic-agent elastic/elastic-agent \
  --version 9.3.2 \
  -n kube-system \
  --set outputs.default.url=https://<your-endpoint>.es.elastic.cloud:443 \
  --set outputs.default.type=ESPlainAuthAPI \
  --set outputs.default.api_key=$(echo "<your-base64-api-key>" | base64 -d) \
  --set kubernetes.enabled=true

Step 3 - Enable the alert rule templates. Go to Observability > Alerts in Kibana. The Kubernetes templates are in the rule library. Enable the templates relevant to your environment, set thresholds, and connect your notification channel.

Step 4 - Let the ML modules warm up. After 24 to 48 hours, anomaly detection modules establish baselines and begin surfacing pattern-based deviations. Longer running jobs usually produce better baselines. Find results in the ML Anomaly Explorer, linked from the Kubernetes dashboards.

Steps 5, 6, and 7 - Agentic content will be covered in Part 2 (forthcoming), Kubernetes observability with Elastic: Agentic Investigations.

What's next

The next step is the layer that runs investigation workflows when an alert fires. That includes skills that encode investigation logic, tools that expose facts like ML state and topology, and MCP apps that render outputs in places like Claude Desktop or VS Code. These technical preview capabilities are available today and will be covered in Part 2 (forthcoming), Kubernetes observability with Elastic: Agentic Investigations.

If you are running Kubernetes on Elastic today, tell us which investigation steps you repeat manually on every incident. Tell us which remediations you would trust a workflow to propose. You can join the Elastic Community Discussion here.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Traffic grows. Teams want metrics derived from traces. The backend goes down for maintenance and you lose an hour of telemetry. A compliance requirement means PII must be stripped before data leaves the cluster. Suddenly, that single collector isn't enough — and the question becomes: what should the architecture actually look like?

The OpenTelemetry Collector is designed to be composed. It can run in multiple deployment modes, be chained into pipelines, and scaled independently at each stage. But the documentation describes individual components, not how to think about assembling them. That thinking is what this article is about.

What follows is a conceptual framework for reasoning about collector architectures — not a set of rigid templates. The building blocks described here are reference points. In practice, they combine, overlap, and adapt to your constraints. A tail sampling tier might also need Kafka-backed resilience. A gateway might absorb the role of a sampling tier at low volumes. The goal is to understand the concepts well enough to compose the right architecture for your situation, not to pick a pre-built one off a shelf.

Three conceptual layers

It helps to think about collector architectures in three layers: edge, processing, and resilience. These aren't physical tiers that must exist as separate deployments — they're categories of concern. A single collector can address multiple layers. A complex deployment might have several components within one layer. The layers are a thinking tool, not a deployment diagram.

Edge: how telemetry enters the pipeline

The edge layer is about the first hop — how telemetry gets from your applications and infrastructure into the pipeline. At this stage, the collector gathers data in two fundamentally different ways. Pull-based receivers like filelog and hostmetrics actively reach out to collect data — tailing log files on disk or scraping system-level metrics from the host. Push-based receivers like otlp listen for data sent to them — applications instrumented with OpenTelemetry SDKs export traces, metrics, and logs directly to the collector's OTLP endpoint. A single edge collector typically runs both: pull receivers for infrastructure telemetry the application doesn't know about, and push receivers for application telemetry the SDK produces. There are several common deployment patterns, and the right one depends on your environment and what you need to collect.

DaemonSet Agent — One OpenTelemetry Collector per Kubernetes node, deployed as a DaemonSet. Applications export to the agent running on the same node (typically via status.hostIP:4317 using the Kubernetes Downward API). The agent also tails container log files from disk via the filelog receiver and scrapes host-level metrics via the hostmetrics receiver. This is the most common Kubernetes pattern because it handles both application and infrastructure telemetry with a single deployment, and applications only need to know about localhost.

Sidecar Agent — One OpenTelemetry Collector per pod, deployed as a sidecar container. Each service gets its own collector with a custom configuration. This is required on managed container platforms like AWS Fargate or Azure Container Apps where DaemonSets aren't available, and it's useful when services have different processing requirements. When running alongside a DaemonSet, the sidecar handles application telemetry while the DaemonSet independently collects node-level telemetry — applications don't send to both.

Host Agent — A standalone OpenTelemetry Collector running as a systemd service on bare-metal or VM hosts. It serves the same role as the DaemonSet agent but outside Kubernetes: collecting host metrics, tailing log files, and receiving OTLP from local applications.

Direct SDK Export — Applications export directly to the next stage (gateway or backend) with no local collector. This is the simplest option but only works when you don't need infrastructure collection. For log collection, the recommended pattern is still to write to stdout and use a collector with the filelog receiver — even if the SDK is exporting traces and metrics directly.

These patterns aren't mutually exclusive. A Kubernetes cluster might run DaemonSet agents for infrastructure collection alongside sidecars for services that need custom processing. A VM environment might use host agents for some services and direct SDK export for others. The edge layer is about matching the collection pattern to the workload, not picking one pattern for everything.

Processing: central policy, sampling, and transformation

Not every architecture needs a processing layer. If your edge collectors can export directly to your backend and you don't need centralized policy, you can skip it to favour simplicity. But several scenarios push you toward central processing — and the way you address them can range from a single gateway to a multi-stage pipeline.

Centralized policy (Gateway) — A pool of OpenTelemetry Collectors that sits between edge collectors and the backend. This is where you enforce consistent filtering, transformation, and PII redaction across all services. It's also where you manage backend credentials — edge collectors export to the gateway over OTLP, and only the gateway holds the API keys. Credential isolation is often the primary reason teams add a gateway.

Replica count scales with data volume. At low volumes (under 1K events/sec), 2 replicas co-located with workloads is sufficient. At medium volumes, 3–5 replicas on a dedicated node pool. At high volumes, 5–20+ replicas, potentially in a separate cluster. This is a general rule of thumb, and you should adapt it to your specific needs as loads might vary significantly between payload types.

Tail-based sampling — Sampling decisions that consider the complete trace (e.g., "keep all traces with errors, sample 10% of successful traces") require that all spans of a trace reach the same collector instance. This is achieved with the loadbalancingexporter using routing_key: traceID, which consistently routes spans from the same trace to the same downstream collector.

There's a critical subtlety here: if you're deriving span metrics (RED metrics) from traces using the spanmetrics connector, the derivation must happen before sampling. Otherwise, your metrics only reflect the sampled subset, not the true traffic. The correct pattern is a two-step pipeline within the sampling stage:

1. Receive traces, derive spanmetrics from 100% of traffic, forward via a forward connector.
2. Apply tail_sampling to the forwarded traces, export only kept traces.
3. A separate metrics pipeline exports the derived RED metrics.

This ensures accurate metrics regardless of your sampling rate.

The key point about processing is that these capabilities — gateway policy, tail sampling, span metrics derivation — are not separate products or fixed modules. They're configurations of the same OpenTelemetry Collector. At low volumes, a single gateway deployment might handle policy enforcement, sampling, and metrics derivation all at once. At high volumes, you might split them into dedicated stages for independent scaling. The architecture adapts to your scale, not the other way around.

Resilience: what happens when the backend is down

The resilience layer determines how much data you're willing to lose during backend outages or collector restarts. This isn't a separate tier you bolt on — it's a property you apply to any stage of the pipeline.

In-Memory Queues — The default. The collector's sending_queue retries failed exports with exponential backoff. If the collector process crashes or restarts, queued data is lost. This is acceptable for development and for workloads where some data loss during incidents is tolerable.

Persistent Queues (WAL) — The file_storage extension writes queued data to disk before export. If the collector crashes, it resumes from where it left off after restart. In Kubernetes, this requires a PersistentVolumeClaim. This is the right choice for most production workloads — it survives collector restarts and brief backend outages without the operational complexity of an external message bus.

Kafka Buffer — An external Kafka cluster sits between collectors and the backend. Producer collectors write to Kafka topics; consumer collectors read from Kafka and export to the backend. This provides the strongest durability guarantee — Kafka can buffer hours of telemetry during extended outages and enables replay. But it adds significant operational complexity.

The important thing to understand is that resilience is orthogonal to the other layers. You can add persistent queues to an edge agent, a gateway, or a sampling tier. You can put Kafka in front of a gateway, in front of a sampling tier, or in front of the backend. A tail sampling deployment that needs to survive extended outages might use Kafka-backed ingestion — combining what might look like two separate "modules" into a single stage. The building blocks compose freely based on what you need to protect against.

Where to start with your architecture

The Agent + Gateway two-tier pattern is the de facto production standard, used by the vast majority of organizations running OpenTelemetry at scale. DaemonSet agents on every node handle local collection — pulling infrastructure telemetry via filelog and hostmetrics, receiving application telemetry via OTLP — while a centralized gateway pool enforces policy, manages credentials, and exports to the backend. Persistent queues (WAL) on the gateway protect against backend outages without external dependencies.

Every other configuration either simplifies this pattern or extends it. Smaller environments might drop the gateway and export directly from agents. Larger ones might add a tail sampling tier with traceID-based load balancing, a Kafka buffer for extended resilience, or span metrics derivation before sampling. The building blocks described in the previous sections — edge, processing, resilience — are the modules you add or remove from this foundation.

The key is to start with the two-tier pattern and evolve incrementally:

• Need credential isolation or centralized PII redaction? You already have the gateway.
• Need tail-based sampling? Add a load-balancing exporter and a sampling tier between agents and gateway.
• Need hours of buffer during extended outages? Insert Kafka between agents and the processing tier.
• Running on Fargate or Azure Container Apps? Swap DaemonSet agents for sidecars — the rest of the pipeline stays the same.

Start here. Add modules as your needs grow. The architecture adapts to your scale, not the other way around.

Decision points that shape where you need to take your architecture

When designing a collector architecture, these are the questions that determine which patterns you need:

| Question | Impact | |----------|--------| | Do I need infrastructure telemetry (host metrics, disk logs)? | Determines whether you need a local collector or can use direct SDK export | | Am I on a managed container platform (Fargate, ACA)? | Forces sidecar pattern instead of DaemonSet | | Do I need centralized filtering, PII redaction, or credential isolation? | Adds a gateway stage | | Do I need tail-based sampling? | Adds a sampling stage with load-balancing exporter and traceID routing | | Do I want span-derived metrics (RED metrics)? | Requires spanmetrics before sampling in a two-step pipeline | | How much data loss is acceptable during outages? | Determines in-memory queues vs. persistent queues vs. Kafka — applied to whichever stage needs protection | | What is my expected data volume? | Determines whether capabilities can be co-located in a single deployment or need dedicated stages |

The answers to these questions don't map to a single "correct" architecture. They constrain the design space, and within those constraints, you make trade-offs between simplicity and capability.

Exploring these patterns interactively

If you'd rather explore how these building blocks compose than assemble them by hand, OpenTelemetry Blueprints is an open-source tool that generates reference architectures from your requirements.

Toggle your environment, signals, volume, resilience, and processing needs — and get a composed diagram with animated data flow, interactive tooltips, and reference collector configurations you can open directly in OTelBin for validation.

The generated configurations export via OTLP, so they work with any OTLP-compatible backend — including Elastic Observability, which natively accepts and stores OTLP traces, metrics, and logs.

The architectures Blueprints generates are reference compositions — starting points for understanding how the building blocks fit together, not turnkey deployments. Every architecture should be adapted to your organisation's scale, security, networking, and compliance requirements. The patterns might combine or overlap differently in your environment than in anyone else's, and that's the point.

Get started

The architectures described here export over OTLP, so they work with any compatible backend. If you don't have one yet, the fastest way to see your telemetry flowing end-to-end is with Elastic Observability — it natively ingests OTLP traces, metrics, and logs with no additional configuration.

1. Start a free trial on Elastic Cloud Serverless — no credit card required.
2. Point your collector's OTLP exporter at the managed OTLP endpoint.
3. Explore your traces, metrics, and logs in Kibana within minutes.

Check out these resources to go further:

• Elastic's managed OTLP endpoint documentation
• EDOT Collector — Elastic's distribution of the OpenTelemetry Collector
• OpenTelemetry Blueprints — generate reference architectures interactively

• Detect Kubernetes pod restarts and OOMKill events using Elastic Agent Builder
• Analyze CPU and memory pressure using ES|QL over Kubernetes metrics
• Generate troubleshooting summaries and remediation guidance

This article explains how to use Elastic Agent Builder to automatically detect, analyze, and remediate Kubernetes pod failures caused by resource pressure (CPU and memory), with a focus on pods experiencing frequent restarts and OOMKilled events. Elastic Agent Builder lets you quickly create precise agents that utilize all your data with powerful tools (such as ES|QL queries), chat interfaces, and custom agents.

Introduction: What is the Elastic Agent Builder?

Elastic has an AI Agent embedded that you can use to get more insights from all of the logs, metrics and traces that you’ve ingested. While that’s great, you can take it one step further and streamline the process by creating tools that the agent can use.

Giving the agent tools means it spends less time ‘thinking’ and quickly gets to assessing what’s important to you. For example, if I have a Kubernetes environment that needs monitoring, and I want to keep an eye on pod restarts and memory and CPU usage without hanging out at the terminal, I can have Elastic alert me if something goes wrong. 

Having an alert is great, but how do I get the bigger picture, faster? You need to know what service is having (or creating) the issues, why, and how to fix it.

Assumptions

This guide assumes:

• A running Kubernetes cluster
• An Elastic Observability deployment
• Kubernetes metrics indexed in Elastic

Step 1: Create a New Elastic Agent

In Elastic Observability, use the top search bar to search for Agents. Create a new agent.

This agent is going to be the Kubernetes Pod Troubleshooter agent, designed to help users troubleshoot pod restarts, OOMKill terminations and evaluate CPU or memory pressure. 

The Kubernetes Pod Troubleshooter agent will:

1. Identify pods that have restarted more than once
2. Filter for pods that are not in a running state
3. Retrieve the container termination reason (e.g., OOMKilled)
4. Analyze CPU and memory utilization for affected services
5. Flag resource utilization above 60% (warning) and 80% (critical)
6. Provide remediation recommendations

The agent requires instructions to guide how the agent behaves when interacting with tools or responding to queries. This description can set tone, priorities or special behaviours. The instructions below tell the agent to execute the steps outlined above. 

You will help users troubleshoot problematic pods by searching the metrics for pods that have restarted more than once and the status is not running. Pods that have the highest number of restarts will be returned to the user.
Once the containers that are not running and have restarted multiple times are found you will use their container ID or image name to to look up the container status reason and reason for the last termination. You will return that reason to the user.
You will also begin basic troubleshooting steps, such as checking  for insufficient cluster resources (CPU or memory) from the metrics and tools available.
Any CPU or memory utilization percentages over 60%, and definitely over 80% should be flagged to the user with remediation steps.

Getting answers quickly is critical when troubleshooting high-value systems and environments. Using Tools ensures that the workflow is repeatable and that you can trust the results. You also get complete oversight of the process, as the Elastic Agent outlines every step and query that it took and you can explore the results in Discover.

You will create custom tools that the agent will run to complete the Kubernetes troubleshooting tasks that the custom instructions references such as: look up the container status reason and reason for the last termination and checking  for insufficient cluster resources (CPU or memory).

Step 2: Create Tools - Pod Restarts

The first tool takes the Kubernetes metrics and assesses if the pod has restarted and it has a last terminated reason, and if it has the agent will present that information to the user.

This pod-restarts tool uses a custom ES|QL query that interrogates the Kubernetes metrics data coming from OTel.

The ES|QL query:

1. Filters for containers that have restarted and have a reason for termination; then
2. Calculates the number of restarts; then
3. Returns the number of restarts and termination reason per service.

FROM metrics-k8sclusterreceiver.otel-default
| WHERE metrics.k8s.container.restarts > 0
| WHERE resource.attributes.k8s.container.status.last_terminated_reason IS NOT NULL
| STATS total_restarts = SUM(metrics.k8s.container.restarts),
        reasons = VALUES(resource.attributes.k8s.container.status.last_terminated_reason) 
  BY resource.attributes.service.name
| SORT total_restarts DESC

Step 3: Create Tools - Service Memory

The custom tools can take input variables, which increases speed and accuracy of the results.

Common reasons for pods not scheduling, or restarting often, is due to the cluster or nodes being under-resourced. The pod-restarts tool returns services that have many restarts and OOMKill termination reasons, which indicate memory pressure.

The eval-pod-memory tool is a custom ES|QL that:

1. Filters for metrics data that match the service name returned from the pod-restarts tool within the last 12 hours; then
2. Converts memory usage, requests, limits and utilization into megabytes; then
3. Calculates the average of each of those metrics; then
4. Groups them into 1 minute groupings and sorts them.

FROM metrics-*
| WHERE resource.attributes.service.name == ?servicename
| WHERE @timestamp >= NOW() - 12 hours
| EVAL
  memory_usage_mb = metrics.container.memory.usage / 1024 / 1024,
   memory_request_mb = metrics.k8s.container.memory_request / 1024 / 1024,
   memory_limit_mb = metrics.k8s.container.memory_limit / 1024 / 1024,
   memory_utilization_pct = metrics.k8s.container.memory_limit_utilization * 100
| STATS
   avg_memory_usage = AVG(memory_usage_mb),
   avg_memory_request = AVG(memory_request_mb),
   avg_memory_limit = AVG(memory_limit_mb),
   avg_memory_utilization = AVG(memory_utilization_pct)
   BY bucket = BUCKET(@timestamp, 1 minute)
| SORT bucket ASC

Step 4: Create Tools: Service CPU

As CPU usage is another common reason for pods to fail scheduling or be stuck in endless restart loops, the next tool will evaluate CPU usage, requests and limits.

The eval-pod-cpu tool is a custom ES|QL that:

1. Filters for metrics data that match the service name returned from the pod-restarts tool within the last 12 hours; then
2. Calculates the average for CPU usage, CPU request utilization and CPU limit utilization.

FROM metrics-kubeletstatsreceiver.otel-default
| WHERE k8s.container.name == ?servicename OR resource.attributes.k8s.container.name == ?servicename
| STATS
  avg_cpu_usage = AVG(container.cpu.usage),
  avg_cpu_request_utilization = AVG(k8s.container.cpu_request_utilization) * 100,
  avg_cpu_limit_utilization = AVG(k8s.container.cpu_limit_utilization) * 100
| LIMIT 100

Step 5: Assign Tools to Kubernetes Pod Troubleshooter Agent

Once all of the tools are built you need to assign them to the agent.

This image shows the Kubernetes Pod Troubleshooter agent with the three tools: pod-restarts, eval-pod-cpu and eval-pod-memory assigned to it and active.

Step 6: Test the Kubernetes Pod Troubleshooter Agent

To simulate memory pressure the Open Telemetry demo is running inside the cluster. Artificially lowering the memory requests and limits and increasing the service load will cause pods to restart.

To do this to the open telemetry demo in your cluster, follow these steps. 

Reduce the cart service to one replica by scaling the deployment. Once that is complete, change the resources on the deployment by lowering the memory requests and limits as shown in this command:

kubectl -n otel-demo scale deploy/cart --replicas=1
kubectl -n otel-demo set resources deploy/cart -c cart --requests=memory=50Mi --limits=memory=60Mi

The OpenTelemetry demo application comes with a load-generator. This is used to simulate requests to the demo site by modifying the users and spawn rate in the load generator deployment, as shown in this command:

kubectl -n otel-demo set env deploy/load-generator LOCUST_USERS=800 LOCUST_SPAWN_RATE=200 LOCUST_BROWSER_TRAFFIC_ENABLED=false

If you list all of your pods in the cluster or namespace, you should begin to see restarts.

You can now chat with the Kubernetes Pod Troubleshooter agent and ask “Are any of my Kubernetes pods having issues?”.

The screenshot shows the final response from the Kubernetes Pod Troubleshooter agent. It provides a problem summary of its findings from each tool, showing which services were experiencing the most restarts and memory and CPU utilization. 

The threshold interpretations were described in the initial agent instructions, where >60% utilization is a warning (sustained pressure) and >80% utilization is critical (high likelihood of restarts or throttling). This aligns with findings presented by the Kubernetes Pod Troubleshooter agent, where the services that had the highest restarts were all above 90% memory utilization. The agent needs clearly defined threshold values to correctly assess the returned memory and CPU utilization values. 

Problem summary returned by the Kubernetes Pod Troubleshooter agent:

Conclusion and Final Thoughts

Elastic Agent Builder enables fast, repeatable Kubernetes troubleshooting by combining ES|QL-driven analysis with constrained AI reasoning.

The creation of custom tools that use specific ES|QL queries combined with downstream queries that take input variables from the output of previous tools eliminates or reduces error propagation and hallucinations. In comparison to generic AI troubleshooting without purpose-built tools, you run the risk of it analyzing too many services (that aren’t relevant to the issue at hand). This will slow down the thinking process and generate longer responses, increasing the likelihood of error propagation and hallucinations. 

With the Elastic Agent Builder, you can inspect the output of every tool if you need to, to explore and verify the outputs.

Having a succinct problem summary is a game-changer, bringing your attention straight to the most affected services.

Reasoning returned by the Kubernetes Pod Troubleshooter agent:

Not only that, but the agent can go one step further and offer recommendations for remediation based on what outputs the tools delivered.

Remediation recommendation returned by the Kubernetes Pod Troubleshooter agent:

Sign up for Elastic Cloud Serverless and try this out with your Kubernetes clusters.

Frequently Asked Questions

1. When to use the Elastic Agent Builder for Troubleshooting

Use the Elastic Agent Builder for Troubleshooting that works best if:

• You need repeatable, auditable troubleshooting workflows

• You want deterministic analysis instead of free-form AI responses

• You’re investigating something that is reported in the logs or metrics (i.e. pod restarts, OOMKills, or resource pressure)

• You want to reduce mean time to resolution (MTTR)

2. Do I need OpenTelemetry to use Elastic Agent Builder for Kubernetes troubleshooting? 

No, you don’t need to use OpenTelemetry. You have two options:

• You can collect logs and metrics from Kubernetes using the Elastic Agent; or 

• You can collect logs, traces and metrics with the Elastic Distro for OTel (EDOT) Collector

When following the steps above, this would change the field names that are used in the tools above. For example, kubernetes.container.memory.usage.bytes vs metrics.container.memory.usage.

3. Can this agent be adapted for node-level failures? 

Yes, Elastic has hundreds of integrations, including AWS (for EKS), Azure (for AKS), Google Cloud (for GKE), as well as host operating system monitoring.

The queries shown above would be modified to use the correct field.

4. Can these tools be reused in automation workflows? 

Yes, Elastic Workflows can reuse the same scripted automations and AI agents you build in Elastic. An agent can handle the initial analysis and investigation (reducing manual effort), and the workflow can continue with structured steps, such as running Elasticsearch queries, transforming data, branching on conditions and calling external APIs or tools like Slack, Jira and PagerDuty. Workflows can also be exposed to Agent Builder as reusable tools, just like the tool created in this guide.

For more advanced automation from a similar scenario as described in this guide, learn how to integrate AI agents into GitHub Actions to monitor K8s health and improve deployment reliability via Observability.

5. Can these tools be triggered by alerts? 

Yes, alerts can trigger Elastic Workflows, and pass the alert context to the workflow. This workflow may be integrated with an Elastic Agent, as described above.

Additionally, Elastic Alerts allow you to publish investigation guides alongside alerts so an SRE has all of the information they need to begin investigating. Any troubleshooting or investigative agents can be linked to from the investigation guide, meaning the SRE doesn’t have to follow manual processes outlined in an investigation guide and instead let the agent handle the manual, repetitive investigations.

6. How can I get started with Agent Builder?

Sign up for Elastic Cloud Serverless, a new fully managed, stateless architecture that auto-scales no matter your data, usage, and performance needs.

Elastic's Site Reliability Engineering team (SRE) observed unstable throughput and packet loss in Elastic Cloud Serverless running on Azure Kubernetes Service (AKS). After investigation, we identified the primary contributing factors to be RX ring buffer overflows and kernel input queue saturation on SR-IOV interfaces. To address this, we increased RX buffer sizes and adjusted the netdev backlog, which significantly improved network stability.

Setting the Scene

Elastic Cloud Serverless is a fully managed solution that allows you to deploy and use Elastic for your use cases without managing the underlying infrastructure. Built on Kubernetes, it represents a shift in how you interact with Elasticsearch. Instead of managing clusters, nodes, data tiers, and scaling, you create serverless projects that are fully managed and automatically scaled by Elastic. This abstraction of infrastructure decisions allows you to focus solely on gaining value and insight from your data.

Elastic Cloud Serverless is generally available (GA) on AWS, GCP and currently in Technical Preview on Azure. As part of preparing Elastic Cloud Serverless GA on Azure, we have been conducting extensive performance and scalability tests to ensure that our users get a consistent and reliable user experience.

In this post, we’ll take you behind the scenes of a deep technical investigation into a surprising performance issue that affected Serverless Elasticsearch in our Azure Kubernetes clusters. At first, the network seemed like the least likely place to look, especially with a high-speed 100 Gb/s interface on the host backing it. But as we dug deeper, with help from the Microsoft Azure team, that’s exactly where the problem led us.

Unexpected Results!

While the high-level architectures and system design patterns of the major cloud provider’s systems are often similar, the implementations are different, and these differences can have dramatic impacts on a system’s performance characteristics.

One of the most significant differences between the different cloud providers is that the underlying hypervisor software and server hardware of the Virtual Machines can vary significantly, even between instance families of the same provider.

There is no way to fully abstract the hardware away from an application like Elasticsearch. Fundamentally, its performance is dictated by the CPU, memory, disks, and network interfaces on the physical server. In preparation for the Elastic Cloud Serverless GA on Azure, our Elasticsearch Performance team kicked off large-scale load testing against Serverless Elasticsearch projects running on Azure Kubernetes Service (AKS), using ARM-based VMs (we’re big fans!). Throughout this process, we relied heavily on Elastic tools to analyse system behaviour, identify bottlenecks, and validate performance under load.

To perform these scale and load tests, the Elasticsearch Performance team use Rally, an open-source benchmarking tool designed to measure the performance of Elasticsearch clusters. The workload (or in Rally nomenclature, ‘Track’) used for these tests was the GitHub Archive Track. Rally collects and sends test telemetry using the official Python client to a separate Elasticsearch cluster running Elastic Observability, which allows for monitoring and analysis during these scale and load tests in real time via Kibana.

When we looked at the results, we observed that the indexing rate (the number of docs/s) for the Serverless projects was not only much lower than we had expected for the given hardware, but the throughput was also quite unstable. There were peaks and valleys, interspersed with frequent errors, whereas we were instead expecting a stable indexing rate for the duration of the test.

These tests are designed to push the system to its limits, and in doing so, they surfaced unexpected behavior in the form of unstable indexing throughput and intermittent errors. This was precisely the kind of problem we'd hoped to uncover prior to going GA — giving us the opportunity to work closely with Azure.

A Kibana visualisation of Rally telemetry, showing fluctuating Elasticsearch indexing rates alongside spikes in 5xx and 4xx HTTP error responses.

Debugging!

Debugging performance issues can feel a little bit like trying to find a ‘Butterfly in a Hurricane’, so it’s crucial that you take a methodological approach to analysing application and system performance.

Using methodologies helps you to be more consistent and thorough in your debugging, and avoids missing things. We started with the Utilisation Saturation and Errors (USE) Method, looking at both the client and server side to identify any obvious bottlenecks in the system.

Elastic's Site Reliability Engineers (SREs) maintain a suite of custom Elastic Observability dashboards designed to visualise data collected from various Elastic Integrations. These dashboards provide deep visibility into the health and performance of Elastic Cloud infrastructure and systems.

For this investigation, we leveraged a custom dashboard built using metrics and log data from the System and Linux Integrations:

One of many Elastic Observability dashboards built and maintained by the SRE team.

Following the USE Method, these dashboards highlight resource utilisation, saturation, and errors across our systems. With their help, we quickly identified that the AKS nodes hosting the Elasticsearch pods under test were dropping thousands of packets per second.

A Kibana visualisation of Elastic Agent's System Integration, showing the rate of packet drops per second for AKS nodes.

Dropping packets forces reliable protocols, such as TCP, to retransmit any missing packets. These retransmissions can introduce significant delays, which kills the throughput of any system where client requests are only triggered upon the previous request completion (known as a Closed System).

To investigate further, we jumped onto one of the AKS nodes exhibiting the packet loss to check the basics. First off, we wanted to identify what type of packet drops or errors we’re seeing; is it for specific pods, or the host as a whole?

root@aks-k8s-node-1:~# ip -s link show
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 7c:1e:52:be:ce:5e brd ff:ff:ff:ff:ff:ff
    RX:    bytes   packets errors dropped  missed   mcast
    373507935420 134292481      0       0       0      15
    TX:    bytes   packets errors dropped carrier collsns
    644247778936 303191014      0       0       0       0
3: enP42266s1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master eth0 state UP mode DEFAULT group default qlen 1000
    link/ether 7c:1e:52:be:ce:5e brd ff:ff:ff:ff:ff:ff
    RX:    bytes   packets errors dropped  missed   mcast
    386782548951 307000571      0       0 5321081       0
    TX:    bytes   packets errors dropped carrier collsns
    655758630548 477594747      0       0       0       0
    altname enP42266p0s2
15: lxc0ca0ec41ecd2@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether f6:f5:5e:c9:4e:fb brd ff:ff:ff:ff:ff:ff link-netns cni-3f90ab53-df66-cac5-bd19-9cea4a68c29b
    RX:    bytes   packets errors dropped  missed   mcast
    627954576078  54297550      0    1600       0       0
    TX:    bytes   packets errors dropped carrier collsns
    372155326349 133538064      0    3927       0       0

In this output you can see the enP42266s1 interface is showing a significant number of packets in the missed column. That’s interesting, sure, but what does missed actually represent? And what is enP42266s1?

To understand, let’s look at roughly what happens when a packet arrives at the NIC:

1. A packet arrives at the NIC from the network.
2. The NIC uses DMA (Direct Memory Access) to place the packet into a receive ring buffer allocated in memory by the kernel, mapped for use by the NIC. Since our NICs supports multiple hardware queues, each queue has its own dedicated ring buffer, IRQ, and NAPI context.
3. The NIC raises a hardware interrupt (IRQ) to notify the CPU that a packet is ready.
4. The CPU runs the NIC driver’s IRQ handler. The driver schedules a NAPI (New API) poll to defer packet processing to a softirq context. A mechanism in the Linux kernel that defers work to be processed outside of the hard IRQ context, for better batching and CPU efficiency, enabling improved scalability.
5. The NAPI poll function is executed in a softirq context (NET_RX_SOFTIRQ) and retrieves packets from the ring buffer. This polling continues either until the driver’s packet budget is exhausted (net.core.netdev_budget) or the time limit is hit (net.core.netdev_budget_usecs).
6. Each packet is wrapped in an sk_buff (socket buffer) structure, which includes metadata such as protocol headers, timestamps, and interface identifiers.
7. If the networking stack is slower than the rate at which NAPI fetches packets, excess packets are queued in a per-CPU backlog queue (via enqueue_to_backlog). The maximum size of this backlog is controlled by the net.core.netdev_max_backlog sysctl.
8. Packets are then handed off to the kernel’s networking stack for routing, filtering, and protocol-specific processing (e.g. TCP, UDP).
9. Finally, packets reach the appropriate socket receive buffer, where they are available for consumption by the user-space application.

Visualised, it looks something like this:

Image © 2018 Leandro Moreira. Used under the BSD 3-Clause License. Source: GitHub repository.

The missed counter is incremented whenever the NIC tries to DMA a packet into a fully occupied ring buffer. The NIC essentially "misses" the chance to deliver the packet to the VM’s memory. However, what’s most interesting is that this counter seldom increments for VMs. This is because Virtual NICs are usually implemented as software via the hypervisor, which typically has much more flexible memory management compared to the physical NICs and can reduce the chance of ring buffer overflow.

We mentioned earlier that we’re building Azure Elasticsearch Serverless on top of Azure’s AKS service, which is important to note because all of our AKS nodes use an Azure feature called Accelerated Networking. In this setup, network traffic is delivered directly to the VM’s network interface, bypassing the hypervisor. This is enabled by single root I/O virtualization (SR-IOV), which offers much lower latency and higher throughput than traditional VM networking. Each node is physically connected to a 100 Gb/s network interface, although the SR-IOV Virtual Function (VF) exposed to the VM typically provides only a fraction of that total bandwidth.

Despite the VM only having a fraction of the 100 Gb/s bandwidth, microbursts are still very possible. These physical interfaces are so fast that they can transmit and receive multiple packets in just nanoseconds, far faster than most buffers or processing queues can absorb. At these timescales, even a short-lived burst of traffic can overwhelm the receiver, leading to dropped packets and unpredictable latency.

Direct access to the SR-IOV interface means that our VMs are responsible for handling the hardware interrupts triggered by the NIC in a timely manner, if there's any delay in handling the hardware interrupt (e.g. waiting to be scheduled onto CPU by the hypervisor) then network packets can be missed!

Firstly - NIC-level Tuning

Since we'd confirmed that our VMs were using SR-IOV, we established that the enP42266s1 and eth0 interfaces were a bonded pair and acted as a single interface. Knowing this, then we reasoned that we should be able to adjust the ring buffer values directly using ethtool.

root@aks-k8s-node-1:~# ethtool -g enP42266s1
Ring parameters for enP42266s1:
Pre-set maximums:
RX:        8192
RX Mini:    n/a
RX Jumbo:    n/a
TX:        8192
Current hardware settings:
RX:        1024
RX Mini:    n/a
RX Jumbo:    n/a
TX:        1024

In the output above, we were using only 1/8th of the available ring buffer descriptors. These values were set by the OS defaults, which generally aim to balance performance and resource usage. Set too low, they risk packet drops under load; set too high, they can lead to unnecessary memory consumption. We knew that the VMs were backed by a virtual function carved out of the directly attached 100 Gb/s network interface, which is fast enough to deliver microbursts that could easily overwhelm small buffers. To better absorb those short, high-intensity bursts of traffic, we increased the NIC’s RX ring buffer size from 1024 to 8192. Using a privileged DaemonSet, we rolled out the change across all of our AKS nodes by installing a udev rule to automatically increase the buffer size:

# Match Mellanox ConnectX network cards and run ethtool to update the ring buffer settings
ENV{INTERFACE}=="en*", ENV{ID_NET_DRIVER}=="mlx5_core", RUN+="/sbin/ethtool -G %k rx ${CONFIG_AZURE_MLX_RING_BUFFER_SIZE} tx ${CONFIG_AZURE_MLX_RING_BUFFER_SIZE}"

A Kibana visualisation of Elastic Agent's System Integration, showing packet loss reduced by ~99% after increasing the NIC's RX ring buffer values.

As soon as the change had been applied to all AKS nodes we stopped ‘missing’ RX packets! Fantastic! As a result of this simple change we observed a significant improvement in our indexing throughput and stability.

A Kibana visualisation of Rally telemetry, showing stable and improved Elasticsearch indexing rates after increasing the RX ring buffer size.

Job done, right? Not quite..

Further improvements - Kernel-level Tuning

Eagle eyed readers may have noticed two things:

1. In the previous screenshot, despite adjusting the physical RX ring buffer values, we still observed a small number of dropped packets on the TX side.
2. In the original ip link -s show output, one of the ‘logical’ interfaces used by the Elasticsearch pod was showing dropped packets on both the TX and RX sides.

15: lxc0ca0ec41ecd2@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether f6:f5:5e:c9:4e:fb brd ff:ff:ff:ff:ff:ff link-netns cni-3f90ab53-df66-cac5-bd19-9cea4a68c29b
    RX:    bytes   packets errors dropped  missed   mcast
    627954576078  54297550      0    1600       0       0
    TX:    bytes   packets errors dropped carrier collsns
    372155326349 133538064      0    3927       0       0

So, we continued to dig. We’d eliminated ~99% of the packet loss, and the remaining loss rate wasn’t as significant as what we’d started with, but we still wanted to understand why it was occurring even after adjusting the RX ring buffer size of the NIC.

So what does dropped represent, and what is this lxc0ca0ec41ecd2 interface? dropped is similar to missed, but only occurs when packets are deliberately dropped by the kernel or network interface. Crucially though, it doesn’t tell you why a packet was dropped. As for the lxc0ca0ec41ecd2 interface, we use the Azure CNI Powered by Cilium to provide the network functionality to our AKS clusters. Any pod spun up on an AKS node gets a ‘logical’ interface, which is a virtual ethernet (veth) pair that connects the pod’s network namespace with the host’s network namespace. It was here that we were dropping packets.

In our experience, packet drops at this layer are unusual, so we started digging deeper into the cause of the drops. There are numerous ways you can debug why a packet is being dropped, but one of the easiest is to use perf attach to the skb:kfree_skb tracepoint. The "socket buffer" (skb) is the primary data structure used to represent network packets in the Linux kernel. When a packet is dropped, its corresponding socket buffer is usually freed, triggering the kfree_skb tracepoint. Using perf to attach to this event allowed us to capture stack traces to analyze the cause of the drops.

# perf record -g -a -e skb:kfree_skb

We left this to run for ~10 minutes or so to capture as many drops as possible, and then ‘heavily inspired’ by this GitHub Gist by Ivan Babrou, we converted the stack traces into an ‘easier’ to read Flamegraphs:

# perf script | sed -e 's/skb:kfree_skb:.*reason:\(.*\)/\n\tfffff \1 (unknown)/' -e 's/^\(\w\+\)\s\+/kernel /' > stacks.txt
cat stacks.txt | stackcollapse-perf.pl --all | perl -pe 's/.*?;//' | sed -e 's/.*irq_exit_rcu_\[k\];/irq_exit_rcu_[k];/' | flamegraph.pl --colors=java --hash --title=aks-k8s-node-1 --width=1440 --minwidth=0.005 > aks-k8s-node-1.svg

A Flamegraph showing the various stack trace ancestry of packet loss.

The flamegraph here shows how often different functions appeared in stack traces for packets drops. Each box represents a function call and wider boxes mean the function appears more frequently in the traces. The stack's ancestry builds upward from the bottom with earlier calls, to the top with later calls.

Firstly, we quickly discovered that unfortunately the skb_drop_reason enum was only added in Kernel 5.17 (Azure’s Node Image at the time was using 5.15). This meant that there was no single human readable message that told us why the packets were being dropped, instead all we got was NOT_SPECIFIED. To work out why packets were being dropped we needed to do a little sleuthing through the stack traces to work out what code paths were being taken when a packet was dropped.

In the flamegraph above you can see that many of the stack traces include veth driver function calls (e.g. veth_xmit), and many end abruptly with a call to the enqueue_to_backlog function. When many stacks end at the same function (like enqueue_to_backlog) it suggests that function is a common point where packets are being dropped. If you go back to the earlier explanation of what happens when a packet arrives at the NIC, you’ll notice that in step 7 we explained:

7. If the networking stack is slower than the rate at which NAPI fetches packets, excess packets are queued in a per-CPU backlog queue (via enqueue_to_backlog). The maximum size of this backlog is controlled by the net.core.netdev_max_backlog sysctl.

Using the same privileged DaemonSet method for the RX ring buffer adjustment, we set the value of the net.core.netdev_max_backlog adjustable kernel parameter from 1000 to 32768:

/usr/sbin/sysctl -w net.core.netdev_max_backlog=32768

This value was based on the fact we knew the hosts were using a 100 Gb/s SR-IOV NIC, even if the VM was allowed only a fraction of the total bandwidth. We acknowledge that it’s worth revisiting this value in the future to see if it can be better optimised to not waste extraneous memory, but at the time “perfect was the enemy of good”.

We re-ran the load tests and compared the three sets of results we’d collected thus far.

A Kibana visualisation of Rally results, comparing impact to median throughput after each configuration change.

| Tuning Step | Packet Loss | Median indexing throughput | |-------------|-------------|-----------------------------| | Baseline | High | ~18,000 docs/s | | +RX Buffer | ~99% drop ↓ | ~26,000 (+ ~40% from baseline) | | +Backlog & +RX Buffer | Near zero | ~29,000 (+ ~60% from baseline) |

Here you can see the P50 of throughput in docs/s over the course of the hours-long load tests. Compared to the baseline, we saw a roughly ~40% increase in throughput by only adjusting the RX ring buffer values, and a ~50-60% increase with both the RX ring buffer and backlog changes! Hooray!

A great result and one more step on our journey towards better Serverless Elasticsearch performance.

Working with Azure

It’s great that we were able to quickly identify and mitigate the majority of our packet loss issues, but since we were using AKS with AKS node images, it made sense to engage with Azure to understand why the defaults weren’t working for our workload.

We walked Azure through our investigation, mitigations and results, and asked for some additional validation of our mitigations. Azure Engineering confirmed that the host NICs were not discarding packets, which confirmed that everything arriving at the host level was passed through to the hypervisor on the host. Further investigation confirmed that no loss or discards were occurring to Azure network fabric, or internal to the hypervisor – which shifted focus from the host to the guest OS and why the guest OS kernel was slow when reading packets off of the enP* SR-IOV interfaces.

Given the complexity of our load testing scenario — which involved configuring multiple systems and tools, including Elastic Observability, we also developed a simplified reproduction of the packet loss issue using iperf3. This simplified test was created specifically to share with Azure for targeted analysis, and added to the broader monitoring and analysis enabled by Elastic Observability and Rally.

With this reproduction Azure was able to confirm the increasing missed and dropped packet counters we had observed, and confirmed the increased RX ring buffer and netdev_max_backlog increase as the recommended mitigations.

Conclusion

While cloud providers offer various abstractions to manage your resources, the underlying hardware ultimately determines your application's performance and stability. High-performance hardware often requires tuning at the operating system level, well beyond the default settings most environments ship with. In managed platforms like AKS, where Azure controls both the node images and infrastructure, it is easy to overlook the impact of low-level configurations such as network device ring buffer sizes or sysctls like net.core.netdev_max_backlog.

Our experience shows that even with the convenience of a managed Kubernetes service, performance issues can still emerge if these hardware parameters are not tuned appropriately. It was tempting to assume that high-speed 100 Gb/s network interfaces, directly attached to the VM using SR-IOV would eliminate any chance of network-related bottlenecks. In reality, that assumption didn’t hold up.

Engaging early with Azure was essential, as they provided deeper visibility into the underlying infrastructure and worked with us to tune low-level, performance-critical settings. Combined with thorough load and scale testing and robust observability using tools like Elastic Observability, this collaboration helped us detect and rectify the issue early in order to deliver a consistent, reliable, and high-performing experience for our users.

OpenTelemetry recently published a blog on how to do Autodiscovery based on Kubernetes Pods' annotations with the OpenTelemetry Collector.

In this blog post, we will talk about how to use this Kubernetes-related feature of the OpenTelemetry Collector, which is already available with the Elastic Distribution of the OpenTelemetry (EDOT) Collector.

In addition to this feature, at Elastic, we heavily invest in making OpenTelemetry the best, standardized ingest solution for Observability. You might already have seen us focusing on:

• Semantic Conventions standardization

• significant log collection improvements

• various other topics around instrumentation

• profiling

Let's walk you through a hands-on journey using the EDOT Collector covering various use cases you might encounter in the real world, highlighting the capabilities of this powerful feature.

Configuring EDOT Collector

The Collector’s configuration is not our main focus here, since based on the nature of this feature it is minimal, letting workloads define how they should be monitored.

To illustrate the point, here is the Collector configuration snippet that enables the feature for both logs and metrics:

receivers:
    receiver_creator/metrics:
      watch_observers: [k8s_observer]
      discovery:
        enabled: true
      receivers:

    receiver_creator/logs:
      watch_observers: [k8s_observer]
      discovery:
        enabled: true
      receivers:

You can include the above in the EDOT’s Collector configuration, specifically the receivers’ section.

Since logs collection in our examples will happen from the discovery feature make sure that the static filelog receiver configuration block is removed and its preset is disabled (i.e. set to false) to avoid having log duplication.

Make sure that the receiver creator is properly added in the pipelines for logs (in addition to removing the filelog receiver completely) and metrics respectively.

Ensure that k8sobserver is enabled as part of the extensions:

extensions:
  k8s_observer:
    observe_nodes: true
    observe_services: true
    observe_ingresses: true

// ...

service:
  extensions: [k8s_observer]

Last but not least, ensure the log files' volume is mounted properly:

volumeMounts:
 - name: varlogpods
   mountPath: /var/log/pods
   readOnly: true

volumes:
  - name: varlogpods
    hostPath:
      path: /var/log/pods

Once the configuration is ready follow the Kubernetes quickstart guides on how to deploy the EDOT Collector. Make sure to replace the values.yaml file linked in the quickstart guide with the file that includes the above-described modifications.

Collecting Metrics from Moving Targets Based on Their Annotations

In this example, we have a Deployment with a Pod spec that consists of two different containers. One container runs a Redis server, while the other runs an NGINX server. Consequently, we want to provide different hints for each of these target containers.

The annotation-based discovery feature supports this, allowing us to specify metrics annotations per exposed container port.

Here is how the complete spec file looks:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    user  nginx;
    worker_processes  1;
    error_log  /dev/stderr warn;
    pid        /var/run/nginx.pid;
    events {
      worker_connections  1024;
    }
    http {
      include       /etc/nginx/mime.types;
      default_type  application/octet-stream;

      log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
      access_log  /dev/stdout main;
      server {
          listen 80;
          server_name localhost;

          location /nginx_status {
              stub_status on;
          }
      }
      include /etc/nginx/conf.d/*;
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deployment
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
      annotations:
        # redis container port hints
        io.opentelemetry.discovery.metrics.6379/enabled: "true"
        io.opentelemetry.discovery.metrics.6379/scraper: redis
        io.opentelemetry.discovery.metrics.6379/config: |
          collection_interval: "20s"
          timeout: "10s"

        # nginx container port hints
        io.opentelemetry.discovery.metrics.80/enabled: "true"
        io.opentelemetry.discovery.metrics.80/scraper: nginx
        io.opentelemetry.discovery.metrics.80/config: |
          endpoint: "http://`endpoint`/nginx_status"
          collection_interval: "30s"
          timeout: "20s"
    spec:
      volumes:
      - name: nginx-conf
        configMap:
          name: nginx-conf
          items:
            - key: nginx.conf
              path: nginx.conf
      containers:
        - name: webserver
          image: nginx:latest
          ports:
            - containerPort: 80
              name: webserver
          volumeMounts:
            - mountPath: /etc/nginx/nginx.conf
              readOnly: true
              subPath: nginx.conf
              name: nginx-conf
        - image: redis
          imagePullPolicy: IfNotPresent
          name: redis
          ports:
            - name: redis
              containerPort: 6379
              protocol: TCP

When this workload is deployed, the Collector will automatically discover it and identify the specific annotations. After this, two different receivers will be started, each one responsible for each of the target containers.

Collecting Logs from Multiple Target Containers

The annotation-based discovery feature also supports log collection based on the provided annotations. In the example below, we again have a Deployment with a Pod consisting of two different containers, where we want to apply different log collection configurations. We can specify annotations that are scoped to individual container names:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-logs-deployment
  labels:
    app: busybox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
      annotations:
        io.opentelemetry.discovery.logs.lazybox/enabled: "true"
        io.opentelemetry.discovery.logs.lazybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints-lazybox
        io.opentelemetry.discovery.logs.busybox/enabled: "true"
        io.opentelemetry.discovery.logs.busybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints-busybox
    spec:
      containers:
        - name: busybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs from busybox at $(date +%H:%M:%S)" && sleep 5s; done
        - name: lazybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs from lazybox at $(date +%H:%M:%S)" && sleep 25s; done

The above configuration enables two different filelog receiver instances, each applying a unique parsing configuration. This is handy when we know how to parse specific technology logs, such as Apache server access logs.

Combining Both Metrics and Logs Collection

In our third example, we illustrate how to define both metrics and log annotations on the same workload. This allows us to collect both signals from the discovered workload. Below is a Deployment with a Pod consisting of a Redis server and a BusyBox container that performs dummy log writing. We can target annotations to the port and container levels to collect metrics from the Redis server using the Redis receiver, and logs from the BusyBox using the filelog receiver. Here’s how:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deployment
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
      annotations:
        io.opentelemetry.discovery.metrics.6379/enabled: "true"
        io.opentelemetry.discovery.metrics.6379/scraper: redis
        io.opentelemetry.discovery.metrics.6379/config: |
          collection_interval: "20s"
          timeout: "10s"

        io.opentelemetry.discovery.logs.busybox/enabled: "true"
        io.opentelemetry.discovery.logs.busybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints
    spec:
      containers:
        - image: redis
          imagePullPolicy: IfNotPresent
          name: redis
          ports:
            - name: redis
              containerPort: 6379
              protocol: TCP
        - name: busybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs at $(date +%H:%M:%S)" && sleep 15s; done

Explore and analyse data coming from dynamic targets in Elastic

Once the target Pods are discovered and the Collector has started collecting telemetry data from them, we can then explore this data in Elastic. In Discover we can search for Redis and NGINX metrics as well as logs collected from the Busybox container. Here is how it looks like:

Summary

The examples above showcase how users of our OpenTelemetry Collector can take advantage of this new feature — one we played a major role in developing.

For this, we leveraged our years of experience with similar features already supported in Metricbeat, Filebeat, and Elastic-Agent. This makes us extremely happy and confident, as it closes the feature gap between Elastic's specific monitoring agents and the OpenTelemetry Collector — making it even better.

Interested in learning more? Visit the documentation and give it a try by following our EDOT quickstart guide.

Elastic provides the ability to ingest OpenTelemetry data with native OTel SDKs, the off the shelf OTel collector, or even Elastic’s Distributions of OpenTelemetry (EDOT). EDOT enables you to bring in logs, metrics and traces for your GenAI application and for K8s. However you will also generally need libraries to help trace specific components in your application. In tracing GenAI applications you can pick from a large set of libraries.

• OpenTelemetry OpenAI Instrumentation-v2 - allows tracing LLM requests and logging of messages made by the OpenAI Python API library. (note v2 is built by OpenTelemetry, the non v2 version is from a specific vendor and not OpenTelemetry)

• OpenTelemetry VertexAI Instrumentation - allows tracing LLM requests and logging of messages made by the VertexAI Python API library

• Langtrace - commercially available library which supports all LLMs in one library, and all traces are also OTel native.

• Elastic’s EDOT - which recently added tracing. See blog.

As you can see OpenTelemetry is the defacto mechanism that is converging to collect and ingest. OpenTelemetry is growing its support for this but it is also early days.

In this blog, we will walk through how to, with minimal code, observe a RAG based chatbot application with tracing using Langtrace. We previously covered Langtrace in a blog to highlight tracing Langchain.

In this blog we used langtrace OpenAI, Amazon Bedrock, Cohere, and others in one library.

Pre-requisites:

In order to follow along, these few pre-requisites are needed

• An Elastic Cloud account — sign up now, and become familiar with Elastic’s OpenTelemetry configuration. With Serverless no version required. With regular cloud minimally 8.17

• Git clone the RAG based Chatbot application and go through the tutorial on how to bring it up and become more familiar.

• An account on your favorite LLM (OpenAI, AzureOpen AI, etc), with API keys

• Be familiar with EDOT to understand how we bring in logs, metrics, and traces from the application through the OTel Collector

• Kubernetes cluster - I’ll be using Amazon EKS

• Look at Langtrace documentation also.

Application OpenTelemetry output in Elastic

Chatbot-rag-app

The first item that you will need to get up and running is the ChatBotApp, and once up you should see the following:

As you select some of the questions you will set a response based on the index that was created in Elasticsearch when the app initializes. Additionally there will be queries that are made to LLMs.

Traces, logs, and metrics from EDOT in Elastic

Once you have OTel Collector with EDOT configuration on your K8s cluster, and Elastic Cloud up and running you should see the following:

Logs:

In Discover you will see logs from the Chatbotapp, and be able to analyze the application logs, any specific log patterns (saves you time in analysis), and view logs from K8s.

Traces:

In Elastic Observability APM, you can also see tha chatbot details, which include transactions, dependencies, logs, errors, etc.

When you look at traces, you will be able to see the chatbot interactions in the trace.

1. You will see the end to end http call

2. Individual calls to elasticsearch

3. Specific calls such as invoke actions, and calls to the LLM

You can also get individual details of the traces, and look at related logs, and metrics related to that trace,

Metrics:

In addition to logs, and traces, any instrumented metrics will also get ingested into Elastic.

Setting it all up

In order to properly set up the Chatbot-app on K8s with telemetry sent over to Elastic, a few things must be set up:

1. Git clone the chatbot-rag-app, and modify one of the python files.

2. Next create a docker container that can be used in Kubernetes. The Docker build here in the Chatbot-app is good to use.

3. Collect all needed env variables. In this example we are using OpenAI, but the files can be modified for any of the LLMs. Hence you will have to get a few environmental variables loaded into the cluster. In the github repo there is a env.example for docker. You can pick and chose what is needed or not needed and adjust appropriately in the K8s file below.

4. Set up your K8s Cluster, and then install the OpenTelemetry collector with the appropriate yaml file and credentials. This will help collect K8s cluster logs and metrics also.

5. Utilize the two yaml files listed below to ensure you can run it on Kubernetes.

• Init-index-job.yaml - Initiates the index in elasticsearch with the local corporate information

• k8s-deployment-chatbot-rag-app.yaml - initializes the application frontend and backend.

1. Open the app on the load balancer URL against the chatbot-app service in K8s

2. Go to Elasticsearch and look at Discover for logs, go to APM and look for your chatbot-app and review the traces, and finally.

Modify the code for tracing with Langtrace

Once you curl the app and untar, go to the chatbot-rag-app directory:

curl https://codeload.github.com/elastic/elasticsearch-labs/tar.gz/main | 
tar -xz --strip=2 elasticsearch-labs-main/example-apps/chatbot-rag-app
cd elasticsearch-labs-main/example-apps/chatbot-rag-app

Next open the app.py file in the api directory and add the following

from opentelemetry.instrumentation.flask import FlaskInstrumentor

from langtrace_python_sdk import langtrace

langtrace.init(batch=False)

FlaskInstrumentor().instrument_app(app)

into the code:

import os
import sys
from uuid import uuid4

from chat import ask_question
from flask import Flask, Response, jsonify, request
from flask_cors import CORS

from opentelemetry.instrumentation.flask import FlaskInstrumentor

from langtrace_python_sdk import langtrace

langtrace.init(batch=False)

app = Flask(__name__, static_folder="../frontend/build", static_url_path="/")
CORS(app)

FlaskInstrumentor().instrument_app(app)

@app.route("/")

See the items in BOLD which will add in the langtrace library, and the opentelemetry flask instrumentation. This combination will provide and end to end trace for the https call all the way down to the calls to Elasticsearch, and to OpenAI (or other LLMs).

Create the docker container

Use the Dockerfile that is in the chatbot-rag-app directory as is and add the following line:

RUN pip3 install --no-cache-dir langtrace-python-sdk

into the Dockerfile:

COPY requirements.txt ./requirements.txt
RUN pip3 install -r ./requirements.txt
RUN pip3 install --no-cache-dir langtrace-python-sdk
COPY api ./api
COPY data ./data

EXPOSE 4000

This enables the langtrace-python-sdk to be installed into the docker container so the langtrace libraries can be used properly.

Collecting the proper env variables:

First collect the env variables from Elastic:

Envs for index initialization in Elastic:

ELASTICSEARCH_URL=https://aws.us-west-2.aws.found.io
ELASTICSEARCH_USER=elastic
ELASTICSEARCH_PASSWORD=elastic

# The name of the Elasticsearch indexes
ES_INDEX=workplace-app-docs
ES_INDEX_CHAT_HISTORY=workplace-app-docs-chat-history

The ELASTICSEARCH_URL can be found in cloud.elastic.co when you bring up your instance. The user and password, you will need to setup in Elastic.

Envs for sending the OTel instrumentation you will need the following:

OTEL_EXPORTER_OTLP_ENDPOINT="https://123456789.apm.us-west-2.aws.cloud.es.io:443"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer xxxxx"

These credentials are found in Elastic under APM integration and under OpenTelemetry

Envs for LLMs

In this example we’re using OpenAI, hence only three variables are needed.

LLM_TYPE=openai
OPENAI_API_KEY=XXXX
CHAT_MODEL=gpt-4o-mini

All these variables will be needed in the Kubernetes yamls in the next step

Setup K8s cluster and load up OTel Collector with EDOT

This step is outlined in the following Blog. It’s a simple three step process.

This step will bring in all the K8s cluster logs and metrics and setup the OTel collector.

Setup secrets, initialize indices, and start the app

Now that the cluster is up, and you have your environmental variables, you will need to

1. Install and run the k8s-deployments.yaml with the variables

2. Initialize the index

Essentially run the following:

kubectl create -f k8s-deployment.yaml
kubectl create -f init-index-job.yaml

Here are the two yamls you should use. Also found here

k8s-deployment.yaml

apiVersion: v1
kind: Secret
metadata:
  name: genai-chatbot-langtrace-secrets
type: Opaque
stringData:
  OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer%20xxxx"
  OTEL_EXPORTER_OTLP_ENDPOINT: "https://1234567.apm.us-west-2.aws.cloud.es.io:443"
 ELASTICSEARCH_URL: "YOUR_ELASTIC_SEARCH_URL"
  ELASTICSEARCH_USER: "elastic"
  ELASTICSEARCH_PASSWORD: "elastic"
  OPENAI_API_KEY: "XXXXXXX"  

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: genai-chatbot-langtrace
spec:
  replicas: 2
  selector:
    matchLabels:
      app: genai-chatbot-langtrace
  template:
    metadata:
      labels:
        app: genai-chatbot-langtrace
    spec:
      containers:
      - name: genai-chatbot-langtrace
        image:65765.amazonaws.com/genai-chatbot-langtrace2:latest
        ports:
        - containerPort: 4000
        env:
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: OTEL_SDK_DISABLED
          value: "false"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=genai-chatbot-langtrace,service.version=0.0.1,deployment.environment=dev"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "http/protobuf"
        envFrom:
        - secretRef:
            name: genai-chatbot-langtrace-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

---
apiVersion: v1
kind: Service
metadata:
  name: genai-chatbot-langtrace-service
spec:
  selector:
    app: genai-chatbot-langtrace
  ports:
  - port: 80
    targetPort: 4000
  type: LoadBalancer

Init-index-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: init-elasticsearch-index-test
spec:
  template:
    spec:
      containers:
      - name: init-index
#update your image location for chatbot rag app
        image: your-image-location:latest
        workingDir: /app/api
        command: ["python3", "-m", "flask", "--app", "app", "create-index"]
        env:
        - name: FLASK_APP
          value: "app"
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: ES_INDEX
          value: "workplace-app-docs"
        - name: ES_INDEX_CHAT_HISTORY
          value: "workplace-app-docs-chat-history"
        - name: ELASTICSEARCH_URL
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_URL
        - name: ELASTICSEARCH_USER
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_USER
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_PASSWORD
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
      restartPolicy: Never
  backoffLimit: 4

Open App with LoadBalancer URL

Run the kubectl get services command and get the URL for the chatbot app

% kubectl get services
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                                                                     AGE
chatbot-langtrace-service            LoadBalancer   10.100.130.44    xxxxxxxxx-1515488226.us-west-2.elb.amazonaws.com   80:30748/TCP                                                                6d23h

Play with app and review telemetry in Elastic

Once you go to the URL, you should see all the screens we described earlier in the beginning of this blog.

Conclusion

With Elastic's Chatbot-rag-app you have an example of how to build out a OpenAI driven RAG based chat application. However, you still need to understand how well it performs, whether its working properly, etc. Using OTel, Elastic’s EDOT and Langtrace gives you the ability to achieve this. Additionally, you will generally run this application on Kubernetes. Hopefully this blog provides the outline of how to achieve this.

Here are the other Tracing blogs:

App Observability with LLM (Tracing)-

• Observing LangChain with Langtrace and OpenTelemetry

• Observing LangChain with OpenLit Tracing

• Instrumenting LangChain with OpenTelemetry

LLM Observability -

• Elevate LLM Observability with GCP Vertex AI Integration

• LLM Observability on AWS Bedrock

• LLM Observability for Azure OpenAI

• LLM Observability for Azure OpenAI v2

This app is also now insturmented with EDOT, and you can visualize the Chatbot's traces to OpenAI, as well as relevant logs, and metrics from the application. By running the app as instructed in the github repo with Docker you can see these traces on a local stack. But how about running it against serverless, Elastic cloud or even with Kubernetes?

In this blog we will walk through how to set up Elastic's RAG Based Chatbot application with Elastic cloud and Kubernetes.

Prerequisites:

In order to follow along, these few pre-requisites are needed

• An Elastic Cloud account — sign up now, and become familiar with Elastic's OpenTelemetry configuration. With Serverless no version required. With regular cloud minimally 8.17

• Git clone the RAG based Chatbot application and go through the tutorial on how to bring it up and become more familiar and how to bring up the application using Docker.

• An account on OpenAI with API keys

• Kubernetes cluster to run the RAG based Chatbot app

• The instructions in this blog are also found in observability-examples in github.

Application OpenTelemetry output in Elastic

Chatbot-rag-app

The first item that you will need to get up and running is the ChatBotApp, and once up you should see the following:

As you select some of the questions you will set a response based on the index that was created in Elasticsearch when the app initializes. Additionally there will be queries that are made to LLMs.

Traces, logs, and metrics from EDOT in Elastic

Once you have the application running on your K8s cluster or with Docker, and Elastic Cloud up and running you should see the following:

Logs:

In Discover you will see logs from the Chatbotapp, and be able to analyze the application logs, any specific log patterns, which saves you time in analysis.

Traces:

In Elastic Observability APM, you can also see tha chatbot details, which include transactions, dependencies, logs, errors, etc.

When you look at traces, you will be able to see the chatbot interactions in the trace.

1. You will see the end to end http call

2. Individual calls to elasticsearch

3. Specific calls such as invoke actions, and calls to the LLM

You can also get individual details of the traces, and look at related logs, and metrics related to that trace,

Metrics:

In addition to logs, and traces, any instrumented metrics will also get ingested into Elastic.

Setting it all up with Docker

In order to properly set up the Chatbot-app on Docker with telemetry sent over to Elastic, a few things must be set up:

1. Git clone the chatbot-rag-app

2. Modify the env file as noted in the github README with the following exception:

Use your Elastic cloud's OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADER instead.

You can find these in the Elastic Cloud under integrations->APM

Envs for sending the OTel instrumentation you will need the following:

OTEL_EXPORTER_OTLP_ENDPOINT="https://123456789.apm.us-west-2.aws.cloud.es.io:443"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20xxxxx"

Notice the %20 in the headers. This will be needed to account for the space in credentials.

1. Set the following to false - OTEL_SDK_DISABLED=false

2. Set the envs for LLMs

In this example we're using OpenAI, hence only three variables are needed.

LLM_TYPE=openai
OPENAI_API_KEY=XXXX
CHAT_MODEL=gpt-4o-mini

1. Run the docker container as noted

docker compose up --build --force-recreate

1. Play with the app at localhost:4000

2. Then log into Elastic cloud and see the output as shown previously.

Run chatbot-rag-app on Kubernetes

In order to set this up, you can follow the following repo on Observability-examples which has the Kubernetes yaml files being used. These will also point to Elastic Cloud.

1. Set up the Kubernetes Cluster (we're using EKS)

2. Get the appropriate ENV variables:

• Find the OTEL_EXPORTER_OTLP_ENDPOINT/HEADER variables as noted in the pervious for Docker.

• Get your OpenAI Key

• Elasticsearch URL, and username and password.

1. Follow the instructions in the following github repo in observability examples to run two Kubernetes yaml files.

Essentially you need only replace the secret variables in k8s-deployment.yaml, and run

kubectl create -f k8s-deployment.yaml
kubectl create -f init-index-job.yaml

The app needs to be running first, then we use the app to initialize Elasticsearch with indices for the app.

Init-index-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: init-elasticsearch-index-test
spec:
  template:
    spec:
      containers:
      - name: init-index
        image: ghcr.io/elastic/elasticsearch-labs/chatbot-rag-app:latest
        workingDir: /app/api
        command: ["python3", "-m", "flask", "--app", "app", "create-index"]
        env:
        - name: FLASK_APP
          value: "app"
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: ES_INDEX
          value: "workplace-app-docs"
        - name: ES_INDEX_CHAT_HISTORY
          value: "workplace-app-docs-chat-history"
        - name: ELASTICSEARCH_URL
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_URL
        - name: ELASTICSEARCH_USER
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_USER
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_PASSWORD
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
      restartPolicy: Never
  backoffLimit: 4

k8s-deployment.yaml

apiVersion: v1
kind: Secret
metadata:
  name: chatbot-regular-secrets
type: Opaque
stringData:
  ELASTICSEARCH_URL: "https://yourelasticcloud.es.us-west-2.aws.found.io"
  ELASTICSEARCH_USER: "elastic"
  ELASTICSEARCH_PASSWORD: "elastic"
  OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer%20xxxx"
  OTEL_EXPORTER_OTLP_ENDPOINT: "https://12345.apm.us-west-2.aws.cloud.es.io:443"
  OPENAI_API_KEY: "YYYYYYYY"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chatbot-regular
spec:
  replicas: 2
  selector:
    matchLabels:
      app: chatbot-regular
  template:
    metadata:
      labels:
        app: chatbot-regular
    spec:
      containers:
      - name: chatbot-regular
        image: ghcr.io/elastic/elasticsearch-labs/chatbot-rag-app:latest
        ports:
        - containerPort: 4000
        env:
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=chatbot-regular,service.version=0.0.1,deployment.environment=dev"
        - name: OTEL_SDK_DISABLED
          value: "false"
        - name: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT
          value: "true"
        - name: OTEL_EXPERIMENTAL_RESOURCE_DETECTORS
          value: "process_runtime,os,otel,telemetry_distro"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "http/protobuf"
        - name: OTEL_METRIC_EXPORT_INTERVAL
          value: "3000"
        - name: OTEL_BSP_SCHEDULE_DELAY
          value: "3000"
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

---
apiVersion: v1
kind: Service
metadata:
  name: chatbot-regular-service
spec:
  selector:
    app: chatbot-regular
  ports:
  - port: 80
    targetPort: 4000
  type: LoadBalancer

Open App with LoadBalancer URL

Run the kubectl get services command and get the URL for the chatbot app

% kubectl get services
NAME                                 TYPE           CLUSTER-IP    EXTERNAL-IP                                                               PORT(S)                                                                     AGE
chatbot-regular-service            LoadBalancer   10.100.130.44    xxxxxxxxx-1515488226.us-west-2.elb.amazonaws.com   80:30748/TCP                                                                6d23h

1. Play with app and review telemetry in Elastic

2. Once you go to the URL, you should see all the screens we described earlier in the beginning of this blog.

Conclusion

With Elastic's Chatbot-rag-app you have an example of how to build out a OpenAI driven RAG based chat application. However, you still need to understand how well it performs, whether its working properly, etc. Using OTel and Elastic’s EDOT gives you the ability to achieve this. Additionally, you will generally run this application on Kubernetes. Hopefully this blog provides the outline of how to achieve this. Here are the other Tracing blogs:

App Observability with LLM (Tracing)-

• Observing LangChain with Langtrace and OpenTelemetry

• Observing LangChain with OpenLit Tracing

• Instrumenting LangChain with OpenTelemetry

LLM Observability -

• Elevate LLM Observability with GCP Vertex AI Integration

• LLM Observability on AWS Bedrock

• LLM Observability for Azure OpenAI

• LLM Observability for Azure OpenAI v2

• Native OTel Integrity: Elastic is now 100% OTel-native, retaining OTel data natively without requiring data translation This eliminates the need for SREs to handle tedious schema conversions and develop customized views. All Elastic Observability capabilities—such as entity discovery, entity-centric insights, APM, infrastructure monitoring, and AI-driven issue analysis— now seamlessly work with native OTel data.

• Powerful end to end OTel based Kubernetes observability with Elastic Distributions of OpenTelemetry (EDOT): Elastic now supports EDOT deployment and management on Kubernetes via the OTel Operator, enabling streamlined EDOT collector deployment, application auto-instrumentation, and lifecycle management. With out-of-the-box OTel-based Kubernetes integration and dashboards, SREs gain instant, real-time visibility into cluster and application metrics, logs, and traces—with no manual configuration needed.

For organizations, it signals our commitment to open standards, streamlined data collection, and delivering insights from native OpenTelemetry data. Bring the power of Elastic Observability to your Kubernetes and OpenTelemetry deployments for maximum visibility and performance. 

Fully native OTel architecture with in-depth data analysis

Elastic’s OpenTelemetry-first architecture is 100% OTel-native, fully retaining the OTel data model, including OTel Semantic Conventions and Resource attributes, so your observability data remains in OpenTelemetry standards. OTel data in Elastic is also backward compatible with the Elastic Common Schema (ECS).

SREs now gain a holistic view of resources, as Elastic accurately identifies entities through OTel resource attributes. For example, in a Kubernetes environment, Elastic identifies containers, hosts, and services and connects these entities to logs, metrics, and traces.

Once OTel data is in Elastic’s scalable vector datastore, Elastic’s capabilities such as the AI Assistant, zero-config machine learning-based anomaly detection, pattern analysis, and latency correlation empower SREs to quickly analyze and pinpoint potential issues in production environments.

Kubernetes insights with Elastic Distributions of OpenTelemetry (EDOT)

EDOT reduces manual effort through automated onboarding and pre-configured dashboards. With EDOT and OpenTelemetry, Elastic makes Kubernetes monitoring straightforward and accessible for organizations of any size.

EDOT paired with Elasticsearch,  enables storage for all signal types—logs, metrics, traces, and soon profiling—while maintaining essential resource attributes and semantic conventions.

Elastic’s OpenTelemetry-native solution enables customers to quickly extract insights from their data rather than manage complex infrastructure to ingest data. Elastic automates the deployment and configuration of observability components to deliver a user experience focused on ease and scalability, making it well-suited for large-scale environments and diverse industry needs.

Let’s take a look at how Elastic’s EDOT enables visibility into Kubernetes environments.

1. Simple 3-step OTel ingest with lifecycle management and auto-instrumentation 

Elastic leverages the upstream OpenTelemetry Operator to automate its EDOT lifecycle management—including deployment, scaling, and updates—allowing customers to focus on visibility into their Kubernetes infrastructure and applications instead of their observability infrastructure for data collection.

The Operator integrates with the EDOT Collector and language SDKs to provide a consistent, vendor-agnostic experience. For instance, when customers deploy a new application, they don’t need to manually configure instrumentation for various languages; the OpenTelemetry Operator manages this through auto-instrumentation, as supported by the upstream OpenTelemetry project.

This integration simplifies observability by ensuring consistent application instrumentation across the Kubernetes environment. Elastic’s collaboration with the upstream OpenTelemetry project strengthens this automation, enabling users to benefit from the latest updates and improvements in the OpenTelemetry ecosystem. By relying on open source tools like the OpenTelemetry Operator, Elastic ensures that its solutions stay aligned with the latest advancements in the OpenTelemetry project, reinforcing its commitment to open standards and community-driven development.

The diagram above shows how the operator can deploy multiple OTel collectors, helping SREs deploy individual EDOT Collectors for specific applications and infrastructure. This configuration improves availability for OTel ingest and the telemetry is sent directly to Elasticsearch servers via OTLP.

Check out our recent blog on how to set this up.

2. Out-of-the-box OTel-based Kubernetes integration with dashboards

Elastic delivers an OTel-based Kubernetes configuration for the OTel collector by packaging all necessary receivers, processors, and configurations for Kubernetes observability. This enables users to automatically collect, process, and analyze Kubernetes metrics, logs, and traces without the need to configure each component individually.

The OpenTelemetry Kubernetes Collector components provide essential building blocks, including receivers like the Kubernetes Receiver for cluster metrics, Kubeletstats Receiver for detailed node and container metrics, along with processors for data transformation and enrichment. By packaging these components, Elastic offers a turnkey solution that simplifies Kubernetes observability and eliminates the need for users to set up and configure individual collectors or processors.

This pre-packaged approach, which includes OTel-native Kibana assets such as dashboards, allows users to focus on analyzing their observability data rather than managing configuration details. Elastic’s Unified OpenTelemetry Experience ensures that users can harness OpenTelemetry’s full potential without needing deep expertise. Whether you’re monitoring resource usage, container health, or API server metrics, users gain comprehensive observability through EDOT.

For more details on OpenTelemetry Kubernetes Collector components, visit OpenTelemetry Collector Components.

3. Streamlined ingest architecture with OTel data and Elasticsearch

Elastic’s ingest architecture minimizes infrastructure overhead by enabling users to forward trace data directly into Elasticsearch with the EDOT Collector, removing the need for the Elastic APM server. This approach:

• Reduces the costs and complexity associated with maintaining additional infrastructure, allowing users to deploy, scale, and manage their observability solutions with fewer resources.

• Allows all OTel data, metrics, logs, and traces to be ingested and stored in Elastic’s singular vector database store enabling further analysis with Elastic’s AI-driven capabilities.

SREs can now reduce operational burdens while also gaining high performance analytics and observability insights provided by Elastic.

Elastic’s ongoing commitment to open source and OpenTelemetry

With Elasticsearch fully open source once again under the AGPL license,  this change reinforces our deep commitment to open standards and the open source community. This aligns with Elastic’s OpenTelemetry-first approach to observability, where Elastic Distributions of OpenTelemetry (EDOT) streamline OTel ingestion and schema auto-detection, providing real-time insights for Kubernetes and application telemetry.

As users increasingly adopt OTel as their schema and data collection architecture for observability, Elastic’s Distribution of OpenTelemetry (EDOT), currently in tech preview, enhances standard OpenTelemetry capabilities and improves troubleshooting while also serving as a commercially supported OTel distribution. EDOT, together with Elastic’s recent contributions of the Elastic Profiling Agent and Elastic Common Schema (ECS) to OpenTelemetry, reinforces Elastic’s commitment to establishing OpenTelemetry as the industry standard.

Customers can now embrace open standards and enjoy the advantages of an open, extensible platform that integrates seamlessly with their environment. End result?  Reduced costs, greater visibility, and vendor independence.

Getting hands-on with Elastic Observability and EDOT

Ready to try out the OTel Operator with EDOT collector and SDKs to see how Elastic utilizes ingested OTel data in APM, Discover, Analysis, and out-of-the-box dashboards? 

• Get an account on Elastic Cloud

• Learn about Elastic Distributions of OpenTelemetry Overview

• Utilize the OpenTelemetry Demo with EDOT

• Understand how you can monitor Kubernetes with EDOT

• Utilize the EDOT Operator and the EDOT OTel collector

If you have your own application and want to configure EDOT the application with auto-instrumentation, read the following blogs on Go, Java, PHP, Python

• Auto-Instrumenting Go Applications with OpenTelemetry

• Elastic Distribution OpenTelemetry Java Agent

• Elastic OpenTelemetry Distribution for PHP

• Elastic OpenTelemetry Distribution for Python

In this second part, I show how to install any Java agent via the OpenTelemetry operator, using the Elastic Java agents as examples.

Installation and configuration recap

Part 1 of this series, Zero config OpenTelemetry auto-instrumentation for Kubernetes Java applications, details the installation and configuration of the OpenTelemetry operator and an Instrumentation resource. Here is an outline of the steps as a reminder:

1. Install cert-manager, eg kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml
2. Install the operator, eg kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
3. Create an Instrumentation resource
4. Add an annotation to either the deployment or the namespace
5. Deploy the application as normal

In that first part, steps 3, 4 & 5 were implemented for the OpenTelemetry Java agent. In this blog I’ll implement them for other agents, using the Elastic APM agents as examples. I assume that steps 1 & 2 outlined above have already been done, ie that the operator is now installed. I will continue using the banana namespace for the examples, so ensure that namespace exists (kubectl create namespace banana). As per part 1, if you use any of the example instrumentation definitions below, you’ll need to substitute my.apm.server.url and my-apm-secret-token with the values appropriate for your collector.

Using the Elastic Distribution for OpenTelemetry Java

From version 0.4.0, the Elastic Distribution for OpenTelemetry Java includes the agent jar at the path /javaagent.jar in the docker image - which is essentially all that is needed for a docker image to be usable by the OpenTelemetry operator for auto-instrumentation. This means the Instrumentation resource is straightforward to define, and as it’s a distribution of the OpenTelemetry Java agent, all the OpenTelemetry environment can apply:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: elastic-otel
  namespace: banana
spec:
  exporter:
    endpoint: https://my.apm.server.url
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  java:
    image: docker.elastic.co/observability/elastic-otel-javaagent:1.10.0
    env:
      - name: OTEL_EXPORTER_OTLP_HEADERS
        value: "Authorization=Bearer my-apm-secret-token"
      - name: ELASTIC_OTEL_INFERRED_SPANS_ENABLED
        value: "true"
      - name: ELASTIC_OTEL_SPAN_STACK_TRACE_MIN_DURATION
        value: "50"

I’ve included environment for switching on several features in the agent, including

1. ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED to switch on the inferred spans implementation feature described in this blog
2. Span stack traces are automatically captured if the span takes more than ELASTIC_OTEL_SPAN_STACK_TRACE_MIN_DURATION (default would be 5ms)

Adding in the annotation …

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "elastic-otel"

… to the pod yaml gets the application traced, and displayed in the Elastic APM UI, including the inferred child spans and stack traces

The additions from the features mentioned above are circled in red - inferred spans (for methodC and methodD) bottom left, and the stack trace top right. (Note that the pod included the OTEL_INSTRUMENTATION_METHODS_INCLUDE environment variable set to "test.Testing[methodB]" so that traces from methodB are shown; for pod configuration see the "Trying it" section in part 1)

Using the Elastic APM Java agent

From version 1.50.0, the Elastic APM Java agent includes the agent jar at the path /javaagent.jar in the docker image - which is essentially all that is needed for a docker image to be usable by the OpenTelemetry operator for auto-instrumentation. This means the Instrumentation resource is straightforward to define:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: elastic-apm
  namespace: banana
spec:
  java:
    image: docker.elastic.co/observability/apm-agent-java:1.55.4
    env:
      - name: ELASTIC_APM_SERVER_URL
        value: "https://my.apm.server.url"
      - name: ELASTIC_APM_SECRET_TOKEN
        value: "my-apm-secret-token"
      - name: ELASTIC_APM_LOG_LEVEL
        value: "INFO"
      - name: ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED
        value: "true"
      - name: ELASTIC_APM_LOG_SENDING
        value: "true"

I’ve included environment for switching on several features in the agent, including

• ELASTIC_APM_LOG_LEVEL set to the default value (INFO) which could easily be switched to DEBUG
• ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED to switch on the inferred spans implementation equivalent to the feature described in this blog
• ELASTIC_APM_LOG_SENDING which switches on sending logs to the APM UI, the logs are automatically correlated with transactions (for all common logging frameworks)

Adding in the annotation …

metadata:
  annotations:
     instrumentation.opentelemetry.io/inject-java: "elastic-apm"

… to the pod yaml gets the application traced, and displayed in the Elastic APM UI, including the inferred child spans

(Note that the pod included the ELASTIC_APM_TRACE_METHODS environment variable set to "test.Testing#methodB" so that traces from methodB are shown; for pod configuration see the "Trying it" section in part 1)

Using an extension with the OpenTelemetry Java agent

Setting up an Instrumentation resource for the OpenTelemetry Java agent is straightforward and was done in part 1 of this two part series - and you can see from the above examples it’s just a matter of deciding on the docker image URL you want to use. However if you want to include an extension in your deployment, this is a little more complex, but also supported by the operator. Basically the extensions you want to include with the agent need to be in docker images - or you have to build an image which includes the extensions that are not already in images. Then you declare the images and the directories the extensions are in, in the Instrumentation resource. As an example, I’ll show an Instrumentation which uses version 2.5.0 of the OpenTelemetry Java agent together with the inferred spans extension from the Elastic OpenTelemetry Java distribution. The distro image includes the extension at path /extensions/elastic-otel-agentextension.jar. The Instrumentation resource allows either directories or file paths to be specified, here I’ll list the directory:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: otel-plus-extension-instrumentation
  namespace: banana
spec:
  exporter:
    endpoint: https://my.apm.server.url
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.5.0
    extensions:
      - image: "docker.elastic.co/observability/elastic-otel-javaagent:1.10.0"
        dir: "/extensions"
    env:
      - name: OTEL_EXPORTER_OTLP_HEADERS
        value: "Authorization=Bearer my-apm-secret-token"
      - name: ELASTIC_OTEL_INFERRED_SPANS_ENABLED
        value: "true"

Note that you can have multiple image … dir pairs, ie include multiple extensions from different images. Note also if you are testing this specific configuration that the inferred spans extension included here will be contributed to the OpenTelemetry contrib repo at some point after this blog is published, after which the extension may no longer be present in a later version of the referred image (since it will be available from the contrib repo instead).

Next steps

Here I’ve shown how to use any agent with the OpenTelemetry Operator for Kubernetes, and configure that for your system. In particular the examples have showcased how to use the Elastic Java agents to auto-instrument Java applications running in your Kubernetes clusters, along with how to enable features, using Instrumentation resources. And you can set it up for either zero config for deployments, or for just one annotation which is generally a more flexible mechanism (you can have multiple Instrumentation resource definitions, and the deployment can select the appropriate one for its application).

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The mechanism to achieve zero-config auto-instrumentation of Java applications in Kubernetes is via the OpenTelemetry Operator for Kubernetes. This operator has many capabilities and the full documentation (and of course source) is available in the project itself. In this blog, I'll walk through installing, setting up and running zero-config auto-instrumentation of Java applications in Kubernetes using the OpenTelemetry Operator.

Installing the OpenTelemetry Operator

At the time of writing this blog, the OpenTelemetry Operator needs the certification manager to be installed, after which the operator can be installed. Installing from the web is straightforward. First install the cert-manager (the version to be installed will be specified in the OpenTelemetry Operator for Kubernetes documentation):

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml

Then when the cert managers are ready (kubectl get pods -n cert-manager)  …

NAMESPACE      NAME                                         READY
cert-manager   cert-manager-67c98b89c8-rnr5s                1/1
cert-manager   cert-manager-cainjector-5c5695d979-q9hxz     1/1
cert-manager   cert-manager-webhook-7f9f8648b9-8gxgs        1/1

… you can install the OpenTelemetry Operator:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

You can, of course, use a specific version of the operator instead of the latest. But here I’ve used the latest version.

An Instrumentation resource

Now you need to add just one further Kubernetes resource to enable auto-instrumentation: an Instrumentation resource. I am going to use the banana namespace for my examples, so I have first created that namespace (kubectl create namespace banana). The auto-instrumentation is specified and configured by these Instrumentation resources. Here is a basic one which will allow every Java pod in the banana namespace to be auto-instrumented with version 2.5.0 of the OpenTelemetry Java agent:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: banana-instr
  namespace: banana
spec:
  exporter:
    endpoint: "https://my.endpoint"
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.5.0
    env:
      - name: OTEL_EXPORTER_OTLP_HEADERS
        value: "Authorization=Bearer MyAuth"

Creating this resource (eg with kubectl apply -f banana-instr.yaml, assuming the above yaml was saved in file banana-instr.yaml) makes the banana-instr Instrumentation resource available for use. (Note you will need to change my.endpoint and MyAuth to values appropriate for your collector.) You can use this instrumentation immediately by adding an annotation to any deployment in the banana namespace:

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"

The banana-instr Instrumentation resource is not yet set to be applied by default to all pods in the banana namespace. Currently it's zero-config as far as the application is concerned, but it requires an annotation added to a pod or deployment. To make it fully zero-config for all pods in the banana namespace, we need to add that annotation to the namespace itself, ie editing the namespace (kubectl edit namespace banana) so it would then have contents similar to

apiVersion: v1
kind: Namespace
metadata:
  name: banana
  annotations:
    instrumentation.opentelemetry.io/inject-java: "banana-instr"
...

Now we have a namespace that is going to auto-instrument every Java application deployed in the banana namespace with the 2.5.0 OpenTelemetry Java agent!

Trying it

There is a simple example Java application at docker.elastic.co/demos/apm/k8s-webhook-test which just repeatedly calls the chain main->methodA->methodB->methodC->methodD with some sleeps in the calls. Running this (kubectl apply -f banana-app.yaml) using a very basic pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: banana-app
  namespace: banana
  labels:
    app: banana-app
spec:
  containers:
    - image: docker.elastic.co/demos/apm/k8s-webhook-test
      imagePullPolicy: Always
      name: banana-app
      env: 
      - name: OTEL_INSTRUMENTATION_METHODS_INCLUDE
        value: "test.Testing[methodB]"

results in the app being auto-instrumented with no configuration changes! The resulting app shows up in any APM UI, such as Elastic APM

As you can see, for this example I also added this env var to the pod yaml, OTEL_INSTRUMENTATION_METHODS_INCLUDE="test.Testing[methodB]" so that there were traces showing from methodB.

The technology behind the auto-instrumentation

To use the auto-instrumentation there is no specific need to understand the underlying mechanisms, but for those of you interested, here’s a quick outline.

1. The OpenTelemetry Operator for Kubernetes installs a mutating webhook, a standard Kubernetes component.
2. When deploying, Kubernetes first sends all definitions to the mutating webhook.
3. If the mutating webhook sees that the conditions for auto-instrumentation should be applied (ie
4. there is an Instrumentation resource for that namespace and
5. the correct annotation for that Instrumentation is applied to the definition in some way, either from the definition itself or from the namespace),
6. then the mutating webhook “mutates” the definition to include the environment defined by the Instrumentation resource.
7. The environment includes the explicit values defined in the env, as well as some implicit OpenTelemetry values (see the OpenTelemetry Operator for Kubernetes documentation for full details).
8. And most importantly, the operator
9. pulls the image defined in the Instrumentation resource,
10. extracts the file at the path /javaagent.jar from that image (using shell command cp)
11. inserts it into the pod at path /otel-auto-instrumentation-java/javaagent.jar
12. and adds the environment variable JAVA_TOOL_OPTIONS=-javaagent:/otel-auto-instrumentation-java/javaagent.jar.
13. The JVM automatically picks up that JAVA_TOOL_OPTIONS environment variable on startup and applies it to the JVM command-line.

Next steps

This walkthrough can be repeated in any Kubernetes cluster to demonstrate and experiment with auto-instrumentation (you will need to create the banana namespace first). In part 2 of this two part series, Using a custom agent with the OpenTelemetry Operator for Kubernetes, I show how to install any Java agent via the OpenTelemetry operator, using the Elastic Java agents as examples.

Site reliability engineers (SREs) face the challenging task of sifting through vast quantities of data, not just from the applications themselves but also from the underlying infrastructure. In addition to data analysis, they are responsible for ensuring the effective use and development of operational tools. The growing volume of data, the daily resolution of issues, and the continuous evolution of tools and processes can detract from the focus on business performance.

Elastic Observability offers a solution to this challenge. It enables SREs to integrate and examine all telemetry data (logs, metrics, traces, and profiling) in conjunction with business metrics. This comprehensive approach to data analysis fosters operational excellence, boosts productivity, and yields critical insights, all of which are integral to maintaining high-performing applications in a demanding digital environment.

To help manage operations and business metrics, Elastic Observability's SLO (Service Level Objectives) feature was introduced in 8.12. This feature enables setting measurable performance targets for services, such as availability, latency, traffic, errors, and saturation or define your own. Key components include:

• Defining and monitoring SLIs (Service Level Indicators)

• Monitoring error budgets indicating permissible performance shortfalls

• Alerting on burn rates showing error budget consumption

Users can monitor SLOs in real-time with dashboards, track historical performance, and receive alerts for potential issues. Additionally, SLO dashboard panels offer customized visualizations.

Service Level Objectives (SLOs) are generally available for our Platinum and Enterprise subscription customers.

In this blog, we will outline the following:

• What are SLOs? A Google SRE perspective

• Several scenarios of defining and managing SLOs

Service Level Objective overview

Service Level Objectives (SLOs) are a crucial component for Site Reliability Engineering (SRE), as detailed in Google's SRE Handbook. They provide a framework for quantifying and managing the reliability of a service. The key elements of SLOs include:

• Service Level Indicators (SLIs): These are carefully selected metrics, such as uptime, latency, throughput, error rates, or other important metrics, that represent the aspects of the service and are important from an operations or business perspective. Hence, an SLI is a measure of the service level provided (latency, uptime, etc.), and it is defined as a ratio of good over total events, with a range between 0% and 100%.

• Service Level Objective (SLO): An SLO is the target value for a service level measured as a percentage by an SLI. Above the threshold, the service is compliant. As an example, if we want to use service availability as an SLI, with the number of successful responses at 99.9%, then any time the number of failed responses is > .1%, the SLO will be out of compliance.

• Error budget: This represents the threshold of acceptable errors, balancing the need for reliability with practical limits. It is defined as 100% minus the SLO quantity of errors that is tolerated.

• Burn rate: This concept relates to how quickly the service is consuming its error budget, which is the acceptable threshold for unreliability agreed upon by the service providers and its users.

Understanding these concepts and effectively implementing them is essential for maintaining a balance between innovation and reliability in service delivery. For more detailed information, you can refer to Google's SRE Handbook.

One main thing to remember is that SLO monitoring is not incident monitoring. SLO monitoring is a proactive, strategic approach designed to ensure that services meet established performance standards and user expectations. It involves tracking Service Level Objectives, error budgets, and the overall reliability of a service over time. This predictive method helps in preventing issues that could impact users and aligns service performance with business objectives.

In contrast, incident monitoring is a reactive process focused on detecting, responding to, and mitigating service incidents as they occur. It aims to address unexpected disruptions or failures in real time, minimizing downtime and impact on service. This includes monitoring system health, errors, and response times during incidents, with a focus on rapid response to minimize disruption and preserve the service's reputation.

Elastic®’s SLO capability is based directly off the Google SRE Handbook. All the definitions and semantics are utilized as described in Google’s SRE handbook. Hence users can perform the following on SLOs in Elastic:

• Define an SLO on an SLI such as KQL (log based query), service availability, service latency, custom metric, histogram metric, or a timeslice metric. Additionally, set the appropriate threshold.

• Utilize occurrence versus time slice based budgeting. Occurrences is the number of good events over the number of total events to compute the SLO. Timeslices break the overall time window into slammer slices of a defined duration and compute the number of good slices over the total slices to compute the SLO. Timeslice targets are more accurate and useful when calculating things like a service’s SLO when trying to meet agreed upon customer targets.

• Manage all the SLOs in a singular location.

• Trigger alerts from the defined SLO, whether the SLI is off, burn rate is used up, or the error rate is X.

• Create unique service level dashboards with SLO information for a more comprehensive view of the service.

SREs need to be able to manage business metrics.

SLOs based on logs: NGINX availability

Defining SLOs does not always mean metrics need to be used. Logs are a rich form of information, even when they have metrics embedded in them. Hence it’s useful to understand your business and operations status based on logs.

Elastic allows you to create an SLO based on specific fields in the log message, which don’t have to be metrics. A simple example is a simple multi-tier app that has a web server layer (nginx), a processing layer, and a database layer.

Let’s say that your processing layer is managing a significant number of requests. You want to ensure that the service is up properly. The best way is to ensure that all http.response.status_code are less than 500. Anything less ensures the service is up and any errors (like 404) are all user or client errors versus server errors.

If we use Discover in Elastic, we see that there are close to 2M log messages over a seven-day time frame.

Additionally, the number of messages with http.response.status_code > 500 is minimal, like 17K.

Rather than creating an alert, we can create an SLO with this query:

We chose to use occurrences as the budgeting method to keep things simple.

Once defined, we can see how well our SLO is performing over a seven-day time frame. We can see not only the SLO, but also the burn rate, the historical SLI, and error budget, and any specific alerts against the SLO.

Not only do we get information about the violation, but we also get:

• Historical SLI (7 days)

• Error budget burn down

• Good vs. bad events (24 hours)

We can see how we’ve easily burned through our error budget.

Hence something must be going on with nginx. To investigate, all we need to do is utilize the AI Assistant, and use its natural language interface to ask questions to help analyze the situation.

Let’s use Elastic’s AI Assistant to analyze the breakdown of http.response.status_code across all the logs from the past seven days. This helps us understand how many 50X errors we are getting.

As we can see, the number of 502s is minimal compared to the number of overall messages, but it is affecting our SLO.

However, it seems like Nginx is having an issue. In order to reduce the issue, we also ask the AI Assistant how to work on this error. Specifically, we ask if there is an internal runbook the SRE team has created.

AI Assistant gets a runbook the team has added to its knowledge base. I can now analyze and try to resolve or reduce the issue with nginx.

While this is a simple example, there are an endless number of possibilities that can be defined based on KQL. Some other simple examples:

• 99% of requests occur under 200ms

• 99% of log message are not errors

Application SLOs: OpenTelemetry demo cartservice

A common application developers and SREs use to learn about OpenTelemetry and test out Observability features is the OpenTelemetry demo.

This demo has feature flags to simulate issues. With Elastic’s alerting and SLO capability, you can also determine how well the entire application is performing and how well your customer experience is holding up when these feature flags are used.

Elastic supports OpenTelemetry by taking OTLP directly with no need for an Elastic specific agent. You can send in OpenTelemetry data directly from the application (through OTel libraries) and through the collector.

We’ve brought up the OpenTelemetry demo on a K8S cluster (AWS EKS) and turned on the cartservice feature flag. This inserts errors into the cartservice. We’ve also created two SLOs to monitor the cartservice’s availability and latency.

We can see that the cartservice’s availability is violated. As we drill down, we see that there aren’t as many successful transactions, which is affecting the SLO.

As we drill into the service, we can see in Elastic APM that there is a higher than normal failure rate of about 5.5% for the emptyCart service.

We can investigate this further in APM, but that is a discussion for another blog. Stay tuned to see how we can use Elastic’s machine learning, AIOps, and AI Assistant to understand the issue.

Conclusion

SLOs allow you to set clear, measurable targets for your service performance, based on factors like availability, response times, error rates, and other key metrics. Hopefully with the overview we’ve provided in this blog, you can see that:

• SLOs can be based on logs. In Elastic, you can use KQL to essentially find and filter on specific logs and log fields to monitor and trigger SLOs.

• AI Assistant is a valuable, easy-to-use capability to analyze, troubleshoot, and even potentially resolve SLO issues.

• APM Service based SLOs are easy to create and manage with integration to Elastic APM. We also use OTel telemetry to help monitor SLOs.

For more information on SLOs in Elastic, check out Elastic documentation and the following resources:

• What’s new in Elastic Observability 8.12

• Introducing the Elastic AI Assistant

• Elastic OpenTelemetry support

Ready to get started? Sign up for Elastic Cloud and try out the features and capabilities I’ve outlined above to get the most value and visibility out of your SLOs.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

In this blog, we’ll walk through a full example from scratch using a Java application. Apart from the Java code and using a JVM for the application, everything else works the same for the other languages supported by the Attacher.

Prerequisites

This walkthrough assumes that the following are already installed on the system: JDK 17, Docker, Kubernetes, and Helm.

The example application

While the application (shown below) is a Java application, it would be easily implemented in any language, as it is just a simple loop that every 2 seconds calls the method chain methodA->methodB->methodC->methodD, with methodC sleeping for 10 milliseconds and methodD sleeping for 200 milliseconds. The choice of application is just to be able to clearly display in the Elastic APM UI that the application is being monitored.

The Java application in full is shown here:

package test;

public class Testing implements Runnable {

  public static void main(String[] args) {
    new Thread(new Testing()).start();
  }

  public void run()
  {
    while(true) {
      try {Thread.sleep(2000);} catch (InterruptedException e) {}
      methodA();
    }
  }

  public void methodA() {methodB();}

  public void methodB() {methodC();}

  public void methodC() {
    System.out.println("methodC executed");
    try {Thread.sleep(10);} catch (InterruptedException e) {}
    methodD();
  }

  public void methodD() {
    System.out.println("methodD executed");
    try {Thread.sleep(200);} catch (InterruptedException e) {}
  }
}

We created a Docker image containing that simple Java application for you that can be pulled from the following Docker repository:

docker.elastic.co/demos/apm/k8s-webhook-test

Deploy the pod

First we need a deployment config. We’ll call the config file webhook-test.yaml, and the contents are pretty minimal — just pull the image and run that as a pod & container called webhook-test in the default namespace:

apiVersion: v1
kind: Pod
metadata:
  name: webhook-test
  labels:
    app: webhook-test
spec:
  containers:
    - image: docker.elastic.co/demos/apm/k8s-webhook-test
      imagePullPolicy: Always
      name: webhook-test

This can be deployed normally using kubectl:

kubectl apply -f webhook-test.yaml

The result is exactly as expected:

$ kubectl get pods
NAME           READY   STATUS    RESTARTS   AGE
webhook-test   1/1     Running   0          10s

$ kubectl logs webhook-test
methodC executed
methodD executed
methodC executed
methodD executed

So far, this is just setting up a standard Kubernetes application with no APM monitoring. Now we get to the interesting bit: adding in auto-instrumentation.

Install Elastic APM K8s Attacher

The first step is to install the Elastic APM K8s Attacher. This only needs to be done once for the cluster — once installed, it is always available. Before installation, we will define where the monitored data will go. As you will see later, we can decide or change this any time. For now, we’ll specify our own Elastic APM server, which is at https://myserver.somecloud:443 — we also have a secret token for authorization to that Elastic APM server, which has value MY_SECRET_TOKEN. (If you want to set up a quick test Elastic APM server, you can do so at https://cloud.elastic.co/).

There are two additional environment variables set for the application that are not generally needed but will help when we see the resulting UI content toward the end of the walkthrough (when the agent is auto-installed, these two variables tell the agent what name to give this application in the UI and what method to trace). Now we just need to define the custom yaml file to hold these. On installation, the custom yaml will be merged into the yaml for the Attacher:

apm:
  secret_token: MY_SECRET_TOKEN
  namespaces:
    - default
webhookConfig:
  agents:
    java:
      environment:
        ELASTIC_APM_SERVER_URL: "https://myserver.somecloud:443"
        ELASTIC_APM_TRACE_METHODS: "test.Testing#methodB"
        ELASTIC_APM_SERVICE_NAME: "webhook-test"

That custom.yaml file is all we need to install the attacher (note we’ve only specified the default namespace for agent auto-installation for now — this can be easily changed, as you’ll see later). Next we’ll add the Elastic charts to helm — this only needs to be done once, then all Elastic charts are available to helm. This is the usual helm add repo command, specifically:

helm repo add elastic https://helm.elastic.co

Now the Elastic charts are available for installation (helm search repo would show you all the available charts). We’re going to use “elastic-webhook” as the name to install into, resulting in the following installation command:

helm install elastic-webhook elastic/apm-attacher --namespace=elastic-apm --create-namespace --values custom.yaml

And that’s it, we now have the Elastic APM K8s Attacher installed and set to send data to the APM server defined in the custom.yaml file! (You can confirm installation with a helm list -A if you need.)

Auto-install the Java agent

The Elastic APM K8s Attacher is installed, but it doesn’t auto-install the APM application agents into every pod — that could lead to problems! Instead the Attacher is deliberately limited to auto-install agents into deployments defined a) by the namespaces listed in the custom.yaml, and b) to those deployments in those namespaces that have a specific annotation “co.elastic.apm/attach.”

So for now, restarting the webhook-test pod we created above won’t have any different effect on the pod, as it isn’t yet set to be monitored. What we need to do is add the annotation. Specifically, we need to add the annotation using the default agent configuration that was installed with the Attacher called “java” for the Java agent (we’ll see later how that agent configuration is altered — the default configuration installs the latest agent version and leaves everything else default for that version). So adding that annotation in to webhook-test yaml gives us the new yaml file contents (the additional config is shown labelled (1)):

apiVersion: v1
kind: Pod
metadata:
  name: webhook-test
  annotations: #(1)
    co.elastic.apm/attach: java #(1)
  labels:
    app: webhook-test
spec:
  containers:
    - image: docker.elastic.co/demos/apm/k8s-webhook-test
      imagePullPolicy: Always
      name: webhook-test

Applying this change gives us the application now monitored:

$ kubectl delete -f webhook-test.yaml
pod "webhook-test" deleted
$ kubectl apply -f webhook-test.yaml
pod/webhook-test created
$ kubectl logs webhook-test
… StartupInfo - Starting Elastic APM 1.45.0 …

And since the agent is now feeding data to our APM server, we can now see it in the UI:

Note that the agent identifies Testing.methodB method as a trace root because of the ELASTIC_APM_TRACE_METHODS environment variable set to test.Testing#methodB in the custom.yaml — this tells the agent to specifically trace that method. The time taken by that method will be available in the UI for each invocation, but we don’t see the sub-methods . . . currently. In the next section, we’ll see how easy it is to customize the Attacher, and in doing so we’ll see more detail about the method chain being executed in the application.

Customizing the agents

In your systems, you’ll likely have development, testing, and production environments. You’ll want to specify the version of the agent to use rather than just pull the latest version whatever that is, you’ll want to have debug on for some applications or instances, and you’ll want to have specific options set to specific values. This sounds like a lot of effort, but the attacher lets you enable these kinds of changes in a very simple way. In this section, we’ll add a configuration that specifies all these changes and we can see just how easy it is to configure and enable it.

We start at the custom.yaml file we defined above. This is the file that gets merged into the Attacher. Adding a new configuration with all the items listed in the last paragraph is easy — though first we need to decide a name for our new configuration. We’ll call it “java-interesting” here. The new custom.yaml in full is (the first part is just the same as before, the new config is simply appended):

apm:
  secret_token: MY_SECRET_TOKEN
  namespaces:
    - default
webhookConfig:
  agents:
    java:
      environment:
        ELASTIC_APM_SERVER_URL: "https://myserver.somecloud:443"
        ELASTIC_APM_TRACE_METHODS: "test.Testing#methodB"
        ELASTIC_APM_SERVICE_NAME: "webhook-test"
    java-interesting:
      image: docker.elastic.co/observability/apm-agent-java:1.55.4
      artifact: "/usr/agent/elastic-apm-agent.jar"
      environment:
        ELASTIC_APM_SERVER_URL: "https://myserver.somecloud:443"
        ELASTIC_APM_TRACE_METHODS: "test.Testing#methodB"
        ELASTIC_APM_SERVICE_NAME: "webhook-test"
        ELASTIC_APM_ENVIRONMENT: "testing"
        ELASTIC_APM_LOG_LEVEL: "debug"
        ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED: "true"
        JAVA_TOOL_OPTIONS: "-javaagent:/elastic/apm/agent/elastic-apm-agent.jar"

Breaking the additional config down, we have:

• The name of the new config java-interesting

• The APM Java agent image docker.elastic.co/observability/apm-agent-java

• With a specific version 1.43.0 instead of latest

• We need to specify the agent jar location (the attacher puts it here)

• artifact: "/usr/agent/elastic-apm-agent.jar"

• And then the environment variables

• ELASTIC_APM_SERVER_URL as before

• ELASTIC_APM_ENVIRONMENT set to testing, useful when looking in the UI

• ELASTIC_APM_LOG_LEVEL set to debug for more detailed agent output

• ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED turning this on (setting to true) will give us additional interesting information about the method chain being executed in the application

• And lastly we need to set JAVA_TOOL_OPTIONS to the enable starting the agent "-javaagent:/elastic/apm/agent/elastic-apm-agent.jar" — this is fundamentally how the attacher auto-attaches the Java agent

More configurations and details about configuration options are here for the Java agent, and other language agents are also available.

The application traced with the new configuration

And finally we just need to upgrade the attacher with the changed custom.yaml:

helm upgrade elastic-webhook elastic/apm-attacher --namespace=elastic-apm --create-namespace --values custom.yaml

This is the same command as the original install, but now using upgrade. That’s it — add config to the custom.yaml and upgrade the attacher, and it’s done! Simple.

Of course we still need to use the new config on an app. In this case, we’ll edit the existing webhook-test.yaml file, replacing java with java-interesting, so the annotation line is now:

co.elastic.apm/attach: java-interesting

Applying the new pod config and restarting the pod, you can see the logs now hold debug output:

$ kubectl delete -f webhook-test.yaml
pod "webhook-test" deleted
$ kubectl apply -f webhook-test.yaml
pod/webhook-test created
$ kubectl logs webhook-test
… StartupInfo - Starting Elastic APM 1.44.0 …
… DEBUG co.elastic.apm.agent. …
… DEBUG co.elastic.apm.agent. …

More interesting is the UI. Now that inferred spans is on, the full method chain is visible.

This gives the details for methodB (it takes 211 milliseconds because it calls methodC - 10ms - which calls methodD - 200ms). The times for methodC and methodD are inferred rather than recorded, (inferred rather than traced — if you needed accurate times you would instead add the methods to trace_methods and have them traced too).

Note on the ECK operator

The Elastic Cloud on Kubernetes operator allows you to install and manage a number of other Elastic components on Kubernetes. At the time of publication of this blog, the Elastic APM K8s Attacher is a separate component, and there is no conflict between these management mechanisms — they apply to different components and are independent of each other.

Try it yourself!

This walkthrough is easily repeated on your system, and you can make it more useful by replacing the example application with your own and the Docker registry with the one you use.

Learn more about real-time monitoring with Kubernetes and Elastic Observability.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Why use OpenShift Logging Operator?

A lot of enterprise customers use OpenShift as their orchestrating solution. The advantages of this approach are:

• It is developed and supported by Red Hat

• It can automatically update the OpenShift cluster along with the Operating system to make sure that they are and remain compatible

• It can speed up developing life cycles with features like source to image

• It uses enhanced security

In our consulting experience, this latter aspect poses challenges and frictions with OpenShift administrators when we try to install an Elastic Agent to collect the logs of the pods. Indeed, Elastic Agent requires the files of the host to be mounted in the pod, and it also needs to be run in privileged mode. (Read more about the permissions required by Elastic Agent in the official Elasticsearch® Documentation). While the solution we explore in this post requires similar privileges under the hood, it is managed by the OpenShift Logging Operator, which is developed and supported by Red Hat.

Which logs are we going to collect?

In OpenShift Container Platform, we distinguish three broad categories of logs: audit, application, and infrastructure logs:

• Audit logs describe the list of activities that affected the system by users, administrators, and other components.

• Application logs are composed of the container logs of the pods running in non-reserved namespaces.

• Infrastructure logs are composed of container logs of the pods running in reserved namespaces like openshift*, kube*, and default along with journald messages from the nodes.

In the following, we will consider only audit and application logs for the sake of simplicity. In this post, we will describe how to format audit and application Logs in the format expected by the Kubernetes integration to take the most out of Elastic Observability.

Getting started

To collect the logs from OpenShift, we must perform some preparation steps in Elasticsearch and OpenShift.

Inside Elasticsearch

We first install the Kubernetes integration assets. We are mainly interested in the index templates and ingest pipelines for the logs-kubernetes.container_logs and logs-kubernetes.audit_logs.

To format the logs received from the ClusterLogForwarder in ECS format, we will define a pipeline to normalize the container logs. The field naming convention used by OpenShift is slightly different from that used by ECS. To get a list of exported fields from OpenShift, refer to Exported fields | Logging | OpenShift Container Platform 4.14. To get a list of exported fields of the Kubernetes integration, you can refer to Kubernetes fields | Filebeat Reference [8.11] | Elastic and Logs app fields | Elastic Observability [8.11]. Further, specific fields like kubernetes.annotations must be normalized by replacing dots with underscores. This operation is usually done automatically by Elastic Agent.

PUT _ingest/pipeline/openshift-2-ecs
{
  "processors": [
    {
      "rename": {
        "field": "kubernetes.pod_id",
        "target_field": "kubernetes.pod.uid",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "kubernetes.pod_ip",
        "target_field": "kubernetes.pod.ip",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "kubernetes.pod_name",
        "target_field": "kubernetes.pod.name",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "kubernetes.namespace_name",
        "target_field": "kubernetes.namespace",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "kubernetes.namespace_id",
        "target_field": "kubernetes.namespace_uid",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "kubernetes.container_id",
        "target_field": "container.id",
        "ignore_missing": true
      }
    },
    {
      "dissect": {
        "field": "container.id",
        "pattern": "%{container.runtime}://%{container.id}",
        "ignore_failure": true
      }
    },
    {
      "rename": {
        "field": "kubernetes.container_image",
        "target_field": "container.image.name",
        "ignore_missing": true
      }
    },
    {
      "set": {
        "field": "kubernetes.container.image",
        "copy_from": "container.image.name",
        "ignore_failure": true
      }
    },
    {
      "set": {
        "copy_from": "kubernetes.container_name",
        "field": "container.name",
        "ignore_failure": true
      }
    },
    {
      "rename": {
        "field": "kubernetes.container_name",
        "target_field": "kubernetes.container.name",
        "ignore_missing": true
      }
    },
    {
      "set": {
        "field": "kubernetes.node.name",
        "copy_from": "hostname",
        "ignore_failure": true
      }
    },
    {
      "rename": {
        "field": "hostname",
        "target_field": "host.name",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "level",
        "target_field": "log.level",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "file",
        "target_field": "log.file.path",
        "ignore_missing": true
      }
    },
    {
      "set": {
        "copy_from": "openshift.cluster_id",
        "field": "orchestrator.cluster.name",
        "ignore_failure": true
      }
    },
    {
      "dissect": {
        "field": "kubernetes.pod_owner",
        "pattern": "%{_tmp.parent_type}/%{_tmp.parent_name}",
        "ignore_missing": true
      }
    },
    {
      "lowercase": {
        "field": "_tmp.parent_type",
        "ignore_missing": true
      }
    },
    {
      "set": {
        "field": "kubernetes.pod.{{_tmp.parent_type}}.name",
        "value": "{{_tmp.parent_name}}",
        "if": "ctx?._tmp?.parent_type != null",
        "ignore_failure": true
      }
    },
    {
      "remove": {
        "field": [
          "_tmp",
          "kubernetes.pod_owner"
          ],
          "ignore_missing": true
      }
    },
    {
      "script": {
        "description": "Normalize kubernetes annotations",
        "if": "ctx?.kubernetes?.annotations != null",
        "source": """
        def keys = new ArrayList(ctx.kubernetes.annotations.keySet());
        for(k in keys) {
          if (k.indexOf(".") >= 0) {
            def sanitizedKey = k.replace(".", "_");
            ctx.kubernetes.annotations[sanitizedKey] = ctx.kubernetes.annotations[k];
            ctx.kubernetes.annotations.remove(k);
          }
        }
        """
      }
    },
    {
      "script": {
        "description": "Normalize kubernetes namespace_labels",
        "if": "ctx?.kubernetes?.namespace_labels != null",
        "source": """
        def keys = new ArrayList(ctx.kubernetes.namespace_labels.keySet());
        for(k in keys) {
          if (k.indexOf(".") >= 0) {
            def sanitizedKey = k.replace(".", "_");
            ctx.kubernetes.namespace_labels[sanitizedKey] = ctx.kubernetes.namespace_labels[k];
            ctx.kubernetes.namespace_labels.remove(k);
          }
        }
        """
      }
    },
    {
      "script": {
        "description": "Normalize special Kubernetes Labels used in logs-kubernetes.container_logs to determine service.name and service.version",
        "if": "ctx?.kubernetes?.labels != null",
        "source": """
        def keys = new ArrayList(ctx.kubernetes.labels.keySet());
        for(k in keys) {
          if (k.startsWith("app_kubernetes_io_component_")) {
            def sanitizedKey = k.replace("app_kubernetes_io_component_", "app_kubernetes_io_component/");
            ctx.kubernetes.labels[sanitizedKey] = ctx.kubernetes.labels[k];
            ctx.kubernetes.labels.remove(k);
          }
        }
        """
      }
    }
    ]
}

Similarly, to handle the audit logs like the ones collected by Kubernetes, we define an ingest pipeline:

PUT _ingest/pipeline/openshift-audit-2-ecs
{
  "processors": [
    {
      "script": {
        "source": """
        def audit = [:];
        def keyToRemove = [];
        for(k in ctx.keySet()) {
          if (k.indexOf('_') != 0 && !['@timestamp', 'data_stream', 'openshift', 'event', 'hostname'].contains(k)) {
            audit[k] = ctx[k];
            keyToRemove.add(k);
          }
        }
        for(k in keyToRemove) {
          ctx.remove(k);
        }
        ctx.kubernetes=["audit":audit];
        """,
        "description": "Move all the 'kubernetes.audit' fields under 'kubernetes.audit' object"
      }
    },
    {
      "set": {
        "copy_from": "openshift.cluster_id",
        "field": "orchestrator.cluster.name",
        "ignore_failure": true
      }
    },
    {
      "set": {
        "field": "kubernetes.node.name",
        "copy_from": "hostname",
        "ignore_failure": true
      }
    },
    {
      "rename": {
        "field": "hostname",
        "target_field": "host.name",
        "ignore_missing": true
      }
    },
    {
      "script": {
        "if": "ctx?.kubernetes?.audit?.annotations != null",
        "source": """
          def keys = new ArrayList(ctx.kubernetes.audit.annotations.keySet());
          for(k in keys) {
            if (k.indexOf(".") >= 0) {
              def sanitizedKey = k.replace(".", "_");
              ctx.kubernetes.audit.annotations[sanitizedKey] = ctx.kubernetes.audit.annotations[k];
              ctx.kubernetes.audit.annotations.remove(k);
            }
          }
          """,
        "description": "Normalize kubernetes audit annotations field as expected by the Integration"
      }
    }
  ]
}

The main objective of the pipeline is to mimic what Elastic Agent is doing: storing all audit fields under the kubernetes.audit object.

We are not going to use the conventional @custom pipeline approach because the fields must be normalized before invoking the logs-kubernetes.container_logs integration pipeline that uses fields like kubernetes.container.name and kubernetes.labels to determine the fields service.name and service.version. Read more about custom pipelines in Tutorial: Transform data with custom ingest pipelines | Fleet and Elastic Agent Guide [8.11].

The OpenShift Cluster Log Forwarder writes the data in the indices app-write and audit-write by default. It is possible to change this behavior, but it still tries to prepend the prefix “app” and the suffix “write”, so we opted to send the data to the default destination and use the reroute processor to send it to the right data streams. Read more about the Reroute Processor in our blog Simplifying log data management: Harness the power of flexible routing with Elastic and our documentation Reroute processor | Elasticsearch Guide [8.11] | Elastic.

In this case, we want to redirect the container logs (app-write index) to logs-kubernetes.container_logs and the Audit logs (audit-write) to logs-kubernetes.audit_logs:

PUT _ingest/pipeline/app-write-reroute-pipeline
{
  "processors": [
    {
      "pipeline": {
        "name": "openshift-2-ecs",
        "description": "Format the Openshift data in ECS"
      }
    },
    {
      "set": {
        "field": "event.dataset",
        "value": "kubernetes.container_logs"
      }
    },
    {
      "reroute": {
        "destination": "logs-kubernetes.container_logs-openshift"
      }
    }
  ]
}



PUT _ingest/pipeline/audit-write-reroute-pipeline
{
  "processors": [
    {
      "pipeline": {
        "name": "openshift-audit-2-ecs",
        "description": "Format the Openshift data in ECS"
      }
    },
    {
      "set": {
        "field": "event.dataset",
        "value": "kubernetes.audit_logs"
      }
    },
    {
      "reroute": {
        "destination": "logs-kubernetes.audit_logs-openshift"
      }
    }
  ]
}

Please note that given that app-write and audit-write do not follow the data stream naming convention, we are forced to add the destination field in the reroute processor. The reroute processor will also fill up the data_stream fields for us. Note that this step is done automatically by Elastic Agent at source.

Further, we create the indices with the default pipelines we created to reroute the logs according to our needs.

PUT app-write
{
  "settings": {
      "index.default_pipeline": "app-write-reroute-pipeline"
   }
}


PUT audit-write
{
  "settings": {
    "index.default_pipeline": "audit-write-reroute-pipeline"
  }
}

Basically, what we did can be summarized in this picture:

Let us take the container logs. When the operator attempts to write in the app-write index, it will invoke the default_pipeline “app-write-reroute-pipeline” that formats the logs into ECS format and reroutes the logs to logs-kubernetes.container_logs-openshift datastreams. This calls the integration pipeline that invokes, if it exists, the logs-kubernetes.container_logs@custom pipeline. Finally, the logs-kubernetes_container_logs pipeline may reroute the logs to another data set and namespace utilizing the elastic.co/dataset and elastic.co/namespace annotations as described in the Kubernetes integration documentation, which in turn can lead to the execution of an another integration pipeline.

Create a user for sending the logs

We are going to use basic authentication because, at the time of writing, it is the only supported authentication method for Elasticsearch in OpenShift logging. Thus, we need a role that allows the user to write and read the app-write, and audit-write logs (required by the OpenShift agent) and auto_configure access to logs-*-* to allow custom Kubernetes rerouting:

PUT _security/role/YOURROLE
{
    "cluster": [
      "monitor"
    ],
    "indices": [
      {
        "names": [
          "logs-*-*"
        ],
        "privileges": [
          "auto_configure",
          "create_doc"
        ],
        "allow_restricted_indices": false
      },
      {
        "names": [
          "app-write",
          "audit-write",
        ],
        "privileges": [
          "create_doc",
          "read"
        ],
        "allow_restricted_indices": false
      }
    ],
    "applications": [],
    "run_as": [],
    "metadata": {},
    "transient_metadata": {
      "enabled": true
    }

}



PUT _security/user/YOUR_USERNAME
{
  "password": "YOUR_PASSWORD",
  "roles": ["YOURROLE"]
}

On OpenShift

On the OpenShift Cluster, we need to follow the official documentation of Red Hat on how to install the Red Hat OpenShift Logging and configure Cluster Logging and the Cluster Log Forwarder.

We need to install the Red Hat OpenShift Logging Operator, which defines the ClusterLogging and ClusterLogForwarder Resources. Afterward, we can define the Cluster Logging resource:

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collection:
    logs:
      type: vector
      vector: {}

The Cluster Log Forwarder is the resource responsible for defining a daemon set that will forward the logs to the remote Elasticsearch. Before creating it, we need to create in the same namespace as the ClusterLogForwarder a secret containing the Elasticsearch credentials for the user we created previously in the namespace, where the ClusterLogForwarder will be deployed:

apiVersion: v1
kind: Secret
metadata:
  name: elasticsearch-password
  namespace: openshift-logging
type: Opaque
stringData:
  username: YOUR_USERNAME
  password: YOUR_PASSWORD

Finally, we create the ClusterLogForwarder resource:

kind: ClusterLogForwarder
apiVersion: logging.openshift.io/v1
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
    - name: remote-elasticsearch
      secret:
        name: elasticsearch-password
      type: elasticsearch
      url: "https://YOUR_ELASTICSEARCH_URL:443"
      elasticsearch:
        version: 8 # The default is version 6 with the _type field
  pipelines:
    - inputRefs:
        - application
        - audit
      name: enable-default-log-store
      outputRefs:
        - remote-elasticsearch

Note that we explicitly defined the version of Elasticsearch to be 8, otherwise the ClusterLogForwarder will send the _type field, which is not compatible with Elasticsearch 8 and that we collect only application and audit logs.

Result

Once the logs are collected and passed through all the pipelines, the result is very close to the out-of-the-box Kubernetes integration. There are important differences, like the lack of host and cloud metadata information that don’t seem to be collected (at least without an additional configuration). We can view the Kubernetes container logs in the logs explorer:

In this post, we described how you can use the OpenShift Logging Operator to collect the logs of containers and audit logs. We still recommend leveraging Elastic Agent to collect all your logs. It is the best user experience you can get. No need to maintain or transform the logs yourself to ECS formatting. Additionally, Elastic Agent uses API keys as the authentication method and collects metadata like cloud information that allow you in the long run to do more.

Learn more about log monitoring with the Elastic Stack.

Have feedback on this blog? Share it here.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Enhanced troubleshooting for SREs

Site reliability engineers (SRE) in large organizations often face challenges in locating necessary information for troubleshooting alerts, monitoring systems, or deriving insights due to scattered and potentially outdated resources. This issue is particularly significant for less experienced SREs who may require assistance even with the presence of a runbook. Recurring incidents pose another problem, as the on-call individual may lack knowledge about previous resolutions and subsequent steps. Mature SRE teams often invest considerable time in system improvements to minimize "fire-fighting," utilizing extensive automation and documentation to support on-call personnel.

Elastic® addresses these challenges by combining generative AI models with relevant search results from your internal data using RAG. The Observability AI Assistant's internal Knowledge Base, powered by our semantic search retrieval model ELSER, can recall information at any point during a conversation, providing RAG responses based on internal knowledge.

This Knowledge Base can be enriched with your organization's information, such as runbooks, GitHub issues, internal documentation, and Slack messages, allowing the AI Assistant to provide specific assistance. The Assistant can also document and store specific information from an ongoing conversation with an SRE while troubleshooting issues, effectively creating runbooks for future reference. Furthermore, the Assistant can generate summaries of incidents, system status, runbooks, post-mortems, or public announcements.

This ability to retrieve, summarize, and present contextually relevant information is a game-changer for SRE teams, transforming the work from chasing documents and data to an intuitive, contextually sensitive user experience.The Knowledge Base (see requirements) serves as a central repository of Observability knowledge, breaking documentation silos and integrating tribal knowledge, making this information accessible to SREs enhanced with the power of LLMs.

Your LLM provider may collect query telemetry when using the AI Assistant. If your data is confidential or has sensitive details, we recommend you verify the data treatment policy of the LLM connector you provided to the AI Assistant.

In this blog post, we will cover different ways to enrich your Knowledge Base (KB) with internal information. We will focus on a specific alert, indicating that there was an increase in logs with “502 Bad Gateway” errors that has surpassed the alert’s threshold.

How to troubleshoot an alert with the Knowledge Base

Before the KB has been enriched with internal information, when the SRE asks the AI Assistant about how to troubleshoot an alert, the response from the LLM will be based on the data it learned during training; however, the LLM is not able to answer questions related to private, recent, or emerging knowledge. In this case, when asking for the steps to troubleshoot the alert, the response will be based on generic information.

However, once the KB has been enriched with your runbooks, when your team receives a new alert on “502 Bad Gateway” Errors, they can use AI Assistant to access the internal knowledge to troubleshoot it, using semantic search to find the appropriate runbook in the Knowledge Base.

In this blog, we will cover different ways to add internal information on how to troubleshoot an alert to the Knowledge Base:

1. Ask the assistant to remember the content of an existing runbook.

2. Ask the Assistant to summarize and store in the Knowledge Base the steps taken during a conversation and store it as a runbook.

3. Import your runbooks from GitHub or another external source to the Knowledge Base using our Connector and APIs.

After the runbooks have been added to the KB, the AI Assistant is now able to recall the internal and specific information in the runbooks. By leveraging the retrieved information, the LLM could provide more accurate and relevant recommendations for troubleshooting the alert. This could include suggesting potential causes for the alert, steps to resolve the issue, preventative measures for future incidents, or asking the assistant to help execute the steps mentioned in the runbook using functions. With more accurate and relevant information at hand, the SRE could potentially resolve the alert more quickly, reducing downtime and improving service reliability.

Your Knowledge Base documents will be stored in the indices .kibana-observability-ai-assistant-kb-*. Have in mind that LLMs have restrictions on the amount of information the model can read and write at once, called token limit. Imagine you're reading a book, but you can only remember a certain number of words at a time. Once you've reached that limit, you start to forget the earlier words you've read. That's similar to how a token limit works in an LLM.

To keep runbooks within the token limit for Retrieval Augmented Generation (RAG) models, ensure the information is concise and relevant. Use bullet points for clarity, avoid repetition, and use links for additional information. Regularly review and update the runbooks to remove outdated or irrelevant information. The goal is to provide clear, concise, and effective troubleshooting information without compromising the quality due to token limit constraints. LLMs are great for summarization, so you could ask the AI Assistant to help you make the runbooks more concise.

Ask the assistant to remember the content of an existing runbook

The easiest way to store a runbook into the Knowledge Base is to just ask the AI Assistant to do it! Open a new conversation and ask “Can you store this runbook in the KB for future reference?” followed by pasting the content of the runbook in plain text.

The AI Assistant will then store it in the Knowledge Base for you automatically, as simple as that.

Ask the Assistant to summarize and store the steps taken during a conversation in the Knowledge Base

You can also ask the AI Assistant to remember something while having a conversation — for example, after you have troubleshooted an alert using the AI Assistant, you could ask to "remember how to troubleshoot this alert for next time." The AI Assistant will create a summary of the steps taken to troubleshoot the alert and add it to the Knowledge Base, effectively creating runbooks for future reference. Next time you are faced with a similar situation, the AI Assistant will recall this information and use it to assist you.

In the following demo, the user asks the Assistant to remember the steps that have been followed to troubleshoot the root cause of an alert, and also to ping the Slack channel when this happens again. In a later conversation with the Assistant, the user asks what can be done about a similar problem, and the AI Assistant is able to remember the steps and also reminds the user to ping the Slack channel.

After receiving the alert, you can open the AI Assistant chat and test troubleshooting the alert. After investigating an alert, ask the AI Assistant to summarize the analysis and the steps taken to root cause. To remember them for the next time, we have a similar alert and add extra instruction like to warn the Slack channel.

The Assistant will use the built-in functions to summarize the steps and store them into your Knowledge Base, so they can be recalled in future conversations.

Open a new conversation, and ask what are the steps to take when troubleshooting a similar alert to the one we just investigated. The Assistant will be able to recall the information stored in the KB that is related to the specific alert, using semantic search based on ELSER, and provide a summary of the steps taken to troubleshoot it, including the last indication of informing the Slack channel.

Import your runbooks stored in GitHub to the Knowledge Base using APIs or our GitHub Connector

You can also add proprietary data into the Knowledge Base programmatically by ingesting it (e.g., GitHub Issues, Markdown files, Jira tickets, text files) into Elastic.

If your organization has created runbooks that are stored in Markdown documents in GitHub, follow the steps in the next section of this blog post to index the runbook documents into your Knowledge Base.

The steps to ingest documents into the Knowledge Base are the following:

Ingest your organization’s knowledge into Elasticsearch

Option 1: Use the Elastic web crawler . Use the web crawler to programmatically discover, extract, and index searchable content from websites and knowledge bases. When you ingest data with the web crawler, a search-optimized Elasticsearch® index is created to hold and sync webpage content.

Option 2: Use Elasticsearch's Index API . Watch tutorials that demonstrate how you can use the Elasticsearch language clients to ingest data from an application.

Option 3: Build your own connector. Follow the steps described in this blog: How to create customized connectors for Elasticsearch.

Option 4: Use Elasticsearch Workplace Search connectors . For example, the GitHub connector can automatically capture, sync, and index issues, Markdown files, pull requests, and repos.

• Follow the steps to configure the GitHub Connector in GitHub to create an OAuth App from the GitHub platform.

• Now you can connect a GitHub instance to your organization. Head to your organization’s Search > Workplace Search administrative dashboard, and locate the Sources tab.

• Select GitHub (or GitHub Enterprise) in the Configured Sources list, and follow the GitHub authentication flow as presented. Upon the successful authentication flow, you will be redirected to Workplace Search and will be prompted to select the Organization you would like to synchronize.

• After configuring the connector and selecting the organization, the content should be synchronized and you will be able to see it in Sources. If you don’t need to index all the available content, you can specify the indexing rules via the API. This will help shorten indexing times and limit the size of the index. See Customizing indexing.

• The source has created an index in Elastic with the content (Issues, Markdown Files…) from your organization. You can find the index name by navigating to Stack Management > Index Management , activating the Include hidden Indices button on the right, and searching for “GitHub.”

• You can explore the documents you have indexed by creating a Data View and exploring it in Discover. Go to Stack Management > Kibana > Data Views > Create data view and introduce the data view Name, Index pattern (make sure you activate “Allow hidden and system indices” in advanced options), and Timestamp field:

• You can now explore the documents in Discover using the data view:

Reindex your internal runbooks into the AI Assistant’s Knowledge Base Index, using it's semantic search pipeline

Your Knowledge Base documents are stored in the indices .kibana-observability-ai-assistant-kb-*. To add your internal runbooks imported from GitHub to the KB, you just need to reindex the documents from the index you created in the previous step to the KB’s index. To add the semantic search capabilities to the documents in the KB, the reindex should also use the ELSER pipeline preconfigured for the KB, .kibana-observability-ai-assistant-kb-ingest-pipeline.

By creating a Data View with the KB index, you can explore the content in Discover.

You execute the query below in Management > Dev Tools , making sure to replace the following, both on “_source” and “inline”:

• InternalDocsIndex : name of the index where your internal docs are stored
• text_field : name of the field with the text of your internal docs
• timestamp : name of the field of the timestamp in your internal docs
• public : (true or false) if true, makes a document available to all users in the defined Kibana Space (if is defined) or in all spaces (if is not defined); if false, document will be restricted to the user indicated in
• (optional) space : if defined, restricts the internal document to be available in a specific Kibana Space
• (optional) user.name : if defined, restricts the internal document to be available for a specific user
• (optional) "query" filter to index only certain docs (see below)

POST _reindex
{
    "source": {
        "index": "<InternalDocsIndex>",
        "_source": [
            "<text_field>",
            "<timestamp>",
            "namespace",
            "is_correction",
            "public",
            "confidence"
        ]
    },
    "dest": {
        "index": ".kibana-observability-ai-assistant-kb-000001",
        "pipeline": ".kibana-observability-ai-assistant-kb-ingest-pipeline"
    },
    "script": {
        "inline": "ctx._source.text=ctx._source.remove(\"<text_field>\");ctx._source.namespace=\"<space>\";ctx._source.is_correction=false;ctx._source.public=<public>;ctx._source.confidence=\"high\";ctx._source['@timestamp']=ctx._source.remove(\"<timestamp>\");ctx._source['user.name'] = \"<user.name>\""
    }
}

You may want to specify the type of documents that you reindex in the KB — for example, you may only want to reindex Markdown documents (like Runbooks). You can add a “query” filter to the documents in the source. In the case of GitHub, runbooks are identified with the “type” field containing the string “file,” and you could add that to the reindex query like indicated below. To add also GitHub Issues, you can also include in the query “type” field containing the string “issues”:

"source": {
        "index": "<InternalDocsIndex>",
        "_source": [
            "<text_field>",
            "<timestamp>",
            "namespace",
            "is_correction",
            "public",
            "confidence"
        ],
    "query": {
      "terms": {
        "type": ["file"]
      }
    }

Great! Now that the data is stored in your Knowledge Base, you can ask the Observability AI Assistant any questions about it:

Conclusion

In conclusion, leveraging internal Observability knowledge and adding it to the Elastic Knowledge Base can greatly enhance the capabilities of the AI Assistant. By manually inputting information or programmatically ingesting documents, SREs can create a central repository of knowledge accessible through the power of Elastic and LLMs. The AI Assistant can recall this information, assist with incidents, and provide tailored observability to specific contexts using Retrieval Augmented Generation. By following the steps outlined in this article, organizations can unlock the full potential of their Elastic AI Assistant.

Start enriching your Knowledge Base with the Elastic AI Assistant today and empower your SRE team with the tools they need to excel. Follow the steps outlined in this article and take your incident management and alert remediation processes to the next level. Your journey toward a more efficient and effective SRE operation begins now.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

As a single platform, Elastic® helps SREs unify and correlate limitless telemetry data, including metrics, logs, traces, and profiling, into a single datastore — Elasticsearch®. By then applying the power of Elastic’s advanced machine learning (ML), AIOps, AI Assistant, and analytics, you can break down silos and turn data into insights. As a full-stack observability solution, everything from infrastructure monitoring to log monitoring and application performance monitoring (APM) can be found in a single, unified experience.

In Elastic 8.11, a technical preview is now available of Elastic’s new piped query language, ES|QL (Elasticsearch Query Language), which transforms, enriches, and simplifies data investigations. Powered by a new query engine, ES|QL delivers advanced search capabilities with concurrent processing, improving speed and efficiency, irrespective of data source and structure. Accelerate resolution by creating aggregations and visualizations from one screen, delivering an iterative, uninterrupted workflow.

Advantages of ES|QL for SREs

SREs using Elastic Observability can leverage ES|QL to analyze logs, metrics, traces, and profiling data, enabling them to pinpoint performance bottlenecks and system issues with a single query. SREs gain the following advantages when managing high dimensionality and high cardinality data with ES|QL in Elastic Observability:

• Improved operational efficiency: By using ES|QL, SREs can create more actionable notifications with aggregated values as thresholds from a single query, which can also be managed through the Elastic API and integrated into DevOps processes.
• Enhanced analysis with insights: ES|QL can process diverse observability data, including application, infrastructure, business data, and more, regardless of the source and structure. ES|QL can easily enrich the data with additional fields and context, allowing the creation of visualizations for dashboards or issue analysis with a single query.
• Reduced mean time to resolution: ES|QL, when combined with Elastic Observability's AIOps and AI Assistant, enhances detection accuracy by identifying trends, isolating incidents, and reducing false positives. This improvement in context facilitates troubleshooting and the quick pinpointing and resolution of issues.

ES|QL in Elastic Observability not only enhances an SRE's ability to manage the customer experience, an organization's revenue, and SLOs more effectively but also facilitates collaboration with developers and DevOps by providing contextualized aggregated data.

In this blog, we will cover some of the key use cases SREs can leverage with ES|QL:

• ES|QL integrated with the Elastic AI Assistant, which uses public LLM and private data, enhances the analysis experience anywhere in Elastic Observability.
• SREs can, in a single ES|QL query, break down, analyze, and visualize observability data from multiple sources and across any time frame.
• Actionable alerts can be easily created from a single ES|QL query, enhancing operations.

I will work through these use cases by showcasing how an SRE can solve a problem in an application instrumented with OpenTelemetry and running on Kubernetes. The OpenTelemetry (OTel) demo is on an Amazon EKS cluster, with Elastic Cloud 8.11 configured.

You can also check out our Elastic Observability ES|QL Demo, which walks through ES|QL functionality for Observability.

ES|QL with AI Assistant

As an SRE, you are monitoring your OTel instrumented application with Elastic Observability, and while in Elastic APM, you notice some issues highlighted in the service map.

Using Elastic AI Assistant, you can easily ask for analysis, and in particular, we check on what the overall latency is across the application services.

My APM data is in traces-apm*. What's the average latency per service over the last hour? Use ESQL, the data is mapped to ECS

The Elastic AI Assistant generates an ES|QL query, which we run in the AI Assistant to get a list of the average latencies across all the application services. We can easily see the top four are:

• load generator
• front-end proxy
• frontendservice
• checkoutservice

With a simple natural language query in the AI Assistant, it generated a single ES|QL query that helped list out the latencies across the services.

Noticing that there is an issue with several services, we decide to start with the frontend proxy. As we work through the details, we see significant failures, and through Elastic APM failure correlation , it becomes apparent that the frontend proxy is not properly completing its calls to downstream services.

ES|QL insightful and contextual analysis in Discover

Knowing that the application is running on Kubernetes, we investigate if there are issues in Kubernetes. In particular, we want to see if there are any services having issues.

We use the following query in ES|QL in Elastic Discover:

from metrics-* | where kubernetes.container.status.last_terminated_reason != "" and kubernetes.namespace == "default" | stats reason_count=count(kubernetes.container.status.last_terminated_reason) by kubernetes.container.name, kubernetes.container.status.last_terminated_reason | where reason_count > 0

ES|QL helps analyze 1,000s/10,000s of metric events from Kubernetes and highlights two services that are restarting due to OOMKilled.

The Elastic AI Assistant, when asked about OOMKilled, indicates that a container in a pod was killed due to an out-of-memory condition.

We run another ES|QL query to understand the memory usage for emailservice and productcatalogservice.

ES|QL easily found the average memory usage fairly high.

We can now further investigate both of these services’ logs, metrics, and Kubernetes-related data. However, before we continue, we create an alert to track heavy memory usage.

Actionable alerts with ES|QL

Suspecting a specific issue, that might recur, we simply create an alert that brings in the ES|QL query we just ran that will track for any service that exceeds 50% in memory utilization.

We modify the last query to find any service with high memory usage:

FROM metrics*
| WHERE @timestamp >= NOW() - 1 hours
| STATS avg_memory_usage = AVG(kubernetes.pod.memory.usage.limit.pct) BY kubernetes.deployment.name | where avg_memory_usage > .5

With that query, we create a simple alert. Notice how the ES|QL query is brought into the alert. We simply connect this to pager duty. But we can choose from multiple connectors like ServiceNow, Opsgenie, email, etc.

With this alert, we can now easily monitor for any services that exceed 50% memory utilization in their pods.

Make the most of your data with ES|QL

In this post, we demonstrated the power ES|QL brings to analysis, operations, and reducing MTTR. In summary, the three use cases with ES|QL in Elastic Observability are as follows:

• ES|QL integrated with the Elastic AI Assistant, which uses public LLM and private data, enhances the analysis experience anywhere in Elastic Observability.
• SREs can, in a single ES|QL query, break down, analyze, and visualize observability data from multiple sources and across any time frame.
• Actionable alerts can be easily created from a single ES|QL query, enhancing operations.

Elastic invites SREs and developers to experience this transformative language firsthand and unlock new horizons in their data tasks. Try it today at https://ela.st/free-trial now in technical preview.

• Elastic Observability Tour
• The power of effective log management
• Transforming Observability with the AI Assistant
• ES|QL announcement blog

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Prometheus is primarily built for short-term metric storage, typically retaining data in-memory or on local disk storage, with a focus on real-time monitoring and alerting rather than historical analysis. While it offers valuable insights into current metric values and trends, it may pose economic challenges and fall short of the robust functionalities and capabilities necessary for in-depth historical analysis, long-term trend detection, and forecasting. This is particularly evident in large environments with a substantial number of targets or high data ingestion rates, where metric data accumulates rapidly.

Numerous organizations assess their unique needs and explore avenues to augment their Prometheus monitoring and observability capabilities. One effective approach is integrating Prometheus with Elastic®. In this blog post, we will showcase the integration of Prometheus with Elastic, emphasizing how Elastic elevates metrics monitoring through extensive historical analytics, anomaly detection, and forecasting, all in a cost-effective manner.

Integrate Prometheus with Elastic seamlessly

Organizations that have configured their cloud-native applications to expose metrics in Prometheus format can seamlessly transmit the metrics to Elastic by using Prometheus integration. Elastic enables organizations to monitor their metrics in conjunction with all other data gathered through Elastic's extensive integrations.

Go to Integrations and find the Prometheus integration.

To gather metrics from Prometheus servers, the Elastic Agent is employed, with central management of Elastic agents handled through the Fleet server.

After enrolling the Elastic Agent in the Fleet, users can choose from the following methods to ingest Prometheus metrics into Elastic.

1. Prometheus collectors

The Prometheus collectors connect to the Prometheus server and pull metrics or scrape metrics from a Prometheus exporter.

2. Prometheus queries

The Prometheus queries execute specific Prometheus queries against Prometheus Query API.

3. Prometheus remote-write

The Prometheus remote_write can receive metrics from a Prometheus server that has configured the remote_write setting.

After your Prometheus metrics are ingested, you have the option to visualize your data graphically within the Metrics Explorer and further segment it based on labels, such as hosts, containers, and more.

You can also query your metrics data in Discover and explore the fields of your individual documents within the details panel.

Storing historical metrics with Elastic’s data tiering mechanism

By exporting Prometheus metrics to Elasticsearch, organizations can extend the retention period and gain the ability to analyze metrics historically. Elastic optimizes data storage and access based on the frequency of data usage and the performance requirements of different data sets. The goal is to efficiently manage and store data, ensuring that it remains accessible when needed while keeping storage costs in check.

After ingesting Prometheus metrics data, you have various retention options. You can set the duration for data to reside in the hot tier, which utilizes high IO hardware (SSD) and is more expensive. Alternatively, you can move the Prometheus metrics to the warm tier, employing cost-effective hardware like spinning disks (HDD) while maintaining consistent and efficient search performance. The cold tier mirrors the infrastructure of the warm tier for primary data but utilizes S3 for replica storage. Elastic automatically recovers replica indices from S3 in case of node or disk failure, ensuring search performance comparable to the warm tier while reducing disk cost.

The frozen tier allows direct searching of data stored in S3 or an object store, without the need for rehydration. The purpose is to further reduce storage costs for Prometheus metrics data that is less frequently accessed. By moving historical data into the frozen tier, organizations can optimize their storage infrastructure, ensuring that the recent, critical data remains in higher-performance tiers while less frequently accessed data is stored economically in the frozen tier. This way, organizations can perform historical analysis and trend detection, identify patterns and make informed decisions, and maintain compliance with regulatory standards in a cost-effective manner.

An alternative way to store your cloud-native metrics more efficiently is to use Elastic Time Series Data Stream (TSDS). TSDS can store your metrics data more efficiently with ~70% less disk space than a regular data stream. The downsampling functionality will further reduce the storage required by rolling up metrics within a fixed time interval into a single summary metric. This not only assists organizations in cutting down on storage expenses for metric data but also simplifies the metric infrastructure, making it easier for users to correlate metrics with logs and traces through a unified interface.

Advanced analytics

Besides Metrics Explorer and Discover, Elasticsearch® provides more advanced analytics capabilities and empowers organizations to gain deeper, more valuable insights into their Prometheus metrics data.

Out of the box, Prometheus integration provides a default overview dashboard.

From Metrics Explorer or Discover, users can also easily edit their Prometheus metrics visualization in Elastic Lens or create new visualizations from Lens.

Elastic Lens enables users to explore and visualize data intuitively through dynamic visualizations. This user-friendly interface eliminates the need for complex query languages, making data analysis accessible to a broader audience. Elasticsearch also offers other powerful visualization methods with aggregations and filters, enabling users to perform advanced analytics on their Prometheus metrics data, including short-term and historical data. To learn more, check out the how-to series: Kibana.

Anomaly detection and forecasting

When analyzing data, maintaining a constant watch on the screen is simply not feasible, especially when dealing with millions of time series of Prometheus metrics. Engineers frequently encounter the challenge of differentiating normal from abnormal data points, which involves analyzing historical data patterns — a process that can be exceedingly time consuming and often exceeds human capabilities. Thus, there is a pressing need for a more intelligent approach to detect anomalies efficiently.

Setting up alerts may seem like an obvious solution, but relying solely on rule-based alerts with static thresholds can be problematic. What's normal on a Wednesday at 9:00 a.m. might be entirely different from a Sunday at 2:00 a.m. This often leads to complex and hard-to-maintain rules or wide alert ranges that end up missing crucial issues. Moreover, as your business, infrastructure, users, and products evolve, these fixed rules don't keep up, resulting in lots of false positives or, even worse, important issues slipping through the cracks without detection. A more intelligent and adaptable approach is needed to ensure accurate and timely anomaly detection.

Elastic's machine learning anomaly detection excels in such scenarios. It automatically models the normal behavior of your Prometheus data, learning trends, and identifying anomalies, thereby reducing false positives and improving mean time to resolution (MTTR). With over 13 years of development experience in this field, Elastic has emerged as a trusted industry leader.

The key advantage of Elastic's machine learning anomaly detection lies in its unsupervised learning approach. By continuously observing real-time data, it acquires an understanding of the data's behavior over time. This includes grasping daily and weekly patterns, enabling it to establish a normalcy range of expected behavior. Behind the scenes, it constructs statistical models that allow accurate predictions, promptly identifying any unexpected variations. In cases where emerging data exhibits unusual trends, you can seamlessly integrate with alerting systems, operationalizing this valuable insight.

Machine learning's ability to project into the future, forecasting data trends one day, a week, or even a month ahead, equips engineers not only with reporting capabilities but also with pattern recognition and failure prediction based on historical Prometheus data. This plays a crucial role in maintaining mission-critical workloads, offering organizations a proactive monitoring approach. By foreseeing and addressing issues before they escalate, organizations can avert downtime, cut costs, optimize resource utilization, and ensure uninterrupted availability of their vital applications and services.

Creating a machine learning job for your Prometheus data is a straightforward task with a few simple steps. Simply specify the data index and set the desired time range in the single metric view. The machine learning job will then automatically process the historical data, building statistical models behind the scenes. These models will enable the system to predict trends and identify anomalies effectively, providing valuable and actionable insights for your monitoring needs.

In essence, Elastic machine learning empowers us to harness the capabilities of data scientists and effectively apply them in monitoring Prometheus metrics. By seamlessly detecting anomalies and predicting potential issues in advance, Elastic machine learning bridges the gap and enables IT professionals to benefit from the insights derived from advanced data analysis. This practical and accessible approach to anomaly detection equips organizations with a proactive stance toward maintaining the reliability of their systems.

Try it out

Start a free trial on Elastic Cloud and ingest your Prometheus metrics into Elastic. Enhance your Prometheus monitoring with Elastic Observability. Stay ahead of potential issues with advanced AI/ML anomaly detection and prediction capabilities. Eliminate data silos, reduce costs, and enhance overall response efficiency.

Elevate your monitoring capabilities with Elastic today!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

OpenTelemetry is more than just becoming the open ingestion standard for observability. As one of the major Cloud Native Computing Foundation (CNCF) projects, with as many commits as Kubernetes, it is gaining support from major ISVs and cloud providers delivering support for the framework. Many global companies from finance, insurance, tech, and other industries are starting to standardize on OpenTelemetry. With OpenTelemetry, DevOps teams have a consistent approach to collecting and ingesting telemetry data providing a de-facto standard for observability.

Elastic^® is strategically standardizing on OpenTelemetry for the main data collection architecture for observability and security. Additionally, Elastic is making a commitment to help OpenTelemetry become the best de facto data collection infrastructure for the observability ecosystem. Elastic is deepening its relationship with OpenTelemetry beyond the recent contribution of Elastic Common Schema (ECS) to OpenTelemetry (OTel).

Today, Elastic supports OpenTelemetry natively, since Elastic 7.14, by being able to directly ingest OpenTelemetry protocol (OTLP) based traces, metrics, and logs.

In this blog, we’ll review the current OpenTelemetry support provided by Elastic, which includes the following:

• Easy ingest of distributed tracing and metrics for applications configured with OpenTelemetry agents for Python, NodeJS, Java, Go, and .NET
• OpenTelemetry logs instrumentation and ingest using various configurations
• Open semantic conventions for logs and more through ECS, which is not part of OpenTelemetry
• Machine learning based AIOps capabilities, such as latency correlations, failure correlations, anomaly detection, log spike analysis, predictive pattern analysis, Elastic AI Assistant support, and more, all apply to native OTLP telemetry.
• Migrate applications to OpenTelemetry at your own speed. Elastic’s APM capabilities all work seamlessly even with a mix of services using OpenTelemetry and/or Elastic APM agents. You can even combine OpenTelemetry instrumentation with Elastic Agent.
• Integrated views and analysis with Kubernetes clusters, which most OpenTelemetry applications are running on. Elastic can highlight specific pods and containers related to each service when analyzing issues for applications based on OpenTelemetry.

Ingesting OpenTelemetry into Elastic

If you’re interested in seeing how simple it is to ingest OpenTelemetry traces and metrics into Elastic, follow the steps outlined in this blog.

Let’s outline what Elastic provides for ingesting OpenTelemetry data. Here are all your options:

Using the OpenTelemetry Collector

When using the OpenTelemetry Collector, which is the most common configuration option, you simply have to add two key variables.

The instructions utilize a specific opentelemetry-collector configuration for Elastic. Essentially, the Elastic values.yaml file specified in the elastic/opentelemetry-demo configure the opentelemetry-collector to point to the Elastic APM Server using two main values:

OTEL_EXPORTER_OTLP_ENDPOINT is Elastic’s APM Server
OTEL_EXPORTER_OTLP_HEADERS Elastic Authorization

These two values can be found in the OpenTelemetry setup instructions under the APM integration instructions (Integrations->APM) in your Elastic Cloud.

Native OpenTelemetry agents embedded in code

If you are thinking of using OpenTelemetry libraries in your code, you can simply point the service to Elastic’s APM server, because it supports native OLTP protocol. No special Elastic conversion is needed.

To demonstrate this effectively and provide some education on how to use OpenTelemetry, we have two applications you can use to learn from:

• Elastic’s version of OpenTelemetry demo: As with all the other observability vendors, we have our own forked version of the OpenTelemetry demo.
• Elastiflix: This demo application is an example to help you learn how to instrument on various languages and telemetry signals.

Check out our blogs on using the Elastiflix application and instrumenting with OpenTelemetry:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation

We have created YouTube videos on these topics as well:

• How to Manually Instrument Java with OpenTelemetry (Part 1)
• How to Manually Instrument Java with OpenTelemetry (Part 2)
• Custom Java Instrumentation with OpenTelemetry
• Elastic APM - Automatic .NET Instrumentation with OpenTelemetry
• How to Manually Instrument .NET Applications with OpenTelemetry

Given Elastic and OpenTelemetry’s vast user base, these provide a rich source of education for anyone trying to learn the intricacies of instrumenting with OpenTelemetry.

Elastic Agents supporting OpenTelemetry

If you’ve already implemented OpenTelemetry, you can still use them with OpenTelemetry. Elastic APM agents today are able to ship OpenTelemetry spans as part of a trace. This means that if you have any component in your application that emits an OpenTelemetry span, it’ll be part of the trace the Elastic APM agent captures.

OpenTelemetry logs in Elastic

If you look at OpenTelemetry documentation, you will see that a lot of language libraries are still in experimental or not implemented yet state. Java is in stable state, per the documentation. Depending on your service’s language, and your appetite for adventure, there exist several options for exporting logs from your services and applications and marrying them together in your observability backend.

In a previous blog, we discussed 3 different configurations to properly get logging data into Elastic for Java. The blog explores the current state of the art of OpenTelemetry logging and provides guidance on the available approaches with the following tenants in mind:

• Correlation of service logs with OTel-generated tracing where applicable
• Proper capture of exceptions
• Common context across tracing, metrics, and logging
• Support for slf4j key-value pairs (“structured logging”)
• Automatic attachment of metadata carried between services via OTel baggage
• Use of an Elastic Observability backend
• Consistent data fidelity in Elastic regardless of the approach taken

Three models, which are covered in the blog, currently exist for getting your application or service logs to Elastic with correlation to OTel tracing and baggage:

• Output logs from your service (alongside traces and metrics) using an embedded OpenTelemetry Instrumentation library to Elastic via the OTLP protocol
• Write logs from your service to a file scrapped by the OpenTelemetry Collector, which then forwards to Elastic via the OTLP protocol
• Write logs from your service to a file scrapped by Elastic Agent (or Filebeat), which then forwards to Elastic via an Elastic-defined protocol

Note that (1), in contrast to (2) and (3), does not involve writing service logs to a file prior to ingestion into Elastic.

OpenTelemetry is Elastic’s preferred schema

Elastic recently contributed the Elastic Common Schema (ECS) to the OpenTelemetry (OTel) project, enabling a unified data specification for security and observability data within the OTel Semantic Conventions framework.

ECS, an open source specification, was developed with support from the Elastic user community to define a common set of fields to be used when storing event data in Elasticsearch^®. ECS helps reduce management and storage costs stemming from data duplication, improving operational efficiency.

Similarly, OTel’s Semantic Conventions (SemConv) also specify common names for various kinds of operations and data. The benefit of using OTel SemConv is in following a common naming scheme that can be standardized across a codebase, libraries, and platforms for OTel users.

The merging of ECS and OTel SemConv will help advance OTel’s adoption and the continued evolution and convergence of observability and security domains.

Elastic Observability APM and machine learning capabilities

All of Elastic Observability’s APM capabilities are available with OTel data (read more on this in our blog, Independence with OpenTelemetry):

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services
• Transactions (traces)
• ML correlations (specifically for latency)
• Service logs

In addition to Elastic’s APM and unified view of the telemetry data, you will now be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR. Here are some of the ML based AIOps capabilities we have:

• Anomaly detection: Elastic Observability, when turned on (see documentation), automatically detects anomalies by continuously modeling the normal behavior of your OpenTelemetry data — learning trends, periodicity, and more.
• Log categorization: Elastic also identifies patterns in your OpenTelemetry log events quickly, so that you can take action quicker.
• High-latency or erroneous transactions: Elastic Observability’s APM capability helps you discover which attributes are contributing to increased transaction latency and identifies which attributes are most influential in distinguishing between transaction failures and successes.
• Log spike detector helps identify reasons for increases in OpenTelemetry log rates. It makes it easy to find and investigate causes of unusual spikes by using the analysis workflow view.
• Log pattern analysis helps you find patterns in unstructured log messages and makes it easier to examine your data.

Elastic allows you to migrate to OTel on your schedule

Although OpenTelemetry supports many programming languages, the status of its major functional components — metrics, traces, and logs — are still at various stages. Thus migrating applications written in Java, Python, and JavaScript are good choices to start with as their metrics, traces, and logs (for Java) are stable.

For the other languages that are not yet supported, you can easily instrument those using Elastic Agents, therefore running your full stack observability platform in mixed mode (Elastic agents with OpenTelemetry agents).

Here is a simple example:

The above shows a simple variation of our standard Elastic Agent application with one service flipped to OTel — the newsletter-otel service. But we can easily and as needed convert each of these services to OTel as development resources allow.

Hence you can migrate what you need to OpenTelemetry with Elastic as specific languages reach a stable state, and you can then continue your migration to OpenTelemetry agents.

Integrated Kubernetes and OpenTelemetry views in Elastic

Elastic manages your Kubernetes cluster using the Elastic Agent, and you can use it on your Kubernetes cluster where your OpenTelemetry application is running. Hence you can not only use OpenTelemetry for your application, but Elastic can also monitor the corresponding Kubernetes cluster.

There are two configurations for Kubernetes:

1. Simply deploying the Elastic Agent daemon set on the kubernetes cluster. We outline this out in the article entitled Managing your Kubernetes cluster with Elastic Observability. This would also push just the Kubernetes metrics and logs to Elastic.

2. Deploying the Elastic Agent with not only the Kubernetes Daemon set, but also Elastic’s APM integration, the Defend (Security) integration, and Network Packet capture integration to provide more comprehensive Kubernetes cluster observability. We outline this configuration in the following article Modern observability and security on Kubernetes with Elastic and OpenTelemetry.

Both OpenTelemetry visualization examples use the OpenTelemetry demo, and in Elastic, we tie the Kubernetes information with the application to provide you an ability to see Kubernetes information from your traces in APM. This provides a more integrated approach when troubleshooting.

Summary

In essence, Elastic's commitment goes beyond mere support for OpenTelemetry. We are dedicated to ensuring our customers not only adopt OpenTelemetry but thrive with it. Through our solutions, expertise, and resources, we aim to elevate the observability journey for every business, turning data into actionable insights that drive growth and innovation.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

AWS Fargate is a serverless pay-as-you-go engine used for Amazon Elastic Container Service (ECS) to run Docker containers without having to manage servers or clusters. The goal of Fargate is to containerize your application and specify the OS, CPU and memory, networking, and IAM policies needed for launch. Additionally, AWS Fargate can be used with Elastic Kubernetes Service (EKS) in a similar manner.

Although the provisioning of servers would be handled by a third party, the need to understand the health and performance of containers within your serverless environment becomes even more vital in identifying root causes and system interruptions. Serverless still requires observability. Elastic Observability can provide observability for not only AWS ECS with Fargate, as we will discuss in this blog, but also for a number of AWS services (EC2, RDS, ELB, etc). See our previous blog on managing an EC2-based application with Elastic Observability.

Gaining full visibility with Elastic Observability

Elastic Observability is governed by the three pillars involved in creating full visibility within a system: logs, metrics, and traces. Logs list all the events that have taken place in the system. Metrics keep track of data that will tell you if the system is down, like response time, CPU usage, memory usage, and latency. Traces give a good indication of the performance of your system based on the execution of requests.

These pillars by themselves offer some insight, but combining them allows for you to see the full scope of your system and how it handles increases in load or traffic over time. Connecting Elastic Observability to your serverless environment will help you deal with outages quicker and perform root cause analysis to prevent any future problems.

In this article, we’ll guide you through how to install the Elastic Agent with the AWS Fargate integration as a sidecar container to send host metrics and logs to Elastic Observability.

Prerequisites:

• AWS account with AWS CLI configured
• GitHub account
• Elastic Cloud account
• An app running on a container in AWS

This tutorial is divided into two parts:

1. Set up the Fleet server to be used by the sidecar container in AWS.
2. Create the sidecar container in AWS Fargate to send data back to Elastic Observability.

Part I: Set up the Fleet server

First, let’s log in to Elastic Cloud.

You can either create a new deployment or use an existing one.

From the Home page, use the side panel to scroll to Management > Fleet > Agent policies. Click Add policy.

Click Create agent policy. Here we’ll create a policy to attach to the Fleet agent.

Give the policy a name and save changes.

Click Create agent policy. You should see the agent policy AWS Fargate in the list of policies.

Now that we have an agent policy, let’s add the integration to collect logs and metrics from the host. Click on AWS Fargate -> Add integration.

We’ll be adding to the policy AWS to collect overall AWS metrics and AWS Fargate to collect metrics from this integration. You can find each one by typing them in the search bar.

Once you click on the integration, it will take you to its landing page, where you can add it to the policy.

For the AWS integration, the only collection settings that we will configure are Collect billing metrics, Collect logs from CloudWatch, Collect metrics from CloudWatch, Collect ECS metrics, and Collect Usage metrics. Everything else can be left disabled.

Another thing to keep in mind when using this integration is the set of permissions required to collect data from AWS. This can be found on the AWS integration page under AWS permissions. Take note of these permissions, as we will use them to create an IAM policy.

Next, we will add the AWS Fargate integration, which doesn’t require further configuration settings.

Now that we have created the agent policy and attached the proper integrations, let’s create the agent that will implement the policy. Navigate back to the main Fleet page and click Add agent.

Since we’ll be connecting to AWS Fargate through ECS, the host type should be set to this value. All the other default values can stay the same.

Lastly, let’s create the enrollment token and attach the agent policy. This will enable AWS ECS Fargate to access Elastic and send data.

Once created, you should be able to see policy name, secret, and agent policy listed.

We’ll be using our Fleet credentials in the next step to send data to Elastic from AWS Fargate.

Part II: Send data to Elastic Observability

It’s time to create our ECS Cluster, Service, and task definition in order to start running the container.

Log in to your AWS account and navigate to ECS.

We’ll start by creating the cluster.

Add a name to the Cluster. And for subnets, only select the first two for us-east-1a and us-eastlb.

For the sake of the demo, we’ll keep the rest of the options set to default. Click Create.

We should see the cluster we created listed below.

Now that we’ve created our cluster to host our container, we want to create a task definition that will be used to set up our container. But before we do this, we will need to create a task role with an associated policy. This task role will allow for AWS metrics to be sent from AWS to the Elastic Agent.

Navigate to IAM in AWS.

Go to Policies -> Create policy.

Now we will reference the AWS permissions from the Fleet AWS integration page and use them to configure the policy. In addition to these permissions, we will also add the GetAtuhenticationToken action for ECR.

You can configure each one using the visual editor.

Or, use the JSON option. Don’t forget to replace the \<account_id> with your own.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "sqs:DeleteMessage",
        "sqs:ChangeMessageVisibility",
        "sqs:ReceiveMessage",
        "ecr:GetDownloadUrlForLayer",
        "ecr:UploadLayerPart",
        "ecr:PutImage",
        "sts:AssumeRole",
        "rds:ListTagsForResource",
        "ecr:BatchGetImage",
        "ecr:CompleteLayerUpload",
        "rds:DescribeDBInstances",
        "logs:FilterLogEvents",
        "ecr:InitiateLayerUpload",
        "ecr:BatchCheckLayerAvailability"
      ],
      "Resource": [
        "arn:aws:iam::<account_id>:role/*",
        "arn:aws:logs:*:<account_id>:log-group:*",
        "arn:aws:sqs:*:<account_id>:*",
        "arn:aws:ecr:*:<account_id>:repository/*",
        "arn:aws:rds:*:<account_id>:target-group:*",
        "arn:aws:rds:*:<account_id>:subgrp:*",
        "arn:aws:rds:*:<account_id>:pg:*",
        "arn:aws:rds:*:<account_id>:ri:*",
        "arn:aws:rds:*:<account_id>:cluster-snapshot:*",
        "arn:aws:rds:*:<account_id>:cev:*/*/*",
        "arn:aws:rds:*:<account_id>:og:*",
        "arn:aws:rds:*:<account_id>:db:*",
        "arn:aws:rds:*:<account_id>:es:*",
        "arn:aws:rds:*:<account_id>:db-proxy-endpoint:*",
        "arn:aws:rds:*:<account_id>:secgrp:*",
        "arn:aws:rds:*:<account_id>:cluster:*",
        "arn:aws:rds:*:<account_id>:cluster-pg:*",
        "arn:aws:rds:*:<account_id>:cluster-endpoint:*",
        "arn:aws:rds:*:<account_id>:db-proxy:*",
        "arn:aws:rds:*:<account_id>:snapshot:*"
      ]
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "sqs:ListQueues",
        "organizations:ListAccounts",
        "ec2:DescribeInstances",
        "tag:GetResources",
        "cloudwatch:GetMetricData",
        "ec2:DescribeRegions",
        "iam:ListAccountAliases",
        "sns:ListTopics",
        "sts:GetCallerIdentity",
        "cloudwatch:ListMetrics"
      ],
      "Resource": "*"
    },
    {
      "Sid": "VisualEditor2",
      "Effect": "Allow",
      "Action": "ecr:GetAuthorizationToken",
      "Resource": "arn:aws:ecr:*:<account_id>:repository/*"
    }
  ]
}

Review your changes.

Now let’s attach this policy to a role. Navigate to IAM -> Roles. Click Create role.

Select AWS service as Trusted entity type and select EC2 as Use case. Click Next.

Under permissions policies, select the policy we just created, as well as CloudWatchLogsFullAccess and AmazonEC2ContainerRegistryFullAccess. Click Next.

Give the task role a name and description.

Click Create role.

Now it’s time to create the task definition. Navigate to ECS -> Task definitions. Click Create new task definition.

Let’s give this task definition a name.

After giving the task definition a name, you’ll add the Fleet credentials to the container section, which you can obtain from the Enrollment Tokens section of the Fleet section in Elastic Cloud. This allows us to host the Elastic Agent on the ECS container as a sidecar and send data to Elastic using Fleet credentials.

• Container name: elastic-agent-container

• Image: docker.elastic.co/beats/elastic-agent:8.19.13

Now let’s add the environment variables:

• FLEET_ENROLL: yes

• FLEET_ENROLLMENT_TOKEN: \

• FLEET_URL: \

For the sake of the demo, leave Environment, Monitoring, Storage, and Tags as default values. Now we will need to create a second container to run the image for the golang app stored in ECR. Click Add more containers.

For Environment, we will reserve 1 vCPU and 3 GB of memory. Under Task role, search for the role we created that uses the IAM policy.

Review the changes, then click Create.

You should see your new task definition included in the list.

The final step is to create the service that will connect directly to the fleet server.
Navigate to the cluster you created and click Create under the Service tab.

Let’s get our service environment configured.

Set up the deployment configuration. Here you should provide the name of the task definition you created in the previous step. Also, provide the service with a unique name. Set the number of desired tasks to 2 instead of 1.

Click Create. Now your service is running two tasks in your cluster using the task definition you provided.

To recap, we set up a Fleet server in Elastic Cloud to receive AWS Fargate data. We then created our AWS Fargate cluster task definition with the Fleet credentials implemented within the container. Lastly, we created the service to send data about our host to Elastic.

Now let’s verify our Elastic Agent is healthy and properly receiving data from AWS Fargate.

We can also view a better breakdown of our agent on the Observability Overview page.

If we drill down to hosts, by clicking on host name we should be able to see more granular data. For instance, we can see the CPU Usage of the Elastic Agent itself that is deployed in our AWS Fargate environment.

Lastly, we can view the AWS Fargate dashboard generated using the data collected by our Elastic Agent. This is an out-of-the-box dashboard that can also be customized based on the data you would like to visualize.

As you can see in the dashboard we’re able to filter based on running tasks, as well as see a list of containers running in our environment. Something else that could be useful to show is the CPU usage per cluster as shown under CPU Utilization per Cluster.

The dashboard can pull data from different sources and in this case shows data for both AWS Fargate and the greater ECS cluster. The two containers at the bottom display the CPU and memory usage directly from ECS.

Conclusion

In this article, we showed how to send data from AWS Fargate to Elastic Observability using the Elastic Agent and Fleet. Serverless architectures are quickly becoming industry standard in offloading the management of servers to third parties. However, this does not alleviate the responsibility of operations engineers to manage the data generated within these environments. Elastic Observability provides a way to not only ingest the data from serverless architectures, but also establish a roadmap to address future problems.

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

More resources on serverless and observability and AWS:

• Analyze your AWS application’s service metrics on Elastic Observability (EC2, ELB, RDS, and NAT)
• Get visibility into AWS Lambda serverless functions with Elastic Observability
• Trace-based testing with Elastic APM and Tracetest
• Sending AWS logs into Elastic via AWS Firehose

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Why do we need alerts?

Logs, metrics, and traces are just the base to build a complete monitoring solution for Kubernetes clusters. Their main goal is to provide debugging information and historical evidence for the infrastructure.

While out-of-the-box dashboards, infrastructure topology, and logs exploration through Kibana are already quite handy to perform ad-hoc analyses, adding notifications and active monitoring of infrastructure allows users to deal with problems detected as early as possible and even proactively take actions to prevent their Kubernetes environments from facing even more serious issues.

How can this be achieved?

By building alerts on top of their infrastructure, users can leverage the data and effectively correlate it to a specific notification, creating a wide range of possibilities to dynamically monitor and observe their Kubernetes cluster.

In this blog post, we will explore how users can leverage Elasticsearch’s search powers to define alerting rules in order to be notified when a specific condition occurs.

SLIs, alerts, and SLOs: Why are they important for SREs?

For site reliability engineers (SREs), the incident response time is tightly coupled with the success of everyday work. Monitoring, alerting, and actions will help to discover, resolve, or prevent issues in their systems.

• An SLA (Service Level Agreement) is an agreement you create with your users to specify the level of service they can expect.
• An SLO (Service Level Objective) is an agreement within an SLA about a specific metric like uptime or response time.
• An SLI (Service Level Indicator) measures compliance with an SLO.

SREs’ day-to-day tasks and projects are driven by SLOs. By ensuring that SLOs are defended in the short term and that they can be maintained in the medium to long term, we lay the basis of a stable working infrastructure.

Having said this, identifying the high-level categories of SLOs is crucial in order to organize the work of an SRE. Then in each category of SLOs, SREs will need the corresponding SLIs that can cover the most important cases of their system under observation. Therefore, the decision of which SLIs we will need demands additional knowledge of the underlying system infrastructure.

One widely used approach to categorize SLIs and SLOs is the Four Golden Signals method. The categories defined are Latency, Traffic, Errors, and Saturation.

A more specific approach is the The RED method developed by Tom Wilkie, who was an SRE at Google and used the Four Golden Signals. The RED method drops the saturation category because this one is mainly used for more advanced cases — and people remember better things that come in threes.

Focusing on Kubernetes infrastructure operators, we will consider the following groups of infrastructure SLIs/SLOs:

• Group 1: Latency of control plane (apiserver,
• Group 2: Resource utilization of the nodes/pods (how much cpu, memory, etc. is consumed)
• Group 3: Errors (errors on logs or events or error count from components, network, etc.)

Creating alerts for a Kubernetes cluster

Now that we have a complete outline of our goal to define alerts based on SLIs/SLOs, we will dive into defining the proper alerting. Alerts can be built using Kibana.

See Elastic documentation.

In this blog, we will define more complex alerts based on complex Elasticsearch queries provided by Watcher’s functionality. Read more about Watcher and how to properly use it in addition to the examples in this blog.

Latency alerts

For this kind of alert, we want to define the basic SLOs for a Kubernetes control plane, which will ensure that the basic control plane components can service the end users without an issue. For instance, facing high latencies in queries against the Kubernetes API Server is enough of a signal that action needs to be taken.

Resource saturation

The next group of alerting will be resource utilization. Node’s CPU utilization or changes in Node’s condition is something critical for a cluster to ensure the smooth servicing of the workloads provisioned to run the applications that end users will interact with.

Error detection

Last but not least, we will define alerts based on specific errors like the network error rate or Pods’ failures like the OOMKilled situation. It’s a very useful indicator for SRE teams to either detect issues on the infrastructure level or just be able to notify developer teams about problematic workloads. One example that we will examine later is having an application running as a Pod and constantly getting restarted because it hits its memory limit. In that case, the owners of this application will need to get notified to act properly.

From Kubernetes data to Elasticsearch queries

Having a solid plan about the alerts that we want to implement, it's time to explore the data we have collected from the Kubernetes cluster and stored in Elasticsearch. For this we will consult the list of the available data fields that are ingested using the Elastic Agent Kubernetes integration (the full list of fields can be found here). Using these fields we can create various alerts like:

• Node CPU utilization
• Node Memory utilization
• BW utilization
• Pod restarts
• Pod CPU/memory utilization

CPU utilization alert

Our first example will use the CPU utilization fields to calculate the Node’s CPU utilization and create an alert. For this alert, we leverage the metrics:

kubernetes.node.cpu.usage.nanocores
kubernetes.node.cpu.capacity.cores.

The following calculation (nodeUsage / 1000000000 ) /nodeCap grouped by node name will give us the CPU utilization of our cluster’s nodes.

The Watcher definition that implements this query can be created with the following API call to Elasticsearch:

curl -X PUT "https://elastic:changeme@localhost:9200/_watcher/watch/Node-CPU-Usage?pretty" -k -H 'Content-Type: application/json' -d'
{
  "trigger": {
    "schedule": {
      "interval": "10m"
    }
  },
  "input": {
    "search": {
      "request": {
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-10m",
                      "lte": "now",
                      "format": "strict_date_optional_time"
                    }
                  }
                },
                {
                  "bool": {
                    "must": [
                      {
                        "query_string": {
                          "query": "data_stream.dataset: kubernetes.node OR data_stream.dataset: kubernetes.state_node",
                          "analyze_wildcard": true
                        }
                      }
                    ],
                    "filter": [],
                    "should": [],
                    "must_not": []
                  }
                }
              ],
              "filter": [],
              "should": [],
              "must_not": []
            }
          },
          "aggs": {
            "nodes": {
              "terms": {
                "field": "kubernetes.node.name",
                "size": "10000",
                "order": {
                  "_key": "asc"
                }
              },
              "aggs": {
                "nodeUsage": {
                  "max": {
                    "field": "kubernetes.node.cpu.usage.nanocores"
                  }
                },
                "nodeCap": {
                  "max": {
                    "field": "kubernetes.node.cpu.capacity.cores"
                  }
                },
                "nodeCPUUsagePCT": {
                  "bucket_script": {
                    "buckets_path": {
                      "nodeUsage": "nodeUsage",
                      "nodeCap": "nodeCap"
                    },
                    "script": {
                      "source": "( params.nodeUsage / 1000000000 ) / params.nodeCap",
                      "lang": "painless",
                      "params": {
                        "_interval": 10000
                      }
                    },
                    "gap_policy": "skip"
                  }
                }
              }
            }
          }
        },
        "indices": [
          "metrics-kubernetes*"
        ]
      }
    }
  },
  "condition": {
    "array_compare": {
      "ctx.payload.aggregations.nodes.buckets": {
        "path": "nodeCPUUsagePCT.value",
        "gte": {
          "value": 80
        }
      }
    }
  },
  "actions": {
    "log_hits": {
      "foreach": "ctx.payload.aggregations.nodes.buckets",
      "max_iterations": 500,
      "logging": {
        "text": "Kubernetes node found with high CPU usage: {{ctx.payload.key}} -> {{ctx.payload.nodeCPUUsagePCT.value}}"
      }
    }
  },
  "metadata": {
    "xpack": {
      "type": "json"
    },
    "name": "Node CPU Usage"
  }
}

OOMKilled Pods detection and alerting

Another Watcher that we will explore is the one that detects Pods that have been restarted due to an OOMKilled error. This error is quite common in Kubernetes workloads and is useful to detect this early on to inform the team that owns this workload, so they can either investigate issues that could cause memory leaks or just consider increasing the required resources for the workload itself.

This information can be retrieved from a query like the following:

kubernetes.container.status.last_terminated_reason: OOMKilled

Here is how we can create the respective Watcher with an API call:

curl -X PUT "https://elastic:changeme@localhost:9200/_watcher/watch/Pod-Terminated-OOMKilled?pretty" -k -H 'Content-Type: application/json' -d'
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-1m",
                      "lte": "now",
                      "format": "strict_date_optional_time"
                    }
                  }
                },
                {
                  "bool": {
                    "must": [
                      {
                        "query_string": {
                          "query": "data_stream.dataset: kubernetes.state_container",
                          "analyze_wildcard": true
                        }
                      },
                      {
                        "exists": {
                          "field": "kubernetes.container.status.last_terminated_reason"
                        }
                      },
                      {
                        "query_string": {
                          "query": "kubernetes.container.status.last_terminated_reason: OOMKilled",
                          "analyze_wildcard": true
                        }
                      }
                    ],
                    "filter": [],
                    "should": [],
                    "must_not": []
                  }
                }
              ],
              "filter": [],
              "should": [],
              "must_not": []
            }
          },
          "aggs": {
            "pods": {
              "terms": {
                "field": "kubernetes.pod.name",
                "order": {
                  "_key": "asc"
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "array_compare": {
      "ctx.payload.aggregations.pods.buckets": {
        "path": "doc_count",
        "gte": {
          "value": 1,
          "quantifier": "some"
        }
      }
    }
  },
  "actions": {
    "ping_slack": {
      "foreach": "ctx.payload.aggregations.pods.buckets",
      "max_iterations": 500,
      "webhook": {
        "method": "POST",
        "url": "https://hooks.slack.com/services/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",
        "body": "{\"channel\": \"#k8s-alerts\", \"username\": \"k8s-cluster-alerting\", \"text\": \"Pod {{ctx.payload.key}} was terminated with status OOMKilled.\"}"
      }
    }
  },
  "metadata": {
    "xpack": {
      "type": "json"
    },
    "name": "Pod Terminated OOMKilled"
  }
}

From Kubernetes data to alerts summary

So far we saw how we can start from plain Kubernetes fields, use them in ES queries, and build Watchers and alerts on top of them.

One can explore more possible data combinations and build queries and alerts following the examples we provided here. A full list of alerts is available, as well as a basic scripted way of installing them.

Of course, these examples come with simple actions defined that only log messages into the Elasticsearch logs. However, one can use more advanced and useful outputs like Slack’s webhooks:

"actions": {
    "ping_slack": {
      "foreach": "ctx.payload.aggregations.pods.buckets",
      "max_iterations": 500,
      "webhook": {
        "method": "POST",
        "url": "https://hooks.slack.com/services/T04SW3JHXasdfasdfasdfasdfasdf",
        "body": "{\"channel\": \"#k8s-alerts\", \"username\": \"k8s-cluster-alerting\", \"text\": \"Pod {{ctx.payload.key}} was terminated with status OOMKilled.\"}"
      }
    }
  }

The result would be a Slack message like the following:

Next steps

In our next steps, we would like to make these alerts part of our Kubernetes integration, which would mean that the predefined alerts would be installed when users install or enable the Kubernetes integration. At the same time, we plan to implement some of these as Kibana’s native SLIs, providing the option to our users to quickly define SLOs on top of the SLIs through a nice user interface. If you’re interested to learn more about these, follow the public GitHub issues for more information and feel free to provide your feedback:

• https://github.com/elastic/package-spec/issues/484
• https://github.com/elastic/kibana/issues/150050

For those who are eager to start using Kubernetes alerting today, here is what you need to do:

1. Make sure that you have an Elastic cluster up and running. The fastest way to deploy your cluster is to spin up a free trial of Elasticsearch Service.
2. Install the latest Elastic Agent on your Kubernetes cluster following the respective documentation.
3. Install our provided alerts that can be found at https://github.com/elastic/integrations/tree/main/packages/kubernetes/docs or at https://github.com/elastic/k8s-integration-infra/tree/main/scripts/alerting.

Of course, if you have any questions, remember that we are always happy to help on the Discuss forums.

Elastic helps reduce the operational work to support more efficient operations, but users still need a way to investigate and understand everything from the cause of an issue to the meaning of specific error messages. As an operations user, if you haven’t run into a particular error before or it's part of some runbook, you will likely go to Google and start searching for information.

OpenAI’s ChatGPT is becoming an interesting generative AI tool that helps provide more information using the models behind it. What if you could use OpenAI to obtain deeper insights (even simple semantics) for an error in your production or development environment? You can easily tie Elastic to OpenAI’s API to achieve this.

Kubernetes, a mainstay in most deployments (on-prem or in a cloud service provider) requires a significant amount of expertise — even if that expertise is to manage a service like GKE, EKS, or AKS.

In this blog, I will cover how you can use Elastic’s watcher capability to connect Elastic to OpenAI and ask it for more information about the error logs Elastic is ingesting from a Kubernetes cluster(s). More specifically, we will use Azure’s OpenAI Service. Azure OpenAI is a partnership between Microsoft and OpenAI, so the same models from OpenAI are available in the Microsoft version.

While this blog goes over a specific example, it can be modified for other types of errors Elastic receives in logs. Whether it's from AWS, the application, databases, etc., the configuration and script described in this blog can be modified easily.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up the configuration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• We used a GCP GKE Kubernetes cluster, but you can use any Kubernetes cluster service (on-prem or cloud based) of your choice.
• We’re also running with a version of the OpenTelemetry Demo. Directions for using Elastic with OpenTelemetry Demo are here.
• We also have an Azure account and Azure OpenAI service configured. You will need to get the appropriate tokens from Azure and the proper URL endpoint from Azure’s OpenAI service.
• We will use Elastic’s dev tools, the console to be specific, to load up and run the script, which is an Elastic watcher.
• We will also add a new index to store the results from the OpenAI query.

Here is the configuration we will set up in this blog:

As we walk through the setup, we’ll also provide the alternative setup with OpenAI versus Azure OpenAI Service.

Setting it all up

Over the next few steps, I’ll walk through:

• Getting an account on Elastic Cloud and setting up your K8S cluster and application
• Gaining Azure OpenAI authorization (alternative option with OpenAI)
• Identifying Kubernetes error logs
• Configuring the watcher with the right script
• Comparing the output from Azure OpenAI/OpenAI versus ChatGPT UI

Step 0: Create an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Once you have the Elastic Cloud login, set up your Kubernetes cluster and application. A complete step-by-step instructions blog is available here. This also provides an overview of how to see Kubernetes cluster metrics in Elastic and how to monitor them with dashboards.

Step 1: Azure OpenAI Service and authorization

When you log in to your Azure subscription and set up an instance of Azure OpenAI Service, you will be able to get your keys under Manage Keys.

There are two keys for your OpenAI instance, but you only need KEY 1 .

Additionally, you will need to get the service URL. See the image above with our service URL blanked out to understand where to get the KEY 1 and URL.

If you are not using Azure OpenAI Service and the standard OpenAI service, then you can get your keys at:

**https** ://platform.openai.com/account/api-keys

You will need to create a key and save it. Once you have the key, you can go to Step 2.

Step 2: Identifying Kubernetes errors in Elastic logs

As your Kubernetes cluster is running, Elastic’s Kubernetes integration running on the Elastic agent daemon set on your cluster is sending logs and metrics to Elastic. The telemetry is ingested, processed, and indexed. Kubernetes logs are stored in an index called .ds-logs-kubernetes.container_logs-default-* (* is for the date), and an automatic data stream logs-kubernetes.container_logs is also pre-loaded. So while you can use some of the out-of-the-box dashboards to investigate the metrics, you can also look at all the logs in Elastic Discover.

While any error from Kubernetes can be daunting, the more nuanced issues occur with errors from the pods running in the kube-system namespace. Take the pod konnectivity agent, which is essentially a network proxy agent running on the node to help establish tunnels and is a vital component in Kubernetes. Any error will cause the cluster to have connectivity issues and lead to a cascade of issues, so it’s important to understand and troubleshoot these errors.

When we filter out for error logs from the konnectivity agent, we see a good number of errors.

But unfortunately, we still can’t understand what these errors mean.

Enter OpenAI to help us understand the issue better. Generally, you would take the error message from Discover and paste it with a question in ChatGPT (or run a Google search on the message).

One error in particular that we’ve run into but do not understand is:

E0510 02:51:47.138292       1 client.go:388] could not read stream err=rpc error: code = Unavailable desc = error reading from server: read tcp 10.120.0.8:46156->35.230.74.219:8132: read: connection timed out serverID=632d489f-9306-4851-b96b-9204b48f5587 agentID=e305f823-5b03-47d3-a898-70031d9f4768

The OpenAI output is as follows:

ChatGPT has given us a fairly nice set of ideas on why this rpc error is occurring against our konnectivity-agent.

So how can we get this output automatically for any error when those errors occur?

Step 3: Configuring the watcher with the right script

What is an Elastic watcher? Watcher is an Elasticsearch feature that you can use to create actions based on conditions, which are periodically evaluated using queries on your data. Watchers are helpful for analyzing mission-critical and business-critical streaming data. For example, you might watch application logs for errors causing larger operational issues.

Once a watcher is configured, it can be:

1. Manually triggered
2. Run periodically
3. Created using a UI or a script

In this scenario, we will use a script, as we can modify it easily and run it as needed.

We’re using the DevTools Console to enter the script and test it out:

The script is listed at the end of the blog in the appendix. It can also be downloaded here .

The script does the following:

1. It runs continuously every five minutes.
2. It will search the logs for errors from the container konnectivity-agent.
3. It will take the first error’s message, transform it (re-format and clean up), and place it into a variable first_hit.

"script": "return ['first_hit': ctx.payload.first.hits.hits.0._source.message.replace('\"', \"\")]"

1. The error message is sent into OpenAI with a query:

What are the potential reasons for the following kubernetes error:
  { { ctx.payload.second.first_hit } }

1. If the search yielded an error, it will proceed to then create an index and place the error message, pod.name (which is konnectivity-agent-6676d5695b-ccsmx in our setup), and OpenAI output into a new index called chatgpt_k8_analyzed.

To see the results, we created a new data view called chatgpt_k8_analyzed against the newly created index:

In Discover, the output on the data view provides us with the analysis of the errors.

For every error the script sees in the five minute interval, it will get an analysis of the error. We could alternatively also use a range as needed to analyze during a specific time frame. The script would just need to be modified accordingly.

Step 4. Output from Azure OpenAI/OpenAI vs. ChatGPT UI

As you noticed above, we got relatively the same result from the Azure OpenAI API call as we did by testing out our query in the ChatGPT UI. This is because we configured the API call to run the same/similar model as what was selected in the UI.

For the API call, we used the following parameters:

"request": {
             "method" : "POST",
             "Url": "https://XXX.openai.azure.com/openai/deployments/pme-gpt-35-turbo/chat/completions?api-version=2023-03-15-preview",
             "headers": {"api-key" : "XXXXXXX",
                         "content-type" : "application/json"
                        },
             "body" : "{ \"messages\": [ { \"role\": \"system\", \"content\": \"You are a helpful assistant.\"}, { \"role\": \"user\", \"content\": \"What are the potential reasons for the following kubernetes error: {{ctx.payload.second.first_hit}}\"}], \"temperature\": 0.5, \"max_tokens\": 2048}" ,
              "connection_timeout": "60s",
               "read_timeout": "60s"
                            }

By setting the role: system with You are a helpful assistant and using the gpt-35-turbo url portion, we are essentially setting the API to use the davinci model, which is the same as the ChatGPT UI model set by default.

Additionally, for Azure OpenAI Service, you will need to set the URL to something similar the following:

https://YOURSERVICENAME.openai.azure.com/openai/deployments/pme-gpt-35-turbo/chat/completions?api-version=2023-03-15-preview

If you use OpenAI (versus Azure OpenAI Service), the request call (against https://api.openai.com/v1/completions) would be as such:

"request": {
            "scheme": "https",
            "host": "api.openai.com",
            "port": 443,
            "method": "post",
            "path": "\/v1\/completions",
            "params": {},
            "headers": {
               "content-type": "application\/json",
               "authorization": "Bearer YOUR_ACCESS_TOKEN"
                        },
            "body": "{ \"model\": \"text-davinci-003\",  \"prompt\": \"What are the potential reasons for the following kubernetes error: {{ctx.payload.second.first_hit}}\",  \"temperature\": 1,  \"max_tokens\": 512,     \"top_p\": 1.0,      \"frequency_penalty\": 0.0,   \"presence_penalty\": 0.0 }",
            "connection_timeout_in_millis": 60000,
            "read_timeout_millis": 60000
          }

If you are interested in creating a more OpenAI-based version, you can download an alternative script and look at another blog from an Elastic community member.

Gaining other insights beyond Kubernetes logs

Now that the script is up and running, you can modify it using different:

• Inputs
• Conditions
• Actions
• Transforms

Learn more on how to modify it here. Some examples of modifications could include:

1. Look for error logs from application components (e.g., cartService, frontEnd, from the OTel demo), cloud service providers (e.g., AWS/Azure/GCP logs), and even logs from components such as Kafka, databases, etc.
2. Vary the time frame from running continuously to running over a specific range.
3. Look for specific errors in the logs.
4. Query for analysis on a set of errors at once versus just one, which we demonstrated.

The modifications are endless, and of course you can run this with OpenAI rather than Azure OpenAI Service.

Conclusion

I hope you’ve gotten an appreciation for how Elastic Observability can help you connect to OpenAI services (Azure OpenAI, as we showed, or even OpenAI) to better analyze an error log message instead of having to run several Google searches and hunt for possible insights.

Here’s a quick recap of what we covered:

• Developing an Elastic watcher script that can be used to find and send Kubernetes errors into OpenAI and insert them into a new index
• Configuring Azure OpenAI Service or OpenAI with the right authorization and request parameters

Ready to get started? Sign up for Elastic Cloud and try out the features and capabilities I’ve outlined above to get the most value and visibility out of your OpenTelemetry data.

Appendix

Watcher script

PUT _watcher/watch/chatgpt_analysis
{
    "trigger": {
      "schedule": {
        "interval": "5m"
      }
    },
    "input": {
      "chain": {
          "inputs": [
              {
                  "first": {
                      "search": {
                          "request": {
                              "search_type": "query_then_fetch",
                              "indices": [
                                "logs-kubernetes*"
                              ],
                              "rest_total_hits_as_int": true,
                              "body": {
                                "query": {
                                  "bool": {
                                    "must": [
                                      {
                                        "match": {
                                          "kubernetes.container.name": "konnectivity-agent"
                                        }
                                      },
                                      {
                                        "match" : {
                                          "message":"error"
                                        }
                                      }
                                    ]
                                  }
                                },
                                "size": "1"
                              }
                            }
                        }
                    }
                },
                {
                    "second": {
                        "transform": {
                            "script": "return ['first_hit': ctx.payload.first.hits.hits.0._source.message.replace('\"', \"\")]"
                        }
                    }
                },
                {
                    "third": {
                        "http": {
                            "request": {
                                "method" : "POST",
                                "url": "https://XXX.openai.azure.com/openai/deployments/pme-gpt-35-turbo/chat/completions?api-version=2023-03-15-preview",
                                "headers": {
                                    "api-key" : "XXX",
                                    "content-type" : "application/json"
                                },
                                "body" : "{ \"messages\": [ { \"role\": \"system\", \"content\": \"You are a helpful assistant.\"}, { \"role\": \"user\", \"content\": \"What are the potential reasons for the following kubernetes error: {{ctx.payload.second.first_hit}}\"}], \"temperature\": 0.5, \"max_tokens\": 2048}" ,
                                "connection_timeout": "60s",
                                "read_timeout": "60s"
                            }
                        }
                    }
                }
            ]
        }
    },
    "condition": {
      "compare": {
        "ctx.payload.first.hits.total": {
          "gt": 0
        }
      }
    },
    "actions": {
        "index_payload" : {
            "transform": {
                "script": {
                    "source": """
                        def payload = [:];
                        payload.timestamp = new Date();
                        payload.pod_name = ctx.payload.first.hits.hits[0]._source.kubernetes.pod.name;
                        payload.error_message = ctx.payload.second.first_hit;
                        payload.chatgpt_analysis = ctx.payload.third.choices[0].message.content;
                        return payload;
                    """
                }
            },
            "index" : {
                "index" : "chatgpt_k8s_analyzed"
            }
        }
    }
}

Additional logging resources:

• Getting started with logging on Elastic (quickstart)
• Ingesting common known logs via integrations (compute node example)
• List of integrations
• Ingesting custom application logs into Elastic
• Enriching logs in Elastic
• Analyzing Logs with Anomaly Detection (ML) and AIOps

Common use case examples with logs:

• Nginx log management
• AWS VPC Flow log management
• PostgreSQL issue analysis with AIOps

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Screenshots of Microsoft products used with permission from Microsoft.

Using full-stack Elastic Observability to understand Kafka and Confluent performance

In the 2023 Dice Tech Salary Report, Elasticsearch and Kakfa are ranked #3 and #5 out of the top 12 most in demand skills at the moment, so it’s no surprise that we are seeing a large number of customers who are implementing data in motion with Kafka.

Kafka comes with some additional complexities that go beyond traditional architectures and which make observability an even more important topic. Understanding where the bottlenecks are in messaging and stream-based architectures can be tough. This is why you need a comprehensive observability solution with machine learning to help you.

In this blog, we will explore how to get Kafka applications instrumented with Elastic APM, how to collect performance data with JMX, and how you can use the Elasticsearch Platform to pull in data from Confluent Cloud — which is by far the easiest and most cost-effective way to implement Kafka architectures.

For this blog post, we will be following the code at this git repository. There are three services here that are designed to run on two clouds and push data from one cloud to the other and finally into Google BigQuery. We want to monitor all of this using Elastic Observability to give you a complete picture of Confluent and Kafka Services performance as a teaser — this is the goal below:

A look at the architecture

As mentioned, we have three multi-cloud services implemented in our example application.

The first service is a Spring WebFlux service that runs inside AWS EKS. This service will take a message from a REST Endpoint and simply put it straight on to a Kafka topic.

The second service, which is also a Spring WebFlux service hosted inside Google Cloud Platform (GCP) with its Google Cloud monitoring, will then pick this up and forward it to another service that will put the message into BigQuery.

These services are all instrumented using Elastic APM. For this blog, we have decided to use Spring config to inject and configure the APM agent. You could of course use the “-javaagent” argument to inject the agent instead if preferred.

Getting started with Elastic Observability and Confluent Cloud

Before we dive into the application and its configuration, you will want to get an Elastic Cloud and Confluent Cloud account. You can sign up here for Elastic and here for Confluent Cloud. There are some initial configuration steps we need to do inside Confluent Cloud, as you will need to create three topics: gcpTopic, myTopic, and topic_2.

When you sign up for Confluent Cloud, you will be given an option of what type of cluster to create. For this walk-through, a Basic cluster is fine (as shown) — if you are careful about usage, it will not cost you a penny.

Once you have a cluster, go ahead and create the three topics.

For this walk-through, you will only need to create single partition topics as shown below:

Now we are ready to set up the Elastic Cloud cluster.

One thing to note here is that when setting up an Elastic cluster, the defaults are mostly OK. With one minor tweak to add in the Machine Learning under “Advanced Settings,” add capacity for machine learning here.

Getting APM up and running

The first thing we want to do here is get our Spring Boot Webflux-based services up and running. For this blog, I have decided to implement this using the Spring Configuration, as you can see below. For brevity, I have not listed all the JMX configuration information, but you can see those details in GitHub.

package com.elastic.multicloud;
import co.elastic.apm.attach.ElasticApmAttacher;
import jakarta.annotation.PostConstruct;
import lombok.Setter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;

import java.util.HashMap;
import java.util.Map;

@Setter
@Configuration
@ConfigurationProperties(prefix = "elastic.apm")
@ConditionalOnProperty(value = "elastic.apm.enabled", havingValue = "true")
public class ElasticApmConfig {

    private static final String SERVER_URL_KEY = "server_url";
    private String serverUrl;

    private static final String SERVICE_NAME_KEY = "service_name";
    private String serviceName;

    private static final String SECRET_TOKEN_KEY = "secret_token";
    private String secretToken;

    private static final String ENVIRONMENT_KEY = "environment";
    private String environment;

    private static final String APPLICATION_PACKAGES_KEY = "application_packages";
    private String applicationPackages;

    private static final String LOG_LEVEL_KEY = "log_level";
    private String logLevel;
    private static final Logger LOGGER = LoggerFactory.getLogger(ElasticApmConfig.class);

    @PostConstruct
    public void init() {
        LOGGER.info(environment);

        Map<String, String> apmProps = new HashMap<>(6);
        apmProps.put(SERVER_URL_KEY, serverUrl);
        apmProps.put(SERVICE_NAME_KEY, serviceName);
        apmProps.put(SECRET_TOKEN_KEY, secretToken);
        apmProps.put(ENVIRONMENT_KEY, environment);
        apmProps.put(APPLICATION_PACKAGES_KEY, applicationPackages);
        apmProps.put(LOG_LEVEL_KEY, logLevel);
        apmProps.put("enable_experimental_instrumentations","true");
          apmProps.put("capture_jmx_metrics","object_name[kafka.producer:type=producer-metrics,client-id=*] attribute[batch-size-avg:metric_name=kafka.producer.batch-size-avg]");


        ElasticApmAttacher.attach(apmProps);
    }
}

Now obviously this requires some dependencies, which you can see here in the Maven pom.xml.

<dependency>
            <groupId>co.elastic.apm</groupId>
            <artifactId>apm-agent-attach</artifactId>
            <version>1.35.1-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>co.elastic.apm</groupId>
            <artifactId>apm-agent-api</artifactId>
            <version>1.35.1-SNAPSHOT</version>
        </dependency>

Strictly speaking, the agent-api is not required, but it could be useful if you have a desire to add your own monitoring code (as per the example below). The agent will happily auto-instrument without needing to do that though.

Transaction transaction = ElasticApm.currentTransaction();
        Span span = ElasticApm.currentSpan()
                .startSpan("external", "kafka", null)
                .setName("DAVID").setServiceTarget("kafka","gcp-elastic-apm-spring-boot-integration");
        try (final Scope scope = transaction.activate()) {
            span.injectTraceHeaders((name, value) -> producerRecord.headers().add(name,value.getBytes()));
            return Mono.fromRunnable(() -> {
                kafkaTemplate.send(producerRecord);
            });
        } catch (Exception e) {
            span.captureException(e);
            throw e;
        } finally {
            span.end();
        }

Now we have enough code to get our agent bootstrapped.

To get the code from the GitHub repository up and running, you will need the following installed on your system and to ensure that you have the credentials for your GCP and AWS cloud.

Java
Maven
Docker
Kubernetes CLI (kubectl)

Clone the project

Clone the multi-cloud Spring project to your local machine.

git clone https://github.com/davidgeorgehope/multi-cloud

Build the project

From each service in the project (aws-multi-cloud, gcp-multi-cloud, gcp-bigdata-consumer-multi-cloud), run the following commands to build the project.

mvn clean install

Now you can run the Java project locally.

java -jar gcp-bigdata-consumer-multi-cloud-0.0.1-SNAPSHOT.jar --spring.config.location=/Users/davidhope/applicaiton-gcp.properties

That will just get the Java application running locally, but you can also deploy this to Kubernetes using EKS and GKE as shown below.

Create a Docker image

Create a Docker image from the built project using the dockerBuild.sh provided in the project. You may want to customize this shell script to upload the built docker image to your own docker repository.

./dockerBuild.sh

Create a namespace for each service

kubectl create namespace aws

kubectl create namespace gcp-1

kubectl create namespace gcp-2

Once you have the namespaces created, you can switch context using the following command:

kubectl config set-context --current --namespace=my-namespace

Configuration for each service

Each service needs an application.properties file. I have put an example here.

You will need to replace the following properties with those you find in Elastic.

elastic.apm.server-url=
elastic.apm.secret-token=

These can be found by going into Elastic Cloud and clicking on Services inside APM and then Add Data , which should be visible in the top right corner.

From there you will see the following, which gives you the config information you need.

You will need to replace the following properties with those you find in Confluent Cloud.

elastic.kafka.producer.sasl-jaas-config=

This configuration comes from the Clients page in Confluent Cloud.

Adding the config for each service in Kubernetes

Once you have a fully configured application properties, you need to add it to your Kubernetes environment as below.

From the aws namespace.

kubectl create secret generic my-app-config --from-file=application.properties

From the gcp-1 namespace.

kubectl create secret generic my-app-config --from-file=application.properties

From the gcp-2 namespace.

kubectl create secret generic bigdata-creds --from-file=elastic-product-marketing-e145e13fbc7c.json

kubectl create secret generic my-app-config-gcp-bigdata --from-file=application.properties

Create a Kubernetes deployment

Create a Kubernetes deployment YAML file and add your Docker image to it. You can use the deployment.yaml file provided in the project as a template. Make sure to update the image name in the file to match the name of the Docker image you just created.

kubectl apply -f deployment.yaml

Create a Kubernetes service

Create a Kubernetes service YAML file and add your deployment to it. You can use the service.yaml file provided in the project as a template.

kubectl apply -f service.yaml

Access your application

Your application is now running in a Kubernetes cluster. To access it, you can use the service's cluster IP and port. You can get the service's IP and port using the following command.

kubectl get services

Now once you know where the service is, you need to execute it!

You can regularly poke the service endpoint using the following command.

curl -X POST -H "Content-Type: application/json" -d '{"name": "linuxize", "email": "linuxize@example.com"}' http://localhost:8080/api/my-objects/publish

With this up and running, you should see the following service map build out in the Elastic APM product.

And traces will contain a waterfall graph showing all the spans that have executed across this distributed application, allowing you to pinpoint where any issues are within each transaction.

JMX for Kafka Producer/Consumer metrics

In the previous part of this blog, we briefly touched on the JMX metric configuration you can see below.

"capture_jmx_metrics","object_name[kafka.producer:type=producer-metrics,client-id=*] attribute[batch-size-avg:metric_name=kafka.producer.batch-size-avg]"

We can use this “capture_jmx_metrics” configuration to configure JMX for any Kafka Producer/Consumer metrics we want to monitor.

Check out the documentation here to understand how to configure this and here to see the available JMX metrics you can monitor. In the example code in GitHub, we actually pull all the available metrics in, so you can check in there how to configure this.

One thing that’s worth pointing out here is that it’s important to use the “metric_name” property shown above or it gets quite difficult to find the metrics in Elastic Discover without being specific here.

Monitoring Confluent Cloud with Elastic Observability

So we now have some good monitoring set up for Kafka Producers and Consumers and we can trace transactions between services down to the lines of code that are executing. The core part of our Kafka infrastructure is hosted in Confluent Cloud. How, then, do we get data from there into our full stack observability solution?

Luckily, Confluent has done a fantastic job of making this easy. It provides important Confluent Cloud metrics via an open Prometheus-based metrics URL. So let's get down to business and configure this to bring data into our observability tool.

The first step is to configure Confluent Cloud with the MetricsViewer. The MetricsViewer role provides service account access to the Metrics API for all clusters in an organization. This role also enables service accounts to import metrics into third-party metrics platforms.

To assign the MetricsViewer role to a new service account:

1. In the top-right administration menu (☰) in the upper-right corner of the Confluent Cloud user interface, click ADMINISTRATION > Cloud API keys.
2. Click Add key.
3. Click the Granular access tile to set the scope for the API key. Click Next.
4. Click Create a new one and specify the service account name. Optionally, add a description. Click Next.
5. The API key and secret are generated for the service account. You will need this API key and secret to connect to the cluster, so be sure to safely store this information. Click Save. The new service account with the API key and associated ACLs is created. When you return to the API access tab, you can view the newly-created API key to confirm.
6. Return to Accounts & access in the administration menu, and in the Accounts tab, click Service accounts to view your service accounts.
7. Select the service account that you want to assign the MetricsViewer role to.
8. In the service account’s details page, click Access.
9. In the tree view, open the resource where you want the service account to have the MetricsViewer role.
10. Click Add role assignment and select the MetricsViewer tile. Click Save.

Next we can head to Elastic Observability and configure the Prometheus integration to pull in the metrics data.

Go to the integrations page in Kibana.

Find the Prometheus integration. We are using the Prometheus integration because the Confluent Cloud metrics server can provide data in prometheus format. Trust us, this works really well — good work Confluent!

Add Prometheus in the next page.

Configure the Prometheus plugin in the following way: In the hosts box, add the following URL, replacing the resource kafka id with the cluster id you want to monitor.

https://api.telemetry.confluent.cloud:443/v2/metrics/cloud/export?resource.kafka.id=lkc-3rw3gw

Add the username and password under the advanced options you got from the API keys step you executed against Confluent Cloud above.

Once the Integration is created, the policy needs to be applied to an instance of a running Elastic Agent.

That’s it! It’s that easy to get all the data you need for a full stack observability monitoring solution.

Finally, let’s pull all this together in a dashboard.

Pulling it all together

Using Kibana to generate dashboards is super easy. If you configured everything the way we recommended above, you should find the metrics (producer/consumer/brokers) you need to create your own dashboard as per the following screenshot.

Luckily, I made a dashboard for you and stored it in GitHub. Take a look below and use this to import it into your own environments.

Adding the icing on the cake: machine learning anomaly detection

Now that we have all the critical bits in place, we are going to add the icing on the cake: machine learning (ML)!

Within Kibana, let's head over to the Machine Learning tab in “Analytics.”

Go to the jobs page, where we’ll get started creating our first anomaly detection job.

The metrics data view contains what we need to create this new anomaly detection job.

Use the wizard and select a “Single Metric.”

Use the full data.

In this example, we are going to look for anomalies in the connection count. We really do not want a major deviation here, as this could indicate something very bad occurring if we suddenly have too many or too few things connecting to our Kafka cluster.

Once you have selected the connection count metric, you can proceed through the wizard and eventually your ML job will be created and you should be able to view the data as per the example below.

Congratulations, you have now created a machine learning job to alert you if there are any problems with your Kafka cluster, adding a full AIOps solution to your Kafka and Confluent observability!

Summary

We looked at monitoring Kafka-based solutions implemented on Confluent Cloud using Elastic Observability.

We covered the architecture of a multi-cloud solution involving AWS EKS, Confluent Cloud, and GCP GKE. We looked at how to instrument Kafka applications with Elastic APM, use JMX for Kafka Producer/Consumer metrics, integrate Prometheus, and set up machine learning anomaly detection.

We went through a detailed walk-through with code snippets, configuration steps, and deployment instructions included to help you get started.

Interested in learning more about Elastic Observability? Check out the following resources:

• An Introduction to Elastic Observability
• Observability Fundamentals Training
• Watch an Elastic Observability demo
• Observability Predictions and Trends for 2023

And sign up for our Elastic Observability Trends Webinar featuring AWS and Forrester, not to be missed!

Elastic is committed to supporting Google Kubernetes Engine (GKE) in all of its delivery modes. In October, during the Google Cloud Next ‘22 event, we announced our intention to integrate and certify Elastic Agent on Anthos, Autopilot, Google Distributed Cloud, and more.

Since that event, we have worked together with Google to get the Elastic Agent certified for use on Anthos, but we didn’t stop there.

Today we are happy to announce that we have been certified for operation on GKE Autopilot.

Hands on with Elastic and GKE Autopilot

Kubernetes observability has never been easier

To show how easy it is to get started with Autopilot and Elastic, let's walk through deploying the Elastic Agent on an Autopilot cluster. I’ll show how easy it is to set up and monitor an Autopilot cluster with the Elastic Agent and observe the cluster’s behavior with Kibana integrations.

One of the main differences between GKE and GKE Autopilot is that Autopilot protects the system namespace “kube-system.” To increase the stability and security of a cluster, Autopilot prevents user space workloads from adding or modifying system pods. The default configuration for Elastic Agent is to install itself into the system namespace. The majority of the changes we will make here are to convince the Elastic Agent to run in a different namespace.

Let’s get started with Elastic Stack!

While writing this article, I used the latest version of Elastic. The best way for you to get started with Elastic Observability is to:

1. Get an account on Elastic Cloud and look at this tutorial to help launch your first stack, or
2. Launch Elastic Cloud on your Google Account

Provisioning an Autopilot cluster and an Elastic stack

To test the agent, I first deployed the recommended, default GKE Autopilot cluster. Elastic’s GKE integration supports kube-state-metrics (KSM), which will increase the number of reported metrics available for reporting and dashboards. Like the Elastic Agent, KSM defaults to running in the system namespace, so I modified its manifest to work with Autopilot. For my testing, I also deployed a basic Elastic stack on Elastic Cloud in the same Google region as my Autopilot cluster. I used a fresh cluster deployed on Elastic’s managed service (ESS), but the process is the same if you are using an Elastic Cloud subscription purchased through the Google marketplace.

Adding Elastic Observability to GKE Autopilot

Because this is a brand new deployment, Elastic suggests adding integrations to it. Let’s add the Kubernetes integration into the new deployment:

Elastic offers hundreds of integrations; filter the list by typing “kub” into the search bar (1) and then click the Kubernetes integration (2).

The Kubernetes integration page gives you an overview of the integration and lets you manage the Kubernetes clusters you want to observe. We haven’t added a cluster yet, so I clicked “Add Kubernetes” to add the first integration.

I changed the integration name to reflect the Kubernetes offering type and then clicked “Save and continue” to accept the integration defaults.

At this point, an Agent policy has been created. Now it’s time to install the agent. I clicked on the “Kubernetes” integration.

Then I selected the “integration policies” tab (1) and clicked “Add agent” (2).

Finally, I downloaded the full manifest for a standard GKE environment.

We won’t be using this manifest directly, but it contains many of the values that we will need to deploy the agent on Autopilot in the next section.

The Elastic stack is ready and waiting for the Autopilot logs, metrics, and events. It’s time to connect Autopilot to this deployment using the Elastic Agent for GKE.

Connect Autopilot to Elastic

From the Google cloud terminal, I downloaded and edited the Elastic Agent manifest for GKE Autopilot.

$ curl -o elastic-agent-managed-gke-autopilot.yaml \
https://github.com/elastic/elastic-agent/blob/autopilotdocumentaton/docs/manifests/elastic-agent-managed-gke-autopilot.yaml

I used the cloud shell editor to configure the manifest for my Autopilot and Elastic clusters. For example, I updated the following:

containers:
  - name: elastic-agent
    image: docker.elastic.co/beats/elastic-agent:8.19.13

I also changed the agent to the version of Elastic that I installed (8.6.0).

From the Integration manifest I downloaded earlier, I copied the values for FLEET_URL and FLEET_ENROLLMENT_TOKEN into this YAML file.

Now it’s time to apply the updated manifest to the Autopilot instance.

Before I commit, I always like to see what’s going to be created (and check for syntax errors) with a dry run.

$ clear
$ kubectl apply --dry-run="client" -f elastic-agent-managed-gke-autopilot.yaml

Everything looks good, so I’ll do it for real this time.

$ clear
$ kubectl apply -f elastic-agent-managed-gke-autopilot.yaml

After several minutes, metrics will start flowing from the Autopilot cluster directly into the Elastic deployment.

Adding a workload to the Autopilot cluster

Observing an Autopilot cluster without a workload is boring, so I deployed a modified version of Google’s Hipster Shop (which includes OpenTelemetry reporting):

$ git clone https://github.com/bshetti/opentelemetry-microservices-demo
$ cd opentelemetry-microservices-demo
$ nano ./deploy-with-collector-k8s/otelcollector.yaml

To get the application’s telemetry talking to our Elastic stack, I replaced all instances of the exporter type from HTTP (otlphttp/elastic) to gRPC (otlp/elastic). I then replaced OTEL_EXPORTER_OTLP_ENDPOINT with my APM endpoint and I replaced OTEL_EXPORTER_OTLP_HEADERS with my APM OTEL Bearer and Token.

Then I deployed the Hipster Shop.

$ kubectl create -f ./deploy-with-collector-k8s/adservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/redis.yaml
$ kubectl create -f ./deploy-with-collector-k8s/cartservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/checkoutservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/currencyservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/emailservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/frontend.yaml
$ kubectl create -f ./deploy-with-collector-k8s/paymentservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/productcatalogservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/recommendationservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/shippingservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/loadgenerator.yaml

Once all of the shop’s pods were running, I deployed the OpenTelemetry collector.

$ kubectl create -f ./deploy-with-collector-k8s/otelcollector.yaml

Observe and visualize Autopilot’s metrics

Now that we have added the Elastic Agent to our Autopilot cluster and added a workload, let's take a look at some of the Kubernetes visualizations the integration provides out of the box.

The “[Metrics Kubernetes] Overview” is a great place to start. It provides a high-level view of the resources used by the cluster and allows me to drill into more specific dashboards that I find interesting:

For example, the “[Metrics Kubernetes] Pods” gives me a high-level view of the pods deployed in the cluster:

The “[Metrics Kubernetes] Volumes” gives me an in-depth view to how storage is allocated and used in the Autopilot cluster:

Creating an alert

From here, I can easily discover patterns in my cluster’s behavior and even create Alerts. Here is an example of an alert to notify me if the the main storage volume (called “volume”) exceeds 80% of its allocated space:

With a little work, I created this view from the standard dashboard:

Conclusion

Today I have shown how easy it is to monitor, observe, and generate alerts on a GKE Autopilot cluster. To get more information on what is possible, see the official Elastic documentation for Autopilot observability with Elastic Agent.

Next steps

If you don’t have Elastic yet, you can get started for free with an Elastic Trial today. Get more from Elastic and Google together with a Marketplace subscription. Elastic does more than just integrate with GKE — check out the almost 300 integrations that Elastic provides.

In order to keep pace with demand, most of these online apps and services (for example, mobile applications, web pages, SaaS) are moving to a distributed microservice-based architecture and Kubernetes. Once you’ve migrated your app to the cloud, how do you manage and monitor production, scale, and availability of the service? OpenTelemetry is quickly becoming the de facto standard for instrumentation and collecting application telemetry data for Kubernetes applications.

OpenTelemetry (OTel) is an open source project providing a collection of tools, APIs, and SDKs that can be used to generate, collect, and export telemetry data (metrics, logs, and traces) to understand software performance and behavior. OpenTelemetry recently became a CNCF incubating project and has a significant amount of growing community and vendor support.

While OTel provides a standard way to instrument applications with a standard telemetry format, it doesn’t provide any backend or analytics components. Hence using OTel libraries in applications, infrastructure, and user experience monitoring provides flexibility in choosing the appropriate observability tool of choice. There is no longer any vendor lock-in for application performance monitoring (APM).

Elastic Observability natively supports OpenTelemetry and its OpenTelemetry protocol (OTLP) to ingest traces, metrics, and logs. All of Elastic Observability’s APM capabilities are available with OTel data. Hence the following capabilities (and more) are available for OTel data:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services
• Transactions (traces)
• ML correlations (specifically for latency)
• Service logs

In addition to Elastic’s APM and unified view of the telemetry data, you will now be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Given its open source heritage, Elastic also supports other CNCF based projects, such as Prometheus, Fluentd, Fluent Bit, Istio, Kubernetes (K8S), and many more.

This blog will show:

• How to get a popular OTel instrumented demo app (Hipster Shop) configured to ingest into Elastic Cloud through a few easy steps
• Highlight some of the Elastic APM capabilities and features around OTel data and what you can do with this data once it’s in Elastic

In follow-up blogs, we will detail how to use Elastic’s machine learning with OTel telemetry data, how to instrument OTel application metrics for specific languages, how we can support Prometheus ingest through the OTel collector, and more. Stay tuned!

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up the configuration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• We used the OpenTelemetry Demo. Directions for using Elastic with OpenTelemetry Demo are here.
• Make sure you have kubectl and helm also installed locally.
• Additionally, we are using an OTel manually instrumented version of the application. No OTel automatic instrumentation was used in this blog configuration.
• Location of our clusters. While we used Google Kubernetes Engine (GKE), you can use any Kubernetes platform of your choice.
• While Elastic can ingest telemetry directly from OTel instrumented services, we will focus on the more traditional deployment, which uses the OpenTelemetry Collector.
• Prometheus and FluentD/Fluent Bit — traditionally used to pull all Kubernetes data — is not being used here versus Kubernetes Agents. Follow-up blogs will showcase this.

Here is the configuration we will get set up in this blog:

Setting it all up

Over the next few steps, I’ll walk through an Opentelemetry visualization:

• Getting an account on Elastic Cloud
• Bringing up a GKE cluster
• Bringing up the application
• Configuring Kubernetes OTel Collector configmap to point to Elastic Cloud
• Using Elastic Observability APM with OTel data for improved visibility

Step 0: Create an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 1: Bring up a K8S cluster

We used Google Kubernetes Engine (GKE), but you can use any Kubernetes platform of your choice.

There are no special requirements for Elastic to collect OpenTelemetry data from a Kubernetes cluster. Any normal Kubernetes cluster on GKE, EKS, AKS, or Kubernetes compliant cluster (self-deployed and managed) works.

Step 2: Load the OpenTelemetry demo application on the cluster

Get your application on a Kubernetes cluster in your cloud service of choice or local Kubernetes platform. The application I am using is available here.

First clone the directory locally:

git clone https://github.com/elastic/opentelemetry-demo.git

(Make sure you have kubectl and helm also installed locally.)

The instructions utilize a specific opentelemetry-collector configuration for Elastic. Essentially, the Elastic values.yaml file specified in the elastic/opentelemetry-demo configure the opentelemetry-collector to point to the Elastic APM Server using two main values:

OTEL_EXPORTER_OTLP_ENDPOINT is Elastic’s APM Server
OTEL_EXPORTER_OTLP_HEADERS Elastic Authorization

These two values can be found in the OpenTelemetry setup instructions under the APM integration instructions (Integrations->APM) in your Elastic cloud.

Once you obtain this, the first step is to create a secret key on the cluster with your Elastic APM server endpoint, and your APM Secret Token with the following instruction:

kubectl create secret generic elastic-secret \
  --from-literal=elastic_apm_endpoint='YOUR_APM_ENDPOINT_WITHOUT_HTTPS_PREFIX' \
  --from-literal=elastic_apm_secret_token='YOUR_APM_SECRET_TOKEN'

Don't forget to replace:

• YOUR_APM_ENDPOINT_WITHOUT_HTTPS_PREFIX: your Elastic APM endpoint ( without https:// prefix ) with OTEL_EXPORTER_OTLP_ENDPOINT
• YOUR_APM_SECRET_TOKEN: your Elastic APM secret token OTEL_EXPORTER_OTLP_HEADERS

Now execute the following commands:

# switch to the kubernetes/elastic-helm directory
cd kubernetes/elastic-helm

# add the open-telemetry Helm repostiroy
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

# deploy the demo through helm install
helm install -f values.yaml my-otel-demo open-telemetry/opentelemetry-demo

Once your application is up on Kubernetes, you will have the following pods (or some variant) running on the default namespace.

kubectl get pods -n default

Output should be similar to the following:

NAME                                                  READY   STATUS    RESTARTS      AGE
my-otel-demo-accountingservice-5c77754b4f-vwph6       1/1     Running   0             5d4h
my-otel-demo-adservice-6b8b7c7dc5-mb7j5               1/1     Running   0             5d4h
my-otel-demo-cartservice-76d94b7dcd-2g4lf             1/1     Running   0             5d4h
my-otel-demo-checkoutservice-988bbdb88-hmkrp          1/1     Running   0             5d4h
my-otel-demo-currencyservice-6cf4b5f9f6-vz9t2         1/1     Running   0             5d4h
my-otel-demo-emailservice-868c98fd4b-lpr7n            1/1     Running   6 (18h ago)   5d4h
my-otel-demo-featureflagservice-8446ff9c94-lzd4w      1/1     Running   0             5d4h
my-otel-demo-ffspostgres-867945d9cf-zzwd7             1/1     Running   0             5d4h
my-otel-demo-frauddetectionservice-5c97c589b9-z8fhz   1/1     Running   0             5d4h
my-otel-demo-frontend-d85ccf677-zg9fp                 1/1     Running   0             5d4h
my-otel-demo-frontendproxy-6c5c4fccf6-qmldp           1/1     Running   0             5d4h
my-otel-demo-kafka-68bcc66794-dsbr6                   1/1     Running   0             5d4h
my-otel-demo-loadgenerator-64c545b974-xfccq           1/1     Running   1 (36h ago)   5d4h
my-otel-demo-otelcol-fdfd9c7cf-6lr2w                  1/1     Running   0             5d4h
my-otel-demo-paymentservice-7955c68859-ff7zg          1/1     Running   0             5d4h
my-otel-demo-productcatalogservice-67c879657b-wn2wj   1/1     Running   0             5d4h
my-otel-demo-quoteservice-748d754ffc-qcwm4            1/1     Running   0             5d4h
my-otel-demo-recommendationservice-df78894c7-lwm5v    1/1     Running   0             5d4h
my-otel-demo-redis-7d48567546-h4p4t                   1/1     Running   0             5d4h
my-otel-demo-shippingservice-f6fc76ddd-2v7qv          1/1     Running   0             5d4h

Step 3: Open Kibana and use the APM Service Map to view your OTel instrumented Services

In the Elastic Observability UI under APM, select servicemap to see your services.

If you are seeing this, then the OpenTelemetry Collector is sending data into Elastic:

Congratulations, you've instrumented the OpenTelemetry demo application using and successfully ingested the telemetry data into the Elastic!

Step 4: What can Elastic show me?

Now that the OpenTelemetry data is ingested into Elastic, what can you do?

First, you can view the APM service map (as shown in the previous step) — this will give you a full view of all the services and the transaction flows between services.

Next, you can now check out individual services and the transactions being collected.

As you can see, the frontend details are listed. Everything from:

• Average service latency
• Throughput
• Main transactions
• Failed traction rate
• Errors
• Dependencies

Let’s get to the trace. In the Transactions tab, you can review all the types of transactions related to the frontend service:

Selecting the HTTP POST transaction, we can see the full trace with all the spans:

Not only can you review the trace but you can also analyze what is related to higher than normal latency for HTTP POST .

Elastic uses machine learning to help identify any potential latency issues across the services from the trace. It’s as simple as selecting the Latency Correlations tab and running the correlation.

This shows that the high latency transactions are occurring in checkout service with a medium correlation.

You can then drill down into logs directly from the trace view and review the logs associated with the trace to help identify and pinpoint potential issues.

Analyze your data with Elastic machine learning (ML)

Once OpenTelemetry metrics are in Elastic, start analyzing your data through Elastic’s ML capabilities.

A great review of these features can be found here: Correlating APM telemetry to determine root causes in transactions. And there are many more videos and blogs on Elastic’s Blog. We’ll follow up with additional blogs on leveraging Elastic’s machine learning capabilities for OpenTelemetry data.

Conclusion

I hope you’ve gotten an appreciation for how Elastic Observability can help you ingest and analyze OpenTelemetry data with Elastic’s APM capabilities.

A quick recap of lessons and more specifically learned:

• How to get a popular OTel instrumented demo app (Hipster Shop) configured to ingest into Elastic Cloud, through a few easy steps
• Highlight some of the Elastic APM capabilities and features around OTel data and what you can do with this once it’s in Elastic

Ready to get started? Sign up for Elastic Cloud and try out the features and capabilities I’ve outlined above to get the most value and visibility out of your OpenTelemetry data.

The day-to-day operations of understanding the status and health of your Kubernetes clusters and applications running on them, through the logs, metrics, and traces they generate, will likely be your biggest challenge. But as an operations engineer you will need all of that important data to help prevent, predict, and remediate issues. And you certainly don’t need that volume of metrics, logs and traces spread across multiple tools when you need to visualize and analyze Kubernetes telemetry data for troubleshooting and support.

Elastic Observability helps manage the sprawl of Kubernetes metrics and logs by providing extensive and centralized observability capabilities beyond just the logging that we are known for. Elastic Observability provides you with granular insights and context into the behavior of your Kubernetes clusters along with the applications running on them by unifying all of your metrics, log, and trace data through OpenTelemetry and APM agents.

Regardless of the cluster location (EKS, GKE, AKS, self-managed) or application, Kubernetes monitoring is made simple with Elastic Observability. All of the node, pod, container, application, and infrastructure (AWS, GCP, Azure) metrics, infrastructure and application logs, along with application traces are available in Elastic Observability.

In this blog we will show:

• How Elastic Cloud can aggregate and ingest metrics and log data through the Elastic Agent (easily deployed on your cluster as a DaemonSet) to retrieve logs and metrics from the host (system metrics, container stats) along with logs from all services running on top of Kubernetes.
• How Elastic Observability can bring a unified telemetry experience (logs, metrics,traces) across all your Kubernetes cluster components (pods, nodes, services, namespaces, and more).

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• While we used GKE, you can use any location for your Kubernetes cluster.
• We used a variant of the ever so popular HipsterShop demo application. It was originally written by Google to showcase Kubernetes across a multitude of variants available such as the OpenTelemetry Demo App. To use the app, please go here and follow the instructions to deploy. You don’t need to deploy otelcollector for Kubernetes metrics to flow — we will cover this below.
• Elastic supports native ingest from Prometheus and FluentD, but in this blog, we are showing a direct ingest from Kubernetes cluster via Elastic Agent. There will be a follow-up blog showing how Elastic can also pull in telemetry from Prometheus or FluentD/bit.

What can you observe and analyze with Elastic?

Before we walk through the steps on getting Elastic set up to ingest and visualize Kubernetes cluster metrics and logs, let’s take a sneak peek at Elastic’s helpful dashboards.

As we noted, we ran a variant of HipsterShop on GKE and deployed Elastic Agents with Kubernetes integration as a DaemonSet on the GKE cluster. Upon deployment of the agents, Elastic starts ingesting metrics from the Kubernetes cluster (specifically from kube-state-metrics) and additionally Elastic will pull all log information from the cluster.

Visualizing Kubernetes metrics on Elastic Observability

Here are a few Kubernetes dashboards that will be available out of the box (OOTB) on Elastic Observability.

In addition to the cluster overview dashboard and pod dashboard, Elastic has several useful OOTB dashboards:

• Kubernetes overview dashboard (see above)
• Kubernetes pod dashboard (see above)
• Kubernetes nodes dashboard
• Kubernetes deployments dashboard
• Kubernetes DaemonSets dashboard
• Kubernetes StatefulSets dashboards
• Kubernetes CronJob & Jobs dashboards
• Kubernetes services dashboards
• More being added regularly

Additionally, you can either customize these dashboards or build out your own.

Working with logs on Elastic Observability

As you can see from the screens above, not only can I get Kubernetes cluster metrics, but also all the Kubernetes logs simply by using the Elastic Agent in my Kubernetes cluster.

Prevent, predict, and remediate issues

In addition to helping manage metrics and logs, Elastic can help you detect and predict anomalies across your cluster telemetry. Simply turn on Machine Learning in Elastic against your data and watch it help you enhance your analysis work. As you can see below, Elastic is not only a unified observability location for your Kubernetes cluster logs and metrics, but it also provides extensive true machine learning capabilities to enhance your analysis and management.

In the top graph, you see anomaly detection across logs and it shows something potentially wrong in the September 21 to 23 time period. Dig into the details on the bottom chart by analyzing a single kubernetes.pod.cpu.usage.node metric showing cpu issues early in September and again, later on in the month. You can do more complicated analyses on your cluster telemetry with Machine Learning using multi-metric analysis (versus the single metric issue I am showing above) along with population analysis.

Elastic gives you better machine learning capabilities to enhance your analysis of Kubernetes cluster telemetry. In the next section, let’s walk through how easy it is to get your telemetry data into Elastic.

Setting it all up

Let’s walk through the details of how to get metrics, logs, and traces into Elastic from a HipsterShop application deployed on GKE.

First, pick your favorite version of Hipstershop — as we noted above, we used a variant of the OpenTelemetry-Demo because it already has OTel. We slimmed it down for this blog, however (fewer services with some varied languages).

Step 0: Get an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 1: Get a Kubernetes cluster and load your Kubernetes app into your cluster

Get your app on a Kubernetes cluster in your Cloud service of choice or local Kubernetes platform. Once your app is up on Kubernetes, you should have the following pods (or some variant) running on the default namespace.

NAME                                    READY   STATUS    RESTARTS   AGE
adservice-8694798b7b-jbfxt              1/1     Running   0          4d3h
cartservice-67b598697c-hfsxv            1/1     Running   0          4d3h
checkoutservice-994ddc4c4-p9p2s         1/1     Running   0          4d3h
currencyservice-574f65d7f8-zc4bn        1/1     Running   0          4d3h
emailservice-6db78645b5-ppmdk           1/1     Running   0          4d3h
frontend-5778bfc56d-jjfxg               1/1     Running   0          4d3h
jaeger-686c775fbd-7d45d                 1/1     Running   0          4d3h
loadgenerator-c8f76d8db-gvrp7           1/1     Running   0          4d3h
otelcollector-5b87f4f484-4wbwn          1/1     Running   0          4d3h
paymentservice-6888bb469c-nblqj         1/1     Running   0          4d3h
productcatalogservice-66478c4b4-ff5qm   1/1     Running   0          4d3h
recommendationservice-648978746-8bzxc   1/1     Running   0          4d3h
redis-cart-96d48485f-gpgxd              1/1     Running   0          4d3h
shippingservice-67fddb767f-cq97d        1/1     Running   0          4d3h

Step 2: Turn on kube-state-metrics

Next you will need to turn on kube-state-metrics.

First:

git clone https://github.com/kubernetes/kube-state-metrics.git

Next, in the kube-state-metrics directory under the examples directory, just apply the standard config.

kubectl apply -f ./standard

This will turn on kube-state-metrics, and you should see a pod similar to this running in kube-system namespace.

kube-state-metrics-5f9dc77c66-qjprz                    1/1     Running   0          4d4h

Step 3: Install the Elastic Agent with Kubernetes integration

Add Kubernetes Integration:

1. 
2. In Elastic, go to integrations and select the Kubernetes Integration, and select to Add Kubernetes.
3. Select a name for the Kubernetes integration.
4. Turn on kube-state-metrics in the configuration screen.
5. Give the configuration a name in the new-agent-policy-name text box.
6. Save the configuration. The integration with a policy is now created.

You can read up on the agent policies and how they are used on the Elastic Agent here.

1. Add Kubernetes integration.
2. Select the policy you just created in the second.
3. In the third step of Add Agent instructions, copy and paste or download the manifest.
4. Add manifest to the shell where you have kubectl running, save it as elastic-agent-managed-kubernetes.yaml, and run the following command.

kubectl apply -f elastic-agent-managed-kubernetes.yaml

You should see a number of agents come up as part of a DaemonSet in kube-system namespace.

NAME                                                   READY   STATUS    RESTARTS   AGE
elastic-agent-qr6hj                                    1/1     Running   0          4d7h
elastic-agent-sctmz                                    1/1     Running   0          4d7h
elastic-agent-x6zkw                                    1/1     Running   0          4d7h
elastic-agent-zc64h                                    1/1     Running   0          4d7h

In my cluster, I have four nodes and four elastic-agents started as part of the DaemonSet.

Step 4: Look at Elastic out of the box dashboards (OOTB) for Kubernetes metrics and start discovering Kubernetes logs

That is it. You should see metrics flowing into all the dashboards. To view logs for specific pods, simply go into Discover in Kibana and search for a specific pod name.

Additionally, you can browse all the pod logs directly in Elastic.

In the above example, I searched for frontendService and cartService logs.

Step 5: Bonus!

Because we were using an OTel based application, Elastic can even pull in the application traces. But that is a discussion for another blog.

Here is a quick peek at what Hipster Shop’s traces for a front end transaction look like in Elastic Observability.

Conclusion: Elastic Observability rocks for Kubernetes monitoring

I hope you’ve gotten an appreciation for how Elastic Observability can help you manage Kubernetes clusters along with the complexity of the metrics, log, and trace data it generates for even a simple deployment.

A quick recap of lessons and more specifically learned:

• How Elastic Cloud can aggregate and ingest telemetry data through the Elastic Agent, which is easily deployed on your cluster as a DaemonSet and retrieves metrics from the host, such as system metrics, container stats, and metrics from all services running on top of Kubernetes
• Show what Elastic brings from a unified telemetry experience (Kubernenetes logs, metrics, traces) across all your Kubernetes cluster components (pods, nodes, services, any namespace, and more).
• Interest in exploring Elastic’s ML capabilities which will reduce your MTTHH (mean time to happy hour)

Ready to get started? Register and try out the features and capabilities I’ve outlined above.