Jeffrey Rengifo - Elastic Observability Labs

AI root cause analysis in Elastic Agent Builder that cites its evidence

Jeffrey Rengifo — Wed, 09 Sep 2026 00:00:00 GMT

Root cause analysis is the work of separating the failure that started an incident from everything that broke because of it, or merely alongside it. A cascading failure makes that hard: a deploy that shipped the same minute and a second service failing at the same time both look like causes.

Three ES|QL tools and about two dozen lines of query text turn AI root cause analysis into something you can check. This article builds them in Elastic Agent Builder and wires them to an agent whose report names the tool behind every number, and a trace.id and document _id behind every root cause claim.

What you need to reproduce this root cause analysis

Elasticsearch and Kibana 9.5, self-managed or on Elastic Cloud.
A generative AI connector for Agent Builder. Ours is Anthropic Claude Sonnet 4.6.
Python 3.12 with opentelemetry-distro[otlp] and the Flask and requests instrumentations.
An Elasticsearch API key with write access for OTLP ingest and Kibana access for the Agent Builder APIs.

Use the companion notebook if you want to reproduce the use case in this article.

The demo environment: four Python services on OpenTelemetry

Four Python services run on one host. checkout-api handles the customer request and calls pricing-api, which calls fx-rates for a currency quote. A fourth service, search-api, serves product search and sits outside that call path.

Each service exports OTLP straight to the Elasticsearch native OTLP endpoint, with no collector in between. Logs land in logs-generic.otel-default and spans in traces-generic.otel-default. Two checkout-api processes run on different versions at once, which one of the tools below depends on.

Starting a service is one command, with the OTLP endpoint and the service identity in the environment:

export OTEL_EXPORTER_OTLP_ENDPOINT="${ES_URL}/_otlp"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey ${ES_API_KEY}"
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true

OTEL_SERVICE_NAME=checkout-api \
OTEL_RESOURCE_ATTRIBUTES="service.version=2026.07.26.2,deployment.environment=production" \
  opentelemetry-instrument python services/checkout_api.py

Why the loudest service is not the root cause

At 09:55:33Z, fx-rates starts refusing about a third of its requests because its cached FX snapshot is older than the max_age it enforces. pricing-api turns each refusal into a 500, and checkout-api returns a 500 to the customer.

Two unrelated things happen in the same window. Release 2026.07.26.2 of checkout-api rolls out at 09:55, the minute the errors start, and 120 milliseconds after fx-rates breaks, search-api starts timing out during a scheduled reindex of its catalog.

One failure propagates, and the two other signals share only the clock:

The on-call view is four unhealthy services and a fresh deploy. This query gives the failure rate per service over the window:

FROM traces-generic.otel-default
| WHERE kind == "Server"
| EVAL failed = CASE(attributes.http.status_code >= 500, 1, 0)
| STATS failures = SUM(failed), requests = COUNT(*)
    BY service = resource.attributes.service.name
| EVAL failure_rate_pct = ROUND(100.0 * failures / requests, 1)
| SORT failure_rate_pct DESC

Nothing in the result ranks the candidates. search-api has the highest failure rate at 36.4%, and it is the one service with no involvement in the failing checkouts.

Automated root cause analysis with the default Elastic AI Agent

Before writing any tools, we gave the incident to the default Elastic AI Agent to see how far the built-in observability skills get on their own:

checkout-api started returning HTTP 500s to customers today. Investigate the
window 2026-07-26T09:50:00.000Z to 2026-07-26T10:02:00.000Z and tell me the root
cause. Context you have from the deploy log: checkout-api release 2026.07.26.2
rolled out at 09:55 UTC, and the on-call channel also reported search-api
timeouts starting at 09:55 UTC.

It got the cascade right. It named fx-rates as the origin, quoted the stale-snapshot message, and separated the search-api timeouts as a different problem. Then it ranked its hypotheses:

Hypothesis 2 claims the release restarted fx-rates without refreshing its snapshot, or lowered its max_age. Neither appears in the telemetry: the release was on checkout-api, and fx-rates reports service.version 2026.07.19.1 across all 7,374 of its spans.

The agent had a timestamp: the deploy and the first error share a minute. Across three runs, it connected them with a different invented mechanism each time:

a restart that dropped the snapshot
a new call path that was not exercised before the release
a previously tolerant code path that stopped tolerating stale data

The numbers have the same problem. The impact summary in that same answer reports the failure rate as "~38%", dividing the 2,043 failures by the 5,331 successes instead of the 7,374 requests, with no query next to the number for a reader to check.

The rest of this article closes that gap with tools.

The 16 built-in observability tools in Agent Builder

Agent Builder ships with tools that cover most of the exploration work, so check the built-in tools reference before writing anything. The catalog is under Agent Builder > Manage components > Tools:

Sixteen of the built-in tools are observability tools, organized around investigation steps instead of index operations:

One platform tool matters here too. platform.streams.investigation_progress_report is what an agent calls to publish its hypothesis list while it works, with a status per hypothesis, a conclusion, and an explicit list of what the data could not settle:

{
  "summary": "string",
  "hypotheses": [
    {
      "candidate": "string",
      "confidence": 0.0,
      "status": "investigating | dismissed | confirmed",
      "reason": "string"
    }
  ],
  "conclusion": "string",
  "gaps_found": ["string"]
}

Three ES|QL tools for AI root cause analysis

Agent Builder's built-in observability tools are shaped for showing the landscape. Ruling a candidate out needs one number that answers one question, and two questions in this incident have no built-in tool. A third returns the document IDs that make a claim checkable.

How to tell one incident from two coincident failures (rca_failure_shapes)

Two services failing in the same minute belong to the same incident only if they appear in the same requests. This tool groups error logs by trace, collapses each trace to the set of services that erred inside it, then counts each shape:

FROM logs-generic.otel-default
| WHERE @timestamp >= TO_DATETIME(?start) AND @timestamp <= TO_DATETIME(?end)
  AND severity_text == "ERROR" AND trace_id IS NOT NULL
| STATS services = VALUES(resource.attributes.service.name) BY trace_id
| EVAL failure_shape = MV_CONCAT(MV_SORT(services), " + ")
| STATS traces = COUNT(*) BY failure_shape
| SORT traces DESC
| LIMIT 20

MV_CONCAT(MV_SORT(...)) matters here. Grouping directly by a multivalue field makes ES|QL expand it, one row per service, which loses the combination. Collapsing the set to a single string first keeps the shape intact.

Two clusters come back, and they share no traces. The three checkout services appear together in 2,043 traces, search-api errs alone in 1,324, and that zero overlap settles the coincidence in one row.

How to rule out a deploy as the root cause (rca_version_split)

A release that caused the failures makes the version carrying it fail at a materially higher rate than the version it replaced. Server spans carry both service.version and the response status, so one query decides it:

FROM traces-generic.otel-default
| WHERE @timestamp >= TO_DATETIME(?start) AND @timestamp <= TO_DATETIME(?end)
  AND resource.attributes.service.name == ?service AND kind == "Server"
| EVAL failed = CASE(attributes.http.status_code >= 500, 1, 0)
| STATS failures = SUM(failed), requests = COUNT(*)
    BY version = resource.attributes.service.version
| EVAL failure_rate_pct = ROUND(100.0 * failures / requests, 1)
| SORT version

27.2% against 28.2%, on roughly 3,700 requests per version, is a difference within noise. The version that was already running fails as often as the one that shipped, which rules the release out.

The tool depends on both versions serving at once. With an instantaneous and total rollout, no query separates a broken new version from something else breaking at the same moment.

How to return document IDs an agent can cite (rca_evidence_sample)

METADATA _id on an ES|QL source command returns the Elasticsearch document ID, so a claim can point at a specific record:

FROM logs-generic.otel-default METADATA _id, _index
| WHERE @timestamp >= TO_DATETIME(?start) AND @timestamp <= TO_DATETIME(?end)
  AND severity_text == "ERROR" AND resource.attributes.service.name == ?service
| KEEP @timestamp, _id, _index, trace_id,
       attributes.error.kind, attributes.upstream.service, body.text
| SORT @timestamp DESC
| LIMIT 5

Each tool is registered with one POST kbn:/api/agent_builder/tools call carrying the query and its typed parameters. Registered, rca_failure_shapes looks like this; the agent fills start and end at call time:

Write the tool description as the decision it supports

The model reads the tool description to decide when to call the tool, so the description does more work than the name.

We describe rca_failure_shapes as the way to test whether two services that broke together are one failure or two. That phrasing got it called whenever a question mentioned a second failing service.

The description also outweighs the agent instructions. An agent with these three tools and one line of instruction, "You are an SRE assistant, help the user find the root cause of production incidents", still ruled out the deploy and the coincidence every time.

Wiring the tools into an incident root cause analysis agent

The agent gets the three custom tools, four built-in observability tools, and platform.streams.investigation_progress_report. Its instructions are five numbered steps:

1. Scope. Establish the affected service, the failure window, and the size of the
   symptom before you name any cause. State the window as an explicit ISO 8601
   range and reuse that same range in every tool call.

2. Enumerate. Write down at least three candidate causes before you test any of
   them. Include the candidate a human on call would reach for first, such as a
   recent deploy or another service that started failing at the same minute.

3. Refute. For each candidate, state the observation that would prove it wrong,
   then run the query that produces that observation. A candidate is dismissed
   when the refuting observation is present, not when a different candidate
   looks better.

4. Cite. Every number you report must name the tool that returned it. Every claim
   about a root cause must carry at least one trace_id and at least one document
   _id. A claim with no citation is not a finding, it is a guess.

5. Report. Put anything the available data cannot settle in gaps_found rather
   than resolving it with reasoning.

Two rules follow the steps. Each one blocks a mistake we saw in the default agent's answers:

Do not rank services by error volume and call the loudest one the cause.
Do not treat time correlation as causation. Two services that start failing in the same minute belong to the same incident only if they appear in the same traces.

The finished agent, with Elastic capabilities switched off so everything it produces comes from these eight tools:

Running the investigation: 11 tool calls, 72 seconds

We gave the new agent the same question, word for word. It opened with a progress report listing its candidates, then ran the refuting queries in parallel batches:

Eleven tool calls and 72 seconds later, the report opens with a citation table:

Three layers, three verbatim messages, three document IDs, and one trace ID shared by all of them. The origin message states the mechanism: StaleQuoteError: fx snapshot age 1215s exceeds max_age 900s (provider=ecb-eod).

The dismissals come next, each with the observation that ruled it out:

The report closes with a gaps section:

The first gap covers the question the default agent answered with an invented mechanism. The snapshot was already 1,127 seconds old at the first error, so the feed stopped refreshing before the window opened, and no log in this system records when. The next place to look is the feed ingestion job.

Verifying the agent's citation by hand in ES|QL

The report gives a trace ID, so open it:

FROM logs-generic.otel-default
| WHERE trace_id == "a131d0cf7343195c0a7a14f6a98da6b7"
| KEEP @timestamp, resource.attributes.service.name,
       attributes.upstream.service, body.text
| SORT @timestamp ASC

Three records, one millisecond apart, in the order the report claimed. fx-rates errs first and names no upstream, because it is the origin. pricing-api names fx-rates, and checkout-api names pricing-api.

Add upstream.service to your error logs

upstream.service is a logging convention that OpenTelemetry does not provide: each service writes it on the error it emits when a dependency fails. With that one key, error records order themselves into a call chain, and the ordering survives the clock skew and sub-millisecond hops that break timestamp sorting.

Two other fields carry the rest of the weight: trace.id arrives free from auto-instrumentation as long as the log is emitted inside an active span, and a structured error.kind lets you count failure modes without matching on message text.

Extracting knowledge indicators from your streams

Knowledge indicators are facts Elastic extracts from your raw data with LLM models. It picks out things like the underlying infrastructure and the dependencies between services.

You find this option under Streams > {your_stream} > Significant events:

Click Generate and the indicators are created for you:

What three ES|QL tools changed about the agent's answer

Three ES|QL tools and about two dozen lines of query text turned an agent's answer into a report that names its candidates, shows the observation that ruled each one out, and carries a record ID behind every number.

Next steps

Run the companion notebook to reproduce the incident, the tools, and the agent in your own cluster.
Write the query that rules out your team's most common wrong answer, the recent deploy or the loudest service, and register it as a tool whose description states the decision it supports.
Audit one service's error logs for trace.id, a structured error.kind, and a declared upstream.service. The last two are a one-line change in your logging.
Register the failure modes your team already knows about as significant events, so the next investigation starts with a history instead of a blank window.
For a different shape of Agent Builder investigation, read From five dashboards to one prompt, which scores APM service health with five ES|QL tools.

Your AI agent needs an alibi: Observability and audit trails for Agent Builder in Elastic

Jeffrey Rengifo — Mon, 31 Aug 2026 15:13:52 GMT

One question to an Elastic Agent Builder agent produced 24 spans, 10 model calls across two models, and roughly 160,000 input tokens. Elastic 9.5 records AI agent observability data without a collector or a scraper. Every run lands as OpenTelemetry traces in your own cluster, on by default, writing to traces-agent_builder.otel- down to each ES|QL query the agent generated and each index it looked up.

Those traces show how the agent reached its recommendation and what it cost. They do not record who approved it. Below: how to read the traces, scope the three identities a run touches, and append the approval decision to a data stream the pipeline cannot rewrite.

Prerequisites

Elastic Stack 9.5 or Serverless
Privileges to manage Kibana advanced settings, needed to install the traces dashboard.

Where Elastic Observability records each part of an AI agent action

Four questions come up in every review of an agentic operations pipeline, and each one is answered by a different record.

The first two are new in 9.5 and cost nothing but a toggle. The last one has no automatic source, so it is the one most pipelines are missing.

The scenario: a stale pricing cache in checkout

Three checkout-service workers serve production traffic. One of them, checkout-worker-1, was rolled to version 2026.07.26.1 and now returns HTTP 500 on every quote because its pricing cache stopped refreshing. The other two stay on 2026.07.25.3 and serve normally.

The SRE control plane pattern behind this setup connects telemetry, an Agent Builder agent that reasons over it, and Elastic Workflows that run known actions, with a human approval gate before the first step that changes production.

Telemetry arrives through the documented OpenTelemetry path, so the agent reads standard OTel fields. Elasticsearch 9.5 exposes a native OTLP endpoint, which lets an OTel SDK write directly to the cluster with no collector in between:

from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource

provider = LoggerProvider(
    resource=Resource.create({
        "service.name": "checkout-service",
        "deployment.environment": "production",
    })
)
provider.add_log_record_processor(
    BatchLogRecordProcessor(
        OTLPLogExporter(
            endpoint=f"{ES_URL}/_otlp/v1/logs",
            headers={"Authorization": f"ApiKey {API_KEY}"},
        )
    )
)

The endpoint speaks OTLP over protobuf and rejects application/json with HTTP 406, so send it through an SDK or collector rather than hand-built JSON. That writes 450 records to logs-generic.otel-default: 360 healthy events across the three workers and 90 PricingCacheStaleError events from the broken one. The agent is given none of that context and has to find it by querying.

Reading AI agent observability traces in Elastic

Everything Agent Builder records about a run lives in two data streams, and all of it is queryable with ES|QL.

How to turn on AI agent tracing in GenAI Settings

Open Stack Management, then GenAI Settings, and find the Agent Builder Traces section. Collect conversation traces is on by default in 9.5.

Two details on that panel matter more than the toggle. Traces are written to traces-agent_builder.otel-, one data stream per Kibana space, with a companion logs-agent_builder.otel- for agent-side events. These are ordinary data streams on the standard OTel index templates, not hidden system indices, so Discover, Lens, and ES|QL query them directly.

The callout states the access model plainly: anyone who can read the index can read every trace in it. Trace access is not scoped per user, so restrict the index pattern through a role before granting access to a space with sensitive conversations.

Reading the LLM trace waterfall for one agent run

A single question through the converse API asks the built-in Elastic AI Agent to investigate logs-generic.otel-default, find which pod and version are returning HTTP 500, and propose one bounded action. It answers correctly, naming checkout-worker-1 on 2026.07.26.1 with a 43% error rate against zero errors on the two workers still on 2026.07.25.3.

Select the trace icon under any agent response to open the waterfall.

One question through the converse API produced 24 spans over 43.7 seconds. The structure is an invoke_agent root, a generate_title side branch, then alternating chat and execute_tool spans as the agent queries, reads the result, and picks the next query.

Three span families carry everything you will aggregate on.

The token breakdown for that run:

| Model | Calls | Input tokens | Output tokens | | :---- | ----: | ----: | ----: | | anthropic-claude-4.6-sonnet | 5 | 113,315 | 2,039 | | anthropic-claude-4.5-haiku | 5 | 47,211 | 651 |

Half the model calls went to the smaller model. That is fast model routing, which sends low-effort steps to a cheaper model, and the split is only visible in the trace.

The single agent investigation consumed roughly 160,000 input tokens in total. Each round replays the accumulated context, so cost scales with conversation length rather than with the length of the question.

How do you query agent tool calls with ES|QL?

Tool invocations are the part of agent behavior most worth watching, because that is where the agent touches your data. Every call is one execute_tool span, and the documented query aggregates them directly:

FROM traces-agent_builder.otel-*
| WHERE span.name LIKE "execute_tool *"
| STATS calls = COUNT(*),
        errors = COUNT(*) WHERE status.code == "Error",
        avg_ms = ROUND(AVG(duration) / 1000000.0, 1)
  BY tool = attributes.gen_ai.tool.name
| SORT calls DESC

Across that run the agent leaned on platform.core.execute_esql, with platform.core.generate_esql and load_skill behind it at two calls each. duration is in nanoseconds on the root of the document, which is why the query divides by a million for milliseconds.

Check the companion logs data stream as well. During a different run in the same session, the agent tried to call a tool that was not in its available set, and the attempt was recorded as an exception event in logs-agent_builder.otel-default, correlated to the trace by trace_id:

{
  "trace_id": "3f8b9722dbd371ac4b7ad75e4bed13b6",
  "event_name": "exception",
  "attributes": {
    "exception.type": "toolNotFoundError",
    "exception.message": "Tool \"platform.streams.query_documents\" called but was not available"
  }
}

A blocked tool attempt is audit-relevant, and it does not appear in the trace waterfall. Querying only the traces data stream will miss it.

For aggregate views there is a managed dashboard, installed per space from the same settings panel.

Over a fifteen-minute window covering these runs, it reported 903,914 input tokens, 11,734 output tokens, and 44 LLM requests, with 35 tool spans at a 100% success rate and 0.42 seconds average duration. The dashboard is managed and read-only, so duplicate it to change a panel, and Elastic can still ship improvements to the original.

What OpenTelemetry LLM traces do not capture by default

By default, a trace records structure and cost, not content. Six toggles under Advanced privacy settings control prompts, responses, tool call details, system prompts, real tool and agent names, and real conversation and workflow IDs, and all six are off.

By default, a trace records structure and cost, not content. Select any span in the waterfall and the detail panel says so directly: "No input/output data available for this span."

Identifiers are hashed rather than dropped. The run with the blocked tool call returned conversation be30fb53-d351-4fa1-b5e1-a569816f85d9 from the API, but its spans carry gen_ai.conversation.id: b1141340d0851a46. That lets you group every span belonging to one conversation and compare conversations against each other, without exposing an identifier that ties back to a user's session.

The consequence is that you cannot join traces to conversations on the conversation ID unless you enable real IDs, and you rarely need to. The converse API hands you the correlation key directly:

{
  "conversation_id": "be30fb53-d351-4fa1-b5e1-a569816f85d9",
  "trace_id": "3f8b9722dbd371ac4b7ad75e4bed13b6",
  "model_usage": {
    "llm_calls": 22,
    "input_tokens": 518251,
    "output_tokens": 6423,
    "model": "anthropic-claude-4.6-sonnet"
  }
}

Store that trace_id in your own decision record, and the join works without weakening the privacy defaults. Enable real IDs only when exact response-to-decision attribution is required, and restrict the trace index in the same change.

Auditing AI agent actions beyond the trace

The traces stop at what the agent did, so the records that show who authorised it have to come from somewhere else.

Whose privileges does an AI agent action run with?

Three identities are involved in an agentic pipeline, and each has its own boundary.

One consequence of the stored key belongs in any review. Deactivating a user or changing their role does not refresh it, and the workflow keeps running with the privileges it captured until someone saves it again or toggles Enabled off and back on. Revoking an engineer's access does not, by itself, stop workflows that still run as them.

Scope the investigation role to reads only:

POST /_security/role/agent-builder-observability-investigator
{
  "cluster": ["monitor_inference"],
  "indices": [
    {
      "names": ["logs-*", "metrics-*", "traces-*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ]
}

Reading the agent's own traces is a separate grant, needing read and view_index_metadata on traces-agent_builder.otel-*. Keep the two roles apart, because the people who investigate incidents and the people who audit the agent are not always the same people.

Recording who approved an agent action in Elasticsearch

Run the workflow, and it stops at the approval gate, where the reviewer sees the agent's structured output rendered into the request rather than a bare confirmation prompt.

The execution record is detailed. It captures resumedAt, resumedBy, the full resumeInput payload, per-step token usage, and a deep link back to itself.

That execution history is an operational view, not an audit store. The underlying .workflows-events data stream is reserved for system operations and rejects user queries outright, so you cannot run ES|QL across a quarter of decisions, and execution history is subject to retention rather than to your compliance policy.

Write the decision to an index you control:

- name: review
  type: waitForInput
  with:
    message: |
      ## Approve the proposed checkout remediation?

      The evidence query matched {{ steps.collect_evidence.output.hits.total.value }} error events in the last hour.

      Agent classification: {{ steps.investigate.output.structured_output.incident_class }}
      Affected pod: {{ steps.investigate.output.structured_output.affected_pod }}
      Proposed action: {{ steps.investigate.output.structured_output.recommended_action }}
    schema:
      type: object
      properties:
        decision:
          type: string
          enum: ["approve", "decline"]
        reason:
          type: string
          enum: ["supported-by-evidence", "insufficient-evidence", "wrong-target", "unsafe-action"]
        notes:
          type: string
      required: ["decision", "reason"]

- name: record_decision
  type: elasticsearch.index
  with:
    index: "agent-action-audit"
    document:
      "@timestamp": "{{ now | date: '%Y-%m-%dT%H:%M:%S.%LZ' }}"
      "event.action": "agent_recommendation_reviewed"
      "incident.id": "{{ consts.incident_id }}"
      "agent.conversation_id": "{{ steps.investigate.output.conversation_id }}"
      "agent.incident_class": "{{ steps.investigate.output.structured_output.incident_class }}"
      "agent.affected_pod": "{{ steps.investigate.output.structured_output.affected_pod }}"
      "agent.recommended_action": "{{ steps.investigate.output.structured_output.recommended_action }}"
      "agent.evidence_count": "{{ steps.collect_evidence.output.hits.total.value }}"
      "review.decision": "{{ steps.review.output.response.decision }}"
      "review.reason": "{{ steps.review.output.response.reason }}"
      "review.notes": "{{ steps.review.output.response.notes }}"
      "review.responded_by": "{{ steps.review.output.respondedBy }}"
      "workflow.execution_id": "{{ execution.id }}"
      "workflow.executed_by": "{{ execution.executedBy }}"
      "workflow.execution_url": "{{ execution.url }}"

Two details in the workflow snippet above differ from the reference page.

The reviewer payload is nested one level deeper. The docs describe steps..output., but the running build returns the submitted values under response, alongside a respondedBy field:

{
  "response": { "decision": "approve", "reason": "supported-by-evidence" },
  "respondedBy": "1506416774"
}

execution.executedBy records who started the run, and respondedBy records who approved the action, which in a human-in-the-loop pipeline are usually different people.

The second detail is the timestamp. {{ now }} renders a JavaScript date string like Sun Jul 26 2026 07:37:11 GMT+0000 (Coordinated Universal Time), which Elasticsearch rejects with failed to parse date field, and execution.startedAt has the same problem. The Liquid date filter fixes it.

The workflow editor also flags steps.review.output.* as an invalid variable before the first run, because the reviewer payload shape is only known once someone responds. The warning clears after the step has real output, and the templates resolve correctly at runtime.

Making the audit data stream append-only

An audit trail the agent's own pipeline can rewrite is not an audit trail. Elasticsearch provides two independent controls, and they compose.

First, write to a data stream rather than an index, because data streams accept appends and nothing else:

PUT _index_template/agent-action-audit
{
  "index_patterns": ["agent-action-audit"],
  "data_stream": {},
  "priority": 500,
  "template": {
    "mappings": {
      "properties": {
        "@timestamp":            { "type": "date" },
        "event.action":          { "type": "keyword" },
        "incident.id":           { "type": "keyword" },
        "agent.conversation_id": { "type": "keyword" },
        "agent.evidence_count":  { "type": "long" },
        "agent.recommended_action": { "type": "keyword" },
        "review.decision":       { "type": "keyword" },
        "review.reason":         { "type": "keyword" },
        "review.responded_by":   { "type": "keyword" },
        "workflow.execution_id": { "type": "keyword" }
      }
    }
  }
}

Second, give the writer create_doc and nothing else, so it can add records but cannot reach for the by-query escape hatches:

PUT _security/role/agent-action-audit-writer
{
  "indices": [
    { "names": ["agent-action-audit"], "privileges": ["create_doc", "auto_configure"] }
  ]
}

Tested against the running cluster, that pair behaves the way an audit store should:

The write-only behaviour in the last row is deliberate. The workflow that writes decisions has no reason to read them, so auditors get a separate read role and the writer stays write-only.

The two controls fail differently, which matters. The 400 comes from the data stream itself and applies to everyone, including a superuser. The 403 rows come from the role, and a superuser could still run them, which is why tamper-resistant retention means shipping records off the cluster the agent's operators administer.

For cluster-level activity, enable Elasticsearch and Kibana security audit logging and forward the logs to a monitoring deployment. On 9.5 xpack.security.audit.enabled became a dynamic cluster setting, so Elasticsearch no longer needs a restart to turn it on, though on orchestrated deployments the logs still have to be shipped somewhere readable.

Query the decision trail with ES|QL

Two runs of the workflow, one approved and one declined, produce two rows you can query alongside everything else in Elastic.

FROM agent-action-audit
| KEEP @timestamp, agent.incident_class, agent.affected_pod, agent.recommended_action,
       agent.evidence_count, review.decision, review.reason, review.responded_by,
       workflow.execution_id
| SORT @timestamp DESC

Both runs saw the same 90 error events and proposed restart-checkout-worker on checkout-worker-1. The first review approved it as supported-by-evidence, and the second declined it as wrong-target, on the argument that restarting the pod hides a pricing-feed problem rather than fixing it.

Because both decisions are structured fields, disagreement between reviews is queryable. You can count rejections per incident class and group them by reason: insufficient-evidence sends you back to the investigation path, and unsafe-action sends you to the workflow and its permission boundary.

AI agent observability limits to design around

Four behaviors are worth designing around, and each is cheaper to handle before the workflows are written.

Trace access is index-level, not per user. A space with sensitive conversations needs a role boundary on traces-agent_builder.otel-* rather than a UI setting.
The managed dashboard is not installed automatically in a new space. Add it to your space provisioning checklist.
The workflow execution carries its own APM traceId. It is not the same trace as the Agent Builder spans its ai.agent step produced, so correlate through the conversation ID or the trace_id returned by the agent rather than expecting one trace to span both.
The waitForInput output shape differs from the reference page. The submitted values arrive under response, alongside respondedBy, as covered above.

None of these blocks the pattern.

Where to start with AI agent observability

Turn trace collection on, install the dashboard in the space your agents run in, and open the waterfall for one real conversation. It shows the tool sequence, the model split, and the latency distribution that the answer text does not.

Then pick the single incident class where you already trust the runbook, and add one elasticsearch.index step after its approval gate. An append-only decision record costs one workflow step and answers the three questions a review needs: who approved this, on what evidence, and what happened next.

For the details, see Collect Agent Builder traces, the traces overview dashboard, Agent Builder permissions, workflow authorization, and the waitForInput reference.

From a 582ms latency spike to the team that owns it, using Kibana Discover

Jeffrey Rengifo — Fri, 28 Aug 2026 15:00:00 GMT

A checkout service is running at 582ms p95 against a 350ms SLO target. One KQL query in Kibana Discover finds it. Working out which team owns that service takes an ES|QL query that joins the metric documents to a small service catalog index using LOOKUP JOIN. Below, that investigation runs in order. A data view narrows the scope and filter pills keep it visible, which matters more than it sounds when someone else has to reconstruct what you searched. KQL does most of the work. Lucene query syntax handles the one case that needs a regex, and ES|QL takes over once filtering stops answering the question.

Prerequisites

To follow along, you need:

An Elasticsearch cluster with Kibana. Everything up to the ES|QL section works on any recent version; LOOKUP JOIN is generally available in Elasticsearch 9.1 and was a technical preview in 9.0, so use 9.1 or later for the last section.
No special license tier. Everything in this article, including LOOKUP JOIN, works on the free basic license.
The two small sample indices created in the next section.

Why checkout latency increased in production

The example starts with a common operations question:

Why did checkout latency increase in production, and which team owns the service?

The metrics index contains 15-minute service measurements for four services across three regions. One service, checkout-api, has higher p95 latency in us-central1 during the investigation window. The goal is to get from all metrics to the small set of documents that explain the issue.

The walkthrough follows these steps:

Select the right data view and time range.
Use UI filters to include, exclude, disable, and pin criteria.
Use KQL for the main field and range search.
Switch to Lucene when regular expression syntax is useful.
Use ES|QL mode with LOOKUP JOIN to enrich metrics with service catalog data.

Set up the sample metrics index

The walkthrough searches a metrics index named o11y-labs-discover-service-metrics. Create it with keyword fields for the service dimensions and numeric fields for the measurements:

PUT o11y-labs-discover-service-metrics
{
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "service": {
        "properties": {
          "name": { "type": "keyword" },
          "environment": { "type": "keyword" },
          "version": { "type": "keyword" }
        }
      },
      "cloud": { "properties": { "region": { "type": "keyword" } } },
      "host": { "properties": { "name": { "type": "keyword" } } },
      "metrics": {
        "properties": {
          "latency": { "properties": { "p95_ms": { "type": "float" } } },
          "cpu": { "properties": { "pct": { "type": "float" } } },
          "error": { "properties": { "rate": { "type": "float" } } }
        }
      }
    }
  }
}

Each document is one 15-minute measurement for one service in one region:

POST o11y-labs-discover-service-metrics/_bulk
{ "index": {} }
{ "@timestamp": "2026-06-30T16:15:00.000Z", "service": { "name": "checkout-api", "environment": "production", "version": "2026.06.30-1" }, "cloud": { "region": "us-central1" }, "host": { "name": "checkout-api-us-central1-01" }, "metrics": { "latency": { "p95_ms": 582.6 }, "cpu": { "pct": 0.81 }, "error": { "rate": 0.041 } } }
{ "index": {} }
{ "@timestamp": "2026-06-30T16:15:00.000Z", "service": { "name": "payments-api", "environment": "production", "version": "2026.06.29-7" }, "cloud": { "region": "us-central1" }, "host": { "name": "payments-api-us-central1-01" }, "metrics": { "latency": { "p95_ms": 231.4 }, "cpu": { "pct": 0.31 }, "error": { "rate": 0.008 } } }

To reproduce the screenshots, index one document per service, region, and 15-minute interval:

Services: checkout-api, checkout-worker, payments-api, inventory-api
Regions: us-central1, us-east4, europe-west1
Window: 14:00 to 19:45 UTC on June 30, 2026, giving 24 intervals of 15 minutes
Documents per interval: 12 production, plus two staging (checkout-api and payments-api, both in us-central1)
Total: 24 intervals × 14 documents = 336 documents

The exact values do not matter, as long as checkout-api in us-central1 reports metrics.latency.p95_ms above 500 between 15:45 and 18:45 UTC and stays well under 500 ms everywhere else.

Instead of indexing everything by hand, you can run the supporting notebook, which generates the full 336-document dataset, creates both indices, and verifies the final ES|QL query.

The ES|QL section also uses a second, four-document lookup index for service catalog data. We will create it when we get there.

Choose a data view in Kibana Discover

The data view is the first filter in Discover. It decides which Elasticsearch indices are searched, which time field drives the histogram, and which fields are available in the left field list.

For this walkthrough, the Discover data view points to:

o11y-labs-discover-service-metrics

The time field is @timestamp. That matters because the time picker limits the documents before you add a query, a filter pill, or a selected field.

Use a narrow data view when you can. For example, a data view that targets only service metrics makes Discover easier to scan than a broad logs-*,metrics-* data view when you already know the question is about metrics.

Once the data view is selected, add the fields that support the investigation:

service.name
service.environment
cloud.region
metrics.latency.p95_ms
metrics.cpu.pct
metrics.error.rate

Filter pills in Discover: include, exclude, disable, and pin

UI filters are useful when you want a visible, editable list of constraints. They are also helpful when you are exploring fields from the document table and want Discover to write the field syntax for you.

In the document table, use the field actions (the +/- icons that appear when you hover over a value) to include or exclude it. For example:

service.environment: production
NOT cloud.region: us-east4
service.version: 2026.06.29-7  (disabled)

These three filters show the main filter controls:

Include a value when you want only matching documents.
Exclude a value when a dimension is not part of the problem.
Temporarily disable a filter when you want to keep it nearby but remove it from the current query.
Pin a filter when it should stay active as you move between Kibana apps.

Pinned filters are useful for investigations that cross app boundaries. For example, you can pin service.environment: production before moving from Discover to dashboards, Lens, or another view. Disabled filters are useful for testing a theory without deleting the context that got you there.

The key habit is to keep the filters readable. If a query has a long search expression and many hidden assumptions, another engineer has to reconstruct your thinking. Filter pills make the major scope decisions visible.

KQL query syntax for field, range, and boolean searches

KQL, the Kibana Query Language, is a good default for Discover searches. It supports field names, exact values, ranges, wildcards, and boolean logic in a readable form.

For the checkout latency example, this KQL query narrows the view to one service, one region, and high p95 latency:

service.name : "checkout-api" and cloud.region : "us-central1" and metrics.latency.p95_ms >= 500

Read it from left to right:

service.name : "checkout-api" keeps one service.
cloud.region : "us-central1" keeps one cloud region.
metrics.latency.p95_ms >= 500 keeps latency samples at or above 500 ms.

You can add the environment in KQL:

service.environment : "production" and service.name : "checkout-api" and cloud.region : "us-central1" and metrics.latency.p95_ms >= 500

Or you can keep service.environment: production as a UI filter. Both approaches are valid. For shared investigations, we prefer stable scope, such as environment and service, as filter pills, and the active hypothesis, such as a latency threshold, in the search bar.

KQL also works well for combining fields:

service.environment : "production" and
(service.name : "checkout-api" or service.name : "payments-api") and
metrics.error.rate > 0.02

This is useful when a user-facing flow crosses multiple services. You can compare a small group of services without switching data views or creating a dashboard first.

Lucene query syntax in Kibana: searching with regular expressions

Lucene query syntax is the option in Kibana that supports regular expressions. KQL does not, so when you need a regex in the search bar, open the query menu at the right of the search bar and switch the language to Lucene.

For example, this Lucene query searches production services whose names start with checkout- and whose p95 latency is above 500 ms:

service.name:/checkout-.*/ AND service.environment:production AND metrics.latency.p95_ms:>500

Lucene syntax is more compact, but it is also easier to misread. Use it when it gives you something you cannot express as clearly in KQL, such as a regex pattern over a field. For everyday field, value, and range filtering, KQL is usually easier for a teammate to review.

How to join two indices in Discover with ES|QL LOOKUP JOIN

Classic Discover mode is good when you want to search, filter, inspect fields, and look at raw documents. ES|QL in Discover is better when the question needs transformation before the result is useful. Use the Query in ES|QL button in the Discover toolbar to switch modes.

In this example, raw metrics tell us that checkout-api latency is high. They do not tell us who owns that service or what latency target the service is expected to meet. That data lives in a small service catalog lookup index.

Create a lookup index for service catalog data

PUT o11y-labs-service-catalog-lookup
{
  "settings": {
    "index.mode": "lookup"
  },
  "mappings": {
    "properties": {
      "service": {
        "properties": {
          "name": {
            "type": "keyword"
          }
        }
      },
      "owner": {
        "properties": {
          "team": {
            "type": "keyword"
          }
        }
      },
      "slo": {
        "properties": {
          "latency_target_ms": {
            "type": "long"
          }
        }
      },
      "runbook": {
        "properties": {
          "url": {
            "type": "keyword"
          }
        }
      }
    }
  }
}

One catalog document can attach ownership and an SLO target to the service:

POST o11y-labs-service-catalog-lookup/_doc/checkout-api
{
  "service": {
    "name": "checkout-api"
  },
  "owner": {
    "team": "checkout-platform"
  },
  "slo": {
    "latency_target_ms": 350
  },
  "runbook": {
    "url": "https://runbooks.example.com/checkout-api/latency"
  }
}

Run the LOOKUP JOIN query

Now Discover can run an ES|QL query that joins the metric documents with that catalog metadata using LOOKUP JOIN. Remember that this command needs Elasticsearch 9.1 or later, that the lookup index must be created with index.mode: lookup, and that the join field, service.name here, must be mapped as keyword in the lookup index.

FROM o11y-labs-discover-service-metrics
| WHERE @timestamp >= "2026-06-30T15:00:00.000Z" AND @timestamp <= "2026-06-30T18:45:00.000Z"
| WHERE service.environment == "production"
| LOOKUP JOIN o11y-labs-service-catalog-lookup ON service.name
| WHERE owner.team == "checkout-platform" AND metrics.latency.p95_ms > slo.latency_target_ms
| KEEP @timestamp, service.name, cloud.region, metrics.latency.p95_ms, slo.latency_target_ms, owner.team
| SORT @timestamp DESC

This is the part classic mode does not cover. Classic Discover can filter the metric documents, but ES|QL can enrich those rows with data from another index before displaying the result.

The result table answers a more operational question than the original search. It shows the affected service, the region, the latency value, the target, and the owning team in one view.

This pattern is useful for more than ownership. You can keep small lookup indices for service tier, deployment ring, escalation channel, business capability, or runbook URL. Then you can join that context into metric searches at investigation time.

How to choose the right Discover search method

The most useful workflow is not one search language for everything. It is a progression from broad scope to specific evidence.

For a real investigation, start with the smallest data view that still contains the data you need. Add filter pills for stable scope. Use KQL for the active search. Switch to Lucene only when regex syntax is worth the extra complexity. Move to ES|QL when the question needs enrichment, aggregation, or reshaping.

Field naming conventions that make metrics easier to search

Metric search works best when the field names carry enough context. The examples above use Elastic Common Schema-style fields where possible:

service.name for the monitored service.
service.environment for production, staging, or development.
cloud.region for the deployment region.
host.name for host-level drill-down.
Numeric metric fields under metrics.*.

You do not need this exact schema to use Discover, but predictable field names make the search bar and filter pills much easier to use. They also make saved searches and screenshots easier to understand during a handoff.

For service catalog data, keep the lookup index small and stable. Fields like service owner, tier, SLO target, and runbook URL change less often than raw metrics. That makes them good candidates for LOOKUP JOIN during analysis.

Run the walkthrough on your own cluster

Use Discover as a drill-down path, not only as a document table. In this walkthrough, we:

Scoped the search with a narrow data view and the time picker before writing any query.
Made the investigation scope visible and shareable with include, exclude, disabled, and pinned filter pills.
Used KQL for readable field, range, and boolean searches.
Switched to Lucene only for the regex case KQL cannot express.
Enriched metric documents with ownership and SLO data from a lookup index using ES|QL LOOKUP JOIN.

To try the full flow on your own cluster, run the supporting notebook, which creates both indices and the incident data used in every example.

From recommendation to remediation in 4 stages: human-in-the-loop automation with Elastic Workflows

Jeffrey Rengifo — Thu, 27 Aug 2026 00:00:00 GMT

An approve button with no evidence behind it is not a control. Human-in-the-loop automation binds five things into one inspectable record: the evidence you observed, the action you propose, the person who decided, the deadline they had, and what actually executed.

In this article, you'll build that gate with Elastic Workflows, reusing the control plane from the companion article Build an SRE Control Plane with Agent Builder and Workflows. This time the incident is narrow enough to act on: a stale pricing cache on a single worker, with a structured approval step sitting between the recommendation and the remediation. Nothing here touches production, so you can run the approved branch and the declined branch and compare what each one leaves in the execution record. That record is how incident response automation earns autonomy one incident class at a time.

Prerequisites

Elastic Stack 9.4+

What an SRE control plane needs before you add approval

This article builds on Build an SRE Control Plane with Agent Builder and Workflows, which defines the control-plane pattern used here: telemetry, investigation context, policy, and a set of known actions, wired together. Agent Builder reasons over logs, traces, metrics, alerts, runbooks, and previous cases. Elastic Workflows executes a defined sequence with explicit inputs and permissions. Human-in-the-loop, or HITL, connects those two responsibilities at exactly the point where evidence becomes action.

A recommendation and a remediation have very different failure modes. A weak recommendation wastes an engineer's time. A weak remediation changes production. This is why the approval gate belongs inside the execution model.

The five bindings that make an approval gate a reliability control

A useful gate preserves five properties. Remove any one of them and the gate gets weaker.

Evidence binding: The reviewer sees the logs, alert details, enrichment, or agent rationale that produced the proposal. An "Approve" button without evidence only asks a person to accept the automation's confidence.
Action binding: The request names the exact bounded action, target, parameters, and expected effect. A request without an explicit target can authorize far more than the reviewer intended.
Identity binding: The record shows who approved or declined the request, and when. Without it, you have an outcome but no accountability.
Time binding: The decision has a deadline, and stale requests fail closed instead of running later without context. An approval without a timeout can outlive the incident evidence that justified it.
Outcome binding: The execution record shows what ran, what was skipped, and whether post-action verification passed.

The form still matters, but the form is only the human-facing part of a much larger control. Approval design is reliability engineering.

The four stages from AI recommendation to auto remediation

Treat autonomy as a sequence of operational states rather than a product toggle.

The important transition is from stage 1 to stage 2. That is where the system stops being an advisor and gains an execution path. That path should be deterministic even when an AI agent contributed to the investigation: the agent summarizes evidence and recommends an action, while the workflow owns the pause, the structured decision, the branch, and the execution record.

Build the human-in-the-loop automation gate in Elastic Workflows

Elastic Workflows gives you the waitForInput step for this. When execution reaches that step, the workflow stops in the WAITING_FOR_INPUT state and waits for a person. The reviewer answers a small form in the Kibana execution view (or through the resume API), and whatever they submit becomes available to later steps at steps..output.

The workflow pulls the last 30 minutes of checkout-api failure logs, hands them to Agent Builder for a root-cause analysis, and writes that analysis into an Observability case. Then it stops and asks one question: should we clear the pricing cache on checkout-worker-07? If you approve, it records the simulated action, runs a verification query, and appends the result to the case. If you decline, it records the decision and runs nothing. Either way, production is untouched.

name: obs-labs-checkout-control-plane-hitl
description: Human approval gate for the OpenTelemetry-grounded checkout control plane.
enabled: false
tags:
  - sre-control-plane
  - agent-builder
  - human-in-the-loop
  - workflows
  - opentelemetry

settings:
  timeout: "30m"

triggers:
  - type: manual

consts:
  incident_id: "obs-labs-checkout-hitl-20260719"

steps:
  - name: collect_evidence
    type: elasticsearch.search
    with:
      index: "logs-*"
      size: 10
      query:
        bool:
          filter:
            - range:
                "@timestamp":
                  gte: "now-30m"
            - term:
                "service.name": "checkout-api"
            - term:
                "attributes.incident.id": "{{ consts.incident_id }}"
            - match_phrase:
                "body.text":
                  query: "checkout failed: stale pricing cache"

  - name: rca_analysis
    type: ai.agent
    agent-id: elastic-ai-agent
    create-conversation: true
    with:
      message: |
        Investigate the checkout-api incident identified by {{ consts.incident_id }}.
        The workflow evidence query found {{ steps.collect_evidence.output.hits.total.value }} matching OpenTelemetry log events in the last 30 minutes.
        Search logs, traces, and metrics for service.name checkout-api and incident.id {{ consts.incident_id }}.
        Identify the affected worker, deployment version, error type, HTTP status, and latency evidence.
        Return the likely cause, supporting evidence, confidence, and whether the bounded simulated action below is consistent with the evidence.
        Proposed simulated action: simulate clearing the pricing cache on checkout-worker-07.
        This workflow must not change production.

  - name: case_title
    type: ai.agent
    agent-id: elastic-ai-agent
    with:
      conversation_id: "{{ steps.rca_analysis.output.conversation_id }}"
      message: "Produce a short case title for this checkout incident. Output only the title."

  - name: case_description
    type: ai.agent
    agent-id: elastic-ai-agent
    with:
      conversation_id: "{{ steps.rca_analysis.output.conversation_id }}"
      message: "Produce a concise case description grounded in the OpenTelemetry evidence. Include the incident ID and say that remediation requires human approval. Output only the description."

  - name: create_case
    type: cases.createCase
    with:
      title: "{{ steps.case_title.output.message }}"
      description: "{{ steps.case_description.output.message }}"
      owner: "observability"
      severity: "medium"
      tags:
        - sre-control-plane
        - agent-builder
        - human-in-the-loop
        - opentelemetry

  - name: add_agent_analysis
    type: cases.addComment
    with:
      case_id: "{{ steps.create_case.output.case.id }}"
      comment: |
        ## Agent Builder RCA and proposed action

        Evidence query matches: {{ steps.collect_evidence.output.hits.total.value }}

        {{ steps.rca_analysis.output.message }}

        Proposed simulated action: simulate clearing the pricing cache on checkout-worker-07.
        No action has run yet.

        Agent conversation: {{ kibanaUrl }}/app/agent_builder/conversations/{{ steps.rca_analysis.output.conversation_id }}

  - name: review
    type: waitForInput
    with:
      message: |
        Approve the bounded checkout response?

        Incident: {{ consts.incident_id }}
        Evidence: {{ steps.collect_evidence.output.hits.total.value }} matching checkout failure logs in the last 30 minutes.
        Target: checkout-worker-07
        Action: simulate clearing the pricing cache on this worker only.
        Expected effect: subsequent checkout requests no longer use the stale cache.
        Blast radius: one worker. This simulated step does not change production.

        Review the Agent Builder analysis in case {{ steps.create_case.output.case.id }} before deciding.
      schema:
        type: object
        properties:
          approved:
            type: boolean
            title: "Approve the simulated cache clear"
          notes:
            type: string
            title: "Reviewer notes"
        required:
          - approved

  - name: approved_action
    type: console
    if: "steps.review.output.approved : true"
    with:
      message: "Approved. Simulated cache clear recorded for checkout-worker-07. Reviewer notes: {{ steps.review.output.notes }}"

  - name: verify_after_approval
    type: elasticsearch.search
    if: "steps.review.output.approved : true"
    with:
      index: "logs-*"
      size: 0
      query:
        bool:
          filter:
            - range:
                "@timestamp":
                  gte: "now-5m"
            - term:
                "service.name": "checkout-api"
            - term:
                "attributes.incident.id": "{{ consts.incident_id }}"
            - match_phrase:
                "body.text":
                  query: "checkout failed: stale pricing cache"

  - name: record_approved
    type: cases.addComment
    if: "steps.review.output.approved : true"
    with:
      case_id: "{{ steps.create_case.output.case.id }}"
      comment: |
        ## Human decision: approved

        Reviewer notes: {{ steps.review.output.notes }}
        Simulated action target: checkout-worker-07
        Verification query matches in the last 5 minutes: {{ steps.verify_after_approval.output.hits.total.value }}

        This walkthrough did not change production.

  - name: record_declined
    type: cases.addComment
    if: "steps.review.output.approved : false"
    with:
      case_id: "{{ steps.create_case.output.case.id }}"
      comment: |
        ## Human decision: declined

        Reviewer notes: {{ steps.review.output.notes }}
        No action ran.

  - name: declined_console
    type: console
    if: "steps.review.output.approved : false"
    with:
      message: "Declined. No action executed. Reviewer notes: {{ steps.review.output.notes }}"

The collect_evidence and rca_analysis steps keep the same evidence-then-reasoning sequence as the first article, and the workflow writes that analysis into the case before it ever reaches the waitForInput boundary. By the time a human is asked to decide, the reasoning is already durable and linkable.

The approval form asks for one boolean and accepts optional notes.

Only the approved branch records the simulated cache clear, runs a bounded verification query, and appends the result to the case. The declined branch records the decision and runs nothing.

The approval request carries the resulting count into the decision, while the linked case retains the full Agent Builder analysis. Treat that count as evidence for the reviewer, not as a root-cause claim or a performance measurement.

Replace the simulated action with real auto remediation

The walkthrough keeps the approved branch as a console step so you can run both branches safely. In production you replace only that step with a narrowly scoped external action, and leave the case, timeout, verification, and decline path exactly as they are.

You have two options, depending on how the target system is reached.

Call an internal remediation API with an HTTP connector

Configure an HTTP connector in Kibana with the base URL, authentication, and any encrypted headers, then reference it by connector-id. Secrets stay in the connector, never in the workflow YAML.

  - name: clear_pricing_cache
    type: http
    connector-id: "checkout-remediation-api"
    if: "steps.review.output.approved : true"
    with:
      path: "/v1/cache/pricing/purge"
      method: "POST"
      body:
        worker: "checkout-worker-07"
        incident_id: "{{ consts.incident_id }}"

Hand off to Jira, Slack or PagerDuty

Kibana connectors are available as workflow steps, so the approved branch can open a jira ticket for a change-managed action, post to slack for the on-call channel, or page through PagerDuty, all using credentials your team already manages centrally.

  - name: notify_oncall
    type: slack
    connector-id: "sre-oncall-channel"
    if: "steps.review.output.approved : true"
    with:
      message: "Approved by {{ steps.review.output.notes }}: pricing cache cleared on checkout-worker-07 for {{ consts.incident_id }}."

Whichever you choose, keep the action bound.

How to design a human-in-the-loop automation gate

The workflow above shows the mechanics. Getting the gate right is a design problem: where the pause goes, what the request tells the reviewer, what happens when nobody answers, and what the execution record has to preserve. Each one maps back to one of the five bindings.

Where should the approval step go in an incident response workflow?

The best place for waitForInput is immediately before the first step that increases impact. Don't pause before gathering evidence, because the workflow can usually search, enrich, classify, and open a draft case without touching the affected service. And don't pause after the remediation, because that just asks a person to ratify something that already happened.

Before anyone enables the workflow, a reviewer can inspect the evidence query, the form schema, the timeout, the branch condition, and the action itself.

What a good approval request tells the reviewer

An on-call engineer should not have to reconstruct the investigation from five other screens. Lead with the decision and include only the evidence needed to make it. A strong approval request answers these questions, in this order:

What exactly am I deciding?
What telemetry supports the proposal?
What target and parameters will the action use?
What is the expected effect and blast radius?
What happens if I decline or do nothing?

One required decision plus optional notes is usually enough. If the reviewer has to type in service names, host identifiers, environment, and action parameters, the proposal wasn't specific enough before the pause.

Paused executions stay discoverable in history and can be resumed by any authorized reviewer, which brings up a queueing requirement. Once a team has more than a few paused executions, reviewers need an inbox or an equivalent filtered view showing pending decisions, age, owner, severity, and target. Otherwise, a safe pause quietly becomes an invisible backlog.

What happens if nobody approves in time

waitForInput has no default timeout; the execution waits indefinitely. The settings.timeout field caps the entire execution, including time spent waiting for input, and in this workflow the 30-minute value limits how long the proposal stays actionable after the evidence was collected.

Choose that value from the incident and the action; a traffic shift during an active outage may need a decision within minutes. A maintenance approval may stay valid for hours. Confirm the timeout behavior on the exact Elastic version you operate, and add an external escalation or cancellation path if what you observe doesn't meet your fail-closed requirement.

Whatever you do, don't convert silence into approval.

What the audit record must capture

The execution history should answer four questions without anyone digging through chat history:

What evidence did the workflow collect?
What exact input did the reviewer submit?
Which branch ran?
What did the action and verification steps return?

A paused execution shows the evidence step and the decision still pending. An approved execution adds the action branch and the verification output to that same view. A declined execution preserves the same evidence and the same decision while skipping every action step. Keeping the walkthrough action simulated lets you inspect both branches without changing production.

For longer-lived incident context, push it into the case. Add the evidence summary, reviewer notes, action result, and verification result as comments, and the case becomes the durable record that outlives the execution.

How to know when a workflow is ready to run without approval

Don't remove the approval gate because a handful of runs succeeded. Review enough executions to understand both the normal and the exceptional paths, and measure at least these outcomes:

Autonomous execution is reasonable only when the incident class, target selection, action, verification, permissions, timeout, and rollback path are all narrow and repeatable. Even then, keep the same workflow structure. Automation should bypass the human wait for the proven path, not bypass evidence collection, authorization, verification, or audit records. Route anything uncertain back to human review.

Conclusion

The approval gate is a small amount of YAML, but it changes what the automation is. waitForInput turns the decision into a structured input the branch logic depends on, so the action, the verification, and the case comment all exist because a named person answered a specific question at a specific time. Placing that pause right before the first step that increases impact, and capping it with settings.timeout, is what makes it a reliability control instead of a confirmation dialog.

Taking this to production is a smaller change than it looks: the simulated console step becomes an http, slack, or jira connector step, and everything else stays as it is. From there you earn autonomy one incident class at a time, letting acceptance rate, approval age, action success, verification success, and rollback rate tell you when a path is proven enough to run without the wait.

Where to start with human-in-the-loop automation

Start with one recurring operational signal and one reversible response. Build the evidence query first, then add a recommendation that names the target and the expected effect, then insert waitForInput immediately before the action and run the workflow in a lab or case-only mode. Review the execution history with SREs, developers, support, security, and the product owners of the affected service.

The approval gate is not the destination. It is the mechanism that lets a team move toward narrow autonomy without giving up evidence, accountability, or control.

From alert to root cause in 3 minutes: automated root cause analysis with Elastic Agent Builder

Jeffrey Rengifo — Tue, 25 Aug 2026 00:00:00 GMT

An alert fires on checkout latency. Three minutes and 22 seconds later there's an Elastic Observability case open with the root cause, the evidence behind it, and how confident the agent was. Nobody moved between Kibana, chat, tickets and a terminal to get there.

In this article, we'll build that loop end to end. We'll use Elastic Agent Builder to feed logs, traces, metrics, alerts, and runbook context into an agent that investigates a problem, and Elastic Workflows to execute the known next steps: opening a case, sending a notification, running an enrichment query, or triggering a remediation path.

We'll work through a checkout latency regression as our example, but the same pattern applies to any incident class where your team already knows the manual steps.

Prerequisites

Elastic Cloud or self-managed cluster running 9.4+

We'll use a checkout latency regression as the running example. If you want to follow along against your own telemetry, point the queries at your service instead. If you'd rather reproduce the exact incident, the supporting notebook simulates it and sets up the role, skill, tool, and workflow for you.

Why dashboards are not enough for incident response

Dashboards show the symptom but cannot choose the next query. A dashboard is still one of the best tools for shared situational awareness, and during an incident the hard work starts after the chart turns red and the engineer still needs to answer a sequence of operational questions.

What changed? You need access to related deploys, alerts, logs, traces, and metrics from the same time window.
What is affected? You need visibility into services, hosts, users, regions, SLOs, and dependency paths.
What is the likely cause? You need evidence from telemetry combined with runbooks or previous incident cases.
What is safe to do next? You need a bounded action that includes proper permissions, an audit trail, and a rollback path.

That last question is where a dashboard stops. It can show the symptom, but it cannot decide which query to run next, which runbook applies, or which workflow should run. Engineers provide that judgment today by moving between Kibana, chat, tickets, terminals, and internal docs.

An SRE control plane keeps the judgment with the engineer while moving more context and more action into the same operational surface.

How automated root cause analysis works: state, policy and action

Automated root cause analysis needs three things in one place: the telemetry, the permissions that bound it, and the actions it can trigger.

State: For SRE work, that state is telemetry in Elasticsearch: logs, traces, metrics, alerts, SLOs, and related operational records.
Policy: Policy defines who can query which data, which tools an agent can call, which workflows can run, and where a human decision is required.
Action: Action is a set of known tools and workflows that run with explicit inputs, permissions, and outputs.

Agent Builder is useful where the system needs reasoning over messy context, and Workflows are useful where the system needs deterministic execution.

The two can work in both directions; a workflow can call an agent with an ai.agent step when it needs analysis before the next step, and an agent can call a workflow through a workflow tool when a conversation needs a repeatable action.

What Elastic Agent Builder adds to AI incident response

Agent Builder skills are reusable capability packs. A skill can include instructions, tools, and context that guide an agent through a specific task.

Reusable skill packs matter for SRE work because incident response is rarely a single query. A good investigation has a shape, and root cause analysis is a good example. The useful unit is not "ask the model what happened." It's a repeatable investigation path that starts from an alert, scopes the time window, checks the right telemetry, records uncertainty, and hands a case or workflow a structured result. The agent needs to decide which signal to start from, query the right index, compare the right time windows, inspect related services, and explain the evidence without hiding what it doesn't know.

Elastic includes a built-in Elastic AI Agent for this pattern. Built-in skills are scoped by solution, so the one that carries an SRE incident loop is observability.investigation, alongside platform skills such as dashboard-management that any solution can use. The list shows the short name, so look for investigation in the UI.

The skill ships as Markdown instructions, the same format we use for our own skill in the next section.

There are also out-of-the-box tools such as platform.core.search, platform.core.get_document_by_id, platform.core.get_index_mapping, platform.core.list_indices, platform.core.get_workflow_execution_status, and platform.core.resume_workflow_execution.

Skills guide the work, tools perform bounded operations, and the agent chooses what to use based on the task.

The scenario: a checkout latency regression

Deployment 2026.07.09.1 ships a connection pool misconfiguration to checkout-api. Within minutes, p95 latency goes from 180ms to over 2s and HTTP 500s appear for the first time. Nobody knows yet that the pool is the cause.

The evidence is spread across three signals, and no single one answers the question:

Correlating those three is the work we want the agent to do. That gives us the contract for the rest of this article:

Everything after this point builds one piece of that contract: the skill shapes the investigation, the tool and role bound the access, and the workflow turns the output into a case.

Build a read-only investigation skill in Elastic Agent Builder

Let's start with a read-only skill that improves investigation quality without touching production:

# Checkout latency investigation

Use this skill when an engineer asks why checkout latency, errors, or failed transactions increased.

Work through the investigation in this order:

1. Identify the affected service, environment, and time range.
2. Query traces for the slowest transactions in that window.
3. Query logs for errors from the same service and dependency path.
4. Compare current error and latency rates with the previous healthy window.
5. Return the likely cause, supporting evidence, confidence level, and the next safe action.

Do not recommend a production change unless there is a workflow tool assigned for that action.

If the evidence is incomplete, say what data is missing.

This kind of skill is a runbook execution guide, and it keeps the agent consistent across incidents. It also helps less experienced engineers ask better follow-up questions, because the agent can show the next query and explain why it matters.

Without step 4, the agent describes what's happening now and stops there. Comparing against the previous healthy window is what makes it an analysis. And the last line lets the agent say the data is missing instead of guessing.

Add AI agent observability tools with narrow permissions

Each tool should expose the smallest operation the agent needs, with the smallest data access that still supports the task.

For a read-only investigation agent, the required privileges usually start with searching observability data and inspecting index structure. The Agent Builder permissions documentation calls out that tools run against Elasticsearch data as the current user, and that read-oriented tools need index privileges such as read and view_index_metadata.

Run this in Dev Tools to create an investigation-scoped role:

POST /_security/role/agent-builder-observability-investigator
{
  "cluster": ["monitor_inference"],
  "indices": [
    {
      "names": ["logs-*", "metrics-*", "traces-*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ],
  "applications": [
    {
      "application": "kibana-.kibana",
      "privileges": ["feature_agentBuilder.read", "feature_actions.read"],
      "resources": ["space:default"]
    }
  ]
}

This role gives the agent enough access to inspect telemetry while keeping production-changing actions out of scope. The monitor_inference cluster privilege is what lets the agent use the inference endpoints behind Agent Builder, and it grants no data access on its own.

When you add a custom tool, describe it in operational language, because the tool description is part of how the agent decides when to call it. Prefer descriptions like this:

Use this tool to search checkout service logs for errors in a bounded time range.

Required inputs:
- service_name
- environment
- start_time
- end_time

Return:
- matching log samples
- error counts by message
- affected host and pod names when present

A narrow tool description is much safer than a broad tool that says "search all logs for anything relevant." The agent gets a clear contract, and reviewers can reason about what the tool can and cannot do.

Use Elastic Workflows for incident response automation

Once the investigation path is useful, we can add Workflows for the actions that should be repeatable. With Workflows the control plane becomes operational because it can query more context, ask an agent to summarize evidence, open a case, notify a channel, or call a remediation endpoint. The key is that each step is explicit.

The Workflows editor gives you a validation loop before you save or run anything. Use it to catch syntax issues before the workflow writes to Cases or calls any action.

Go to Workflows > Create workflow and paste the following:

name: obs-labs-checkout-control-plane
description: Checkout regression investigation with Agent Builder and case creation.
tags: ["sre-control-plane", "agent-builder", "workflows"]

triggers:
  - type: manual

inputs:
  - name: service_name
    type: string
    default: "checkout-api"
  - name: alert_summary
    type: string
    default: "Checkout API p95 latency increased above 2s and HTTP 500s rose in the last 15 minutes after deployment 2026.07.09.1."

steps:
  - name: rca_analysis
    type: ai.agent
    agent-id: elastic-ai-agent
    create-conversation: true
    with:
      message: |
        Investigate this checkout incident as an SRE would.

        Service: {{ inputs.service_name }}
        Alert: {{ inputs.alert_summary }}

        Search the available logs, traces, and metrics for this service.
        Compare the window before and after the most recent deployment.

        Return a concise likely cause, supporting evidence, confidence, and next safe action.
        If the evidence is incomplete, say what data is missing.

  - name: case_title
    type: ai.agent
    agent-id: elastic-ai-agent
    with:
      conversation_id: "{{ steps.rca_analysis.output.conversation_id }}"
      message: "Produce a short case title for this incident. Output only the title."

  - name: case_description
    type: ai.agent
    agent-id: elastic-ai-agent
    with:
      conversation_id: "{{ steps.rca_analysis.output.conversation_id }}"
      message: "Produce a concise case description. Output only the description."

  - name: create_case
    type: cases.createCase
    with:
      title: "{{ steps.case_title.output.message }}"
      description: "{{ steps.case_description.output.message }}"
      owner: "observability"
      severity: "medium"
      tags: ["sre-control-plane", "agent-builder", "workflows"]

  - name: add_agent_analysis
    type: cases.addComment
    with:
      case_id: "{{ steps.create_case.output.case.id }}"
      comment: |
        ## Agent Builder RCA

        {{ steps.rca_analysis.output.message }}

        Agent conversation: {{ kibanaUrl }}/app/agent_builder/conversations/{{ steps.rca_analysis.output.conversation_id }}

Each step feeds the next one through its output. ai.agent steps emit a message with the model's text and a conversation_id, and cases.createCase emits the new case.id. Those three fields are the whole contract:

This workflow doesn't restart anything. It asks Agent Builder to investigate, reuses the same conversation to generate the case title and description, creates an Observability case, and writes the agent analysis back as a case comment.

Two details are worth calling out. The create-conversation: true flag on the first step is what makes the next two steps cheap: case_title and case_description pass the same conversation_id, so the agent already has the investigation in context and doesn't repeat the queries. And we use a manual trigger with a default alert_summary so you can run the sequence before attaching it to a live alert rule. In production, you'd switch the trigger to alert and attach the workflow to the rule that owns that incident class.

Run the workflow with the play button. Our run took 3 minutes and 22 seconds, with rca_analysis, case_title, case_description, create_case, and add_agent_analysis all marked as successful. Almost all of that is the investigation itself: rca_analysis alone took 3 minutes and 5 seconds, while the two case writes finished in about a second each.

The workflow then wrote an Observability case. The case list shows one open case with the generated checkout title, our tags, medium severity, and one comment.

The case detail is the audit artifact for the investigation. It records the evidence considered, the affected hosts and deployment version, the likely error type, the agent's confidence, and any missing signals.

A useful operational control plane surfaces the limits of its evidence instead of turning uncertainty into a confident claim. If your agent never reports missing data or a lower confidence, that's a signal to tighten the skill instructions, not a sign that every investigation went well.

The read-only automated root cause analysis pattern improves response quality without changing the affected service. Add remediation only when the action is well understood, narrowly scoped, and paired with verification and rollback: clearing one cache key, restarting one worker, shifting traffic away from one unhealthy instance, or running a pre-approved maintenance task.

Turn Elastic Workflows into agent tools

Workflow tools let an Agent Builder conversation trigger an Elastic Workflow and use its output. This is the bridge from "the agent recommended a next step" to "the agent can offer a known action."

A workflow tool should have a narrow description:

Use this tool only when checkout errors are caused by connection pool exhaustion on a single worker.

The workflow drains and recycles the connection pool for one worker, then verifies that the worker resumes successful requests.

Required input:
- host_name

Do not use this tool for database outages, deploy regressions affecting all hosts, or multi-host failures.

The description matters because it sets the agent's selection boundary. Note how the last line excludes the very scenario we just investigated: our incident hit both hosts and was caused by a deploy, so the agent should not offer this tool. That's the point. A workflow tool that matches every incident is a workflow tool with no boundary.

The workflow still owns execution. The agent doesn't need to know how to recycle the pool. It only needs to recognize when a known workflow may apply, collect the required input, and present the action to the engineer.

How do you stop an AI agent from changing production?

An SRE control plane should be built around blast-radius control, which means every action path needs a clear boundary. Use these checks before exposing a workflow as an agent tool:

For the review boundary itself, Workflows gives you wait steps, timeouts, and execution history, so a risky path can pause for an approval and still leave an audit trail.

An agent can help gather evidence and propose the next step, but production action should stay inside known workflow paths.

Validate against one incident class first

For a real rollout, validate the control plane against one recurring incident class. Track whether the agent finds the right evidence, whether the workflow output is complete enough for review, and whether engineers trust the recommended next step.

Use a simple validation plan:

Pick one alert type with a known runbook.
Build a read-only investigation skill for that alert.
Add one or two query tools with scoped index permissions.
Run the agent against historical incidents and compare its summary with the actual case notes.
Add a case-creation workflow and review the output with the owning SRE team.
Only then consider a workflow tool that performs a bounded remediation step.

The main failure mode is not that the model gives an imperfect summary. It's granting broad action before the investigation path is proven. Keep the first version boring, scoped, and reviewable.

Conclusion

What we covered:

An SRE control plane combines state (telemetry in Elasticsearch), policy (permissions and review boundaries), and action (known tools and workflows).
Agent Builder handles reasoning over messy context, while Workflows handles deterministic execution, and the two can call each other.
A read-only investigation skill turns a runbook into a repeatable investigation path that records uncertainty instead of hiding it.
Scoped roles with read and view_index_metadata on logs-*, metrics-*, and traces-* keep the agent useful without letting it change production.
Reusing a conversation_id across ai.agent steps lets later steps build on the investigation instead of repeating it.
A case-only workflow gives you the full audit artifact before you enable any remediation.
Tool descriptions are a security boundary, not documentation, because they decide when the agent offers an action.

Resources

One edit, every dashboard updated: managing Kibana observability at scale with Terraform

Jeffrey Rengifo — Mon, 03 Aug 2026 00:00:00 GMT

Elastic ships a Kibana Dashboards API and a native Terraform resource for managing dashboards as code. This capability was introduced as a technical preview in Elastic 9.4 and was made generally available in Elastic 9.5. You define a golden signals panel library once in HCL, and for_each generates a dashboard for every team from it. When you need to change an error threshold, a panel layout or a query, one pull request updates every team at once. If something drifts or breaks, you roll back with git.

Why managing observability dashboards by hand breaks down at scale

Large organizations often end up with hundreds of dashboards. Teams build similar panels and maintain them using the Kibana UI.

When a small change comes in (a panel rename, a field fix, a new error threshold), there is no easy way to apply it across all of them. You either open each dashboard and edit it in the UI one by one, or you export the NDJSON, run a string replace, and re-import it.

Dashboards are code now

Elastic ships a typed Kibana Dashboards API and a native elasticstack_kibana_dashboard Terraform resource. You define a dashboard in an HCL file and then manage versions and changes as if it was regular code.

Golden signals dashboard: one definition for every team

The platform team owns a standard dashboard built on the four golden signals: latency, traffic, errors, and saturation. Every team should get that standard, and some teams add a panel or two of their own.

We want one definition of the standard, each team's dashboard generated from it, and a single change that reaches every team.

Prerequisites

An Elastic Cloud deployment or self-managed cluster running Elastic 9.4 or newer, or an Elastic Cloud Serverless project
Terraform installed
An Elasticsearch API key

The full Terraform configuration, the seed script, and the captured terraform plan outputs used in this article are available in the companion repo.

Configure the Elastic Terraform provider

Create a provider.tf next to the rest of your Terraform files:

terraform {
  required_providers {
    elasticstack = {
      source  = "elastic/elasticstack"
      version = "~> 0.11"
    }
  }
}

variable "elasticsearch_endpoint" {
  type = string
}

variable "elasticsearch_api_key" {
  type      = string
  sensitive = true
}

variable "kibana_endpoint" {
  type = string
}

variable "kibana_api_key" {
  type      = string
  sensitive = true
}

provider "elasticstack" {
  elasticsearch {
    endpoints = [var.elasticsearch_endpoint]
    api_key   = var.elasticsearch_api_key
  }
  kibana {
    endpoints = [var.kibana_endpoint]
    api_key   = var.kibana_api_key
  }
}

Provide your credentials through a local terraform.tfvars file (and add it to .gitignore so the keys never reach the repo):

elasticsearch_endpoint = "https://...es.region.cloud.es.io"
elasticsearch_api_key  = "..."
kibana_endpoint        = "https://...kb.region.cloud.es.io"
kibana_api_key         = "..."

You can use the same API key for both elasticsearch_api_key and kibana_api_key as long as it has dashboard write privileges in the target space.

Then initialize the working directory:

terraform init

Define a single-team Kibana dashboard in HCL

Start with a baseline dashboard for a single team. Panels sit on a 48-column grid, and each one is a Lens visualization configured inline. Use config_json for KPI tiles (it exposes secondary metrics and value coloring) and xy_chart_config for time-series charts.

Add the baseline resource to a new dashboards.tf:

resource "elasticstack_kibana_dashboard" "golden_signals" {
  title            = "Golden Signals - payments"
  description      = "Latency, traffic, errors"
  query            = { language = "kql", text = "" }
  refresh_interval = { pause = false, value = 60000 }
  time_range       = { from = "now-15m", to = "now" }

  panels = [
    {
      type = "vis"
      grid = { x = 0, y = 0, w = 12, h = 5 }
      config_json = jsonencode({
        type        = "metric"
        data_source = {
          type  = "esql"
          query = "FROM logs-payments-* | STATS `5xx errors` = COUNT(CASE(status >= 500, 1, null))"
        }
        metrics = [{ type = "primary", column = "5xx errors" }]
      })
    },
    # More panels follow the same shape: other metric tiles, xy_chart_config line charts, and a breakdown datatable. See the companion repo for the full file.
  ]
}

Each panel sets a type and grid position, then picks one chart kind. KPI tiles serialize the whole Lens config into config_json; the ES|QL query lives under data_source and the metric column is referenced by name in metrics[*].column. The dashboard time picker already scopes ES|QL panels, so the query needs no explicit @timestamp range filter.

Preview Kibana dashboard changes with terraform plan

Run terraform plan to see what Terraform will create:

terraform plan

The plan output lists the new elasticstack_kibana_dashboard.golden_signals resource and every attribute it will set: the top-level dashboard fields and one entry per panel with its grid position, chart kind, and data source.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # elasticstack_kibana_dashboard.golden_signals will be created
  + resource "elasticstack_kibana_dashboard" "golden_signals" {
      + description      = "Latency, traffic, errors"
      + title            = "Golden Signals - payments"
      + query            = { language = "kql", text = "" }
      + refresh_interval = { pause = false, value = 60000 }
      + time_range       = { from = "now-15m", to = "now" }
      + panels           = [
          # Every panel described in full: KPI tiles (config_json),
          # line charts (xy_chart_config), and the breakdown datatable.
        ]
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Reviewing the plan is your last check before anything ships to Kibana.

Don't apply yet. The next section extends the file with per-team dashboards, and then a single terraform apply ships everything.

Generate per-team observability dashboards from a shared panel library

On top of the baseline, each team gets the standard set of panels, with the option to add a few of their own. Hardcoding one resource per team does not scale. Instead, define a panel library and a teams map as locals, then build the dashboards with for_each. Each library entry describes a chart kind, a title, and the data it needs; the resource emits the right Lens block (config_json for metric tiles, xy_chart_config for line charts) based on chart_type.

Replace the contents of dashboards.tf with:

locals {
  panel_library = {
    errors = {
      chart_type     = "metric"
      title          = "Error rate"
      esql_query_tpl = "FROM {idx} | STATS `5xx errors` = COUNT(CASE(status >= 500, 1, null))"
      esql_column    = "5xx errors"
    }
    saturation = {
      chart_type     = "metric"
      title          = "Saturation (CPU)"
      # Saturation reads from the metrics TSDB, so this query is not parameterized by {idx}.
      esql_query_tpl = "TS metrics-payments-* | STATS avg_cpu = AVG(cpu.pct)"
      esql_column    = "avg_cpu"
    }
    latency = {
      chart_type = "xy"
      title      = "Latency p95"
      x_json     = jsonencode({
        operation          = "date_histogram"
        field              = "@timestamp"
        suggested_interval = "auto"
      })
      y_json = jsonencode({
        operation  = "percentile"
        field      = "duration_ms"
        percentile = 95
      })
    }
    # ... more entries (traffic, cart_value) in the companion repo.
  }

  teams = {
    payments = {
      index  = "logs-payments-*"
      panels = ["errors", "saturation", "latency", "traffic"]
    }
    checkout = {
      index  = "logs-checkout-*"
      panels = ["errors", "cart_value", "latency", "traffic"]
    }
  }
}

resource "elasticstack_kibana_dashboard" "golden_signals" {
  for_each         = local.teams
  title            = "Golden Signals - ${each.key}"
  description      = "Latency, traffic, and errors for the ${each.key} service"
  query            = { language = "kql", text = "" }
  refresh_interval = { pause = false, value = 60000 }
  time_range       = { from = "now-15m", to = "now" }

  sections = [
    {
      title     = "KPIs"
      grid      = { y = 0 }
      collapsed = false
      panels = [
        for i, p in [for q in each.value.panels : q if local.panel_library[q].chart_type == "metric"] : {
          type        = "vis"
          grid        = { x = (i % 4) * 12, y = 0, w = 12, h = 5 }
          config_json = jsonencode({ ... }) # one metric tile per panel; see the companion repo for the full config
        }
      ]
    },
    {
      title     = "Trends"
      grid      = { y = 1 }
      collapsed = false
      panels = [
        for i, p in [for q in each.value.panels : q if local.panel_library[q].chart_type == "xy"] : {
          type       = "vis"
          grid       = { x = (i % 3) * 16, y = 0, w = 16, h = 10 }
          vis_config = { by_value = { xy_chart_config = { ... } } }
        }
      ]
    },
    # A third "Breakdown" section holds the request-by-status datatable. See the companion repo.
  ]
}

Adding a team is one entry in teams. Adding a panel to every team is one entry in panel_library and one reference per team. The full config (data source ES|QL queries, metrics, layers, axis defaults, and legend placement) lives in dashboards.tf.

The saturation panel queries the metrics data stream with the ES|QL TS command, which is designed for TSDB. For the query to work, data streams matching metrics-payments-* must use time_series mode, so the configuration also ships an index template (metrics_tsdb.tf) that enables that.

Apply dashboards as code to Kibana with terraform apply

Run terraform plan to confirm both team dashboards (payments and checkout) will be created then apply:

terraform apply

Open Kibana and you'll see one Golden Signals dashboard per team, each backed by its own index pattern.

Dashboards as code in the GitOps loop: review changes in pull requests

Dashboards are now an artifact in version control, like the rest of your infrastructure.

You edit the library or a team's selection, open a pull request, your reviewer reads the terraform plan diff and sees which dashboards change.

For example, say you tighten the "critical error" threshold from status >= 500 to status >= 503 in panel_library.errors.esql_query_tpl. Running terraform plan shows the change reaching both teams at once:

Note: Full output in terraform-plan-update.txt.

A single edit to panel_library.errors propagates to every team that references it. After the PR merges, it's time to run terraform apply.

After the apply finishes, refresh the dashboards in Kibana and the new threshold is in effect:

Detect dashboard drift and roll back with git

If someone edits a dashboard using the UI, the next terraform plan shows the difference, because the code and the live state no longer match.

To see this in action, open Golden Signals - payments in Kibana, rename the Latency p95 panel to Latency p95 (EDITED), and save the dashboard.

Then run terraform plan:

Terraform reads the panel title from the live dashboard, compares it against the code, and proposes reverting the UI rename. You decide whether to keep the change (update the code to match) or revert it by running terraform apply.

You can commit the new version, or rollback one or many versions using git.

Replaying the earlier example: if you reopen the PR that changed panel_library.errors to broaden the error threshold and add a clearer title, git diff dashboards.tf shows the entire intent in two lines:

Every team that references errors picks up the new threshold on the next terraform apply, and reverting that commit rolls the change back across all of them at once.

Wrap up

Managing Kibana observability dashboards by hand does not scale past a few teams. With the Kibana Dashboards API and Terraform, you define a standard once, compose each team's dashboard from a shared library, and review every change in a pull request. One edit reaches every team, and you can roll back by reverting a commit.

The proposed file structure is only one of many ways you can organize your dashboards depending on how much information they share.

Next steps

Skip writing alert rules: 6 ready-made ES|QL templates ship inside the NGINX OTel integration

Jeffrey Rengifo — Fri, 31 Jul 2026 00:00:00 GMT

The NGINX OpenTelemetry Assets integration ships six alerting rule templates. Each one is an ES|QL query with a threshold already tuned. Install the integration, create a rule from one of the templates, and adjust the threshold to match your traffic. You get working alerts in minutes instead of writing them from scratch. This walkthrough covers the full setup, threshold tuning, and how to use an idle data streams rule to catch a service that stops sending data.

Prerequisites for Elastic integration alerting rule templates

Elastic Stack 9.4.0 or later.

Alerting rule templates have been available since 9.2.1, under the integration Assets tab. This article covers three things that need 9.4.0: the dedicated Alerting tab, idle data streams rules, and the NGINX OpenTelemetry Assets package, which is in technical preview.

Step 1: Send NGINX logs and metrics to Elasticsearch with OpenTelemetry

First, get NGINX metrics and logs into Elasticsearch.

Enable the NGINX stub_status module and make the access and error logs readable by the collector. Then, configure an EDOT or upstream OpenTelemetry Collector with the nginx and filelog receivers to export metrics and logs to Elasticsearch.

The integration setup has the full receiver and pipeline configuration.

If you want to reproduce this example, you can use the companion repository.

Both signals matter for alerting, and each group of templates reads a different data stream:

The log-based templates (4xx and 5xx error rates, error log spike) query logs-nginx.access.otel-* and logs-nginx.error.otel-*, which come from the filelog receiver.
The metric-based templates (active connections, dropped connections) query metrics-nginxreceiver.otel-*, which comes from the nginx receiver.

This is easy to get wrong: the Fleet Nginx (OpenTelemetry) input package collects stub_status metrics only. Its companion for logs is the classic Nginx integration, which writes ECS-based nginx.access and nginx.error datasets, not the *.otel-* data streams the log-based templates query. If you rely on that pairing alone, the log-based rules have nothing to evaluate and silently never fire. Run the filelog receiver too, not just the nginx receiver.

Step 2: Install the NGINX OpenTelemetry Assets integration

NGINX OpenTelemetry Assets is a content-only package. It ships the dashboards, alerting rule templates, and SLO templates, but it does not collect data itself. The data comes from the collector you set up in Step 1.

You don't need to install it by hand. Once the NGINX OTel data from Step 1 starts arriving, Elastic detects it and installs the Assets package for you, which takes a minute or two. Confirm it under Management > Integrations > Installed integrations, where NGINX OpenTelemetry Assets should appear.

Step 3: Create Elasticsearch alert rules from a rule template

Open the integration and select the Alerting tab.

This package ships six templates: high 4xx and 5xx error rates, high active connections, an error log spike, dropped connections, and a generic High error rate by service template that points at a placeholder logs-myservicereceiver.otel-* index for you to repoint and rename. The five NGINX rules run ES|QL every minute and group results by host.name, so an alert points at the host with the problem. The generic template groups by service.name instead, since it is meant to be repointed at whichever service you choose.

Select a template, for example [Nginx OTel] High 5xx error rate. Kibana opens a prefilled Create rule form built on an Elasticsearch query rule. It runs the template's ES|QL on a schedule and alerts when the query returns rows.

The query looks like this:

FROM logs-nginx.access.otel-*
// Flag each access log entry as a server error (5xx) or not
| EVAL is_5xx = CASE(http.response.status_code >= 500, 1, 0)
// Aggregate total requests and 5xx count per NGINX host
| STATS total = COUNT(*), errors_5xx = SUM(is_5xx) BY host.name
// Minimum sample size to avoid noisy low-traffic hosts
| WHERE total > 50
// Calculate 5xx error rate as a percentage
| EVAL error_rate_pct = ROUND(TO_DOUBLE(errors_5xx) / TO_DOUBLE(total) * 100.0, 2)
// Alert threshold: adjust to tune sensitivity
| WHERE error_rate_pct > 5.0
| SORT error_rate_pct DESC
| LIMIT 10

It counts requests and 5xx responses per host, keeps hosts with enough traffic to matter, and returns those above five percent.

Three things to get right while the form is open:

Send data first. ES|QL validates column names against the indices that exist when the query runs. Open a template before any NGINX data has been ingested and the editor reports Unknown column "http.response.status_code" and the form shows errors. Once data is flowing (Step 1), the same query validates and the error clears, so collect data before you create the rule.
Set the time field to @timestamp.
Leave "Create an alert for each row" selected. Because the query groups by host.name, this makes every affected host raise its own alert.

Add a connector and an action so the alert reaches Slack, email, or PagerDuty, then save and enable the rule.

Step 4: Tune alerting rule template thresholds in ES|QL

The thresholds are starting points, so confirm them against your own traffic. The threshold lives in the ES|QL WHERE clause.

To make the rule stricter, change error_rate_pct > 5.0 to error_rate_pct > 2.0. To require more traffic before it fires, raise total > 50. Use Test query in the rule form to confirm the edited query parses and returns rows before you save.

Three more settings are worth a look:

Time window: the look-back period the query runs over. A shorter window reacts faster but is noisier on bursty traffic.
Rule schedule: how often the query runs, every minute by default.
Alert delay: the number of consecutive runs the condition must hold before an alert is created, which filters out single-run blips.

How do you detect idle data streams in Elasticsearch?

Threshold rules only fire while data keeps arriving. When an agent goes offline or an output breaks, the data stops, and a threshold rule has nothing to evaluate.

Many Elastic integrations include a dynamically generated idle data streams template for exactly this case. It is named [{Integration name}] Idle data streams and appears in the same Alerting tab, though it is generated automatically rather than bundled with the integration. It alerts when no data is written to any of the integration's data stream patterns within a set period.

The NGINX OpenTelemetry packages do not include an idle data streams template, which is why no such template appears in the Alerting tab from Step 3. The end of this section covers what to do instead.

The default period is 24 hours, which is usually too long. A production service can go quiet for most of a day before you hear about it.

When you create the rule, drop the period to match how fast you need to know. Fifteen minutes to one hour works for a critical service. For a batch job, set a period comfortably longer than its run interval, so the quiet gaps between runs do not trigger it.

So why is this example left out? The template is generated from the data stream patterns an integration defines, and it is not generated for input-only packages. A content-only package like NGINX OpenTelemetry Assets defines no data streams of its own either. To catch silence in a collector-based setup like this, recreate the rule by hand with an Elasticsearch query rule. Use the query DSL or KQL variant rather than ES|QL, because an ES|QL rule fires on returned rows and so cannot alert on the absence of data. Point it at the OTel data streams (logs-nginx.access.otel-*, or metrics-nginxreceiver.otel-*) and set the condition to fire when the number of matching documents is below 1 over a window of, say, the last 15 minutes. That reproduces what an idle data streams template does, scoped to the data streams your collector writes.

Get started with Elastic integration alerting rule templates

Alerting rule templates turn alert setup into a few steps: send data, install the integration, create a rule from a template, and adjust the threshold. Treat the bundled thresholds as defaults to confirm, not numbers to trust blindly. And where an idle data streams template is available, reduce its 24-hour default so you find out quickly when a service goes silent.

Resources

Companion repository, to generate the NGINX demo data used here
Alerting rule templates
NGINX OpenTelemetry Assets integration
End-to-end tracing for NGINX with OpenTelemetry
Elasticsearch query rule
Elastic Agent built-in alerts, for monitoring the agents themselves

Common ES|QL queries for Kubernetes monitoring

Jeffrey Rengifo — Wed, 08 Jul 2026 00:00:00 GMT

This post has nine ES|QL queries for diagnosing Kubernetes problems in Elasticsearch. They cover crash-looping pods, memory pressure before an OOM kill, saturated nodes, and error spikes by namespace. Every query runs against Kubernetes data collected with the Elastic Distributions of OpenTelemetry (EDOT) and pastes into Discover with little to no editing. Turn any of them into a dashboard panel or an alert once you've found the one you need. Jump straight to the query that matches what you're seeing or read through the whole set to get a feel for your cluster.

What you need before running these Kubernetes ES|QL queries

To follow the queries in this article, you need:

Elasticsearch 9.2 or later.
EDOT Collectors running in your cluster and shipping data to Elasticsearch, with the kubeletstats, k8s_cluster, and filelog receivers enabled.

Why ES|QL for Kubernetes monitoring

Elasticsearch ES|QL is a piped query language that lets you start from a data source and then add one operation per line: filter, compute, aggregate, sort. That structure fits investigation work well because you refine a query step by step as you narrow down a problem.

When to use the ES|QL TS command for Kubernetes metrics

The TS command understands time series. This matters because most Kubernetes metrics are counters or gauges sampled over time. Using TS for metrics avoids the common mistake of summing cumulative counters across pods and getting a meaningless number.

EDOT receivers these Kubernetes queries depend on

The queries below assume EDOT Collectors are running in your cluster and shipping data to Elasticsearch. A typical setup uses:

The kubeletstats receiver for pod, container, and node resource metrics.
The k8s_cluster receiver for object state such as pod phase and container restarts.
The filelog receiver with the k8sattributes processor for container logs.

Resource attributes follow the OpenTelemetry semantic conventions. Pod, namespace, and node identifiers appear as k8s.pod.name, k8s.namespace.name, and k8s.node.name. Metric names are defined by the receiver that emits them, not by the semconv spec itself: k8s.pod.phase comes from the k8s_cluster receiver, while utilization metrics like k8s.container.memory_limit_utilization come from kubeletstats. EDOT preserves all of these names natively in Elasticsearch. The exact fields you have depend on which receivers you enabled, so treat the queries as templates and adjust field names if a metric is missing.

Exploring the cluster

Start broad. Before investigating a specific symptom, it helps to see what the cluster looks like in your data.

This query counts the pods reporting metrics in each namespace over the past hour.

FROM metrics-*
| WHERE @timestamp >= NOW() - 1 hour
| STATS pod_count = COUNT_DISTINCT(k8s.pod.name) BY k8s.namespace.name
| SORT pod_count DESC

COUNT_DISTINCT collapses the many metric samples per pod into a single count per namespace. The result is a quick inventory: which namespaces are busy and whether anything you expected to be running is missing.

Finding pod restarts and crash loops

Restarts are usually the first signal that something is wrong. The k8s.container.restarts metric is a gauge that reports the current restart count for each container.

This query surfaces the containers that have restarted the most in the last 24 hours.

FROM metrics-*
| WHERE @timestamp >= NOW() - 24 hours AND k8s.container.restarts IS NOT NULL
| STATS restarts = MAX(k8s.container.restarts)
    BY k8s.namespace.name, k8s.pod.name, k8s.container.name
| WHERE restarts > 0
| SORT restarts DESC
| LIMIT 20

MAX takes the highest restart count seen in the window, which reflects the latest value of the gauge. A container with a high and climbing restart count is the textbook sign of a crash loop. Once you have the pod name, you can pivot straight to its logs with the queries further down.

Spotting pods that are not running

A restart count tells you a pod recovered. The pod phase tells you whether it is healthy right now. The k8s.pod.phase metric encodes the phase as a number:

| Value | Phase | | :---- | :---- | | 1 | Pending | | 2 | Running | | 3 | Succeeded | | 4 | Failed | | 5 | Unknown |

This query uses TS to read the latest phase per pod and keeps anything that is not Running.

TS metrics-*
| WHERE TRANGE(15m)
| STATS phase = MAX(LAST_OVER_TIME(k8s.pod.phase))
    BY k8s.namespace.name, k8s.pod.name
| WHERE phase != 2
| EVAL phase_name = CASE(
    phase == 1, "Pending",
    phase == 3, "Succeeded",
    phase == 4, "Failed",
    phase == 5, "Unknown",
    "Other")
| SORT phase_name

LAST_OVER_TIME picks the most recent sample for each pod's time series, so you compare the current state rather than an average. Pods stuck in Pending often point to scheduling problems, such as insufficient CPU or memory on the nodes. Pods in Failed or Unknown are worth an immediate look.

Catching memory pressure before the OOM kill

Out-of-memory kills are one of the most common Kubernetes failures, and they are easier to prevent than to debug after the fact. When you enable limit metadata on the kubeletstats receiver, EDOT reports k8s.container.memory_limit_utilization as a fraction between 0 and 1 of the container's memory limit.

This query finds containers that ran close to their limit in the last hour.

TS metrics-*
| WHERE TRANGE(1h)
| STATS peak_mem_pct = MAX(MAX_OVER_TIME(k8s.container.memory_limit_utilization))
    BY k8s.namespace.name, k8s.pod.name, k8s.container.name
| EVAL peak_mem_pct = ROUND(peak_mem_pct * 100, 1)
| WHERE peak_mem_pct > 85
| SORT peak_mem_pct DESC

MAX_OVER_TIME finds the peak within each container's series, and the outer MAX keeps that peak per container. A container that repeatedly touches 95% or higher is a strong candidate for the next OOM kill. Pair this with the restart query above: a container with both a rising restart count and high memory utilization was very likely OOM killed.

Tracking CPU usage and node pressure

CPU problems show up as throttling and slow response times rather than crashes. The k8s.pod.cpu.node.utilization metric reports each pod's CPU use as a fraction of total node capacity.

This query charts the busiest pods over the last hour in five-minute buckets.

TS metrics-*
  | WHERE TRANGE(1h)
  | STATS avg_cpu = AVG(AVG_OVER_TIME(k8s.pod.cpu.node.utilization))
      BY k8s.pod.name, TBUCKET(5m)
  | SORT avg_cpu DESC

AVG_OVER_TIME averages the samples inside each pod's series for the bucket, and the outer AVG combines series that share a pod name. TBUCKET(5m) produces one point every five minutes, which renders cleanly as a time series chart.

To check whether the nodes themselves are saturated, query node utilization directly.

TS metrics-*
  | WHERE TRANGE(1h)
  | STATS cpu = AVG(AVG_OVER_TIME(k8s.node.cpu.usage)),
          mem = AVG(LAST_OVER_TIME(k8s.node.memory.usage))
      BY k8s.node.name, TBUCKET(5m)
  | SORT cpu DESC

A node sitting near full CPU explains throttled pods across many namespaces at once, which is easy to misread as an application bug when you only look at a single pod.

Investigating container logs

Once metrics point you at a pod, logs explain what it was doing. EDOT stores the log message in body.text and the level in severity_text, alongside the same k8s.* fields as the metrics.

This query ranks namespaces and pods by error volume in the last hour.

FROM logs-*
| WHERE @timestamp >= NOW() - 1 hour
  AND severity_text IN ("ERROR", "FATAL")
| STATS errors = COUNT(*)
    BY k8s.namespace.name, k8s.pod.name
| SORT errors DESC
| LIMIT 20

Counting by pod tells you whether errors are concentrated in one workload or spread across the cluster. A single noisy pod and a cluster-wide spike call for very different responses.

To read what a specific pod is logging, filter by pod name and search the message text.

FROM logs-*
| WHERE @timestamp >= NOW() - 1 hour
  AND k8s.pod.name == "checkout-"
  AND body.text LIKE "*timeout*"
| KEEP @timestamp, severity_text, body.text
| SORT @timestamp DESC
| LIMIT 50

LIKE "*timeout*" does a simple wildcard match on the message. For full-text relevance instead of wildcards, swap it for MATCH(body.text, "timeout").

Chaining ES|QL queries to investigate a Kubernetes incident

Real Kubernetes investigations chain multiple ES|QL queries together instead of running one in isolation.

A useful loop looks like this:

Count errors by pod to find the noisy workload.
Check that pod's restart count and memory utilization to see if it is crashing or starved.
Read its recent logs to find the specific failure.

Because every query uses the same k8s.namespace.name and k8s.pod.name fields, you can carry a pod name straight from one query to the next. The same fields let you build a single dashboard where a metrics panel and a logs panel filter together as you click through namespaces.

Turning queries into alerts and dashboards

Any ES|QL query in this post that produces an aggregated value can back an alert, not just support ad hoc investigation.

For example, the restart query becomes an alert when you keep only containers above a threshold and trigger on a non-empty result.

FROM metrics-*
| WHERE @timestamp >= NOW() - 15 minutes AND k8s.container.restarts IS NOT NULL
| STATS restarts = MAX(k8s.container.restarts)
    BY k8s.namespace.name, k8s.pod.name, k8s.container.name
| WHERE restarts >= 5

Wire this into an Elasticsearch query rule and you get notified the moment a container crosses five restarts in fifteen minutes, instead of finding out when a user does. The same pattern applies to memory utilization, node saturation, and error counts.

Building a Kubernetes monitoring toolkit with ES|QL

ES|QL gives you one language for every Kubernetes signal, from object state to resource metrics to container logs. Start with the exploration query to understand your cluster's shape, then keep the restart, phase, memory, CPU, and log queries close for the next incident. Use TS when you need time series functions like MAX_OVER_TIME or TBUCKET to aggregate correctly within each pod or container series. For counting distinct values or taking a simple MAX on a gauge, FROM is enough.

From here, you can adapt the field names to your own receivers, save the most useful queries as dashboard panels, and promote the critical ones to alerts.

To go deeper, see the ES|QL reference, the TS command documentation, and the EDOT Kubernetes guide.

Contextual AI: Stop pinging the SRE: three MCP tools that turn Elastic Agent Builder into your team's runbook

Jeffrey Rengifo — Thu, 04 Jun 2026 00:00:00 GMT

A developer asks their editor, "Is it safe to merge this PR?" and gets a real answer in seconds, not a 10–15 minute dashboard hunt or a Slack ping to an SRE. This post shows how to build three MCP tools in Elastic Agent Builder that read endpoint health, recent deploys, and SLO burn rate, and encode the platform team's interpretation rules, error rate thresholds, deploy warm-up windows, and burn rate limits directly into the tool descriptions. The result is contextual AI: an agent that reasons over production signals using the runbook the platform team wrote once.

Prerequisites for Elastic Agent Builder MCP tools

An Elastic Cloud deployment with Elastic Stack 9.3+ (or Elastic Cloud Serverless) with Agent Builder enabled.
An APM-ingested service. If your cluster does not already have APM data, the companion notebook includes instructions to generate synthetic traffic using elastic/apm-integration-testing with the opbeans-node demo app.
An MCP-compatible client: Claude Code, Cursor, or VS Code with an MCP extension.
Basic familiarity with ES|QL syntax.
Node.js 18+ (for the mcp-remote bridge).

If you are new to MCP or need to set up the Elastic MCP server for the first time, check out Connect Agent Builder tools to any AI agent with Elastic MCP server for the full setup walkthrough. This article assumes the MCP server is already configured.

The problem: why developers fly blind

A developer is about to merge a pull request. The change looks simple: increasing the timeout for the downstream recommendations service call from 2 seconds to 5 seconds. But before hitting the merge button, a question lingers: is the service healthy enough to absorb this change right now?

To answer that question today, the developer has two options:

Check dashboards manually. Open the APM UI, look at error rates, scan latency charts, find the SLO page, and look for recent deploys. This takes 10-15 minutes and requires knowing what to look for and how to interpret it.
Ask an SRE. Ping the platform team on Slack: "Hey, is checkout healthy? I want to merge something." This creates an interruption, adds latency to the decision, and doesn't scale.

The core problem is not the data. Elastic already collects everything: traces, metrics, error logs, deploy markers, and SLO budgets. The problem is that correlating multiple signals requires mental overhead and domain knowledge that most developers don't have.

An SRE knows that a p99 spike after a deploy is normal for 5 minutes, that an error rate under 0.5% is acceptable during a release window, and that merging when the SLO budget is below 20% is risky. That knowledge lives in runbooks, tribal memory, and experience.

What if the platform engineer could encode that knowledge into tools that any developer can query from their editor?

How MCP tools in Elastic Agent Builder encode your runbook

The key insight is this: a tool is not just a query; it is a query plus interpretation. A dashboard shows you a p99 of 450ms. A well-designed tool tells you "p99 is 450ms, which is within normal range for this service, and has been stable since the last deploy 2 hours ago."

The difference is that the tool description carries the domain knowledge. When a platform engineer creates a tool in Agent Builder, they write descriptions like: "Error rate above 1% typically indicates a regression. If this coincides with a recent deploy, the deploy is the likely cause." That description becomes part of the context the AI agent uses when reasoning across multiple tool results.

This is what we mean by contextual AI: the AI agent does not just fetch data; it reasons over it using the interpretation rules that the platform team encoded.

Here is the architecture:

The platform engineer authors the tools once. Every developer on the team benefits from their own editor, without needing to learn ES|QL or understand APM data models.

Setting up the Elastic Agent Builder sample environment

The full end-to-end setup (traffic generation with opbeans-node, deploy annotations, SLO creation, and the three Agent Builder tools) is available as a runnable notebook at this repository: notebook.ipynb. The sections below focus on the ES|QL queries and tool descriptions: the why behind each tool, not the mechanics of posting them.

Building Tool 1: get_endpoint_health

This tool answers the question: "How is this endpoint performing right now?" It returns error rate, latency percentiles (p50, p95, p99), and throughput for a given service and endpoint within a time window.

Here is the full tool configuration as created in Agent Builder:

{
  "id": "get_endpoint_health",
  "type": "esql",
  "description": "Returns the current health of a service endpoint: error rate, latency percentiles (p50/p95/p99), and throughput. Use this tool to assess whether a service is healthy before making changes. Interpretation guide: error rate below 0.5% is healthy, 0.5-1% is elevated (check for recent deploys), above 1% indicates a problem. For latency, compare p99 against the service baseline: checkout is typically under 500ms, product-search under 200ms. A sudden p99 spike within 15 minutes of a deploy suggests the deploy caused a regression.",
  "tags": ["apm", "reliability", "health"],
  "configuration": {
    "query": "FROM traces-apm-* | WHERE service.name == ?serviceName AND @timestamp >= NOW() - ?timeWindow AND transaction.duration.us IS NOT NULL | STATS total_transactions = COUNT(*), error_count = SUM(CASE(event.outcome == \"failure\", 1, 0)), p50_latency_ms = PERCENTILE(transaction.duration.us, 50) / 1000, p95_latency_ms = PERCENTILE(transaction.duration.us, 95) / 1000, p99_latency_ms = PERCENTILE(transaction.duration.us, 99) / 1000 BY service.name | EVAL error_rate_pct = ROUND(error_count / total_transactions * 100, 2) | EVAL throughput_per_min = ROUND(total_transactions / ?windowMinutes, 1)",
    "params": {
      "serviceName": {
        "type": "keyword",
        "description": "The APM service name to check (e.g., opbeans-node)"
      },
      "timeWindow": {
        "type": "keyword",
        "description": "Time window to analyze, in ES|QL duration format (e.g., 30 minutes, 1 hour, 6 hours)"
      },
      "windowMinutes": {
        "type": "integer",
        "description": "Time window in minutes, used to calculate throughput per minute"
      }
    }
  }
}

The query uses the traces-apm-* data stream, which contains raw transaction data. We filter with transaction.duration.us IS NOT NULL to select only transaction events (excluding spans). Using traces-apm-* is more portable than the pre-aggregated metrics-apm.transaction.1m-* stream, which only populates after sustained traffic.

Notice the description field. It is not just "returns health metrics." It includes interpretation rules: what error rate thresholds mean, what latency baselines look like, and how to correlate spikes with deploys. This is the runbook encoded in the tool.

Building Tool 2: get_recent_deploys

This tool answers: "What has been deployed recently?" Deploy history is a critical context because most production issues correlate with code changes. The agent needs this to reason about whether current metrics are normal or reflect a recent deployment.

Deploy annotations are stored in the observability-annotations index. Here is the full tool configuration:

{
  "id": "get_recent_deploys",
  "type": "esql",
  "description": "Returns the deployment history for a service over the last 24 hours, including version numbers, timestamps, and deploy messages. Use this tool to understand the deployment timeline when assessing service health. Key patterns: if a deploy happened within the last 15 minutes, elevated error rates or latency may be expected (warm-up period). If metrics degraded immediately after a deploy, the deploy is the likely cause. Multiple deploys in a short window (under 2 hours) increase risk because it becomes harder to isolate which change caused an issue.",
  "tags": ["apm", "deploys", "change-tracking"],
  "configuration": {
    "query": "FROM observability-annotations | WHERE service.name == ?serviceName AND @timestamp >= NOW() - 24 hours | SORT @timestamp DESC | KEEP @timestamp, service.version, service.environment, message | LIMIT 10",
    "params": {
      "serviceName": {
        "type": "keyword",
        "description": "The APM service name to check deploy history for"
      }
    }
  }
}

Again, the description encodes domain knowledge: the 15-minute warm-up window, the correlation between deploys and metric changes, and the risk of multiple rapid deploys. This is how a platform engineer transfers their intuition into something an AI agent can reason with.

Building Tool 3: get_slo_status

This tool answers: "How much error budget do we have left?" SLO budget is the platform team's quantified way of expressing risk tolerance. If the budget is nearly spent, even a small change could cause a violation.

Unlike the previous tools that query APM data, this one queries the internal SLO indices where Elastic stores pre-computed SLI data. The query calculates the current burn rate, that is, how fast the service is consuming error budget relative to the allowed threshold:

{
  "id": "get_slo_status",
  "type": "esql",
  "description": "Returns the current SLO burn rate for a service over the last hour. The response includes: SLI value (current performance), error budget target, and burn rate percentage. The burn rate tells you how fast the service is consuming error budget relative to the allowed threshold. Interpretation: a burn rate below 100% means the service is consuming budget slower than the limit (sustainable). Between 100-200%, the service is burning budget faster than planned (proceed with caution). Above 200%, the service is burning budget at double the allowed rate (delay non-critical changes). Above 500%, investigate immediately. Note: this measures the current burn rate over the last hour, not cumulative budget consumption over the full SLO window. A temporarily high burn rate does not mean the overall budget is exhausted.",
  "tags": ["slo", "reliability", "budget"],
  "configuration": {
    "query": "FROM .slo-observability.sli-v* | WHERE slo.id == ?sloId AND @timestamp >= NOW() - 1 hour | STATS sli_value = AVG(slo.numerator) / AVG(slo.denominator) BY slo.id, slo.name | EVAL error_budget_target = 0.995 | EVAL burn_rate_pct = ROUND((1 - sli_value) / (1 - error_budget_target) * 100, 1)",
    "params": {
      "sloId": {
        "type": "keyword",
        "description": "The SLO identifier. Use the SLO ID for the service you are evaluating."
      }
    }
  }
}

Note on the SLI index: the version suffix in .slo-observability.sli-v* depends on your Stack release (e.g., v3.6 in Stack 9.3). Verify with GET _cat/indices/.slo-observability.*?v and adjust the pattern if your cluster uses a different version.

The burn rate interpretation rules in the description are the most valuable part. A raw number like "burn rate 85%" means nothing to a developer without context. The tool description translates that into actionable guidance: "below 100% means sustainable, above 200% means delay non-critical changes."

Connecting to your editor via MCP

With all three tools created in Agent Builder, they are automatically available through the MCP server endpoint. Configure your MCP client to connect.

Claude Code configuration

Add the Elastic MCP server to your Claude Code settings:

{
  "mcpServers": {
    "elastic-agent-builder": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://your-kibana-url/api/agent_builder/mcp",
        "--header",
        "Authorization: ApiKey your-base64-api-key"
      ]
    }
  }
}

Cursor configuration

For Cursor, add the server in Settings > MCP Servers:

{
  "mcpServers": {
    "elastic-agent-builder": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://your-kibana-url/api/agent_builder/mcp",
        "--header",
        "Authorization: ApiKey your-base64-api-key"
      ]
    }
  }
}

Once connected, your editor's AI agent will discover all three tools automatically. You can verify by asking: "What Elastic tools do you have available?" The agent should list get_endpoint_health, get_recent_deploys, and get_slo_status.

API key permissions: the API key needs the feature_agentBuilder.read Kibana privilege and read access to the relevant indices (traces-apm.*, observability-annotations, .slo-observability.*). For production use, set the key expiry to 30-90 days and follow the principle of least privilege.

The scenario: "Is it safe to merge this PR?"

A developer on the team has a pull request that increases the timeout for the downstream recommendations service call from 2 seconds to 5 seconds in opbeans-node. Before merging, they ask the agent:

Developer: "I'm about to merge PR #42, which increases the recommendations service timeout from 2s to 5s in opbeans-node. Is it safe to merge right now?"

The agent begins its multi-signal reasoning chain. Here is what happens.

Step 1: the agent calls get_endpoint_health

The agent checks the current health of the service:

Step 2: the agent calls get_recent_deploys

Next, it checks for recent deployments:

Step 3: the agent calls get_slo_status

The agent's response

After correlating all three results, the agent produces a recommendation:

The agent pulled the current p99, checked recent deploys, and read the SLO burn rate. It combined those signals with the timeout change in the PR, flagged the merge as risky, and recommended next steps.

Conclusion: when to use MCP tools instead of pinging an SRE

With Elasticsearch, Agent Builder, and MCP, a developer can answer questions like "is it safe to merge this PR?" from inside their editor, in seconds, without pinging an SRE. Elasticsearch holds the signals: traces, deploy markers, and SLO budgets. Agent Builder is where the platform team encodes how to read those signals: the thresholds, the warm-up windows, the correlation rules. MCP is what carries those tools into the developer's editor.

The query pulls the data. The description tells the agent how to read it. The platform engineer writes the runbook once, and every developer on the team gets to use it.

Next steps: extend Elastic Agent Builder MCP tools to CI/CD

Explore the Elastic Agent Builder documentation for more tool types and configuration options.
See Agentic CI/CD: Kubernetes Deployment Gates with Elastic MCP Server for extending this pattern into your CI/CD pipeline.
Check out Agent Skills for Elastic Observability for a complementary approach using pre-packaged observability skills.

How to trace MCP server tool calls with OpenTelemetry and Elastic APM

Jeffrey Rengifo — Thu, 14 May 2026 00:00:00 GMT

An MCP server is just a Node process, which means OpenTelemetry instrumentation is one --import flag away. What is new is what happens after the traces land in Elastic APM. The same Claude Desktop session that produced them can query them back through the Elastic Agent Builder MCP. The agent analyzes its own tool-call latency, identifies slow tools, and explains failures without leaving the chat. Observability stops being a dashboard that the human checks after the fact and becomes the context the agent uses while working. This post walks through the OTel semantic conventions for MCP, the wrapper pattern for tool spans, and how the loop closes on the Elastic side.

Prerequisites

Elastic Cloud hosted (9.3+) or serverless
Claude Desktop
An MCP server instrumented with the Elastic Distribution of OpenTelemetry (EDOT). We cover how to instrument one below.

The observability gap in MCP servers

MCP servers ship with no built-in observability, which means tool-call latency, errors, and performance baselines are invisible to developers. MCP (Model Context Protocol) servers are increasingly used as infrastructure for AI-powered applications, giving AI models access to databases, APIs, internal tools, and business data. The MCP SDK does not instrument any of it.

The gap shows up in three concrete ways:

When a tool call takes 3 seconds, you don't know if the bottleneck is in your business logic, a downstream API, or the data layer.
When a tool call fails, you get the error message but no context about what the server was doing before it failed.
When you add a new tool, you have no baseline to compare its performance against.

These are the same problems any backend service faces. The answer is the same one backend developers have used for years: distributed tracing with OpenTelemetry.

MCP servers are standard programmatic processes. There is nothing special about them from an instrumentation perspective. You add the OTel SDK, define spans around your tool handlers, and ship traces to your backend. The only new part is knowing which span names and attributes to use so your traces are meaningful and consistent.

What we built

For this article, we use the @modelcontextprotocol/server-everything package, the official reference MCP server published by Anthropic. It ships with a set of tools that cover the common patterns you will find in real-world MCP servers: simple request/response, parameterized calls, long-running operations, and calls that return structured data.

The server exposes several tools. In this article we use three of them:

echo: receives a string and returns it unchanged. A minimal request/response tool, useful for verifying that the instrumentation pipeline works end to end.
get-sum: receives two numbers and returns their sum. Represents a parameterized tool with simple business logic.
trigger-long-running-operation: starts a multi-step operation that takes several seconds to complete. Simulates tools that call downstream APIs or run expensive computations.

We instrumented it with EDOT Node.js (@elastic/opentelemetry-node), which is Elastic's distribution of the OpenTelemetry SDK. EDOT replaces the five or six individual OTel packages you would otherwise install and adds the elasticapm connector that the Kibana APM UI needs to build its service maps, transaction groupings, and latency charts. Without that connector, raw OTLP data arrives in Elasticsearch but the APM views have nothing to build from.

The architecture looks like this:

This is the loop: Claude executes tools, generates telemetry, and then uses a second MCP to analyze that telemetry. The observability data becomes something the AI can reason about, not just something that sits in a dashboard waiting for a human to check it.

Both MCP servers are active simultaneously in Claude Desktop. The instrumented MCP generates telemetry. The Agent Builder MCP lets us query it.

To connect Claude Desktop to both servers, the claude_desktop_config.json looks like this:

{
  "mcpServers": {
    "everything": {
      "command": "node",
      "args": [
        "--import",
        "/path/to/node_modules/@elastic/opentelemetry-node/import.mjs",
        "/path/to/everything/dist/index.js",
        "stdio"
      ],
      "env": {
        "OTEL_SERVICE_NAME": "everything-mcp-server",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "https://",
        "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=ApiKey ",
        "OTEL_LOG_LEVEL": "none"
      }
    },
    "elastic-agent-builder": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https:///api/agent_builder/mcp",
        "--header",
        "Authorization:ApiKey "
      ]
    }
  }
}

The --import /path/to/@elastic/opentelemetry-node/import.mjs flag is all it takes for zero-code auto-instrumentation. But auto-instrumentation only captures HTTP calls, database queries, and other Node.js instrumented libraries. MCP tool calls are application logic, and application logic needs manual spans.

What traces MCP tool calls generate

The OpenTelemetry specification includes official semantic conventions for MCP. Following them means your traces are consistent, searchable by name across tools and teams, and compatible with any OTel-aware backend, including Elastic APM.

Span naming follows the pattern {mcp.method.name} {target}. For a tool call, this becomes tools/call echo or tools/call get-sum. This is what you will see as the transaction name in Kibana APM.

Key attributes on each span:

| Attribute | Value | Purpose | | ----------------------- | ---------------- | ------------------------- | | mcp.method.name | tools/call | The MCP protocol method | | gen_ai.tool.name | echo | The specific tool invoked | | gen_ai.operation.name | execute_tool | GenAI semantic convention | | error.type | error class name | Set only on failure |

The wrapper pattern that creates these spans looks like this:

const tracer = trace.getTracer('everything-mcp-server', '1.0.0');

function withToolSpan(toolName, fn) {
  return tracer.startActiveSpan(`tools/call ${toolName}`, (span) => {
    span.setAttribute('mcp.method.name', 'tools/call');
    span.setAttribute('gen_ai.tool.name', toolName);
    span.setAttribute('gen_ai.operation.name', 'execute_tool');

    try {
      const result = fn();
      span.end();
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      span.setAttribute('error.type', err.constructor.name);
      span.end();
      throw err;
    }
  });
}

Each tool handler wraps its logic in withToolSpan. The result is a named span in Elastic APM for every tool invocation, with duration, status, and error details attached.

Security note: The OTel spec defines two optional attributes for tool calls: gen_ai.tool.call.arguments and gen_ai.tool.call.result_. Both are flagged as potentially containing sensitive data. The_ get-env tool in the everything server is a good example of why this matters: it returns all environment variables, which may include API keys and credentials. Capture these attributes only if you have confirmed the data is safe to store in your observability backend, and consider masking or filtering at the SDK level before export.

Kibana APM: exploring MCP server performance

After starting Claude Desktop with both MCPs configured and triggering a few tool calls, the everything-mcp-server service appears in Kibana under Observability > Applications > Services Inventory:

Transactions view

Kibana groups traces by transaction name. Because we follow the semantic conventions, each tool gets its own row: tools/call echo, tools/call get-sum, tools/call trigger-long-running-operation. You can immediately see latency, throughput, and error rate per tool without any configuration.

This is where the value of consistent span naming becomes concrete. If you have five different developers adding tools to an MCP server and everyone follows tools/call {toolName}, the APM UI stays organized automatically.

Trace waterfall

Clicking on a specific trace shows the waterfall view. For a single tool call, the waterfall is straightforward: one span covering the full execution. If your tool handler makes downstream HTTP requests or database queries that are auto-instrumented, those appear as child spans. You can see exactly how much time was spent in business logic versus waiting for external calls.

Latency distribution

The latency chart shows p50, p95, and p99 distribution across all executions of a given tool. This makes it easy to distinguish between tools that are consistently fast and those that have occasional outliers. The trigger-long-running-operation tool, for example, shows a wide distribution depending on how many steps were requested: a useful baseline for understanding expected execution time ranges before setting alerts.

Error tracking

Failed tool calls appear in the Errors panel with their full stack trace, the span attributes attached at the time of failure, and a count of how many times the error has occurred. If you record the exception with span.recordException(err), Kibana links the error directly to the trace that produced it.

Closing the loop with the Agent Builder MCP

The Elastic Agent Builder MCP lets Claude query its own trace data from Elasticsearch in the same chat session that produced the traces. The Agent Builder MCP can query any Elasticsearch index the API key has access to, and APM traces are stored under .ds-traces-apm.otel-default-*. Granting the Agent Builder API key read access to those indices is what closes the loop: the agent that executed the tool calls can now reason about how they performed.

Here is what this looks like in practice. To generate traces, let's ask in Claude Desktop: "Use the echo tool to say hello, then use get-sum to add 1337 and 42, then run a long-running operation with 3 steps."

Claude executes three tool calls on the instrumented MCP. Three spans land in Elastic APM.

Now, without leaving the chat, let's try querying the traces by asking: "Search the APM trace data from the last 10 minutes. What tool calls were made, and how long did each one take?"

We can confirm the information against the services data:

Claude uses the Agent Builder MCP to run a query against the traces index. It returns the tool names, durations, and status from the actual trace data, then synthesizes an answer in natural language.

You can go further by asking:

"Which of those tool calls had the highest p95 latency?"
"Did any tool calls fail? If so, what was the error message?"
"Compare the latency of the echo tool vs get-sum across all calls in the last hour."

Each of these questions translates into an ES|QL query via the Agent Builder's platform.core.execute_esql tool, run against the APM trace indices.

Why Elastic for MCP observability

The Agent Builder closes the loop: this is the part that is specific to the Elastic ecosystem. Because APM data lives in Elasticsearch, and Elasticsearch is queryable via the Agent Builder MCP, you can bring your AI agent's own observability data back into the conversation. Your AI can reflect on its own performance and spot anomalies.

APM UI built for distributed tracing: Kibana's APM interface is designed for exactly this kind of data: named transactions, trace waterfalls, latency percentiles, error tracking with stack traces, and service maps.

Managed OTLP endpoint: Elastic APM accepts OTLP directly since Elastic 8.x. You point OTEL_EXPORTER_OTLP_ENDPOINT at your APM server and it works.

EDOT simplifies the setup: the elasticapm connector is included, which means the APM UI views work without any additional configuration.

Conclusion

MCP servers do not need special observability tooling. They are programmatic processes, and OpenTelemetry is the right instrument for processes. The OTel MCP semantic conventions are stable and give you a consistent naming scheme that scales across tools and teams.

What makes the Elastic setup interesting is not the instrumentation itself. It is the second MCP. When your observability data lives in Elasticsearch, you can query it from the same AI session that generated it. That feedback loop is new, and it opens up use cases that dashboards alone cannot cover: real-time anomaly questions, automated triage, and AI-assisted incident investigation from the chat interface your team is already using.

Next steps

Frequently asked questions

How do I add tracing to an MCP server? Use the OpenTelemetry SDK and follow the official OTel MCP semantic conventions. With the Elastic Distribution of OpenTelemetry (EDOT) for Node.js, a single --import flag enables auto-instrumentation for HTTP and database calls. Tool-call spans need to be added manually using a wrapper that sets mcp.method.name, gen_ai.tool.name, and gen_ai.operation.name.

Why are my MCP tool calls slow and how do I find the bottleneck? Without tracing, an MCP tool call is a black box: you see the result but not where the time went. Instrument the server with OpenTelemetry, ship traces to Elastic APM, and use the trace waterfall view in Kibana to see exactly how much time was spent in business logic versus downstream HTTP or database calls.

Can I send MCP server traces to Elastic APM without a custom collector? Yes. Elastic APM accepts OTLP directly. Set OTEL_EXPORTER_OTLP_ENDPOINT to your APM endpoint and OTEL_EXPORTER_OTLP_HEADERS with an API key, and traces flow in. EDOT bundles the elasticapm connector that the Kibana APM UI needs for service maps and transaction grouping.

What span names and attributes should I use for MCP tool calls? Follow the OpenTelemetry MCP semantic conventions: name spans {mcp.method.name} {target} (for example, tools/call echo), and set mcp.method.name, gen_ai.tool.name, and gen_ai.operation.name=execute_tool. On failure, set error.type to the error class name. Consistent naming means the Elastic APM transactions view groups your tool calls automatically.

How is this different from sending MCP traces to Datadog or Grafana? Any OTel-compatible backend can receive the traces. The Elastic-specific part is the Agent Builder MCP, which lets the same AI agent that generated the traces query them back from Elasticsearch in natural language. That feedback loop, where the AI reasons about its own tool-call performance, is not available with backends that do not expose their data through an MCP server.

Should I capture MCP tool call arguments and results in my traces? The OTel spec defines gen_ai.tool.call.arguments and gen_ai.tool.call.result as optional and warns they may contain sensitive data. Tools like get-env, which returns environment variables, illustrate the risk: API keys and credentials can land in your observability backend. Capture these only when the data is safe to store, and consider masking at the SDK level before export.

Does this work for MCP servers written in languages other than Node.js? The OpenTelemetry MCP semantic conventions are language-agnostic. EDOT is available for Node.js, Java, Python, .NET, and other languages, and any of them can send OTLP to Elastic APM. The wrapper pattern shown in this post translates directly: open a span around the tool handler, set the standard attributes, record exceptions on failure.

Connecting Cursor to Production Logs via the Elastic MCP Server

Jeffrey Rengifo — Wed, 29 Apr 2026 00:00:00 GMT

Prerequisites

Elasticsearch 9.3+ (or Elastic Cloud Serverless)
Elasticsearch API KEY and Kibana URL
An application instrumented with Elastic APM: the RUM agent for frontend interactions (populates traces-apm-*) and the APM agent for backend errors (populates logs-apm.error-*
Cursor (version 2.6+) installed

The problem with two worlds

Application logs and code are two separate worlds that don't talk to each other. If you want to apply log insights into the application you have to analyze the logs, and then come back to the editor and apply your findings.

The Model Context Protocol (MCP) changes this. MCP is an open standard that lets AI clients like Cursor connect to external tools and data sources through a standardized interface. Instead of your IDE only knowing about your local code, it can also talk to your Elasticsearch cluster, query your APM data, and reason about production behavior alongside your source files.

Elastic ships a built-in MCP server as part of Agent Builder. You define tools in Kibana, expose them via the MCP endpoint, and any MCP-compatible client can call them. Cursor supports MCP natively, which means you can set this up in minutes.

What we're building

We're working with an eCommerce search app instrumented with Elastic APM. The RUM JS agent tracks filter click interactions from the browser, stored in traces-apm-default. The Node.js APM agent captures backend errors, stored in logs-apm.error-default.

Two situations come up during development:

Use case 1: The product team wants to simplify the search page. There are six filters but we don't know which ones users actually click. We need usage data to decide which to keep.
Use case 2: Users report intermittent 500 errors on search. The errors are not constant and started two days ago. We need the error details to find the root cause.

To bring that data into Cursor, we'll build two Agent Builder tools in Kibana and connect them via the Elastic Agent Builder MCP Server:

get_filter_usage: queries traces-apm-default for filter click events and returns a usage breakdown by filter name
get_recent_errors: queries logs-apm.error-default for the most recent error groups for a given service, including the exception message and stack trace culprit

For a deeper look at the overall architecture, see the Agent Builder reference guide.

Setting up the Elastic MCP Server

Step 1: Create the Agent Builder tools

We create both tools via the Kibana Agent Builder API. Each tool is an ES|QL query with a name and description that Cursor uses to decide when to call it. The full implementation of the tools is in the following notebook.

Tool 1: get_filter_usage

The product team needs to know which filters users actually click before deciding which ones to remove. The query reads RUM interaction events from traces-apm-default and groups them by filter name:

    {
      "id": "get_filter_usage",
      "type": "esql",
      "description": "Returns the usage count for each search filter in the ecommerce-search-ui service, sorted by most used first.",
      "configuration": {
        "query": "FROM traces-apm-default | WHERE service.name == \"ecommerce-search-ui\" | WHERE transaction.type == \"user-interaction\" | WHERE labels.filter_name IS NOT NULL | STATS count = COUNT(*) BY labels.filter_name | SORT count DESC"
      }
    }

Tool 2: get_recent_errors

For the error debugging use case, we need to surface the most frequent recent errors for a service, along with where in the code they originate. STATS ... BY groups errors by their fingerprint (grouping_key), surfaces the exception message and the line of code that caused it (culprit), and ranks by frequency:

    {
      "id": "get_recent_errors",
      "type": "esql",
      "description": "Returns the most frequent error groups for ecommerce-search-ui, ranked by occurrence count, with the exception message and code location.",
      "configuration": {
        "query": "FROM logs-apm.error-default | WHERE service.name == \"ecommerce-search-ui\" | WHERE processor.name == \"error\" | STATS count = COUNT(*) BY error.grouping_key, error.exception.0.message, error.culprit | SORT count DESC | LIMIT 5"
      }
    }

Both tools are created with POST /api/agent_builder/tools. You can learn more about the Kibana API endpoints for Elastic Agent Builder here.

Step 2: Connect to Cursor

Open ~/.cursor/mcp.json and add the Elastic server. For detailed information, see the Cursor documentation. The Agent Builder MCP endpoint uses Server-Sent Events (SSE) transport, so we connect via mcp-remote, a lightweight bridge that Cursor invokes as a local process:

    {
      "mcpServers": {
        "elastic-agent-builder": {
          "command": "npx",
          "args": [
            "-y",
            "mcp-remote",
            "https://YOUR_KIBANA_URL/api/agent_builder/mcp",
            "--header",
            "Authorization: ApiKey YOUR_API_KEY"
          ]
        }
      }
    }

Replace YOUR_KIBANA_URL and YOUR_API_KEY with your values.

Restart Cursor, open the Agent panel, and confirm that get_filter_usage and get_recent_errors appear in the available tools list.

Use case 1: Data-driven UI optimization

The eCommerce search page has six filters: category, manufacturer, price range, customer gender, day of week, and region. The product team wants to simplify the UI by removing filters that users don't use as much. Rather than guessing, we ask Cursor to check.

When you type a prompt in Cursor's Agent panel, the model sees the name and description of every connected MCP tool. It matches your intent to the best-fitting tool and calls it automatically. This is why the description field we set in Step 1 matters: it's what the model reads to decide which tool answers your question. If you are interested in learning more about Cursor’s MCP tools management, read the following documentation.

Open a Cursor chat and ask: "Show me how often each search filter is used." Cursor calls the tool and returns something like:

The category and manufacturer filters get most of the clicks. The bottom three filters (customer_gender, day_of_week, region) are rarely used.

Ask Cursor to act on this: "Based on this data, simplify the SearchFilters component. Keep the top 3 filters visible, collapse the others under a 'More filters' toggle."

Cursor opens src/components/SearchFilters.jsx, reads the current implementation, and proposes the change.

Before:

After:

The entire loop took one chat prompt. The decision was backed by production data, not a team discussion about what users probably care about.

Use case 2: Production error debugging

A bug report comes in: intermittent 500 errors on the search endpoint. The errors started appearing two days ago but they're not constant. The developer opens Cursor and asks: "Show me what errors ecommerce-search-ui is throwing."

Cursor calls the tool and returns the error groups:

The error message is explicit: category is a text field and can't be used in terms of aggregation. The correct field is category.keyword.

With APM data available alongside your code, the debugging session becomes a conversation: you describe the symptom, the agent pulls the relevant logs, and you work through what's happening together. You can ask follow-up questions, check whether the error correlates with a recent deployment, or ask which endpoints are most affected, all within the same context where you'll make the fix. If you want to go further, Elastic also provides pre-built observability tools in Agent Builder that you can use alongside custom tools like the ones we created here. For a complementary approach to AI-driven observability, see how to monitor web AI agents with OpenLIT and Elastic.

Conclusion

What we covered:

How to create Agent Builder tools in Kibana that wrap APM data queries
How to connect the Elastic Agent Builder MCP Server to Cursor in three lines of JSON
Using production telemetry to make a UI decision backed by real usage data
Debugging a production error from the same window where you fix it

These two use cases are a starting point. The same pattern works for any data you have in Elasticsearch: performance metrics, A/B test results, audit logs, feature flag usage, user session data. Define the Agent Builder tool, connect it via MCP, and it becomes part of your development context in Cursor. For other examples of what's possible, see automating synthetic monitoring with MCP and agentic CI/CD deployment gates.

Next steps

How to cut Elasticsearch log storage costs with LogsDB

Jeffrey Rengifo — Thu, 09 Apr 2026 00:00:00 GMT

LogsDB is a specialized Elasticsearch index mode that gives you full functionality at a fraction of the storage cost. Your Kibana dashboards, searches, alerts, and visualizations all continue to work exactly as before. No data is discarded. No queries need to be updated. No workflows break. It is one setting, and everything else gets cheaper.

In benchmarks, LogsDB brought a dataset from 162.7 GB down to 39.4 GB — a 76% reduction in storage. You can explore the full nightly benchmark results at elasticsearch-benchmarks.elastic.co.

In this tutorial you'll reproduce the experiment yourself using Kibana Dev Tools and an Apache logs dataset. You'll create two identical indices, ingest the same documents into both, and measure the storage difference with the _stats API. By the end, you'll see a 44% reduction on your test data — and understand exactly why production numbers push even higher.

Already on Elasticsearch 9.2+? Any data stream with a logs- prefix already uses LogsDB by default. Jump to What about your existing logs? to verify your setup.

Want the full picture? For the engineering history behind these savings — how Lucene doc values, synthetic _source, index sorting, and ZSTD were developed and stacked over twelve years — see Elasticsearch over the years: how LogsDB cuts index size by up to 75%.

Prerequisites

Elasticsearch 8.17+ cluster, Elastic Cloud deployment, or Serverless
Kibana with Dev Tools access
Some logs
Basic familiarity with running API calls in Kibana Dev Tools

How LogsDB saves storage

LogsDB stacks three mechanisms to achieve its storage reduction:

Index sorting — documents are sorted by host.name then @timestamp, grouping similar log lines so compression codecs find far more repeated patterns. Sorting alone accounts for roughly 30% of the savings.
ZSTD compression with delta/GCD/run-length encoding — best_compression switches from LZ4 to Zstandard and applies numeric codecs to each doc values column. The standard index in this tutorial uses LZ4, so part of what you're measuring is the full package LogsDB delivers automatically.
Synthetic _source — Elasticsearch skips storing the raw JSON blob entirely and reconstructs _source on demand from doc values, adding another 20–40% of savings on top.

Synthetic _source trade-offs: Field ordering in returned documents may differ from the original, and some edge cases around multi-value array fields behave differently. For most log analytics workloads these differences are invisible, but check the synthetic _source documentation before enabling it in latency-sensitive applications.

For a deep dive into the architecture behind each mechanism, see Elasticsearch over the years: how LogsDB cuts index size by up to 75%.

Let's now walk through the steps you can take to enable LogsDB and measure the storage savings.

Step 1: Collect logs with Elastic Agent

The recommended way to ingest Apache logs into Elasticsearch is through Elastic Agent with the Apache integration. It handles collection, parsing, ECS field mapping, and routing automatically.

Browse all available integrations in the Elastic integrations catalog.

Once the Agent is collecting logs and routing them to logs-apache.access-*, move to the next step.

Step 2: Create the two indices

All commands in this tutorial are run in Kibana Dev Tools.

Create one standard index and one LogsDB index with identical field mappings. The only difference is "index.mode": "logsdb".

Standard index:

PUT /apache-standard
{
  "mappings": {
    "properties": {
      "@timestamp":                  { "type": "date" },
      "host.name":                   { "type": "keyword" },
      "http.request.method":         { "type": "keyword" },
      "url.path":                    { "type": "keyword" },
      "http.version":                { "type": "keyword" },
      "http.response.status_code":   { "type": "integer" },
      "http.response.bytes":         { "type": "integer" },
      "http.request.referrer":       { "type": "keyword" },
      "user_agent.original":         { "type": "keyword" }
    }
  }
}

LogsDB index:

PUT /apache-logsdb
{
  "settings": {
    "index.mode": "logsdb"
  },
  "mappings": {
    "properties": {
      "@timestamp":                  { "type": "date" },
      "host.name":                   { "type": "keyword" },
      "url.path":                    { "type": "keyword" },
      "http.request.method":         { "type": "keyword" },
      "http.version":                { "type": "keyword" },
      "http.response.status_code":   { "type": "integer" },
      "http.response.bytes":         { "type": "integer" },
      "http.request.referrer":       { "type": "keyword" },
      "user_agent.original":         { "type": "keyword" }
    }
  }
}

That single "index.mode": "logsdb" line activates all three storage mechanisms. Elasticsearch enables these additional settings behind the scenes — you don't set any of them manually:

{
  "index.sort.field":              ["host.name", "@timestamp"],
  "index.sort.order":              ["asc", "desc"],
  "index.codec":                   "best_compression",
  "index.mapping.ignore_malformed": true,
  "index.mapping.ignore_above":    8191
}

Step 3: Reindex the logs

Use the _reindex API to copy the same documents into both test indices:

POST /_reindex
{
  "source": { "index": "logs-apache.access-*" },
  "dest":   { "index": "apache-standard" }
}

POST /_reindex
{
  "source": { "index": "logs-apache.access-*" },
  "dest":   { "index": "apache-logsdb" }
}

Both indices now hold identical documents, so the storage comparison in the next step reflects only the index mode difference.

Step 4: Force merge for a fair comparison

Before measuring, force merge both indices to a single segment:

POST /apache-standard/_forcemerge?max_num_segments=1

POST /apache-logsdb/_forcemerge?max_num_segments=1

These calls block until the merge finishes. Wait for both responses before continuing.

Why this matters: Elasticsearch writes data into multiple Lucene segments before merging them in the background. Measuring mid-merge gives artificially inflated numbers because each segment is compressed independently. Forcing a single segment shows the real steady-state storage footprint you'd see in a mature production index.

Only run _forcemerge on indices that are no longer being written to. Force merging an index that is still receiving writes is resource-intensive and can impact ingestion performance. In production, you can use Index Lifecycle Management (ILM) to automate force merges as part of the warm or cold phase, once an index is rolled over and no longer actively ingested into.

Step 5: Measure the difference

GET /apache-standard/_stats?filter_path=indices.*.primaries.store

GET /apache-logsdb/_stats?filter_path=indices.*.primaries.store

The filter_path parameter keeps the response focused. Look for primaries.store.size_in_bytes in each response.

In our test with Apache log records, the results were:

| Index | Documents | Size | |------------------|-----------|----------| | apache-standard | 111,818 | 15.37 MB | | apache-logsdb | 111,818 | 8.6 MB | | Reduction | | 44% |

To put this in perspective: at 1 TB of log data, LogsDB brings that down to around 560 GB. That's 450 GB saved without any changes to your queries. At production scale with billions of documents and synthetic _source enabled, savings push to 76% — taking 162.7 GB down to 39.4 GB in our benchmark.

Visualize in Kibana

To see the storage difference visually, open Kibana and go to Management → Stack Management → Index Management. You'll see both indices listed with their current sizes side by side.

Why Kibana shows larger numbers than _stats: Kibana Index Management displays the total index size including all replica shards. The _stats query above uses primaries to report primary shards only. The ratio between the two indices remains the same either way.

What about your existing logs?

Elasticsearch 9.2+ (already enabled by default)

Since 9.2, any data stream matching the logs-* naming pattern automatically uses LogsDB. You're likely already saving storage without any configuration change.

Verify your existing data streams:

GET /.ds-logs-*/_settings?filter_path=*.settings.index.mode

If you see "index.mode": "logsdb" in the responses, you're already getting the savings.

Elasticsearch 8.x or 9.0–9.1 (enable per data stream via index template)

For earlier versions, enable LogsDB on a data stream by updating its index template. This affects all new indices created from that template — existing indices are not changed, so the transition is safe and gradual.

Option A — Update an existing template:

PUT _index_template/logs-myapp-template
{
  "index_patterns": ["logs-myapp-*"],
  "data_stream": {},
  "template": {
    "settings": {
      "index.mode": "logsdb"
    }
  },
  "priority": 200
}

Option B — Check and patch an existing integration template:

First, find the template managing your data stream:

GET _index_template/logs-apache*

Then add the index.mode setting to the template.settings block using a PUT _index_template/ call with the full template body including your addition.

After updating the template, the next index rollover will use LogsDB. Trigger a rollover immediately if you don't want to wait:

POST /logs-myapp-default/_rollover

Upgrading from 8.x to 9.0+: Existing data streams are not changed automatically. Only new rollovers will use LogsDB. There is no data loss and no reindexing required — the savings accumulate as new indices roll over.

What about query performance?

LogsDB does not significantly impact query performance for typical log analytics workloads. The index sorting by host.name and @timestamp can actually improve range query and aggregation performance on those fields, since matching documents are stored adjacently. Queries that don't filter on those fields perform comparably to a standard index.

For indexing throughput data across releases, see the performance section of the companion article.

Conclusion

LogsDB activates with a single "index.mode": "logsdb" setting and delivers measurable storage savings immediately: 44% in our hands-on test, and 76% (162.7 GB → 39.4 GB) in production benchmarks with synthetic _source. On Elasticsearch 9.2+, logs-* data streams already use LogsDB by default. For 8.x or earlier 9.x clusters, a one-line index template change enables it on your next rollover with no data loss and no reindexing required.