AI root cause analysis in Elastic Agent Builder that cites its evidence

Root cause analysis is the work of separating the failure that started an incident from everything that broke because of it, or merely alongside it. A cascading failure makes that hard: a deploy that shipped the same minute and a second service failing at the same time both look like causes.

Three ES|QL tools and about two dozen lines of query text turn AI root cause analysis into something you can check. This article builds them in Elastic Agent Builder and wires them to an agent whose report names the tool behind every number, and a trace.id and document _id behind every root cause claim.

What you need to reproduce this root cause analysis

• Elasticsearch and Kibana 9.5, self-managed or on Elastic Cloud.
• A generative AI connector for Agent Builder. Ours is Anthropic Claude Sonnet 4.6.
• Python 3.12 with opentelemetry-distro[otlp] and the Flask and requests instrumentations.
• An Elasticsearch API key with write access for OTLP ingest and Kibana access for the Agent Builder APIs.

Use the companion notebook if you want to reproduce the use case in this article.

The demo environment: four Python services on OpenTelemetry

Four Python services run on one host. checkout-api handles the customer request and calls pricing-api, which calls fx-rates for a currency quote. A fourth service, search-api, serves product search and sits outside that call path.

Each service exports OTLP straight to the Elasticsearch native OTLP endpoint, with no collector in between. Logs land in logs-generic.otel-default and spans in traces-generic.otel-default. Two checkout-api processes run on different versions at once, which one of the tools below depends on.

Starting a service is one command, with the OTLP endpoint and the service identity in the environment:

export OTEL_EXPORTER_OTLP_ENDPOINT="${ES_URL}/_otlp"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey ${ES_API_KEY}"
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true

OTEL_SERVICE_NAME=checkout-api \
OTEL_RESOURCE_ATTRIBUTES="service.version=2026.07.26.2,deployment.environment=production" \
  opentelemetry-instrument python services/checkout_api.py

Why the loudest service is not the root cause

At 09:55:33Z, fx-rates starts refusing about a third of its requests because its cached FX snapshot is older than the max_age it enforces. pricing-api turns each refusal into a 500, and checkout-api returns a 500 to the customer.

Two unrelated things happen in the same window. Release 2026.07.26.2 of checkout-api rolls out at 09:55, the minute the errors start, and 120 milliseconds after fx-rates breaks, search-api starts timing out during a scheduled reindex of its catalog.

One failure propagates, and the two other signals share only the clock:

The on-call view is four unhealthy services and a fresh deploy. This query gives the failure rate per service over the window:

FROM traces-generic.otel-default
| WHERE kind == "Server"
| EVAL failed = CASE(attributes.http.status_code >= 500, 1, 0)
| STATS failures = SUM(failed), requests = COUNT(*)
    BY service = resource.attributes.service.name
| EVAL failure_rate_pct = ROUND(100.0 * failures / requests, 1)
| SORT failure_rate_pct DESC

Nothing in the result ranks the candidates. search-api has the highest failure rate at 36.4%, and it is the one service with no involvement in the failing checkouts.

Automated root cause analysis with the default Elastic AI Agent

Before writing any tools, we gave the incident to the default Elastic AI Agent to see how far the built-in observability skills get on their own:

checkout-api started returning HTTP 500s to customers today. Investigate the
window 2026-07-26T09:50:00.000Z to 2026-07-26T10:02:00.000Z and tell me the root
cause. Context you have from the deploy log: checkout-api release 2026.07.26.2
rolled out at 09:55 UTC, and the on-call channel also reported search-api
timeouts starting at 09:55 UTC.

It got the cascade right. It named fx-rates as the origin, quoted the stale-snapshot message, and separated the search-api timeouts as a different problem. Then it ranked its hypotheses:

Hypothesis 2 claims the release restarted fx-rates without refreshing its snapshot, or lowered its max_age. Neither appears in the telemetry: the release was on checkout-api, and fx-rates reports service.version 2026.07.19.1 across all 7,374 of its spans.

The agent had a timestamp: the deploy and the first error share a minute. Across three runs, it connected them with a different invented mechanism each time:

• a restart that dropped the snapshot
• a new call path that was not exercised before the release
• a previously tolerant code path that stopped tolerating stale data

The numbers have the same problem. The impact summary in that same answer reports the failure rate as "~38%", dividing the 2,043 failures by the 5,331 successes instead of the 7,374 requests, with no query next to the number for a reader to check.

The rest of this article closes that gap with tools.

The 16 built-in observability tools in Agent Builder

Agent Builder ships with tools that cover most of the exploration work, so check the built-in tools reference before writing anything. The catalog is under Agent Builder > Manage components > Tools:

Sixteen of the built-in tools are observability tools, organized around investigation steps instead of index operations:

One platform tool matters here too. platform.streams.investigation_progress_report is what an agent calls to publish its hypothesis list while it works, with a status per hypothesis, a conclusion, and an explicit list of what the data could not settle:

{
  "summary": "string",
  "hypotheses": [
    {
      "candidate": "string",
      "confidence": 0.0,
      "status": "investigating | dismissed | confirmed",
      "reason": "string"
    }
  ],
  "conclusion": "string",
  "gaps_found": ["string"]
}

Three ES|QL tools for AI root cause analysis

Agent Builder's built-in observability tools are shaped for showing the landscape. Ruling a candidate out needs one number that answers one question, and two questions in this incident have no built-in tool. A third returns the document IDs that make a claim checkable.

How to tell one incident from two coincident failures (rca_failure_shapes)

Two services failing in the same minute belong to the same incident only if they appear in the same requests. This tool groups error logs by trace, collapses each trace to the set of services that erred inside it, then counts each shape:

FROM logs-generic.otel-default
| WHERE @timestamp >= TO_DATETIME(?start) AND @timestamp <= TO_DATETIME(?end)
  AND severity_text == "ERROR" AND trace_id IS NOT NULL
| STATS services = VALUES(resource.attributes.service.name) BY trace_id
| EVAL failure_shape = MV_CONCAT(MV_SORT(services), " + ")
| STATS traces = COUNT(*) BY failure_shape
| SORT traces DESC
| LIMIT 20

MV_CONCAT(MV_SORT(...)) matters here. Grouping directly by a multivalue field makes ES|QL expand it, one row per service, which loses the combination. Collapsing the set to a single string first keeps the shape intact.

Two clusters come back, and they share no traces. The three checkout services appear together in 2,043 traces, search-api errs alone in 1,324, and that zero overlap settles the coincidence in one row.

How to rule out a deploy as the root cause (rca_version_split)

A release that caused the failures makes the version carrying it fail at a materially higher rate than the version it replaced. Server spans carry both service.version and the response status, so one query decides it:

FROM traces-generic.otel-default
| WHERE @timestamp >= TO_DATETIME(?start) AND @timestamp <= TO_DATETIME(?end)
  AND resource.attributes.service.name == ?service AND kind == "Server"
| EVAL failed = CASE(attributes.http.status_code >= 500, 1, 0)
| STATS failures = SUM(failed), requests = COUNT(*)
    BY version = resource.attributes.service.version
| EVAL failure_rate_pct = ROUND(100.0 * failures / requests, 1)
| SORT version

27.2% against 28.2%, on roughly 3,700 requests per version, is a difference within noise. The version that was already running fails as often as the one that shipped, which rules the release out.

The tool depends on both versions serving at once. With an instantaneous and total rollout, no query separates a broken new version from something else breaking at the same moment.

How to return document IDs an agent can cite (rca_evidence_sample)

METADATA _id on an ES|QL source command returns the Elasticsearch document ID, so a claim can point at a specific record:

FROM logs-generic.otel-default METADATA _id, _index
| WHERE @timestamp >= TO_DATETIME(?start) AND @timestamp <= TO_DATETIME(?end)
  AND severity_text == "ERROR" AND resource.attributes.service.name == ?service
| KEEP @timestamp, _id, _index, trace_id,
       attributes.error.kind, attributes.upstream.service, body.text
| SORT @timestamp DESC
| LIMIT 5

Each tool is registered with one POST kbn:/api/agent_builder/tools call carrying the query and its typed parameters. Registered, rca_failure_shapes looks like this; the agent fills start and end at call time:

Write the tool description as the decision it supports

The model reads the tool description to decide when to call the tool, so the description does more work than the name.

We describe rca_failure_shapes as the way to test whether two services that broke together are one failure or two. That phrasing got it called whenever a question mentioned a second failing service.

The description also outweighs the agent instructions. An agent with these three tools and one line of instruction, "You are an SRE assistant, help the user find the root cause of production incidents", still ruled out the deploy and the coincidence every time.

Wiring the tools into an incident root cause analysis agent

The agent gets the three custom tools, four built-in observability tools, and platform.streams.investigation_progress_report. Its instructions are five numbered steps:

1. Scope. Establish the affected service, the failure window, and the size of the
   symptom before you name any cause. State the window as an explicit ISO 8601
   range and reuse that same range in every tool call.

2. Enumerate. Write down at least three candidate causes before you test any of
   them. Include the candidate a human on call would reach for first, such as a
   recent deploy or another service that started failing at the same minute.

3. Refute. For each candidate, state the observation that would prove it wrong,
   then run the query that produces that observation. A candidate is dismissed
   when the refuting observation is present, not when a different candidate
   looks better.

4. Cite. Every number you report must name the tool that returned it. Every claim
   about a root cause must carry at least one trace_id and at least one document
   _id. A claim with no citation is not a finding, it is a guess.

5. Report. Put anything the available data cannot settle in gaps_found rather
   than resolving it with reasoning.

Two rules follow the steps. Each one blocks a mistake we saw in the default agent's answers:

• Do not rank services by error volume and call the loudest one the cause.
• Do not treat time correlation as causation. Two services that start failing in the same minute belong to the same incident only if they appear in the same traces.

The finished agent, with Elastic capabilities switched off so everything it produces comes from these eight tools:

Running the investigation: 11 tool calls, 72 seconds

We gave the new agent the same question, word for word. It opened with a progress report listing its candidates, then ran the refuting queries in parallel batches:

Eleven tool calls and 72 seconds later, the report opens with a citation table:

Three layers, three verbatim messages, three document IDs, and one trace ID shared by all of them. The origin message states the mechanism: StaleQuoteError: fx snapshot age 1215s exceeds max_age 900s (provider=ecb-eod).

The dismissals come next, each with the observation that ruled it out:

The report closes with a gaps section:

The first gap covers the question the default agent answered with an invented mechanism. The snapshot was already 1,127 seconds old at the first error, so the feed stopped refreshing before the window opened, and no log in this system records when. The next place to look is the feed ingestion job.

Verifying the agent's citation by hand in ES|QL

The report gives a trace ID, so open it:

FROM logs-generic.otel-default
| WHERE trace_id == "a131d0cf7343195c0a7a14f6a98da6b7"
| KEEP @timestamp, resource.attributes.service.name,
       attributes.upstream.service, body.text
| SORT @timestamp ASC

Three records, one millisecond apart, in the order the report claimed. fx-rates errs first and names no upstream, because it is the origin. pricing-api names fx-rates, and checkout-api names pricing-api.

Add upstream.service to your error logs

upstream.service is a logging convention that OpenTelemetry does not provide: each service writes it on the error it emits when a dependency fails. With that one key, error records order themselves into a call chain, and the ordering survives the clock skew and sub-millisecond hops that break timestamp sorting.

Two other fields carry the rest of the weight: trace.id arrives free from auto-instrumentation as long as the log is emitted inside an active span, and a structured error.kind lets you count failure modes without matching on message text.

Extracting knowledge indicators from your streams

Knowledge indicators are facts Elastic extracts from your raw data with LLM models. It picks out things like the underlying infrastructure and the dependencies between services.

You find this option under Streams > {your_stream} > Significant events:

Click Generate and the indicators are created for you:

What three ES|QL tools changed about the agent's answer

Three ES|QL tools and about two dozen lines of query text turned an agent's answer into a report that names its candidates, shows the observation that ruled each one out, and carries a record ID behind every number.

Next steps

• Run the companion notebook to reproduce the incident, the tools, and the agent in your own cluster.

• Write the query that rules out your team's most common wrong answer, the recent deploy or the loudest service, and register it as a tool whose description states the decision it supports.

• Audit one service's error logs for trace.id, a structured error.kind, and a declared upstream.service. The last two are a one-line change in your logging.

• Register the failure modes your team already knows about as significant events, so the next investigation starts with a history instead of a blank window.

• For a different shape of Agent Builder investigation, read From five dashboards to one prompt, which scores APM service health with five ES|QL tools.