This post builds that pipeline from a real debugging session and ends with Agent Builder diagnosing a Jellyfin playback failure from the enriched logs. It also covers the alternatives, journald and the OpenTelemetry Collector, and when to prefer them.

How rootless Podman logs differ from Docker logs

The Docker integration reads Docker's format, paths and socket, and rootless Podman uses different ones for all three. Each produced a distinct symptom during our debugging session.

The log format is CRI, not Docker JSON

Podman's default file-based log driver is k8s-file, and even --log-driver=json-file is just an alias for it.

Instead of Docker's one-JSON-object-per-line format, Podman writes the format used by Kubernetes container runtimes (CRI): a timestamp, the stream name, a partial/full flag, and then the message.

An example follows:

A parser expecting Docker JSON breaks on this immediately.

Where rootless Podman stores container logs

Rootful Docker writes logs under /var/lib/docker/containers/<container-id>/. Rootless Podman stores everything under the user's home instead:

Any integration with a hardcoded /var/lib/... path finds nothing.

The Podman API socket is per-user

In rootless Podman there is no /var/run/docker.sock. Instead, it exposes a Docker-compatible API on a user-scoped socket, typically /run/user/<uid>/podman/podman.sock.

Where Elastic Agent runs: host install vs containerized

This walkthrough assumes Elastic Agent is installed directly on the host, not in a container. In our case, the Agent is Fleet-managed and installed directly on a Ubuntu host. It is not running in a container. That matters because the paths in the Agent Policy are resolved by the Agent process itself: a containerized Agent only sees its own mount namespace, so the host directories above would have to be volume-mounted into the Agent container and referenced by their in-container paths. Running as root with default privileges is also what lets the Agent read files under another user's home directory.

How the rootless Podman log pipeline fits together

The setup we ended up with is simple once all the pieces are known:

1. Elastic Agent tails the raw CRI log files from the rootless user's container storage.
2. A container parser with format: cri strips the timestamp, stream, and flag prefix from each line.
3. A processor extracts the container ID from a fixed position in the log file path.
4. The add_docker_metadata processor resolves that ID against the rootless Podman socket and attaches container name, image, and labels.
5. Enriched events are shipped to Elasticsearch.

Let's now build the log ingestion pipeline step by step.

Step 1: Confirm the Podman log driver

First, check which log driver Podman uses on your host, because the default varies by distribution and containers.conf:

If it prints journald, either switch to the file-based driver or jump to the journald alternative at the end of this post. To make k8s-file the default for all containers of this user, set it in ~/.config/containers/containers.conf:

You can also set it per container with podman run --log-driver=k8s-file. Remember that json-file is accepted but silently behaves as k8s-file: do not expect Docker-formatted JSON from it. Containers pick up the driver at creation time, so you will need to recreate any containers that were started with a different driver.

Step 2: Enable the rootless Podman API socket

The metadata enrichment in step 4 needs the Podman API. Enable the user-scoped socket as the user that runs the containers:

The enable-linger call keeps the user's systemd instance, and with it the socket, alive when the user is not logged in. Verify the socket answers Docker-compatible API calls:

You should get a JSON array describing the running containers. This compatibility layer is exactly why a processor named add_docker_metadata will work against Podman later.

Step 3: Collect rootless Podman logs with a Custom Logs integration

In Fleet, add the Custom Logs (Filestream) integration to your agent policy instead of the Docker integration.

Set the file path pattern to the rootless storage location:

Then, in the advanced options, configure the parser so the agent understands the CRI format:

The container parser removes the <timestamp> <stream> <flag> prefix, reassembles partial lines (the P flag marks a message that was split), and stores the original timestamp and stream in the event.

For standalone agents, the equivalent input configuration looks like this:

At this point you should see clean log messages in Discover, but without any container context: no name, no image, no labels. A raw container ID buried in log.file.path is all you have, which makes the data hard to filter and nearly impossible to correlate.

Step 4: Enrich with container metadata using add_docker_metadata

The add_docker_metadata processor enriches our logs with information such as the container name. It extracts a container ID from the log file path, queries the Docker-compatible API for that container, and attaches the container's metadata to each event.

Add it to the integration's processors field (or under processors: in a standalone input):

The match_source_index value deserves an explanation, as it is not a commonly used one. The processor splits the log file path on /, discards the empty leading element, and picks the component at the given index as the container ID.

For the rootless Podman path, the indices work out like this:

Index

Component

0

home

1

<user-id>

2

.local

3

share

4

containers

5

storage

6

overlay-containers

7

<container-id>

8

userdata

The default value of 4 exists because Docker's path is /var/lib/docker/containers/<container-id>/..., where index 4 lands on the ID. However, as you can see from the table above, in our case the proper value is 7.

Note that the correct index depends on the depth of the home directory. /home/<user-id>/... puts the ID at index 7, but a nonstandard home location shifts it. Count the components of your actual path to verify whether 7 is the right value for you.

Once the ID resolves correctly, the processor calls the Podman socket and each event gains the familiar container.* fields. Here is a real document from a Jellyfin container managed with podman-compose:

The metadata is more complete than it first appears: alongside container.id and container.image.name, every label on the container arrives too, including the com_docker_compose_* and io_podman_compose_* labels set by podman-compose and the standard org_opencontainers_image_* labels baked into the image. Filtering all logs of a compose project is now one query on container.labels.com_docker_compose_project.

Step 5: Validate the rootless Podman log pipeline

Generate some traffic and check Discover:

A healthy pipeline produces events where:

• message contains only the application log line, with no CRI prefix.
• @timestamp matches the timestamp Podman wrote.
• log.file.path points into overlay-containers.
• container.id is the full 64-character ID, and container.name and container.image.name are populated.

If metadata is missing but messages parse fine, test the socket with the curl command from step 2 and re-check match_source_index. If messages still carry a timestamp prefix, the CRI parser is not applied; verify the parser YAML made it into the integration policy.

Alternatives for collecting rootless Podman logs

The file-based approach above yields the same ECS container.* fields as the Docker integration, but three alternatives are worth weighing first.

Approach

Metadata

ECS container.*

Throughput

Best when

Custom Logs + add_docker_metadata

Name, image, and all labels from the Podman socket

Native

High, plain file tailing

You want the same fields the Docker integration produces

journald log driver

Container ID, name, and image as journal fields

Needs a rename step

Lower, rate limited by RateLimitBurst

Containers run as systemd services via Quadlet

Docker integration on the compat socket

None for logs

Metrics datasets only

Not applicable to logs

You only need metrics, validated per dataset

OpenTelemetry Collector (filelog)

container.id via a regex operator, names and images need custom work

Partial

High, plain file tailing

You are standardizing on EDOT

Collect Podman logs with the journald log driver

Podman integrates with systemd natively, and on many distributions journald is already the default log driver. With --log-driver=journald, Podman writes each log line to the journal and attaches CONTAINER_ID_FULL, CONTAINER_NAME, and the image name as structured journal fields.

You can collect these with the Custom Journald logs integration. The metadata comes for free, with no socket, no path counting, and no match_source_index.

However, journald applies rate limiting that can drop bursts from chatty containers unless you raise RateLimitBurst. Throughput is lower than plain file tailing. The container fields also arrive with journald's naming, so you need a rename step (an ingest pipeline or processors) to get ECS-style container.* fields. Rootless containers also write to the per-user journal rather than the system one. An agent running as root still sees those entries, because reading the default journal as root includes user journals, but enable journald persistence (Storage=persistent in journald.conf) if you want container logs to survive a reboot.

If your containers are managed as systemd services via Quadlet, this is a natural fit.

Point the Docker integration at the Podman compat socket (metrics only)

Since Podman exposes a Docker-compatible API, it is tempting to point the Docker integration's host at unix:///run/user/<uid>/podman/podman.sock.

• Metrics datasets can work, because they only talk to the API.
• Logs do not, because the integration reads Docker's JSON log files from Docker's paths, and rootless Podman uses neither. Treat this as a partial option for metrics, and validate each dataset you enable.

Collect Podman logs with the OpenTelemetry Collector

If you are standardizing on OpenTelemetry, for instance by using EDOT (Elastic Distributions of OpenTelemetry) the Collector's filelog receiver reads the same files with its container operator, which auto-detects CRI-style formats:

Set add_metadata_from_filepath: false because that option expects Kubernetes pod log paths, which don't match Podman's layout.

Here you will miss out on some data enrichment: there is no Podman equivalent of the k8sattributes processor, so container names and labels are not attached automatically.. You can extract container.id from the file path with a regex operator, but resolving it to names and images requires custom work. The contrib Collector does include a podman_stats receiver for container metrics over the same socket, so a Collector-based setup covers metrics well and logs with reduced metadata.

Parsing application logs from Podman containers

The Custom Logs pipeline above ships raw log lines without application-level parsing. For instance, Nginx logs will end up in your cluster plainly, without proper processing. If you'd like to enable proper processing, you can use Ingest Pipelines to "redirect" specific containers (e.g. nginx) to the Ingest Pipeline defined by the Elastic Integration, and get full parsing of that data "for free".

Root cause analysis on Podman logs with Agent Builder

Was it worth the effort? Absolutely! Getting clean, metadata-rich container logs into Elasticsearch lets an LLM investigate issues against the data directly, and Elastic Agent Builder is the quickest way to see that in action.

Agent Builder, available on Elastic Cloud Serverless and the Enterprise tier for Elastic Cloud Hosted and self-managed, provides a chat interface in Kibana, backed by an LLM, with built-in tools to explore your indices and run ES|QL queries against them.

Here is a real example from the host we just configured. Among other containers, it runs Jellyfin, a media server, as a rootless Podman container. One day, playback of a title fails instantly on an Android TV client: the screen goes black and drops back to the menu, with no error apart from a cryptic "Playback error".

In Discover, filtering on container.name: "jellyfin" narrows the view to a few hundred verbose .NET log lines around the playback attempts. The histogram shows a burst of activity at each failed attempt, but the actual cause is buried somewhere in the noise.

Instead of scanning the messages manually, open the Agent Builder panel directly from Discover and ask it to investigate the issue. The agent queries the log data stream on its own, reconstructs the timeline, and returns a diagnosis:

In this run it worked out that the client requested playback six times. Jellyfin tried to convert a 4K Dolby Vision/HEVC video to H.264 through VA-API hardware transcoding while burning in a subtitle stream, and FFmpeg exited with code 187 each time before producing the first HLS segment. That is why playback stopped at zero milliseconds. It also stated its uncertainty honestly: the available logs don't contain FFmpeg's stderr, so it proposed concrete steps to discriminate between a broken transcoding combination and missing GPU access, including checking whether the container can reach the render device /dev/dri/renderD128.

The pipeline we just built made each step of this investigation possible. The agent can slice by container.name only because add_docker_metadata resolved it from the Podman socket, the message field is queryable because the CRI parser stripped the prefixes, and the timeline is trustworthy because @timestamp comes from Podman, not from ingestion time. Without that work, the LLM would face a pile of prefixed raw lines, likely resulting in worse performance and higher token usage.

Conclusion: three settings for rootless Podman logs

Rootless Podman is Docker-compatible where it matters the most: the API. For log collection it uses its own format, file paths, and socket location, which is what the Elastic Docker integration reads.

Once you know that, the fix is three configuration decisions: parse cri instead of Docker JSON, tail the user's overlay-containers storage, and point add_docker_metadata at the per-user Podman socket with a corrected match_source_index.

The same recipe applies to any rootless user on the host; only the home path and UID in the socket change.

If you want to try this workflow end to end to collect your Podman logs, spin up an Elastic Cloud trial, install the Elastic Agent on your host, and you can go from raw CRI files to fully enriched container logs in one integration policy. If you want to push this further, from a one-off chat to automated investigations that open cases with evidence attached, see automated root cause analysis with Elastic Agent Builder.

Those traces show how the agent reached its recommendation and what it cost. They do not record who approved it. Below: how to read the traces, scope the three identities a run touches, and append the approval decision to a data stream the pipeline cannot rewrite.

Prerequisites

• Elastic Stack 9.5 or Serverless
• Privileges to manage Kibana advanced settings, needed to install the traces dashboard.

Where Elastic Observability records each part of an AI agent action

Four questions come up in every review of an agentic operations pipeline, and each one is answered by a different record.

| Question | Where the answer lives | Who creates it | | :---- | :---- | :---- | | How did the agent reach its recommendation? | traces-agent_builder.otel-* spans | Agent Builder, automatically | | Which tools did it call, and did they fail? | execute_tool spans in the same data stream | Agent Builder, automatically | | Whose privileges did the run execute with? | Workflow execution record and Elasticsearch security audit logs | Kibana, partly | | What did a human decide, and did the action run? | An index you write to yourself | You |

The first two are new in 9.5 and cost nothing but a toggle. The last one has no automatic source, so it is the one most pipelines are missing.

The scenario: a stale pricing cache in checkout

Three checkout-service workers serve production traffic. One of them, checkout-worker-1, was rolled to version 2026.07.26.1 and now returns HTTP 500 on every quote because its pricing cache stopped refreshing. The other two stay on 2026.07.25.3 and serve normally.

The SRE control plane pattern behind this setup connects telemetry, an Agent Builder agent that reasons over it, and Elastic Workflows that run known actions, with a human approval gate before the first step that changes production.

Telemetry arrives through the documented OpenTelemetry path, so the agent reads standard OTel fields. Elasticsearch 9.5 exposes a native OTLP endpoint, which lets an OTel SDK write directly to the cluster with no collector in between:

from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource

provider = LoggerProvider(
    resource=Resource.create({
        "service.name": "checkout-service",
        "deployment.environment": "production",
    })
)
provider.add_log_record_processor(
    BatchLogRecordProcessor(
        OTLPLogExporter(
            endpoint=f"{ES_URL}/_otlp/v1/logs",
            headers={"Authorization": f"ApiKey {API_KEY}"},
        )
    )
)

The endpoint speaks OTLP over protobuf and rejects application/json with HTTP 406, so send it through an SDK or collector rather than hand-built JSON. That writes 450 records to logs-generic.otel-default: 360 healthy events across the three workers and 90 PricingCacheStaleError events from the broken one. The agent is given none of that context and has to find it by querying.

Reading AI agent observability traces in Elastic

Everything Agent Builder records about a run lives in two data streams, and all of it is queryable with ES|QL.

How to turn on AI agent tracing in GenAI Settings

Open Stack Management, then GenAI Settings, and find the Agent Builder Traces section. Collect conversation traces is on by default in 9.5.

Two details on that panel matter more than the toggle. Traces are written to traces-agent_builder.otel-<space-id>, one data stream per Kibana space, with a companion logs-agent_builder.otel-<space-id> for agent-side events. These are ordinary data streams on the standard OTel index templates, not hidden system indices, so Discover, Lens, and ES|QL query them directly.

The callout states the access model plainly: anyone who can read the index can read every trace in it. Trace access is not scoped per user, so restrict the index pattern through a role before granting access to a space with sensitive conversations.

Reading the LLM trace waterfall for one agent run

A single question through the converse API asks the built-in Elastic AI Agent to investigate logs-generic.otel-default, find which pod and version are returning HTTP 500, and propose one bounded action. It answers correctly, naming checkout-worker-1 on 2026.07.26.1 with a 43% error rate against zero errors on the two workers still on 2026.07.25.3.

Select the trace icon under any agent response to open the waterfall.

One question through the converse API produced 24 spans over 43.7 seconds. The structure is an invoke_agent root, a generate_title side branch, then alternating chat and execute_tool spans as the agent queries, reads the result, and picks the next query.

Three span families carry everything you will aggregate on.

| Span name prefix | What it represents | Key attributes | | :---- | :---- | :---- | | invoke_agent | A conversation round (CHAIN) or an agent execution (AGENT) | elastic.inference.span.kind, gen_ai.agent.id | | chat | One model call | gen_ai.request.model, gen_ai.provider.name, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens | | execute_tool | One tool invocation | gen_ai.tool.name, gen_ai.tool.call.id, status.code |

The token breakdown for that run:

| Model | Calls | Input tokens | Output tokens | | :---- | ----: | ----: | ----: | | anthropic-claude-4.6-sonnet | 5 | 113,315 | 2,039 | | anthropic-claude-4.5-haiku | 5 | 47,211 | 651 |

Half the model calls went to the smaller model. That is fast model routing, which sends low-effort steps to a cheaper model, and the split is only visible in the trace.

The single agent investigation consumed roughly 160,000 input tokens in total. Each round replays the accumulated context, so cost scales with conversation length rather than with the length of the question.

How do you query agent tool calls with ES|QL?

Tool invocations are the part of agent behavior most worth watching, because that is where the agent touches your data. Every call is one execute_tool span, and the documented query aggregates them directly:

FROM traces-agent_builder.otel-*
| WHERE span.name LIKE "execute_tool *"
| STATS calls = COUNT(*),
        errors = COUNT(*) WHERE status.code == "Error",
        avg_ms = ROUND(AVG(duration) / 1000000.0, 1)
  BY tool = attributes.gen_ai.tool.name
| SORT calls DESC

Across that run the agent leaned on platform.core.execute_esql, with platform.core.generate_esql and load_skill behind it at two calls each. duration is in nanoseconds on the root of the document, which is why the query divides by a million for milliseconds.

Check the companion logs data stream as well. During a different run in the same session, the agent tried to call a tool that was not in its available set, and the attempt was recorded as an exception event in logs-agent_builder.otel-default, correlated to the trace by trace_id:

{
  "trace_id": "3f8b9722dbd371ac4b7ad75e4bed13b6",
  "event_name": "exception",
  "attributes": {
    "exception.type": "toolNotFoundError",
    "exception.message": "Tool \"platform.streams.query_documents\" called but was not available"
  }
}

A blocked tool attempt is audit-relevant, and it does not appear in the trace waterfall. Querying only the traces data stream will miss it.

For aggregate views there is a managed dashboard, installed per space from the same settings panel.

Over a fifteen-minute window covering these runs, it reported 903,914 input tokens, 11,734 output tokens, and 44 LLM requests, with 35 tool spans at a 100% success rate and 0.42 seconds average duration. The dashboard is managed and read-only, so duplicate it to change a panel, and Elastic can still ship improvements to the original.

What OpenTelemetry LLM traces do not capture by default

By default, a trace records structure and cost, not content. Six toggles under Advanced privacy settings control prompts, responses, tool call details, system prompts, real tool and agent names, and real conversation and workflow IDs, and all six are off.

By default, a trace records structure and cost, not content. Select any span in the waterfall and the detail panel says so directly: "No input/output data available for this span."

Identifiers are hashed rather than dropped. The run with the blocked tool call returned conversation be30fb53-d351-4fa1-b5e1-a569816f85d9 from the API, but its spans carry gen_ai.conversation.id: b1141340d0851a46. That lets you group every span belonging to one conversation and compare conversations against each other, without exposing an identifier that ties back to a user's session.

The consequence is that you cannot join traces to conversations on the conversation ID unless you enable real IDs, and you rarely need to. The converse API hands you the correlation key directly:

{
  "conversation_id": "be30fb53-d351-4fa1-b5e1-a569816f85d9",
  "trace_id": "3f8b9722dbd371ac4b7ad75e4bed13b6",
  "model_usage": {
    "llm_calls": 22,
    "input_tokens": 518251,
    "output_tokens": 6423,
    "model": "anthropic-claude-4.6-sonnet"
  }
}

Store that trace_id in your own decision record, and the join works without weakening the privacy defaults. Enable real IDs only when exact response-to-decision attribution is required, and restrict the trace index in the same change.

Auditing AI agent actions beyond the trace

The traces stop at what the agent did, so the records that show who authorised it have to come from somewhere else.

Whose privileges does an AI agent action run with?

Three identities are involved in an agentic pipeline, and each has its own boundary.

| Identity | Runs with | Determined by | How to scope | | :---- | :---- | :---- | :---- | | Agent Builder tools | The privileges of whoever is chatting | The current user, so two people can get different data from the same question | Roles, as described in Agent Builder permissions | | Workflow steps | One stored API key shared by every elasticsearch.* and kibana.* step | The trigger: manual runs use the person who started them, scheduled runs use whoever last saved the workflow | Workflow authorization | | Trace readers | Index-level access, all or nothing | A role grant on traces-agent_builder.otel-* | A role boundary on the trace index pattern |

One consequence of the stored key belongs in any review. Deactivating a user or changing their role does not refresh it, and the workflow keeps running with the privileges it captured until someone saves it again or toggles Enabled off and back on. Revoking an engineer's access does not, by itself, stop workflows that still run as them.

Scope the investigation role to reads only:

POST /_security/role/agent-builder-observability-investigator
{
  "cluster": ["monitor_inference"],
  "indices": [
    {
      "names": ["logs-*", "metrics-*", "traces-*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ]
}

Reading the agent's own traces is a separate grant, needing read and view_index_metadata on traces-agent_builder.otel-*. Keep the two roles apart, because the people who investigate incidents and the people who audit the agent are not always the same people.

Recording who approved an agent action in Elasticsearch

Run the workflow, and it stops at the approval gate, where the reviewer sees the agent's structured output rendered into the request rather than a bare confirmation prompt.

The execution record is detailed. It captures resumedAt, resumedBy, the full resumeInput payload, per-step token usage, and a deep link back to itself.

That execution history is an operational view, not an audit store. The underlying .workflows-events data stream is reserved for system operations and rejects user queries outright, so you cannot run ES|QL across a quarter of decisions, and execution history is subject to retention rather than to your compliance policy.

Write the decision to an index you control:

- name: review
  type: waitForInput
  with:
    message: |
      ## Approve the proposed checkout remediation?

      The evidence query matched {{ steps.collect_evidence.output.hits.total.value }} error events in the last hour.

      Agent classification: {{ steps.investigate.output.structured_output.incident_class }}
      Affected pod: {{ steps.investigate.output.structured_output.affected_pod }}
      Proposed action: {{ steps.investigate.output.structured_output.recommended_action }}
    schema:
      type: object
      properties:
        decision:
          type: string
          enum: ["approve", "decline"]
        reason:
          type: string
          enum: ["supported-by-evidence", "insufficient-evidence", "wrong-target", "unsafe-action"]
        notes:
          type: string
      required: ["decision", "reason"]

- name: record_decision
  type: elasticsearch.index
  with:
    index: "agent-action-audit"
    document:
      "@timestamp": "{{ now | date: '%Y-%m-%dT%H:%M:%S.%LZ' }}"
      "event.action": "agent_recommendation_reviewed"
      "incident.id": "{{ consts.incident_id }}"
      "agent.conversation_id": "{{ steps.investigate.output.conversation_id }}"
      "agent.incident_class": "{{ steps.investigate.output.structured_output.incident_class }}"
      "agent.affected_pod": "{{ steps.investigate.output.structured_output.affected_pod }}"
      "agent.recommended_action": "{{ steps.investigate.output.structured_output.recommended_action }}"
      "agent.evidence_count": "{{ steps.collect_evidence.output.hits.total.value }}"
      "review.decision": "{{ steps.review.output.response.decision }}"
      "review.reason": "{{ steps.review.output.response.reason }}"
      "review.notes": "{{ steps.review.output.response.notes }}"
      "review.responded_by": "{{ steps.review.output.respondedBy }}"
      "workflow.execution_id": "{{ execution.id }}"
      "workflow.executed_by": "{{ execution.executedBy }}"
      "workflow.execution_url": "{{ execution.url }}"

Two details in the workflow snippet above differ from the reference page.

The reviewer payload is nested one level deeper. The docs describe steps.<name>.output.<field>, but the running build returns the submitted values under response, alongside a respondedBy field:

{
  "response": { "decision": "approve", "reason": "supported-by-evidence" },
  "respondedBy": "1506416774"
}

execution.executedBy records who started the run, and respondedBy records who approved the action, which in a human-in-the-loop pipeline are usually different people.

The second detail is the timestamp. {{ now }} renders a JavaScript date string like Sun Jul 26 2026 07:37:11 GMT+0000 (Coordinated Universal Time), which Elasticsearch rejects with failed to parse date field, and execution.startedAt has the same problem. The Liquid date filter fixes it.

The workflow editor also flags steps.review.output.* as an invalid variable before the first run, because the reviewer payload shape is only known once someone responds. The warning clears after the step has real output, and the templates resolve correctly at runtime.

Making the audit data stream append-only

An audit trail the agent's own pipeline can rewrite is not an audit trail. Elasticsearch provides two independent controls, and they compose.

First, write to a data stream rather than an index, because data streams accept appends and nothing else:

PUT _index_template/agent-action-audit
{
  "index_patterns": ["agent-action-audit"],
  "data_stream": {},
  "priority": 500,
  "template": {
    "mappings": {
      "properties": {
        "@timestamp":            { "type": "date" },
        "event.action":          { "type": "keyword" },
        "incident.id":           { "type": "keyword" },
        "agent.conversation_id": { "type": "keyword" },
        "agent.evidence_count":  { "type": "long" },
        "agent.recommended_action": { "type": "keyword" },
        "review.decision":       { "type": "keyword" },
        "review.reason":         { "type": "keyword" },
        "review.responded_by":   { "type": "keyword" },
        "workflow.execution_id": { "type": "keyword" }
      }
    }
  }
}

Second, give the writer create_doc and nothing else, so it can add records but cannot reach for the by-query escape hatches:

PUT _security/role/agent-action-audit-writer
{
  "indices": [
    { "names": ["agent-action-audit"], "privileges": ["create_doc", "auto_configure"] }
  ]
}

Tested against the running cluster, that pair behaves the way an audit store should:

| Attempt as the audit writer | Result | | :---- | :---- | | Append a decision record | 201 Created | | Overwrite a record by ID | 400, only op_type: create is allowed in data streams | | _update_by_query to change a decision | 403, action unauthorized | | _delete_by_query to erase history | 403, action unauthorized | | _search to read the trail back | 403, action unauthorized |

The write-only behaviour in the last row is deliberate. The workflow that writes decisions has no reason to read them, so auditors get a separate read role and the writer stays write-only.

The two controls fail differently, which matters. The 400 comes from the data stream itself and applies to everyone, including a superuser. The 403 rows come from the role, and a superuser could still run them, which is why tamper-resistant retention means shipping records off the cluster the agent's operators administer.

For cluster-level activity, enable Elasticsearch and Kibana security audit logging and forward the logs to a monitoring deployment. On 9.5 xpack.security.audit.enabled became a dynamic cluster setting, so Elasticsearch no longer needs a restart to turn it on, though on orchestrated deployments the logs still have to be shipped somewhere readable.

Query the decision trail with ES|QL

Two runs of the workflow, one approved and one declined, produce two rows you can query alongside everything else in Elastic.

FROM agent-action-audit
| KEEP @timestamp, agent.incident_class, agent.affected_pod, agent.recommended_action,
       agent.evidence_count, review.decision, review.reason, review.responded_by,
       workflow.execution_id
| SORT @timestamp DESC

Both runs saw the same 90 error events and proposed restart-checkout-worker on checkout-worker-1. The first review approved it as supported-by-evidence, and the second declined it as wrong-target, on the argument that restarting the pod hides a pricing-feed problem rather than fixing it.

Because both decisions are structured fields, disagreement between reviews is queryable. You can count rejections per incident class and group them by reason: insufficient-evidence sends you back to the investigation path, and unsafe-action sends you to the workflow and its permission boundary.

AI agent observability limits to design around

Four behaviors are worth designing around, and each is cheaper to handle before the workflows are written.

1. Trace access is index-level, not per user. A space with sensitive conversations needs a role boundary on traces-agent_builder.otel-* rather than a UI setting.
2. The managed dashboard is not installed automatically in a new space. Add it to your space provisioning checklist.
3. The workflow execution carries its own APM traceId. It is not the same trace as the Agent Builder spans its ai.agent step produced, so correlate through the conversation ID or the trace_id returned by the agent rather than expecting one trace to span both.
4. The waitForInput output shape differs from the reference page. The submitted values arrive under response, alongside respondedBy, as covered above.

None of these blocks the pattern.

Where to start with AI agent observability

Turn trace collection on, install the dashboard in the space your agents run in, and open the waterfall for one real conversation. It shows the tool sequence, the model split, and the latency distribution that the answer text does not.

Then pick the single incident class where you already trust the runbook, and add one elasticsearch.index step after its approval gate. An append-only decision record costs one workflow step and answers the three questions a review needs: who approved this, on what evidence, and what happened next.

For the details, see Collect Agent Builder traces, the traces overview dashboard, Agent Builder permissions, workflow authorization, and the waitForInput reference.

In this article, you'll build that gate with Elastic Workflows, reusing the control plane from the companion article Build an SRE Control Plane with Agent Builder and Workflows. This time the incident is narrow enough to act on: a stale pricing cache on a single worker, with a structured approval step sitting between the recommendation and the remediation. Nothing here touches production, so you can run the approved branch and the declined branch and compare what each one leaves in the execution record. That record is how incident response automation earns autonomy one incident class at a time.

Prerequisites

• Elastic Stack 9.4+

What an SRE control plane needs before you add approval

This article builds on Build an SRE Control Plane with Agent Builder and Workflows, which defines the control-plane pattern used here: telemetry, investigation context, policy, and a set of known actions, wired together. Agent Builder reasons over logs, traces, metrics, alerts, runbooks, and previous cases. Elastic Workflows executes a defined sequence with explicit inputs and permissions. Human-in-the-loop, or HITL, connects those two responsibilities at exactly the point where evidence becomes action.

A recommendation and a remediation have very different failure modes. A weak recommendation wastes an engineer's time. A weak remediation changes production. This is why the approval gate belongs inside the execution model.

The five bindings that make an approval gate a reliability control

A useful gate preserves five properties. Remove any one of them and the gate gets weaker.

• Evidence binding: The reviewer sees the logs, alert details, enrichment, or agent rationale that produced the proposal. An "Approve" button without evidence only asks a person to accept the automation's confidence.
• Action binding: The request names the exact bounded action, target, parameters, and expected effect. A request without an explicit target can authorize far more than the reviewer intended.
• Identity binding: The record shows who approved or declined the request, and when. Without it, you have an outcome but no accountability.
• Time binding: The decision has a deadline, and stale requests fail closed instead of running later without context. An approval without a timeout can outlive the incident evidence that justified it.
• Outcome binding: The execution record shows what ran, what was skipped, and whether post-action verification passed.

The form still matters, but the form is only the human-facing part of a much larger control. Approval design is reliability engineering.

The four stages from AI recommendation to auto remediation

Treat autonomy as a sequence of operational states rather than a product toggle.

| Autonomy stage | System behavior | Human responsibility | Promotion evidence | | :---- | :---- | :---- | :---- | | 0. Observe | Search logs and assemble context. | Investigate and act manually. | Queries find the right incident evidence. | | 1. Recommend | Propose one bounded next step with rationale. | Decide and execute outside the workflow. | Recommendations are accurate enough to review quickly. | | 2. Approve | Pause before action and resume only with structured input. | Approve or decline the exact proposal. | Approval quality, execution success, and rollback behavior are measured. | | 3. Automate narrowly | Run the same action automatically for a proven incident class. | Review exceptions and audit samples. | Scope, permissions, timeout, verification, and rollback remain enforced. |

The important transition is from stage 1 to stage 2. That is where the system stops being an advisor and gains an execution path. That path should be deterministic even when an AI agent contributed to the investigation: the agent summarizes evidence and recommends an action, while the workflow owns the pause, the structured decision, the branch, and the execution record.

Build the human-in-the-loop automation gate in Elastic Workflows

Elastic Workflows gives you the waitForInput step for this. When execution reaches that step, the workflow stops in the WAITING_FOR_INPUT state and waits for a person. The reviewer answers a small form in the Kibana execution view (or through the resume API), and whatever they submit becomes available to later steps at steps.<step_name>.output.

The workflow pulls the last 30 minutes of checkout-api failure logs, hands them to Agent Builder for a root-cause analysis, and writes that analysis into an Observability case. Then it stops and asks one question: should we clear the pricing cache on checkout-worker-07? If you approve, it records the simulated action, runs a verification query, and appends the result to the case. If you decline, it records the decision and runs nothing. Either way, production is untouched.

name: obs-labs-checkout-control-plane-hitl
description: Human approval gate for the OpenTelemetry-grounded checkout control plane.
enabled: false
tags:
  - sre-control-plane
  - agent-builder
  - human-in-the-loop
  - workflows
  - opentelemetry

settings:
  timeout: "30m"

triggers:
  - type: manual

consts:
  incident_id: "obs-labs-checkout-hitl-20260719"

steps:
  - name: collect_evidence
    type: elasticsearch.search
    with:
      index: "logs-*"
      size: 10
      query:
        bool:
          filter:
            - range:
                "@timestamp":
                  gte: "now-30m"
            - term:
                "service.name": "checkout-api"
            - term:
                "attributes.incident.id": "{{ consts.incident_id }}"
            - match_phrase:
                "body.text":
                  query: "checkout failed: stale pricing cache"

  - name: rca_analysis
    type: ai.agent
    agent-id: elastic-ai-agent
    create-conversation: true
    with:
      message: |
        Investigate the checkout-api incident identified by {{ consts.incident_id }}.
        The workflow evidence query found {{ steps.collect_evidence.output.hits.total.value }} matching OpenTelemetry log events in the last 30 minutes.
        Search logs, traces, and metrics for service.name checkout-api and incident.id {{ consts.incident_id }}.
        Identify the affected worker, deployment version, error type, HTTP status, and latency evidence.
        Return the likely cause, supporting evidence, confidence, and whether the bounded simulated action below is consistent with the evidence.
        Proposed simulated action: simulate clearing the pricing cache on checkout-worker-07.
        This workflow must not change production.

  - name: case_title
    type: ai.agent
    agent-id: elastic-ai-agent
    with:
      conversation_id: "{{ steps.rca_analysis.output.conversation_id }}"
      message: "Produce a short case title for this checkout incident. Output only the title."

  - name: case_description
    type: ai.agent
    agent-id: elastic-ai-agent
    with:
      conversation_id: "{{ steps.rca_analysis.output.conversation_id }}"
      message: "Produce a concise case description grounded in the OpenTelemetry evidence. Include the incident ID and say that remediation requires human approval. Output only the description."

  - name: create_case
    type: cases.createCase
    with:
      title: "{{ steps.case_title.output.message }}"
      description: "{{ steps.case_description.output.message }}"
      owner: "observability"
      severity: "medium"
      tags:
        - sre-control-plane
        - agent-builder
        - human-in-the-loop
        - opentelemetry

  - name: add_agent_analysis
    type: cases.addComment
    with:
      case_id: "{{ steps.create_case.output.case.id }}"
      comment: |
        ## Agent Builder RCA and proposed action

        Evidence query matches: {{ steps.collect_evidence.output.hits.total.value }}

        {{ steps.rca_analysis.output.message }}

        Proposed simulated action: simulate clearing the pricing cache on checkout-worker-07.
        No action has run yet.

        Agent conversation: {{ kibanaUrl }}/app/agent_builder/conversations/{{ steps.rca_analysis.output.conversation_id }}

  - name: review
    type: waitForInput
    with:
      message: |
        Approve the bounded checkout response?

        Incident: {{ consts.incident_id }}
        Evidence: {{ steps.collect_evidence.output.hits.total.value }} matching checkout failure logs in the last 30 minutes.
        Target: checkout-worker-07
        Action: simulate clearing the pricing cache on this worker only.
        Expected effect: subsequent checkout requests no longer use the stale cache.
        Blast radius: one worker. This simulated step does not change production.

        Review the Agent Builder analysis in case {{ steps.create_case.output.case.id }} before deciding.
      schema:
        type: object
        properties:
          approved:
            type: boolean
            title: "Approve the simulated cache clear"
          notes:
            type: string
            title: "Reviewer notes"
        required:
          - approved

  - name: approved_action
    type: console
    if: "steps.review.output.approved : true"
    with:
      message: "Approved. Simulated cache clear recorded for checkout-worker-07. Reviewer notes: {{ steps.review.output.notes }}"

  - name: verify_after_approval
    type: elasticsearch.search
    if: "steps.review.output.approved : true"
    with:
      index: "logs-*"
      size: 0
      query:
        bool:
          filter:
            - range:
                "@timestamp":
                  gte: "now-5m"
            - term:
                "service.name": "checkout-api"
            - term:
                "attributes.incident.id": "{{ consts.incident_id }}"
            - match_phrase:
                "body.text":
                  query: "checkout failed: stale pricing cache"

  - name: record_approved
    type: cases.addComment
    if: "steps.review.output.approved : true"
    with:
      case_id: "{{ steps.create_case.output.case.id }}"
      comment: |
        ## Human decision: approved

        Reviewer notes: {{ steps.review.output.notes }}
        Simulated action target: checkout-worker-07
        Verification query matches in the last 5 minutes: {{ steps.verify_after_approval.output.hits.total.value }}

        This walkthrough did not change production.

  - name: record_declined
    type: cases.addComment
    if: "steps.review.output.approved : false"
    with:
      case_id: "{{ steps.create_case.output.case.id }}"
      comment: |
        ## Human decision: declined

        Reviewer notes: {{ steps.review.output.notes }}
        No action ran.

  - name: declined_console
    type: console
    if: "steps.review.output.approved : false"
    with:
      message: "Declined. No action executed. Reviewer notes: {{ steps.review.output.notes }}"

The collect_evidence and rca_analysis steps keep the same evidence-then-reasoning sequence as the first article, and the workflow writes that analysis into the case before it ever reaches the waitForInput boundary. By the time a human is asked to decide, the reasoning is already durable and linkable.

The approval form asks for one boolean and accepts optional notes.

Only the approved branch records the simulated cache clear, runs a bounded verification query, and appends the result to the case. The declined branch records the decision and runs nothing.

The approval request carries the resulting count into the decision, while the linked case retains the full Agent Builder analysis. Treat that count as evidence for the reviewer, not as a root-cause claim or a performance measurement.

Replace the simulated action with real auto remediation

The walkthrough keeps the approved branch as a console step so you can run both branches safely. In production you replace only that step with a narrowly scoped external action, and leave the case, timeout, verification, and decline path exactly as they are.

You have two options, depending on how the target system is reached.

Call an internal remediation API with an HTTP connector

Configure an HTTP connector in Kibana with the base URL, authentication, and any encrypted headers, then reference it by connector-id. Secrets stay in the connector, never in the workflow YAML.

  - name: clear_pricing_cache
    type: http
    connector-id: "checkout-remediation-api"
    if: "steps.review.output.approved : true"
    with:
      path: "/v1/cache/pricing/purge"
      method: "POST"
      body:
        worker: "checkout-worker-07"
        incident_id: "{{ consts.incident_id }}"

Hand off to Jira, Slack or PagerDuty

Kibana connectors are available as workflow steps, so the approved branch can open a jira ticket for a change-managed action, post to slack for the on-call channel, or page through PagerDuty, all using credentials your team already manages centrally.

  - name: notify_oncall
    type: slack
    connector-id: "sre-oncall-channel"
    if: "steps.review.output.approved : true"
    with:
      message: "Approved by {{ steps.review.output.notes }}: pricing cache cleared on checkout-worker-07 for {{ consts.incident_id }}."

Whichever you choose, keep the action bound.

How to design a human-in-the-loop automation gate

The workflow above shows the mechanics. Getting the gate right is a design problem: where the pause goes, what the request tells the reviewer, what happens when nobody answers, and what the execution record has to preserve. Each one maps back to one of the five bindings.

Where should the approval step go in an incident response workflow?

The best place for waitForInput is immediately before the first step that increases impact. Don't pause before gathering evidence, because the workflow can usually search, enrich, classify, and open a draft case without touching the affected service. And don't pause after the remediation, because that just asks a person to ratify something that already happened.

Before anyone enables the workflow, a reviewer can inspect the evidence query, the form schema, the timeout, the branch condition, and the action itself.

What a good approval request tells the reviewer

An on-call engineer should not have to reconstruct the investigation from five other screens. Lead with the decision and include only the evidence needed to make it. A strong approval request answers these questions, in this order:

1. What exactly am I deciding?
2. What telemetry supports the proposal?
3. What target and parameters will the action use?
4. What is the expected effect and blast radius?
5. What happens if I decline or do nothing?

One required decision plus optional notes is usually enough. If the reviewer has to type in service names, host identifiers, environment, and action parameters, the proposal wasn't specific enough before the pause.

Paused executions stay discoverable in history and can be resumed by any authorized reviewer, which brings up a queueing requirement. Once a team has more than a few paused executions, reviewers need an inbox or an equivalent filtered view showing pending decisions, age, owner, severity, and target. Otherwise, a safe pause quietly becomes an invisible backlog.

What happens if nobody approves in time

waitForInput has no default timeout; the execution waits indefinitely. The settings.timeout field caps the entire execution, including time spent waiting for input, and in this workflow the 30-minute value limits how long the proposal stays actionable after the evidence was collected.

Choose that value from the incident and the action; a traffic shift during an active outage may need a decision within minutes. A maintenance approval may stay valid for hours. Confirm the timeout behavior on the exact Elastic version you operate, and add an external escalation or cancellation path if what you observe doesn't meet your fail-closed requirement.

Whatever you do, don't convert silence into approval.

What the audit record must capture

The execution history should answer four questions without anyone digging through chat history:

• What evidence did the workflow collect?
• What exact input did the reviewer submit?
• Which branch ran?
• What did the action and verification steps return?

A paused execution shows the evidence step and the decision still pending. An approved execution adds the action branch and the verification output to that same view. A declined execution preserves the same evidence and the same decision while skipping every action step. Keeping the walkthrough action simulated lets you inspect both branches without changing production.

For longer-lived incident context, push it into the case. Add the evidence summary, reviewer notes, action result, and verification result as comments, and the case becomes the durable record that outlives the execution.

How to know when a workflow is ready to run without approval

Don't remove the approval gate because a handful of runs succeeded. Review enough executions to understand both the normal and the exceptional paths, and measure at least these outcomes:

| Measure | Question it answers | | :---- | :---- | | Proposal acceptance rate | Does the workflow recognize the right incident class? | | Reviewer edits or declines | Which evidence or action parameters are still wrong? | | Approval age | Can the team respond before evidence becomes stale? | | Action success rate | Does the bounded action execute reliably? | | Verification success rate | Did the service improve after the action? | | Rollback rate | How often did the response create a new problem? |

Autonomous execution is reasonable only when the incident class, target selection, action, verification, permissions, timeout, and rollback path are all narrow and repeatable. Even then, keep the same workflow structure. Automation should bypass the human wait for the proven path, not bypass evidence collection, authorization, verification, or audit records. Route anything uncertain back to human review.

Conclusion

The approval gate is a small amount of YAML, but it changes what the automation is. waitForInput turns the decision into a structured input the branch logic depends on, so the action, the verification, and the case comment all exist because a named person answered a specific question at a specific time. Placing that pause right before the first step that increases impact, and capping it with settings.timeout, is what makes it a reliability control instead of a confirmation dialog.

Taking this to production is a smaller change than it looks: the simulated console step becomes an http, slack, or jira connector step, and everything else stays as it is. From there you earn autonomy one incident class at a time, letting acceptance rate, approval age, action success, verification success, and rollback rate tell you when a path is proven enough to run without the wait.

Where to start with human-in-the-loop automation

Start with one recurring operational signal and one reversible response. Build the evidence query first, then add a recommendation that names the target and the expected effect, then insert waitForInput immediately before the action and run the workflow in a lab or case-only mode. Review the execution history with SREs, developers, support, security, and the product owners of the affected service.

The approval gate is not the destination. It is the mechanism that lets a team move toward narrow autonomy without giving up evidence, accountability, or control.

• Human-in-the-loop workflow guide
• waitForInput flow-control reference
• External systems and apps steps
• Kibana connectors reference
• Workflow monitoring guide

In this article, we'll build that loop end to end. We'll use Elastic Agent Builder to feed logs, traces, metrics, alerts, and runbook context into an agent that investigates a problem, and Elastic Workflows to execute the known next steps: opening a case, sending a notification, running an enrichment query, or triggering a remediation path.

We'll work through a checkout latency regression as our example, but the same pattern applies to any incident class where your team already knows the manual steps.

Prerequisites

• Elastic Cloud or self-managed cluster running 9.4+

We'll use a checkout latency regression as the running example. If you want to follow along against your own telemetry, point the queries at your service instead. If you'd rather reproduce the exact incident, the supporting notebook simulates it and sets up the role, skill, tool, and workflow for you.

Why dashboards are not enough for incident response

Dashboards show the symptom but cannot choose the next query. A dashboard is still one of the best tools for shared situational awareness, and during an incident the hard work starts after the chart turns red and the engineer still needs to answer a sequence of operational questions.

• What changed? You need access to related deploys, alerts, logs, traces, and metrics from the same time window.
• What is affected? You need visibility into services, hosts, users, regions, SLOs, and dependency paths.
• What is the likely cause? You need evidence from telemetry combined with runbooks or previous incident cases.
• What is safe to do next? You need a bounded action that includes proper permissions, an audit trail, and a rollback path.

That last question is where a dashboard stops. It can show the symptom, but it cannot decide which query to run next, which runbook applies, or which workflow should run. Engineers provide that judgment today by moving between Kibana, chat, tickets, terminals, and internal docs.

An SRE control plane keeps the judgment with the engineer while moving more context and more action into the same operational surface.

How automated root cause analysis works: state, policy and action

Automated root cause analysis needs three things in one place: the telemetry, the permissions that bound it, and the actions it can trigger.

• State: For SRE work, that state is telemetry in Elasticsearch: logs, traces, metrics, alerts, SLOs, and related operational records.
• Policy: Policy defines who can query which data, which tools an agent can call, which workflows can run, and where a human decision is required.
• Action: Action is a set of known tools and workflows that run with explicit inputs, permissions, and outputs.

Agent Builder is useful where the system needs reasoning over messy context, and Workflows are useful where the system needs deterministic execution.

The two can work in both directions; a workflow can call an agent with an ai.agent step when it needs analysis before the next step, and an agent can call a workflow through a workflow tool when a conversation needs a repeatable action.

What Elastic Agent Builder adds to AI incident response

Agent Builder skills are reusable capability packs. A skill can include instructions, tools, and context that guide an agent through a specific task.

Reusable skill packs matter for SRE work because incident response is rarely a single query. A good investigation has a shape, and root cause analysis is a good example. The useful unit is not "ask the model what happened." It's a repeatable investigation path that starts from an alert, scopes the time window, checks the right telemetry, records uncertainty, and hands a case or workflow a structured result. The agent needs to decide which signal to start from, query the right index, compare the right time windows, inspect related services, and explain the evidence without hiding what it doesn't know.

Elastic includes a built-in Elastic AI Agent for this pattern. Built-in skills are scoped by solution, so the one that carries an SRE incident loop is observability.investigation, alongside platform skills such as dashboard-management that any solution can use. The list shows the short name, so look for investigation in the UI.

The skill ships as Markdown instructions, the same format we use for our own skill in the next section.

There are also out-of-the-box tools such as platform.core.search, platform.core.get_document_by_id, platform.core.get_index_mapping, platform.core.list_indices, platform.core.get_workflow_execution_status, and platform.core.resume_workflow_execution.

Skills guide the work, tools perform bounded operations, and the agent chooses what to use based on the task.

The scenario: a checkout latency regression

Deployment 2026.07.09.1 ships a connection pool misconfiguration to checkout-api. Within minutes, p95 latency goes from 180ms to over 2s and HTTP 500s appear for the first time. Nobody knows yet that the pool is the cause.

The evidence is spread across three signals, and no single one answers the question:

| Signal | What it shows | | :---- | :---- | | Logs | PoolExhaustedException and HTTP 500s, only on the new version | | Traces | The payment-gateway span goes from ~180ms to ~2500ms | | Metrics | Connection pool pinned at 20 of 20 right after the deploy |

Correlating those three is the work we want the agent to do. That gives us the contract for the rest of this article:

| Contract | Detail | | :---- | :---- | | Input | Service name and the alert summary | | Access | Read-only search over logs-*, traces-*, and metrics-* | | Output | Likely cause, supporting evidence, confidence, and the next safe action | | Side effect | One Observability case with the analysis attached |

Everything after this point builds one piece of that contract: the skill shapes the investigation, the tool and role bound the access, and the workflow turns the output into a case.

Build a read-only investigation skill in Elastic Agent Builder

Let's start with a read-only skill that improves investigation quality without touching production:

# Checkout latency investigation

Use this skill when an engineer asks why checkout latency, errors, or failed transactions increased.

Work through the investigation in this order:

1. Identify the affected service, environment, and time range.
2. Query traces for the slowest transactions in that window.
3. Query logs for errors from the same service and dependency path.
4. Compare current error and latency rates with the previous healthy window.
5. Return the likely cause, supporting evidence, confidence level, and the next safe action.

Do not recommend a production change unless there is a workflow tool assigned for that action.

If the evidence is incomplete, say what data is missing.

This kind of skill is a runbook execution guide, and it keeps the agent consistent across incidents. It also helps less experienced engineers ask better follow-up questions, because the agent can show the next query and explain why it matters.

Without step 4, the agent describes what's happening now and stops there. Comparing against the previous healthy window is what makes it an analysis. And the last line lets the agent say the data is missing instead of guessing.

Add AI agent observability tools with narrow permissions

Each tool should expose the smallest operation the agent needs, with the smallest data access that still supports the task.

For a read-only investigation agent, the required privileges usually start with searching observability data and inspecting index structure. The Agent Builder permissions documentation calls out that tools run against Elasticsearch data as the current user, and that read-oriented tools need index privileges such as read and view_index_metadata.

Run this in Dev Tools to create an investigation-scoped role:

POST /_security/role/agent-builder-observability-investigator
{
  "cluster": ["monitor_inference"],
  "indices": [
    {
      "names": ["logs-*", "metrics-*", "traces-*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ],
  "applications": [
    {
      "application": "kibana-.kibana",
      "privileges": ["feature_agentBuilder.read", "feature_actions.read"],
      "resources": ["space:default"]
    }
  ]
}

This role gives the agent enough access to inspect telemetry while keeping production-changing actions out of scope. The monitor_inference cluster privilege is what lets the agent use the inference endpoints behind Agent Builder, and it grants no data access on its own.

When you add a custom tool, describe it in operational language, because the tool description is part of how the agent decides when to call it. Prefer descriptions like this:

Use this tool to search checkout service logs for errors in a bounded time range.

Required inputs:
- service_name
- environment
- start_time
- end_time

Return:
- matching log samples
- error counts by message
- affected host and pod names when present

A narrow tool description is much safer than a broad tool that says "search all logs for anything relevant." The agent gets a clear contract, and reviewers can reason about what the tool can and cannot do.

Use Elastic Workflows for incident response automation

Once the investigation path is useful, we can add Workflows for the actions that should be repeatable. With Workflows the control plane becomes operational because it can query more context, ask an agent to summarize evidence, open a case, notify a channel, or call a remediation endpoint. The key is that each step is explicit.

The Workflows editor gives you a validation loop before you save or run anything. Use it to catch syntax issues before the workflow writes to Cases or calls any action.

Go to Workflows > Create workflow and paste the following:

name: obs-labs-checkout-control-plane
description: Checkout regression investigation with Agent Builder and case creation.
tags: ["sre-control-plane", "agent-builder", "workflows"]

triggers:
  - type: manual

inputs:
  - name: service_name
    type: string
    default: "checkout-api"
  - name: alert_summary
    type: string
    default: "Checkout API p95 latency increased above 2s and HTTP 500s rose in the last 15 minutes after deployment 2026.07.09.1."

steps:
  - name: rca_analysis
    type: ai.agent
    agent-id: elastic-ai-agent
    create-conversation: true
    with:
      message: |
        Investigate this checkout incident as an SRE would.

        Service: {{ inputs.service_name }}
        Alert: {{ inputs.alert_summary }}

        Search the available logs, traces, and metrics for this service.
        Compare the window before and after the most recent deployment.

        Return a concise likely cause, supporting evidence, confidence, and next safe action.
        If the evidence is incomplete, say what data is missing.

  - name: case_title
    type: ai.agent
    agent-id: elastic-ai-agent
    with:
      conversation_id: "{{ steps.rca_analysis.output.conversation_id }}"
      message: "Produce a short case title for this incident. Output only the title."

  - name: case_description
    type: ai.agent
    agent-id: elastic-ai-agent
    with:
      conversation_id: "{{ steps.rca_analysis.output.conversation_id }}"
      message: "Produce a concise case description. Output only the description."

  - name: create_case
    type: cases.createCase
    with:
      title: "{{ steps.case_title.output.message }}"
      description: "{{ steps.case_description.output.message }}"
      owner: "observability"
      severity: "medium"
      tags: ["sre-control-plane", "agent-builder", "workflows"]

  - name: add_agent_analysis
    type: cases.addComment
    with:
      case_id: "{{ steps.create_case.output.case.id }}"
      comment: |
        ## Agent Builder RCA

        {{ steps.rca_analysis.output.message }}

        Agent conversation: {{ kibanaUrl }}/app/agent_builder/conversations/{{ steps.rca_analysis.output.conversation_id }}

Each step feeds the next one through its output. ai.agent steps emit a message with the model's text and a conversation_id, and cases.createCase emits the new case.id. Those three fields are the whole contract:

This workflow doesn't restart anything. It asks Agent Builder to investigate, reuses the same conversation to generate the case title and description, creates an Observability case, and writes the agent analysis back as a case comment.

Two details are worth calling out. The create-conversation: true flag on the first step is what makes the next two steps cheap: case_title and case_description pass the same conversation_id, so the agent already has the investigation in context and doesn't repeat the queries. And we use a manual trigger with a default alert_summary so you can run the sequence before attaching it to a live alert rule. In production, you'd switch the trigger to alert and attach the workflow to the rule that owns that incident class.

Run the workflow with the play button. Our run took 3 minutes and 22 seconds, with rca_analysis, case_title, case_description, create_case, and add_agent_analysis all marked as successful. Almost all of that is the investigation itself: rca_analysis alone took 3 minutes and 5 seconds, while the two case writes finished in about a second each.

The workflow then wrote an Observability case. The case list shows one open case with the generated checkout title, our tags, medium severity, and one comment.

The case detail is the audit artifact for the investigation. It records the evidence considered, the affected hosts and deployment version, the likely error type, the agent's confidence, and any missing signals.

A useful operational control plane surfaces the limits of its evidence instead of turning uncertainty into a confident claim. If your agent never reports missing data or a lower confidence, that's a signal to tighten the skill instructions, not a sign that every investigation went well.

The read-only automated root cause analysis pattern improves response quality without changing the affected service. Add remediation only when the action is well understood, narrowly scoped, and paired with verification and rollback: clearing one cache key, restarting one worker, shifting traffic away from one unhealthy instance, or running a pre-approved maintenance task.

Turn Elastic Workflows into agent tools

Workflow tools let an Agent Builder conversation trigger an Elastic Workflow and use its output. This is the bridge from "the agent recommended a next step" to "the agent can offer a known action."

A workflow tool should have a narrow description:

Use this tool only when checkout errors are caused by connection pool exhaustion on a single worker.

The workflow drains and recycles the connection pool for one worker, then verifies that the worker resumes successful requests.

Required input:
- host_name

Do not use this tool for database outages, deploy regressions affecting all hosts, or multi-host failures.

The description matters because it sets the agent's selection boundary. Note how the last line excludes the very scenario we just investigated: our incident hit both hosts and was caused by a deploy, so the agent should not offer this tool. That's the point. A workflow tool that matches every incident is a workflow tool with no boundary.

The workflow still owns execution. The agent doesn't need to know how to recycle the pool. It only needs to recognize when a known workflow may apply, collect the required input, and present the action to the engineer.

How do you stop an AI agent from changing production?

An SRE control plane should be built around blast-radius control, which means every action path needs a clear boundary. Use these checks before exposing a workflow as an agent tool:

| Check | Importance | | :---- | :---- | | Read-only first | Proves the investigation path before adding production action | | Narrow input schema | Prevents vague prompts from becoming vague actions | | Explicit permissions | Keeps the agent limited to the current user's allowed data and actions | | Dry-run or case-only mode | Lets teams review outputs before enabling remediation | | Human review for risky steps | Keeps judgment in the loop where impact is high | | Post-action verification | Confirms that the workflow improved the service instead of only executing a command |

For the review boundary itself, Workflows gives you wait steps, timeouts, and execution history, so a risky path can pause for an approval and still leave an audit trail.

An agent can help gather evidence and propose the next step, but production action should stay inside known workflow paths.

Validate against one incident class first

For a real rollout, validate the control plane against one recurring incident class. Track whether the agent finds the right evidence, whether the workflow output is complete enough for review, and whether engineers trust the recommended next step.

Use a simple validation plan:

1. Pick one alert type with a known runbook.
2. Build a read-only investigation skill for that alert.
3. Add one or two query tools with scoped index permissions.
4. Run the agent against historical incidents and compare its summary with the actual case notes.
5. Add a case-creation workflow and review the output with the owning SRE team.
6. Only then consider a workflow tool that performs a bounded remediation step.

The main failure mode is not that the model gives an imperfect summary. It's granting broad action before the investigation path is proven. Keep the first version boring, scoped, and reviewable.

Conclusion

What we covered:

• An SRE control plane combines state (telemetry in Elasticsearch), policy (permissions and review boundaries), and action (known tools and workflows).
• Agent Builder handles reasoning over messy context, while Workflows handles deterministic execution, and the two can call each other.
• A read-only investigation skill turns a runbook into a repeatable investigation path that records uncertainty instead of hiding it.
• Scoped roles with read and view_index_metadata on logs-*, metrics-*, and traces-* keep the agent useful without letting it change production.
• Reusing a conversation_id across ai.agent steps lets later steps build on the investigation instead of repeating it.
• A case-only workflow gives you the full audit artifact before you enable any remediation.
• Tool descriptions are a security boundary, not documentation, because they decide when the agent offers an action.

Resources

• Agent Builder for Observability
• Agent Builder skills
• Agent Builder tools
• Workflow tools
• AI-augmented workflows
• Root cause analysis workflow for observability alerts

What changed in Elastic Agent 9.3: A native OTel Collector under the hood

Previously, Elastic Agent acted as a supervisor process, spinning up Beats-based subprocesses, such as Filebeat or Metricbeat. From 9.3 onward, that architecture has been replaced. Elastic Agent itself is now built on the EDOT Collector, turning it into a first-class OTel Collector under the hood while preserving its original functionality.

Key benefits of this architectural shift include:

• Reduced footprint: Fewer subprocesses mean significantly less memory overhead and a simpler deployment model. In future releases, this footprint will be even further reduced.
• Unified telemetry pipeline: Logs and metrics flow through a single, standards-based OTel pipeline, as do traces.
• Ecosystem interoperability: Elastic Agent can now receive data from any OTel-compatible source out of the box. It can also be configured to emit to OTel-compatible destinations.
• Aligned with the OTel ecosystem: As the OTel ecosystem matures with new receivers, processors, and exporters, Elastic Agent deployments gain access to those capabilities automatically.

When you deploy or update Elastic Agent from version 9.3 onward, you're deploying an OpenTelemetry Collector. EDOT is the technology foundation; Elastic Agent is the product.

How existing Beats configurations run inside the OTel Collector pipeline

Elastic has introduced Beats Receivers, which are Beat inputs and processors that execute natively inside the new OTel Collector pipeline. For your teams and customers, this means:

• Existing elastic-agent.yml configurations require no modification.
• Fleet-managed agents automatically translate policy configurations into OTel format internally.
• All integrations, dashboards, ingest pipelines, and alerting rules continue to function exactly as before.
• Data written via Beats Receivers lands in the same data streams as always.

Upgrading to 9.3 is transparent because it uses the same inputs and produces the same outputs.

Running Beats and OTel Collector pipelines in one Elastic Agent

The new Elastic Agent is a collector capable of simultaneously running traditional Beats-based collections alongside native OTel pipelines, all in a single deployment. One agent policy can collect Elastic Common Schema–schematized (ECS-schematized) data via Beats Receivers and ingest native OpenTelemetry Protocol (OTLP) data from OTel-instrumented applications and infrastructure. This same agent policy can apply OTel processing stages across all telemetry before export.

OTel integrations from Elastic's catalog can be added to any agent policy. When native OTel data is ingested, Elastic automatically installs the relevant dashboards and alerts, in addition to necessary content packs, without any manual setup.

What's the relationship between Elastic Agent and EDOT?

You may be familiar with EDOT, the Elastic Distribution of OpenTelemetry Collector, as a stand-alone product. With this architectural change, EDOT is the technology foundation that now powers Elastic Agent, not a separate product that users need to track or deploy independently.

Going forward, Elastic Agent is the supported, Fleet-manageable, fully featured product. A stand-alone deployment remains available for specific niche scenarios (environments where the full version of Elastic Agent cannot be installed), but it isn't the recommended path for the vast majority of users.

Elastic Agent deployment options: Fleet-managed vs. stand-alone

| | Fleet-managed Elastic Agent | Stand-alone Elastic Agent | | :------------------------- | :-----------------------------: | :-----------------------------------------------------------------------------------------------------: | | Fleet lifecycle management | Yes | Can enroll into Fleet in-field without reinstallation | | Beats Receivers | Yes | Yes | | Elastic Defend | Yes | No | | Cloud Security | Yes | No | | Profiler support | Yes | No | | OTel-native pipeline | Yes | Yes | | Best for | Most deployments | Environments where full Elastic Agent cannot be installed or management is handled by other tools |

Do I need to change anything when upgrading to Elastic Agent 9.3?

For users running Elastic Agent today, upgrading to 9.3 requires no changes to configurations or integrations, and no changes to workflows. For customers evaluating OTel adoption, Elastic Agent now provides a fully supported, production-ready OTel Collector with Fleet management and rich integrations, along with Elastic's full support matrix, and none of this requires a separate OTel deployment.

With Elastic Agent 9.3, Elastic's data collection is fully OpenTelemetry-native. Elastic Agent is now an OpenTelemetry Collector. Everything you have today still works, and you also get all the capabilities of OTel.

Prerequisites for Elastic Agent Builder MCP tools

• An Elastic Cloud deployment with Elastic Stack 9.3+ (or Elastic Cloud Serverless) with Agent Builder enabled.
• An APM-ingested service. If your cluster does not already have APM data, the companion notebook includes instructions to generate synthetic traffic using elastic/apm-integration-testing with the opbeans-node demo app.
• An MCP-compatible client: Claude Code, Cursor, or VS Code with an MCP extension.
• Basic familiarity with ES|QL syntax.
• Node.js 18+ (for the mcp-remote bridge).

If you are new to MCP or need to set up the Elastic MCP server for the first time, check out Connect Agent Builder tools to any AI agent with Elastic MCP server for the full setup walkthrough. This article assumes the MCP server is already configured.

The problem: why developers fly blind

A developer is about to merge a pull request. The change looks simple: increasing the timeout for the downstream recommendations service call from 2 seconds to 5 seconds. But before hitting the merge button, a question lingers: is the service healthy enough to absorb this change right now?

To answer that question today, the developer has two options:

1. Check dashboards manually. Open the APM UI, look at error rates, scan latency charts, find the SLO page, and look for recent deploys. This takes 10-15 minutes and requires knowing what to look for and how to interpret it.
2. Ask an SRE. Ping the platform team on Slack: "Hey, is checkout healthy? I want to merge something." This creates an interruption, adds latency to the decision, and doesn't scale.

The core problem is not the data. Elastic already collects everything: traces, metrics, error logs, deploy markers, and SLO budgets. The problem is that correlating multiple signals requires mental overhead and domain knowledge that most developers don't have.

An SRE knows that a p99 spike after a deploy is normal for 5 minutes, that an error rate under 0.5% is acceptable during a release window, and that merging when the SLO budget is below 20% is risky. That knowledge lives in runbooks, tribal memory, and experience.

What if the platform engineer could encode that knowledge into tools that any developer can query from their editor?

How MCP tools in Elastic Agent Builder encode your runbook

The key insight is this: a tool is not just a query; it is a query plus interpretation. A dashboard shows you a p99 of 450ms. A well-designed tool tells you "p99 is 450ms, which is within normal range for this service, and has been stable since the last deploy 2 hours ago."

The difference is that the tool description carries the domain knowledge. When a platform engineer creates a tool in Agent Builder, they write descriptions like: "Error rate above 1% typically indicates a regression. If this coincides with a recent deploy, the deploy is the likely cause." That description becomes part of the context the AI agent uses when reasoning across multiple tool results.

This is what we mean by contextual AI: the AI agent does not just fetch data; it reasons over it using the interpretation rules that the platform team encoded.

Here is the architecture:

The platform engineer authors the tools once. Every developer on the team benefits from their own editor, without needing to learn ES|QL or understand APM data models.

Setting up the Elastic Agent Builder sample environment

The full end-to-end setup (traffic generation with opbeans-node, deploy annotations, SLO creation, and the three Agent Builder tools) is available as a runnable notebook at this repository: notebook.ipynb. The sections below focus on the ES|QL queries and tool descriptions: the why behind each tool, not the mechanics of posting them.

Building Tool 1: get_endpoint_health

This tool answers the question: "How is this endpoint performing right now?" It returns error rate, latency percentiles (p50, p95, p99), and throughput for a given service and endpoint within a time window.

Here is the full tool configuration as created in Agent Builder:

{
  "id": "get_endpoint_health",
  "type": "esql",
  "description": "Returns the current health of a service endpoint: error rate, latency percentiles (p50/p95/p99), and throughput. Use this tool to assess whether a service is healthy before making changes. Interpretation guide: error rate below 0.5% is healthy, 0.5-1% is elevated (check for recent deploys), above 1% indicates a problem. For latency, compare p99 against the service baseline: checkout is typically under 500ms, product-search under 200ms. A sudden p99 spike within 15 minutes of a deploy suggests the deploy caused a regression.",
  "tags": ["apm", "reliability", "health"],
  "configuration": {
    "query": "FROM traces-apm-* | WHERE service.name == ?serviceName AND @timestamp >= NOW() - ?timeWindow AND transaction.duration.us IS NOT NULL | STATS total_transactions = COUNT(*), error_count = SUM(CASE(event.outcome == \"failure\", 1, 0)), p50_latency_ms = PERCENTILE(transaction.duration.us, 50) / 1000, p95_latency_ms = PERCENTILE(transaction.duration.us, 95) / 1000, p99_latency_ms = PERCENTILE(transaction.duration.us, 99) / 1000 BY service.name | EVAL error_rate_pct = ROUND(error_count / total_transactions * 100, 2) | EVAL throughput_per_min = ROUND(total_transactions / ?windowMinutes, 1)",
    "params": {
      "serviceName": {
        "type": "keyword",
        "description": "The APM service name to check (e.g., opbeans-node)"
      },
      "timeWindow": {
        "type": "keyword",
        "description": "Time window to analyze, in ES|QL duration format (e.g., 30 minutes, 1 hour, 6 hours)"
      },
      "windowMinutes": {
        "type": "integer",
        "description": "Time window in minutes, used to calculate throughput per minute"
      }
    }
  }
}

The query uses the traces-apm-* data stream, which contains raw transaction data. We filter with transaction.duration.us IS NOT NULL to select only transaction events (excluding spans). Using traces-apm-* is more portable than the pre-aggregated metrics-apm.transaction.1m-* stream, which only populates after sustained traffic.

Notice the description field. It is not just "returns health metrics." It includes interpretation rules: what error rate thresholds mean, what latency baselines look like, and how to correlate spikes with deploys. This is the runbook encoded in the tool.

Building Tool 2: get_recent_deploys

This tool answers: "What has been deployed recently?" Deploy history is a critical context because most production issues correlate with code changes. The agent needs this to reason about whether current metrics are normal or reflect a recent deployment.

Deploy annotations are stored in the observability-annotations index. Here is the full tool configuration:

{
  "id": "get_recent_deploys",
  "type": "esql",
  "description": "Returns the deployment history for a service over the last 24 hours, including version numbers, timestamps, and deploy messages. Use this tool to understand the deployment timeline when assessing service health. Key patterns: if a deploy happened within the last 15 minutes, elevated error rates or latency may be expected (warm-up period). If metrics degraded immediately after a deploy, the deploy is the likely cause. Multiple deploys in a short window (under 2 hours) increase risk because it becomes harder to isolate which change caused an issue.",
  "tags": ["apm", "deploys", "change-tracking"],
  "configuration": {
    "query": "FROM observability-annotations | WHERE service.name == ?serviceName AND @timestamp >= NOW() - 24 hours | SORT @timestamp DESC | KEEP @timestamp, service.version, service.environment, message | LIMIT 10",
    "params": {
      "serviceName": {
        "type": "keyword",
        "description": "The APM service name to check deploy history for"
      }
    }
  }
}

Again, the description encodes domain knowledge: the 15-minute warm-up window, the correlation between deploys and metric changes, and the risk of multiple rapid deploys. This is how a platform engineer transfers their intuition into something an AI agent can reason with.

Building Tool 3: get_slo_status

This tool answers: "How much error budget do we have left?" SLO budget is the platform team's quantified way of expressing risk tolerance. If the budget is nearly spent, even a small change could cause a violation.

Unlike the previous tools that query APM data, this one queries the internal SLO indices where Elastic stores pre-computed SLI data. The query calculates the current burn rate, that is, how fast the service is consuming error budget relative to the allowed threshold:

{
  "id": "get_slo_status",
  "type": "esql",
  "description": "Returns the current SLO burn rate for a service over the last hour. The response includes: SLI value (current performance), error budget target, and burn rate percentage. The burn rate tells you how fast the service is consuming error budget relative to the allowed threshold. Interpretation: a burn rate below 100% means the service is consuming budget slower than the limit (sustainable). Between 100-200%, the service is burning budget faster than planned (proceed with caution). Above 200%, the service is burning budget at double the allowed rate (delay non-critical changes). Above 500%, investigate immediately. Note: this measures the current burn rate over the last hour, not cumulative budget consumption over the full SLO window. A temporarily high burn rate does not mean the overall budget is exhausted.",
  "tags": ["slo", "reliability", "budget"],
  "configuration": {
    "query": "FROM .slo-observability.sli-v* | WHERE slo.id == ?sloId AND @timestamp >= NOW() - 1 hour | STATS sli_value = AVG(slo.numerator) / AVG(slo.denominator) BY slo.id, slo.name | EVAL error_budget_target = 0.995 | EVAL burn_rate_pct = ROUND((1 - sli_value) / (1 - error_budget_target) * 100, 1)",
    "params": {
      "sloId": {
        "type": "keyword",
        "description": "The SLO identifier. Use the SLO ID for the service you are evaluating."
      }
    }
  }
}

Note on the SLI index: the version suffix in .slo-observability.sli-v* depends on your Stack release (e.g., v3.6 in Stack 9.3). Verify with GET _cat/indices/.slo-observability.*?v and adjust the pattern if your cluster uses a different version.

The burn rate interpretation rules in the description are the most valuable part. A raw number like "burn rate 85%" means nothing to a developer without context. The tool description translates that into actionable guidance: "below 100% means sustainable, above 200% means delay non-critical changes."

Connecting to your editor via MCP

With all three tools created in Agent Builder, they are automatically available through the MCP server endpoint. Configure your MCP client to connect.

Claude Code configuration

Add the Elastic MCP server to your Claude Code settings:

{
  "mcpServers": {
    "elastic-agent-builder": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://your-kibana-url/api/agent_builder/mcp",
        "--header",
        "Authorization: ApiKey your-base64-api-key"
      ]
    }
  }
}

Cursor configuration

For Cursor, add the server in Settings > MCP Servers:

{
  "mcpServers": {
    "elastic-agent-builder": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://your-kibana-url/api/agent_builder/mcp",
        "--header",
        "Authorization: ApiKey your-base64-api-key"
      ]
    }
  }
}

Once connected, your editor's AI agent will discover all three tools automatically. You can verify by asking: "What Elastic tools do you have available?" The agent should list get_endpoint_health, get_recent_deploys, and get_slo_status.

API key permissions: the API key needs the feature_agentBuilder.read Kibana privilege and read access to the relevant indices (traces-apm.*, observability-annotations, .slo-observability.*). For production use, set the key expiry to 30-90 days and follow the principle of least privilege.

The scenario: "Is it safe to merge this PR?"

A developer on the team has a pull request that increases the timeout for the downstream recommendations service call from 2 seconds to 5 seconds in opbeans-node. Before merging, they ask the agent:

Developer: "I'm about to merge PR #42, which increases the recommendations service timeout from 2s to 5s in opbeans-node. Is it safe to merge right now?"

The agent begins its multi-signal reasoning chain. Here is what happens.

Step 1: the agent calls get_endpoint_health

The agent checks the current health of the service:

Step 2: the agent calls get_recent_deploys

Next, it checks for recent deployments:

Step 3: the agent calls get_slo_status

The agent's response

After correlating all three results, the agent produces a recommendation:

The agent pulled the current p99, checked recent deploys, and read the SLO burn rate. It combined those signals with the timeout change in the PR, flagged the merge as risky, and recommended next steps.

Conclusion: when to use MCP tools instead of pinging an SRE

With Elasticsearch, Agent Builder, and MCP, a developer can answer questions like "is it safe to merge this PR?" from inside their editor, in seconds, without pinging an SRE. Elasticsearch holds the signals: traces, deploy markers, and SLO budgets. Agent Builder is where the platform team encodes how to read those signals: the thresholds, the warm-up windows, the correlation rules. MCP is what carries those tools into the developer's editor.

The query pulls the data. The description tells the agent how to read it. The platform engineer writes the runbook once, and every developer on the team gets to use it.

Next steps: extend Elastic Agent Builder MCP tools to CI/CD

• Explore the Elastic Agent Builder documentation for more tool types and configuration options.
• See Agentic CI/CD: Kubernetes Deployment Gates with Elastic MCP Server for extending this pattern into your CI/CD pipeline.
• Check out Agent Skills for Elastic Observability for a complementary approach using pre-packaged observability skills.

Problem context

Outages often show up as correlated errors on multiple services like checkout, frontend, and recommendation in Elasticsearch. That pattern can mean a shared dependency, or it can mean Kubernetes is misleading callers: wrong targetPort, empty Endpoints, or pods that never become ready. Observability tools like Elasticsearch tell you that callers fail and which edges look wrong. They generally do not fetch the live Service spec or compare containerPort to targetPort for you.

The Elastic AI Agent in Agent Builder is built for APM, logs, metrics, dependencies, and service maps. It is not a full EKS operations console. You could attach all EKS MCP tools to the same agent, but long tool lists increase wrong-tool calls, slow planning, and widen blast radius if a prompt accidentally asks for mutating actions.

Solution overview

Use Elastic AI Agent as the only agent your SRE chats with. It reasons from Elasticsearch first. When evidence points to cluster config, it calls a workflow tool that invokes the K8s Troubleshooter agent over /api/agent_builder/converse with a structured user_prompt. The K8s Troubleshooter agent carries only the EKS MCP tools, and cluster access stays scoped to one specialist identity, IAM, and RBAC. You can audit like any other integration.

Elasticsearch reaches EKS through an in-cluster bridge, exposed to Kibana as an MCP connector with a shared secret.

Before you start

You need:

• An EKS cluster with kubectl configured.
• An Elasticsearch 9.3+ deployment, an OTLP endpoint, an Elasticsearch API key, Agent Builder, and rights to create agents, MCP tools, and Workflows.
• An AI Connector in Elasticsearch for your chosen LLM.
• Budget two to four hours the first time you run these steps.

Implementation walkthrough

Step 1: deploy the Elastic OpenTelemetry Demo and ship telemetry to Elastic Observability

Follow elastic/opentelemetry-demo for Kubernetes and deploy elastic-opentelemetry-demo application to your EKS cluster. Configure your Elasticsearch OTLP endpoint and API key, confirm workloads are running, and note the namespace. In Kibana (APM, Logs, or Service Map), confirm data for checkout, frontend, recommendation, and product-catalog.

If you see healthy traffic to product-catalog, you are ready for the failure drill.

Step 2: run the EKS MCP bridge, register the connector, and bulk import EKS MCP tools

Complete the steps in aws-eks-mcp-setup end to end. The flow you would be following is:

1. Build and push the EKS MCP Bridge image
2. Create IAM policies
3. Create IRSA Service Account
4. Map IRSA role in aws-auth and apply Kubernetes RBAC
5. Deploy the bridge with a strong API_ACCESS_TOKEN to the EKS cluster
6. Connect Elastic Agent Builder with EKS MCP

A green MCP connector proves Kibana can reach the bridge.

For production, restrict LoadBalancer security groups to known Elasticsearch egress, prefer TLS on real paths, store tokens in Kubernetes Secrets, and use read-only MCP modes when you only diagnose.

Step 3: create a K8s Troubleshooter agent with EKS tools only

In Agent Builder, create an agent with agent ID k8s_troubleshooter, display name K8s Troubleshooter, and custom instructions from k8s_troubleshooter_agent. Attach only EKS MCP tools to this agent.

Chat directly with K8s Troubleshooter agent once and confirm a harmless read (for example list pods in the demo namespace).

Step 4 (Elasticsearch 9.3 only): clone the Observability Agent without EKS tools

Clone the bundled Observability Agent (Agent Builder → Manage Agents → Observability Agent → Clone) and name it Elastic AI Agent so it keeps the stock Observability system instructions and tools. Do not attach EKS MCP tools to this copy.

The parent Elastic AI Agent stays an observability-first interface for whoever chats with it.

Step 5: create the workflow and make it a callable tool

Create a new Elastic Workflow by importing k8s_troubleshooter_workflow.yaml and enable it.

Create a new tool in Agent Builder of type Workflow. Select the k8s_troubleshooter workflow, set tool ID custom.k8s_troubleshooter, and set the description to Tool to triage and troubleshoot kubernetes related issues (or equivalent wording your team standardizes on).

On Elastic AI Agent, attach the custom.k8s_troubleshooter workflow tool that you just created.

The parent’s tool list should show the custom.k8s_troubleshooter workflow tool attached, and K8s Troubleshooter agent should still answer when invoked on its own.

Step 6: inject the product-catalog service misconfiguration

Save the original targetPort, then patch to a wrong value (for example 9999).

kubectl get svc -A | grep product-catalog
kubectl get svc product-catalog -n YOUR_NAMESPACE -o yaml

kubectl patch svc product-catalog -n YOUR_NAMESPACE --type='json' \
  -p='[{"op": "replace", "path": "/spec/ports/0/targetPort", "value": 9999}]'

kubectl rollout restart deployment/checkout deployment/recommendation deployment/frontend -n YOUR_NAMESPACE

Callers still resolve Endpoints, but traffic lands on a port the container does not listen on, so Elasticsearch shows downstream errors on checkout, frontend, and recommendation.

You now have symptoms in Elasticsearch and a clear kubernetes cluster-side fault.

Step 7: run two prompts on the parent agent

Use AI Agent chat on Elastic AI Agent, not on the specialist.

Note: If you are using Elasticsearch 9.3, make sure you use the Elastic AI Agent that you created, not the stock agent.

Prompt 1: Why are failure transactions increasing for services like checkout and frontend?

Expect Elastic AI Agent to narrow the issue to the product-catalog service and note possible configuration issues as one of the probable causes, without yet invoking the custom.k8s_troubleshooter tool.

Prompt 2: Why is product-catalog service not servicing any requests in (insert your k8s cluster name) cluster? Is there any misconfiguration in the service?

Expect Elastic AI Agent to call the custom.k8s_troubleshooter tool, which invokes the K8s Troubleshooter agent and reads Service, Endpoints, and pods, compares targetPort to containerPort, and explains the mismatch with evidence. Expect to also see the recommended remediation steps.

Note: depending on the LLM you are using, the response from the agents may vary.

You get agent-led triage in Elastic Observability and cluster-grounded confirmation in the same thread, along with recommended remediation steps.

Step 8: patch the product-catalog service

Prompt 3: Patch the product-catalog service in (your EKS cluster name) cluster to have 8080 as the targetPort

Expect Elastic AI Agent to call the custom.k8s_troubleshooter tool, which invokes the K8s Troubleshooter agent and patches the product-catalog service.

Prompt 4: Rollout restart upstream services of product-catalog service

Expect Elastic AI Agent to identify all upstream services of product-catalog and call the custom.k8s_troubleshooter tool, which invokes the K8s Troubleshooter agent to roll out restarts for upstream services such as checkout, frontend, and recommendation.

Confirm product-catalog and upstream services recover in Elasticsearch.

Validation and trade-offs

You validated that Elastic AI Agent stays the main surface, that ~20 EKS tools live on one specialist K8s Troubleshooter agent, and that the Workflow plus Agent Builder converse API keeps a clear boundary for audits and reviews.

Trade-offs: MCP bridges need ongoing token and network hygiene, and you should keep mutating tools off or tightly RBAC-scoped until you accept the risk.

Frequently asked questions

Why is Elastic AI Agent not triaging the issues as explained in this article?

There could be two primary reasons. (A) If you are on Elasticsearch 9.3, make sure you chat on the Elastic AI Agent that you created, and not on the stock agent. (B) Make sure to use one of the LLM models rated Excellent or Great in Large language model performance matrix for Observability

Why do my services look unhealthy in Elasticsearch when the app code did not change?

Kubernetes can mislead HTTP clients: a bad Service targetPort, empty Endpoints, or pods that never become ready can fan out as errors on multiple edges in traces and service maps. Elastic Observability shows which dependencies fail; confirming the live Service spec usually needs cluster access.

How do I give Kubernetes access to Elastic AI Agent without putting every EKS tool on it?

Run two Agent Builder agents: keep the stock tools on the parent (Elastic AI Agent), and attach only EKS MCP tools to a specialist agent(K8s Troubleshooter agent). Invoke the specialist through a workflow that calls the Agent Builder converse API so the boundary is explicit and auditable.

Why chain agents with Elastic Workflows instead of one long system prompt?

Workflows give a callable, reviewable step between observability reasoning and cluster actions, which helps with governance and keeps the parent agent’s tool list short. Long unified tool lists often increase mistaken tool use and broaden blast radius if a prompt requests a mutating operation.

How does this compare to kubectl or a cloud console for incident response?

Consoles and kubectl stay the source of truth for live object state. This pattern automates the handoff from Elastic Observability signals to those checks through MCP, while still relying on IAM and Kubernetes RBAC on the specialist identity.

What are the main limitations or risks of an EKS MCP bridge with Agent Builder?

MCP bridges need token rotation, network restrictions, and TLS discipline on real paths. Mutating EKS tools should stay off or tightly RBAC-scoped until you accept operational risk.

Why do we need an EKS MCP bridge?

The managed EKS MCP server authenticates via AWS SigV4 through a stdio-based proxy (mcp-proxy-for-aws). Elastic's MCP connector requires an HTTP/SSE endpoint. The bridge pod runs mcp-proxy to expose the stdio proxy as an SSE/HTTP endpoint.

Can I reuse the same layout on GKE, AKS, or self-managed Kubernetes?

Yes. The separation principle is the same: observability data in Elasticsearch plus a specialist agent with cluster-scoped tools and a workflow-mediated handoff. Swap the MCP server or bridge, adjust RBAC, and parameterize cluster name or region in workflow inputs where needed.

• Elasticsearch 9.3+ (or Elastic Cloud Serverless)

• Elasticsearch API KEY and Kibana URL

• An application instrumented with Elastic APM: the RUM agent for frontend interactions (populates traces-apm-*) and the APM agent for backend errors (populates logs-apm.error-*

• Cursor (version 2.6+) installed

The problem with two worlds

Application logs and code are two separate worlds that don't talk to each other. If you want to apply log insights into the application you have to analyze the logs, and then come back to the editor and apply your findings.

The Model Context Protocol (MCP) changes this. MCP is an open standard that lets AI clients like Cursor connect to external tools and data sources through a standardized interface. Instead of your IDE only knowing about your local code, it can also talk to your Elasticsearch cluster, query your APM data, and reason about production behavior alongside your source files.

Elastic ships a built-in MCP server as part of Agent Builder. You define tools in Kibana, expose them via the MCP endpoint, and any MCP-compatible client can call them. Cursor supports MCP natively, which means you can set this up in minutes.

What we're building

We're working with an eCommerce search app instrumented with Elastic APM. The RUM JS agent tracks filter click interactions from the browser, stored in traces-apm-default. The Node.js APM agent captures backend errors, stored in logs-apm.error-default.

Two situations come up during development:

• Use case 1: The product team wants to simplify the search page. There are six filters but we don't know which ones users actually click. We need usage data to decide which to keep.

• Use case 2: Users report intermittent 500 errors on search. The errors are not constant and started two days ago. We need the error details to find the root cause.

To bring that data into Cursor, we'll build two Agent Builder tools in Kibana and connect them via the Elastic Agent Builder MCP Server:

• get_filter_usage: queries traces-apm-default for filter click events and returns a usage breakdown by filter name

• get_recent_errors: queries logs-apm.error-default for the most recent error groups for a given service, including the exception message and stack trace culprit

For a deeper look at the overall architecture, see the Agent Builder reference guide.  

Setting up the Elastic MCP Server  

Step 1: Create the Agent Builder tools

We create both tools via the Kibana Agent Builder API. Each tool is an ES|QL query with a name and description that Cursor uses to decide when to call it. The full implementation of the tools is in the following notebook.

Tool 1: get_filter_usage

The product team needs to know which filters users actually click before deciding which ones to remove. The query reads RUM interaction events from traces-apm-default and groups them by filter name:

    {
      "id": "get_filter_usage",
      "type": "esql",
      "description": "Returns the usage count for each search filter in the ecommerce-search-ui service, sorted by most used first.",
      "configuration": {
        "query": "FROM traces-apm-default | WHERE service.name == \"ecommerce-search-ui\" | WHERE transaction.type == \"user-interaction\" | WHERE labels.filter_name IS NOT NULL | STATS count = COUNT(*) BY labels.filter_name | SORT count DESC"
      }
    }

Tool 2: get_recent_errors

For the error debugging use case, we need to surface the most frequent recent errors for a service, along with where in the code they originate. STATS ... BY groups errors by their fingerprint (grouping_key), surfaces the exception message and the line of code that caused it (culprit), and ranks by frequency:

    {
      "id": "get_recent_errors",
      "type": "esql",
      "description": "Returns the most frequent error groups for ecommerce-search-ui, ranked by occurrence count, with the exception message and code location.",
      "configuration": {
        "query": "FROM logs-apm.error-default | WHERE service.name == \"ecommerce-search-ui\" | WHERE processor.name == \"error\" | STATS count = COUNT(*) BY error.grouping_key, error.exception.0.message, error.culprit | SORT count DESC | LIMIT 5"
      }
    }

Both tools are created with POST /api/agent_builder/tools. You can learn more about the Kibana API endpoints for Elastic Agent Builder here.

Step 2: Connect to Cursor

Open ~/.cursor/mcp.json and add the Elastic server. For detailed information, see the Cursor documentation. The Agent Builder MCP endpoint uses Server-Sent Events (SSE) transport, so we connect via mcp-remote, a lightweight bridge that Cursor invokes as a local process:

    {
      "mcpServers": {
        "elastic-agent-builder": {
          "command": "npx",
          "args": [
            "-y",
            "mcp-remote",
            "https://YOUR_KIBANA_URL/api/agent_builder/mcp",
            "--header",
            "Authorization: ApiKey YOUR_API_KEY"
          ]
        }
      }
    }

Replace YOUR_KIBANA_URL and YOUR_API_KEY with your values.

Restart Cursor, open the Agent panel, and confirm that get_filter_usage and get_recent_errors appear in the available tools list. 

Use case 1: Data-driven UI optimization

The eCommerce search page has six filters: category, manufacturer, price range, customer gender, day of week, and region. The product team wants to simplify the UI by removing filters that users don't use as much. Rather than guessing, we ask Cursor to check.

When you type a prompt in Cursor's Agent panel, the model sees the name and description of every connected MCP tool. It matches your intent to the best-fitting tool and calls it automatically. This is why the description field we set in Step 1 matters: it's what the model reads to decide which tool answers your question. If you are interested in learning more about Cursor’s MCP tools management, read the following documentation.

Open a Cursor chat and ask: "Show me how often each search filter is used." Cursor calls the tool and returns something like:

The category and manufacturer filters get most of the clicks. The bottom three filters (customer_gender, day_of_week, region) are rarely used.

Ask Cursor to act on this: "Based on this data, simplify the SearchFilters component. Keep the top 3 filters visible, collapse the others under a 'More filters' toggle."

Cursor opens src/components/SearchFilters.jsx, reads the current implementation, and proposes the change.

Before: 

After: 

The entire loop took one chat prompt. The decision was backed by production data, not a team discussion about what users probably care about.

Use case 2: Production error debugging

A bug report comes in: intermittent 500 errors on the search endpoint. The errors started appearing two days ago but they're not constant. The developer opens Cursor and asks: "Show me what errors ecommerce-search-ui is throwing."

Cursor calls the tool and returns the error groups:

The error message is explicit: category is a text field and can't be used in terms of aggregation. The correct field is category.keyword. 

With APM data available alongside your code, the debugging session becomes a conversation: you describe the symptom, the agent pulls the relevant logs, and you work through what's happening together. You can ask follow-up questions, check whether the error correlates with a recent deployment, or ask which endpoints are most affected, all within the same context where you'll make the fix. If you want to go further, Elastic also provides pre-built observability tools in Agent Builder that you can use alongside custom tools like the ones we created here. For a complementary approach to AI-driven observability, see how to monitor web AI agents with OpenLIT and Elastic.

Conclusion

What we covered:

• How to create Agent Builder tools in Kibana that wrap APM data queries

• How to connect the Elastic Agent Builder MCP Server to Cursor in three lines of JSON

• Using production telemetry to make a UI decision backed by real usage data

• Debugging a production error from the same window where you fix it

These two use cases are a starting point. The same pattern works for any data you have in Elasticsearch: performance metrics, A/B test results, audit logs, feature flag usage, user session data. Define the Agent Builder tool, connect it via MCP, and it becomes part of your development context in Cursor. For other examples of what's possible, see automating synthetic monitoring with MCP and agentic CI/CD deployment gates.

Next steps

• Elastic Agent Builder MCP server documentation

• Model Context Protocol specification

• Elastic Agent Builder overview

Elastic Ramen (Root-cause Analysis & Monitoring Engine) bridges that gap. It is a local CLI agent that connects directly to Elastic Agent Builder, carrying the same conversation, skills, and Elastic context into the terminal. Ramen operates directly in the environment where fixes actually happen. No handoff. No re-auth. No translation layer. Ramen is open source and available at elastic/elastic-ramen.

Why the terminal matters

Agent Builder gives engineers a strong environment for querying observability data. Ramen takes that same capability to the two workflows that need it most.

Onboarding. Configuring collectors, managing credentials, and validating data flow all happen in the shell. A local agent can guide that work right where the credentials and tools already live.

Mitigation. The actual fix, whether restarting pods, scaling deployments, or rolling back releases, requires kubectl, gcloud, git, or internal scripts. A CLI agent runs on hardware the team already trusts, using the credentials already present on the engineer's machine.

How Ramen works

Ramen is a CLI client for Agent Builder. It is not a separate assistant with its own memory. It connects your local environment to the same conversations, skills, and tools you already use in Kibana through a simple authentication flow.

On first launch, Ramen connects to your Elastic deployment and gives you everything out of the box:

• LLM inference through the Kibana gateway, using your existing AI connector
• Native Kibana tools for managing workflows and agents
• The Agent Builder MCP server for ES|QL queries and documentation search
• An embedded elastic CLI for cluster health, data streams, and SLOs
• Built-in skills for root cause analysis and SLO management

The agent carries your investigation history across surfaces, so you never re-explain the incident when moving from the UI to the CLI. Terminal interactions sync back to Elastic automatically, building a searchable record of operational knowledge for the team.

Get started

You need an Elastic Observability Serverless project. In Kibana, open Stack Management, then Advanced Settings, or go directly to https://<your-kibana-url>/app/management/kibana/settings?query=ramen. Enable elasticRamen:enabled, then install the CLI:

npm i -g @elastic/ramen
bun add -g @elastic/ramen

You can also use the install script or download a pre-built binary from GitHub Releases:

curl -fsSL https://raw.githubusercontent.com/elastic/elastic-ramen/dev/install | bash

Once installed, connect to your deployment:

elastic-ramen --kibana-base=https://<your-kibana-url>

Ramen opens a browser auth flow, generates credentials, and stores them locally. After that, it reconnects automatically. Start a conversation in Agent Builder and resume it in the terminal with /kibana-conversations.

What is next

Ramen is the first surface of a multi-surface agent system. The same architecture extends to every surface engineers already use:

• Space-scoped collaboration for shared agent context during outages
• Slack, Teams, Jira, PagerDuty integration: start from an alert, collaborate in chat, mitigate in the terminal, one thread
• Shared memory: progressively distill conversations into durable operational context that improves future investigations

Beyond incident response, the same model applies to deployment risk analysis, production debugging, CI/CD policy checks, and cost anomaly investigation.

Summary

Ramen connects signal to action: Elastic data and Agent Builder context, plus the ability to act with local tools, in one continuous thread. Elastic as the persistent context layer, every surface you use as the interface.

Try it out on GitHub and let us know what you think.

• Detect Kubernetes pod restarts and OOMKill events using Elastic Agent Builder
• Analyze CPU and memory pressure using ES|QL over Kubernetes metrics
• Generate troubleshooting summaries and remediation guidance

This article explains how to use Elastic Agent Builder to automatically detect, analyze, and remediate Kubernetes pod failures caused by resource pressure (CPU and memory), with a focus on pods experiencing frequent restarts and OOMKilled events. Elastic Agent Builder lets you quickly create precise agents that utilize all your data with powerful tools (such as ES|QL queries), chat interfaces, and custom agents.

Introduction: What is the Elastic Agent Builder?

Elastic has an AI Agent embedded that you can use to get more insights from all of the logs, metrics and traces that you’ve ingested. While that’s great, you can take it one step further and streamline the process by creating tools that the agent can use.

Giving the agent tools means it spends less time ‘thinking’ and quickly gets to assessing what’s important to you. For example, if I have a Kubernetes environment that needs monitoring, and I want to keep an eye on pod restarts and memory and CPU usage without hanging out at the terminal, I can have Elastic alert me if something goes wrong. 

Having an alert is great, but how do I get the bigger picture, faster? You need to know what service is having (or creating) the issues, why, and how to fix it.

Assumptions

This guide assumes:

• A running Kubernetes cluster
• An Elastic Observability deployment
• Kubernetes metrics indexed in Elastic

Step 1: Create a New Elastic Agent

In Elastic Observability, use the top search bar to search for Agents. Create a new agent.

This agent is going to be the Kubernetes Pod Troubleshooter agent, designed to help users troubleshoot pod restarts, OOMKill terminations and evaluate CPU or memory pressure. 

The Kubernetes Pod Troubleshooter agent will:

1. Identify pods that have restarted more than once
2. Filter for pods that are not in a running state
3. Retrieve the container termination reason (e.g., OOMKilled)
4. Analyze CPU and memory utilization for affected services
5. Flag resource utilization above 60% (warning) and 80% (critical)
6. Provide remediation recommendations

The agent requires instructions to guide how the agent behaves when interacting with tools or responding to queries. This description can set tone, priorities or special behaviours. The instructions below tell the agent to execute the steps outlined above. 

You will help users troubleshoot problematic pods by searching the metrics for pods that have restarted more than once and the status is not running. Pods that have the highest number of restarts will be returned to the user.
Once the containers that are not running and have restarted multiple times are found you will use their container ID or image name to to look up the container status reason and reason for the last termination. You will return that reason to the user.
You will also begin basic troubleshooting steps, such as checking  for insufficient cluster resources (CPU or memory) from the metrics and tools available.
Any CPU or memory utilization percentages over 60%, and definitely over 80% should be flagged to the user with remediation steps.

Getting answers quickly is critical when troubleshooting high-value systems and environments. Using Tools ensures that the workflow is repeatable and that you can trust the results. You also get complete oversight of the process, as the Elastic Agent outlines every step and query that it took and you can explore the results in Discover.

You will create custom tools that the agent will run to complete the Kubernetes troubleshooting tasks that the custom instructions references such as: look up the container status reason and reason for the last termination and checking  for insufficient cluster resources (CPU or memory).

Step 2: Create Tools - Pod Restarts

The first tool takes the Kubernetes metrics and assesses if the pod has restarted and it has a last terminated reason, and if it has the agent will present that information to the user.

This pod-restarts tool uses a custom ES|QL query that interrogates the Kubernetes metrics data coming from OTel.

The ES|QL query:

1. Filters for containers that have restarted and have a reason for termination; then
2. Calculates the number of restarts; then
3. Returns the number of restarts and termination reason per service.

FROM metrics-k8sclusterreceiver.otel-default
| WHERE metrics.k8s.container.restarts > 0
| WHERE resource.attributes.k8s.container.status.last_terminated_reason IS NOT NULL
| STATS total_restarts = SUM(metrics.k8s.container.restarts),
        reasons = VALUES(resource.attributes.k8s.container.status.last_terminated_reason) 
  BY resource.attributes.service.name
| SORT total_restarts DESC

Step 3: Create Tools - Service Memory

The custom tools can take input variables, which increases speed and accuracy of the results.

Common reasons for pods not scheduling, or restarting often, is due to the cluster or nodes being under-resourced. The pod-restarts tool returns services that have many restarts and OOMKill termination reasons, which indicate memory pressure.

The eval-pod-memory tool is a custom ES|QL that:

1. Filters for metrics data that match the service name returned from the pod-restarts tool within the last 12 hours; then
2. Converts memory usage, requests, limits and utilization into megabytes; then
3. Calculates the average of each of those metrics; then
4. Groups them into 1 minute groupings and sorts them.

FROM metrics-*
| WHERE resource.attributes.service.name == ?servicename
| WHERE @timestamp >= NOW() - 12 hours
| EVAL
  memory_usage_mb = metrics.container.memory.usage / 1024 / 1024,
   memory_request_mb = metrics.k8s.container.memory_request / 1024 / 1024,
   memory_limit_mb = metrics.k8s.container.memory_limit / 1024 / 1024,
   memory_utilization_pct = metrics.k8s.container.memory_limit_utilization * 100
| STATS
   avg_memory_usage = AVG(memory_usage_mb),
   avg_memory_request = AVG(memory_request_mb),
   avg_memory_limit = AVG(memory_limit_mb),
   avg_memory_utilization = AVG(memory_utilization_pct)
   BY bucket = BUCKET(@timestamp, 1 minute)
| SORT bucket ASC

Step 4: Create Tools: Service CPU

As CPU usage is another common reason for pods to fail scheduling or be stuck in endless restart loops, the next tool will evaluate CPU usage, requests and limits.

The eval-pod-cpu tool is a custom ES|QL that:

1. Filters for metrics data that match the service name returned from the pod-restarts tool within the last 12 hours; then
2. Calculates the average for CPU usage, CPU request utilization and CPU limit utilization.

FROM metrics-kubeletstatsreceiver.otel-default
| WHERE k8s.container.name == ?servicename OR resource.attributes.k8s.container.name == ?servicename
| STATS
  avg_cpu_usage = AVG(container.cpu.usage),
  avg_cpu_request_utilization = AVG(k8s.container.cpu_request_utilization) * 100,
  avg_cpu_limit_utilization = AVG(k8s.container.cpu_limit_utilization) * 100
| LIMIT 100

Step 5: Assign Tools to Kubernetes Pod Troubleshooter Agent

Once all of the tools are built you need to assign them to the agent.

This image shows the Kubernetes Pod Troubleshooter agent with the three tools: pod-restarts, eval-pod-cpu and eval-pod-memory assigned to it and active.

Step 6: Test the Kubernetes Pod Troubleshooter Agent

To simulate memory pressure the Open Telemetry demo is running inside the cluster. Artificially lowering the memory requests and limits and increasing the service load will cause pods to restart.

To do this to the open telemetry demo in your cluster, follow these steps. 

Reduce the cart service to one replica by scaling the deployment. Once that is complete, change the resources on the deployment by lowering the memory requests and limits as shown in this command:

kubectl -n otel-demo scale deploy/cart --replicas=1
kubectl -n otel-demo set resources deploy/cart -c cart --requests=memory=50Mi --limits=memory=60Mi

The OpenTelemetry demo application comes with a load-generator. This is used to simulate requests to the demo site by modifying the users and spawn rate in the load generator deployment, as shown in this command:

kubectl -n otel-demo set env deploy/load-generator LOCUST_USERS=800 LOCUST_SPAWN_RATE=200 LOCUST_BROWSER_TRAFFIC_ENABLED=false

If you list all of your pods in the cluster or namespace, you should begin to see restarts.

You can now chat with the Kubernetes Pod Troubleshooter agent and ask “Are any of my Kubernetes pods having issues?”.

The screenshot shows the final response from the Kubernetes Pod Troubleshooter agent. It provides a problem summary of its findings from each tool, showing which services were experiencing the most restarts and memory and CPU utilization. 

The threshold interpretations were described in the initial agent instructions, where >60% utilization is a warning (sustained pressure) and >80% utilization is critical (high likelihood of restarts or throttling). This aligns with findings presented by the Kubernetes Pod Troubleshooter agent, where the services that had the highest restarts were all above 90% memory utilization. The agent needs clearly defined threshold values to correctly assess the returned memory and CPU utilization values. 

Problem summary returned by the Kubernetes Pod Troubleshooter agent:

Conclusion and Final Thoughts

Elastic Agent Builder enables fast, repeatable Kubernetes troubleshooting by combining ES|QL-driven analysis with constrained AI reasoning.

The creation of custom tools that use specific ES|QL queries combined with downstream queries that take input variables from the output of previous tools eliminates or reduces error propagation and hallucinations. In comparison to generic AI troubleshooting without purpose-built tools, you run the risk of it analyzing too many services (that aren’t relevant to the issue at hand). This will slow down the thinking process and generate longer responses, increasing the likelihood of error propagation and hallucinations. 

With the Elastic Agent Builder, you can inspect the output of every tool if you need to, to explore and verify the outputs.

Having a succinct problem summary is a game-changer, bringing your attention straight to the most affected services.

Reasoning returned by the Kubernetes Pod Troubleshooter agent:

Not only that, but the agent can go one step further and offer recommendations for remediation based on what outputs the tools delivered.

Remediation recommendation returned by the Kubernetes Pod Troubleshooter agent:

Sign up for Elastic Cloud Serverless and try this out with your Kubernetes clusters.

Frequently Asked Questions

1. When to use the Elastic Agent Builder for Troubleshooting

Use the Elastic Agent Builder for Troubleshooting that works best if:

• You need repeatable, auditable troubleshooting workflows

• You want deterministic analysis instead of free-form AI responses

• You’re investigating something that is reported in the logs or metrics (i.e. pod restarts, OOMKills, or resource pressure)

• You want to reduce mean time to resolution (MTTR)

2. Do I need OpenTelemetry to use Elastic Agent Builder for Kubernetes troubleshooting? 

No, you don’t need to use OpenTelemetry. You have two options:

• You can collect logs and metrics from Kubernetes using the Elastic Agent; or 

• You can collect logs, traces and metrics with the Elastic Distro for OTel (EDOT) Collector

When following the steps above, this would change the field names that are used in the tools above. For example, kubernetes.container.memory.usage.bytes vs metrics.container.memory.usage.

3. Can this agent be adapted for node-level failures? 

Yes, Elastic has hundreds of integrations, including AWS (for EKS), Azure (for AKS), Google Cloud (for GKE), as well as host operating system monitoring.

The queries shown above would be modified to use the correct field.

4. Can these tools be reused in automation workflows? 

Yes, Elastic Workflows can reuse the same scripted automations and AI agents you build in Elastic. An agent can handle the initial analysis and investigation (reducing manual effort), and the workflow can continue with structured steps, such as running Elasticsearch queries, transforming data, branching on conditions and calling external APIs or tools like Slack, Jira and PagerDuty. Workflows can also be exposed to Agent Builder as reusable tools, just like the tool created in this guide.

For more advanced automation from a similar scenario as described in this guide, learn how to integrate AI agents into GitHub Actions to monitor K8s health and improve deployment reliability via Observability.

5. Can these tools be triggered by alerts? 

Yes, alerts can trigger Elastic Workflows, and pass the alert context to the workflow. This workflow may be integrated with an Elastic Agent, as described above.

Additionally, Elastic Alerts allow you to publish investigation guides alongside alerts so an SRE has all of the information they need to begin investigating. Any troubleshooting or investigative agents can be linked to from the investigation guide, meaning the SRE doesn’t have to follow manual processes outlined in an investigation guide and instead let the agent handle the manual, repetitive investigations.

6. How can I get started with Agent Builder?

Sign up for Elastic Cloud Serverless, a new fully managed, stateless architecture that auto-scales no matter your data, usage, and performance needs.

Elastic has been leading the charge with AIOps, especially in the recent 9.2 update of Elastic Observability with Streams. The conversation around AIOps has shifted dramatically as we move through the year. DevOps and SRE teams aren't asking whether they need AIOps, they're asking how to leverage it more effectively to stay ahead of exponentially growing complexity.

The current challenge of AIOps is that modern cloud-native environments generate massive volumes of telemetry data that are magnitudes larger than past environments. But here's what many teams overlook: logs are the richest source of operational intelligence you have. Logs are able to tell you exactly what happened and why, while metrics only tell you something is wrong, and traces only tell you where. The problem is that most organizations are drowning in logs. Microservices, such as user authentications or inventories, serverless functions, and Kubernetes generate millions of log entries daily. Without AI and machine learning, finding meaningful patterns in this data takes too much time and energy.

Log Intelligence Improvement: What's New in 2025

Historically in observability, unlocking your log intelligence included long manual effort that required not only parsing through logs, but also structuring those logs. Elastic Observability has drastically changed how teams extract value from logs. Observability is not just simple signal analysis - modern tools need to have proactive, log-driven investigations. At Elastic, this modernity is Streams.

Streams, a new release from Elastic, is a collection of AI-driven tools that identify significant events in parsed raw logs by enriching logs with meaningful fields. With Streams, SREs can maximize the value of their data, their logs, and their systems. With system reliability as the goal, Streams helps to reduce pipeline management overhead and accelerates observability analysis. And it takes nearly no time to set up!

Here is how Streams powers the Elastic Observability capabilities available now.

Advanced Log Rate Analysis

Log rate analysis can go far beyond only detecting spikes. Elastic's machine learning automatically identifies when log volumes deviate from expected baselines, then contextualizes these changes within your broader system performance. When your application suddenly generates more error logs, Elastic’s AIOps doesn't just alert you, it also determines whether it's a critical issue requiring immediate attention or just a temporary anomaly.

This matters to your analysis because not all log spikes are equal. A 10x increase in DEBUG logs might indicate verbose logging accidentally enabled in production. A 2x increase in ERROR logs could signal a cascading failure. Log rate analysis distinguishes between these scenarios automatically, giving your team the context needed to respond appropriately.

Intelligent Log Categorization with Streams

This is where AIOps shines with log data. Streams uses machine learning algorithms in order to automatically classify and group similar log patterns, dramatically reducing noise. Instead of manually parsing millions of entries, the system identifies common structures, groups related events, and surfaces the categories that matter most.

Logs are unstructured by nature, making them difficult to analyze at scale. Streams corrals chaotic log streams into organized, queryable patterns. Instantly, you can see that 80% of your errors fall into three categories, helping you prioritize where to focus remediation efforts. This approach helps you reduce noise and accelerate analysis, allowing teams to act on insights faster.

Multi-Dimensional Anomaly Detection

Anomaly detection now simultaneously examines relationships between logs, metrics, and traces. A slight increase in response time might not trigger an alert by itself, but when correlated with unusual log patterns and memory consumption changes, the system recognizes it as an early warning sign.

Logs contain a myriad of contextual information that metrics and traces can't capture: stack traces, user IDs, transaction details, error messages, etc. By correlating log anomalies with other signals, you get the full picture of what's happening in your system. This whole holistic view enables teams to catch issues earlier, as well as understand their full impact across the stack.

Enhanced Root Cause Analysis Powered by Significant Events

When an issue occurs, Elastic's Streams accelerates root cause analysis through AI-assisted parsing of logs and bringing about “Significant events.” Significant event queries can be defined by AI or manually, depending on if you know what logs you are looking for or not. Then, Elastic’s AIOps traces the problem through your entire stack using these events, as well as enriched log data combined with distributed tracing. This system is able to correlate failed transactions with specific log entries, deployment events, and infrastructure changes. This helps you understand not just what broke, but why and when.

Streams makes the analysis of your logs quick and automatic by going across your entire distributed system within seconds, grabbing relevant log entries such as stack traces, state information, error messages, and more. What used to require hours of manual investigation and deduction now happens automatically, freeing you and your team from tedious detective work and enabling faster resolution. 

Logs in Action: Real-World Impact

Let's look at how these capabilities work together in practice. Imagine your payment processing service is experiencing intermittent failures - only 0.5% of transactions, but enough to concern your team. Traditional monitoring shows everything is mostly okay, but customers are still complaining.

Without Streams, an SRE might initially run some broad queries, manually sift through thousands of logs, struggle to connect all the dots, and ultimately not understand the correlation between the errors and recent system changes. 

With Elastic Streams and AIOps, many of these potential problems are instantly mitigated:

• Streams automatically parse the payment service, adding connection timeouts to a new category of significant events

• Log rate analysis with Streams reveal that this significant event category has been slowly growing over the past month, showing growth of the timeouts from a small number of occurrences into a larger amount

• Elastic’s built-in anomaly detection correlates these significant events with deployment data, and identifies that they started appearing after a recent load balancer configuration

• Root analysis pinpoints the exact database connection pool setting that is too restrictive for peak load by tracing affected transactions through previously enriched logs

What usually takes 4-8 hours of manual log analysis is resolved in minutes, with Elastic automatically highlighting the relevant log entries that tell the complete story. This is the power of AIOps and Streams as applied to log intelligence.

The Power of Unified Log Intelligence

What sets Elastic apart is treating logs as a priority in your observability strategy. Elastic provides comprehensive log ingestion that centralizes petabytes of logs from across your infrastructure with flexible parsing and enrichment. The platform uses purpose-built machine learning models that understand log patterns, not generic algorithms retrofitted for log analysis.

Logs don't exist in isolation, which is why Elastic correlates log data with metrics, traces, and business events to provide complete context. And because log volumes can be massive, Elastic's tiered storage approach means you can retain years of logs for compliance and historical analysis without breaking the budget.

Why Logs Matter More Than Ever

Logs have become the cornerstone of effective AIOps for three critical reasons.

First off, logs capture what metrics can't. A metric tells you the CPU is at 80%, but a log tells you which process is consuming resources and why. This level of detail is essential for understanding not just that something is wrong, but what specifically is causing the problem.

Second, logs provide business context. Error messages contain user IDs, transaction ldetails, and business logic failures that help you understand customer impact. When you're troubleshooting an issue, knowing which customers are affected and what they were trying to do is invaluable for prioritizing your response.

Third, logs enable true root cause analysis. Stack traces, error messages, and application state captured in logs are essential for understanding the why behind every incident. Without this information, teams are left guessing at root causes rather than definitively identifying and fixing them.

The teams winning with AIOps in 2025 aren't just monitoring metrics, they're extracting intelligence from their logs at scale, turning operational data into actionable insights.

Transform Your Log Strategy Today

Every hour your team spends manually searching through logs is an hour they're not spending on innovation. Every incident that could have been prevented through intelligent log analysis represents both technical debt and business risk.

Elastic Observability provides the foundation you need to unlock the intelligence hidden in your logs. With automatic categorization, anomaly detection, and ML-powered analysis, you can start seeing value immediately. Check out this recent article to get started with Elastic Streams and Observability today!

Anomaly Detection

A production environment is ingesting millions of log lines per minute. Elastic’s AIOps jobs continuously profile normal log throughput and content without any manual rules. When log volume or message structure deviates beyond learned baselines, the platform automatically fires a high‑fidelity anomaly alert. Because the models are unsupervised, they adapt to changing traffic patterns and flag both sudden spikes (e.g., 10× error surge) and rare new log categories.

In addition to looking directly for Log Spikes, Elastic trains seasonal/univariant models to predict expected event counts per bucket and applies statistical tests to classify outliers. Simultaneously, log categorization clusters similar messages with cosine similarity on token embeddings, making it trivial to identify a previously unseen error string.

Investigating Alerts: Automated Pattern Analysis

Clicking the alert reveals more than a timestamp. Elastic’s ML job already correlates the spike with the dominant new log pattern ERROR 1114 (HY000): table "orders" is full and surfaces example lines. Instead of grep‑driven hunting, engineers get an immediate hypothesis about what subsystem is failing and why.

If deeper context is needed, the builtin Elastic AI Assistant can be invoked directly from the alert. Thanks to Retrieval‑Augmented Generation (RAG) over your telemetry, the assistant explains the anomaly in plain language, references the exact log events, and proposes next steps without hallucinating.

AI‑Assisted Root Cause Verification

From within the same chat, you might ask, “Using lens create a single graph of all http response status codes =400 from logs-nginx.access-default over the last 3 hours..”  The assistant translates that intent into an ES|QL aggregation, retrieves the data, and renders a bar chart with no DSL knowledge required. If there are a number of errors with a status code above 400, you’ve validated that end‑users are impacted.

Global Impact Analysis with Enriched Logs

Structured log enrichment (e.g., GeoIP, user ID, service tags) lets the assistant answer business questions on the fly. A query like “What are the top 10 source.geo.country_name with http.response.status.code>=400 over the last 3 hours. Use logs-nginx.access-default. Provide counts for each country name.” surfaces whether the incident is regional or global.

Quantifying Business Impact

Technical metrics alone rarely sway executives. Suppose historical data shows the application normally processes $1,000 in transactions per minute. The assistant can combine that baseline with real‑time failure counts to estimate revenue loss. Presenting financial impact alongside error graphs sharpens prioritization and justifies extraordinary remediation steps.

Pinpointing Infrastructure & Ownership

Every log is automatically enriched with Kubernetes, cloud, and custom metadata. A single question “Which pod and cluster emit the ‘table full’ error, and who owns it?” returns the full information about the pod, namespace and owner as shown below.

Immediate, accurate routing replaces frantic Slack threads, cutting minutes (or hours) off of downtime.

Some of the magic happening here is because we can put instructions in the Elastic AI Assistants knowledge base to guide the AI assistant. For example this simple entry in the knowledge base is what allows the assistant to populate the response in the previous screenshot.

``markdown ## Kubernetes Information Query Instructions If asked about Kubernetes pod, namespace, cluster, location, or owner run the "query" tool. 1. Use the index logs-mysql.error-default unless another log location is specified. 2. Include the following fields in the query:    - Pod: agent.name    - Namespace: data_stream.namespace    - Cluster Name: orchestrator.cluster.name    - Cloud Provider: cloud.provider    - Region: cloud.region    - Availability Zone: cloud.availability_zone    - Owner: cloud.account.id` 3. Use the ES|QL query format:    esql    FROM logs-mysql.error-default    | KEEP agent.name, data_stream.namespace, orchestrator.cluster.name, cloud.provider, cloud.region, cloud.availability_zone, cloud.account.id     4. Ensure the query is executed within the appropriate time range and context. 

## Leveraging Institutional Knowledge with RAG

Elastic can index runbooks, GitHub issues, and wikis alongside telemetry. Asking “Find documentation on fixing a full orders table” retrieves and summarizes a prior runbook that details archiving old rows and adding a partition. Grounding remediation in proven procedures avoids guesswork and accelerates fixes.

![](https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/bltba6ebcc6e2f451ad/6a7f01b233fa8ae5f62021c9/image6.png)

## Automated Communication & Documentation

Good incident response includes timely stakeholder updates. A prompt such as “Draft an incident update email with root cause, impact, and next steps” lets the assistant assemble a structured message and send it via the alerting framework’s email or Slack connector complete with dashboard links and next‑update timelines. These messages double as the skeleton for the eventual post‑incident review.

![](https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/blt5e7b81ee98b1a9a5/6a7f01b5227b1c583c598102/image8.png)

Again as before, some of the magic happening here is because we can put instructions in the Elastic AI Assistants knowledge base to guide the AI assistant. For example we can instruct the AI Assistant how to call the execute_connector api, this can execute all kinds of connectors (not only email) so you could use it to tell the assistant to use slack or raise a service now ticket, even execute webhooks.

markdown Here are specific instructions to send an email. Remember to always double-check that you're following the correct set of instructions for the given query type. Provide clear, concise, and accurate information in your response.

Email Instructions

If the user's query requires sending an email:

1. Use the Elastic-Cloud-SMTP connector with ID elastic-cloud-email.
2. Prepare the email parameters:    - Recipient email address(es) in the to field (array of strings)    - Subject in the subject field (string)    - Email body in the message field (string)
3. Include

• Details for the alert along with a link to the alert
• Root cause analysis
• Revenue impact
• Remediation recommendations
• Link to GitHub issue
• All relevant information from this conversation
• Link to the Business Health Dashboard

1. Send the email immediately. Do not ask the user for confirmation.
2. Execute the connector using this format:

   execute_connector(      id="elastic-cloud-email",      params={        "to": ["recipient@example.com"],        "subject": "Your Email Subject",        "message": "Your email content here."      }    )

1. Check the response and confirm if the email was sent successfully. ```

Conclusion & Key Takeaways

Elastic Observability's combination of unsupervised ML, schema-aware data ingestion, and a context-rich RAG powered AI assistant enables teams to transform incident response from reactive firefighting into proactive, data-driven operations. By automatically detecting anomalies, correlating patterns, and providing contextual insights, teams can:

• Preserve revenue by quantifying business impact in real-time and prioritizing accordingly
• Scale expertise by embedding institutional knowledge into RAG-powered recommendations
• Improve continuously through automated documentation that feeds back into the knowledge base

The key is to collect logs broadly, maintain a unified observability store, and let ML and AI handle the heavy lifting. The payoff isn't just reduced downtime, it's the transformation of incident response from a source of organizational stress into a competitive advantage.

Try out this exact scenario and get hands in with this Elastic Logging Workshop: https://play.instruqt.com/elastic/invite/rx4yvknhpfci

As SREs, we've grown accustomed to jumping between dashboards, log aggregators, and trace visualizers when troubleshooting issues. But what if there was a better way? What if AI could analyze all your observability data holistically, answering complex questions in natural language, and identifying root causes automatically?

This is the next evolution of observability. But to harness this power, we need to rethink how we collect, store, and analyze our telemetry data.

The problem: siloed data limits AI effectiveness

Traditional observability setups separate data into distinct types:

• Metrics: Numeric measurements over time (CPU, memory, request rates)
• Logs: Detailed event records with timestamps and context
• Traces: Request journeys through distributed systems
• Profiles: Code-level execution patterns showing resource consumption and performance bottlenecks at the function/line level

This separation made sense historically due to the way the industry evolved. Different data types have traditionally had different cardinality, structure, access patterns and volume characteristics. However, this approach creates significant challenges for AI-powered analysis:

Metrics (Prometheus) → "CPU spiked at 09:17:00"
Logs (ELK) → "Exception in checkout service at 09:17:32" 
Traces (Jaeger) → "Slow DB queries in order-service at 09:17:28"
Profiles (pyroscope) -> "calculate_discount() is taking 75% of CPU time"

When these data sources live in separate systems, AI tools must either:

1. Work with an incomplete picture (seeing only metrics but not the related logs)
2. Rely on complex, brittle integrations that often introduce timing skew
3. Force developers to manually correlate information across tools

Imagine asking an AI, "Why did checkout latency spike at 09:17?" To answer comprehensively, it needs access to logs (to see the stack trace), traces (to understand the service path), and metrics (to identify resource strain). With siloed tools, the AI either sees only fragments of the story or requires complex ETL jobs that are slower than the incident itself.

Why traditional machine learning (ML) falls short

Traditional machine learning for observability typically focuses on anomaly detection within a single data dimension. It can tell you when metrics deviate from normal patterns, but struggles to provide context or root cause.

ML models trained on metrics alone might flag a latency spike, but can't connect it to a recent deployment (found in logs) or identify that it only affects requests to a specific database endpoint (found in traces). They behave like humans with extreme tunnel vision, seeing only a fraction of the relevant information and only the information that a specific vendor has given you an opinionated view into.

This limitation becomes particularly problematic in modern microservice architectures where problems frequently cascade across services. Without a unified view, traditional ML can detect symptoms but struggles to identify the underlying cause.

The solution: unified data with enriched logs

The solution is conceptually simple but transformative: unify metrics, logs, and traces into a single data store, ideally with enriched logs that contain all signals about a request in a single JSON document. We're about to see a merging of signals.

Think of traditional logs as simple text lines:

[2025-05-19 09:17:32] ERROR OrderService - Failed to process checkout for user 12345

Now imagine an enriched log that contains not just the error message, but also:

• The complete distributed trace context
• Related metrics at that moment
• System environment details
• Business context (user ID, cart value, etc.)

This approach creates a holistic view where every signal about the same event sits side-by-side, perfect for AI analysis.

How generative AI changes things

Generative AI differs fundamentally from traditional ML in its ability to:

1. Process unstructured data: Understanding free-form log messages and error text
2. Maintain context: Connecting related events across time and services
3. Answer natural language queries: Translating human questions into complex data analysis
4. Generate explanations: Providing reasoning alongside conclusions
5. Surface hidden patterns: Discovering correlations and anomalies in log data that would be impractical to find through manual analysis or traditional querying

With access to unified observability data, GenAI can analyze complete system behavior patterns and correlate across previously disconnected signals.

For example, when asked "Why is our checkout service slow?" a GenAI model with access to unified data can:

• Analyze unified enriched logs to identify which specific operations are slow and to find errors or warnings in those components
• Check attached metrics to understand resource utilization
• Correlate all these signals with deployment events or configuration changes
• Present a coherent explanation in natural language with supporting graphs and visualizations

Implementing unified observability with OpenTelemetry

OpenTelemetry provides the perfect foundation for unified observability with its consistent schema across metrics, logs, and traces. Here's how to implement enriched logs in a Java application:

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.metrics.Meter;
import io.opentelemetry.api.metrics.DoubleHistogram;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
import java.lang.management.ManagementFactory;
import java.lang.management.OperatingSystemMXBean;

public class OrderProcessor {
    private static final Logger logger = LoggerFactory.getLogger(OrderProcessor.class);
    private final Tracer tracer;
    private final DoubleHistogram cpuUsageHistogram;
    private final OperatingSystemMXBean osBean;

    public OrderProcessor(OpenTelemetry openTelemetry) {
        this.tracer = openTelemetry.getTracer("order-processor");
        Meter meter = openTelemetry.getMeter("order-processor");
        this.cpuUsageHistogram = meter.histogramBuilder("system.cpu.load")
                                      .setDescription("System CPU load")
                                      .setUnit("1")
                                      .build();
        this.osBean = ManagementFactory.getOperatingSystemMXBean();
    }

    public void processOrder(String orderId, double amount, String userId) {
        Span span = tracer.spanBuilder("processOrder").startSpan();
        try (Scope scope = span.makeCurrent()) {
            // Add attributes to the span
            span.setAttribute("order.id", orderId);
            span.setAttribute("order.amount", amount);
            span.setAttribute("user.id", userId);
            // Populate MDC for structured logging
            MDC.put("trace_id", span.getSpanContext().getTraceId());
            MDC.put("span_id", span.getSpanContext().getSpanId());
            MDC.put("order_id", orderId);
            MDC.put("order_amount", String.valueOf(amount));
            MDC.put("user_id", userId);
            // Record CPU usage metric associated with the current trace context
            double cpuLoad = osBean.getSystemLoadAverage();
            if (cpuLoad >= 0) {
                cpuUsageHistogram.record(cpuLoad);
                MDC.put("cpu_load", String.valueOf(cpuLoad));
            }
            // Log a structured message
            logger.info("Processing order");
            // Simulate business logic
            // ...
            span.setAttribute("order.status", "completed");
            logger.info("Order processed successfully");
        } catch (Exception e) {
            span.recordException(e);
            span.setAttribute("order.status", "failed");
            logger.error("Order processing failed", e);
        } finally {
            MDC.clear();
            span.end();
        }
    }
}

This code demonstrates how to:

1. Create a span for the operation
2. Add business attributes
3. Add current CPU usage
4. Link everything with consistent IDs
5. Record exceptions and outcomes in the backend system

When configured with an appropriate exporter, this creates enriched logs that contain both application events and their complete context.

Powerful queries across previously separate data

With data that has not yet been enriched, there is still hope. Firstly with GenAI powered ingestion it is possible to extract key fields to help correlate data such as a session id's. This will help you enrich your logs so they get the structure they need to behave like other signals. Below we can see Elastic's Auto Import mechanism that will automatically generate ingest pipelines and pull unstructured information from logs into a structured format perfect for analytics.

Once you have this data in the same data store, you can perform powerful join queries that were previously impossible. For example, finding slow database queries that affected specific API endpoints:

FROM logs-nginx.access-default 
| LOOKUP JOIN .ds-logs-mysql.slowlog-default-2025.05.01-000002 ON request_id 
| KEEP request_id, mysql.slowlog.query, url.query 
| WHERE mysql.slowlog.query IS NOT NULL

This query joins web server logs with database slow query logs, allowing you to directly correlate user-facing performance with database operations.

For GenAI interfaces, these complex queries can be generated automatically from natural language questions:

"Show me all checkout failures that coincided with slow database queries"

The AI translates this into appropriate queries across your unified data store, correlating application errors with database performance.

Real-world applications and use cases

Natural language investigation

Imagine asking your observability system:

"Why did checkout latency spike at 09:17 yesterday?"

A GenAI-powered system with unified data could respond:

"Checkout latency increased by 230% at 09:17:32 following deployment v2.4.1 at 09:15. The root cause appears to be increased MySQL query times in the inventory-service. Specifically, queries to the 'product_availability' table are taking an average of 2300ms compared to the normal 95ms. This coincides with a CPU spike on database host db-03 and 24 'Lock wait timeout' errors in the inventory service logs."

Here's an example of Claude Desktop connected to Elastic's MCP (Model Context Protocol) Server which demonstrates how powerful natural language investigations can be. Here we ask Claude "analyze my web traffic patterns" and as you can see it has correctly identified that this is in our demo environment.

Unknown problem detection

GenAI can identify subtle patterns by correlating signals that would be missed in siloed systems. For example, it might notice that a specific customer ID appears in error logs only when a particular network path is taken through your microservices—indicating a data corruption issue affecting only certain user flows.

Predictive maintenance

By analyzing the unified historical patterns leading up to previous incidents, GenAI can identify emerging problems before they cause outages:

"Warning: Current load pattern on authentication-service combined with increasing error rates in user-profile-service matches 87% of the signature that preceded the April 3rd outage. Recommend scaling user-profile-service pods immediately."

The future: agentic AI for observability

The next frontier is agentic AI, systems that not only analyze but take action automatically.

These AI agents could:

1. Continuously monitor all observability signals
2. Autonomously investigate anomalies
3. Implement fixes for known patterns
4. Learn from the effectiveness of previous interventions

For example, an observability agent might:

• Detect increased error rates in a service
• Analyze logs and traces to identify a memory leak
• Correlate with recent code changes
• Increase the memory limit temporarily
• Create a detailed ticket with the root cause analysis
• Monitor the fix effectiveness

This is about creating systems that understand your application's behavior patterns deeply enough to maintain them proactively. See how this works in Elastic Observability, in the screenshot at the end of the RCA we are sending an email summary but this could trigger any action.

Business outcomes

Unifying observability data for GenAI analysis delivers concrete benefits:

• Faster resolution times: Problems that previously required hours of manual correlation can be diagnosed in seconds
• Fewer escalations: Junior engineers can leverage AI to investigate complex issues before involving specialists
• Improved system reliability: Earlier detection and resolution of emerging issues
• Better developer experience: Less time spent context-switching between tools
• Enhanced capacity planning: More accurate prediction of resource needs

Implementation steps

Ready to start your observability transformation? Here's a practical roadmap:

1. Adopt OpenTelemetry: Standardize on OpenTelemetry for all telemetry data collection and use it to generate enriched logs.
2. Choose a unified storage solution: Select a platform that can efficiently store and query metrics, logs, traces and enriched logs together
3. Enrich your telemetry: Update application instrumentation to include relevant context
4. Create correlation IDs: Ensure every request has identifiers
5. Implement semantic conventions: Follow consistent naming patterns across your telemetry data
6. Start with focused use cases: Begin with high-value scenarios like checkout flows or critical APIs
7. Leverage GenAI tools: Integrate tools that can analyze your unified data and respond to natural language queries

Remember, AI can only be as smart as the data you feed it. The quality and completeness of your telemetry data will determine the effectiveness of your AI-powered observability.

Generative AI: an evolutionary catalyst for observability

The unification of observability data for GenAI analysis represents an evolutionary leap forward comparable to the transition from Internet 1.0 to 2.0. Early adopters will gain a significant competitive advantage through faster problem resolution, improved system reliability, and more efficient operations. GAI is a huge step for increasing observability maturity and moving your team to a more proactive stance.

Think of traditional observability as a doctor trying to diagnose a patient while only able to see their heart rate. Unified observability with GenAI is like giving that doctor a complete health picture, vital signs, lab results, medical history, and genetic data all accessible through natural conversation.

As SREs, we stand at the threshold of a new era in system observability. The asteroid of GenAI isn't a threat to be feared, it's an opportunity to evolve our practices and tools to build more reliable, understandable systems. The question isn't whether this transformation will happen, but who will lead it.

Will you?

In managing NGINX Elastic provides several capabilities:

1. Easy ingest, parsing, and out-of-the-box dashboards. Check out the simple how-to in our docs. Based on logs, these dashboards show several items over time, response codes, errors, top pages, data volume, browsers used, active connections, drop rates, and much more.

2. Out-of-the-box ML-based anomaly detection jobs for your NGINX logs. These jobs help pinpoint anomalies against request rates, IP address request rates, URL access, status codes, and visitor rate anomalies.

3. ES|QL which helps work through logs and build out charts during analysis.

4. Elastic’s GenAI Assistant provides a simple natural language interface that helps analyze all the logs and can pull out issues from ML jobs and even create dashboards. The Elastic AI Assistant also automatically uses ES|QL.

5. NGINX SLOs - Finally Elastic provides the ability to define and monitor SLOs for your NGINX logs. While most SLOs are metrics-based, Elastic allows you to create logs-based SLOs. We detailed this in a previous blog.

NGINX logs are another example of why logs are great.  Logging is an important part of Observability, for which we generally think of metrics and tracing. However, the amount of logs an application and the underlying infrastructure output can be significantly daunting and NGINX is usually the starting point for most analyses. 

In today’s blog, we’ll cover how the out-of-the-box ML-based anomaly detection jobs can help RCA, and how Elastic’s GenAI Assistant helps easily work through logs to pinpoint issues in minutes. 

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).

• Bring up an NGINX server on a host. OR run an application with NGINX as a front end and drive traffic.

• Install the NGINX integration and assets and review the dashboards as noted in the docs.

• Ensure you have an ML node configured in your Elastic stack

• To use the AI Assistant you will need a trial or upgrade to Platinum.

In our scenario, we use data from 3 months from our Elastic environment to help highlight the features. Hence you might need to run your application with traffic for a specific time frame to follow along.

Analyzing the issues with AI Assistant

As detailed in a previous blog, you can get alerted on issues via SLO monitoring against NGINX logs. Let’s assume you have an SLO based on status codes as we outlined in the previous blog. You can immediately analyze the issue via the AI Assistant. Because it's a chat interface we simply open the AI Assistant and work through some simple analysis: (See Animated GIF for a demo)

AI Assistant analysis:

• Using lens graph all http response status codes < 400 and > =400 from filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer - We wanted to simply understand the amount of requests resulting in status code >= 400 and graph the results. We see that 15% of the requests were not successful, hence an SLO alert being triggered.

• Which ip address (field source.adress) has the highest number of http.response.status.code >= 400 from filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer  - We were curious is there was a specific IP address not having successful requests. 72.57.0.53, with a count of 25,227 occurrences is daily high but not the ensure 2 failed requests.

• What country (source.geo.country_iso_code) is source.address=72.57.0.53 coming from. Use filebeat-nginx-elasticco-anon-2017. - Again we were curious if this came from a specific country. And the IP address 72.57.0.53 is coming from the country with the ISO code IN, which corresponds to India. Nothing out of the ordinary.

• Did source.address=72.57.0.53 have any (http.response.status.code < 400) from filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer -  Oddly the IP address in question only had 4000+ successful responses. Meaning its not malicious, and points to something else.

• What are the different status codes (http.response.status.code>=400), from source.address=72.57.0.53. Use filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer. Provide counts for each status code - We are curious whether or not we see any 502, which there were none, but most of the failures were 404. 

• What are the different status codes (http.response.status.code>=400). Use filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer. Provide counts for each status code - Regardless of a specific address, what is the largest number of status code occurrences > 400. This also points to 404. 

• What does a high 404 count from a specific IP address mean from NGINX logs? - Asking this question, we need to understand the potential causes of this from our application. From the answers, we can rule out security probing and web scraping, as we validated that a specific address 72.57.0.53 has a low non-success request status code. It also rules out User error. Hence this points potentially to Broken Links or Missing Resources.

Watch the flow:

Potential issue:

It seems that we potentially have an issue with the backend serving specific answers or having issues with resources (database, or broken links). This is cursing the higher-than-normal non-successful status codes>=400.

Key highlights from AI Assistant:

As you watched this video you will notice a few things:

1. We analyzed millions of logs in a matter of minutes using a set of simple natural language queries. 

2. We didn’t need to know any special query language. The AI Assistant used Elastic’s ES|QL but can similarly use KQL also. 

3. The AI Assistant easily builds out graphs

4. The AI Assistant is accessing and using internal information stored in Elastic’s indices. Vs a simple “google foo” based AI Assistant. This is enabled through RAG, and the AI Assistant can also bring up known issues in github, runbooks, and other useful internal information.

Check out the following blog on how the AI Assistant uses RAG to retrieve internal information. Specifically using github and runbooks.

Locating anomalies with ML

While using the AI Assistant is great for analyzing information, another important aspect of NGINX log management is to ensure you can manage log spikes and anomalies. Elastic has a machine learning platform that allows you to develop jobs to analyze specific metrics or multiple metrics to look for anomalies.When using NGINX, there are several out-of-the-box anomaly detection jobs. These work specifically on NGINX access logs.

• Low_request_rate_nginx - Detect low request rates

• Source_ip_request_rate_nginx - Detect unusual source IPs - high request rates

• Source_ip_url_count_nginx - Detect unusual source IPs - high distinct count of URLs

• Status_code_rate_nginx - Detect unusual status code rates

• Visitor_rate_nginx - Detect unusual visitor rates

Being right out of the box, lets look at the job - Status_code_rate_nginx, which is related to our previous analysis.

With a few simple clicks we immediately get an analysis showing a specific IP address - 72.57.0.53, having higher than normal non-successful requests. Oddly we also found this is using the AI Assistant.

We can take this further with conversations with the AI Assistant, look at the logs, and/or even look at the other ML anomaly jobs.

Conclusion:

You’ve now seen how easily Elastic’s RAG-based AI Assistant can help analyze NGINX logs without even the need to know query syntax, understand where the data is, and understand even the fields. Additionally, you’ve also seen how we can alert you when a potential issue or degradation in service (SLO). 

Check out other resources on NGINX logs:

Out-of-the-box anomaly detection jobs for NGINX

Using the NGINX integration to ingest and analyze NGINX Logs

NGINX Logs based SLOs in Elastic

Using GitHub issues, runbooks, and other internal information for RCAs with Elastic’s RAG based AI Assistant

Try it out

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on the cloud? Start a free trial.

All of this is also possible in your environment. Learn how to get started today.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

1. Where traffic is coming in from and going out to from the deployment, and within the deployment. This helps identify unusual or unauthorized communications

2. Traffic volumes detecting spikes or drops which could indicate service issues in production or an increase in customer traffic

3. Latency and Performance bottlenecks - with VPC Flow logs, you can look at latency for a flow (in and outflows), and understand patterns

4. Accepted and rejected traffic helps determine where potential security threats and misconfigurations lie. 

AWS VPC Logs is a great example of how logs are great. Logging is an important part of Observability, for which we generally think of metrics and tracing. However, the amount of logs an application and the underlying infrastructure output can be significantly daunting with VPC Logs. However, it also provides a significant amount of insight.

Before we proceed, it is important to understand what Elastic provides in managing AWS and VPC Flow logs:

1. A full set of integrations to manage VPC Flows and the entire end-to-end deployment on AWS. 

2. Elastic has a simple-to-use AWS Firehose integration. 

3. Elastic’s tools such as Discover, spike analysis,  and anomaly detection help provide you with better insights and analysis.

4. And a set of simple Out-of-the-box dashboards

In today’s blog, we’ll cover how Elastics’ other features can support analyzing and RCA for potential VPC flow logs even more easily. Specifically, we will focus on managing the number of rejects, as this helps ensure there weren’t any unauthorized or unusual activities:

1. Set up an easy-to-use SLO (newly released) to detect when things are potentially degrading

2. Create an ML job to analyze different fields of the VPC Flow log

3. Using our newly released RAG-based AI Assistant to help analyze the logs without needing to know Elastic’s query language nor how to even graph on Elastic

4. ES|QL will help understand and analyze add latency for patterns.

In subsequent blogs, we will use AI Assistant and ESQL to show how to get other insights beyond just REJECT/ACCEPT from VPC Flow log.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).

• Follow the steps in the following blog to get AWS’s three-tier app installed instructed in git, and bring in the AWS VPC Flow logs.

• Ensure you have an ML node configured in your Elastic stack

• To use the AI Assistant you will need a trial or upgrade to Platinum.

SLO with VPC Flow Logs

Elastic’s SLO capability is based directly on the Google SRE Handbook. All the definitions and semantics are utilized as described in Google’s SRE handbook. Hence users can perform the following on SLOs in Elastic:

• Define an SLO on Logs not just metrics - Users can use KQL (log-based query), service availability, service latency, custom metric, histogram metric, or a timeslice metric.
• Define SLO, SLI, Error budget and burn rates. Users can also use occurrence versus time slice-based budgeting. 
• Manage, with dashboards, all the SLOs in a singular location.
• Trigger alerts from the defined SLO, whether the SLI is off, the burn rate is used up, or the error rate is X.

Setting up an SLO for VPC is easy. You simply create a query you want to trigger off. In our case, we look for all the good events where aws.vpcflow.action=ACCEPT and we define the target at 85%. 

As the following example shows, over the last 7 days, we have exceeded our budget by 43%. Additionally, we have not complied for the last 7 days.

Analyzing the SLO with AI Assistant

Now that we see that there is an issue with the VPC Flows, we immediately work with the AI Assistant to start analyzing the SLO. Because it's a chat interface we simply open the AI Assistant and work through some simple analysis: (See Animated GIF for a demo below)

AI Assistant analysis:

• what were the top 3 source.address that had aws.vpcflow.action=REJECT over the last 7 days, which is causing this SLO issue? - We wanted to simply see what could be causing the loss in error budget. Were there any particular source.addresses causing a heavy reject rate.

• The answer: A table with the highest count = 42670 and source.address = 79.110.62.185

• There is one singular source.address that is causing the loss in SLO. 

• What is the largest number of  aws.vpcflow.action=REJECT in a 30 min time frame for the last 3 days where the source.address=79.110.62.185? - After understanding that a specific source.address is causing the loss in SLO, we want to understand the averages. 

• **The answer: ** "The largest number of aws.vpcflow.action=REJECT in a 30-minute time frame for the last 3 days where the source.address is 79.110.62.185 is 229. This occurred on 2024-06-01T04:00:00.000Z.”

• It means there must be a low REJECT rate but fairly consistent vs spiky over the last 7 days.

• for the logs with source.address="79.110.62.185" was there any country code of source.geo.country_iso_code field present. If yes what is the value - Given the last question showed a low REJECT rate, it only means that this was fairly consistent vs spiky over the last 7 days.

• The answer: Yes, there is a country code present in the source.geo.country_iso_code field for logs with source.address="79.110.62.185". The value is BG (Bulgaria).

• Is there a specific destination.address where source.address=79.110.62.185 is getting a aws.vpcflow.action=REJECT. Give me both the destination.address and the number of REJECTs for that destination.address?

• The answer: destination.address of 10.0.0.27 is giving a reject number of 53433 in this time frame.

• Graph the number of REJECT vs ACCEPT for source.address="79.110.62.185" over the last 7 days. The graph is on a daily basis in a singular graph - We asked this question to see what the comparison is between ACCEPT and REJECT. 

• The answer: See the animated GIF to see that the generated graph is fairly stable

• Were there any source.address that had a spike, high reject rate in. a 30min period over the 30 days? - We wanted to see if there was any other spike 

• The answer - Yes, there was a source.address that had a spike in high reject rates in a 30-minute period over the last 30 days. source.address: 185.244.212.67, Reject Count: 8975, Time Period: 2024-05-22T03:00:00.000Z

Watch the flow

Potential issue:

he server handling requests from source 79.110.62.185 is potentially having an issue.

Again using logs, we essentially asked the AI Assistant to give the eni ids where the internal ip address was 10.0.0.27

From our AWS console, we know that this is the webserver. Further analysis in Elastic, and with the developers we realized there is a new version that was installed recently causing a problem with connections.

Locating anomalies with ML

While using the AI Assistant is great for analyzing information, another important aspect of VPC flow management is to ensure you can manage log spikes and anomalies. Elastic has a machine learning platform that allows you to develop jobs to analyze specific metrics or multiple metrics to look for anomalies.

VPC Flow logs come with a large amount of information. The full set of fields is listed in AWS docs. We will use a specific subset to help detect anomalies.

We were setting up anomalies for aws.vpcflow.action=REJECT, which requires us to use multimetric anomaly detection in Elastic.

The config we used utilizes:

Detectors:

• destination.address

• destination.port

Influencers:

• source.address

• aws.vpcflow.action

• destination.geo.region_iso_code

The way we set this up will help us understand if there is a large spike in REJECT/ACCEPT against destination.address values from a specific source.address and/or destination.geo.region_iso_code location.

The job once run reveals something interesting:

Notice that source.address 185.244.212.67 has had a high REJECT rate in the last 30 days. 

Notice where we found this before? In the AI Assistant!!!!!

While we can run the AI Assistant and find this sort of anomaly, the ML job can be setup to run continuously and alert us on such spikes. This will help us understand if there are any issues with the webserver like we found above or even potential security attacks.

Conclusion:

You’ve now seen how easily Elastic’s RAG-based AI Assistant can help analyze VPC Flows without even the need to know query syntax, understand where the data is, and understand even the fields. Additionally, you’ve also seen how we can alert you when a potential issue or degradation in service (SLO). Check out our other blogs on AWS VPC Flow analysis in Elastic:

1. A full set of integrations to manage VPC Flows and the entire end-to-end deployment on AWS. 

2. Elastic has a simple-to-use AWS Firehose integration. 

3. Elastic’s tools such as Discover, spike analysis,  and anomaly detection help provide you with better insights and analysis.

4. And a set of simple Out-of-the-box dashboards

Try it out

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on the cloud? Start a free trial.

All of this is also possible in your environment. Learn how to get started today.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

You might have heard the term Agent pop up recently in different open source projects or vendors focusing their go-to-market on GenAI. Indeed, while most GenAI applications are focused on RAG applications today, there is an increasing interest in isolating tasks that could be achieved with a more special model into what is called an Agent.

To be clear, an agent will be given a task, which could be a prompt, and execute the task by leveraging other models, data sources, and a knowledge base. Depending on the field of application, the results should ultimately look like generated text, pictures, charts, or sounds.

Now, what the multi-Agent Architecture, is the process of leveraging multiple agents around a given task by:

• Orchestrating complex system oversight with multiple agents
• Analyzing and strategizing in real-time with strategic reasoning
• Specializing agents, tasks are decomposed into smaller focused tasks into expert-handled elements
• Sharing insights for cohesive action plans, creating collaborative dynamics

In a nutshell, multi-agent architecture's superpower is tackling intricate challenges beyond human speed and solving complex problems. It enables a couple of things:

• Scale the intelligence as the data and complexity grows. The tasks are decomposed into smaller work units, and the expert network grows accordingly.
• Coordinate simultaneous actions across systems, scale collaboration
• Evolving with data allows continuous adaptation with new data for cutting-edge decision-making.
• Scalability, high performance, and resilience

Single Agent Vs Multi-Agent Architecture

Before double-clicking on the multi-agent architecture, let’s talk about the single-agent architecture. The single-agent architecture is designed for straightforward tasks and a late feedback loop from the end user. There are multiple single-agent frameworks such as ReAct (Reason+Act), RAISE (ReAct+ Short/Long term memory), Reflexion, AutoGPT+P, and LATS (Language Agent Tree Search). The general process these architectures enable is as follows:

The Agent takes action, observes, executes, and self-decides whether or not it looks complete, ends the process if finished, or resubmits the new results as an input action, the process keeps going.

While simple tasks are ok with this type of agent, such as a RAG application where a user will ask a question, and the agent returns an answer based on the LLM and a knowledge base, there are a couple of limitations:

• Endless execution loop: the agent is never satisfied with the output and reiterates.
• Hallucinations
• Lack of feedback loop or enough data to build a feedback loop
• Lack of planning

For these reasons, the need for a better self-evaluation loop, externalizing the observation phase, and division of labor is rising, creating the need for a multi-agent architecture.

Multi-agent architecture relies on taking a complex task, breaking it down into multiple smaller tasks, planning the resolution of these tasks, executing, evaluating, sharing insights, and delivering an outcome. For this, there is more than one agent; in fact, the minimum value for the network size N is N=2 with:

• A Manager
• An Expert

When N=2, the source task is simple enough only to need one expert agent as the task can not be broken down into multiple tasks. Now, when the task is more complex, this is what the architecture can look like:

With the help of an LLM, the Manager decomposes the tasks and delegates the resolutions to multiple agents. The above architecture is called Vertical since the agents directly send their results to the Manager. In a horizontal architecture, agents work and share insight together as groups, with a volunteer-based system to complete a task, they do not need a leader as shown below:

A very good paper covering these two architectures with more insights can be found here: https://arxiv.org/abs/2404.11584

Application Vertical Multi-Agent Architecture to Observability

Vertical Multi-Agent Architecture can have a manager, experts, and a communicator. This is particularly important when these architectures expose the task's result to an end user.

In the case of Observability, what we envision in this blog post is the scenario of an SRE running through a Root Cause Analysis (RCA) process. The high-level logic will look like this:

• Communicator:
  • Read the initial command from the Human
  • Pass command to Manager
  • Provide status updates to Human
  • Provide a recommended resolution plan to the Human
  • Relay follow-up commands from Human to Manager
• Manager:
  • Read the initial command from the Communicator

  • Create working group

  • Assign Experts to group

  • Evaluate signals and recommendations from Experts

  • Generate recommended resolution plan

  • Execute plan (optional)
• Expert:
  • Each expert task with singular expertise tied to Elastic integration

  • Use o11y AI Assistant to triage and troubleshoot data related to their expertise

  • Work with other Experts as needed to correlate issues

  • Provide recommended root cause analysis for their expertise (if applicable)

  • Provide recommended resolution plan for their expertise (if applicable)

We believe that breaking down the experts by integration provides enough granularity in the case of observability and allows them to focus on a specific data source. Doing this also gives the manager a breakdown key when receiving a complex incident involving multiple data layers (application, network, datastores, infrastructures).

For example, a complex task initiated by an alert in an e-commerce application could be “Revenue dropped by 30% in the last hour.” This task would be submitted to the manager, who will look at all services, applications, datastores, network components, and infrastructure involved and decompose these into investigation tasks. Each expert would investigate within their specific scope and provide observations to the manager. The manager will be responsible for correlating and providing observations on what caused the problem.

Core Architecture

In the above example, we have decided to deploy the architecture on the below software architecture:

• The agent manager and expert agent are deployed on GCP or your favorite cloud provider
• Most of the components are written in Python
• A task management layer is necessary to queue the task to the expert
• Expert agents are specifically deployed by integration/data source and converse with the Elastic AI Assistant deployed in Kibana.
• The AI Assistant can access a real-time context to help the expert resolve their task.
• Elasticsearch is used as the AI Assistant context and as the expert memory to build its experience.
• The backend LLM here is GPT-4, now GTP-4o, running on Azure.

Agent Experience

Agent experience is built based on previous events stored in Elasticsearch, to which the expert can look semantically for similar events. When they find one, they get the execution path stored in memory to execute it.

The beauty of using the Elasticsearch Vector Database for this is the semantic query the agent will be able to execute against the memory and how the memory itself can be managed. Indeed, there is a notion of short—and long-term memory that could be very interesting in the case of observability, where some events often happen and probably worth to be stored in the short-term memory because they are questioned more often. Less queried but important events can be stored in a longer-term memory with more cost-effective hardware.

The other aspect of the Agent Experience is the semantic reranking feature with Elasticsearch. When the agent executes a task, reranking is used to surface the best outcome compared to past experience:

If you are looking for a working example of the above, check this blog post where 2 agents are working together with the Elastic Observability AI Assistant on an RCA:

Enhanced troubleshooting for SREs

Site reliability engineers (SRE) in large organizations often face challenges in locating necessary information for troubleshooting alerts, monitoring systems, or deriving insights due to scattered and potentially outdated resources. This issue is particularly significant for less experienced SREs who may require assistance even with the presence of a runbook. Recurring incidents pose another problem, as the on-call individual may lack knowledge about previous resolutions and subsequent steps. Mature SRE teams often invest considerable time in system improvements to minimize "fire-fighting," utilizing extensive automation and documentation to support on-call personnel.

Elastic® addresses these challenges by combining generative AI models with relevant search results from your internal data using RAG. The Observability AI Assistant's internal Knowledge Base, powered by our semantic search retrieval model ELSER, can recall information at any point during a conversation, providing RAG responses based on internal knowledge.

This Knowledge Base can be enriched with your organization's information, such as runbooks, GitHub issues, internal documentation, and Slack messages, allowing the AI Assistant to provide specific assistance. The Assistant can also document and store specific information from an ongoing conversation with an SRE while troubleshooting issues, effectively creating runbooks for future reference. Furthermore, the Assistant can generate summaries of incidents, system status, runbooks, post-mortems, or public announcements.

This ability to retrieve, summarize, and present contextually relevant information is a game-changer for SRE teams, transforming the work from chasing documents and data to an intuitive, contextually sensitive user experience.The Knowledge Base (see requirements) serves as a central repository of Observability knowledge, breaking documentation silos and integrating tribal knowledge, making this information accessible to SREs enhanced with the power of LLMs.

Your LLM provider may collect query telemetry when using the AI Assistant. If your data is confidential or has sensitive details, we recommend you verify the data treatment policy of the LLM connector you provided to the AI Assistant.

In this blog post, we will cover different ways to enrich your Knowledge Base (KB) with internal information. We will focus on a specific alert, indicating that there was an increase in logs with “502 Bad Gateway” errors that has surpassed the alert’s threshold.

How to troubleshoot an alert with the Knowledge Base

Before the KB has been enriched with internal information, when the SRE asks the AI Assistant about how to troubleshoot an alert, the response from the LLM will be based on the data it learned during training; however, the LLM is not able to answer questions related to private, recent, or emerging knowledge. In this case, when asking for the steps to troubleshoot the alert, the response will be based on generic information.

However, once the KB has been enriched with your runbooks, when your team receives a new alert on “502 Bad Gateway” Errors, they can use AI Assistant to access the internal knowledge to troubleshoot it, using semantic search to find the appropriate runbook in the Knowledge Base.

In this blog, we will cover different ways to add internal information on how to troubleshoot an alert to the Knowledge Base:

1. Ask the assistant to remember the content of an existing runbook.

2. Ask the Assistant to summarize and store in the Knowledge Base the steps taken during a conversation and store it as a runbook.

3. Import your runbooks from GitHub or another external source to the Knowledge Base using our Connector and APIs.

After the runbooks have been added to the KB, the AI Assistant is now able to recall the internal and specific information in the runbooks. By leveraging the retrieved information, the LLM could provide more accurate and relevant recommendations for troubleshooting the alert. This could include suggesting potential causes for the alert, steps to resolve the issue, preventative measures for future incidents, or asking the assistant to help execute the steps mentioned in the runbook using functions. With more accurate and relevant information at hand, the SRE could potentially resolve the alert more quickly, reducing downtime and improving service reliability.

Your Knowledge Base documents will be stored in the indices .kibana-observability-ai-assistant-kb-*. Have in mind that LLMs have restrictions on the amount of information the model can read and write at once, called token limit. Imagine you're reading a book, but you can only remember a certain number of words at a time. Once you've reached that limit, you start to forget the earlier words you've read. That's similar to how a token limit works in an LLM.

To keep runbooks within the token limit for Retrieval Augmented Generation (RAG) models, ensure the information is concise and relevant. Use bullet points for clarity, avoid repetition, and use links for additional information. Regularly review and update the runbooks to remove outdated or irrelevant information. The goal is to provide clear, concise, and effective troubleshooting information without compromising the quality due to token limit constraints. LLMs are great for summarization, so you could ask the AI Assistant to help you make the runbooks more concise.

Ask the assistant to remember the content of an existing runbook

The easiest way to store a runbook into the Knowledge Base is to just ask the AI Assistant to do it! Open a new conversation and ask “Can you store this runbook in the KB for future reference?” followed by pasting the content of the runbook in plain text.

The AI Assistant will then store it in the Knowledge Base for you automatically, as simple as that.

Ask the Assistant to summarize and store the steps taken during a conversation in the Knowledge Base

You can also ask the AI Assistant to remember something while having a conversation — for example, after you have troubleshooted an alert using the AI Assistant, you could ask to "remember how to troubleshoot this alert for next time." The AI Assistant will create a summary of the steps taken to troubleshoot the alert and add it to the Knowledge Base, effectively creating runbooks for future reference. Next time you are faced with a similar situation, the AI Assistant will recall this information and use it to assist you.

In the following demo, the user asks the Assistant to remember the steps that have been followed to troubleshoot the root cause of an alert, and also to ping the Slack channel when this happens again. In a later conversation with the Assistant, the user asks what can be done about a similar problem, and the AI Assistant is able to remember the steps and also reminds the user to ping the Slack channel.

After receiving the alert, you can open the AI Assistant chat and test troubleshooting the alert. After investigating an alert, ask the AI Assistant to summarize the analysis and the steps taken to root cause. To remember them for the next time, we have a similar alert and add extra instruction like to warn the Slack channel.

The Assistant will use the built-in functions to summarize the steps and store them into your Knowledge Base, so they can be recalled in future conversations.

Open a new conversation, and ask what are the steps to take when troubleshooting a similar alert to the one we just investigated. The Assistant will be able to recall the information stored in the KB that is related to the specific alert, using semantic search based on ELSER, and provide a summary of the steps taken to troubleshoot it, including the last indication of informing the Slack channel.

Import your runbooks stored in GitHub to the Knowledge Base using APIs or our GitHub Connector

You can also add proprietary data into the Knowledge Base programmatically by ingesting it (e.g., GitHub Issues, Markdown files, Jira tickets, text files) into Elastic.

If your organization has created runbooks that are stored in Markdown documents in GitHub, follow the steps in the next section of this blog post to index the runbook documents into your Knowledge Base.

The steps to ingest documents into the Knowledge Base are the following:

Ingest your organization’s knowledge into Elasticsearch

Option 1: Use the Elastic web crawler . Use the web crawler to programmatically discover, extract, and index searchable content from websites and knowledge bases. When you ingest data with the web crawler, a search-optimized Elasticsearch® index is created to hold and sync webpage content.

Option 2: Use Elasticsearch's Index API . Watch tutorials that demonstrate how you can use the Elasticsearch language clients to ingest data from an application.

Option 3: Build your own connector. Follow the steps described in this blog: How to create customized connectors for Elasticsearch.

Option 4: Use Elasticsearch Workplace Search connectors . For example, the GitHub connector can automatically capture, sync, and index issues, Markdown files, pull requests, and repos.

• Follow the steps to configure the GitHub Connector in GitHub to create an OAuth App from the GitHub platform.

• Now you can connect a GitHub instance to your organization. Head to your organization’s Search > Workplace Search administrative dashboard, and locate the Sources tab.

• Select GitHub (or GitHub Enterprise) in the Configured Sources list, and follow the GitHub authentication flow as presented. Upon the successful authentication flow, you will be redirected to Workplace Search and will be prompted to select the Organization you would like to synchronize.

• After configuring the connector and selecting the organization, the content should be synchronized and you will be able to see it in Sources. If you don’t need to index all the available content, you can specify the indexing rules via the API. This will help shorten indexing times and limit the size of the index. See Customizing indexing.

• The source has created an index in Elastic with the content (Issues, Markdown Files…) from your organization. You can find the index name by navigating to Stack Management > Index Management , activating the Include hidden Indices button on the right, and searching for “GitHub.”

• You can explore the documents you have indexed by creating a Data View and exploring it in Discover. Go to Stack Management > Kibana > Data Views > Create data view and introduce the data view Name, Index pattern (make sure you activate “Allow hidden and system indices” in advanced options), and Timestamp field:

• You can now explore the documents in Discover using the data view:

Reindex your internal runbooks into the AI Assistant’s Knowledge Base Index, using it's semantic search pipeline

Your Knowledge Base documents are stored in the indices .kibana-observability-ai-assistant-kb-*. To add your internal runbooks imported from GitHub to the KB, you just need to reindex the documents from the index you created in the previous step to the KB’s index. To add the semantic search capabilities to the documents in the KB, the reindex should also use the ELSER pipeline preconfigured for the KB, .kibana-observability-ai-assistant-kb-ingest-pipeline.

By creating a Data View with the KB index, you can explore the content in Discover.

You execute the query below in Management > Dev Tools , making sure to replace the following, both on “_source” and “inline”:

• InternalDocsIndex : name of the index where your internal docs are stored
• text_field : name of the field with the text of your internal docs
• timestamp : name of the field of the timestamp in your internal docs
• public : (true or false) if true, makes a document available to all users in the defined Kibana Space (if is defined) or in all spaces (if is not defined); if false, document will be restricted to the user indicated in
• (optional) space : if defined, restricts the internal document to be available in a specific Kibana Space
• (optional) user.name : if defined, restricts the internal document to be available for a specific user
• (optional) "query" filter to index only certain docs (see below)

POST _reindex
{
    "source": {
        "index": "<InternalDocsIndex>",
        "_source": [
            "<text_field>",
            "<timestamp>",
            "namespace",
            "is_correction",
            "public",
            "confidence"
        ]
    },
    "dest": {
        "index": ".kibana-observability-ai-assistant-kb-000001",
        "pipeline": ".kibana-observability-ai-assistant-kb-ingest-pipeline"
    },
    "script": {
        "inline": "ctx._source.text=ctx._source.remove(\"<text_field>\");ctx._source.namespace=\"<space>\";ctx._source.is_correction=false;ctx._source.public=<public>;ctx._source.confidence=\"high\";ctx._source['@timestamp']=ctx._source.remove(\"<timestamp>\");ctx._source['user.name'] = \"<user.name>\""
    }
}

You may want to specify the type of documents that you reindex in the KB — for example, you may only want to reindex Markdown documents (like Runbooks). You can add a “query” filter to the documents in the source. In the case of GitHub, runbooks are identified with the “type” field containing the string “file,” and you could add that to the reindex query like indicated below. To add also GitHub Issues, you can also include in the query “type” field containing the string “issues”:

"source": {
        "index": "<InternalDocsIndex>",
        "_source": [
            "<text_field>",
            "<timestamp>",
            "namespace",
            "is_correction",
            "public",
            "confidence"
        ],
    "query": {
      "terms": {
        "type": ["file"]
      }
    }

Great! Now that the data is stored in your Knowledge Base, you can ask the Observability AI Assistant any questions about it:

Conclusion

In conclusion, leveraging internal Observability knowledge and adding it to the Elastic Knowledge Base can greatly enhance the capabilities of the AI Assistant. By manually inputting information or programmatically ingesting documents, SREs can create a central repository of knowledge accessible through the power of Elastic and LLMs. The AI Assistant can recall this information, assist with incidents, and provide tailored observability to specific contexts using Retrieval Augmented Generation. By following the steps outlined in this article, organizations can unlock the full potential of their Elastic AI Assistant.

Start enriching your Knowledge Base with the Elastic AI Assistant today and empower your SRE team with the tools they need to excel. Follow the steps outlined in this article and take your incident management and alert remediation processes to the next level. Your journey toward a more efficient and effective SRE operation begins now.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

As a single platform, Elastic® helps SREs unify and correlate limitless telemetry data, including metrics, logs, traces, and profiling, into a single datastore — Elasticsearch®. By then applying the power of Elastic’s advanced machine learning (ML), AIOps, AI Assistant, and analytics, you can break down silos and turn data into insights. As a full-stack observability solution, everything from infrastructure monitoring to log monitoring and application performance monitoring (APM) can be found in a single, unified experience.

In Elastic 8.11, a technical preview is now available of Elastic’s new piped query language, ES|QL (Elasticsearch Query Language), which transforms, enriches, and simplifies data investigations. Powered by a new query engine, ES|QL delivers advanced search capabilities with concurrent processing, improving speed and efficiency, irrespective of data source and structure. Accelerate resolution by creating aggregations and visualizations from one screen, delivering an iterative, uninterrupted workflow.

Advantages of ES|QL for SREs

SREs using Elastic Observability can leverage ES|QL to analyze logs, metrics, traces, and profiling data, enabling them to pinpoint performance bottlenecks and system issues with a single query. SREs gain the following advantages when managing high dimensionality and high cardinality data with ES|QL in Elastic Observability:

• Improved operational efficiency: By using ES|QL, SREs can create more actionable notifications with aggregated values as thresholds from a single query, which can also be managed through the Elastic API and integrated into DevOps processes.
• Enhanced analysis with insights: ES|QL can process diverse observability data, including application, infrastructure, business data, and more, regardless of the source and structure. ES|QL can easily enrich the data with additional fields and context, allowing the creation of visualizations for dashboards or issue analysis with a single query.
• Reduced mean time to resolution: ES|QL, when combined with Elastic Observability's AIOps and AI Assistant, enhances detection accuracy by identifying trends, isolating incidents, and reducing false positives. This improvement in context facilitates troubleshooting and the quick pinpointing and resolution of issues.

ES|QL in Elastic Observability not only enhances an SRE's ability to manage the customer experience, an organization's revenue, and SLOs more effectively but also facilitates collaboration with developers and DevOps by providing contextualized aggregated data.

In this blog, we will cover some of the key use cases SREs can leverage with ES|QL:

• ES|QL integrated with the Elastic AI Assistant, which uses public LLM and private data, enhances the analysis experience anywhere in Elastic Observability.
• SREs can, in a single ES|QL query, break down, analyze, and visualize observability data from multiple sources and across any time frame.
• Actionable alerts can be easily created from a single ES|QL query, enhancing operations.

I will work through these use cases by showcasing how an SRE can solve a problem in an application instrumented with OpenTelemetry and running on Kubernetes. The OpenTelemetry (OTel) demo is on an Amazon EKS cluster, with Elastic Cloud 8.11 configured.

You can also check out our Elastic Observability ES|QL Demo, which walks through ES|QL functionality for Observability.

ES|QL with AI Assistant

As an SRE, you are monitoring your OTel instrumented application with Elastic Observability, and while in Elastic APM, you notice some issues highlighted in the service map.

Using Elastic AI Assistant, you can easily ask for analysis, and in particular, we check on what the overall latency is across the application services.

My APM data is in traces-apm*. What's the average latency per service over the last hour? Use ESQL, the data is mapped to ECS

The Elastic AI Assistant generates an ES|QL query, which we run in the AI Assistant to get a list of the average latencies across all the application services. We can easily see the top four are:

• load generator
• front-end proxy
• frontendservice
• checkoutservice

With a simple natural language query in the AI Assistant, it generated a single ES|QL query that helped list out the latencies across the services.

Noticing that there is an issue with several services, we decide to start with the frontend proxy. As we work through the details, we see significant failures, and through Elastic APM failure correlation , it becomes apparent that the frontend proxy is not properly completing its calls to downstream services.

ES|QL insightful and contextual analysis in Discover

Knowing that the application is running on Kubernetes, we investigate if there are issues in Kubernetes. In particular, we want to see if there are any services having issues.

We use the following query in ES|QL in Elastic Discover:

from metrics-* | where kubernetes.container.status.last_terminated_reason != "" and kubernetes.namespace == "default" | stats reason_count=count(kubernetes.container.status.last_terminated_reason) by kubernetes.container.name, kubernetes.container.status.last_terminated_reason | where reason_count > 0

ES|QL helps analyze 1,000s/10,000s of metric events from Kubernetes and highlights two services that are restarting due to OOMKilled.

The Elastic AI Assistant, when asked about OOMKilled, indicates that a container in a pod was killed due to an out-of-memory condition.

We run another ES|QL query to understand the memory usage for emailservice and productcatalogservice.

ES|QL easily found the average memory usage fairly high.

We can now further investigate both of these services’ logs, metrics, and Kubernetes-related data. However, before we continue, we create an alert to track heavy memory usage.

Actionable alerts with ES|QL

Suspecting a specific issue, that might recur, we simply create an alert that brings in the ES|QL query we just ran that will track for any service that exceeds 50% in memory utilization.

We modify the last query to find any service with high memory usage:

FROM metrics*
| WHERE @timestamp >= NOW() - 1 hours
| STATS avg_memory_usage = AVG(kubernetes.pod.memory.usage.limit.pct) BY kubernetes.deployment.name | where avg_memory_usage > .5

With that query, we create a simple alert. Notice how the ES|QL query is brought into the alert. We simply connect this to pager duty. But we can choose from multiple connectors like ServiceNow, Opsgenie, email, etc.

With this alert, we can now easily monitor for any services that exceed 50% memory utilization in their pods.

Make the most of your data with ES|QL

In this post, we demonstrated the power ES|QL brings to analysis, operations, and reducing MTTR. In summary, the three use cases with ES|QL in Elastic Observability are as follows:

• ES|QL integrated with the Elastic AI Assistant, which uses public LLM and private data, enhances the analysis experience anywhere in Elastic Observability.
• SREs can, in a single ES|QL query, break down, analyze, and visualize observability data from multiple sources and across any time frame.
• Actionable alerts can be easily created from a single ES|QL query, enhancing operations.

Elastic invites SREs and developers to experience this transformative language firsthand and unlock new horizons in their data tasks. Try it today at https://ela.st/free-trial now in technical preview.

• Elastic Observability Tour
• The power of effective log management
• Transforming Observability with the AI Assistant
• ES|QL announcement blog

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Preconfigured machine learning models for observability and security are available. If those don't work well enough on your data, in-tool wizards guide you through the few steps needed to configure custom anomaly detection and train your model with supervised learning. To get you started, there are several key features built into Elastic Observability to aid in analysis, bypassing the need to run specific ML models. These features help minimize the time and analysis of logs.

Let’s review the set of machine learning-based observability features in Elastic:

Anomaly detection: Elastic Observability, when turned on (see documentation), automatically detects anomalies by continuously modeling the normal behavior of your time series data — learning trends, periodicity, and more — in real time to identify anomalies, streamline root cause analysis, and reduce false positives. Anomaly detection runs in and scales with Elasticsearch and includes an intuitive UI.

Log categorization: Using anomaly detection, Elastic also identifies patterns in your log events quickly. Instead of manually identifying similar logs, the logs categorization view lists log events that have been grouped, based on their messages and formats, so that you can take action more quickly.

High-latency or erroneous transactions: Elastic Observability’s APM capability helps you discover which attributes are contributing to increased transaction latency and identifies which attributes are most influential in distinguishing between transaction failures and successes. Read APM correlations in Elastic Observability: Automatically identifying probable causes of slow or failed transactions for an overview of this capability.

AIOps Labs: AIOps Labs provides two main capabilities using advanced statistical methods:

• Log spike detector helps identify reasons for increases in log rates. It makes it easy to find and investigate the causes of unusual spikes by using the analysis workflow view. Examine the histogram chart of the log rates for a given data view, and find the reason behind a particular change possibly in millions of log events across multiple fields and values.
• Log pattern analysis helps you find patterns in unstructured log messages and makes it easier to examine your data. It performs categorization analysis on a selected field of a data view, creates categories based on the data, and displays them together with a chart that shows the distribution of each category and an example document that matches the category.

As we showed in the last blog, using machine learning-based features helps minimize the extremely tedious and time-consuming process of analyzing data using traditional methods, such as alerting and simple pattern matching (visual or simple searching, etc.). Trying to find the needle in the haystack requires the use of some level of artificial intelligence due to the increasing amounts of telemetry data (logs, metrics, and traces) being collected across ever-growing applications.

In this blog post, we’ll cover two capabilities found in Elastic’s AIOps Labs: log spike detector and log pattern analysis. We’ll use the same data from the previous blog and analyze it using these two capabilities.

We will cover log spike detector and log pattern analysis against the popular Hipster Shop app developed by Google, and modified recently by OpenTelemetry.

Overviews of high-latency capabilities can be found here, and an overview of AIOps labs can be found here.

Below, we will examine a scenario where we use anomaly detection and log categorization to help identify a root cause of an issue in Hipster Shop.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here) on AWS. Deploying this on AWS is required for Elastic Serverless Forwarder.
• Utilize a version of the popular Hipster Shop demo application. It was originally written by Google to showcase Kubernetes across a multitude of variants available, such as the OpenTelemetry Demo App. The Elastic version is found here.
• Ensure you have configured the app for either Elastic APM agents or OpenTelemetry agents. For more details, please refer to these two blogs: Independence with OTel in Elastic and Observability and Security with OTel in Elastic. Additionally, review the OTel documentation in Elastic.
• Look through an overview of Elastic Observability APM capabilities.
• Look through our anomaly detection documentation for logs and log categorization documentation.

Once you’ve instrumented your application with APM (Elastic or OTel) agents and are ingesting metrics and logs into Elastic Observability, you should see a service map for the application as follows:

In our example, we’ve introduced issues to help walk you through the root cause analysis features. You might have a different set of issues depending on how you load the application and/or introduce specific feature flags.

As part of the walk-through, we’ll assume we are DevOps or SRE managing this application in production.

Root cause analysis

While the application has been running normally for some time, you get a notification that some of the services are unhealthy. This can occur from the notification setting you’ve set up in Elastic or other external notification platforms (including customer-related issues). In this instance, we’re assuming that customer support has called in multiple customer complaints about the website.

How do you as a DevOps or SRE investigate this? We will walk through two avenues in Elastic to investigate the issue:

• Log spike analysis
• Log pattern analysis

While we show these two paths separately, they can be used in conjunction and are complementary, as they are both tools Elastic Observability provides to help you troubleshoot and identify a root cause.

Starting with the service map, you can see anomalies identified with red circles and as we select them, Elastic will provide a score for the anomaly.

In this example, we can see that there is a score of 96 for a specific anomaly for the productCatalogService in the Hipster Shop application. An anomaly score indicates the significance of the anomaly compared to previously seen anomalies. Rather than jump into anomaly detection (see previous blog), let’s look at some of the potential issues by reviewing the service details in APM.

What we see for the productCatalogService is that there are latency issues, failed transactions, a large number of issues, and a dependency to PostgreSQL. When we look at the errors in more detail and drill down, we see they are all coming from PQ - which is a PostgreSQL driver in Go.

As we drill further, we still can’t tell why the productCatalogService is not able to pull information from the PostgreSQL database.

We see that there is a spike in errors, so let's see if we can gleam further insight using one of our two options:

• Log rate spikes
• Log pattern analysis

Log rate spikes

Let’s start with the log rate spikes detector capability from Elastic’s AIOps Labs section of Elastic’s machine learning capabilities. We also pre-select analyzing the spike against a baseline history.

The log rate spikes detector has looked at all the logs from the spike and compared them to the baseline, and it's seeing higher-than-normal counts in specific log messages. From a visual inspection, we see that PostgreSQL log messages are high. We further filter this with postgres.

We immediately notice that this issue is potentially caused by pgbench, a popular PostgreSQL tool to help benchmark the database. pgbench runs the same sequence of SQL commands over and over, possibly in multiple, concurrent database sessions. While pgbench is definitely a useful tool, it should not be used in a production environment as it causes a heavy load on the database host, likely causing higher latency issues on the site.

While this may or may not be the ultimate root cause, we have rather quickly identified a potential issue that has a high probability of being the root cause. An engineer likely intended to run pgbench against a staging database to evaluate its performance, and not the production environment.

Log pattern analysis

Instead of log rate spikes, let’s use log pattern analysis to investigate the spike in errors we saw in productCatalogService. In AIOps Labs, we simply select Log Pattern Analysis, use Logs data, filter the results with postgres (since we know it's related to PostgreSQL), and look at information from the message field of the logs we are processing. We see the following:

Almost immediately we see the biggest pattern it finds is a log message where pgbench is updating the database. We can further directly drill into this log message from log pattern analysis into Discover and review the details and further analyze the messages.

As we mentioned in the previous section, while it may or may not be the root cause, it quickly gives us a place to start and a potential root cause. A developer likely intended to run pgbench against a staging database to evaluate its performance, and not the production environment.

Conclusion

Between the first blog and this one, we’ve shown how Elastic Observability can help you further identify and get closer to pinpointing the root cause of issues without having to look for a “needle in a haystack.” Here’s a quick recap of what you learned in this blog.

• Elastic Observability has numerous capabilities to help you reduce your time to find the root cause and improve your MTTR (even MTTD). In particular, we reviewed the following two main capabilities (found in AIOps Labs in Elastic) in this blog:

1. Log rate spikes detector helps identify reasons for increases in log rates. It makes it easy to find and investigate the causes of unusual spikes by using the analysis workflow view. Examine the histogram chart of the log rates for a given data view, and find the reason behind a particular change possibly in millions of log events across multiple fields and values.
2. Log pattern analysis helps you find patterns in unstructured log messages and makes it easier to examine your data. It performs categorization analysis on a selected field of a data view, creates categories based on the data, and displays them together with a chart that shows the distribution of each category and an example document that matches the category.

• You learned how easy and simple it is to use Elastic Observability’s log categorization and anomaly detection capabilities without having to understand machine learning (which helps drive these features) or having to do any lengthy setups.

Ready to get started? Register for Elastic Cloud and try out the features and capabilities outlined above.

Additional logging resources:

• Getting started with logging on Elastic (quickstart)
• Ingesting common known logs via integrations (compute node example)
• List of integrations
• Ingesting custom application logs into Elastic
• Enriching logs in Elastic
• Analyzing Logs with Anomaly Detection (ML) and AIOps

Common use case examples with logs:

• Nginx log management
• AWS VPC Flow log management
• Using OpenAI to analyze Kubernetes errors
• PostgreSQL issue analysis with AIOps

Elastic and Elasticsearch are trademarks, logos or registered trademarks of Elasticsearch B.V. in the United States and other countries.

In this blog post, we’ll cover some of Elastic’s artificial intelligence for IT operations (AIOps) and machine learning (ML) capabilities for root cause analysis.

Elastic’s machine learning will help you investigate performance issues by providing anomaly detection and pinpointing potential root causes through time series analysis and log outlier detection. These capabilities will help you reduce time in finding that “needle” in the haystack.

Elastic’s platform enables you to get started on machine learning quickly. You don’t need to have a data science team or design a system architecture. Additionally, there’s no need to move data to a third-party framework for model training.

Preconfigured machine learning models for observability and security are available. If those don't work well enough on your data, in-tool wizards guide you through the few steps needed to configure custom anomaly detection and train your model with supervised learning. To help get you started, there are several key features built into Elastic Observability to aid in analysis, helping bypass the need to run specific ML models. These features help minimize the time and analysis for logs.

Let’s review some of these built-in ML features:

Anomaly detection: Elastic Observability, when turned on (see documentation), automatically detects anomalies by continuously modeling the normal behavior of your time series data — learning trends, periodicity, and more — in real time to identify anomalies, streamline root cause analysis, and reduce false positives. Anomaly detection runs in and scales with Elasticsearch and includes an intuitive UI.

Log categorization: Using anomaly detection, Elastic also identifies patterns in your log events quickly. Instead of manually identifying similar logs, the logs categorization view lists log events that have been grouped, based on their messages and formats, so that you can take action quicker.

High-latency or erroneous transactions: Elastic Observability’s APM capability helps you discover which attributes are contributing to increased transaction latency and identifies which attributes are most influential in distinguishing between transaction failures and successes. An overview of this capability is published here: APM correlations in Elastic Observability: Automatically identifying probable causes of slow or failed transactions.

AIOps Labs: AIOps Labs provides two main capabilities using advanced statistical methods:

• Log spike detector helps identify reasons for increases in log rates. It makes it easy to find and investigate causes of unusual spikes by using the analysis workflow view. Examine the histogram chart of the log rates for a given data view, and find the reason behind a particular change possibly in millions of log events across multiple fields and values.
• Log pattern analysis helps you find patterns in unstructured log messages and makes it easier to examine your data. It performs categorization analysis on a selected field of a data view, creates categories based on the data, and displays them together with a chart that shows the distribution of each category and an example document that matches the category.

In this blog, we will cover anomaly detection and log categorization against the popular “Hipster Shop app” developed by Google, and modified recently by OpenTelemetry.

Overviews of high-latency capabilities can be found here, and an overview of AIOps labs can be found here.

In this blog, we will examine a scenario where we use anomaly detection and log categorization to help identify a root cause of an issue in Hipster Shop.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here) on AWS. Deploying this on AWS is required for Elastic Serverless Forwarder.
• Utilize a version of the ever so popular Hipster Shop demo application. It was originally written by Google to showcase Kubernetes across a multitude of variants available, such as the OpenTelemetry Demo App. The Elastic version is found here.
• Ensure you have configured the app for either Elastic APM agents or OpenTelemetry agents. For more details, please refer to these two blogs: Independence with OTel in Elastic and Observability and security with OTel in Elastic. Additionally, review the OTel documentation in Elastic.
• Look through an overview of Elastic Observability APM capabilities.
• Look through our Anomaly detection documentation for logs and log categorization documentation.

Once you’ve instrumented your application with APM (Elastic or OTel) agents and are ingesting metrics and logs into Elastic Observability, you should see a service map for the application as follows:

In our example, we’ve introduced issues to help walk you through the root cause analysis features: anomaly detection and log categorization. You might have a different set of anomalies and log categorization depending on how you load the application and/or introduce specific issues.

As part of the walk-through, we’ll assume we are a DevOps or SRE managing this application in production.

Root cause analysis

While the application has been running normally for some time, you get a notification that some of the services are unhealthy. This can occur from the notification setting you’ve set up in Elastic or other external notification platforms (including customer related issues). In this instance, we’re assuming that customer support has called in multiple customer complaints about the website.

How do you as a DevOps or SRE investigate this? We will walk through two avenues in Elastic to investigate the issue:

• Anomaly detection
• Log categorization

While we show these two paths separately, they can be used in conjunction and are complementary, as they are both tools Elastic Observability provides to help you troubleshoot and identify a root cause.

Machine learning for anomaly detection

Elastic will detect anomalies based on historical patterns and identify a probability of these issues.

Starting with the service map, you can see anomalies identified with red circles and as we select them, Elastic will provide a score for the anomaly.

In this example, we can see that there is a score of 96 for a specific anomaly for the productCatalogService in the Hipster Shop application. An anomaly score indicates the significance of the anomaly compared to previously seen anomalies. More information on anomaly detection results can be found here. We can also dive deeper into the anomaly and analyze the details.

What you will see for the productCatalogService is that there is a severe spike in average transaction latency time, which is the anomaly that was detected in the service map. Elastic’s machine learning has identified a specific metric anomaly (shown in the single metric view). It’s likely that customers are potentially responding to the slowness of the site and that the company is losing potential transactions.

One step to take next is to review all the other potential anomalies that we saw in the service map in a larger picture. Use an anomaly explorer to view all the anomalies that have been identified.

Elastic is identifying numerous services with anomalies. productCatalogService has the highest score and a good number or others: frontend, checkoutService, advertService, and others, also have high scores. However, this analysis is looking at just one metric.

Elastic can help detect anomalies across all types of data, such as kubernetes data, metrics, and traces. If we analyze across all these types (via individual jobs we’ve created in Elastic machine learning), we will see a more comprehensive view as to what is potentially causing this latency issue.

Once all the potential jobs are selected and we’ve sorted by service.name, we can see that productCatalogService is still showing a high anomaly influencer score.

In addition to the chart giving us a visual of the anomalies, we can review all the potential anomalies. As you will notice, Elastic has also categorized these anomalies (see category examples column). As we scroll through the results, we notice a potential postgreSQL issue from the categorization, which also has a high 94 score. Machine learning has identified a “rare mlcategory,” meaning that it has rarely occurred, hence pointing to a potential cause of the issue customers are seeing.

We also notice that this issue is potentially caused by pgbench , a popular postgreSQL tool to help benchmark the database. pgbench runs the same sequence of SQL commands over and over, possibly in multiple, concurrent database sessions. While pgbench is definitely a useful tool, it should not be used in a production environment as it causes heavy load on the database host, likely causing the higher latency issues on the site.

While this may or may not be the ultimate root cause, we have rather quickly identified a potentially issue that has a high probability of being the root cause. An engineer likely intended to run pgbench against a staging database to evaluate its performance, and not the production environment.

Machine learning for log categorization

Elastic Observability’s service map has detected an anomaly, and in this part of the walk-through, we take a different approach by investigating the service details from the service map versus initially exploring the anomaly. When we explore the service details for productCatalogService, we see the following:

The service details are identifying several things:

1. There is an abnormally high latency compared to expected bounds of the service. We see that recently there was a higher than normal (upward of 1s latency) compared to the average to 275ms on average.
2. There is also a high failure rate for the same time frame as the high latency (lower left chart “ Failed transaction rate ”).
3. Additionally, we can see the transactions and one in particular /ListProduct has an abnormally high latency, in addition to a high failure rate.
4. We see productCatalogService has a dependency on postgreSQL.
5. We also see errors all related to postgreSQL.

We have an option to dig through the logs and analyze in Elastic or we can use a capability to identify the logs more easily.

If we go to Categories under Logs in Elastic Observability and search for postgresql.logto help identify postgresql logs that could be causing this error, we see that Elastic’s machine learning has automatically categorized the postgresql logs.

We notice two additional items:

• There is a high count category (message count of 23,797 with a high anomaly of 70) related to pgbench (which is odd to see in production). Hence we search further for all pgbench related logs in Categories .
• We see an odd issue regarding terminating the connection (with a low count).

While investigating the second error, which is severe, we can see logs from Categories before and after the error.

This troubleshooting shows postgreSQL having a FATAL error, the database shutting down prior to the error, and all connections terminating. Given the two immediate issues we identified, we have an idea that someone was running pgbench and this potentially overloaded the database, causing the latency issue that customers are seeing.

The next steps here could be to investigate anomaly detection and/or work with the developers to review the code and identify pgbench as part of the deployed configuration.

Conclusion

I hope you’ve gotten an appreciation for how Elastic Observability can help you further identify and get closer to pinpointing root cause of issues without having to look for a “needle in a haystack.” Here’s a quick recap of lessons and what you learned:

• Elastic Observability has numerous capabilities to help you reduce your time to find root cause and improve your MTTR (even MTTD). In particular, we reviewed the following two main capabilities in this blog:

1. Anomaly detection: Elastic Observability, when turned on (see documentation), automatically detects anomalies by continuously modeling the normal behavior of your time series data — learning trends, periodicity, and more — in real time to identify anomalies, streamline root cause analysis, and reduce false positives. Anomaly detection runs in and scales with Elasticsearch and includes an intuitive UI.
2. Log categorization: Using anomaly detection, Elastic also identifies patterns in your log events quickly. Instead of manually identifying similar logs, the logs categorization view lists log events that have been grouped based on their messages and formats so that you can take action quicker.

• You learned how easy and simple it is to use Elastic Observability’s log categorization and anomaly detection capabilities without having to understand machine learning (which help drive these features), nor having to do any lengthy setups. Ready to get started? Register for Elastic Cloud and try out the features and capabilities I’ve outlined above.

Additional logging resources:

• Getting started with logging on Elastic (quickstart)
• Ingesting common known logs via integrations (compute node example)
• List of integrations
• Ingesting custom application logs into Elastic
• Enriching logs in Elastic
• Analyzing Logs with Anomaly Detection (ML) and AIOps

Common use case examples with logs:

• Nginx log management
• AWS VPC Flow log management
• Using OpenAI to analyze Kubernetes errors
• PostgreSQL issue analysis with AIOps