If tracing is on, the agent-builder-traces skill is already loaded on every agent in your space and there’s nothing to install. The first post in our series covered enabling OTel tracing and the out-of-the-box (OOTB) dashboards, along with threshold alerts. This post is about asking.

What is the agent-builder-traces skill?

agent-builder-traces is a built-in Agent Builder skill. It takes a natural-language question, generates an ES|QL query from it, executes that query against your trace index, and returns a plain-language summary. The index it targets is traces-agent_builder.otel-<space-id>, where <space-id> is the Kibana space that you’re working in. Using the exact space-scoped index pattern, rather than a wildcard, keeps data from other spaces out of your results.

Under the hood, the skill uses a single inline tool: agent-builder-traces.generate_esql. Instead of calling this tool directly, you ask a question and the agent calls the tool on your behalf, passing your question as the prompt for ES|QL generation. The tool resolves the current space's trace index, builds a query using the default model, executes it against Elasticsearch, and returns the result.

Which privacy settings control what your traces capture?

Several fields containing sensitive information are off by default. You can enable them in Gen AI Settings (Stack Management > AI Assistants), under the "Agent Builder Traces" section. Expand the Advanced privacy settings to find them.

Setting key

What it captures

agentBuilder:tracing:includeUserPrompts

User messages

agentBuilder:tracing:includeLlmResponses

Large language model (LLM) response text

agentBuilder:tracing:includeToolDetails

Tool call arguments and results

agentBuilder:tracing:includeSystemPrompt

The agent's system prompt

agentBuilder:tracing:includeRealNames

Real tool, agent, and conversation names (hashed when off)

agentBuilder:tracing:includeRealIds

Real conversation and workflow IDs (hashed when off)

agentBuilder:tracing:includeUserData

Real user IDs and usernames (hashed when off)

Why is my trace analysis query returning empty rows?

If the skill returns empty rows or tells you that a field is unavailable, check two things: whether the relevant setting is enabled in your configuration, and whether your query time window overlaps with any recorded spans. The skill reports exactly what the query returned; it doesn’t fabricate content when fields are empty..

What agent performance questions can you ask?

Examples of questions that you can ask include:

How many tokens have my agents used, by model?

Ask: How many input and output tokens have my agents used in the last 24 hours, broken down by model?

ES|QL: 

The TO_LONG() cast around each token field is required. These fields can surface as mixed integer and long types across index generations, and ES|QL’s SUM() needs an explicit numeric conversion before it can aggregate them. If you write your own queries against this index and hit unexpected type errors, this is usually why.

Which tools are failing most often?

Ask: What is the error rate for each tool over the last 7 days?

ES|QL: 

The tool error rate query aggregates across spans named ‘execute_tool’ and compares status.code == "Error" counts to total counts per tool name. The result shows which tools are failing most often. 

Which conversation turns are slowest?

Ask: Show me the 10 slowest conversation turns in the last hour.

ES|QL: 

The skill queries spans where span.name LIKE "invoke_agent *" and attributes.elastic.inference.span.kind == "CHAIN". These correspond to individual conversation turns, from the moment a user sends a message to when the agent returns a response. Durations are stored in nanoseconds, so the skill converts to seconds before sorting.

What have users been asking my agents?

Ask: What questions have users been asking in the last 30 minutes?

ES|QL: 

The recent-prompts query returns data only when agentBuilder:tracing:includeUserPrompts is set to true. When the setting is off, the attributes.gen_ai.input.messages field is empty and the skill will suggest that you check your Include User Prompts privacy setting.

When should you use Discover or Kibana Lens instead?

The skill targets one index pattern: traces-agent_builder.otel-<space-id>. Two scenarios fall outside that boundary.

Can the skill query my own application indices?

If you want to run ad hoc questions against data that you've indexed yourself, use a general data exploration skill or write ES|QL directly in Discover. The agent-builder-traces skill won’t query outside the Agent Builder traces index.

Can the skill build or edit dashboards?

The skill focuses on ad hoc queries and generating summary text. If your goal is to build a permanent visualization from your trace data, use Lens or the dashboard-management skill instead. We covered that specific process in depth in the first post in our series.

The skill sits alongside two other ways of watching your agents.

Tool

Best for

You use it when

agent-builder-traces skill

Ad hoc agent performance questions

You notice something and want an answer without leaving the chat

OOTB dashboards

Ongoing visibility

You want trends over time without asking anything

Threshold alerts

Automated monitoring

You define a condition once and get notified when it's breached

How to build evaluation pipelines from agent trace data

Spans in traces-agent_builder.otel-<space-id> capture real execution data from every conversation turn and LLM call running in your space. That record is the raw material for evaluation pipelines: automated checks on response quality and latency budgets per agent type, along with regression detection when you update a system prompt. The next post in our series covers how to build those eval loops from trace data that Agent Builder is already collecting.

Elasticsearch Workflows solves this with a persistent execution state. A workflow can pause at a human approval gate for days and resume exactly where it left off. Every decision is persisted in Elasticsearch as searchable data.

In this article, we’ll build this in practice with a real scenario: Documents that failed during ingestion get stuck in a data stream’s failure store. 

In this scenario, every step is reproducible. By the end, you’ll have an end-to-end workflow triggered by alerts, with pre-execution approval, automatic remediation, post-execution verification, and a rejection-and-revision path.

What you’ll learn

We’ll cover how to:

• Use the failure store to capture and remediate documents that failed ingestion.

• Build a long-running workflow with structured and binary approval gates using waitForInput and waitForApproval.

• Invoke Elastic Agent Builder AI agents automatically after human approval.

• Trigger workflows automatically from alerting rules.

• Handle approval and rejection paths (real human-in-the-loop, not just a checkbox).

What’s a long-running AI agent?

The difference between a session-bound agent and a long-running agent is the nature of the work rather than the speed of the large language model (LLM).

A session-bound agent lives within a session: It receives a question and processes it. Then it responds to the question. If the process dies, the context dies with it. 

Operational processes, like data remediation or infrastructure provisioning, or like change review, are different. The processing itself is fast. What takes time are human decisions at unpredictable moments. The person who needs to approve a reindex might be handling an incident or in a different time zone. Or they may simply be at lunch. Long-running agents handle these breaks in workflow by assembling separate infrastructure: a database to store execution state, a server for the orchestrator, a UI for approvals, and integrations for notifications. 

Elasticsearch Workflows runs where the data already lives. The workflow's long-term execution state is persisted in Elasticsearch, surviving restarts and waiting days between steps. AI agents come from Agent Builder and maintain a session only for the duration of each step, such as when the step ends or when the session ends. The workflow holds the context that spans across steps and days, and alerts come from the same rules already monitoring your data. The approval UI is in Kibana, and there’s no additional infrastructure to provision.

The workflow's internal execution state is managed by Kibana. The remediation-runs index is the audit trail that the workflow itself writes and that you can query like any other Elasticsearch index.

What’s the Elasticsearch failure store?

When Elasticsearch receives a document it cannot index, it has two options: Reject the document with an error, or store it somewhere safe for later analysis. The failure store is that second option.

Imagine a logs-demo-app data stream with the price field mapped as float. If the source application sends "price": "N/A" instead of a number, Elasticsearch cannot index the document. With the failure store enabled, it’s redirected to a dedicated index within the data stream itself rather than losing that document.

Documents in the failure store preserve the original content and include information about the error, including the exception type, the message, the pipeline and processor that failed, and even the stack trace. You can query the failure store using the data_stream::failures syntax. 

A typical document looks like this:

The document.source field contains the original document that failed, and the error field describes what went wrong. This is exactly the structure that the failure-analyst agent will read to diagnose the problem.

But the documents remain stuck there. Someone needs to diagnose the problem and fix the pipeline or mapping. They also need to reindex the documents back into the data stream. This is exactly the kind of work that combines AI automation with human approval, and it’s what we’re going to build.

How the AI agent orchestration works, from alert to resolution

The workflow connects four building blocks: an alerting rule, two AI agents, human approval gates, and an audit index. The diagram below shows how these components interact.

The alerting rule monitors logs-demo-app::failures every minute. When it finds documents written within the previous five minutes, it starts a workflow execution.

The workflow first runs read_failures to retrieve the failed documents. It then passes them to the failure-analyst agent through the diagnose step. The agent inspects the destination mapping and identifies the root cause. It then produces a structured remediation plan without making any changes.

The workflow records this plan in the remediation-runs index with the status awaiting_fix_approval and pauses at Gate 1.

Gate 1 uses waitForInput, a structured form that collects the reviewer's decision and optional notes. While the workflow waits, its state remains persisted in Elasticsearch. No agent session or polling loop needs to remain active, which means that the gate can wait for hours or days without continuously consuming compute resources.

If the reviewer approves the plan, the workflow invokes the remediation-executor agent through the execute_fix step. The agent calls the execute-failure-store-fix skill and creates the ingest pipeline. It also performs the bounded reindex and returns a detailed execution report.

Gate 2 appears only after the execution finishes. It uses waitForApproval to present the agent's report and asks the reviewer to choose between Yes, mark as resolved and No, escalate. The workflow then records the final outcome in remediation-runs and includes the complete report in agent_report.

If the reviewer rejects the initial plan at Gate 1, the workflow sends the diagnosis and the reviewer's feedback back to the failure-analyst agent, which then produces a revised plan that’s presented at Gate 1b. If approved, the revised plan follows the same automatic execution and verification path. If the plan is rejected again, the workflow records the case as fix_rejected and ends without applying any changes.

The remediation-runs index records the diagnosis, revision requests, revised plans, execution reports, and terminal outcomes as searchable audit documents. The shared workflow execution_id correlates records belonging to the same remediation run.

Production consideration: Exception-based verification

The diagram above shows the deliberately conservative implementation used in this tutorial: Every completed remediation pauses at Gate 2, including executions that appear fully successful. This makes the human-verification mechanism explicit and may be appropriate for low-volume or high-risk environments, but it can create review fatigue at scale. A production extension would add a deterministic verification step after execution. When the expected documents are present in the destination and the reindex reports no failures or policy-defined errors, the workflow can record resolved automatically. Only partial, failed, or ambiguous outcomes should pause at Gate 2 for site reliability engineering (SRE) review and possible escalation.

The following sections walk through each part: the data stream setup, the agents and skills, the alerting rule, and the workflow.

Prerequisites

To follow this tutorial, you’ll need:

• An Elastic Cloud Serverless project or an Elastic Stack 9.4+ deployment.

• Agent Builder (generally available [GA] on Elasticsearch projects, enabled by default).

• The following environment variables configured:

Setting up a data stream with the failure store enabled

All code in this tutorial, including the workflow YAML, setup scripts, and agent instructions, is available in this repository. 

Start by getting the code from the repository:

Then run the setup script:

The script prepares the test environment: It removes previous test resources, creates a failure-store-enabled index template, ingests three valid baseline documents, initializes the failure store, and creates the remediation-runs audit index. The price field is mapped as float with ignore_malformed: false ; nonnumeric values generate a parsing error and get redirected to the failure store instead of being silently dropped. The data stream itself is created automatically when the first document is indexed.

The failure store is enabled at the index template level, not on the data stream directly. The template includes a data_stream_options block that tells Elasticsearch to activate the failure store for any data stream created from that template:

The data stream is created automatically upon the first ingestion request, inheriting the failure store configuration from the index template. From that point on, any document that cannot be indexed is redirected to the failure store instead of being rejected with an indexing error.

The script also ingests three valid documents with numeric price values. These documents establish the healthy baseline: The data stream exists and contains valid data before any ingestion failures occur.

After the 01-setup-failure-store.sh script completes, the environment is ready:

The invalid documents that trigger the alerting rule and start the workflow are ingested later, when you run 02-trigger-test.sh in the testing section. This sequence shows a realistic failure scenario: The data stream is operating normally with valid data, and then an upstream change causes malformed documents to arrive.

Importing the agents and skills into Agent Builder

Run the restore script to import the two agents and two skills, along with the workflow into your Kibana instance:

Expected output:

Running the script again updates existing components without creating duplicates.

The AI agents that diagnose and execute the fix

After running the restore script, two agents will be available in Agent Builder: one to diagnose failures and another to apply approved fixes. In Kibana, go to Agent Builder and verify that both agents and their associated skills were created correctly.

failure-analyst, the read-only diagnosis agent

This agent reads failed documents from the failure store and inspects the destination mapping. It produces a structured remediation plan that includes an ingest pipeline definition and bounded replay instructions. The plan also includes a risk assessment. The agent doesn’t execute the proposed remediation.

The agent instructions establish its objective and read-only boundary. Key instructions include:

Its associated failure-store-remediation-planner skill provides the specialized rules used to construct a safe replay. For example:

The response contract is also constrained so that the workflow can consume the plan reliably:

Together, the agent instructions and skill allow failure-analyst to produce an actionable but bounded remediation plan while leaving execution under the control of the workflow and its human approval gate.

remediation-executor, the agent that applies the approved fix

The workflow invokes this agent after the remediation plan has been approved at Gate 1. Unlike the first agent, remediation-executor is allowed to perform the write operations required to apply the approved remediation.

Its agent instructions intentionally keep this role focused on immediate execution:

The associated execute-failure-store-fix skill defines how the approved plan is retrieved and executed. Key instructions include:

The skill searches by both execution_id and approval status. This ensures that the executor retrieves the plan approved for the current workflow execution rather than an unrelated remediation run. The status determines whether it uses the original diagnosis or the revised_diagnosis produced during the revision path.

This separation gives the two agents distinct responsibilities: failure-analyst investigates the failure and proposes a bounded plan without making changes, while remediation-executor applies the selected plan only after the workflow has completed the required human approval step.

The setup scripts import the complete agent instructions and skill definitions. The excerpts above highlight the instructions most relevant to understanding the responsibilities and behavior of each agent. The complete definitions are available in backup/save_agents.json and backup/save_skills.json in the repository.

After verifying that the agents and their skills have been imported correctly, the next step is to configure the alerting rule that detects new documents in the failure store and starts the remediation workflow.

How do I trigger a workflow from an Elasticsearch alert

The remediation process starts with an Elasticsearch alerting rule that monitors the failure store and starts the workflow when new failures are detected. Because the restore.py script has already imported and enabled failure_store_remediation, you can connect the workflow while creating the rule.

Creating the alerting rule that watches the failure store

The rule checks the failure store every minute and creates an alert when it finds documents written within the configured five-minute time window.

1. Go to Management → Rules → Create rule.

2. Select Elasticsearch query.

3. Configure:

• Query type: ES|QL.

• Query:

• Select time field: @timestamp.

• Select alert group: Create an alert if matches are found.

• Time window: 5 minutes.

• Check every: 1 minute.

4. In the Actions section, click Add action → Workflows.

5. Select the failure_store_remediation workflow.

6. Under Run workflow for, select New alerts.

7. Set Action frequency to Run per alert.

8. Save and enable the rule.

Whenever the rule creates a new alert, it starts one failure_store_remediation workflow execution for that alert. The failure store is still empty at this stage; the invalid documents are ingested later by 02-trigger-test.sh.

With the alert trigger configured, let’s examine the workflow that performs the diagnosis, approval, remediation, and verification steps.

Building the agentic workflow, step by step

At this point, the workflow has already been created in Kibana by the restore.py script executed earlier. To understand how the workflow operates, we’ll examine it incrementally, using relevant YAML excerpts to explain the role of each step in the remediation process. The complete workflow definition is available in the tutorial repository.

Declaring the alert trigger and the 7-day timeout

The workflow declares an alert trigger that allows it to be initiated by the alerting rule created earlier.

Two details are important here. First, the timeout: 7d defines that the workflow can remain active for up to seven days. We set this explicitly because the workflow will pause at human approval gates that may take hours or days to respond.

Second, declaring type: alert makes the workflow available to alerting. The workflow action configured in the rule establishes the connection between the alert and failure_store_remediation.

Reading failed documents from the failure store

The workflow begins by retrieving failed documents using the ::failures syntax:

The range query limits the search to failure-store documents created during the same five-minute window monitored by the alerting rule. Within that window, size: 50 bounds the batch to the 50 most recent matching documents. For larger volumes, use search_after or pagination.

How the AI agent diagnoses the root cause

The next step sends the failed documents to the failure-analyst agent:

The {{ steps.read_failures.output.hits.hits | json }} template injects the documents retrieved in the previous step into the agent's prompt. The agent analyzes the error patterns and identifies that the price field received string values where floating-point values were expected. It then proposes a remediation pipeline.

The agent's output arrives in steps.diagnose.output.message as a single JSON object. The exact diagnosis, proposed remediation, pipeline definition, and reindex request may vary between executions because they’re generated by the AI agent from the failures it receives. In our tests, the agent consistently identified the root cause and generated working remediation pipelines.

Despite this natural variability, the response contract remains fixed so that the workflow can persist the plan and the executor can reliably extract its executable components. The response always contains the six required top-level keys: root_cause, failure_type, affected_docs, proposed_fix, remediation_pipeline, and risk_assessment.

Recording the diagnosis for audit

Before requesting human approval, the workflow writes the diagnosis and proposed remediation plan to the remediation-runs index. This creates a durable audit record that can later be retrieved by the remediation-executor if the plan is approved:

The execution_id correlates the diagnosis with the current workflow execution. The record stores both the number of failure documents loaded into the bounded batch and the total number of failures matching the query. After human approval, the executor uses the execution_id together with the approval status to retrieve the correct remediation plan.

How do human approval gates work in Elasticsearch Workflows?

The workflow uses two kinds of pause steps for different kinds of human decisions. Gate 1 uses waitForInput to collect a structured approval decision and reviewer feedback. Gate 2 uses waitForApproval because verification requires a binary choice: Mark the case as resolved, or escalate it.

Gate 1

Gate 1b

Gate 2

Step name

gate_fix

gate_fix_revised

gate_verify (approved path) / gate_verify_v2 (revised path)

Step type

waitForInput

waitForInput

waitForApproval

Timeout

72h

72h

72h

Decision shape

Approve or reject, plus notes

Approve or reject, plus notes

Yes, mark as resolved / No, escalate

Presents

AI diagnosis and proposed plan

Revised plan

remediation-executor report

Appears

Always

Only after Gate 1 rejection

Only after execution completes

Gate 1 is defined like this:

When the workflow reaches this step, execution pauses, and its state is persisted in Elasticsearch. No agent session or polling loop needs to remain active while the gate waits. Every gate accepts a response for up to 72 hours, subject to the workflow's overall seven-day timeout.

The schema collects an approval decision and an optional notes field. When the proposal is rejected, the workflow passes the notes to the failure-analyst agent as context for revising the remediation plan. The testing section shows the approval and rejection payloads at the point where each is submitted.

What happens when the reviewer approves or rejects the plan

When Gate 1 is submitted, route_fix resumes the workflow without rerunning the completed steps. The simplified excerpt below shows the first transition in each branch. (The complete executable YAML is available in the tutorial repository.)

Approval sends the current workflow execution_id to the remediation-executor. The executor uses this identifier together with the approval status to retrieve the remediation plan associated with this specific execution before applying it and continuing to Gate 2.

If Gate 1 is rejected, the workflow stores the reviewer’s feedback and asks the failure-analyst agent to produce a revised proposal. Gate 1b (gate_fix_revised) then presents that proposal for another human decision. When the revised plan is approved, execute_fix_revised sends the same workflow execution ID to the executor:

The executor uses the approval status to determine which plan belongs to the current path: the original diagnosis associated with awaiting_fix_approval or the revised_diagnosis associated with awaiting_fix_approval_v2. Both paths then continue to Gate 2. If the revised proposal is rejected at Gate 1b, the workflow records fix_rejected and ends without applying any changes.

Verifying the result at Gate 2 before closing the case

Gate 2 is a verification checkpoint. By the time it appears, the remediation-executor agent has already created the ingest pipeline and run the reindex. It has also returned a detailed report. The gate embeds the executor's report, steps.execute_fix.output.message on the approved path, or steps.execute_fix_revised.output.message when the revised plan was executed, and asks the reviewer to select either Yes (mark as resolved) or No (escalate).

Gate 2 uses waitForApproval because the workflow only needs a binary verification decision. The operation details, including the pipeline ID, documents matched and reindexed, version conflicts, failures, and execution errors, come from the remediation-executor agent’s report rather than from fields entered manually by the reviewer.

In our validated execution, the agent created the logs-demo-app-price-remediation pipeline and matched and reindexed all five approved documents. It reported zero version conflicts and zero failures. The logs-demo-app data stream increased from three to eight documents, while the five original records remained in logs-demo-app::failures. This is expected because reindex copies documents; it doesn’t remove them from the failure store.

If the reviewer selects Yes, mark as resolved, route_verify writes a new document to remediation-runs with status resolved and stores the complete agent report in agent_report. If the reviewer selects No, escalate; the workflow writes an escalated record with the same report. The revised path mirrors this through route_verify_v2, producing the same statuses and the same agent_report field.

Gate 2 exists because a technically completed reindex can still produce partial or unexpected results. Escalating instead of automatically marking the remediation as resolved keeps the audit trail honest. Whether to keep this second gate depends on your environment and risk tolerance.

The complete workflow

After reviewing the workflow, remember that the complete code is available in the repository in the failure-store-remediation.yaml file.

The workflow can finish in five ways, depending on the decisions made at Gate 1, Gate 1b, and Gate 2. The diagram below shows how the original and revised remediation paths lead to resolved, escalated, or fix_rejected.

Every terminal outcome is appended to the remediation-runs audit history. Across the workflow paths, the audit records preserve the diagnosis, reviewer feedback when a revision is requested, the executor report, and the terminal outcome.

With the rule and its workflow action already configured, the end-to-end flow is ready to test.

Testing the workflow end to end

With the alerting rule enabled, run the trigger script to ingest five documents with invalid price values and current timestamps:

The rule searches the previous five minutes, so it detects the new failure-store documents during its next evaluation. Go to Workflows → Executions, and open the new execution. Confirm that read_failures, diagnose, and record_diagnosis have completed and that gate_fix is waiting for input.

Approving the fix at Gate 1

At Gate 1, approve the proposed remediation by submitting:

The workflow resumes at route_fix and invokes the remediation-executor agent through execute_fix. It pauses at Gate 2 with the execution report.

The following clip shows Gate 1 being approved and the workflow resuming automatically at route_fix and execute_fix.

Before completing Gate 2, verify that the five remediated documents were added to the data stream:

Expected result:

The data stream started with three valid documents, so a successful replay increases the total to eight. The original records remain in the failure store because reindex copies documents to the destination rather than removing the source records.

Complete Gate 2 by selecting Yes, mark as resolved or No, escalate. The workflow stores the selected outcome and the execution report in remediation-runs.

Rejecting the plan and reviewing a revised one

To test the rejection path, reset the environment and start another execution:

At Gate 1, reject the proposal with specific feedback:

Although notes is optional in the current schema, provide specific feedback whenever you reject a proposal. The workflow passes this value to the failure-analyst agent when requesting a revised plan.

The workflow records the feedback and asks the agent to revise the proposal. It pauses at Gate 1b. To approve the revised plan, submit:

To reject it definitively, submit:

Approving the revision sends it through the same execution and Gate 2 verification path described above. Rejecting it records fix_rejected and ends the workflow without applying the remediation.

Conclusion

The pattern built in this tutorial combines three elements: persistent workflow state, specialized AI agents, and human control at the points where judgment matters. The failure-analyst agent diagnoses the problem and proposes a bounded remediation. Gate 1 pauses before any change is made, giving the reviewer control over execution. After approval, the remediation-executor agent applies the fix automatically, and Gate 2 pauses again so the result can be verified before the case is marked as resolved or escalated.

Persistent execution state solves more than the problem of expiring agent sessions. The diagnosis, reviewer feedback, execution report, and final decision are stored as searchable data in Elasticsearch and correlated by the workflow execution_id. This correlation allows the executor to retrieve the plan approved for the current workflow run, even when multiple remediation cases exist in remediation-runs. It also makes it possible to analyze how many remediations were approved on the first attempt and which failure types are most frequently rejected, along with how long approval gates remain open and which cases require escalation.

This workflow pattern also applies to index promotions, mapping changes, enrichment pipeline validation, infrastructure operations, and other processes where automation and human judgment must coexist: Diagnose the problem, pause for approval, execute the approved action automatically, pause for verification, and record the final outcome.

Resources

• Elasticsearch Workflows documentation

• Agent Builder documentation

• Failure store documentation

• Tutorial repository

Start a free Elastic Cloud Serverless trial or log in to Elastic Cloud to follow along, and install the CLI via npm: 

Designing the Elastic CLI for people and AI agents

A useful side effect of building flexible tools for developers is that they’re more useful to AI agents, too. It’s also the tool that our Agent Skills now use to get things done, closing the loop that we opened in March 2026, when we said that a CLI for agent workflows was coming.

The CLI gives every public API across Elasticsearch, Kibana, and Elastic Cloud one shape: the same flags, input and output conventions, authentication method, and failure mode. Consistency is the ergonomic feature; everything else is built on it.

Agents need the same thing, only stricter. An agent won’t know how to craft a valid CLI command or notice if a tool “feels” wrong; it needs output it can parse and input it can validate before sending, along with failures it can branch on. Agents are now a first-class interface to Elastic, alongside people, whether they live on the platform or in your editor and terminal. Agent Skills, and now the CLI, are how we serve the second kind, so those needs are built into the core of the CLI rather than tacked on.

JSON input and output for every command

Agents love structured text, and almost every Elastic API already speaks JSON, so first-class JSON support was a hard requirement. Developers who are quick with a jq query will be equally satisfied.

• JSON output: Any command, like elastic version and elastic es indices delete ..., supports --json, which prints JSON-parseable output to stdout and nothing else. Failed commands print {"error": {"code": "...", "message": "..."}} to stderr.

• JSON input: Every command that takes input accepts JSON on stdin or via --input-file. Every top-level key in that JSON also works as a CLI argument, and inline arguments take precedence, so you can keep a big request body in a file and tweak a value or two per invocation.

• JSON Schema as --help output: Pass --help --json to any command, and it prints a valid JSON Schema for its input, which also feeds nicely into codegen tools. elastic cli-schema prints the whole command tree.

Exit codes that an AI agent can branch on

Agents loop on exit codes as much as they do over stdout. All failure modes are distinguishable from success, even if stdout and stderr are never read.

Safety rails: Keychain storage, allow lists, and validation

No model uses tools perfectly 100% of the time, so an agent-friendly CLI should provide safety rails wherever possible.

• Contexts and secret storage: Connection details live in named contexts in ~/.elasticrc.yml, kubectl-style; switch with --use-context. API commands never take credentials as flags. elastic config context add writes API keys to your OS keychain (macOS, Linux, Windows) and leaves a $(keychain:...) reference in the YAML; $(env:...), $(cmd:...), and $(file:...) work, too. Creating a Serverless project with --save-as writes its credentials straight to the keychain and never prints them, so nothing leaks into logs or LLM transcripts.

• Allowlists/blocklists: A commands.allowed (or commands.blocked) list in the config file, globally or per context, ensures that only the commands an administrator wants are runnable.

• Validation: Every command has a JSON Schema, so inputs are validated before any request is sent. Add --dry-run to any command that takes input, and it validates and exits without sending anything.

• Confirmation: Destructive commands prompt in a terminal. In a noninteractive session, where agents live, they refuse to run without --yes and say so in a structured error.

• Sanitization: Index, field, and pipeline names have length limits and forbidden characters. elastic sanitize index-name '<value>' (and field-name, pipeline-name, …) prints a version stripped of anything invalid.

Keeping API responses inside an agent's context window

Elastic APIs return a lot of data, and an agent’s context window is finite. Three controls help keep unnecessary text out of the context window:

• Field masks: --output-fields takes a comma-separated list, with dot notation for nested fields.

• String templates: For total control, --output-template takes a mustache-style template.

• Command profiles: --command-profile serverless (or default_profile: serverless in your config) hides Elastic Cloud Hosted commands and the Elasticsearch namespaces that don’t exist on Serverless. That means less to scroll past and less for an agent to guess wrong. It’s the profile we recommend for agents.

Control

What it does

Syntax

When to use

Field mask

Returns only the fields you name, using dot notation for nested fields

--output-fields 'name,version.number'

You want valid JSON back, just less of it. This is the default choice for agents parsing structured output.

String template

Renders the response through a mustache-style template

--output-template 'ES version: {{ version.number }}'

You need one value in a specific shape, for a shell variable, a log line, or a prompt.

Command profile

Hides commands and namespaces that don't apply to your deployment

--command-profile serverlessor default_profile: serverless

You want a smaller command surface so an agent has less to scroll past and less to guess wrong. This is recommended for agents.

Helpers for bulk ingest, scroll search, and msearch

Some of Elasticsearch’s most popular APIs have a learning curve, so elastic es helpers wraps them:

• scroll-search: Stream a large result set as NDJSON with paging handled for you.

• bulk-ingest: Ingest from a file, a directory, or stdin (NDJSON, JSON arrays, or CSV) with streaming, batching, concurrency, and retries.

• msearch: Send multiple searches in one request.

• watch: Print new documents from an index to stdout as they’re indexed. This is great for piping into logging tools.

elastic es and elastic kb are aliases for elastic stack elasticsearch and elastic stack kibana. If we don’t ship a command you need, elastic extension create scaffolds one for you.

Searching Elastic docs from the terminal

If you or your agents don’t know which API to use, elastic docs search (plus docs read and docs ask) searches Elastic’s documentation from the terminal, returning Markdown or --json. These are experimental. You’ll see a warning until you pass --accept-experimental, so explore, but don’t script against them yet.

Shell completion for Bash, Zsh, and Fish

Autocomplete hooks are available for Bash, Zsh, and Fish, and they always respect your commands.allowed or commands.blocked policy.

How Elastic Agent Skills use the CLI

Agent Skills teach an AI coding agent how an Elastic expert approaches a job; for example, which cluster health field is the verdict or how to stage a reindex so it doesn’t fall over. They capture process and judgment but not transport. A skill that embeds curl with an auth header has hard-coded a hostname, key, and runtime, and it breaks when any of those change.

So our skills now use a universal format that runs unchanged in any runtime that can execute the elastic CLI, including Claude Code, Codex, Cursor, and GitHub Copilot. The body refers to operations in HTTP shorthand (GET /_cluster/health, POST /_query), and an operations table at the end binds each to a CLI command. That table is the only place transport appears:

HTTP API (shorthand)

elastic CLI command

GET /{index}/_mapping

elastic es indices get-mapping --index '<index>'

POST /_query

elastic es esql query --format tsv --query "<esql>"

POST cloud:/api/v1/serverless/projects/elasticsearch

elastic cloud serverless projects search create --input-file <json> --wait --save-as <ctx>

Every universal skill also inherits a blunt preamble; that is, use the CLI, don’t guess credentials, don’t call the HTTP API directly, and never ask the user to paste an API key into the chat.

The two halves need each other. The skill supplies the expertise that the model doesn’t have, and the CLI supplies a way to act on it that’s validated, credential-safe, and scoped by your allowlist. Tell your agent to spin up a Serverless project and load products.csv into it, and the provisioning skill creates it with --save-as. The ingest skill dry-runs a mapping and loads with elastic es bulk, and the Elasticsearch Query Language (ES|QL) skill writes a query that parses on the first try. Every step returns JSON, exits non-zero on failure, and can only do what your policy allows.

Skills for Elastic Cloud onboarding and provisioning, Elastic Workflows, and Kubernetes investigation are available today. Skills for Elasticsearch query, ingest, reindex, and index design, plus Kibana dashboards and alerting, are close behind.

What’s in the Elastic CLI technical preview, and what’s next

The preview covers all public Elasticsearch Serverless, Kibana Serverless, and Elastic Cloud APIs. Hosted-only 9.x Elasticsearch API coverage is nearly 100%, and hosted-only 9.x Kibana APIs will be added soon.

We’re actively planning more developer experience work, including broader coverage for all supported stack releases, more helpers for common workflows, more skills in the public catalog, and loading the same skills into agents that run on the Elastic platform itself. What shapes that list is hearing how you and your agents use the CLI. Tell us what’s awkward and what’s missing, along with what you’d automate next.

Install the Elastic CLI and Agent Skills

The Elastic CLI is available now on npm (Node.js 22+). Install it and the skills together:

Then, add a context and check it:

Even without a project, you can start a free Serverless trial in about a minute, with no credit card. If you already have a project, log in and create API keys for Elastic Cloud and your Elasticsearch clusters. Before pointing an agent at anything real, start with a trial project, a read-only key, and a scoped commands.allowed list. Be sure to take five minutes to read the security notes in the skills repo.

Replace all those curl commands in your Bash scripts, and add some usage instructions to your AGENTS.md. Then let your agent’s skills work efficiently and accurately with our APIs. Let us know what you think, and don’t hesitate to open an issue if you find a bug or if your use case isn’t well supported. Your feedback directly shapes what we build next.

Elastic CLI and Agent Skills resources

• Elastic CLI on GitHub and CLI documentation

• Elastic Agent Skills on GitHub

• agentskills.io specification

• Elastic Cloud Serverless documentation

• Report a CLI issue · Report a skills issue · Discuss

However, with the rapid progression of coding agents, performance optimization has become a task we can tackle semiautomatically. Unlike many software engineering challenges, optimizing code offers a cheap and objective verifier. If you ask an AI model to make code faster, there’s a hard number at the end telling you exactly what happened, backed by profiling tools that explain why. This makes performance a perfect candidate for automation, provided you can actually trust the numbers.

If you simply point a coding agent at a benchmark, you typically get low signal-to-noise: wins that fall inside the variance of the environment, or variations caused by thermal throttling rather than better code. To capture optimizations that actually benefit Elasticsearch users, we had to bridge the gap between "checkable in principle" and "checked in practice." We built a highly trustworthy measurement loop: a harness that assumes the agent will be wrong a good fraction of the time but reliably catches and proves it when it’s right and then helps guide it where to look next.

Once the machinery is in place, the results speak for themselves. By letting this harness loose on the codebase, we've already begun uncovering meaningful wins across the stack. In part 2 of this post, we’ll dive into some examples it has found so far, including string conversion inefficiencies in Elasticsearch Query Language (ES|QL), an improvement to our NEON vector dot product implementation, and an upgrade opportunity for the gzip library we were using. In this part, we’ll take a look at the design choices we made and how they relate to the broader topic of effective harness development.

The AI code optimization pipeline architecture

The first step in any software engineering problem is to identify the correct high-level components. We made an architectural choice that turned out to be very helpful for this problem: separate understanding where opportunities exist from the loop making code changes. The agent starts with a real workload but only uses it to mine information about where to seek performance improvements. At this stage, it’s instructed to go broad and consider a range of performance-related signals. Once it has found and classified the hot spots, the agent reads the context of the code around them to understand the optimization opportunities. We use a separate task to condense the ranked list of hot spots into artifacts that a loop can iterate against in minutes: a microbenchmark that we prove exercises the hot path in its real operating regime. Finally, we use a proposer-verifier loop to actually make changes to the codebase to improve performance on the benchmark. This hands off to validation to assess the impact on real workloads at the end. Our CLI (atune) supplies the tools this process needs, and the rest is largely automated by a set of task-specific instructions.

For context, our high-level architecture looks like the following. Pink boxes are the humans, and teal boxes are the agent. There are three task types, one skeleton loop, and one referee.

An exploration task profiles a real workload and produces ranked opportunities; a human promotes one into an exploitation task, which iterates against an approved microbenchmark and commits each accepted experiment; a validation run on the real workload guards the result before a human reviews and opens the PR. Where no benchmark covers the hot path, a benchmark task authors one and a human approves it into a registry. A performance atlas informs every task and accumulates what each one learns.

Why performance optimization suits autonomous agents

Three properties make a task ideally suited for autonomous work, and it's worth being explicit about them because they provide a checklist you can use to evaluate automation candidates. You want:

1. An objective verdict so that the agent can be held to something other than its own opinion.

2. A dense guiding signal so that it knows where to look next instead of guessing.

3. A bounded blast radius so that being wrong is affordable.

Performance gives you all three. Benchmarks provide the verdict and profilers provide the gradient, while a rejected patch costs you wall-clock time rather than correctness. The change gets reverted, and the reason gets recorded. Life goes on. Regarding the first two, we've come to think the gradient matters more than the verdict. ″This got faster″ is binary, whereas a profile hints at what to try next. An agent can generate its next hypothesis conditioned on a rich guiding signal, the richer the better, rather than grinding through a list.

Signals are what the agent gets to see

A useful mental model is that the CLI is the agent's sensory apparatus. That changes how you design each command. Rather than exposing a capability, you design it to return a clear and concise answer to a question about the task at hand, and, when relevant, an explanation the model can reason over, instead of raw data it has to parse and interpret. These are the signals that the agent acts on, and we ended up with the following for our harness:

Signal

Question it answers

Facet-decomposed macro profile

Where in the code does real workload time go, per query type?

Allocation and lock sampling, in the same capture

Is the cost cycles, garbage, or contention?

Cost-composition classification

Is this in scope compute, other product code, GC, JIT tax, or parked threads?

Input-shape instrumentation

What does the workload actually feed this code?

Statistical verdict

Did this change help, at this measured noise floor?

Allocation-rate comparison

Did the new code end up allocating more?

Interpreted disassembly

Why did that result happen?

End-to-end A/B guard, with differential profile attribution

Did anything appear to break, and was it us?

Environment check

Is this machine even fit to measure right now?

Upstream duplicate search

Has somebody already reported or fixed this?

Four of these signals are worth dwelling on, because in each case the tool encodes a judgment that the agent would otherwise have had to keep making by hand.

Facet-decomposition is the clearest example. Blended CPU shares hide breadth: if you profile a mixed query workload, the grouping hash map insert and the percentiles sketch update can both show up as single-digit percentages of the total run time and look comparable. They aren't comparable at all, because the hash map insert is paid for by nearly every aggregation query, while the sketch update is only paid for when somebody asks for percentiles. So the profiler runs each named query facet as its own race, and every opportunity the agent records carries a breadth field (universal, broad, or narrow) and gets ranked by headroom × tractability × breadth. A universal 3% beats a narrow 10%. Putting the ranking function in the tool prevents it from having to be rediscovered on every run.

The cost-composition classification works as a router. Rather than handing the agent a flat top-N list of frame names, it buckets every sampled stack into ″in scope compute,″ ″other product compute,″ ″GC,″ ″JIT and safepoint overhead,″ ″off CPU waiting,″ and ″parked threads.″ Each of those buckets implies a different kind of investigation. If GC is above about 15%, the real target is allocation rate, and the CPU top frames will actively mislead you, because they show where objects were collected rather than where they were created. If JIT and safepoint overhead is above about 25%, you're looking at a ceiling rather than an opportunity, since no in-scope code change will move it. If threads are parked and core utilization is low, this is a concurrency problem and CPU flame graphs are the wrong instrument entirely. We wrote that mapping into the playbook as a table, so the model (even a cheap one) reads a profile the way that an experienced engineer would, rather than reaching straight for the top frame. The classification is a prior for forming a hypothesis, though, not a substitute for evidence, so the agent still has to cite specific frames when it proposes an experiment.

Interpreted disassembly is a tool we hadn't originally provided, but it most definitely earns its place. It helps answer why, the question that unblocks the next hypothesis. Flame graphs tell you where the time goes; they rarely tell you why a change made things worse. So atune asm runs the benchmark briefly with the JIT told to print the assembly for one hot method, captures both sides of the working-tree diff, reduces each to the final C2 compilation, normalizes the addresses, and diffs them. The diff alone would likely still be 4,000 lines of aarch64, so on top of it sits an interpretation layer: a per-mnemonic delta, a net instruction count, the compilation tier that was actually captured, and a vectorization signal that counts vector register references on each side and raises a warning when they're eliminated, halved, or narrowed from Advanced Vector Extensions (AVX) to Streaming SIMD Extensions (SSE) width. The playbook then maps mnemonic patterns to causes:

Pattern in the diff

Likely cause

b.eq/b.ne up, csel down

New unpredictable branches

Clusters of str/ldr against the stack pointer

The compiler ran out of registers

NEON loads replaced by scalar compares

The vector path degraded

In one experiment, the agent fused two SIMD mask extractions into one, and the benchmark regressed by 26%. The vectorization warning explained it in about 10 seconds. Without that tool, the agent has a dead end and no working model of the machine; with it, it has a corrected model and several new ideas.

The fourth signal is less a single tool than a habit; the instruments check themselves. Core utilization is derived two independent ways, from sample density and from process sampling, so the two can be compared. The classification is rejected if the unclassified bucket exceeds a budget, on the grounds that a breakdown which can't account for its own samples shouldn't be reasoned over. The disassembly capture warns when the compilation it caught isn't the steady-state one. The upstream duplicate search is restricted to read-only commands, and that restriction is enforced by a test that greps the source, so no future edit can quietly reintroduce the ability to file anything. Each of these exists because a tool that can be confidently wrong is worse than a tool that is merely absent.

One small piece of design is worth highlighting as a specific instance of good return practice. atune compare returns 0 for improved, 1 for no change, 2 for regressed, and 3 for error, and the loop branches on this code. That means no parsing and no ambiguity about what the verdict was. Plus, no tokens are spent interpreting prose.

If you take one thing away from our CLI design, it’s a broader design principle. In a general setting, the interesting thing isn't the individual signals we found useful to understand performance; it's that the CLI is capturing and packaging the judgment of an experienced performance engineer into tools that return answers rather than raw data. Structurally imposing good judgment about the problem an agent is tasked with improves outcomes. The right CLI is as much part of that story as the instructions. Furthermore, tokens are saved by tools that return decisions and digests rather than data. That means a comparison verdict instead of raw JMH output; a triage summary sitting on top of a 4,000-line disassembly diff.

From 20-second probes to hours of validation

Building a verifier that you can afford to consult is a separate problem from building one that you can trust. This covers the affordability part. Or, if you like aphorisms, real workloads are where truth lives and where iteration goes to die. A macro profile takes 45 to 60 minutes, and an end-to-end validation run takes hours. But a microbenchmark takes minutes. That's why the exploration-then-exploitation split works; you go broad on the real workload once and then hand off to a microbenchmark that you can iterate against in minutes.

The catch is that the handoff is only sound if the microbenchmark exercises the hot path in a realistic operating regime; that is, with the right cardinality and right data distribution. If you get that wrong, your fast loop spins fast but in the wrong direction. Until we finalized the handoff procedure, we saw cases where the agent accepted changes on a benchmark whose key distributions happened to flatter it, and only the end-to-end run caught the problem.

Handing off from exploration to exploitation

The handoff between the two phases is structured rather than informal. An exploration task's primary deliverable is a set of opportunity records, and each one carries the scope paths that an exploitation task would be allowed to edit, a headroom estimate, a classification (constant factor, structural,

allocation, or concurrency), and the benchmark it would gate on (or an explicit "no benchmark coverage" flag, if none exists). Each also carries a narrow test pattern so that the correctness gate stays cheap. The test pattern field exists because of a specific incident; an exploration task omitted it, and the resulting exploitation task ran a very heavy test suite on every experiment. The fix was to change the upstream artifact rather than add an instruction downstream. This is a pattern we use repeatedly; make and record decisions as early as possible rather than re-derive them each time.

Validating a new benchmark before it can gate anything

Where benchmark coverage is genuinely missing, a dedicated benchmark task authors one, and that new benchmark has to pass validity checks before anything can rely on it. Two of them are mechanical: what fraction of the benchmark’s hot self-time comes from frames that actually appear in the production profile and whether the parameters fall inside the input shapes we measured. The third is a checklist that the agent has to attest item by item. It exists to avoid the JIT getting a simpler world than production.

1. Inputs have to be reshuffled rather than fixed or sorted, so the branch predictor doesn’t get too good.

2. Results have to be consumed, or dead-code elimination deletes the thing that you meant to measure.

3. Inputs must not be compile-time constants, or they get folded away.

4. Call sites have to see roughly the product mix of types, because a monomorphic call site inlines, whereas a megamorphic one doesn’t.

A human then approves it into a hash-pinned registry. Until that happens, it's inert, because task setup refuses any task citing an unapproved benchmark. We think of that approval as the strongest gate in the system, and it's deliberately placed. An approved benchmark can decide accept or reject in every future task, so it's the one place where we ask for a human signature on an artifact rather than on a decision.

The validation ladder

Underneath all of this sits a hierarchy of feedback mechanisms with the property that each rung is cheaper and weaker than the one below it, and the cheap tiers are for rejection only.

Tier

Cost

Role

probe

~20–60 s

Directionally right? Can never accept

codegen capture

~4 min

Why did that happen?

screen

~5–15 min

Cheap statistical filter

confirm

~20–60 min

The accept decision

end-to-end

hours

Regression guard, advisory

The asymmetry between accept and reject is doing real work here. A probe is a single paired fork, whereas the accept predicate requires a full confirm run with a matching calibration record. Agents are very good at telling believable stories; indeed, they're trained on many tasks judged by both LLMs and by humans, so being convincing is actively rewarded. We don't want an agent to be able to promote a cheap signal into a decision by being persuasive about it.

For this task benchmark, wall-clock is a real cost consideration. A confirm run might take an hour. So one has to weigh carefully all the costs involved when choosing the setup. A model that lands one hypothesis in four typically beats a cheaper one landing one in 10 by a margin on end-to-end metrics. The usual instinct to down-spec the model on a long-running loop is exactly backward here. When your loop has an uncertain outcome and significant costs beyond the tokens it consumes, you may well find yourself in the same situation.

How do you know a performance improvement is real?

Is it faster? is a statistical question. So we made the accept predicate code rather than judgment and put it somewhere the agent can't bypass. There are four ideas in the accept decision, and in each case, the alternative we rejected is as informative as the choice we made.

Forks are the statistical unit

Each JMH fork collapses to its mean, and verdicts come from an exact two-sided Mann-Whitney U test at α = 0.05 plus a seeded bootstrap confidence interval over three to five fork means per side. The reason is that iterations within a fork share JIT and heap state and are therefore autocorrelated, so treating them as independent samples manufactures significance out of nothing. We rejected comparing single-run scores, which is pure noise, and iteration-level t-tests, which can be confidently wrong. Seeding the bootstrap means that a rerun reproduces the verdict exactly because the agent needs to be able to tell the difference between a result that changed and a result that was never stable.

Pair candidate and baseline in time

The screen (three forks, optionally over a subset of parameters) exists only to kill bad hypotheses in maybe 10 minutes instead of an hour. The accept decision itself comes from a confirm run that measures candidate and baseline back to back using a stash-flip. In an unstable environment, thermal and background drift only cancels if both sides ran under the same conditions, so an hours-old baseline is really a different experiment.

The noise floor is measured, not assumed

The minimum effect size that a task will accept has to clear the A/A-calibrated coefficient of variation for that specific benchmark on that specific machine, and both the confirm run and the comparison refuse to proceed without a matching calibration record. A 1–2% improvement on a laptop is indistinguishable from noise, and because the floor varies by benchmark and by JDK, any global constant you pick will be too loose somewhere and too tight somewhere else.

The accept rule is composite and deliberately conservative

A parameter combination counts as improved only if p < α, the effect clears the calibrated floor, and the confidence interval excludes zero. Overall acceptance then requires that nothing regressed (not the primary benchmarks and not the guards) and that at least one primary combination improved. The headline figure is the geometric mean of the per-combination speedup ratios, which is always positive and composes across experiments, so a task's cumulative improvement is a meaningful number rather than a sum of incomparable percentages.

What must not get slower

Guards deserve a note of their own, because they answer a different question to the primary benchmarks; not Did this get faster? but What must not get slower while it does? The task definition lists them separately for that reason, and they're typically the operations and the input regimes that the change isn't aimed at. If you're optimizing insert throughput on a hash table, iteration is a guard and so is a collision-heavy key distribution. A change that improves the common case by weakening the hash function will look good on uniformly distributed keys while catastrophically degrading more adversarial inputs. We know that because it happened; the collision distribution was missing from the matrix that accepted one of our early experiments, and the task now carries a comment telling future readers never to drop it again for a hash-quality-sensitive scope. While guards cost wall-clock on every confirm, they also surface edge-case regressions, and omitting them can be much more costly in the long run.

Runtime isn't the only thing worth guarding, either. The confirm run also captures normalized allocation rate on both sides and flags any change that buys speed with more than about 15% extra garbage. That one is advisory rather than blocking, because sometimes the trade is the right one. However, it's the kind of regression a purely time-based accept rule would happily wave through but might raise a red flag to an experienced performance engineer with better understanding of the calling context.

The end-to-end gate is one-sided and default open

The end-to-end gate is a different statistical problem: small n, high noise, and a very strong prior that we should accept based on our microbenchmark results. Our first design treated accept and reject on an equal footing, and it produced multiple clearly spurious rejections, so the redesign is one-sided and default open. An operation is flagged only when the median regression exceeds the threshold and every candidate repetition is slower than every base repetition. That full-separation criterion is the nonparametric one-sided test at this sample size, and it's robust to the single outlier repetition that would occasionally fool us otherwise. A flag then also has to be corroborated against a differential CPU flame graph, where only a rise in the task's own in-scope CPU share counts as real. Near misses get reported for transparency but don't trigger triage, and nothing is ever auto-rejected; a flag is a request for human attention rather than a verdict. We have a final backstop which is the large suite of performance tests that we already run against Elasticsearch on a daily basis.

The one-sided gate lesson generalizes beyond benchmarking. A noisy gate should be one-sided and default open where there is strong prior reason to accept. A symmetric threshold on a noisy signal doesn't just cost you real wins, it teaches the loop to distrust its own instruments, and that’s a much more expensive failure.

Exploration and exploitation need different permissions

Exploration and exploitation might look like two phases of one activity, but they have different inputs (a macro workload versus a pinned scope) and different outputs (ranked opportunities versus commits). They also have different failure modes, which means they want different permissions. We made the split a first-class property of a task, which lets us enforce it; an exploration task literally cannot commit. The baselining, comparison, checkpointing, and validation CLI all refuse exploration tasks, benchmarking allows probes only, and every probe diff is always reverted. A broad, speculative survey is safe because nothing it does can edit the code.

A nice ancillary benefit is prompt focus. Each type reads one playbook, in full, with the others explicitly not loaded. If you try to write a single document covering both "find where the headroom is" and "land a validated win inside this scope", you get something that does neither well, because the instructions for good exploration (follow the profile, widen the net, a broad survey is preferred) are close to the opposite of the instructions for good exploitation (one hypothesis at a time, minimal diff, never widen scope).

AI agent memory: Journals, knowledge bases, and postmortems

Sessions are ephemeral, but what you can learn from them isn't, so the harness accumulates three durable assets, plus one disposable view derived from them. These are a journal of the code changes we’ve tried, a knowledge base of how the code performs, postmortems of when the harness failed and, because sessions can be stopped and resumed, a session summary. What makes them work together is a clean ownership rule about which kind of fact goes where.

The journal records what we tried and measured. It's append-only, one file per task, and one record per experiment, and it's written before the code is edited. Rejections carry a forward-looking note in the form "do not retry X because Y", which is probably the highest value line, because it's what stops the next session re-deriving a dead end. Records also carry the environment and the driving model, which means that hypothesis hit rates are comparable across models.

The knowledge base records how the code works and how it performs. It's an indexed collection of per-area summaries, each stamped with the commit it was written against, and every playbook ends with an upkeep step that appends whatever durable facts the run turned up. Because the performance characteristics of the JDK also change from time to time, for example, a new Vector API might implement vector masking more efficiently on AArch64, findings from profile data are also tagged with the JVM version they apply to.

The postmortems record mistakes that the agent has made in the past, and they're indexed by symptom rather than by date. The question a session actually has when a number looks wrong is Have I seen this shape of wrongness before?, and a chronological list doesn't answer it. So the rows read like "validation fails on operations structurally unrelated to your diff", or "screen reports no matching combinations".

Keeping the journal and the knowledge base distinct sounds pedantic but isn't because without the rule, both of them turn into a diary that has a tendency to bloat the context window or miss critical information in context.

The disposable state is a session handoff, and our advice is to never write it by hand and to avoid asking a model for a session summary, if possible. Our harness regenerates a one-page digest mechanically from the journal, so a resumed session doesn’t spend time and tokens reconstructing where the task had got to. Because it’s derived rather than authored, it can’t drift from the record in the way hand-maintained content does. That’s also why it isn’t part of the audit trail; because it’s cheap to regenerate it from the journal.

Mechanical session handoffs point to two lessons, and they turn out to be the same one. Every time a person appears in the loop, it’s a source of friction and an opportunity for error. And every time you reach for a model, ask whether code can do the same job. This seems like an odd thing to advocate in a project whose primary premise is delegating to a model, but the habit is easy to fall into once you have one to hand. Judgment is expensive, wherever it happens to sit, so the person and model both have to earn their place on merit.

Two further things are critical for durable agent memory. The first is that knowledge rots, so you have to lint it. Elasticsearch's main branch moves daily, which means the knowledge base's file citations go stale, so one linter checks them. Another checks that every command, flag, and path cited in the agent-facing docs actually exists, that every postmortem is linked from the index, and that every task type has a playbook, and it runs as part of the test suite. Treating prose written for an agent as a testable artifact is the reason it stays true. The second is that loading discipline is half of memory. The instruction is to load the index and then the one or two summaries matching the task's scope; never to bulk-load the rest. Memory you can't afford to read isn't memory.

Coding agent guardrails: Containment, scope, and stop conditions

Elasticsearch is millions of lines of code, and an unscoped "make it faster" run against a codebase that size is unreviewable and unfalsifiable. It’s also expensive. The converse is that a verifier only protects what it can see, so we have to apply the same boundary to what it can change. The outcome is a task that fixes its goal, allowed paths, benchmarks, thresholds, and stop conditions before any code is edited, and none of those are things the agent may change during a run.

The human owns the safety envelope, defining scope, thresholds, and stop conditions before any code is edited, and owning every handoff that crosses a trust boundary. The agent session owns judgment: reading profiles, forming one hypothesis at a time, editing inside scope. The atune CLI owns mechanical enforcement: scope checks, correctness gates, statistics, calibration, stop conditions, environment checks, and the signals. Persisted state is the record: a worktree pinned at a base commit, an append-only journal, and generated reports.

Containment happens in two layers. The first layer is coarse and task-agnostic; it’s a single static permissions file lets the session edit the per-task worktree (which includes a local branch of Elasticsearch, all journal entries, and CLI artifacts) and the knowledge base, and it denies the pristine Elasticsearch clone, the task definitions, the harness config, and the audit trail. The second is precise and per-task: a git-level scope check that runs over tracked and untracked files before any build, so an out-of-scope edit is rejected before it can be benchmarked or committed.

Two containment layers have a nice corollary. Widening the first layer to the whole worktree costs nothing, because nothing out of scope survives the second one. Coarse containment plus precise scope beats trying to make a single mechanism do both jobs, which is what we tried first and which left us with a permissions file that needed editing for every new task.

Stop conditions are mechanical. There's a maximum number of experiments and of consecutive rejections, along with a cumulative improvement target, and proposing a new experiment is refused once one of them fires. A human can override; the agent can't. That asymmetry is what makes the gates independent of which model happens to be driving.

Tests are add-only. New test files ship with the checkpoint, and modifying or deleting an existing test is blocked by the scope check. This closes the single most tempting shortcut in the entire problem space by construction rather than by instruction, which seems like the right way to handle any shortcut you'd otherwise have to keep asking an agent not to take.

The harness lives outside the code it optimizes. The Elasticsearch clone is a separate, gitignored directory, and each task gets a worktree pinned at a base commit. The payoffs compound; the subject repo stays pristine and upstream mergeable with nothing related to the harness leaking into a PR, and the audit trail is versioned independently of a codebase that moves daily. Also, tasks are isolated from each other and from any developer checkout. Targeting a newer Elasticsearch means creating a new task rather than repointing an old one, because that task's numbers are tied to its base. It also means that the harness is retargetable in principle, since the Elasticsearch-specific parts are configuration, knowledge base, and benchmark registry rather than architecture.

The decisions the agent never makes

The rule we settled on is that the agent runs the loops and a human owns every step that crosses a trust boundary. That is creating work, blessing a measurement instrument, publishing a branch, or acting outside the repo. None of those are in the agent's allowlist, and each has a reason worth stating:

• Humans have to sign off on the task because the thing being constrained can't set its own limits.

• Calibrating the noise floor needs to be done once per benchmark, per machine, and by default is measured rather than assumed. However, it's rather expensive and we allow a human to override if they know the environment well.

• Deciding what to optimize next by promoting an opportunity is a judgment and a new scope.

• Since benchmarks go on to gate other tasks, we consider reviewing this artifact part of the correctness safety net.

• We leave outward-facing actions, such as pushing a branch or filing an issue, to a human until we're confident in the process.

• We allow actions to be forced, but the override has to sit outside the thing being overridden.

How the human actions get surfaced in the workflow matters. The generated report and the session handoff both print the human actions currently due, at the moment they become due, rather than leaving them to be inferred from the playbook.

We're deliberately not taking a position on how permanent the manual processes are. The right amount of supervision for a new technology is an empirical question, and we'd rather measure it than argue about it. We’ve started with a relatively high degree of supervision because that's the cheap direction in which to be wrong (a gate you never needed is easier to remove than a regression you shipped) and because the harness makes the question answerable. Every gate is a named, logged transition, so over time we can see which of them ever changed an outcome and which only ever cost friction. In summary, measure first, and then refine.

Building the harness is the same kind of loop

A lot of the harness design didn't fall out of an initial design document. The signal set, the ranking function, the shape of a task, and the exact wording of a playbook rule each came from watching a run go wrong. If there's one piece of advice here that generalizes, it's to use the thing before it's ready and to instrument your own disappointment.

The clearest example is a rule we now call distrust surprising results. A validation run reported that every operation had regressed, the worst of them by 14.8%. It was wrong twice over. A target operation pattern had overmatched a completely different code path, and a stale output directory from an earlier run was being read alongside the new one. Offered a coherent story, the agent took it and reverted a change that was actually good.

What went into the playbook after that incident is not "be careful." It's a three-step check to run before acting on a surprising verdict:

1. Trace the code path, and confirm that the thing which moved can even reach your diff.

2. Read the raw per-repetition data rather than the summary, and recompute one headline number by hand.

3. Compare the report's shape against a known good run, because a structurally different report implicates the pipeline rather than the code.

Alongside that, there’s another important rule, which is if the agent concludes that the harness is buggy, it must not fix it mid-run, because a mid-run harness change makes every result in that run incomparable. It should journal the evidence and stop.

Improving the harness is itself a loop worth describing. Asking the model to review its own transcripts and the harness documents, and to propose the rule itself, usually works well. It's good at spotting where its own instructions were ambiguous, in a way that's hard to reproduce by rereading the instructions yourself. What makes that output useful is having somewhere for it to land: a terse rule in the playbook, the narrative in a dated postmortem, a symptom keyed index row, and a linter that keeps the citations honest.

Restraint turns out to be part of the same discipline. The design document carries an explicit list of extension points that we've deliberately not built, because the need for them is still speculative. That's the same "don't guess, wait for evidence" rule we impose on the optimization loop, applied to ourselves.

How this applies beyond performance optimization

A few of these themes aren't specific to performance work or to Elasticsearch.

Verifiable work is the current frontier. The same insight drives reinforcement learning with verifiable rewards, and it’s what AlphaEvolve is built around. The tasks agents are consistently good at are the ones that come with a cheap oracle (tests, compilers, benchmarks), and the interesting move isn't finding more such domains but manufacturing oracles for domains that lack them. Performance is an instructive case precisely because the oracle is, in some senses, obvious, and yet building it still took most of the engineering.

The referee pattern generalizes, too. Separating a fallible optimizer from mechanical enforcement is the same shape as sandboxed execution and policy engines: judgment in the model, invariants in code. The practical consequence is that the system's safety doesn't depend on which model drives it. A weaker model wastes benchmark time, but it can't corrupt the code or accept a bogus win.

Goodhart is a standing adversary for any optimization task that uses an agent, and it has a formal treatment worth reading. An agent optimizes exactly what you tell it to, so the two-tier benchmark structure, add-only tests, benchmark approval registry, adversarial guard distributions, and a marker file that makes timing an instrumented build mechanically impossible are all one design theme wearing different clothes.

Tools are context engineering. The consensus is drifting away from "expose everything" and toward a few well-shaped tools that return digests: the principles of progressive disclosure, self-documenting interfaces, and verdicts rather than payloads.

Memory is becoming architecture. A rules file, per-task playbooks, a durable knowledge base, and an append-only journal form a hierarchy with different lifetimes, owners, and loading rules, and the hard part is eviction and staleness rather than storage; hence, the linters.

Finally, human-in-the-loop is a dial rather than a switch, so where it should sit is something to measure per domain rather than assert.

What's in part 2 of this post

The harness design is a set of hypotheses about what autonomous performance work needs, and the harness was built so that we could test them. Part 2 is that test: the first four PRs we raised using it, their gains, how many hypotheses it took to get each one, which gates actually caught something, and where the harness got in its own way. That includes the changes which didn't survive end-to-end validation, since as usual, the rejections are as informative.

Managing context depends on good retrieval. Rather than have agents rediscover the same content for every question, burning tokens by retracing similar steps over and over again, Elastic’s agentic AI capabilities enable us to precompute these details and store them in a structured, searchable form, and they let agents load that context directly. We call this precomputed unit of context a Knowledge Indicator.

The default agentic retrieval augmented generation (RAG) pattern does the opposite. It retrieves whole documents and dumps them into the model's context at query time, paying for that retrieval in tokens and latency on every single question. Precomputing the answer as a KI moves that cost out of the hot path and does it once.

How it works: AI Index, Kibana Workflows, and the query-ki skill

Building context through AI indices has three main parts: the AI Index (a special Elasticsearch index where KIs live), Kibana Workflows to create your KIs, and a query-ki skill to help agents directly query KIs using ES|QL: 

This blog post is similar to Part 1 in that we’re using the same core building blocks. But in this post, we’re demonstrating a very different use case. Instead of precomputing index metadata, we’re distilling specific facts from our indexed documents that may be used to directly answer agents’ questions without subsequent searches. We've also provided a notebook, if you'd like to create the same KIs yourself, end to end, as you go through these examples. 

Prerequisites: Elasticsearch Serverless and an LLM API key

This tutorial assumes you have:

1. An Elasticsearch Serverless project. You can sign up for a trial if you don't have one.

2. An API key to access your Elasticsearch project.

3. An OpenAI-compatible large language model (LLM) API key, to access AI indices via Deep Agents scripts.

Load the BrowseComp-Plus sample corpus into Elasticsearch

First, we’ll need some sources. Sources can be data that already exists in your Elasticsearch indices or external data accessed via connectors or ES|QL data sources.  For this blog, we’ll create an index, browsecomp-plus, to hold our example data, with the following mappings:

and populate it with a small sample of BrowseComp-Plus data via the _bulk API. You can use the supporting notebook to load a sample of this data in your project. 

Create the AI Index that stores your KIs

Just like in Part 1, the first step is to create an AI Index:

This is preconfigured with the same required mappings as we listed out in Part 1. We perform hybrid search here using semantic_text out of the box.

How agents retrieve KIs using ES|QL

A KI is a document in the AI Index. What makes KIs useful is retrieval, or querying the AI Index to find the right content. This query is packaged within a small, portable skill that’s harness-agnostic and can be run in any agent harness. 

Here’s a sample query-ki skill:

Save this asskills/query-ki/SKILL.md.

Here’s what this skill is doing: 

• We’re defining corpus_entry as our KI use case.

• We’re performing a hybrid ES|QL search on our AI indices, filtering by the appropriate type, using reciprocal rank fusion (RRF) as the default method to fuse results.

• The KI results will directly ground the agent’s answer when determining what facts are relevant to the users’ query.

When we say that AI indices and KIs are harness-agnostic, it’s because the skill is just instructions plus a query. It will work in Elastic Agent Builder, a Kibana workflow agent, Claude Code, or any other harness. We’ll be using Deep Agents for examples of how to query it outside the Kibana ecosystem. Since an AI Index is, at its core, an Elasticsearch index, you can also explore your data directly. 

Precompute facts as KIs for agentic RAG

In this example, we extract actual facts so agents can retrieve an answer without consuming a full document. We generate one fact-based KI per selected document, though the actual number and structure of KIs you generate are completely customizable.

We'll use a sample of the BrowseComp-Plus corpus, indexed into a browsecomp-plus index, with docid, url, title, and text fields.

Baseline: Retrieving whole documents with RRF

As a baseline, here's a simple RRF query:

This drops several hundred words of raw body text into the model's context. It may work, but it's expensive, and the cost compounds with every miss.

Build the Kibana workflow

The workflow below reads a batch of documents with a single ES|QL query and writes one fact-level KI per document into the AI Index. Each iteration runs two steps: generate_ki distills a raw document into a structured KI, and sink_ki writes it to the AI Index keyed on docid so reruns are idempotent.

Copy and paste the following YAML into the Elastic Workflows editor:

Here’s what this workflow is doing: 

• query_corpus runs an ES|QL query against the browsecomp-plus index, applying some rules, like dropping documents with empty bodies and trimming each body to 12,000 chars so the agent prompt stays inside the context window.

• Note: In this example, we’re cherry-picking some concrete KI IDs, because generating KIs for every document in the index would take a long time, and we want this exercise to be short for those following along.

• loop_corpus_docs iterates over every returned document, running the following two steps per document: 

• generate_ki reads the document and calls an LLM to emit a strictly grounded, structured KI.

• sink_ki bulk-writes each KI into the AI Index (ai-index-idx-my-corpus) as a KI of type corpus_entry. It forces _id to be the same as the document’s docid so rerunning the workflow is idempotent.

To summarize, this workflow turns each raw corpus document into a compact, searchable metadata record that agents can find and judge without reading the full source into the context window.

This workflow is used for example purposes, and the same foreach caveat as in Part 1 applies. For scale, use workflow.executeAsync or native parallel support. The cheat sheet is useful for optimizing Workflows. There could also be cost and efficiency gains in production by using ai.prompt or by choosing different models with which to create KIs. 

Inspect the KIs in your AI Index

Once the workflow runs, you can query the AI Index to browse what was written:

Here’s an example of what one of the KI documents looks like: 

Query KIs from LangChain Deep Agents

We’ll use LangChain Deep Agents with an OpenAI-compatible key to show that AI indices and KIs will work with any agent harness, inside and outside of Kibana’s Agent Builder ecosystem. 

First, let’s create facts_baseline_agent.py to measure our baseline before applying KIs: 

When I ran this baseline query, What was the actress who played Torvi from Vikings also known for?, it output the following:

(Note: Deep Agents automatically adds the read_file tool to handle paginated tool results, which is why it shows up in the output.) 

Next, let’s create an agent that knows how to use our query-ki skill, facts_ki_agent.py: 

This agent will query the KI indices to generate the answer, rather than pulling every document into context. 

When I ran these results against the same query, here was the output: 

How much can precomputing facts reduce agent token usage?

Both agents had similar conclusions, but they took far different paths to get there: 

The same question and the same grounded answer result in 93% fewer tokens and two tool calls instead of eight, when answering from KIs.

Baseline (No AI Index)

With AI Index

Total tool calls

8

2

read_file calls

2

1

esql_query calls

6, all against the browsecomp-plus index

1, from ai-index-idx-*

Tokens consumed

386,187

27,625

Latency

44.86s

15.22s

Answer

Grounded, correct

Grounded, correct

Exact tool call counts, latency, and answers will vary between runs and using different agents. 

Both agents produced solid, grounded answers. The difference is cost. Querying KIs from the AI Index cut token use by 93% and cut latency by roughly two thirds. Here’s how both paths went, side by side:

That was in-depth, but it shows what AI indices and Workflows do together: the same answer, at a fraction of the tokens.

Build precomputed context in Elasticsearch Serverless

This walkthrough shows how to generate more sophisticated KIs based on documented facts and query them for knowledge retrieval use cases using Elasticsearch primitives. 

Managing context is critical in agentic search systems. And at its core, context is a retrieval problem. AI indices help you manage context within the Elastic Stack. Try it out in Serverless, and let us know what you think in our Discuss forums or the #stack-kibana channel in our Community Slack.

We’d also love to hear from you about what use cases you’d like to solve using AI indices.

NASA Aviation Safety Reporting System (ASRS) reports describe unusual events during flights, such as missed altitudes, confusing clearances, runway issues, or mechanical problems. Each report already has an official category, like altitude deviation, course deviation, or ground encounter. In this article, we ask a different question: What does this report reveal about the pilot who wrote it? The idea is to ask a model to infer a schema grounded on the data to classify the report based on criteria that help us figure out information about the report writers. Then ask a second model to apply the labels.

You can find the full workflow definitions and helper scripts here.

What you need to run this LLM pipeline

• Elastic Stack 9.4+ or Elastic Cloud Serverless. Elastic Workflows has been generally available (GA) since 9.4.

• Elastic Agent Builder enabled in your deployment.

• A Kibana generative AI (GenAI) connector pointing at Claude Sonnet (or an equivalent reasoning model). This is the planner.

• A Mistral API key. We’ll use it to register an Elasticsearch inference endpoint.

• Python 3.10+ with elasticsearch>=9.0 and pandas. Used by the dataset loader.

How two-tier LLM orchestration works

The workflow has two jobs: Decide what labels should exist, and then apply those labels to every report.

The first job is open-ended. A large model reads a varied sample of reports and proposes a small schema of categorical fields. A field is one way to describe the writer, such as attribution_style or procedure_orientation. Each field has a few allowed values, such as self_critical, system_attributing, or balanced.

The second job is repeatable. After a human approves the schema, a smaller model reads each report and chooses one value for each field.

We use Elastic Workflows because the steps are known ahead of time: sample reports, propose labels, wait for approval, classify every document, and store the results. Writing those steps in YAML makes the process reproducible, observable, and cheaper to rerun.

Why split LLM work across two model tiers?

A small model could handle classification, but schema discovery is a different shape of problem. It requires reading a diverse sample, spotting latent patterns, and proposing complex structures. In practice, smaller models over-anchor on surface keywords and produce redundant or nonexclusive fields.

Classification is simpler, the schema exists, the values are enumerated, and the task is to pick one per field. A smaller model handles this reliably and at a fraction of the cost, since it runs once per document across the entire corpus.

Large and small here mean reasoning capability. In this article, Claude Sonnet plays the planner and Mistral Small 3.1 plays the executor.

Classifying NASA pilot reports with a two-tier LLM pipeline

We’ll use the NASA ASRS database, which collects voluntary, anonymous incident reports from pilots, controllers, and mechanics. The dataset is public, and the reports are written as free-text narratives.

What we want to ask is:

What does this report reveal about the pilot who wrote it?

The planner reads a varied sample of reports and decides which distinctions are meaningful based on how the reports are actually written.

Step

Role

Model tier

sample

Pull a diverse subset of reports from the corpus.

(no LLM)

discover

Read the question and the sample, propose a schema of fields with enum values.

Large

approve

Human reviews the proposed schema and approves or edits it.

(Human via waitForInput)

apply

Iterate over the corpus, assign one value per field to each report.

Small

store

Write the schema and the per-document field values to Elasticsearch.

(No LLM)

Registering Mistral and Claude as Elasticsearch inference endpoints

The small model will be registered as an Elasticsearch inference endpoint using the native mistral service integration. 

The alias mistral-small-latest resolves to Mistral Small 3.1. It has a 128k context window and supports JSON-mode output.

The large model will be an AI connector pointing at Claude Sonnet. The Agent Builder UI walks you through creating the connector. Take a note of the connector ID since we’ll reference it from the workflow.

Indexing NASA ASRS incident reports into Elasticsearch

The ASRS dataset is indexed with keyword mappings for aggregation fields and text mappings for the narratives the models will read.

Download the ASRS CSV (the database publishes quarterly extracts at the ASRS Database Online page), and index it. The mappings are:

The mapping types follow how each field is used. flight_phase and anomaly are mapped as keyword because we’ll run terms aggregations on them to build the sample, and aggregations need exact, non-analyzed values. narrative and synopsis are mapped as text because they hold free-form prose that the models will read. The companion notebook has the full loader script that reads the CSV and bulk-indexes the documents.

Building a stratified sample for the planning LLM

The YAML snippets in this and the following sections are steps of the workflow definition that the notebook registers via the Workflows API. The first two steps generate a representative sample: They aggregate by flight phase and by anomaly and pull a few documents per bucket with top_hits.

How the large LLM discovers a classification schema from the data

The prompt needs both the question and the sample. A question alone may produce generic labels disconnected from the corpus, and a sample alone produces descriptive clusters that ignore the angle of the question. 

When both are present and the output is structured, the model produces labels that are grounded in the data and oriented to the task: a schema of categorical fields, each with two to four mutually exclusive value options backed by evidence from the sample.

Here’s the planner step from the workflow:

The structured output schema enforces the shape of the response:

• name: Identifier for the categorical field.

• definition: What this field measures, in one sentence.

• why_useful: How this field serves the question; this also helps the downstream classifier understand the intent.

• values: Two to four mutually exclusive options. Each has a value and a definition.

Here’s an example of the produced schema. We can see how the writer is being classified and the reasons why the model decided to create the category. definitionand why_useful fields are used by the second model to classify the documents.

Human-in-the-loop schema approval with waitForInput

The proposed schema is now passed to a person for approval. Elastic Workflows has a waitForInput step that pauses the workflow with a schema, exposes a form, and resumes when the input is submitted.

waitForInput has no timeout of its own, so if nobody responds, the execution waits indefinitely. To put a limit on that, set a workflow-level settings.timeout; if it elapses before the reviewer submits the form, the execution is canceled.

When the workflow reaches this step, the execution pauses and the Kibana UI shows an "Action is required" badge. Clicking Provide action opens a form where the reviewer can paste or edit the schema JSON. Since waitForInput cannot be prepopulated from a previous step, the code polls the discover step output and prints a paste-ready JSON block that can be copied directly into this form.

The step_output helper (in the notebook) polls the execution via GET /api/workflows/executions/{id} until the discover step completes and then returns its output.

Code JSON output pasted on Kibana:

The reviewer can keep useful fields, rewrite unclear ones, merge overlapping values, and add notes. After approval, the workflow resumes and sends the final schema to the executor step.

For a new corpus, keep this human gate in place. Once the schema is stable, you can auto-approve and only fall back to review when it’s worth it: Route just the low-confidence extractions to a person, or compare a new discovery run against the schema stored in the schemas index and trigger review only when fields or values change beyond a threshold.

Classifying the full corpus with a smaller LLM

By the time the workflow reaches this step, the open-ended part of the job is over. From here, the small model takes over and classifies each report against the approved schema.

Note: The classification step uses ai.agent instead of ai.prompt because ai.agent accepts an inference-id, which lets it call the Elasticsearch _inference endpoint directly, while ai.prompt only accepts a connector-id.

The fetch_corpus step is the third elasticsearch.request in the workflow, so it’s worth saying why we read from the index again. The first two (by_phase and by_anomaly) only pulled a small stratified sample for the planner to reason over, not the data to label. Now that the schema is approved, fetch_corpus pulls the documents we actually want to classify. We cap it at 100 with match_all to keep the demo fast; this is where you would page through the full corpus.

For every field, it returns a value (or null), a confidence, and a short quote. Setting additionalProperties: true in the JSON schema lets the step return one entry per field without the workflow having to know the field names ahead of time. A stored extraction looks like this:

Here, review_required is true because procedure_orientation came back at 0.44 confidence, below our 0.5 threshold, which is the signal a confidence-based quality gate would act on.

The fetch_corpus step pulls the documents to classify. The foreachstep iterates over them sequentially, and iteration-on-failure handles the errors: retry covers transient API errors from the inference endpoint, and, if all attempts fail, the fallback step posts to Slack so the failure doesn’t pass silently. (An email connector works the same way.) continue: true then lets the loop move on to the next document instead of failing the whole run. 

For production-scale corpora, consider using executeAsync, which is the fan-out version of execute.

Writing schemas and extractions back to Elasticsearch

The workflow produces two things: the approved schema and the per-document field values. The store_schema step runs right after the human gate, before the classification step fans out:

Each extraction is written inside the foreach loop, so results are persisted as they’re produced rather than batched at the end.

The schemas index holds one document per discovery run (question, approved fields, reviewer notes). The extractions index holds one document per report per schema version. 

What this two-tier LLM orchestration pattern gives you

We built one Elastic workflow that pulls a stratified sample from an incident report index, sends it with a question to a large reasoning model to generate a classification schema based on the data and a user-defined angle, pauses for human approval, and then iterates over the full corpus with a small Mistral model that assigns one value per field. 

The approved schema and per-document field values are written back to Elasticsearch as structured data.

The point of the exercise is that two different shapes of work, schema discovery, and schema application can use two different model tiers and that a workflow lets you write the routing decision down.

Next steps for your own LLM pipeline

• Try it on a corpus of your own. The pattern doesn’t care whether the input is incident reports, customer feedback, weekly status updates, or property listings.

• Promote the foreach step to workflow.executeAsync once you’re comfortable for parallel fan-out at scale.

• Schedule the rediscovery workflow on a cron trigger so you can discover different schema variations based on the data that comes in.

• Read the Elastic Workflows documentation for the full step catalog.

Related reading

• Build AI agents with Elastic Inference Service (EIS) covers the broader multi-model wiring pattern via EIS, complementary to the Workflows-orchestrated split shown here.

• How to build AI agentic workflows with Elasticsearch is a higher-level survey of how Agent Builder and Workflows fit together.

• LangExtract and Elasticsearch tutorial explores a different extraction pattern using a hand-authored schema; useful for contrast with the discover-then-apply approach above.

Does App A v1.2.3 support Feature X? Is it compatible with App B v9.8.7 when running on Kubernetes? Will I need more JVM heap space?

These are the kinds of questions that our field teams handle constantly. Answering them is harder than it looks. Documentation offers context, but it's an abstraction that can't anticipate every possible question. When we hit one it doesn't cover, our options are to interrupt an engineer who should be developing code or to hunt through that code ourselves. Often we don't have the time or expertise to navigate that much of it.

Coding agents do this well for a single repository on your laptop. Wouldn't it be great if we could scale that to our entire code estate? As a field engineer, I wanted that capability to serve my customers: agentic code intelligence across every project, dependency, platform, and version that we support. And I wanted it grounded in linked citations and always available to everyone as a service.

So I built it with Elasticsearch and Elastic Agent Builder and packaged it into a command line interface (CLI). I released it under an Apache 2.0 license and called it Sourcerer. This blog post reports multiple performance benchmarks of Sourcerer as a code research agent and walks through the design and rationale of its implementation.

Sourcerer

Sourcerer explores code like a frontier coding agent, searching across many versioned repositories as fast as it would in a single repository, and it generates answers with linked citations that establish trust.

Sourcerer consists of:

1. A set of configuration files for tools, skills, and agents in Agent Builder.

2. A set of index templates to store and search code from Git commit snapshots.

3. A CLI to install those assets and index and prune commit snapshots from remote Git repositories.

At Elastic, we're using Sourcerer to support our customers with verifiable information about our software directly from the source. Our internal deployment has indexed over a billion lines of code from our own public and private repositories. It also includes our core dependencies, such as Apache Lucene and OpenJDK, along with our common integrations, like Kubernetes and OpenTelemetry. Our solution architects, customer architects, consulting architects, and support engineers no longer have to hunt for answers in documentation or reach out to our engineers who should be building software rather than supporting it.

Code search benchmarks

Agentic code retrieval

SWE-Explore is a new benchmark, published on June 5, 2026, by Zhang et al., that evaluates "how well coding agents explore, localize, and rank repository context." It appears to be the only benchmark that specifically tests agentic code retrieval quality. I ran the benchmark with Sourcerer to see how it performs and compares to the other coding agents from the original paper, and again with Claude Code to measure and compare its token usage and task durations with Sourcerer's.

Retrieval scores

Sourcerer performs as well as frontier coding agents on relevance metrics for code retrieval (see Figure 1). The composite retrieval score is the arithmetic mean of all retrieval metrics weighed by their Pearson correlations (r) as reported in the paper; I did this to rank the agents by a measurement of overall retrieval quality. Sourcerer trailed Claude Code by 0.002 and surpassed Codex by 0.022 on a 0.0–1.0 scale, which should be interpreted as a statistical tie, given that the results vary slightly on each run due to the indeterminism of large language models (LLMs).

Generally, all the coding agents, including Sourcerer, performed well on precision metrics and suboptimally on recall metrics, although recall metrics had lower Pearson correlations and thus less importance. Table 1 shows Sourcerer's retrieval scores alongside the scores of the other agents tested in the original paper (page 8, table 6). "SignalReg" is the inverse of what the authors called "NoiseReg" (that is, 1 – NoiseReg); I did this to keep that metric consistent with the other metrics whose ranges imply that higher is better.

Token usage and task duration

Zhang et al. didn’t publish metrics for token usage or task durations, so I ran the benchmark again to capture those metrics for Claude Code. Given the high token cost of running the benchmark with any agent that uses an LLM, I opted to test only one coding agent, and Claude Code was the one that I expected most people would find useful as a comparison.

Compared to Claude Code, Sourcerer used ~13.9% more tokens to complete all 848 benchmark tasks. Sourcerer used 156,379,977 input tokens and 1,612,175 output tokens, while Claude Code used 137,629,435 input tokens and 1,137,176 output tokens. Sourcerer took ~8.9% longer to complete all 848 benchmark tasks. Sourcerer took 40,717 seconds, and Claude Code took 37,460 seconds.

I view these results on token usage and task duration as an acceptable modest tax in exchange for efficiently searching across multiple repositories and versions. That said, there’s room to explore optimizations to Sourcerer's tools, skills, and system prompt, the harness of Agent Builder, or the search engine of Elasticsearch and Lucene.

Single-repo vs. multi-repo scope

Critically, the SWE-Explore benchmarks only measure the retrieval scores, token usage, and task duration of agents searching within the boundaries of a single commit snapshot of a repository for any given task. This is the typical search space of a development coding agent. Sourcerer's intended scope is much broader, covering many commit snapshots of many repositories. The benchmark on "search speed and scalability," covered next in this report, shows Sourcerer's unique advantage when searching across many repositories.

Retrieval benchmark methodology

Appendix A explains the configuration of these benchmarks in detail.

Sourcerer searched all indexed representations of the SWE-Explore-Bench dataset (see Appendix A). Both benchmark runs used the same GPT-5.4 model that was used in the original paper. Sourcerer communicated with GPT-5.4 through Elastic Inference Service (EIS), while Claude Code communicated with GPT-5.4 through a shim proxy to be compatible with the OpenAI API.

I made a best effort to ensure that Sourcerer's benchmark task prompt was like-for-like with Claude Code's (see Appendix A). Both agents received identical instructions for their roles and tasks, along with output formats, and differed only in their brief harness-specific instructions. I instructed Sourcerer not to use its repo discovery skill and instead gave it explicit repo filtering instructions, ensuring that it was on a level playing field with Claude Code, which already receives the resolved directories. An alternative could have been to leave Sourcerer's repo discovery skill active while instructing Claude Code to find the repository in a filesystem that has all the repositories for the benchmark. I left Sourcerer's system prompts and skills, in addition to its tools, unmodified from their defaults, given that we're comparing the two harnesses overall, and much of which in Claude Code is closed source and not visible or controllable anyway.

Code search speed and scalability

Coding agents tend to use external tools to match substrings or regular expressions as a first line of retrieval. LLMs are trained to use shell commands, like ls and grep, when exploring code on a filesystem. Claude Code's own built-in Grep tool invokes ripgrep. This is the behavior I wanted to reproduce in Elasticsearch.

Elasticsearch has a wildcard field type that can scale regular expression matching to billions of documents. Sourcerer mimics the inputs and outputs of grep in Elasticsearch using sourcerer.code.grep, an Elasticsearch Query Language (ES|QL) tool that performs anRLIKE query on awildcard field of an index where each document has the contents of a single line of code. Sourcerer also providessourcerer.code.search, which performs a BM25-ranked MATCH query against an analyzed text field, for relevance-ranked discovery rather than exact substring retrieval.

I benchmarked the speed of both approaches against the speed of ripgrep and grep on a filesystem using two different corpus sizes and pattern rarities. Corpus sizes included one commit (7,215,509 lines of code) and 52 commits (200,325,684 lines of code) from the elastic/elasticsearch repository. The single commit covers the release tag for v9.4.3. The 52 commits cover the latest patch release tag for every major and minor release from v6.0.1 to v9.4.3. Sourcerer searched the corpus as indexed in Elasticsearch, while ripgrep and grep searched the corpus as stored on a filesystem, reflecting their respective use cases. The regular expression patterns included one that appears rarely among the commits (DiskBBQ) and one that appears commonly among the commits (XContentType).

sourcerer.code.grep

The first search I benchmarked was a rare pattern for DiskBBQ that appears only in some commits:

.*[dD][iI][sS][kK][-_]?[bB][bB][qQ].*

Search latency (in seconds) spanning a single commit (605 matches found from 7,215,509 lines of code):

Retrieval method

Cache

p0

p50

p100

stdev

vs. sourcerer.code.grep

sourcerer.code.grep

Cold

0.069s

0.124s

0.167s

0.020s

-

sourcerer.code.grep

Warm

0.027s

0.029s

0.046s

0.005s

-

ripgrep

Cold

0.788s

0.800s

0.809s

0.005s

~6.5x slower

ripgrep

Warm

0.081s

0.088s

0.109s

0.009s

~3.0x slower

grep

Cold

3.565s

3.580s

3.821s

0.055s

~28.9x slower

grep

Warm

0.825s

0.827s

0.833s

0.002s

~28.5x slower

Search latency (in seconds) spanning 52 commits (1,041 matches found from 200,325,684 lines of code):

Retrieval method

Cache

p0

p50

p100

stdev

vs. sourcerer.code.grep

sourcerer.code.grep

Cold

0.159s

0.164s

0.270s

0.028s

-

sourcerer.code.grep

Warm

0.027s

0.031s

0.053s

0.006s

-

ripgrep

Cold

22.356s

22.364s

22.459s

0.026s

~136.4x slower

ripgrep

Warm

16.017s

16.123s

16.297s

0.058s

~520.1x slower

grep

Cold

101.962s

102.386s

104.281s

0.752s

~624.3x slower

grep

Warm

73.082s

73.507s

74.915s

0.536s

~2,371.2x slower

Table 2. p0/p50/p100/stdev retrieval speeds of sourcerer.code.grep, ripgrep , and grep, under cold and warm caches, at two corpus scopes (20 runs per method per cache state; three warmup runs discarded before each warm-cache measurement). All percentiles computed via linear interpolation. Ratios are computed against sourcerer.code.grep's p50 at the matching cache state. See the “Methodology” section for cache definitions and query/command syntax.

The wildcard field type indexes trigrams of each value and uses them as a filter to narrow the candidate set for a regular expression before verifying full matches. For a selective pattern like this one (605 matches out of 7.2 million lines, 1041 out of 200 million) this approaches sublinear time complexity relative to corpus size. ripgrep and grep both perform scans with linear time complexity, with ripgrep using multithreading and single instruction, multiple data–accelerated (SIMD-accelerated) literal prefiltering to speed up searches, but neither has a mechanism to skip the vast majority of a corpus the way that an indexed trigram search can.

This shows up starkly in how each approach scales. Going from the single-commit corpus to the all-commits corpus is a 27.8x increase in line count. sourcerer.code.grep warm-cache time barely moves from 0.029 seconds to 0.031 seconds. ripgrep's warm-cache time goes from 0.089 seconds to 16.1 seconds, a 183x change; and grep's goes from 0.83 seconds to 1.2 minutes, an 89x change. Both filesystem tools scale worse than linearly with corpus size on this hardware, while the indexed approach is nearly flat.

The second search I benchmarked was a common pattern for XContentType that appears in all commits:

.*[xX][cC][oO][nN][tT][eE][nN][tT][tT][yY][pP][eE].*

Search latency (in seconds) spanning a single commit (7,999 matches found from 7,215,509 lines of code):

Retrieval method

Cache

p0

p50

p100

stdev

vs. sourcerer.code.grep

sourcerer.code.grep

Cold

0.158s

0.172s

0.224s

0.015s

-

sourcerer.code.grep

Warm

0.055s

0.057s

0.140s

0.023s

-

ripgrep

Cold

0.794s

0.804s

0.824s

0.007s

~4.7x slower

ripgrep

Warm

0.093s

0.095s

0.096s

0.001s

~1.7x slower

grep

Cold

3.385s

3.399s

3.421s

0.010s

~19.8x slower

grep

Warm

0.681s

0.683s

0.686s

0.001s

~12.1x slower

Search latency (in seconds) spanning 52 commits (290,662 matches found from 200,325,684 lines of code):

Retrieval method

Cache

p0

p50

p100

stdev

vs. sourcerer.code.grep

sourcerer.code.grep

Cold

1.587s

1.644s

1.756s

0.047s

-

sourcerer.code.grep

Warm

1.480s

1.541s

1.627s

0.040s

-

ripgrep

Cold

22.426s

22.440s

22.548s

0.028s

~13.6x slower

ripgrep

Warm

16.021s

16.276s

16.498s

0.119s

~10.6x slower

grep

Cold

97.699s

97.922s

99.150s

0.404s

~59.6x slower

grep

Warm

68.883s

69.184s

70.374s

0.360s

~44.9x slower

Table 3. p0/p50/p100/stdev retrieval speeds for the pattern .*[xX][cC][oO][nN][tT][eE][nN][tT][tT][yY][pP][eE].* under the same conditions as Table 2.

The relative search latencies of sourcerer.code.grep compared to grep shows why the pattern's match count matters as much as the corpus size:

Corpus

Corpus size

Cache

Speed of sourcerer.code.grep with a rare pattern (DiskBBQ)

Speed of sourcerer.code.grep with a common pattern (XContentType)

1 commit

7,215,509 lines

Cold

~28.9x faster

~19.8x faster

1 commit

7,215,509 lines

Warm

~28.5x faster

~12.1x faster

52 commits

200,325,684 lines

Cold

~624.3x faster

~59.6x faster

52 commits

200,325,684 lines

Warm

~2,371.2x faster

~44.9x faster

Table 4. This table shows how much faster sourcerer.code.grep was compared to grep when searching across two different corpus sizes and two different pattern rarities.

The pattern with far more matches shows a dramatically smaller Elasticsearch advantage, most strikingly at all-commits scope, where the advantage drops from 2,371x to 45x. The reason is visible in the absolute numbers: sourcerer.code.grep's warm-cache time at all-commits scope jumps from 0.031 seconds (DiskBBQ) to 1.541 seconds (XContentType), a 49.7x increase for a 279x increase in match count, while grep's warm-cache time barely changes (73.5 seconds to 69.2 seconds, effectively flat, since it scans the same number of bytes regardless of how many of them match). The wildcard field's trigram index has sublinear-in-corpus-size behavior that comes specifically from narrowing the candidate set before verification. Once a pattern matches hundreds of thousands of lines, the bottleneck shifts from narrowing candidates to collecting and serializing all of them, a cost that scales with match count rather than corpus size. sourcerer.code.grep still wins by a wide margin even in this less favorable case, but the margin depends heavily on how selective the search is, not just how large the corpus is.

sourcerer.code.search

Sourcerer's other retrieval tool, sourcerer.code.search, performs a BM25-ranked MATCH query rather than an exact-substring regex match. Its speed on both patterns is included below for reference.

Pattern

Corpus scope

p0

p50

p100

stdev

Matches found

DiskBBQ

One commit

0.013s

0.017s

0.018s

0.002s

288

DiskBBQ

52 commits

0.014s

0.020s

0.046s

0.007s

493

XContentType

One commit

0.062s

0.063s

0.154s

0.020s

7,177

XContentType

52 commits

2.203s

2.359s

2.611s

0.125s

249,737

Table 5. p0/p50/p100/stdev retrieval speeds of sourcerer.code.search, warm cache only, for both patterns at both corpus scopes (20 runs per row; cold-cache figures omitted; see Methodology). Match counts are sourcerer.code.search's own, not the regex-based methods' BM25 matches on tokens rather than substrings, so these aren’t directly comparable to Tables 2–4.

What drives the speed advantage

sourcerer.code.grep outperformed ripgrep and grep at every corpus scale and pattern rarity tested, along with every cache state tested. But it wasn’t by a fixed margin. The advantage ranged from ~3–30x on a common pattern to over 2,300x on a rare pattern, because indexed and brute-force search respond to different things. The trigram narrowing of the wildcard field does less work as a pattern gets more selective, while grep and ripgrep do the same amount of work regardless of how much of the corpus happens to match. sourcerer.code.search is a third option for the cases where the exact string isn't known at all.

Figure 2: Sourcerer's sourcerer.code.grep tool outperformed ripgrep and grep in every combination of corpus size and pattern rarity benchmarked, sometimes by multiple orders of magnitude. Its outperformance was strongest when searching rare patterns in large corpora and weakest when searching common patterns in small corpora.

These search speed benchmarks show what scalable means in practice for an enterprise code search agent. It's being able to search the histories of any number of repositories at interactive speeds, just like how a developer coding agent searches the working state of a single repository on a filesystem.

Speed benchmark methodology

Appendix B explains the configuration of this benchmark. Note that while the Elasticsearch deployment had two data nodes each with the same specs as the virtual machine used for ripgrep and grep, the benchmark consisted of one index with one primary shard and one replica shard. Each search ran on a single shard, which means that sourcerer.code.grep had the same amount of vCPUs and memory available per search as ripgrep and grep, despite having twice as much capacity across the overall deployment. Having two data nodes actually incurred a slight latency penalty compared to just one data node. For brevity, I've omitted results from the benchmark with one data node. We don't recommend single-node deployments in production, so it's worth including the realistic latency overhead that comes with a multi-node deployment in this benchmark.

I compared all retrieval methods under both cold and warm caches, 20 cold runs and 20 warm runs per method. The definitions of cold and warm caches weren’t like-for-like between the Elasticsearch and filesystem benchmarks. For ripgrep and grep, a cold cache meant performing a full-page cache drop (sync; echo 3 > /proc/sys/vm/drop_caches) immediately before each cold run; and a warm cache meant three discarded warmup executions immediately followed by the 20 measured runs, with no cache drops in between. For ES|QL, a cold cache meant calling POST /_cache/clear before each run. This only clears Elasticsearch's internal caches, not the OS page caches of the data nodes, which can't be cleared by hand on Elastic Cloud Hosted (ECH). A warm cache for ES|QL meant three discarded warmup queries immediately before the 20 measured runs, to help ensure that the caches on both data nodes would be warm. sourcerer.code.search's cold-cache figures are omitted from Table 5 for the same reason discussed elsewhere in this post: Its cold-cache measurements came back statistically indistinguishable from its own warm-cache measurements, evidence that the OS-level cache-clearing limitation affects it more than it affects sourcerer.code.grep, which showed a consistent, physically sensible cold/warm gap throughout.

Code search indexing throughput

I didn't conduct a formal benchmark of indexing throughout. I'll share my general observations instead.

Typically, I see a sustained indexing throughput of 20K–25K lines per second on data nodes that each have ~16GiB RAM and ~8 vCPU on c4a-highcpu instances on Google Cloud Platform (GCP). That includes writing to a primary shard and its replica shard. I've seen throughput as high as ~60K lines per second on Elastic Cloud Serverless with Search Power set to "Performant."

An engineering team at Elastic compared Sourcerer's indexing throughput to semantic code search implementations that used either sparse vector generation (Elastic Learned Sparse EncodeR [ELSER]) or dense vector embedding generation (Jina). Sourcerer indexed ~25x faster than .elser-2-elastic and ~15x faster than .jina-embeddings-v5-text-small, while retrieval quality was similar among all of them. More concretely, what took ~6 hours to index with ELSER took ~14 minutes to index with Sourcerer.

Code search solution design and rationale

The remainder of this blog post explains the rationale for my design decisions of Sourcerer, giving expert insights for practitioners of Elasticsearch and generative AI (GenAI).

Goals

Ultimately, we want an agent that answers questions about deployed software and its supporting infrastructure by searching the primary sources of truth (the code itself) and generating verifiable responses that cite those sources so they can be trusted. Inspired by the success of coding agents with grep, my main functional goal for Sourcerer was to reproduce the search behavior of a coding agent and generate responses with citations, all using Agent Builder. My nonfunctional goals were to keep it fast and scalable, as well as accurate, when searching across many versioned repositories, while maintaining acceptable costs and ease of use. Of these goals, reproducing the search behaviors of coding agents would be the most consequential, as it would dictate the access pattern, index design, and query design, plus their effects on nonfunctional goals.

Access pattern

From a human perspective, the intended access pattern is simple: We expect to ask natural language questions about software and receive plain language answers grounded in the source of truth. From the perspective of the agent handling those questions, the intended access pattern is to reproduce the search behavior of coding agents to find what it needs. The LLMs used by coding agents are heavily trained to explore code with shell commands, likelsor find, grep or ripgrep, cat, head, tail, and so on. Claude Code's built-in tools, such as Glob and Grep, provide similar functions.

I chose to go with the grain of how models are trained. So my intended access pattern for Sourcerer was to reproduce the names, inputs, and outputs of shell commands as ES|QL tools in Agent Builder. That way, the Agent Builder harness would allow an LLM to use its trained intuition to achieve similar results as a frontier harness, like Claude Code or Codex, without having to fill the LLM's limited context window with instructions for using a different search interface. This was the main context engineering problem to solve with Sourcerer. Solving it would enable faster searches that scale across many repositories at once, allowing an agent to answer questions about deployments in which many different versioned software projects work together.

Index design

I decided on three index templates to fulfill this access pattern:

• sourcerer-refs: Each document indexes the high-level metadata for a single Git reference or ref identified by its unique commit hash, which can have a tag name or branch name associated with it. A ref represents an entire snapshot of a repository at a point in time. This is a small index. The agent mainly uses this to discover the repositories and snapshots that are available to search.

• sourcerer-files: Each document indexes the metadata for a single file of a given ref. This is a larger index. The agent mainly uses this to navigate files and directories using ls semantics.

• sourcerer-lines: Each document indexes the contents of a single, numbered line of code for a given file of a given ref. Yes, every line of code becomes a document. This is the largest index (but perhaps not as large as you might expect). The agent mainly uses this to search and view code using grep and cat semantics.

The design is almost entirely denormalized. Each index has the same namespacing fields for fast, joinless filtering. Each indexed ref stores all files and lines from its commit snapshot, rather than storing diffs and reconstructing them at search time or storing unique files and lines with a mutable array of ref names associated with each. These choices trade duplicative storage (the cheapest compute resource) for faster searches and less segment merging pressure.

Namespacing

All three indices use four fields to namespace the ref, file, or line of code:

• git.host: A Git hosting provider (for example, github, gitlab).

• git.org: An account name (for example, elastic).

• git.repo: A Git repository (for example, elasticsearch, kibana).

• git.commit: A commit hash, stored as the full 40-character SHA-1 digest for integrity.

The document _id hashes for files and lines are also namespaced by {git.host}, {git.org}, {git.repo}, and {git.commit} to allow for idempotent indexing. That means you can safely rerun an indexing job without duplicating any documents.

Likewise, the index names are namespaced with the same semantics, using tildes (~) as a reliable separator since it's a disallowed character in Git repository names and organization names:

• sourcerer-v*-files~{git.host}~{git.org}~{git.repo}

• sourcerer-v*-lines~{git.host}~{git.org}~{git.repo}

This namespace convention has many benefits:

• Agents can quickly narrow the search space for refs and files, along with lines of code, by these common scoping fields, keeping searches fast and focused.

• The semantics reflect common permission boundaries. You can reproduce the access policies of your Git hosting provider by implementing your choice of index-level security and/or document-level security based on host or organization or based on repository.

• You can instantly delete indices for a whole repository or organization, or for a host, without an expensive _delete_by_query.

• The index names are future proofed for different levels of granularity. Sourcerer might eventually allow indexing code by {git.host}, {git.host}~{git.org}, or {git.host}~{git.org}~{git.repo}~{git.commit} for selective shard sizing optimizations. The query syntax would be unaffected because they target index aliases (sourcerer-files and sourcerer-lines), not individual indices.

Settings

Three index settings help to optimize storage costs and search speed:

• Index sorting gives faster searches and better compression at the cost of reduced indexing throughput. Each index sorts documents on disk by git.host, git.org, git.repo, git.commit. File and line documents are further sorted by file.path, and line documents are further sorted by line.number.

• Synthetic _source discards _source and instead reconstructs it as needed when reindexing. None of the queries access _source, which makes it dead weight. Enabling this setting reclaims ~50% storage space in the files and lines indices. While it requires an Enterprise license, enabling it on a non-licensed deployment won’t prevent the index from being created; instead the setting will be ignored.

• best_compression is a fallback for Elastic deployments that lack an Enterprise license to use synthetic _source. It provides decent compression for _source (~11% storage savings by my observations) in exchange for a modest tax on indexing throughput (~15% slower), while search speeds are essentially unaffected because the queries don't fetch _source.

I use index aliases to support zero-downtime upgrades when reindexing to a new schema.

Mappings

The indices mainly use keyword fields. They facilitate efficient filtering with basic wildcard support and aggregations, along with optimal storage usage and indexing throughput.

The line.content field is indexed both as a wildcard field and as a text field with tokenization and similarity settings tuned for code search. This gives agents the option to search code using familiar and effective grep-like regular expressions on the wildcard field or using the inverted index of the text field to return only the highest ranking matched lines to reduce token usage and increase search speed. Both options are remarkably fast and scalable, taking milliseconds to finish in most cases.

Shards

While shards are becoming less relevant with the rise of stateless platforms like Elastic Cloud Serverless, I want the solution to accommodate all deployment modes of Elasticsearch. So I've given attention to the effect of the index design on the number and size of primary shards. Based on my observations, I expect that most users won’t need to give much attention to shards.

By default, Elasticsearch enforces a soft limit of 1,000 shards per data node (including replicas). That means this solution will hit a soft limit of just under 250 repositories indexed per data node, because each repository is written to a files index and a lines index, each with one primary shard and one replica. Additionally, there’s a conventional best practice of limiting shard sizes to ~50GB, which affects how many refs you can index per repository. Both of these limits can be pushed a bit. But they reveal that this solution design really is optimized for its intended use case of searching the commit snapshots of supported, deployed software. You wouldn't use Sourcerer to index every repository on the Internet, and you shouldn't use it to index every ephemeral development branch. Plus, you should decide how many refs are worth retaining for each repository.

Repository-level granularity of indices appears to strike the right balance of shard counts and shard sizes. For reference, Kibana is one of the largest repositories on GitHub (source). I observed its shard size to be a manageable 60GB–75GB when retaining only the latest patch release for every major and minor version release from v6.0.0 to v9.5.0. That's great coverage for the Elastic deployments we see in the wild. If that's one of the largest repositories out there, you can expect just about any other repository to fit in a single shard as long as you have a reasonable retention policy, which I discuss in the next section (“Pruning”).

Pruning

By default, Sourcerer retains everything you index. Pruning lets you delete old refs to prevent unbounded growth. You can define ref retention policies based on the age of refs and the number of refs indexed in the repo. You can also define these policies based on the number of semantic versions indexed in the repo at any level of granularity for major, minor, patch, build, and prerelease versions. Some common configurations are to retain only the latest commit of the default branch or the most recent patch release tag for every major and minor version release tag.

Elastic Agent Builder tools

With the index design in place, we can review the tools that query those indices.

ES|QL tools in Agent Builder are parameterized ES|QL queries with descriptions to guide the agent's use of them. Sourcerer has tools for several purposes: repo discovery, file discovery, code search, and code display. These tools reproduce the names, inputs, and outputs of shell commands that coding agents prefer to use when exploring code, making them intuitive enough for the LLM to use with minimal instructions passed into its context window.

Repo discovery

These are typically the first tools that the agent calls. Unlike most coding agents, which search within a single repository on a filesystem, Sourcerer is aware that its search space likely has multiple repositories and versions, and so its first step is to decide which repos and refs to scope its searches to.

• sourcerer.repos.list: Lists the repos that are available to search.

• sourcerer.repos.search: Lists the repos whose file contents best match a given query.

• sourcerer.refs.list: Lists the repos and refs that are available to search.

File discovery

All file discovery tools support glob matching (* and **) on file paths.

• sourcerer.files.ls: Lists files and directories that match a given pattern.

• sourcerer.files.tree: Lists files and directories that match a given pattern in a tree-like format.

• sourcerer.files.wc: Counts lines, words, characters, bytes, and longest lines for each matching file.

Code search

All file code search tools support glob matching (* and **) on file paths.

• sourcerer.code.grep: Searches lines of code using RLIKE on a wildcard field for rapid execution of regular expressions.

• sourcerer.code.search: Searches lines of code using MATCH on a text field that has been tuned for code search.

Code retrieval

All file code retrieval tools concatenate the desired lines of any matching file and return them as a single, contiguous block of code in grep -n format, which is a format preferred by coding agents, including Claude Code's built-in Grep tool. This lets the agent see a faithful representation of file contents with line-level attribution for precise citations, without requiring the agent to reconstruct the contents or infer line numbers through reasoning.

To illustrate, here's how the sourcerer.files.head tool formats the first five lines of Kibana's README.md file, reconstructed from five documents from the lines index:

All file code retrieval tools support glob matching (* and **) on file paths. Agents typically use these tools to display the contents of a single file.

• sourcerer.files.cat: Concatenates and displays all lines for each matching file.

• sourcerer.files.head: Concatenates and displays the first n lines for each matching file.

• sourcerer.files.tail: Concatenates and displays the last n lines for each matching file.

• sourcerer.files.read_lines: Concatenates and displays the range of lines between two given line numbers for each matching file.

Agent Builder skills

With the tools implemented, we can review the skills that guide the agent's proper use of them. Agents typically invoke the follow skills in this order:

• sourcerer-repo-discovery: Guides the agent in discovering and selecting the repositories that are available to search for a given prompt. While this is typically the first skill an agent invokes, the agent might return to it when tracing dependencies from other repositories or when answering questions that span multiple repositories or versions.

• sourcerer-ref-resolution: Guides the agent in resolving the names of tags or branches to their unique, immutable commit hashes. This lets the agent reliably filter its searches to a single commit snapshot.

• sourcerer-code-search: Guides the agent in exploring code, with basic best practices on when and how to use the available tools for maximum efficiency.

• sourcerer-code-citations: Guides the agent in citing files, directories, lines of code, and ranges of lines of code. Sourcerer auto-generates more specific citation skills for each major Git hosting provider, so that its citation links conform to the URL formats of each respective host.

Agent system prompt

The final packaging of the agent comes with a system prompt that succinctly describes the agent's role and its high-level instructions. The configure file that defines the system prompt also defines the tools and skills that are made available to the agent, so that it can only execute what we permit it to execute.

Sourcerer CLI

The Sourcerer CLI assists with setup and indexing, along with pruning to keep operations simple. Configuration is managed through a sourcerer.yml configuration file.

• sourcerer setup: Idempotently loads the index templates and Agent Builder configurations, in addition to Kibana dashboards. This is typically a one-time operation and takes a few seconds.

• sourcerer index: Checks for new refs that match patterns defined in sourcerer.yml and then idempotently indexes them, skipping any refs that have already been indexed or that qualify for pruning based on retention policies. It calls git to clone repos and to list remote refs, as well as to  check out refs.

• sourcerer prune: Checks the retention policies for any refs that qualify for pruning and then deletes them from all three indices using _delete_by_query.

You can easily schedule indexing and pruning using external schedulers, like cron. For our internal use at Elastic, we maintain sourcerer.yml files in a private Git repository and schedule indexing and pruning with GitHub Actions. I prefer to index code frequently, while pruning outside of normal working hours to prevent agents from suddenly losing context in the middle of a conversation.

Feature license summary

For transparency, here’s a summary of the license levels for all non–open source software (non-OSS) features referenced in this solution design.

Subscription features:

• Agent Builder (optional; you can query the indices from a different harness using the Elasticsearch API).

• Synthetic _source (optional).

• Document-level security (optional).

Free features, proprietary to Elastic (not OSS as defined by the Open Source Initiative [OSI]):

• ES|QL.

• wildcard field.

A subscription gives you the magic of "everything just works" with Agent Builder, along with resource optimizations, finer security permissions, and platform support. Without a subscription, you can still index and prune code with the Sourcerer CLI and search the code using the Elasticsearch API.

Conclusion

Sourcerer demonstrates that Elasticsearch + Agent Builder is an exceptional solution for agentic enterprise code intelligence, as evidenced in many ways:

• Accurate: Sourcerer's agentic code retrieval quality is at parity with frontier coding agents as demonstrated by its performance on the academic benchmark SWE-Explore.

• Fast: Sourcerer's query speed ranges from milliseconds in a single repository to multiple seconds across a billion lines of code. Indexing is also much faster than could be achieved with vector embedding generations.

• Scalable: Elasticsearch sharding enables horizontal scaling, making it possible to search the current and historical states of an entire enterprise software estate. Alternatively, the stateless architecture of Serverless naturally scales without having to plan shards.

• Resilient: Elasticsearch replication enables high availability to keep the agent operational 24/7. Likewise, the stateless architecture of Serverless naturally provides high availability.

• Polyglot: Sourcerer's indexing and retrieval methods are completely language-agnostic and tolerant of malformed code.

• Secure: Elasticsearch document-level security enforces access policies for humans and agents at the organization, repository, and commit levels.

• Efficient: Sourcerer demonstrates an efficient use of storage, memory, compute, and token consumption for its intended use case.

• Manageable: The Sourcerer CLI, and its use of native git commands and Elastic REST APIs, makes it easy to get started with and operate, as well as schedule.

• Universal: Organizations from all industries build software, and Git is the source of truth for ~85% of them (source). Sourcerer has value to all of these organizations. 

Sourcerer is currently less than two months old, and the benchmark results suggest that there’s room to improve recall and token efficiency, along with task duration. I'll continue to work on this project in the near future.

Try it yourself

Sourcerer depends on Elasticsearch and Kibana. You can get started with those in a couple ways:

• Elastic Cloud (includes an Enterprise trial).

• Local setup with Docker (includes an Enterprise trial).

Then you can get started with Sourcerer.

I built this with Agent Builder. What will you build?

References

Sen, S. (2026). Is Grep All You Need? How Agent Harnesses Reshape Agentic Search. arXiv. https://arxiv.org/abs/2605.15184

Zhang, S. (2026). SWE-Explore: Benchmarking How Coding Agents Explore Repositories. arXiv. https://arxiv.org/abs/2606.07297

Appendices

Appendix A. SWE-Explore benchmark configuration

Benchmark environment:

• Sourcerer version: v1.0.0 (commit hash: 26d2e84e3f9e4c1e1598a48532eba9a739465c3e)

• Elastic Cloud Hosted (ECH):

• Region: GCP - Los Angeles (us-west2)

• CPU Optimized hardware (c4a-highcpu)

• 3x Elasticsearch data nodes (each with 16GiB RAM, 8 vCPUs)

• 2x Kibana instances (each with 2GiB RAM)

• Elastic stack version: v9.5.0 (commit hash: dbedef007f580447413782705acb9afec41f945b)

• Claude Code version: 2.1.202

• LLM: GPT-5.4

Indices as reported by GET /_cat/indices:

Benchmark task prompt:

Table 6. This table compares Claude Code's task prompt used in the original paper and Sourcerer's task prompt used in this benchmark run. The highlights indicate where Sourcerer's task prompt differed from Claude Code's: red indicates an instruction from Claude Code's task prompt that wasn't used by Sourcerer's, and green indicates an instruction that was unique to Sourcerer's task prompt.

Appendix B. Code search speed and scalability benchmark configuration

Benchmark environment:

• Sourcerer version: v2.0.0 (commit hash: dcc373b3bf7b24168e00ed7b172aea7227f705f9)

• Elastic Cloud Hosted (ECH) for ES|QL:

• Region: GCP - Los Angeles (us-west2)

• CPU Optimized hardware (c4a-highcpu)

• 2x Elasticsearch data nodes (each with 16GiB RAM, 8 vCPUs)

• 1x Kibana instances (2GiB RAM)

• Elastic stack version: v9.5.0 (commit hash: 8d4246a64bc255212407b1b313fe402391299c88)

• Virtual machine for ripgrep and grep:

• Region: GCP - Los Angeles (us-west2-a)

• CPU Optimized hardware (c4a-highcpu-8-lssd)

• 16GiB RAM

• Image: projects/ubuntu-os-cloud/global/images/ubuntu-2404-noble-arm64-v20260717

• Provisioned IOPS: 3300

• Provisioned throughput: 215

Indices as reported by GET /_cat/indices:

Cluster settings adjusted to avoid truncating the returned match count on frequent patterns:

Regular expression used by ripgrep and grep:

• DiskBBQ: .*[dD][iI][sS][kK][-_]?[bB][bB][qQ].*

• XContentType: .*[xX][cC][oO][nN][tT][eE][nN][tT][tT][yY][pP][eE].*

ES|QL query syntax used for sourcerer.code.grep:

ES|QL query syntax used for sourcerer.code.search:

ES|QL query parameters used in all tests:

• git_host="github"

• git_org="elastic"

• git_repo="elasticsearch"

• n=1000000

ES|QL query parameters used for specific tests:

• Corpus with single commit: git_commit="45f6a06b1b441b41fe711059b8720013173e7c89"

• Corpus with all commits: git_commit="*"

• sourcerer.code.grep pattern for rare pattern (DiskBBQ): regex=".*[dD][iI][sS][kK][-_]?[bB][bB][qQ].*"

• sourcerer.code.grep pattern for common pattern (XContentType): regex=".*[xX][cC][oO][nN][tT][eE][nN][tT][tT][yY][pP][eE].*"

• sourcerer.code.search pattern for common pattern (DiskBBQ): q="DiskBBQ"

• sourcerer.code.search pattern for common pattern (XContentType): q="XContentType"

n was set high enough to recover the true, uncapped match count for each pattern (605 and 1,041 matches for DiskBBQ;  7,999 and 290,662 matches for XContentType), confirmed in each case by comparing hits_returned against ripgrep's and grep's own match counts on the same underlying data.

Command syntax used for ripgrep :

rg -n --no-ignore -j 6

Command syntax used for grep:

grep -E -r -n --binary-files=without-match --exclude-dir=.git

Directories of each cloned repository snapshot and the approximate total sizes of their nonbinary files tracked by Git, which is the search space of ripgrep and grep:

Why OAuth is better than API keys for MCP server authentication

With OAuth, tokens are short-lived credentials that expire on their own, so a leaked token is a narrowing window rather than a standing grant. Every token traces back to an explicit consent: which user, which client, which time.

In contrast, an API key is a long-lived credential. It lives in a configuration file on the machine that runs the agent, working for whoever uses it until someone rotates or deletes it. That model is manageable for a CI pipeline you wrote and deployed for your team. It gets uncomfortable when the key is held by an AI agent that assembles its own requests, retrieves untrusted content that may contain injected instructions, and sometimes passes context to sub-agents.

The failure mode is familiar from every credential-leak postmortem: the key ends up somewhere it shouldn't (a log file, a prompt), and from that moment anyone who has the key can access everything its creator could. The audit trail doesn't help much: API key logs tell you that a key with a given name did something, but not who authorized the client that used it or when.

Attribute

OAuth 2.1

API Key

Credential lifetime

Short-lived, auto-refreshing

Long-lived until manually rotated

Audit trail

User, client and timestamp per connection

Key name only

Revocation

Per connection, immediate

Requires key rotation

Blast radius

Single client connection

Everything the key creator can access

How to set up OAuth 2.1 for the Elasticsearch MCP server

Setup is a one-time step per project:

1. In your Elastic Cloud Serverless project, open Agent Builder → Tools library → MCP clients → Create MCP client (OAuth). This gives you a client ID and the MCP server URL.

2. Add the server to your MCP host. In Cursor or Claude Desktop, the configuration file entry looks like this:

3. The first time the agent calls a tool, the host opens your browser on an Elastic Cloud consent screen. You sign in with your normal Elastic Cloud credentials and see what the agent is requesting access to.

4. Click Authorize. This creates an application connection between you, your machine, and your project, and Elastic Cloud issues the agent a short-lived access token.

From there, your agent has access to your project’s data. The mechanics are invisible. When the access token expires, the host refreshes it using a refresh token with a 30-day rolling expiry, so you are not re-authenticating every hour. If you stop trusting a connection, open the application connections page in Elastic Cloud and revoke the connection. Both tokens die immediately, and nothing else about your account changes.

The agent acts with the permissions of the user who consented. Calls to Agent Builder tools, such as ES|QL queries, Workflows, and Streams, are evaluated based on your role. If you want an agent with less access than your own, authorize the connection as a user with a tighter role, or keep using a scoped API key for that workload. 

How to manage and revoke MCP server connections in Elastic Cloud

Org owners can list every active application connection across an organization or a specific project: which MCP hosts were authorized, by whom, and when. Revocation is per connection, so cutting off one misbehaving agent does not disturb the authorizing user's account or any other integration. There is no shared credential to rotate and no blast radius beyond the one client.

This is the practical difference for teams. An OAuth app connection records who consented, to which client, and when. An API key log shows that agent-key-3 queried an index, but not who authorized that agent. 

What's next for OAuth authentication across the Elastic API

Agent Builder is the first Elastic surface behind OAuth, and the same authorization model will carry to the rest of the Elastic API surface, including Elasticsearch and Elastic Cloud management, as we work toward a single Elastic MCP endpoint.

To try it now, open Agent Builder in an Elastic Cloud Serverless project and create an OAuth client, or start with the documentation. If you don't have a project yet, you can start a free trial.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

While AI indices will be included in future Stack releases, today we recommend using Serverless.

How it works: AI Index, Kibana Workflows, and the query-ki skill

Building context in this walkthrough has three moving parts:

1. An AI Index, where KIs live. It's a regular Elasticsearch index or data stream with a specific naming convention triggering component templates to configure the right mappings automatically.

2. Kibana Workflows, which read from your data sources, run an LLM to structure content into KIs, and write those KIs into the AI Index.

3. A query-ki skill, a skill that queries KIs directly from the AI Index using ES|QL, and that a chat agent can call as a tool.

Prerequisites

This tutorial assumes you have:

1. An Elasticsearch Serverless project. You can sign up for a trial if you don't have one.

2. An API key to access your Elasticsearch project. 

Create sample indices for agent routing

First, we’ll need some sources. Sources can be data that already exists in your Elasticsearch indices, or external data accessed via connectors or ES|QL data sources. 

For this blog, we’ll create some indices with example data. We’ll start with an example using three datasets: BEIR/fiqa (financial), beir-nfcorpus (biomedical/nutrition), and beir-scifact (scientific fact-checking). Each index is populated with its own _meta.description. 

Here are the mappings we define for these indices:

Then, using the above convenience scripts, load a handful of documents into each index with the _bulk API.

Now imagine an agent with a question and the indices we’ve just created. The agent has no idea which one is relevant at the start. Without pre-computed context, it either performs exploratory lookups (mappings, test searches) to figure out which source to use, or searches all three and hopes the merged results contain something useful. Either approach costs tokens, and if you multiply that inefficiency across every query an agent makes, it adds up.

Create your AI Index

Before generating any KIs, you need an index to store them. We call this an AI Index.

The naming convention is what triggers automatic configuration. Any index whose name starts with ai-index-idx- is a regular index; ai-index-ds- is a data stream. You’ll want to choose data streams for observability use cases, time series data, and when recency is important. Conversely, standard indices are a good choice for static data that will exist for a long while, where recency is not as much of a concern, and may need to occasionally be updated on demand. This naming convention is required for AI indices. 

When Elasticsearch sees the ai-index- prefixes, it automatically applies component templates that configure the right mappings and settings.

Creating an AI Index is a single call:

To see exactly what the component templates applied, inspect the mappings:

The response shows the fields every AI Index gets out of the box:

title, description, and content are each a text field with a .semantic sub-field of type semantic_text, supporting hybrid retrieval.

Data stream indices (ai-index-ds-*) additionally carry a default 90-day data retention policy. This blog uses a standard index (ai-index-idx-*).

Index Metadata as a Knowledge Indicator

The target use case for this example is how the query-index-metadata-ki skill can route an agent to the correct Elasticsearch index, even when index or field names are vague. This reduces mistakes from choosing the wrong index or formulating queries based on incomplete schema exploration.

Since we're creating KIs for our own indices, we can give the LLM a head start: annotate index mappings with human-written _meta.description content. The workflow generates better KIs with more context to work from.

To address this, we'll manually create a Kibana Workflow that profiles each index and writes routing KIs into the AI Index. The workflow chains four steps:

Step

Type

What it does

get_mapping

elasticsearch.request

Read the mapping, including _meta.description and per-field descriptions.

sample_docs

elasticsearch.search

Pull a few real documents so the profile reflects actual value shapes.

profile_index

ai.agent

Generate a structured index profile as structured output.

sink_index_ki

elasticsearch.bulk

Write the profile into the AI Index as a KI.

Paste the following YAML into the Workflows editor:

Let's walk through what this workflow does. We loop over three specified indices with a foreach loop. For each:

1. get_mapping fetches the Elasticsearch index mappings, including any _meta.description annotations we added earlier.

2. sample_docs pulls 3 real documents. Concrete examples give the LLM much better signal than schema alone.

3. profile_index calls ai.agent with the index name, mappings, and sample documents. The LLM returns structured output describing the index's purpose, key fields, and an example ES|QL query showing how to use it.

4. sink_index_ki writes the result into the AI Index as a KI of type index_metadata_entry, keyed on the index name so re-runs are idempotent.

A few things to point out: 

• This workflow hard-codes a specific set of indices. In practice, you could derive the list from an index pattern or a dynamic source. 

• The foreach loop also runs iterations sequentially, which is fine for this guide but slow in production because each iteration involves an LLM call. For scale, use workflow.executeAsync or native parallel support. The cheat sheet has tips on both.

• In the profile_index step, the agent prompt is the special sauce. This is what shapes the accuracy and usefulness of the KIs. 

• Using ai.prompt can improve workflow efficiency (and cost) if you don’t need to load other tools. 

• Cost can be controlled in multiple ways. Richer prompts and structured output often result in higher token utilization, and of course the model you choose significantly impacts total costs. The Elastic Inference Service (EIS) can be a great playground to test different models against the profile_index’s ai.agent step to compare how different models stack up against each other when generating KIs. 

Query your AI Index to verify Knowledge Indicators

Once the beir-index-profile-ki workflow runs, query the AI Index directly in the Discover tab using the following ES|QL query to confirm what got written:

This will result in the following output: 

Build a portable skill to retrieve AI agent context

Retrieval is a critical component in an AI index. A KI is a document in the AI Index, and finding one is a single ES|QL query. We package that query as a small, portable skill so any agent can call it, regardless of the harness it runs in.

We write the skill as a SKILL.md: a YAML header with a name and description, followed by markdown instructions. This is the same Agent Skills format that many harnesses, including Claude Code, LangChain's Deep Agents, and others, load directly. 

The harness reads the header content up front, and only pulls in the full instructions when a question matches the description. The one thing the skill asks of the harness is a way to run ES|QL against Elasticsearch.

Here is a sample query-index-metadata-ki skill:

Let’s break down what the skill is doing: 

• We’re defining index-metadata-entry as a KI type/use case.

• We’re performing a hybrid ES|QL search on our AI indices, filtering by the appropriate type using RRF as the default method to fuse results.

• The KI results will directly ground the agent’s answer when determining what indices are relevant to the query.

Because the skill is just instructions plus a query, it travels wherever your agent does. You can point the same file at a Kibana Workflow agent, Claude Code, LangChain Deep Agents, or any other harness without changing a line of it.

Connect your AI Index to an agent harness 

We want to demonstrate how you can use AI indices to query your data with any harness. For these examples, we’ll use LangChain Deep Agents and an OpenAI-compatible key, but any other agent harness can be easily substituted in, including Elastic Agent Builder.

First, let’s create a baseline to see how an agent will perform without using KIs:

Here’s a modified example that could run the same agent, but now with the ability to search AI indices to return KIs: 

This agent will always query the KI indices to get the answer.

How much do Knowledge Indicators reduce agent token usage?

Since we’re using agents, the results of these scripts are non-deterministic. However, when I ran these results against the query Is there scientific evidence that vitamin D supplementation prevents cancer?, both agents led to the same conclusion, but they took different paths to get there: 

Baseline (No AI Index)

With AI Index

Total tool calls

12

8

read_file calls

0

2

get_mapping calls

3

0

esql_query calls

9

6 

Total indices queried

2 (bounced between beir-scifact and beir-nfcorpus)

1 (beir-nfcorpus)

Tokens consumed

167,763

92,711

Latency

39.58s

36.15s

Answer

Grounded, correct

Grounded, correct

The KI answers were both grounded and correct, but an interesting datapoint is the fact that the overall tool usage and token utilization was smaller when using KIs (latency was roughly equivalent). Here’s how both paths went, side by side: 

Run the full AI Index pipeline in Serverless

This walkthrough offered a deep dive into the build-it-yourself version of AI indices and KIs. In production, you wouldn't hand-write these workflows; a setup agent would generate them, and a feedback loop would refine KIs from the agent's own traces. But the primitives are exactly what you just used: extract KIs with a workflow, store them in an AI Index, and retrieve them with a skill.

Managing context is key to a relevant and efficient agentic search system, and AI indices are a way to manage this context with the full power of the Elastic stack. Try it out in Serverless and let us know what you think in our Discuss forums or the #stack-kibana channel in our Community Slack! 

Before adding precomputed context, the Support team at Elastic used an agent for case investigation, root cause analysis (RCA), triage, and related work. To answer a question, the agent had to discover relevant indices, inspect schemas, issue several queries, reconcile conflicting updates, and assemble a response. Questions ranged from narrow state lookups to multisource investigations:

• What is the current status and priority of support case 1234567?

• After client ABC’s cluster migrated certificates, it returned Secure Sockets Layer (SSL) handshake and certificate_unknown errors. What caused the failure, and how should it be fixed? Which knowledge base articles are relevant?

• Client ABC’s cluster stopped processing indexing requests and returned 'rejected execution of primary operation' errors. What was the root cause, and how was service restored?

The next question about the same case or about a related case often required much of the same orientation and synthesis work. That repetition consumed input tokens and added latency. It also increased the chance of invalid queries or wrong source selection.

Earlier work on precomputed context provided a starting point. It described extracting useful context ahead of time and storing it as structured Knowledge Indicators (KIs). Then it explained how an agent retrieves those compact records before scanning raw documents.

For this support use case, we wanted to understand what context was worth preparing in advance and how the choice of context affected response efficiency and reliability. We also wanted to know what retrieval rules were needed when an agent could use several KI types. This article follows that design process and reports the findings.

Starting with index profiles

We began with index profile KIs. A workflow read each support index mapping and sampled a few documents. It generated a compact profile describing the index’s purpose, the questions it could answer, important fields, exclusions, verified joins, time coverage, example Elasticsearch Query Language (ES|QL) queries, and more. A representative KI document with selected fields looked like this:

The profiles helped the agent choose a source and form a query. They reduced orientation work, especially when index or field names were unclear, but they didn’t remove the main cost of a case investigation. After selecting the indices, the agent still had to gather the case record and reconstruct the conversation. They also still had to follow links to engineering and knowledge sources and then reconcile the evidence. Routing was useful, but the repeated work was synthesis.

Changing the unit of context

That observation led to case-level KIs. The workflow materializes one evidence-aware snapshot per eligible case. It gathers the case record, recent material conversation events, linked engineering issues and comments, and linked knowledge articles. The resulting KI document contains a stable case identifier and source references. It also contains tags and freshness metadata.

The design became more detailed as we worked through support specific nuances. Support conversations contain provisional theories, later corrections, administrative status changes, and occasionally separate incidents inside one case. A single fluent summary can flatten those distinctions. The case schema therefore records incident phases, a hypothesis ledger with confirmed, inferred, contradicted, and unresolved states, root cause status and confidence, technical outcome, reusable learning, and more.

Since large language models (LLMs) can hallucinate some of these details, we added deterministic checks after generation. For example:

• An unresolved root cause must be empty and have low confidence. 

• An inferred root cause cannot have high confidence. 

• A case marked closed in a customer relationship management (CRM) system is evaluated separately from whether the technical issue was resolved. 

These checks repair structural contradictions without requiring another model call to reinterpret the evidence. A representative KI document with selected fields looked like this:

Generating KIs only when they’re useful

After a few iterations, we realized that generating a KI for every case adds cost and noise. Many cases have little usable evidence. In some, there’s only a short intake message; in others, there’s no substantive feed or acknowledgement from a support engineer. So we added gates and filters to the workflow. For example:

• A case proceeds when it has linked engineering evidence or at least two material conversational events, including evidence of support participation.

• The workflow compares the stored KI watermark with the newest timestamp across the case and its material feed. An unchanged case reuses its existing KI instead of regenerating it every time.

• The workflow limits corpus selection by focusing generation on recent cases or those otherwise likely to be queried, especially when the source corpus is large.

• The workflow also refreshes the case KI when linked sources can change independently, rather than relying on an incomplete watermark.

Deferred cases remain available through raw lookup and can be reconsidered after new activity arrives. Provenance is part of the stored KI: References point back to linked sources, and the output records what was missing or unverifiable. The KI shortens routine investigation, while raw records remain available for current status, complete history, attachments, and evidence outside the generated snapshot.

The retrieval skill is part of the system

While workflows generate KIs, the agent requires retrieval guidance to use them effectively. To address this, we developed a retrieval skill which provides ES|QL templates for querying KIs stored in the index and guidance on when to use each retrieval method. For instance, a known case number is searched lexically because it’s an exact identifier. Questions about similar cases or precedents use hybrid lexical and semantic retrieval, while questions about official knowledge, engineering issues, comments, or other non-case entities begin with an index profile and then query the selected raw source.

The skill also assigns a clear role to each KI family. An index profile provides routing and field guidance, while a case KI provides a bounded snapshot of the evidence relevant to one case. If no case KI is available, the agent treats that as a cache miss and follows the index profile guidance back to the underlying data, rather than assuming that the available context is complete.

That distinction matters during a typical case lookup. For a question such as What is the current priority of case 1234567?, the agent retrieves the case KI by case _id to understand the investigation and its supporting evidence. It then checks the live case record for values that can change, including status and priority, along with updated_at. If the live record is newer than the KI’s last update, the live value takes precedence. The KI remains useful for durable evidence, while the source record remains the reference for current state. The next eligible refresh rebuilds the KI using timestamps from the case and its material feed, in addition to linked sources.

This separation was shaped by an evaluation failure. In an earlier version, the agent used the case KI to plan retrieval but issued a raw source query using an inferred schema instead of the fields supplied by the index profile. The query failed, causing retries and additional model work. Later versions of the skill made the precedence rules, source roles, field guidance, and recovery steps for mapping errors explicit.

What we observed 

We evaluated the agent using 20 distinct questions, with approximately 10 for each workflow: case investigation/postmortem summarization and shorter technical support Q&A across several source types. We used three trials per question to measure variation from run to run. The examples in this post were tested on an Elastic Cloud Hosted deployment running Elasticsearch 9.4.2.

Different agent configurations used the same answering model and comparable tool conditions. They varied only in whether the agent had access to KIs and, if so, which ones: index-profile KIs, case-level KIs, or both. We measured factuality against expected answers, input and output tokens, latency, and tool execution failures. A separate LLM judge scored factuality.

Case-level KIs produced the clearest signal of operational efficiency in both workflows. Relative to the raw index baseline, observed input token use and latency changed as follows:

Workflow

Input tokens

Latency

Case investigation

About 58% lower

About 40% lower

Technical support Q&A

About 43% lower

About 17% lower

We noticed that index profiles shortened source orientation but left consolidation to the agent, whereas case-level KIs supplied a compact evidence bundle aligned with the requested output. That led to fewer raw queries and reduced the opportunity for schema- and partition-related tool errors.

Factuality varied by workflow, but we didn’t observe a statistically significant reduction in this evaluation.

A note on interpreting these results: The evaluation was performed on a small dataset, which is common and useful early in the agent development lifecycle when large labeled evaluation datasets aren’t yet available. We therefore treat the results as directional signals for comparing variants. This mirrors strategies outlined in engineering blogs by Elastic and other companies, such as Anthropic: Start with a small number of tasks drawn from real failures, and then expand the suite as effects become smaller and the product matures.

Why case-level KIs fit support work

The case-level strategy matched the shape of support investigation work in several ways:

• A case number is a stable retrieval anchor.

• The expensive operation is repeated consolidation across the same source relationships.

• Much of the evidence used for explanation is durable, while volatile states can be verified separately.

• The KI schema mirrors the type of questions an engineer asks during case summary and RCA work.

• Maturity and freshness controls limit generation to cases with enough evidence and likely reuse.

Index profiles still have value. They help with source discovery, schema orientation, non-case questions, and fallback. For case investigation, they leave the cross-source reconstruction inside the response path. Case-level KIs remove part of that recurring work, which is the main reason we expected lower token use and latency in this domain.

The combined strategy exposed a composition problem

While evaluating the responses, we made a counterintuitive observation: Providing both index profiles and case KIs didn’t improve on using case KIs alone. Inspection of the trace showed that the agent understood some of the case context but lacked a dependable rule for composing the two KI families. It used a case-level KI for planning but then ignored the index profile’s schema guidance when querying raw data, leading to failures and suboptimal responses.

This resulted in a practical lesson for improving guidance in the retrieval skill: Context sources need roles and precedence. The agent must know which source can support an answer, which source only routes to evidence, when verification is required, and what to do after a cache miss or mapping error. Two individually useful context types can create additional work when those contracts are implicit.

What we learned about precomputed context

In this support workflow, the most useful unit of context was a case and the evidence connected to it: the case record, conversation, linked engineering work, and relevant knowledge. Preparing that evidence ahead of time meant that the agent didn’t have to reconstruct the same relationships for every response. Compared with the raw index baseline, the case-level approach was associated with lower observed input token use and latency, while factuality didn’t show a statistically significant reduction.

Because case state can change, the system still checks source data for values, such as priority and case status, and falls back to raw data when the available evidence doesn’t justify a case-level KI. Next, we plan to test how well this design holds with support data from external systems, such as Salesforce, and to identify any adjustments needed.

What is an Agent Builder OTel trace and what does it capture?

When your agent runs, Agent Builder records everything that happened as an OpenTelemetry (OTel) trace. Think of a trace as a receipt for a single conversation turn. Every LLM request, tool call, and agent action is recorded as an individual span in Elasticsearch, which is a unit of work or operation. When opted in, additional details like user prompts, LLM responses, tool outputs, and conversation IDs are captured as structured span attributes on the chat span. All of this is scoped to your Kibana space.

How to enable agent tracing and privacy controls in Kibana

To begin capturing trace data, ensure the following toggles under Agent Traces in Gen AI Settings are active within your environment:

• agentBuilder:tracing:enabled — This gen AI setting manages the collection of traces and is enabled by default.

Advanced privacy controls, located under the default tracing toggle, also let you collect message content. While prompts and tool outputs are masked by default, you may choose to enable them to support more robust traces:

• agentBuilder:tracing:includeUserPrompts

• agentBuilder:tracing:includeLlmResponses

• agentBuilder:tracing:includeToolDetails

• agentBuilder:tracing:includeSystemPrompt

• agentBuilder:tracing:includeRealNames: Retains real agent/tool names instead of anonymizing to custom

• agentBuilder:tracing:includeRealIds: Retains the actual conversation identifiers instead of the default hashed versions. This means trace data collects original IDs, which can link traces to specific user sessions (PII).

Only enable these if you understand what data your agents handle and have appropriate data governance in place.

How Agent Builder stores OTel trace data in Elasticsearch

The Agent Builder utilizes OpenTelemetry semantic conventions. This results in a structured hierarchy of spans that provides a granular view of the agent's internal logic:

Span

Type

What it captures

`invoke_agent <name>`

CHAIN

Full turn lifecycle, from user input to final reply

`invoke_agent <name>`

AGENT

Single agent execution: reasoning, tool calls, reply

`chat <model>`

LLM

One LLM request: model, latency, token counts

`execute_tool <toolName>`

TOOL

Tool invocation: arguments, duration, result

Trace data is written to a dedicated data stream per Kibana space, keeping conversation data cleanly isolated. To query your traces in Discover, target the index for your space directly:

For the default space, that’s traces-agent_builder.otel-default. If the advanced privacy controls are turned on, then those span attributes will also be shipped to the traces data stream with the original spans. This index lets you query the raw message content to see what's actually being said in conversations. It is best practice to avoid using wildcards to prevent mixing data from unrelated spaces.

Agent Builder ships with a built-in skill called agent-builder-traces, installed automatically whenagentBuilder:tracing:enabled is on. You can use it to ask questions directly about your trace data, making it easy to explore agent behavior without writing ES|QL from scratch.

How to debug agent behaviour with the OTel trace waterfall view

The trace waterfall shows every step of an Agent Builder session as a timeline. To open it, navigate to the specific turn in the conversation UI and select the trace icon.

This launches a waterfall timeline breaking down every step of your agent's execution. At the top level, you'll see the invoke_agent parent span with the full end-to-end duration of your agent run. Nested beneath it are chat spans, each representing a single LLM request and showing exactly how long the model took to respond. Alongside those are execute_tool spans, one per tool call, where you can see which tool was called, what arguments it received, and how long it ran. This allows you to trace the exact sequence your agent followed, pinpoint timing bottlenecks, and see where errors occurred.

How to build token cost dashboards from trace data

Discover gives you raw trace data, but most teams want answers to operational questions like "how many tokens did we burn today?", "which tool is called most often?", and "how many unique users interacted with the agent this week?". These would require a dashboard built directly against the trace data.

There is an Elastic-managed out-of-the-box dashboard called [Elastic] Agent Builder Overview that can be installed by clicking in the top-right corner of the Agent Traces section within GenAI Settings.

It contains basic details spanning Token Usage and Cost, Conversation Volume and Latency, Agent Execution, and Tool Call Frequency and Errors.

However, a custom dashboard may be more efficient. If you want something more tailored, build Lens panels directly against the OTel trace index. Some useful panels to start with:

1. Most active conversations by token spend

create a horizontal bar chart against traces-agent_builder.otel-<space-id>. Set the x-axis to gen_ai.conversation.id and sort descending and limit to the top 10. Set the y-axis to a sum of gen_ai.usage.input_tokensplusgen_ai.usage.output_tokens. Input the formula as: 

Conversations with the most LLM round-trips

Option to create this visualization with a simple ES|QL query that would look like:

There is also a dashboard-management skill that can be used to help create traces visualizations using natural language.

How to set token cost alerts for Agent Builder conversations

Token consumption is the most direct cost lever for LLM-based agents. A single runaway conversation can blow through your monthly budget before anyone notices.

Elastic alerting lets you define a threshold rule directly against the trace data. Navigate toObservability > Alerts > Manage Rules > Create Ruleand selectElasticsearch queryas the rule type.

A rule that fires when any single conversation exceeds 256,000 tokens looks like this as an ES|QL rule:

Set the schedule to run every 15 minutes and configure the action to send a Slack notification or open a PagerDuty incident. The gen_ai.conversation.id value in the alert payload gives you the exact conversation to inspect.

What’s coming next for Agent Builder observability

Agent traces give you visibility that goes far beyond debugging. Once you've built dashboards and configured alerts against Agent Builder trace data, you have a live pulse on how your agents are behaving in production. If you haven't already, spin up Agent Builder in your Kibana space, make sure tracing is enabled, and run a few conversations. Check Discover, pull up the waterfall view, and see what your agent is actually doing under the hood.

This is the first in a series of posts on Agent Builder observability. Coming up, we'll go deeper on using theagent-builder-traces skill to query your data conversationally, building custom evaluation pipelines from trace data, and using traces to feed conversation history back into your agents. Your agents have been keeping secrets. It's time to make them talk.

Elastic Workflows now writes its own YAML. You type what you want automated in plain language, the Elastic AI Agent generates a complete workflow against a typed schema, and nothing runs until you've read it. YAML is why this works: it gives the model a constrained, well-typed target, so what comes back is actual building blocks you can edit and run.

What is new in Elastic Workflows 9.5:

• Natural language authoring is GA and on by default: describe an automation, and the Elastic AI Agent writes the workflow, you review and run it.

• Versioning with diff and one-click rollback is GA.

• Three new experimental previews (behind an advanced setting): a visual mode that renders a workflow as a graph, human-in-the-loop steps that reach people in Slack for input or approval, and parallel execution.

• More to build on: new connectors, event triggers that react to Cases activity, token metering for AI steps, and a queue strategy for concurrency.

Workflows is the automation engine built into the Elastic platform. It reached general availability in 9.4, enabled by default and running against your Elasticsearch data with the connectors and access controls you already have. This post walks through what 9.5 adds.

Why YAML makes AI workflow automation work

YAML is the authoring language for Elastic Workflows because it's declarative, version-controllable, diffable, and portable across environments. It reads the same in a pull request as it does in the editor.

It was also a bet. Large language models (LLMs) are very good at generating structured, well-typed content, and a workflow language is close to an ideal target for that. Ask for prose and a model can wander. Ask for a workflow against a typed schema, with named step types and validated inputs, and there is a right shape for the answer.

In 9.5 that bet pays off, and it is GA. Inside the workflow editor, you write what you want in plain language:

When a detection alert fires for a host, pull the last 24 hours of related logs, ask the AI step to summarize what happened, and post the summary to the on-call Slack channel.

The Elastic AI Agent generates the workflow: the trigger, the Elasticsearch query, the AI summarize step, the Slack step, wired together with the right inputs and outputs. You get inspectable, editable YAML back. Nothing runs until you read it, adjust it, and decide to run it. You can also point it at a workflow you already have and describe the change you want, and it edits in place.

The output is worth reading because there is a rich, well-typed language underneath it. A short prompt expands into real building blocks:

• foreach and while loops, with guardrails that stop runaway execution.

• switch for clean multi-way branching.

• data steps like data.filter and data.aggregate for in-flight transforms.

• on-failure handling on every step, so you can retry, continue, or abort.

• workflow.execute, so one workflow can call another and you assemble new automation from pieces you have already tested.

Natural language gets you the first draft fast; the language underneath is what makes that draft real.

Workflow versioning with diff and one-click rollback

Versioning is GA in 9.5. Every workflow now has version history: every change is tracked and diffable, and you can roll back to any prior version in one click. You see who changed what and when, compare any two versions side by side, and undo a bad edit without reconstructing it by hand. This is the change-control foundation teams asked for before they would run automation against production systems.

Versioning pairs with the production controls already in place: granular role-based access control (RBAC) over who creates, edits, runs, and views workflows, every management action written to the security audit log, and import/export that moves workflows between environments with their connector references intact.

Versioning in the product is the near end of a longer arc. A workflow is a declarative YAML definition, plain text with a well-defined schema, which means it already fits the tooling built for code: it can be diffed, reviewed, and version-controlled. Where we are headed is full, bidirectional integration with the version control systems you already use, so that a workflow could live in your repository, move through review, and deploy the same way the rest of your software does. That is coming, and the same bet that made natural language authoring work, a declarative and well-typed language, is what will let you manage workflows as code.

Visual mode, human-in-the-loop workflows, and parallel execution

Three of the newest 9.5 additions ship in Experimental. To try them, turn on Elastic Workflows: Experimental Features in Stack Management → Advanced Settings (it requires a page reload). Here is what each one does.

Visual workflow editor: see the logic as a graph

You can now switch a workflow between the YAML editor and a visual mode that renders the workflow as a graph. The graph lays out your steps, branches, and flow control, so you can see the logic and the paths a run can take at a glance, alongside the YAML. It is read-only in 9.5: you still author in YAML, and the graph stays in sync as you edit. It is the first step toward a full drag-and-drop builder, which is coming next.

Human-in-the-loop workflows with approval steps in Slack

Workflows could already pause for a person in 9.4 with waitForInput, which presents a schema-defined form and lets the response drive what happens next. 9.5 adds waitForApproval for the most common case of all, a binary approve or reject with labels you choose, and it extends both steps to their first external surface: Slack. Both pause the run until someone answers, with a timeout so a workflow never hangs forever. Not every decision should be fully automated, and this is how you put the human on exactly the steps that need one.

waitForInput is the flexible one: define a schema for the input you want back, and the response comes through typed. Reach for it when the choice is more than yes or no. Here, an Observability workflow has caught a service-level objective (SLO) burn-rate alert on payment-service and asks the on-call engineer which mitigation to run:

waitForApproval is the new one in 9.5, for a straight approve or reject. Here a security workflow has decided a host should be contained, but gates that destructive action on a human before it runs:

The new piece is the channels block. A wait step can now deliver to where people already are, and in 9.5 that means Slack: the workflow posts the request to a Slack channel, the person responds from Slack, and the workflow resumes with their answer. No one has to be sitting in Kibana for the automation to move. Slack is the first external surface, and richer delivery experiences across more channels are on the way.

Parallel execution: run independent workflow steps at once

By default a workflow runs one step after another, which is what you want when each step depends on the last. But plenty of work does not: enriching an alert from three sources, checking a file against several reputation services, investigating a handful of leads. Run those sequentially and the workflow is only as fast as the sum of its parts, when it could be as fast as the slowest one. The new parallel step lets you run independent work at the same time. It works two ways.

The first is when you know the work ahead of time. You define a fixed set of tasks, and they run at the same time, so you gather all the results in one step instead of waiting for each in turn. Enriching a security alert from two sources at once is the classic case:

The second is when you do not know the work ahead of time. You give the step a list, and it runs the same work once per item, all at the same time, up to a concurrency limit you set. Root cause analysis is a good example. An earlier AI step generates a set of hypotheses for why a service is degrading, and you do not know in advance how many there will be or what they are. Rather than investigate them one after another, you pass the list into a parallel step, and it investigates every hypothesis at once:

You control how many run at once with concurrency, and the engine caps both the concurrency and the total number of parallel tasks so a workflow cannot spawn unbounded work. All the results are available to the next step, so the workflow runs the parallel work, then continues once every task finishes.

More in Elastic Workflows: connectors, triggers, token metering and concurrency

Beyond the headline features, 9.5 widens what a workflow can reach and react to.

New connectors: BigQuery, Snowflake, HubSpot, Cortex XSOAR and more

The connector catalog keeps growing, with native connectors added in 9.5 for:

• BigQuery

• Snowflake

• Box

• Dropbox

• OneDrive

• Outlook

• Azure Blob

• Google Cloud Functions

• HubSpot

• Cortex XSOAR connector for security automation

More are on the way, and when there is not a dedicated connector for the system you need, the http step is the escape hatch: it can securely call any API endpoint, with credentials supplied by a connector rather than written into the YAML.

Event-driven triggers for Elastic Cases

A workflow starts from a trigger, and 9.5 widens what a workflow can respond to. Cases now emit events a workflow can subscribe to:

• A case is created.

• A case is updated.

• Its status changes.

• A comment is added.

• An attachment is added.

So a workflow can run the moment a case opens, to enrich it, tag it, or notify the right channel, or when its status flips to a state you care about, rather than polling for changes. Alert-triggered workflows also receive richer rule context now, including the rule's tags, type, and parameters, so the workflow has more to work with before it acts.

Token usage and cost tracking for AI workflow steps

Workflows can call AI steps: ai.prompt for a freeform prompt, ai.classify to sort something into categories, ai.agent to hand a task to an Agent Builder agent. In an automation that runs thousands of times a day, those calls add up. 9.5 now reports token usage for every AI step, input, output, cached, and total, both per step and for the whole run. You can see exactly what the AI in a workflow consumes, track it over time, and tune a prompt or a model choice with the numbers in front of you.

Workflow concurrency: cancel, drop or queue

A workflow's concurrency setting decides what happens when a new execution starts before the last one finishes. 9.5 adds a third strategy, so you can pick the behavior that fits the workflow:

Strategy

Use it when

Cancel-in-progress

Only the latest execution matters, like recomputing a current state

Drop

An execution already in flight covers the situation and extras are redundant

Queue (new in 9.5)

Every execution matters and order does, so they line up and run one after another, with a queue size and time-to-live you control

Queue is what you reach for when executions touch the same resource or should not overlap. Audit logging also covers more of the lifecycle in 9.5, including restoring a workflow from its version history.

Get started with Elastic Workflows

The fastest way to see this is to describe something you want automated. Open the workflow editor in 9.5, type it in plain language, and read the YAML that comes back. Natural language authoring is on by default. To try the visual mode, human-in-the-loop steps, and parallel execution, turn on Elastic Workflows: Experimental Features in Stack Management → Advanced Settings.

The theme across 9.5 is a shorter path from idea to running automation. You describe what you want and AI drafts it, you see it as a graph and version it as you go, you pause it for a person in Slack when a step needs judgment, and you run independent work in parallel. For the full details, see the Workflows documentation.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

This is where Elasticsearch earns its place in the stack. It is already where a great deal of operational telemetry lives (logs, metrics, traces, events), and it exposes them through an expressive query and aggregation layer. That makes it a natural tool for an agent debugging a live system: it can slice by attribute, aggregate over time, correlate across signals, and drill from a symptom down to the documents that produced it.

We've been building an agentic layer on top of Elasticsearch that continuously monitors a system and root-causes issues as they arise. A recurring primitive in that workflow is time series event analysis. For example, given an error rate, a p99 latency, a queue depth and a throughput counter, tell me whether something happened, what it was and when. A transient spike in errors, a regime change in latency, and a step up in CPU usage are the signatures of the underlying fault, and they're typically what an agent examines first as it forms and tests hypotheses.

Elasticsearch has shipped a single-change-point aggregation for some time. It answers "did this series change?" with one verdict and the most significant change it found. That's a good fit for a dashboard, but less so for an agent, which often wants to interrogate a long window and enumerate everything of interest in it: the error spike at 02:14, the latency regime shift at 02:30, the throughput dip while the pod was being rescheduled. So we upgraded the capability to detect and report multiple events of multiple kinds in a single series. At the same time, we took the opportunity to further harden it to work reliably against whatever telemetry the agent points it at. This post describes how it works.

Why single change point detection isn't enough for agents

Concretely, we want a single entry point that takes a numeric time series and returns a small list of interesting events, each with a type, a location, a significance, and some key characteristics. We care about three classes of event, because they map onto three different kinds of underlying fault:

Event type

What changes

Detection channel

Example fault

Structural change

Level (step) or slope (trend) shifts to a new sustained regime

Value channel

Config push doubles baseline latency; memory leak turns a flat curve into a ramp

Distribution change

Noise level (variance) shifts while the mean holds steady

Dispersion channel

Service responds erratically at the same average latency

Point anomaly

Isolated spike or dip against a stable background

Value channel (pulse detector)

Single burst of errors; one-minute throughput drop during GC pause

The hard part is not detecting any one of these on clean, well-behaved data. The hard part is doing it on arbitrary telemetry without per-series tuning. The agent does not know in advance whether the series it is examining is near-constant, smoothly drifting, heteroscedastic (quiet in places and noisy in others), sparsely populated, or has a magnitude of . It is not scalable to hand-pick parameters for every series it needs to analyze. Whatever we build has to be robust to all of that while maintaining excellent recall and precision. If it fails to detect important events it runs the risk of missing key corroborating evidence for a working hypothesis. Conversely, an analysis tool that reports an event for every minor fluctuation will pollute the context the agent reasons over.

The design goals, in priority order, are: correct on diverse data out of the box, parsimonious (report only what matters), and cheap enough to run interactively across many series.

How PELT and BIC power change point detection

Change-point detection is a well studied field. The classical offline formulation searches for the segmentation of a series that minimizes a penalized cost: a per-segment goodness-of-fit term plus a penalty for each added break to stop the optimizer from putting a boundary between every pair of points. Solved naively, this is combinatorial, but PELT (Pruned Exact Linear Time, Killick et al.) finds the optimal partition in roughly linear time by using a dynamic program to prune candidate boundaries that can never be optimal. On the labeling side, comparing nested models by an information criterion such as the Bayesian Information Criterion (BIC) gives a principled, scale-aware way to decide whether a candidate break is really important and what sort of change it constitutes.

These are good building blocks, and we use them. However, the textbook recipe assumes more than telemetry gives you. It typically assumes a single change type (a mean shift), a known and stationary noise level, and reasonably benign numerics. Real telemetry violates all three: variance changes matter as much as mean changes, the noise level is unknown, often heavy-tailed and changing, and the data spans extreme magnitudes and degenerate cases, such as perfectly constant segments, that wreck an ill-conditioned polynomial fit or a fixed-variance cost. Most of the engineering I describe below is about closing that gap.

Splitting one time series into three detection channels

Rather than trying to find one detector that does everything, we run three focused detectors and then merge their findings. Two of the three are the same structural detector applied to two different views of the data, or channels.

Two detectors run on the value channel: a structural detector that flags step and trend changes, and a pulse detector that identifies point spikes and dips. The dispersion channel, a windowed measure of spread, is fed to a second copy of the structural detector, where a variance change shows up as a level shift and is relabeled a distribution change. The two channels are complementary by construction: a step is a level shift the value channel flags but only has a single large first difference the dispersion channel ignores, while a variance change is invisible to the value channel yet shows up clearly in the dispersion channel. A thin orchestration layer then merges and de-duplicates the streams.

Keeping the three concerns separate makes each one tractable. A mean-shift detector and a variance-shift detector pull in opposite directions if you try to fuse them; a point-anomaly detector and a regime detector need opposite robustness settings. Separated, each can be tuned to its job.

PELT with a scale-free cost

For the structural channel, we fit each candidate segment with a low-order polynomial (constant or linear) and score it with the profiled-variance Gaussian cost. If a segment of length has residual sum of squares , the cost is

This is the negative log-likelihood of a Gaussian segment after profiling out the variance, i.e., after substituting the maximum-likelihood estimate back into the log-likelihood. The total objective PELT minimizes is the sum of segment costs plus a per-break penalty,

Here, is the BIC complexity term (number of parameters times where is the number of values in the time series) and the scale factor lets us trade sensitivity against parsimony in one place.

Using the profiled cost rather than a cost against a fixed global variance is a deliberate and important choice. The global noise level of telemetry is unreliable: on a smoothly varying series the natural estimate (the spread of first differences) can collapse toward zero, and a fixed-variance cost then treats every wiggle as enormously significant and over-segments. The profiled cost depends only on the ratio , so it is invariant to the absolute scale and immune to that failure. A small floor on keeps the logarithm finite, so a zero-residual segment is not rewarded without bound.

Keeping the fit stable using robust local weighting

Before PELT runs, we robustly down-weight points so that an excursion does not create spurious breaks or drag a segment boundary onto itself. Crucially, these weights enter into the weighted residual moments, so they shape not just the segment fit but its residual variance, and so the segment costs themselves. Each point gets a Cauchy weight, , of its residual from a rolling-median baseline against a robust scale : points near the local median keep full weight and points far from it are progressively discounted. A nice side effect of measuring excursions from the median on a window centered on each point is that clean structural breaks do not get down-weighted at all, because the majority of values land on the same side of the break as the point whose residual is being computed. So a sustained regime keeps full weight, but a lone spike does not.

There is a nice justification for scoring a weighted Gaussian cost when what we really want is robustness to a heavy tail. The Cauchy weight is exactly the iteratively reweighted least squares (IRLS) weight of its loss: , so the weighted normal equations are identical to the Cauchy M-estimator's estimating equations . A weighted-mean (or weighted-line) fit at those weights is therefore a Cauchy M-estimate, not a Gaussian one. The cost we actually evaluate inherits the same properties. Because is concave in , its tangent at the current residual lies above it. This gives a pointwise bound with the weight at the tangent point; summing, the weighted residual sum of squares is a tangent upper bound on the total Cauchy loss, touching it in both value and gradient at the weights' anchor point. Minimizing the weighted RSS is thus one step of a majorize–minimize scheme that provably decreases the true Cauchy objective, and the profiled-variance cost we feed the BIC is that majorizer standing in for the Cauchy deviance. The only approximation is that we anchor the weights once, at the rolling-median baseline, rather than iterating IRLS to its fixed point; this is exact for the inliers that sit near the baseline, and correct for gross outliers, whose vanishing weight removes them from the cost wherever the bound is loosest.

Finally, the trick that makes this work on heteroscedastic data is that the residual is judged primarily against a local robust scale, not a global one: the MAD of residuals in the same sliding window. On a series that is quiet in one stretch and noisy in another, this means a spike in the quiet stretch that is multiple local sigmas is correctly suppressed. The local MAD can collapse on quiet stretches, so we use a backstop that is a fraction of a global composite of robust scales and a floor related to the quantization error for discrete series and numerical precision otherwise.

From candidates to labeled events using BIC verification

PELT gives a globally optimal penalized segmentation, so we take its boundaries as candidates and verify each one. For a candidate at index we look at the window of length spanning to its nearest neighboring candidates and compare a no-change null against step and trend alternatives by BIC,

where counts the fitted parameters (the same parameter count as in the PELT penalty above). We map the BIC gain of an alternative over the null to a significance via , to turn a threshold into a decision boundary). We treat this as a significance score for ranking and thresholding, not as a calibrated tail probability.

At this stage we allow higher-degree models to avoid splitting smoothly varying trends. These are problematic in PELT itself because it considers short segments, which they overfit. We keep the most parsimonious alternative that clears the significance threshold and survives a persistence check. The persistence check re-scores with the weights immediately around the candidate muted, and if the evidence collapses, the "change" was driven by a few extreme points – an excursion, not a regime change – and we reject it. The polynomial order is applied symmetrically to the null and to each side of the split, so the alternative is always the same model class merely split at the candidate, and therefore strictly more flexible. The whole process can be thought of as Bayesian model selection with a preference for the null.

When no candidate survives, we still say something useful: we report the series as "stationary" (best no-change model is a constant) or "non-stationary" (best model has a slope), with the trend direction. For an agent, "this series is cleanly trending up over the window" is itself a finding.

Detecting distribution changes with a dispersion channel

A variance change is indirectly visible to the mean channel – worse, the robust weighting there actively mutes the excursions that signal it. So we detect it on a separate dispersion channel and reuse the exact same structural detector, because on this channel a variance change is just an ordinary level change.

The channel is built from one sample per non-overlapping window. Within a window, we take the inter-quartile range of the first differences, rescaled to a standard-deviation equivalent (), and pass it through :

Then . Three choices matter here. First-differencing cancels level and slope, so a mean step contributes a single large difference rather than inflating the whole window, and a steady ramp produces a flat channel. Non-overlapping windows keep the samples independent; overlapping windows autocorrelates the channel and makes the segmenter over-detect. And the IQR is used rather than the median (which is too robust and will miss a window that is 40% noisy then flatlines) or the raw standard deviation (which is not robust enough since one spike's two large differences inflate the window). Because the dispersion channel is a fraction of the original length, the verifier there is restricted to a lower-order null so a genuine low-high-low variance bump is not absorbed.

The functional form is worth dwelling on, because each part earns its place. The log makes the channel respond to ratios of noise level rather than absolute differences. Variance changes in telemetry are typically multiplicative: a regime is "twice as noisy". On a raw-scale channel, a doubling shows up as an enormous absolute jump at a high baseline and a negligible one at a low baseline, so an additive step-cost detector would find variance changes trivially in loud series and miss them in quiet ones. Under a log, a factor- change in scale is the same offset wherever it occurs, which is exactly the additive-step behavior the structural detector is built for. The " is a soft floor. A bare diverges to as the scale goes to zero, which is precisely what happens on a near constant stretch, and would manufacture a huge spurious step at the first noisy window after it. Conversely, is finite and smooth at zero, behaves linearly () while the noise is small, and recovers the multiplicative behavior once the noise is appreciable. This gives graceful degradation instead of a singularity, and with no tuned epsilon to pick. Note that squaring the scale (using a variance instead) only doubles the dynamic range; it makes no difference to the detector either way, since differs only by a constant the threshold absorbs.

Detecting point anomalies as excursions from a local baseline

Spikes and dips are detected as point excursions from the local rolling-median baseline. Working from the local residual rather than raw values means level structure is removed and smooth curvature is tracked; even for time series that change significantly the detector is sensitive to significant local deviations.

The pipeline is a generous proposer followed by a strict gate:

1. Propose every point whose residual exceeds a threshold number of robust sigmas. The scale is the larger of the global first-difference noise (which stays meaningful on smooth data where most residuals are exactly zero) and a composite of robust scales of the residuals (which inflates once a frequent large-residual population appears).

2. Merge adjacent same-sign candidates into excursions, dropping any that span a full minimum segment: that is a regime, and is owned by the structural channels.

3. Rank and cap the excursions by peak z-score, keeping the top , so a pathological series cannot drown the output.

4. Gate using one shared null: build a Gaussian KDE from the series with all of the retained excursions removed, and keep an excursion only if its peak's Bonferroni-corrected upper-/lower-tail probability under that null clears the threshold.

Removing all the tested excursions from the single null at once is a key trick. The leave-one-out alternative – score each excursion against a null containing the others – lets the largest spike and dip mask everything else. Removing them together means several genuinely distinct excursions are each judged against the remainder and all survive, while a recurring population is still rejected.

The proposer and the gate ask deliberately different questions, and that distinction drives two further choices. The proposer works on residuals from the rolling median since it wants recall, and a residual is what tells you a point stands out from its local neighborhood. The gate is value-based: it asks "is this magnitude one we see at other times in the series?", so a spike to a level that recurs elsewhere — such as periodic batch jobs — is suppressed even though it is a large local residual. Those are the right semantics for an agent, but they expose a heteroscedasticity problem, because telemetry noise is almost always a function of magnitude. Periodic spikes can sit orders of magnitude above the background, and a single KDE bandwidth fitted to the whole value range is then far too narrow up in the high tail. So ordinary large values come back as significant, a steady source of false positives.

The fix is a variance-stabilizing transform. We run the value gate in space, where is a robust measure of the spread of the background. is linear for and logarithmic for , which turns a multiplicative (magnitude-dependent) spread into a roughly constant one, so a single bandwidth is valid across orders of magnitude. It is also odd and finite at zero, so exact zeros and sign changes (dips below a small baseline) need no special handling, unlike a bare log. Crucially, it is monotone and so does not change what is tested (for any monotone function , ) so it only fixes the estimate of that tail probability.

One subtlety closes the loop. The KDE null and the kernel bandwidth are taken from different scales, on purpose. The null is the stabilized background values: so it models any mode in the data, which is what makes a recurring large magnitude unsurprising. However, the bandwidth is taken from the stabilized residuals, not the stabilized values, because a genuine level change makes the value distribution bimodal, and a bandwidth computed from that bimodal spread would balloon, masking a real spike sitting on top of a shifted regime. The residual removes the step, so the bandwidth always reflects within-regime noise and the gate stays sensitive to a deviation that is extreme relative to its own neighborhood if it is also outside the envelope for the series as a whole.

Merging structural, distribution, and point anomaly events

Finally, an orchestration layer merges the structural, distribution, and point-anomaly event streams. Structural and distribution events that mark the same regime boundary are de-duplicated to the more significant one (a boundary that shifts both level and spread is one event, not two). Pulses are a separate stream added after de-duplication, because a spike that lands on a structural boundary is a real, separate finding and must not be suppressed. Everything is then mapped back from the internal value-array index space to source-bucket indices.

Handling extreme magnitudes and edge cases

What makes this usable as an unattended tool is a collection of defensive choices for the cases that break naive implementations:

• Variance computed as loses all precision at large magnitudes: a constant series at can manufacture phantom change points purely from floating-point error. We center every PELT input by a constant offset first; the polynomial RSS is invariant to that shift in exact arithmetic, but the working magnitudes drop from to .

• Using raw indices as the regressor results in poor condition polynomial fits: the largest moment is , which is about for a cubic over a 2000-point window. This trips the SVD singularity guard and silently degrades the fit. Mapping affinely onto leaves the fit identical (RSS is invariant under reparametrization) but every moment becomes .

• Scale-free cost, as described, means the segmentation cost doesn't depend on tuning a noise estimate.

• Down-weighting wants a primarily local scale: suppress whatever is anomalous in its own neighbourhood. It uses the maximum of MAD and a small global floor on the differences from the rolling median. Conversely, spike/dip detection wants a global scale: we care about global outliers. It uses a composite of robust scales of all differences. Using the wrong one in either place produces characteristic failures – irrelevant spikes in a quiet segment, or locally large excursions creating spurious breaks – and maintaining separate channels allows us to pick appropriately.

• Using one p-value threshold, Bonferroni-corrected by the number of candidates, applied consistently across all three detectors, means that "how surprised should I be" is consistent everywhere.

The recurring theme is that the difference between a detector that works in a notebook and one that works on a firehose of production time series is mainly in handling the edge cases gracefully.

Performance: ~1ms per series on a single core

Detection cost is dominated by PELT. Its segment cost is a profiled-variance linear fit, which we evaluate in constant time from prefix-summed weighted moments rather than maintaining a regression per candidate boundary, so a single segment cost is a handful of array reads and a 2×2 solve. Cost grows a little faster than linearly with series length since PELT's pruned candidate set does not stay constant on noisy data. Therefore, to bound the worst case on very long series, we downsample ahead of detection: above a cap (2000 samples), the series is collapsed into macro-buckets, keeping two samples per bucket: the median and the largest local deviation. This is inspired by the M4 downsampling scheme, but because we need only the median and the largest excursion for structural-change and outlier detection, respectively, we can then afford to double the bucket resolution. The downsampled series carries its original bucket indices, so every reported event still maps back to a real source bucket; below the cap it is a no-op. The whole analysis is a single pass over the (possibly downsampled) series with no per-series configuration. This lets the agent call it freely across many signals.

In absolute terms, this means the analysis is comfortably interactive. On a single core, post-warmup, a typical series of 140–350 buckets is analysed in about 1 ms (≈220,000 buckets/s), and a series long enough to hit the downsample cap (say 5,000 buckets, collapsed to 2,000) takes about 40 ms, which is the effective worst case per call. For an agent issuing a handful of these calls per investigation, and parallelising across the many series in a `BY` query, the latency is negligible.

Evaluation on synthetic and production telemetry

Synthetic benchmark

We evaluate first on a synthetic generator with known ground truth, because it lets us measure the things that matter precisely. The generator produces random time series with diverse behaviors, which we group into three families. The positive family injects known events of each type: clean and noisy step changes (up and down, SNR around 10, at several positions), trend onsets and ramps (including a flat–ramp–flat sequence with two boundaries), variance changes with a constant mean (single steps and a low–high–low bump), and isolated spikes and dips. The null family ideally produces no event: stationary noise, perfectly constant series, smooth quadratic drift and clean ramps, and periodic signals. (A variance change with a constant mean is not null: it is a distribution change, an abrupt step on the dispersion channel, and so it belongs in the positive family above. The related null requirement, that such a change does not surface on the value channel as a step, is checked separately.) The adversarial family stresses the robustness machinery: a perfectly flat series at a magnitude of (for which a naive variance arithmetic manufactures phantom breaks here through catastrophic cancellation) and, conversely, a genuine spike or step in a noisy baseline as high as (which must still be found and located, the high baseline notwithstanding); a spike on top of a step; a within-regime spike after a 100 level jump; a recurring train of equal peaks (a population, not individual spikes); wide sustained excursions (a structural change, not a spike or dip); and fuzzed random level-shift series. On these series we track recall per event type, precision and the false-positive rate on the null family, localization error, parsimony (events per series and adherence to the count limit), and invariance under constant offsets, rescaling and extreme magnitudes.

The table below summarizes what the suite tests.

Event family

Representative scenarios

Required outcome

Localization tolerance

Step

clean / noisy, up / down, single and multiple, several positions

detected

≤ 4–8 buckets

Trend

slope change and flat–ramp–flat, clean and in noise

detected

≤ 8–12 buckets

Distribution

variance step (mean constant), low–high–low bump

detected

≤ 1 dispersion window

Spike / dip

isolated, multiple distinct, on heavy-tailed and high-magnitude series, within-regime after a step

detected, capped at max(5, 2% of n)

≤ 2 buckets

Null series

stationary noise, constant, smooth drift / ramp, periodic

no event reported

n/a

Invariance / robustness

constant offset, rescaling, 10^5–10^9 magnitudes, spike-on-step, recurring population, wide excursion

result unchanged / no spurious event

n/a

Running this over 400 series per scenario, just over half a million buckets in total, and scoring with the same two categories we use for the real-data evaluation below (any regime change versus point spikes and dips) gives:

Event type

Recall

Precision

Median localization error

Structural change

0.994

0.664

0

Spike / dip

0.847

0.746

0

Two things stand out. When an event is detected, it is placed essentially exactly: the median localization error is zero buckets for both categories; and the point-wise accuracy, 0.998, is directly comparable to the 0.995 we report on real telemetry below: the overwhelming majority of buckets are correctly left unmarked.

The precision figures are lower than on real data, and understandably so. A sixth of this population is a hostile null family (periodic signals, smooth drift, clean ramps) chosen precisely because they tempt a detector into a spurious break, and every false alarm on them counts against precision. The resulting false-positive rate is 0.5% per bucket, with 15% of null series carrying at least one spurious event. We treat this as the conservative end of the range: on the real-telemetry mix below, where the null series are less adversarial, precision rises to 0.85 (structural) and 0.90 (spikes/dips). Finally, offset and scale invariance holds on all 400 series: the same events, to within a few buckets, whether the series is shifted by a constant or rescaled by up to three orders of magnitude.

That raw precision also misses how the result is consumed. The agent reads events most-significant-first, so a false positive only does harm if it outranks a genuine one, and by and large it does not. The median p-value of a true positive is about , against about for a false positive: the genuine events are typically overwhelmingly more significant. Concretely, if we keep only the top events per series by significance (with the true count) precision rises to 0.94, and a randomly chosen true positive is more significant than a randomly chosen false positive 93% of the time. It is not a perfectly clean separation: a strong periodicity or a sharp curve genuinely can contain a significant-looking break, which is why that figure is 0.94 rather than 1. However, the ranking is reliable enough that an agent reading from the top, or applying a stricter significance cut-off, sees the real events first and the false alarms as a lower-significance tail. This is also why exposing the detector's significance to the agent (see What's next) matters more than squeezing the raw precision higher.

Real cloud telemetry

To evaluate on real data, we scraped around 300 metrics from our production cloud environment. These cover HTTP status-code counts, failed memory allocations, memory usage, network usage, page faults, CPU usage, and throttling metrics, measured both per instance and aggregated across the fleet as a whole. Their values range over more than 12 orders of magnitude, and they display a variety of behaviors including ramps, periodicity, step changes, distribution changes, and trend changes. The figure below shows a sample of series together with the detections we make on them.

To get a sense of the accuracy on this set data, we labeled a subset of 150 most interesting time series, marking the visually clearest features in each. This labeling is not necessarily optimized for our target use case, where false negatives are typically more problematic than false positives: an agent consumes these results as part of a broader investigation and can pull additional information to corroborate them. Even so, we find excellent agreement with human judgment on these series. Since each series comprises between 140 and 350 points, the point-wise accuracy, at 0.995, is extremely high: the great majority of points are correctly identified as neither a change, a spike, nor a dip. But the more telling metrics are recall and precision on the human-labeled points, shown in the table below. The human labels did not attempt to categorize each change, so we break the results down only into "structural changes" and "spikes / dips" — the same two categories, and the same point-wise accuracy and recall/precision metrics, as the synthetic benchmark above.

Event type

Recall

Precision

Structural change

0.94

0.89

Spike / dip

0.97

0.92

It is worth covering exactly why we get disagreements. These largely fall into three categories: spikes and dips in context, isolated breaks, and small-magnitude breaks in stable series. We deliberately do not try to detect spikes and dips that are unusual only in their immediate context; that is, not globally unusual but visually significant relative to an inferred periodicity in the data, for example. Trying to account for these without fully modeling the seasonality in the data hurt precision more than it helped recall, and we have a separate persistent anomaly-detection process that builds more complete models of baseline behavior over time. Isolated breaks are an artifact we decided to live with: PELT's cost function tends to isolate a change point with a few values intermediate between the two regimes, because absorbing it into either neighboring span inflates that span's cost. Humans are good at judging such situations visually and assign a single change point. Finally, small-magnitude changes are simply not visually obvious. We detect them deliberately and regard this as a strength of a quantitative approach, since they are often the early precursors of an incident whose later, larger effects drown them out.

How the agent uses change point results in ES|QL

To the agent, all of this is one tool call: it points it at a collection of time series and gets back a typed, located, ranked list of events. That list is small by construction, which means it drops cleanly into the model's context without crowding out everything else it is reasoning about. Because the result contains multiple events, a single call over a window can hand the agent the whole local story – "error spike at 02:14, latency regime change at 02:30, throughput dip at 02:31" – and let it correlate across signals to a root cause.

Operationally, we expose this through both the ES|QL CHANGE_POINT command and the change_point aggregation. Currently, we have not extended the change_point aggregation to return multiple change points since it breaks backwards compatibility of the output schema. It just returns the most significant event. We don't have the same restriction for ES|QL since it returns change points annotated onto the table rows to which they apply. We do plan to revisit the output schema for both ES|QL and the aggregation in a later version. We'd like to migrate to optionally returning significance in log-space, which doesn't underflow, and including a short verbal description of each change, which we expect to help agents when seeing just the change points themselves.

ES|QL is Elasticsearch's piped query language, and CHANGE_POINT runs the detector as one stage in a pipeline. Its BY clause enables it to analyze many series at once (one per group) so the agent can, in a single query, segment every service's latency or every host's error rate side by side rather than issuing a call per series. The actual leverage, compared to the change_point aggregation, is composability: the events come back as ordinary rows in the pipeline, so the agent then has the entire ES|QL language to manipulate them downstream. It can filter to a window, join change points against deploy markers, count events per service, rank by significance, feed the survivors into a further aggregation, and so on.

For example, suppose the agent wants to find out whether any servers have recently seen a sudden CPU spike or a prolonged step change in CPU usage over the last 12 hours, and whether that might point to a load-balancing issue. It could use the following query:

Here it's using a STATS ... BY host.pod to see how the detected events cluster across other dimensions of the data, such as the Kubernetes pod, and so judge whether they share a common cause.

What's next for ES|QL time series analysis

As far as detecting events of interest in time series, the foundation is in place: a single, robust, parsimonious tool that turns a raw telemetry series into the short list of events that actually matter, which is exactly the kind of reach an agentic SRE needs. Going forward, we plan to explore the best mechanism for feeding the detector's uncertainty to the agent, so that a borderline event can be flagged as "worth a second look" rather than silently included or dropped. Also, this is the first of several analytical tools we plan to build into the ES|QL query language to enable agents to triage and RCA issues more effectively; so stay tuned for further updates.

AI shopping agents fail not because the model is wrong but because the agent arrives at every conversation not knowing your catalog, vocabulary, or business rules. It has to discover all of this context through tool calls, and that discovery is the cost. Precomputing a structured context layer from signals you already hold (vocabulary, policies, user profiles, session behavior) cuts the exploratory work the agent does before it can answer and makes its behavior governed and predictable.

In the analogous document-retrieval case, precomputing context reduced input tokens by up to 75% on a controlled benchmark; we expect comparable savings in ecommerce because the exploration pattern is the same, though the exact figure will vary by catalog and query mix. The signals are richer in ecommerce than in almost any other domain, and most retailers already have them. The question is whether or not these signals are assembled in a form that the agent can use before it starts reasoning. Elastic’s broad mix of search capabilities, semantic, hybrid, keyword, filtering, and aggregations make it a compelling choice for not only building your core retrieval tools but also as the vital context engine.

Why AI shopping agents fail in production

Retailers investing in AI shopping assistants are discovering an uncomfortable gap between what the demos promise and what the first few months in production deliver.

The assistant takes four seconds to respond. It confidently recommends a product in a size range that doesn't exist for that item. It tells a returning customer about a coat style they bought 18 months ago and returned. It filters by a category name that doesn't match the internal taxonomy and returns zero results. The customer gives up and abandons the chat altogether.

These aren't model failures. The frontier models powering these agents are capable of extraordinary reasoning when they have the right information in front of them. The problem is that the agent arrives at the conversation knowing nothing about the retailer's catalog, customer, or business rules that govern what should and shouldn't be recommended. It has to learn all of this through the conversation itself, making exploratory tool calls to discover what departments exist, what filter values are valid, and what the brand's policies are on certain product types. Every one of those discovery calls costs latency and tokens before the agent has said a single useful thing to the customer.

Latency matters in ecommerce in a way it doesn't in many other contexts. Shoppers expect response times measured in seconds, not the minutes that enterprise knowledge-base agents routinely take. It’s well established that slower responses reduce engagement and conversion in online retail. An AI shopping assistant that thinks visibly for five seconds before answering a question about gift ideas isn't a feature; it's friction.

The fix isn't a faster model or a bigger context window. The agent's latency and cost problem is a context problem. This can be solved by carefully computing context before the agent call, not during it.

What an AI shopping agent knows before it searches

Imagine you’re the agent. You have a search tool connected to the catalog, and this query arrives:

"an outfit for an autumn wedding"

What do you actually do with that?

Start with what you don't know. An outfit for a man or a woman? Is "autumn" a color, a season, a style of fabric, or just when the wedding happens? And who is this shopper, someone who has bought from you for years, or a stranger? Do they buy expensive designer brands or do they always hunt out a bargain on sale? You have none of these answers. So you do what an agent does when it's working blind: You ask loads of follow-up questions, guess, or fire off a series of exploratory searches to find out what departments and filter values even exist, watching the seconds tick by before you've offered the customer anything at all.

Hold onto that feeling of working blind. The rest of this article is about what changes when the agent is handed the answers first.

Why ecommerce is different from general RAG

Most of the published work on reducing agent costs focuses on document retrieval: question-answering over corpora of articles, reports, or knowledge-base entries. A recent experiment from the Elastic team (Cutting agent costs with pre-computed context) demonstrated that pre-extracting structured facts from documents before the agent call reduced input token consumption by up to 75% and improved answer accuracy from 60% to 92% on a hard factual benchmark. That improvement came in stages, with the largest jump driven by feeding the agent's own wrong answers back into the extraction step rather than by precomputing context alone, which is a distinction we'll return to when we discuss governance.

Ecommerce applies the same principle to a fundamentally different structure. A product catalog isn't a document corpus. It's a highly structured index of items with strict field semantics, a domain-specific vocabulary of brand names, color codes, and category hierarchies, and a layer of business rules that override pure relevance in specific situations.

The failure modes that result are distinct from document retrieval augmented generation (RAG) failures:

• Vocabulary mismatch: A customer asks for a "navy jumper." The agent constructs a filter against a field where the canonical value is NAVY and the category is stored as Knitwear & Jumpers. Without a vocabulary mapping, the agent either guesses colors and categories and gets it wrong, or makes multiple exploratory calls to discover what values exist before it can filter correctly.

• Hallucinated filter values: Without knowing which filter dimensions are valid for a given query, agents can construct queries against fields that don't exist or with values that return zero results. A filter like category: knitwear looks reasonable; masterCategoryNames: "Knitwear & Jumpers" is what the index actually contains. If the agent doesn’t know, it either hallucinates or has to do a separate tool call to find out, causing another LLM loop, which costs time and tokens.

• Context-free personalization: The same query from two different customers, one who typically shops in the premium range and dresses for formal occasions, and one who buys primarily casualwear under £40, should return different results. Without profile context, the agent treats every query identically, which is worse than a well-tuned keyword search because it creates the impression of a personal assistant while delivering generic answers.

The context layer: What signals it needs

The reason ecommerce is particularly well suited to precomputed context is that retailers already hold an unusually rich set of signals. The challenge isn't data availability; it's assembly.

The signals fall into two groups. Two of them, the catalog vocabulary and the business policies, are the genuinely original work and the heart of this approach. The rest, live facet state, user profiles, and session history, are valuable but closer to table stakes, signals that most teams already understand how to fetch. Here's what most mid-to-large retailers have and what each signal prevents.

Signal

What it prevents

Effort to build

Catalog vocabulary

Vocabulary mismatch and hallucinated filter values; the agent guessing at colors, categories, or brand names instead of resolving them to canonical field values

One-time engineering effort (full-catalog aggregation); incremental to maintain as new categories and brands are added

Business policies

Recommendations that ignore legal or trading requirements, for example, missing age verification on alcohol queries or routing that misses a gluten-free range

Human-authored and governed, not automated; ongoing review as new policy types are added

Live facet state

Recommending filters that return zero results or out-of-stock options for the current query

Runs in parallel with vocabulary and policy lookups; leans on existing catalog and retrieval infrastructure

User profile

Making a returning customer restate sizes, budget, or brand preferences they've already given

Fastest signal to retrieve, a single document lookup by user ID

Session and purchase history

Re-recommending an item the customer already dismissed or bought and returned

Most aspirational layer; depends on customer relationship management (CRM) and analytics integration, best added once the agent is already live

The reason we say semantic metadata and not just metadata is that we’re trying to match the semantic (meaning) of the intent rather than the exact words. If a user is searching for “teal,” we should be able to understand that this is a color and which colors exist in our products that are semantically similar to teal, even if none of them are actually teal. So, if we search for “teal,” we might want to return:

ProductColours = “aquamarine, turquoise”

Hopefully, you can see how this semantic metadata is bridging the gap between user intent and the agent’s knowledge of the products.

Catalog vocabulary: The foundation of context engineering

Catalog vocabulary is the layer that does the most work, and it's the one most worth getting right first.

A vocabulary index maps natural language to the exact field values and category paths used in the product index. It answers questions like: What does "navy" map to? Which categories fall under "knitwear"? Is "Autograph" a brand or a range? What's the correct spelling of a competitor brand the agent might encounter in a query?

What makes this more than a synonym list is how it's queried. The interesting thing a shopper says is rarely an exact field value. They say "something cozy for fall," not colour: NAVY and masterCategoryNames: "Knitwear & Jumpers". So the vocabulary index needs to resolve fuzzy, natural language intent into precise, exact-match filters, and that requires both kinds of matching at once: semantic search to understand that "cozy" leans toward knitwear and fleece, and exact keyword matching to pin the result to the canonical values the product index actually stores. A metadata index that supports both on the same documents is, in effect, a translation layer between how customers talk and how the catalog is structured.

This is also where the index earns the description "semantic metadata layer" rather than "lookup table." Each entry is a small natural language description of a facet value or schema concept, so the agent can match against meaning and then read back the exact filter to use. For a typical fashion retailer, this covers hundreds of color values, brand aliases, category synonyms, and size-range conventions. Building the semantic metadata layer from a full-catalog aggregation is a one-time engineering effort; maintaining it is incremental as new categories and brands are added. Without it, an agent encountering an unfamiliar term must either guess or make exploratory tool calls to discover what's there.

Business policies in the context layer

The second original layer is policy. Some queries carry implicit business requirements that pure relevance cannot handle. A query for "wine gift for a friend" should trigger an age-verification reminder in markets where it's legally required. A query for "gluten-free food gift" should route away from general confectionery toward the specific gluten-free range. A query mentioning "wedding guest outfit" in spring should apply different weighting than the same query in November.

These are policies, and most retail search teams already write them. They just call them boost rules, merchandising overlays, or synonym configurations. The difference in an agentic context is that instead of being applied silently as query modifications, they're surfaced as readable hints the agent can use when deciding how to frame its answer and which products to surface. The agent doesn't have to infer your trading rules from the catalog; it's handed them.

The critical point: These policies encode business intent, not just relevance. A policy that routes alcohol queries through an age-appropriate flow isn't a retrieval optimization; it's a trading requirement. That's why this layer must be human-authored and governed, not generated automatically from traffic patterns, a point we return to in the governance section.

Together, vocabulary and policy are what make the agent behave like it understands your business rather than just your data. The remaining three signals sharpen the experience, but they're more familiar engineering.

Facet, profile, and session signals in the context layer

• Live facet state: Before the agent recommends filters, it should know which filters are available and how many results each returns for this specific query. An agent that suggests "filter by size 8" without knowing that size 8 is out of stock for this query undermines the customer's trust immediately. A facet state query against the product index, run in parallel with the vocabulary and policy lookups, returns the counts, ranges, and available values specific to the current query.

• User profile: A persistent profile (sizes, color preferences, budget range, brand affinities) lets a returning customer skip restating what they've already told you. It's typically the fastest signal to retrieve, a single document lookup by user ID.

• Session and purchase history: Within a session, the agent should know what the customer has already seen, dismissed, or added to their basket, so it doesn't re-recommend a dismissed item or repeat itself. Longer-term purchase history extends this, and signals like return history are richer still, but using them well depends on data most retailers hold in systems that aren't yet wired into their search path. This is the most aspirational layer and the one best approached last, once the earlier layers are delivering value. There must also be balance when building the initial context not to overinflate the size, which will slow down the first reply and increase token costs. There is, therefore, a careful balance to strike between providing information like purchase history in the initial context or providing it as a tool for the agent to use during conversation, but users will expect that, if they’re logged in, the agent should know what they’ve purchased. The precise optimal context is likely to be specific to each implementation and customer experience and will require careful testing.

How context engineering works before the LLM call

The pattern that makes this work is simple to describe and moderately involved to implement:

Without this context build, the agent makes the same discoveries on its own, but through LLM-driven tool calls that each cost a full inference round trip. As a rough rule of thumb, an exploratory tool (for example, GetFilterValues) call tends to land somewhere in the region of 600ms to 1 second of latency in practice; an agent that discovers the vocabulary, checks policy hints, and retrieves facet state through three separate tool calls before it even begins answering can, therefore, add two to three seconds to the response time, and that's before it performs the actual product search. These are order-of-magnitude estimates, not benchmarked figures, and the real numbers depend heavily on the model, the network path, and how the tools are implemented.

Replacing those discovery calls with a parallel fetch (multiple context-building queries can be done in parallel) that runs before the LLM is invoked removes most of that cost. The context build takes roughly the same wall time as a single LLM tool call, but it replaces three or four of them. The LLM would either need to repeat failed searches or do its own context-building tool calls sequentially to get enough context to be successful. Precomputing context is also deterministic rather than subject to the model's tool-selection choices, which gives the business more control to fine-tune the experience.

The token reduction follows the same logic: Each exploratory tool call returns raw data the model must process. A preassembled context summary replaces that raw data with structured facts the model can consume in one pass. At the scale of usage possible in public-facing retail websites, this token cost saving can be significant. This work on document search (Cutting agent costs with pre-computed context) showed up to 75% input token reduction on a controlled benchmark. That benchmark was document retrieval rather than ecommerce, and its authors are explicit that the multiplier isn't a fixed number you can expect everywhere. We expect the direction to hold in ecommerce, because the exploration pattern is the same; it just runs against a structured catalog rather than a document corpus. The magnitude is something each team should measure against its own traffic.

The AI shopping agent with full context

Remember the query that left you guessing: an outfit for an autumn wedding. Run it again, but this time, before you have to think, you’re handed a precomputed context as a short brief:

• This shopper’s name is Sarah, female, age 32, and she buys womenswear, size 12.

• She typically buys your mid-tier ranges.

• "Autumn" here matches these specific color labels: “rust”, “burgundy”, “forest green”, “camel”.

• Matching departments: “Womenswear”, “Menswear”.

• Matching tags: "occasion dresses”, “trouser suits”, “wedding”.

• She already has a burgundy bag in her basket.

• House rule for wedding-guest looks: Complete the outfit. Show hats and accessories, not just the dress.

Suddenly, you’re not guessing; you’re styling for this customer. And the interesting part is how those facts combine rather than just stack. The season proposes a whole autumn palette; the burgundy bag already in her basket narrows that palette to the few tones that coordinate with it; the house rule tells you to finish the look with a matching fascinator rather than stopping at the dress. Together, these facts let you answer like someone who knows both this customer and this shop, in a single pass, with nothing invented.

That short brief is exactly what the signal stack produces: the shopper's profile, the resolved vocabulary, the live basket, and the business policy. These are assembled in parallel and placed in front of the agent before its first move, so the "What do I even do with this?" problem never has to be solved one expensive tool call at a time.

Governing the context layer without automated drift

One difference between the approach described here and automated knowledge extraction systems is worth addressing directly: In ecommerce, the context index cannot self-update without human review.

The policies that govern how an agent responds to gift queries, alcohol queries, or queries from customers in certain age brackets aren't just relevance configurations; they're trading decisions with potential legal and brand implications. An automated system that generates new policies from traffic patterns, without review, is a compliance risk before it's a technical asset.

This is actually the right constraint for most retail organizations, and it aligns with how search teams already work. Merchandisers write boost rules. Search teams maintain synonym configurations. Content teams approve what language appears in automated recommendations. The context policy layer is the same kind of governed configuration; it just serves a different consumer, namely, the agent's reasoning step rather than the query pipeline.

It's worth noting where this differs from the document-retrieval work referenced earlier. In that experiment, the biggest accuracy gain came from an automated feedback loop that fed the agent's wrong answers straight back into the extractor. That works well for factual question-answering, where "right" and "wrong" are unambiguous. In ecommerce, the equivalent signals still surface automatically, but a human decides what to do with them, because the changes carry trading and compliance weight. The loop is the same shape; the publication step has a person in it.

The governance loop that works in practice has two tiers:

• Automatic signal surfacing: Zero-result queries, repeated reformulations on the same topic, and sessions that end without a purchase after an agent interaction are all signals that something in the context layer is missing or wrong. These surface automatically as candidates for improving the experience, for example: a vocabulary term that didn't resolve or a policy that didn't fire on a query type it should have covered. To do this, you need a thorough log of conversations, including the reasoning and tool call trace in a platform like Elastic. This allows you to analyze the performance of the agent using both structured tools, for example, percentage increase in thumbs-down conversations and semantically. You could also run an automated review of conversations about "gifts" around December to characterize the thumbs-up/down ratio across an AB test of two agents.

• Human authorship and review: The search or merchandising team reviews candidates and authors the appropriate vocabulary entry or policy. Policies go through approval before publication. This typically mirrors the workflow that already exists for synonym changes or boost rule modifications; the tooling is the only new part.

How to implement context engineering in phases

• Phase 1: Vocabulary layer (highest leverage, bounded engineering task).

• Phase 2: Facet state and initial policies (leans on the same catalog and retrieval primitives).

• Phase 3: User profiles and session signals (requires CRM and analytics integration; best added when the agent is active).

• Phase 4: Governed feedback loop (shifts to organizational alignment; surfaces gaps for merchandising teams).

The full signal stack described above doesn't need to be built at once, and the order isn’t arbitrary. The highest-leverage starting point is also the lowest in implementation complexity: the vocabulary layer. A semantic metadata index built from a full-catalog aggregation (canonical color values, brand aliases, category paths, field names) is a bounded engineering task, and an agent that can resolve "navy jumper" to color: NAVY, masterCategoryNames: "Knitwear & Jumpers" before its first tool call is materially better than one that discovers this through trial and error. If you build nothing else, build this.

Facet state and the first set of policies can follow close behind, often in parallel, because they lean on the same catalog and the same retrieval primitives. The later layers, such as user profiles, session signals, and the governed feedback loop, are where the work shifts from search engineering to organizational alignment. To implement CRM system integration, merchandising workflow changes, and the analytics needed to surface gaps can take a significant amount of work. Those layers are more valuable once the agent is already in regular use and generating the traffic signals that make the governed loop worth running. The important feature is that each layer stands on its own, so a retailer gets real value from phase one without committing to phase six.

What infrastructure does a context layer need?

Precomputing context at the depth described here places specific requirements on the underlying platform. It's worth being explicit about these, because the temptation in early agent builds is to reach for the simplest available tool for each capability.

• Semantic search to match natural language queries against the vocabulary index and surface the right canonical values. Fuzzy keyword matching alone won't resolve ambiguity between similar brand names or color terms.

• A percolator to implement the policy layer. A percolator reverses the usual search direction: Instead of matching a query against stored documents, it stores the queries and matches an incoming piece of text (here, the customer's message) against them. That's exactly what policy matching needs, because each policy is essentially a saved pattern that says "When a query looks like this, surface this hint."

• Real-time aggregations over the full product catalog to produce accurate facet state at query time. Precomputed facet snapshots go stale quickly in active catalogs; query-time aggregations are the more reliable source.

• Document retrieval by key for user profiles: fast, single-document lookups by user ID that must complete within the context build window.

• Structured and semantic logging and analytics over query traces and agent interactions, which are the raw material for the governed loop's automatic signal surfacing.

These are standard capabilities of a mature search and analytics platform, rather than six separate systems, and Elasticsearch provides all of them in one place. That matters less as a procurement point than as an architectural one: When semantic matching, percolation, aggregations, profile lookups, and analytics all run against the same catalog in the same cluster, the context layer stays consistent with the search layer by construction. Splitting these across a separate vector store and a separate analytics platform is a legitimate choice, but it adds operational surface and introduces a consistency problem between two systems that are reasoning about the same products. The context infrastructure is simplest to run when it lives where the product data already lives and can be updated without a separate extract, transform, load (ETL) step.

Conclusion: Context engineering is a search team's job to own

The retailers who run effective AI shopping experiences at scale aren't the ones with the largest models or the most generous token budgets. They're the ones who have done the work to make their catalog, vocabulary, and tpolicies legible to an agent before it starts reasoning.

The good news is that most of the work is already done. The vocabulary is implicit in the catalog. The policies exist as merchandising rules and compliance guidelines. The user profiles are in the CRM. The session signals are in the analytics stream. The gap isn't data; it's the assembly layer that turns those signals into a structured context the agent can consume before it starts reasoning.

The search team already owns the vocabulary, policies, and merchandising workflows. The context layer is the right home for work the search team is already doing, in a form that serves the agent as well as the query pipeline. And, because it grows every time a gap is found and filled, it behaves less like a setup cost and more like an asset that compounds.

To begin building a context layer in Elastic, you can start a trial of Elastic Cloud or run locally. You should become familiar with configuring semantic search, and if you’re interested in how to build, store, and match search policies at query time, you will enjoy this blog.

Every year, natural disasters force emergency managers, military commanders, and public safety officials to make high-stakes decisions in compressed time frames. These decisions traditionally rely on phone trees, spreadsheets, and institutional knowledge spread across dozens of people. The coordination overhead alone costs critical time.

This post demonstrates how Elastic can power a responsive, agentic coordination system for disaster response that detects a threat, reasons through the logistics, and takes action automatically. To make it concrete, we built a simulation: a fictitious Category 4 hurricane threatening the Hampton Roads coastline triggers the automated relocation of over 137,000 personnel across seven military installations.

Disclaimer: This is an entirely fictitious scenario built for demonstration purposes. Hurricane ELARA-26 doesn’t exist. Installation locations are based on real, publicly available geographic data (the U.S. Department of Defense [DoD] Military Installations, Ranges, and Training Areas [MIRTA] dataset), but all operational data, like personnel counts, housing capacity, assets, contact emails, and mission profiles, are completely fabricated. Nothing in this demo reflects actual military readiness, capability, or operational procedures.

Why automated disaster response requires geospatial and agentic coordination

When a natural disaster threatens critical infrastructure, the coordination challenge is immediate:

• Which facilities are in the impact zone?

• How many personnel need to move?

• Where can they go, and do those facilities have capacity?

• Who needs to be notified right now?

These questions don't wait. Neither should the answers.

Deploy the pipeline: prerequisites and setup

Follow the instructions here in the example repo to deploy a local Elastic cluster with Elastic Inference Service (EIS) via Cloud Connect.

How the Elasticsearch agentic disaster response pipeline works

The pipeline has seven layers that work together end to end:

1. Data ingestion: Global Disaster Alert and Coordination System (GDACS) disaster events sent to Elasticsearch

2. Ingest pipeline: GeoJSON is ingested and normalized to Elastic Common Schema (ECS).

3. Geospatial enrichment: The event's affected area polygon is matched against indexed military installation boundaries.

4. Alerting: A Kibana detection rule fires when a disaster intersects any installation.

5. Workflow automation: The alert triggers a Kibana workflow that starts an AI agent conversation.

6. AI reasoning: The agent reasons through affected facilities, their assets, and nearest supporting facilities to determine relocation of all assets and personnel.

7. Email notifications: The agent dispatches emails to all recipients for incoming and outgoing personnel and or assets.

Let's walk through each layer.

Step 1: Indexing military installations with geo boundaries

The foundation is the DoD MIRTA dataset from source.coop/seerai/hifld. This dataset provides a geo_shape of type Point for each installation; centroid coordinates rather than full boundary polygons.

Each installation document in the mitra-facilities index is enriched with operational profile data (all fictitious), beyond what MIRTA provides:

This rich index is what enables the AI agent to make intelligent allocation decisions; not just "here are nearby bases," but "here are bases with available housing capacity, compatible mission types, and the logistics to receive incoming assets."

Step 2: Ingesting and normalizing GDACS events

GDACS publishes real-time GeoJSON for earthquakes, tropical cyclones, floods, wildfires, volcanoes, and droughts. We ingest this feed into a data stream (logs-gdacs.events-*) using a custom ingest pipeline that normalizes the raw GeoJSON to ECS fields.

The GDACS ingest pipeline does several things worth noting:

Geometry extraction: The centroid is stored as a geo_point for map display, and the impact polygon is stored as a geo_shape in gdacs.affected_area, which is the field used for intersection queries later.

Severity normalization: Each disaster type has a different severity scale. A tropical cyclone is measured in km/h wind speed; an earthquake in Richter magnitude. The pipeline maps all of them to a normalized 0–100 score:

The normalized severity score then maps to a severity_level label (low, medium, high, critical) used for alert severity mapping in the detection rule.

ECS alignment: event.kind: alert, event.category: threat, timestamps mapped to event.start/event.end, and a stable fingerprint-based _id for deduplication.

Step 3: Geospatial enrichment: Finding affected facilities at index time

Elasticsearch's geo_match enrich policy matches the disaster polygon against every installation boundary at index time, with no query-time join required. Instead of querying at search time, we use an enrich processor in the ingest pipeline to match the disaster's impact polygon against every installation boundary as the document is indexed.

The enrich policy is a geo_match policy:

The processor runs at the end of the ingest pipeline:

INTERSECTS catches any installation whose boundary touches or overlaps the disaster polygon and even partial intersections. The result is that every GDACS event document is stored with an affected_facilities nested array that tells us exactly which installations are in the impact zone. No join query needed.

Step 4: Detection rule: Alerting on facility impact

A Kibana detection rule watches the logs-gdacs.events-* data stream and fires when a GDACS event has been enriched with at least one affected facility:

The rule runs on an hourly schedule (covering a now-1h to now window) and uses dynamic severity mapping; the gdacs.severity_level field computed by the ingest pipeline drives the alert severity automatically.

Alert severity also drives the risk score via field mapping:

When the rule fires, it passes the full alert context, including the enriched affected_facilities array with installation names, types, and locations, downstream to a Kibana workflow.

Step 5: Workflow automation: Bridging alert to agent

Kibana Workflows handle the handoff from detection to response. The natural disaster response workflow is triggered by the alert:

The entire alert payload (disaster type, severity, affected area, and the list of impacted installations) is forwarded to the AI agent as its initial context. The agent takes it from there.

Step 6: The AI agent: From data to coordinated action

The mitra.response agent takes the full alert payload and, in a single agentic loop, assesses scope, finds receiving facilities, allocates personnel and dispatches evacuation and intake notifications, all without human intervention.

The agent has two tools available:

• mitra.nearest_facilityqueries the mitra-facilities index using a geo_shape query, sorted by distance from a given coordinate, returning up to 50 nearby active installations with available capacity.

• mitra.send_email iterates over a JSON array of facility objects and dispatches formatted evacuation or receiving notifications.

The agent's instruction set defines a clear workflow:

1. Assess the situation. Parse the alert, identify affected facilities, and determine disaster scope.

2. Inventory what needs to move. Personnel counts, critical assets, housing requirements per facility.

3. Find destination facilities. Call mitra.nearest_facility for each affected installation, filtering out facilities still in the danger zone.

4. Make allocation decisions. Reason through single versus multi-facility solutions, branch compatibility, housing capacity, asset support.

5. Send coordination emails. Dispatch evacuation orders to source facilities and intake notifications to receiving facilities.

6. Produce a summary report. Produces a short summary of all affected facilities, total personnel, assets moved, destination facilities, and any concerns to the chat for review.

The agent's allocation logic follows real-world constraints: Don't exceed housing capacity, prefer same-branch relocations when possible, use joint bases for multi-branch overflow, and prioritize distance to minimize transit time.

The nearest facility tool

The underlying workflow query uses geo_shape with a circle filter and _geo_distance sorting:

Available capacity is computed at query time via a script field which calculates housing capacity minus current personnel count. The agent uses this to allocate personnel across destinations without exceeding limits.

Hurricane ELARA-26: agentic coordination of 137,000 personnel, end to end

Hurricane ELARA-26 is a Category 4 storm (213 km/h maximum winds) projected to make landfall in the Hampton Roads area of Virginia. When the GDACS event is ingested, the affected area polygon intersects seven major military installations in the region. The detection rule fires. The workflow kicks off an agent conversation.

Within a single agentic loop, the agent:

• Identified seven facilities in the impact zone, with a combined 137,372 personnel.

• Called mitra.nearest_facility to find receiving facilities outside the storm track.

• Distributed personnel across nine receiving facilities based on available housing capacity and distance.

• Generated and dispatched evacuation orders to all seven affected installations.

• Generated and dispatched intake notifications to all nine receiving facilities.

• Produced a full coordination summary, similar to below:

Facilities evacuated:

Facility

Personnel

Naval Station Norfolk

50,000

Joint Expeditionary Base Little Creek-Fort Story

18,000

Naval Air Station Oceana

15,355

Naval Air Station Oceana Dam Neck Annex

17,509

NG State Military Reservation Camp Pendleton

9,707

Joint Base Langley-Eustis

15,000

Naval Weapons Station Yorktown

11,801

Receiving facilities:

Facility

Distance

Incoming personnel

Fort Gregg-Adams

97 km

~40,000

Marine Corps Base Quantico

148 km

~30,000

Naval Support Facility Indian Head

151 km

~30,000

Joint Base Andrews

180 km

~30,000

Naval Air Station Patuxent River

141 km

~10,000

NG MTA Camp Butner

174 km

~5,000

NG Bethany Beach Training Site

209 km

~4,707

Rivanna Station

140 km

~7,500

Def Gen Supply Center

22 km

~6,000

Assets relocated include transport vehicles, helicopters, patrol boats, medical units, engineering vehicles, generators, water trailers, shelter kits, and communication systems.

Automated email notifications

Once the agent finalized its allocation plan, it called mitra.send_email and dispatched 16 emails in a single pass; that is, evacuation orders to all seven affected installations and intake notifications to all nine receiving facilities. Each message included destination facility, incoming personnel count, assets to move, and a coordination contact. What would have taken hours of phone trees completed automatically the moment the agent finished reasoning.

Extending agentic disaster response with RAG and policy grounding

This demo is purely from structured data, like capacity numbers, distances, and operational status. Elastic's semantic search and retrieval augmented generation (RAG) capabilities can make the agent significantly smarter, with two additions:

Historical response retrieval: Index past after-action reports, Federal Emergency Management Agency (FEMA) incident summaries, and disaster response records as vector embeddings. When a new event fires, the agent can semantically retrieve how similar events were handled, informing allocation decisions with institutional knowledge rather than capacity math alone.

Policy and doctrine grounding: Index DoD emergency management directives, installation continuity of operations plans, and commander guidance. The agent can retrieve and cite the actual policies governing a response, ensuring every decision is grounded in doctrine rather than inference.

Both follow the same Elastic-native approach:; An inference pipeline generates embeddings at index time, and a semantic search tool is exposed to the agent. The coordination pipeline stays the same. The agent just gets smarter.

Why Elasticsearch is the right platform for agentic public sector response

This isn’t a chatbot. It isn’t a dashboard. It's a responsive agentic workflow system, one that detected a threat, reasoned through a complex logistics problem, and coordinated the relocation of 137,000 people without a human in the loop. That kind of outcome is only possible because every capability it depends on lives in a single, unified platform.

Elasticsearch's geospatial support (geo_point, geo_shape, enrichment policies, and distance-based sorting) handles the spatial reasoning that makes intersection detection and facility lookup possible at scale. Semantic search and vector embeddings ground agents in truth, ensuring AI reasoning is based on what's actually in your data rather than hallucinated assumptions. Kibana's detection engine, Workflows, Agent Builder, and Agent Builder tools wire it all together into a pipeline that goes from raw event to coordinated action with no external glue code required.

No other platform brings this together the way Elastic does. The combination of real-time indexing, geospatial precision, semantic retrieval, and agentic orchestration, all in one stack, with enterprise-grade security and observability built in, is what separates Elastic from tools that do one of these things well but require you to stitch the rest together yourself.

Agentic geospatial response for emergency management, fire, law enforcement and public health

The same architecture applies wherever people, facilities, and real-time events intersect. The specific data changes. The pipeline doesn't.

Emergency management: FEMA and state offices of emergency management can map shelter locations, staging areas, and vulnerable populations against incoming National Weather Service (NWS) severe weather polygons, triggering automated resource pre-positioning before a storm makes landfall.

Fire and emergency medical services: Fire departments can overlay unit locations and response zones against wildfire perimeters or structure fire clusters, automatically routing mutual aid requests to the nearest available units with the right equipment.

Law enforcement: Agencies can correlate active incident locations with school zones, critical infrastructure, and officer positions, triggering geo-aware lockdown notifications or resource dispatch without waiting for manual triage.

Public school safety: School districts can monitor real-time threat feeds against campus boundaries. When a threat intersects a school's perimeter, an agent can immediately notify administration, initiate lockdown communications, and coordinate law enforcement response, all before a dispatcher picks up a phone.

Public health: Health departments can match disease surveillance data or environmental hazard zones against clinic locations, population density layers, and supply depot inventories to route resources where they're needed most.

Sector

Use case

Elastic capability

Emergency management

Match shelter locations against NWS severe weather polygons

geo_shape enrichment + Kibana Workflows

Fire and EMS

Overlay unit locations against wildfire perimeters

geospatial routing + nearest-facility query

Law enforcement

Correlate incidents with school zones and officer positions

geo-aware alert rules + agent dispatch

Public school safety

Monitor threat feeds against campus perimeters

detection rules + automated notification

Public health

Match hazard zones against clinic locations and supply depots

semantic search + geospatial enrichment

The data is different in every scenario. The underlying pattern of ingest, enrich at index time, detect intersection, trigger agentic response, and act is all the same. Elastic gives public sector organizations the platform to build it once and adapt it everywhere.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this article, we’ll explore the benefits of building a custom Elasticsearch MCP server and show how to create one in TypeScript that connects Elasticsearch to LLM-powered applications.

Why build a custom Elasticsearch MCP server?

Elastic provides some alternatives for MCP servers:

• Elastic Agent Builder MCP server for Elasticsearch 9.2+

• Elasticsearch MCP server for older versions (Python)

If you need more control over how your MCP server interacts with Elasticsearch, building your own custom server gives you the flexibility to tailor it exactly to your needs. For example, Agent Builder's MCP endpoint is limited to Elasticsearch Query Language (ES|QL) queries, while a custom server allows you to use the full Query DSL. You also gain control over how results are formatted before being passed to the LLM and can integrate additional processing steps, like the OpenAI-powered summarization we'll implement in this tutorial.

By the end of this article, you’ll have an MCP server in TypeScript that searches for information stored in an Elasticsearch index, summarizes it, and provides citations. We'll use Elasticsearch for retrieval, OpenAI's gpt-4o-mini model to summarize and generate citations, and Claude Desktop as the MCP client and UI to take in user queries and give responses. The end result is an internal knowledge assistant that helps engineers discover and synthesize best practices across their organization’s technical docs.

Prerequisites:

• Node.js 20 +

• Elasticsearch

• OpenAI API key

• Claude Desktop

What is MCP?

MCP is an open standard, created by Anthropic, that provides secure, bidirectional connections between LLMs and external systems, like Elasticsearch. You can read more about the current state of MCP in this article.

The MCP landscape is evolving every day, with servers available for a wide range of use cases. On top of that, it’s easy to build your own custom MCP server, as we’ll show in this article.

MCP clients

There’s a long list of available MCP clients, each with its own characteristics and limitations. For simplicity and popularity, we’ll use Claude Desktop as our MCP client. It will serve as the chat interface where users can ask questions in natural language, and it will automatically invoke the tools exposed by our MCP server to search documents and generate summaries.

Creating an Elasticsearch MCP server

Using the TypeScript SDK, we can easily create a server that understands how to query our Elasticsearch data based on a user query input.

Here are the steps in this article to integrate the Elasticsearch MCP server with the Claude Desktop client:

1. Configure MCP server for Elasticsearch.

2. Load the MCP server into Claude Desktop.

3. Test it out.

Configure MCP server for Elasticsearch

To begin, let's initialize a node application:

This will create a package.json file, and with it, we can start installing the necessary dependencies for this application.

• @elastic/elasticsearch will give us access to the Elasticsearch Node.js library.

• @modelcontextprotocol/sdk provides the core tools to create and manage an MCP server, register tools, and handle communication with MCP clients.

• openai allows interaction with OpenAI models to generate summaries or natural language responses.

• zodhelps define and validate structured schemas for input and output data in each tool.

ts-node, @types/node, and typescript will be used during development to type the code and compile the scripts.

Set up the dataset

To provide the data that Claude Desktop can query using our MCP server, we’ll use a mock internal knowledge base dataset. Here’s what a document from this dataset will look like:

To ingest the data, we prepared a script that creates an index in Elasticsearch and loads the dataset into it. You can find it here.

MCP server

Create a file named index.ts and add the following code to import the dependencies and handle environment variables:

Also, let’s initialize the clients to handle the Elasticsearch and OpenAI calls:

To make our implementation more robust and ensure structured input and output, we'll define schemas using zod. This allows us to validate data at runtime, catch errors early, and make the tool responses easier to process programmatically:

Learn more about structured outputs here.

Now let’s initialize the MCP server:

Defining the MCP tools

With everything configured, we can start writing the tools that will be exposed by our MCP server. This server exposes two tools:

• search_docs: Searches for documents in Elasticsearch using full-text search.

• summarize_and_cite: Summarizes and synthesizes information from previously retrieved documents to answer a user question. This tool also adds citations referencing the source documents.

Together, these tools form a simple “retrieve-then-summarize” workflow, where one tool fetches relevant documents and the other uses those documents to generate a summarized, cited response.

Tool response format

Each tool can accept arbitrary input parameters, but it must respond with the following structure:

• Content: This is the response of the tool in an unstructured format. This field is usually used to return text, images, audio, links, or embeddings. For this application, it will be used to return formatted text with the information generated by the tools.

• structuredContent: This is an optional return used to provide the results of each tool in a structured format. This is useful for programmatic purposes. Although it isn't used in this MCP server, it can be useful if you want to develop other tools or process the results programmatically.

With that structure in mind, let’s dive into each tool in detail.

Search_docs tool

This tool performs a full-text search in the Elasticsearch index to retrieve the most relevant documents based on the user query. It highlights key matches and provides a quick overview with relevance scores.

We configure fuzziness: “AUTO” to have a variable typo tolerance based on the length of the token that’s being analyzed. We also set title^2 to increase the score of the documents where the match happens on the title field.

summarize_and_cite tool

This tool generates a summary based on documents retrieved in the previous search. It uses OpenAI’s gpt-4o-mini model to synthesize the most relevant information to answer the user’s question, providing responses derived directly from the search results. In addition to the summary, it also returns citation metadata for the source documents used.

Finally, we need to start the server using stdio. This means the MCP client will communicate with our server by reading and writing to its standard input and output streams. stdio is the simplest transport option and works well for local MCP servers launched as subprocesses by the client. Add the following code at the end of the file:

Now compile the project using the following command:

This will create a dist folder, and inside it, an index.js file.

Load the MCP server into Claude Desktop

Follow this guide to configure the MCP server with Claude Desktop. In the Claude configuration file, we need to set the following values:

The args value should point to the compiled file in the dist folder. You also need to set the environment variables in the configuration file with the exact same names defined in the code.

Test it out

Before executing each tool, click on Search and Tools to make sure that the tools are enabled. Here you can also enable or disable each one:

Finally, let’s test the MCP server from the Claude Desktop chat and start asking questions:

For the question “Search for documents about authentication methods and role-based access control”, the search_docs tool is executed and returns the following results:

The response is, “Great! I found 5 relevant documents about authentication methods and role-based access control. Here's what was found:”

The tool call returns the source documents as part of its response payload, which are later used to generate citations.

It’s also possible to chain multiple tools in a single interaction. In this case, Claude Desktop analyzes the user's question and determines that it needs to first call search_docs to retrieve relevant documents and to then pass those results to summarize_and_cite to generate the final answer, all without requiring separate prompts from the user:

In this case, for the query “What are the main recommendations to improve authentication and access control across our systems? Include references.”, we obtained the following results:

As in the previous step, we can see the response from each tool for this question:

Note: If a submenu appears asking whether you approve the use of each tool, select Always allow or Allow once.

Conclusion

MCP servers represent a significant step toward standardizing LLM tools for both local and remote applications. Though full compatibility is still in the works, we’re moving fast in that direction.

In this article, we learned how to build a custom MCP server in TypeScript that connects Elasticsearch to LLM-powered applications. Our server exposes two tools: search_docs for retrieving relevant documents using Query DSL; and summarize_and_cite for generating summaries with citations via OpenAI models and Claude Desktop as client UI.

The future of compatibility between different client and server providers looks promising. Next steps include adding more functionalities and flexibility to your agent. There’s a practical article on how you can parameterize your queries using search templates to gain precision and flexibility.

Prerequisites

• Elasticsearch 9.3 or Elastic Cloud Serverless: You can create a cloud deployment following these instructions, or you can use the start-local quickstart instead.

• Python 3.12: Download Python here.

• Hugging Face access token.

Chat completions using a Hugging Face inference endpoint

First, we’ll build a practical example that connects Elasticsearch to a Hugging Face inference endpoint to generate AI-powered recommendations from a collection of blog posts. For the app knowledge base, we’ll use a dataset of company blog articles, which contains valuable but often hard-to-navigate information.

With this endpoint, semantic search retrieves the most relevant articles for a given query, and a Hugging Face LLM generates short, contextual recommendations based on those results.

Let’s take a look at a high-level overview of the information flow we’re going to build:

In this article, we’ll test SmolLM3-3B capacity tocombine its compact size with strong multilingual reasoning and tool-calling capabilities. Based on a search query, we’ll send all the matching content (in English and Spanish) to the LLM to generate a list of recommended articles with a custom-made description based on the search query and results.

Here’s what the UI of an article site with an AI recommendations generation system could look like.

You can find the full implementation of this application in the linked notebook.

Configuring Elasticsearch inference endpoints

To use the Elasticsearch Hugging Face inference endpoint, we need two important elements: a Hugging Face API key and a running Hugging Face endpoint URL. It should look like this:

The Hugging Face inference endpoint in Elasticsearch supports different task types: text_embedding, completion, chat_completion, and rerank. In this blog post, we use chat_completion because we need the model to generate conversational recommendations based on the search results and a system prompt.This endpoint allows us to perform chat completions directly from Elasticsearch in a simple way using the Elasticsearch API:

This will serve as the core of the application, receiving the prompt and the search results that will pass through the model. With the theory covered, let’s start implementing the application.

Setting up ​​inference endpoint on Hugging Face

To deploy the Hugging Face model, we’re going to use Hugging Face one-click deployments, an easy and fast service for deploying model endpoints. Keep in mind that this is a paid service, and using it may incur additional costs. This step will create the model instance that will be used to generate the recommendations of the articles.

You can pick a model from the one-click catalog:

Let’s pick the SmolLM3-3B model:

From here, grab the Hugging Face endpoint URL:

As mentioned in the Elasticsearch Hugging Face inference endpoints documentation, text generation requires a model that’s compatible with the OpenAI API. For that reason, we need to append the /v1/chat/completions subpath to the Hugging Face endpoint URL. The final result will look like this:

With this in place, we can start coding in a Python notebook.

Generating Hugging Face API key

Create a Hugging Face account, and obtain an API token by following these instructions. You can choose between three token types: fine-grained (recommended for production, as it provides access only to specific resources); read (for read-only access); or write (for read and write access). For this tutorial, a read token is sufficient, since we only need to call the inference endpoint. Save this key for the next step.

Setting up Elasticsearch inference endpoint

First, let’s declare an Elasticsearch Python client:

Next, let’s create an Elasticsearch inference endpoint that uses the Hugging Face model. This endpoint will allow us to generate responses based on the blog posts and the prompt passed to the model.

Dataset

The dataset contains the blog posts that will be queried, representing a multilingual content set used throughout the workflow:

Elasticsearch mappings

With the dataset defined, we need to create a data schema that properly fits the blog post structure. The following index mappings will be used to store the data in Elasticsearch:

Here, we can see more clearly how the data is structured. We’ll use semantic search to retrieve results based on natural language, along with the copy_to property to copy the field contents into the semantic_text field. Additionally, the title field contains two subfields: the original subfield stores the title in either English or Spanish, depending on the original language of the article; and the translated_title subfield is present only for Spanish articles and contains the English translation of the original title.

Ingesting data

The following code snippet ingests the blog posts dataset into Elasticsearch using the bulk API:

Now that we have the articles ingested into Elasticsearch, we need to create a function capable of searching against the semantic_text field:

We also need a function that calls the inference endpoint. In this case, we’ll call the endpoint using the chat_completiontask type to get streaming responses:

Now we can write a function that calls the semantic search function, along with the chat_completions inference endpoint and the recommendations endpoint, to generate the data that will be allocated in the cards:

Finally, we need to extract the information and format it to be printed:

Let’s test this by asking a question about the security blog posts:

Here we can see the cards in the console generated by the workflow:

You can see the full results, including all hits and the LLM response, in this file.

We’re asking for articles related to: “Security and vulnerabilities.” This question is used as the search query against the documents stored in Elasticsearch. The retrieved results are then passed to the model, which generates recommendations based on their content. As we can see, the model did a great job generating engaging short text that can motivate the reader to click on it.

Conclusion

This example shows how Elasticsearch and Hugging Face can be combined to create a fast and efficient centralized system for AI applications. This approach reduces manual effort and provides flexibility, thanks to Hugging Face’s extensive model catalog. Using SmolLM3-3B, in particular, shows how compact, multilingual models can still deliver meaningful reasoning and content generation when paired with semantic search. Together, these tools offer a scalable and effective foundation for building intelligent content analysis and multilingual applications.

What real problem does it solve for engineers?

OpenClaw is a self-hosted gateway for AI agents: a single runtime that coordinates execution, treats agents as isolated processes, and uses skills (structured instructions in markdown files) as the unit of integration. Conceptually, this isn’t entirely different from what we already do with command line interfaces (CLIs) and scripts, but it’s now formalized around agent-driven workflows.

This led to a practical exploration within the Elastic Stack:

If we treat OpenClaw as an orchestration runtime, how does it behave when Elasticsearch is the back end? And how straightforward is integration using OpenClaw skills?

Let's build an integration using composable skills.

Solution architecture

In this tutorial, we’ll teach OpenClaw how to access and query Elasticsearch data through a custom read-only skill, and we’ll then demonstrate how it composes multiple skills together; for example, combining Elasticsearch queries with real-time weather data to generate dynamic reports.

Before diving into the hands-on steps, let’s look at what we’re building. The solution is composed of three integrated layers that work together through OpenClaw orchestration.

Layer 1: Storage and search (Elasticsearch)

The data layer runs on Elasticsearch via start-local, a single command that spins up Elasticsearch and Kibana locally with Docker.

Two sample indices demonstrate different use cases:

• fresh_produce: 10 products with semantic search (ecommerce scenario)

• app-logs-synthetic: 30 log entries across four services (observability scenario)

The same read-only skill works with both indices without any reconfiguration; the agent inspects the mapping and adapts its queries accordingly.

Layer 2: Orchestration (OpenClaw Gateway)

The gateway receives natural language requests and loads the Elasticsearch skill, and the large language model (LLM) decides which queries to construct. The skill is a pure SKILL.md with reference docs, meaning that its operations require no custom code.

To understand how the gateway organizes this, two core OpenClaw concepts are worth knowing:

• Agents: Independent AI instances, each with its own configuration, workspace, and set of skills. You can run multiple agents for different purposes.

• Workspace: A folder that defines an agent’s context:AGENTS.md (the agent’s permanent briefing), .env(credentials), and a skills/ directory. Think of it as the agent’s working environment.

Layer 3: Skills (composable capabilities)

Skills are structured instructions in markdown files (SKILL.md) that teach the agent how to use specific tools or APIs. They can be global (available to all agents), workspace-specific, or bundled with OpenClaw. The agent selectively loads only the skills relevant to each request.

This tutorial uses two skills:

• Elasticsearch-openclaw (custom, built for this tutorial): A read-only skill that teaches the agent how to search, filter, aggregate, and explore Elasticsearch indices using curl.

• Weather (community skill, used for composition demo): A skill that fetches current weather conditions from external APIs.

Later in the tutorial, we'll demonstrate how OpenClaw composes both skills in a single request, querying Elasticsearch products based on real-time weather data without any custom integration code.

Read-only by design

The elasticsearch-openclaw skill is read-only by design. It provides patterns for searching, filtering, and aggregating data, but it never writes, updates, or deletes. This minimizes the security footprint when giving AI agents access to your Elasticsearch cluster.

Even if the agent environment is compromised, your data remains safe from modification or deletion. This is enforced through:

• Skill design: No write operation patterns in SKILL.md or reference files.

• API key permissions: The tutorial uses a read-only API key with only read and view_index_metadata privileges.

• Agent instructions: AGENTS.md explicitly states "You can SEARCH, FILTER, and AGGREGATE data, but you can NEVER write, update, or delete."

This security-first approach is why infrastructure setup (index creation, data loading) must be done manually; by design, the agent cannot do it for you.

Prerequisites

To follow this tutorial, you’ll need:

Software and tools:

• Docker Desktop installed and running (Docker Engine with Compose V2).

• Elasticsearch running locally via start-local. (We’ll set this up in the next section.)

• Jina API key (free): https://jina.ai/embeddings.

• OpenClaw installed: https://openclaw.ai.

Setting up the environment

Start by cloning the starter project, which contains the skill, workspace configuration, and Dev Tools scripts:

The repository contains:

Note: The devtools*.md files contain Kibana Dev Tools commands formatted as reference documentation.

Installing OpenClaw

OpenClaw is a self-hosted gateway. This means you maintain full control over execution and data, but you need to prepare your local environment or server.

I installed OpenClaw on a separate machine, which is why I included the disclaimer below.

** Security and responsibility disclaimer **

Since OpenClaw is an early-stage, rapidly evolving open-source project, the community has raised important discussions about potential security vulnerabilities, especially around token handling and third-party script execution.

Deployment recommendations:

• Isolated environments: If you’re not an advanced infrastructure security user, we recommend installing OpenClaw strictly in isolated, controlled environments (such as a dedicated virtual machine [VM], a rootless Docker container, or a test machine).

• Do not use in production: Avoid running the gateway on servers containing sensitive data or with unrestricted access to your corporate network until the project reaches a more stable, audited version.

• Least privilege: We reinforce the need to use Elasticsearch API keys with restricted permissions (read-only) to mitigate risks, in case the environment is compromised.

• Network segmentation: Both Elasticsearch and OpenClaw bind to localhost by default. Keep it that way, unless you have a specific reason to expose them.

• Credential rotation: Rotate API keys periodically. OpenClaw stores credentials locally, so treat the machine’s security as the perimeter.

• Audit logging: Enable Elasticsearch audit logging to track all API calls made by OpenClaw. This creates a full trail of what the agent accessed and when.

• Keep the installation up to date.

For a deeper analysis of the security architecture and deployment options, consult the official OpenClaw documentation.

Runtime installation

OpenClaw manages daemons and skill isolation via CLI. Since it’s a recent project that has undergone naming changes, we recommend strictly following the official documentation to ensure installation compatibility.

Preparing the Elasticsearch back end

Before connecting any agent runtime, we need a working Elasticsearch environment with data to query and a secure, read-only access layer. In the next two sections, we’ll spin up Elasticsearch locally using start-local, create an index with semantic_text and Jina v5 embeddings, load sample data, validate that semantic search works, and generate a read-only API key. Once this foundation is in place, the Elasticsearch side is complete and we can focus entirely on teaching the agent how to use it.

Part 1: Setting up Elasticsearch locally

Start a local Elasticsearch and Kibana instance with a single command:

Once complete: Elasticsearch at http://localhost:9200, Kibana at http://localhost:5601, and credentials in elastic-start-local/.env.

Part 2: Configuring the index in Kibana Dev Tools

Open http://localhost:5601 → Dev Tools and run devtools_fresh_produce.md in order.

• Step 1: Replace YOUR_JINA_API_KEY with your actual Jina API key (free).

• Step 2: Save the encoded field immediately; it cannot be retrieved later.

The key commands in the Dev Tools file are:

Create the Jina inference endpoint:

Create the index with semantic_text:

The semantic_text field type handles embedding generation automatically at index time.

Index sample products using the bulk API (see devtools_fresh_produce.md for the full dataset of 10 products).

Validate semantic search:

The semantic query type handles inference on the query side automatically; no need to specify model IDs or embedding details.

Create a read-only API key:

Save the encoded value from the response. This is your API key for the OpenClaw configuration.

Connecting to OpenClaw

With the Elasticsearch back end ready, we can now wire it into OpenClaw. Several Elasticsearch integrations already exist in the ecosystem, from Elastic’s own Model Context Protocol (MCP) server to community-built MCP servers. However, most of these offer full CRUD access or are designed for different agent runtimes. Given that the technology is still in its early stages and security remains a primary concern, I chose to build a dedicated skill, simple, read-only, and purpose-built for OpenClaw. This approach ensures that the agent can search, filter, and aggregate data but never modify it, keeping the blast radius minimal even if the environment is compromised.

In the next sections, we’ll configure credentials, install the skill, create a dedicated agent, and explore how the workspace ties everything together.

Install the skill and create the agent

Step 1: Configure credentials

From the cloned repository, configure the credentials by copying the environment template and filling in your Elasticsearch URL and the read-only API key:

Edit the .env file with these two values:

Example .env file:

Step 2: Install the skill from ClawHub

ClawHub is OpenClaw's public skill registry. Think of it as npm for AI agent skills. At the time of this writing, ClawHub hosts over 3,200 skills, covering everything from Slack and GitHub integrations to Internet of Things (IoT) device automation. For this tutorial, we created elasticsearch-openclaw, a custom skill focused on read-only queries using semantic_text, aggregations, and observability on Elasticsearch 9.x. It’s published on ClawHub so you can install it directly. As a best practice, only install skills from trusted sources with known provenance; as with any package manager, review the content before granting access to your agent.

The elasticsearch-openclaw skill is published on ClawHub.

Recommended: Open the OpenClaw Web UI (http://127.0.0.1:18789/) and ask:

OpenClaw will:

• Fetch the skill from ClawHub.

• Install it in the appropriate directory.

• Confirm when ready to use.

Step 3: Create the agent

Do this by registering a dedicated agent with its own workspace, and then restart the gateway to load the new configuration:

Understanding the workspace

Now that the agent is running, let’s look at what makes it tick.

AGENTS.md

The AGENTS.md file is the agent’s permanent briefing. It defines who the agent is, what it can do, and how it should behave. For our Elasticsearch agent, this file instructs the agent about the available indices, the read-only constraint, and the preferred query patterns.

Skills: When they make a difference

Without skill

With `elasticsearch-openclaw` skill

Agent has no knowledge of Elasticsearch query syntax.

Agent knows semantic, full-text, filtered, and aggregation patterns.

Agent might attempt write operations.

Agent is instructed to never write, update, or delete.

Agent guesses field names and types.

Agent inspects mappings first and then constructs appropriate queries.

Generic curl commands with trial and error.

Structured query templates with best practices for Elasticsearch 9.x.

Exploring with the agent

With the Elasticsearch back end configured and the OpenClaw agent connected, it’s time to see what the agent can actually do. In the next sections, we’ll test natural language queries, explore observability data, and compose multiple skills together.

Testing in OpenClaw

Open the OpenClaw web UI, and try some natural language queries. The agent will inspect the index mapping, choose the appropriate query type, and return results.

Type:

Result:

Others ideas to explore:

• Index exploration: > “What indices do I have in Elasticsearch? Show me the fields of fresh_produce.”

• Filtered search: > “Show me all products on sale under $15.”

• Aggregations: > “What’s the average price by category?”

Observability

To demonstrate that the skill works beyond a single use case, the repository includes a second index: app-logs-synthetic, with 30 synthetic log entries across four fictional services, created from devtools_app_logs_synthetic.md.

Setting up the log data

Since the skill is read-only, you need to populate the index first. The devtools_app_logs_synthetic.md file contains five commands (three for setup and two for verification):

• Create ingest pipeline: Adds @timestamp to log entries automatically.

• Create index mapping: Defines the app-logs-synthetic structure (classic fields only, no semantic_text).

• Bulk insert logs: Loads 30 synthetic log entries across four services.

• Count query: Verify 30 documents were indexed.

• Sample search: Quick test to confirm that data is queryable.

How to run:

1. Open Kibana Dev Tools: http://localhost:5601 → Dev Tools.

2. Copy each numbered block from the .md file.

3. Paste into the Dev Tools console.

4. Press Ctrl/Cmd+Enter to execute.

5. Wait for a successful response before continuing to the next block.

This creates the app-logs-synthetic index with sample data ready for querying.

Try this query in the OpenClaw web UI:

Result:

Other ideas to explore:

• “How many 500 errors do I have in app-logs-synthetic? Which services are failing?”

• “Which endpoints have the slowest response times?”

• “What happened with the payment-service in the last 24 hours?”

This is the same skill, same agent, same setup, just pointed at different data. The agent inspects the new index mapping, adapts its queries, and returns relevant results without any reconfiguration.

Composing skills in action

This is where composable skills truly shine. Start by asking the agent:

OpenClaw will search for the weather skill, automatically attempt the installation, and guide you through the process. Just follow the on-screen instructions; no new API key is required for the weather skill. Afterward, try this:

In a single request, the agent chains multiple skills: the weather skill to check current conditions, the Elasticsearch skill to run a hybrid search on products that match the context, and its built-in file and browser tools to generate an HTML report and open it. No custom integration code, no glue scripts, just skills composed by the LLM at runtime.

This is what makes OpenClaw different from a traditional automation framework. You don’t preprogram the workflow. You describe the outcome, and the agent figures out the composition.

Conclusion

SearchClaw started as a simple experiment and ended up demonstrating what composable, LLM-driven integration looks like in practice. The key takeaway is not the individual tools (all are familiar) but the approach. Instead of writing a specific application with hardcoded queries, we gave the agent capabilities and let it compose solutions dynamically. This is what makes OpenClaw native: composable, LLM-driven, and local-first.

As with any early-stage project, OpenClaw should be used thoughtfully, especially regarding security and environment isolation. The read-only skill approach demonstrated here is one way to limit risk while still unlocking the value of your Elasticsearch data.

The full code is available in the repository and can serve as a starting point for your own integrations: https://github.com/salgado/elasticsearch-openclaw-start-blog.

If agents can call tools on demand and models can reason over massive context windows, why not just fetch data live from every system and let the LLM figure it out?

It’s a reasonable question. It’s also the wrong conclusion.

The reality is that MCP and agent tooling don’t eliminate the need for search. They make the quality of search more critical than ever. In this blog, we’ll explore why MCP, federated search, and large context windows don’t replace search engines and why indexes remain the foundational layer for scalable, accurate, enterprise-grade AI.

What MCP actually is (and what it is not)

MCP is a coordination protocol. It standardizes how an agent requests information or actions from external systems.

What MCP doesn’t do:

• Rank results across systems.

• Understand relevance across heterogeneous data.

• Normalize schemas or metadata.

• Data transformations or enrichments at scale.

• Apply consistent security and permissions.

• Optimize for latency, cost, or scale.

In other words, MCP tells agents how to ask for data, not which data matters most.

Modern retrieval requires query intelligence, not just data access

In modern enterprise search architectures, retrieval quality is determined long before a query reaches an index. Raw queries — especially those generated by agents — may be incomplete, overly literal, schema-driven rather than intent-driven, and at times syntactically invalid.

This is why mature search platforms introduce a query intelligence layer that performs query rewriting, entity normalization, synonym expansion, and intent disambiguation before retrieval even begins.

For example, an agent-generated request such as: “Show severity 2 authentication failures from last sprint” may be rewritten to include authentication synonyms (login, SSO, OAuth), normalized severity mappings, and sprint-to-date-range translation. The result is not just more matches — it is more relevant matches.

In enterprise AI, retrieval is not a single step. It is a controlled pipeline.

This distinction is crucial because once MCP-based agents start pulling information live from multiple tools, they recreate a familiar pattern under a new name: federated search.

MCP-based retrieval is federated search in disguise

Federated search isn’t new. Enterprises have tried it for decades.

The model is simple:

• Send the user’s query to multiple systems in parallel (SharePoint, GitHub, Jira, customer relationship management [CRM]).

• Collect the responses.

• Merge and present the results.

MCP-driven tool calls follow the same pattern, except that the caller is now an agent instead of a user interface.

And the same problems resurface.

Why federated search breaks down at enterprise scale

• Latency becomes unpredictable: A federated query is only as fast as its slowest system. Enterprise systems can have wildly different response times and rate limits, so federated queries tend to be slow and jittery. Agents must wait for multiple round trips before reasoning can even begin. The result is a laggy experience and unpredictable wait times.

• Relevance is fragmented: Because each system ranks results on its own, there’s no unified relevance model. Federated search cannot apply a single ranking or semantic understanding across all content, so results often seem disjointed or incomplete. Agents may retrieve correct information but not the most useful information.

• Context is shallow and incomplete: Federated systems typically expose only what’s directly accessible through an API call.They rarely surface:

  • Usage signals, like clicks, dwell time, recency of access, popularity, or authority.

  • Relationships between documents across different systems to correlate the insights.

  • Organizational knowledge beyond a single silo. This strips agents of the broader context required for high-quality reasoning.

• Limited filtering and features: In a federated setup, you can only filter on fields that every system supports (the “lowest common denominator”). If one system doesn’t support a particular filter or facet, you lose that functionality entirely. This severely limits rich search features, like date ranges, categories, or tags.

The power of an indexed search

Search engines achieve millisecond-level retrieval at massive scale by using specialized data structures, including inverted indexes for lexical search and k‑dimensional trees (k-d trees) for vector-based retrieval. The approach is to crawl or ingest every source into search engines, creating a central place of company knowledge. This brings big advantages:

• Speed by design: Searching an index is lightning fast. Queries hit inverted indexes and specialized data structures, avoiding the need to poll each backend system.

• Relevance that compounds over time: Search engines that support semantic search are capable of comprehending the intent, and machine learning models can rerank results for enterprise contexts. In one Elastic experiment, Elastic users see more accurate results when combining vector search with a question-answering (QA) model to extract answers. It gives better precision than keyword matching.

• Advanced features: Elastic’s Graph retrieval augmented generation (RAG) solution shows how structuring an index as a knowledge graph can power more contextual retrieval. In other words, indexes aren’t just backward-looking dumps of text; they can also encode relationships and ontologies that let AI connect the dots across documents.

• Permission-aware search: Enterprise AI cannot compromise on security. Indexed search allows:

  • Document-level security.

  • Role-based access control.

  • Permission-aware retrieval for RAG and agents.

Agents see only what users are allowed to see, without leaking data into model prompts or training. Elasticsearch is suitable for the indexed search layer in the diagram below, as it provides the essential components for context engineering.

Retrieval consistency through search templates and governed execution

At scale, retrieval must be predictable, secure, and repeatable. This is where search templates become critical.

Search templates act as retrieval contracts between applications, agents, and the search platform. Instead of dynamically constructing queries at runtime, agents invoke pre-defined retrieval patterns that enforce:

• Consistent relevance logic

• Mandatory security filters

• Cost and latency guardrails

• Business-specific ranking rules

• Explicit index and field scope boundaries

In MCP-driven architectures, this becomes even more important. Agents should not dynamically invent retrieval strategies. Instead, MCP tool calls can map directly to approved search templates, ensuring that every retrieval request adheres to enterprise relevance and governance standards.

This approach shifts retrieval from ad-hoc query execution to controlled retrieval orchestration.

Retrieval is now a multi-layer engineering discipline

Modern enterprise retrieval is no longer a simple query-to-index operation. It typically includes multiple coordinated layers:

• Query understanding — rewriting, expansion, entity resolution

• Retrieval strategy selection — hybrid search, vector search, graph retrieval, or synthetic query techniques such as Hypothetical Document Embeddings (HyDE), where the system generates a representative answer or expanded context first and retrieves documents using that richer semantic signal.

• Execution governance — templates, security enforcement, and performance guardrails

• Ranking and re-ranking — blending lexical precision, semantic similarity, and interaction-derived relevance signals such as click-through patterns, dwell time, and document usage frequency.

When these layers are implemented upstream, agents receive clean, high-confidence context rather than raw, fragmented data.

This is what makes large-scale agent systems reliable in production environments.

Advanced retrieval techniques improve context quality before reasoning begins

Modern retrieval systems increasingly use AI-assisted techniques to improve recall and semantic coverage before ranking is applied.

One example is Hypothetical Document Embeddings (HyDE). Instead of embedding only the original query, the system first generates a hypothetical answer or expanded context, embeds that representation, and retrieves documents based on that richer semantic signal.

This is particularly useful in enterprise environments where:

• Users or agents may not know the exact terminology

• Knowledge is distributed across silos

• Important context is implied rather than explicitly stated

Techniques like HyDE improve the probability that relevant documents are retrieved even when the original query is underspecified.

This reinforces a key principle of enterprise AI: better context retrieval produces better reasoning outcomes.

Agents aren’t data engineers; they’re reasoning systems

They shouldn’t be responsible for stitching together raw data, reconciling schemas, or compensating for poor retrieval.

This is where a search platform such as Elasticsearch becomes foundational.

By ingesting data once and normalizing it upstream (through pipelines, mappings, enrichment processors, and prebuilt indexes), Elasticsearch resolves schema mismatches, joins signals across sources, and materializes retrieval-ready views of the data. At query time, the agent receives clean, ranked, semantically enriched results rather than fragmented raw records.

For example, instead of an agent pulling independently from CRM, ticketing, and documentation systems and attempting to reconcile customer IDs, timestamps, and formats in real time, Elasticsearch can pre-index these sources into a unified customer interaction index with hybrid (keyword + vector) search and relevance ranking. The agent then queries a single, coherent interface and immediately reasons over the most relevant context.

This separation of concerns, that is, Elasticsearch handling data integration and retrieval, and agents focusing on reasoning, planning, and decision-making,is what makes agent systems scalable, reliable, and production ready.

Elastic’s role in the AI stack

Elastic sits at the intersection of search and AI by design.

• Connectors and crawlers ingest data continuously from enterprise systems.

• Semantic and vector search enable intent-based retrieval.

• Hybrid search blends lexical precision with semantic understanding.

• RAG workflows ground LLMs in authoritative, permission-aware data.

Elastic does not compete with agents or MCP. It makes them effective.

Bigger models don’t eliminate retrieval

Some have wondered whether huge new LLMs can bypass traditional search, perhaps by letting the model read everything in one go. Large context windows feel powerful, but they introduce:

• Higher latency.

• Higher cost.

• Lower precision due to noise.

• A higher propensity for confusion, context clash, and context poisoning.

RAG wins because it filters first and then reasons.In another Elastic Search Labs experiment, RAG achieved answers in about 1 second, versus 45 seconds for the raw-LM approach, at 1/1250th the cost, and with far higher accuracy. In other words, giving an LLM a million tokens of documents is slower, more expensive, and actually less precise than filtering through an index first.

Conclusion: MCP changes the interface, not the fundamentals

MCP is a meaningful step forward in how agents interact with tools. But it doesn’t replace the need for fast, relevant, governed retrieval.

In enterprise AI:

• Context quality determines answer quality.

• Indexes create that context.

• Search is the foundation, not the legacy.

Indexes aren’t obsolete in the era of MCP. They’re the reason that MCP-based agents can work at all.

What are subagents?

Subagents are specialized assistants that can be called to execute a specific task, using their own context window. They complete a task and give the results to the main agent, preventing it from saving information that isn’t relevant for the rest of the conversation in the context window.

Their four core principles are:

• Context preservation: Each subagent uses its own context window.

• Specialized expertise: Each subagent is designed for a specific task.

• Reusability: You can reuse a subagent in different sessions and projects.

• Flexible access: You can limit the subagent access to specific tools.

Each subagent can have access to Claude Code tools to work with the terminal, such as glob, read, write, grep, or bash, or to access the internet, like search, fetch, or call external tools with Model Context Protocol (MCP) servers.

A subagent uses the following schema:

You can call subagents implicitly by talking about the task they run, and Claude will call them automatically. For example, you can say, "I want to plan my new functionality."

You can also call them explicitly by directly asking Claude Code to use a subagent and telling it, "Use the planning subagent to plan my new functionality."

Another important feature is that subagents are stateful, so once you give one a task, it will generate an ID. This way, when you use it again, you can start from scratch or provide the ID to give it context from its previous tasks.

You can read the full documentation here.

When are subagents used?

Subagents are useful when you need to delegate tasks that require specialized context but you don't want to clutter the main chat window. Considering our example of coding, the most common subtasks include:

Subtask type

Description

Typical tools

Exploration / research

Searching and analyzing code without modifying it.

Read, grep, glob

Planning

Running deep analysis to create implementation plans.

Read, grep, glob, bash

Code review

Reviewing quality, safety, and best practices.

Read, grep, glob, bash

Code modification

Writing and editing code.

Read, edit, write, grep, glob

Testing / debugging

Running tests and analyzing issues.

Bash, read, grep, edit

Retrieval

Getting information from external sources (APIs, databases).

MCP tools, bash

Claude Code includes three built-in agents that showcase these use cases:

• Explore: Quick agents for read-only search in the codebase. It's great for answering questions like, "Where are the client's errors handled?"

• Plan: Research agent that activates in plan mode to analyze the codebase before proposing changes.

• General-purpose: The most capable agent for complex tasks that require multiple steps and can include modifications.

Context management: Ensuring subagents have the right information

One of the most important decisions when designing subagents is how to handle context. There are three key considerations:

1. Which context the subagent should get

The prompt you give to the subagent must contain all of the necessary information to complete the task since the subagent doesn’t have access to the main chat. You need to be specific:

• Do NOT say, "Review the code."

• SAY, "Review the changes to src/auth/index.ts, focusing on JWT token validation."

Providing the exact file name makes a difference between using the read tool against the file directly and making a wide search using grep and thus wasting time and tokens.

Also consider what not to include. Irrelevant context can distract the subagent or bias results. It’s tempting to ask for multiple things in one pass, but focused tasks yield better results:

• Do NOT say, “Review src/auth/index.ts. Here is also the database schema and our API docs for reference, fix bugs and suggest improvements about the architecture decisions.”

• SAY, “Fix the token refresh bug in src/auth/index.ts that's throwing AUTH_TOKEN_EXPIRED unexpectedly.”

2. What tools to provide

Limit the tools to what’s strictly necessary. This improves security, keeps the subagent focused, and reduces unnecessary tool calls and execution costs.

If you don't specify a tools field, the subagent inherits all tools from the main agent, including MCP tools.

You can learn all Claude Code tools here.

3. How to keep context between calls

Subagents can be resumed using their agentId:

You can ask Claude for the agent ID or find it in ~/.claude/projects/{project}/{sessionId}/subagents/

This is especially useful for long research tasks or multistep workflows.

Another way to keep context consistent is to ask the agent to write a Markdown checklist with what it's doing and its current progress. Then you can execute /clear without losing the initial instruction. In that request, you can define the task granularity or details to retain that make sense for your use case.

After you clear the conversation, the next agent can pick it up from here. This is very useful when you want an agent to run a script over a list and watch the output record by record.

Orchestration patterns

It’s important to see subagents as a context optimization mechanism. The way in which you coordinate them determines the efficiency of the whole system. There are different orchestration patterns.

Sequential (chaining)

Here, a subagent completes a task, and its results feed the next one in a sequence of tasks, similar to traditional Linux piping.

Call example:

Parallel

In this pattern, multiple subagents run independent tasks simultaneously. The main Claude Code agent invokes them since subagents cannot spawn other subagents.

This approach reduces the execution time for tasks like code review since it allows you to work with the same code from different angles without impacting the running time.

Hub-and-spoke (delegation)

In this approach, the main agent acts as an orchestrator, delegates tasks to specialized agents, and then consolidates the results.

This is the pattern we’ll implement in our example: The main Claude Code agent will delegate the gathering of business information to a retrieval agent built with Elastic Agent Builder, while the explore agent will look into local files and the planning agent builds a plan.

Why use an agent instead of a single query?

Before building our retrieval subagent, it's worth understanding when an agent adds value versus when a simple Elasticsearch Query Language (ES|QL) query suffices.

If you need a single aggregation, like "What's our most visited page?" just run the query directly. The agent adds value when your question requires:

• Multiple queries that build on each other: The answer from query 1 informs query 2.

• Cross-index reasoning: Correlating data from different sources.

• Ambiguity resolution: The agent interprets and follows leads.

• Synthesis: Combining quantitative data with qualitative knowledge.

Our example will demonstrate all of these capabilities.

Agent Builder as subagent

Generating code using AI is very quick, but the problem is having a good planning phase to set the boundaries for our coding agent. To help with that, Claude created a subagent that specializes in planning to perform deep analysis and create a to-do list for the main agent to execute.

With this flow, you can plan based on what Claude Code can see both in local files and on the internet. However, there's still knowledge available in Elasticsearch that you cannot access via standard tools.

To access our internal knowledge during the planning phase, we'll create a Claude Code subagent by making a retrieval agent using Agent Builder.

You can configure the agent using the UI or an API. In this example, we'll use the latter.

Prerequisites

• Claude Code 2.0.76+

• Elasticsearch 9.2

• Elasticsearch API key

The scenario: Technical debt sprint planning

You're a tech lead. You have two weeks and two developers. Your TECH_DEBT.md lists 12 items. You can tackle maybe three or four. Which ones should you prioritize?

The complexity is that you need to optimize across multiple dimensions simultaneously:

• User impact: How many users hit this issue?

• Business impact: Does it affect paying customers? Enterprise tier?

• Severity: Errors? Performance? Just ugly code?

• Effort: Quick win or rabbit hole?

• Dependencies: Does fixing A unlock fixing B?

• Strategic alignment: Does it align with Q1 priorities?

A single query like, "What's the most important tech debt item?" fails because this requires:

1. Reading TECH_DEBT.md to understand what the 12 items even are.

2. For EACH item, querying error_logsto get error frequency.

3. Cross-referencing with customer_data to see tier breakdown.

4. Checking support_ticketsto see complaint volume.

5. Reading engineering_standards in the knowledge base to see whether any items violate core principles.

6. Reading Q1_roadmap to check strategic alignment.

7. Synthesizing all of this into a prioritized recommendation.

This is where a retrieval agent can be helpful in orchestrating multiple queries across different indices and synthesizing the results.

Steps

Preparing the test dataset

We'll create four indices: a knowledge base with internal documentation, error logs, support tickets, and customer data.

You can create the indices, index the data, and create the agent using one of the following:

• Kibana Dev Tools: Using the Elasticsearch requests provided below.

• Jupyter Notebook: Using the complete notebook written for this article.

Create the indices

Open Kibana Dev Tools, and run the following requests to create each index with its mapping and bulk data. Here's an example of the knowledge index structure and data to be indexed:

Full requests for all indices:

• Knowledge index: knowledge.txt

• Error logs index: error_logs.txt

• Support tickets index: support_tickets.txt

• Customer data index: customer_data.txt

The raw JSON files with the dataset are also available:

• knowledge.json

• error_logs.json

• support_tickets.json

• customer_data.json

Local project files

Create the following Markdown (MD) files in your project. These files look like this:

Full files:

• TECH_DEBT.md: Tech debt items list.

• REQUIREMENTS.md: FlowDesk Q1 2025 requirements.

This ties directly to the tech debt items and gives the agent clear priorities to work with when cross-referencing with the Elasticsearch data.

Create an agent with Agent Builder

We'll now create an agent capable of running analytics queries with ES|QL to provide us with app usage information while also capable of searching to provide us info from Knowledge Base (KB) in unstructured text format.

We're using the built-in tools since they cover search and analytics on any index. Agent Builder also supports custom tools for more specialized operations, like scoping an index or adding ES|QL dynamic parameters, but that's beyond our scope here.

You can create the agent using the curl request in create_agent.txt.

You’ll get this response if everything went OK:

The agent will be available in Kibana, so you can now chat with it if you want:

Configure the agent as Claude Code tool

The agent we just created will expose an MCP server. Let's add the MCP server to Claude Code using the already-generated API key:

We can check the connection status using claude mcp get agentbuilder.

Create a subagent that uses the tool

Now that we have the Agent Builder available as a set of MCP tools, we can create a subagent in Claude Code that will use all or some of those tools, in combination with Claude Code ones.

Claude Code recommends using its agent creator tool for this step:

1. Type /agents in Claude Code.

2. Choose Create new agent.

3. Select Project scope so that it's only available for this project. (This is the recommended setting to avoid agent overflow.)

4. Select Generate with Claude (recommended).

5. Type in the description: "Agent that analyzes technical debt by querying Elasticsearch for error logs, support tickets, customer data, and engineering knowledge base. Use this agent when you need to prioritize tech debt items based on business impact."

6. In “Select tools,” choose Advanced options and select the tools we defined on the agent creation.

7. Select [ Continue ].

Now choose the model. For planning tasks, the recommendation is to use Opus due to its significant reasoning capacity. So let's select that and continue.

Finally, choose the background color for our subagent text and confirm.

Claude automatically names our subagent based on the description (for example, tech-debt-analyzer).

Testing the agent

Once the agent has been created, we can test it with a complex prioritization question that requires multistep reasoning:

Watch how the agent orchestrates multiple queries:

And will give you a comprehensive analysis of the local files combined with Elasticsearch data:

This demonstrates why a single query fails and an agent succeeds: It orchestrates five or more queries across different indices, correlates the data, and synthesizes a recommendation that contradicts the naive "fix highest error count" approach.

By typing /context, we can see how much context each of the MCP tool's definitions uses and our subagent's prompt. Keep an eye on this overhead when creating subagents.

Start planning

We can now start planning using local files, the internet, and our Elasticsearch knowledge as information sources.

Ask something like:

Note that Claude decides to run the Elasticsearch data analysis and the local documentation reading in parallel, following the hub-and-spoke orchestration pattern.

After the analysis, you should get a plan that prioritizes based on actual business data rather than on assumptions. This context will make your AI coding experience much more reliable, as you can feed this plan directly to the agent and execute step by step:

The more details you provide and the more focused the instructions are, the better the quality of the plan will be. If you have an existing codebase, it will suggest the code changes.

Conclusion

Subagents are a great tool to offload specific tasks where we only need the final result for the main chat (without going through how we got there), keeping the chat flow focused.

By choosing the right orchestration pattern (sequential, parallel, or hub-and-spoke) and handling the context properly, we can build efficient and maintainable agent systems.

Elastic Agent Builder and its MCP feature allow us to access our data using a retrieval subagent to facilitate planning and coding by combining local (files, source code), external (internet), and internal (Elasticsearch) sources. The key insight is that agents add value not for simple queries but when you need multistep reasoning that builds on previous results and synthesizes information from multiple sources.

Resources

• Claude Code Subagents

• Elastic Agent Builder

• Agent Builder MCP

At a recent DevFest, we demonstrated how to bridge the gap between natural language requests and structured IT workflows. By combining Elastic Agent Builder with Elastic Workflows, we can create AI assistants that not only answer questions but also perform complex actions.

In this post, we’ll dive into the architecture from that talk, specifically looking at how we built an automated "Laptop Refresh" workflow. We’ll demonstrate how to configure an agent that collects user requirements and triggers a server-side automation to interact directly with ServiceNow APIs.

Watch the full breakdown: This post is based on our presentation at Google DevFest. You can watch the full session here to see the demo in action.

The architecture: From chat to fulfillment

Note: The technical implementation described in this document is a streamlined version of the full production environment. While the architecture diagram provided serves as an accurate structural reference for the actual deployment, the accompanying text and code snippets have been simplified for illustrative purposes and may differ from the final, complex configurations used in the live implementation.

The goal is to move from a manual, form-heavy process to a conversational interface. Instead of a user navigating a catalog, they simply tell the AI assistant that they’re due for a laptop upgrade.

As illustrated above, the flow consists of three distinct layers:

1. Interaction layer (ElasticGPT/Agent Builder): The user interacts naturally with an interface powered by ElasticGPT. Behind the scenes, Agent Builder processes this conversation, handling intent detection and slot filling, to structure the data and orchestrate interactions with other internal systems.

• Intent detection

  • Mechanism: System prompt instruction.

  • Implementation: The agent is explicitly told its single purpose in the MISSION statement. It doesn’t need to "detect" other intents because it’s scoped strictly to IT provisioning.

    • Code reference: MISSION: You are a specialized agent designed to collect complete employee onboarding information...

  • Constraint: If a user asks about non-IT topics (for example, "What is the weather?"), the MISSION implies that the agent should pivot back to data collection or decline, depending on the large language model’s (LLM's) default safety alignment.

• Slot filling (data collection)

  • Mechanism: Phased conversation flow.

  • Implementation: Instead of asking for all slots at once, the DATA COLLECTION STRATEGY breaks the slots into five logical phases. This prevents the context switching fatigue mentioned above.

    • Code reference:PHASE 1: Personal information, PHASE 2: Employment Details, and so on.

  • Validation: The prompt enforces immediate validation (for example, Validate inputs immediately), acting as a gatekeeper before moving to the next slot.

2. Automation layer ( Workflows): Once the agent has the data, it triggers a workflow. This workflow handles the logic: checking device eligibility, enforcing policy (for example, "Is the laptop > 3 years old?"), and making API calls.

3. System of record (ServiceNow): The workflow reads and writes directly to the ITSM tool to maintain audit trails and initiate fulfillment.

Step 1: Configuring the agent

The first step is defining the "brain" of the operation using Agent Builder. We need an agent that acts strictly within the bounds of IT provisioning. We don't want a general chatbot; we want a data collection machine that feels like a helpful colleague.

We achieve this via a robust system prompt. The prompt dictates the agent's operating protocol, enforcing a step-by-step data collection strategy.

Here’s the refined structure of the prompt we used. Notice how it enforces validation and logically groups questions to avoid overwhelming the user:

For a sample system prompt or instructions, please refer here.

By explicitly instructing the agent to send the data in a specific JSON format at the end of the conversation, we ensure that the input matches exactly what our automation layer expects.

Step 2: The automation layer (Workflows)

The agent provides the intent and the data, but Workflows provides the muscle.

We define a workflow using a YAML configuration. This workflow acts as the bridge between the AI agent and the ServiceNow REST APIs. It handles authentication, data retrieval, and the ordering process.

Below is the workflow definition. We’ve refined the code to use secure variable handling for credentials rather than hardcoding them.

Workflow inputs

First, we define the inputs the workflow expects to receive from the agent:

Interacting with ServiceNow

The workflow executes a series of HTTP steps. Crucially, we first need to identify the user's current asset to link the refresh request correctly.

1. Fetching computer data

We query the cmdb_ci_computer table in ServiceNow to find the asset currently assigned to the user.

2. Adding to cart

Once we have the asset details and the user's preferences, we don't just create a generic ticket. We use the ServiceNow Service Catalog API to programmatically add the specific item to a cart.

3. Indexing the transaction

Finally, we want to keep a record of this transaction within Elasticsearch for analytics and future reference. We use the elasticsearch.index step to store the request details immediately after submission.

For detailed workflow yaml, please refer here.

The result

By stitching these components together, we create a seamless experience:

1. The user chats naturally with the agent to provide details.

2. The agent structures this unstructured conversation into a JSON object.

3. Workflow receives the JSON, validates the user's current hardware via ServiceNow, creates the order, and indexes the result.

This approach reduces a process that traditionally took users 5–10 minutes of form navigation into a quick conversation, while ensuring that IT operations retains full visibility and control.

Video demo:

Ready to build?

This pattern, using an agent for the interface and using Workflows for the execution, can be applied to almost any ITSM task, from password resets to software provisioning.

If you’re interested in trying this out, be sure to watch the DevFest talk for the full context, and check out the Elastic AI Agent Builder documentation to get started building your own agents today.

After all, everything in the context influences the AI’s responses. Garbage in, garbage out holds true.

In this article, we’ll introduce what short-term and long-term memory mean for AI agents, specifically:

• The difference between short- and long-term memory.

• How they relate to retrieval-augmented generation (RAG) techniques with vector databases, like Elasticsearch, and why careful memory management is necessary.

• The risks of neglecting memory, including context overflow and context poisoning.

• Best practices, like context pruning, summarizing, and retrieving only what’s relevant, to keep an agent’s memory both useful and safe.

• Finally, we’ll touch on how memory can be shared and propagated in multi-agent systems to enable agents to collaborate without confusion using Elasticsearch.

Short-term versus long-term memory in AI agents

Short-term memory in an AI agent typically refers to the immediate conversational context or state—essentially, the current chat history or recent messages in the active session. This includes the user’s latest query and recent back-and-forth exchanges. It’s very similar to the information a person holds in mind during an ongoing conversation.

AI frameworks often maintain this transient memory as part of the agent’s state (for example, using a checkpointer to store the conversation state as covered by this example from LangGraph). Short-term memory is session-scoped; that is, it exists within a single conversation or task and is reset or cleared when that session ends, unless explicitly saved elsewhere. An example of session-bound short-term memory would be the temporary chatavailable in ChatGPT.

Long-term memory, on the other hand, refers to information that persists across conversations or sessions. This is the knowledge an agent retains over time, facts it learned earlier, user preferences, or any data we’ve told it to remember permanently.

Long-term memory is usually implemented by storing and fetching it from an external source, such as a file or vector database that’s outside the immediate context window. Unlike short-term chat history, long-term memory isn’t automatically included in every prompt. Instead, based on a given scenario, the agent must recall or retrieve it when relevant tools are invoked. In practice, long-term memory might include a user’s profile info, prior answers or analyses the agent produced, or a knowledge base the agent can query.

For instance, if you have a travel-planner agent, the short-term memory would contain details of the current trip inquiry (dates, destination, budget) and any follow-up questions in that chat; whereas the long-term memory could store the user’s general travel preferences, past itineraries, and other facts shared in previous sessions. When the user returns later, the agent can pull from this long-term store (for example, the user loves beaches and mountains, has an average budget of INR 100,000, has a bucket list to visit, and prefers to experience history and culture rather than kid-friendly attractions) so that it doesn’t treat the user as a blank slate each time.

The short-term memory (chat history) provides immediate context and continuity, while long-term memory provides a broader context that the agent can draw upon when needed. Most advanced AI agent frameworks enable both: They keep track of recent dialogue to maintain context and offer mechanisms to look up or store information in a longer-term repository. Managing short-term memory ensures it stays within the context window, while managing long-term memory helps the agent to ground the answers based on prior interactions and personas.

Memory and RAG in context engineering

How do we give an AI agent a useful long-term memory in practice?

One prominent approach for long-term memory is semantic memory, often implemented via retrieval-augmented generation (RAG). This involves coupling the LLM with an external knowledge store or vector-enabled datastore, like Elasticsearch. When the LLM needs information beyond what’s in the prompt or its built-in training, it performs semantic retrieval against Elasticsearch and injects the most relevant results into the prompt as context. This way, the model’s effective context includes not only the recent conversation (short-term memory) but also pertinent long-term facts fetched on the fly. The LLM then grounds its answer on both its own reasoning and the retrieved information, effectively combining short-term memory and long-term memory to produce a more accurate, context-aware response.

Elasticsearch can be used to implement long-term memory for AI agents. Here’s a high-level example of how context can be retrieved from Elasticsearch for long-term memory.

This way, the agent “remembers” by searching for relevant data rather than by storing everything in its limited prompt, where it leads to different risks.

Using RAG with Elasticsearch or any vector stores offers multiple benefits:

First, it extends the knowledge of the model beyond its training cutoff. The agent can retrieve up-to-date information or domain-specific data that the LLM might not know. This is crucial for questions about recent events or specialized topics.

Second, retrieving context on demand helps reduce hallucinations, especially since LLMs aren’t trained on the proprietary or highly specialized data relative to your niche use case, which is highly likely to expose it to hallucinations. Instead of the LLM guessing or inventing new information as it has been incentivised through evaluation, as highlighted in a recent OpenAI paper (Why Language Models Hallucinate), the model can be grounded by factual references from Elasticsearch. Naturally, the LLM depends on the reliability of the data in the vector store to truly prevent misinformation and the relevant data is retrieved as per the core relevance measures.

Third, RAG allows an agent to work with knowledge bases far larger than anything you could ever fit into a prompt. Instead of pushing entire documents, like long research papers or policy documents, into the context window and risking overload or irrelevant information context poisoning the model’s reasoning, RAG relies on chunking. Large documents are broken into smaller, semantically meaningful pieces, and the system retrieves only the few chunks most relevant to the query. This way, the model doesn’t need a million-token context to appear knowledgeable; it just needs access to the right chunks of a much larger corpus.

It’s worth noting that as LLM context windows have grown (some models now support hundreds of thousands or even millions of tokens), a debate arose about whether RAG is “dead.” Why not push all the data into the prompt? If you feel likewise, refer to this wonderful article by my colleagues, Jeffrey Rengifo and Eduard Martin, Longer context ≠ better: Why RAG still matters. This avoids the “garbage in, garbage out” problem: The LLM stays focused on the few chunks that matter, rather than running through noise.

That said, integrating Elasticsearch or any vector store into an AI agent architecture provides long-term memory. The agent stores knowledge externally and pulls it in as memory context when needed. This could be implemented as an architecture, where after each user query, the agent performs a search on Elasticsearch for relevant info and then appends the top results to the prompt before calling the LLM. The response might also be saved back into the long-term store if it contains useful new information (creating a feedback loop of learning). By using such retrieval-based memory, the agent remains informed and up to date, without having to cram everything it knows into every prompt, even though the context window supports one million tokens. This technique is a cornerstone of context engineering, combining the strengths of information retrieval and generative AI.

Here’s an example of a managed in-memory conversation state using LangGraph's checkpoint system for short-term memory during the session. (Refer to our supporting context engineering app.)

Here’s how it stores checkpoints:

For long-term memory, here's how we perform semantic search on Elasticsearch to retrieve relevant previous conversations using vector embeddings after summarizing and indexing the checkpoints to Elasticsearch.

Now that we’ve explored how short-term memory and long-term memory are indexed and fetched using LangGraph’s checkpoints in Elasticsearch, let’s take some time to understand why indexing and dumping the complete conversations can be risky.

Risks of not managing context memory

As we’re talking much about context engineering, along with short-term and long-term memory, let’s understand what happens if we don’t manage an agent’s memory and context well.

Unfortunately, many things can go wrong when an AI’s context grows extremely long or contains bad information. As context windows get larger, new failure modes emerge, like:

• Context poisoning

• Context distraction

• Context confusion

• Context clash

• Context leakage and knowledge conflicts

• Hallucinations and misinformation

Let’s break down these issues and other risks that arise from poor context management:

Context poisoning

Context poisoning refers to when incorrect or harmful information ends up in the context and “poisons” the model’s subsequent outputs. A common example is a hallucination by the model that gets treated as fact and inserted into the conversation history. The model might then build on that error in later responses, compounding the mistake. In iterative agent loops, once a false information makes it into the shared context (for example, in a summary of the agent’s working notes), it can be reinforced over and over.

Researchers at DeepMind, in the release of the Gemini 2.5 report (TL;DR, check here), observed this in a long-running Pokémon-playing agent: If the agent hallucinated a wrong game state and that got recorded into its context (its memory of goals), the agent would form nonsensical strategies around an impossible goal and get stuck. In other words, a poisoned memory can send the agent down the wrong path indefinitely.

Context poisoning can happen innocently (by mistake) or even maliciously, for instance, via prompt injection attacks where a user or third-party sneaks in a hidden instruction or false fact that the agent then remembers and follows.

Recommended countermeasures:

Based on insights from Wiz, Zerlo, and Anthropic, countermeasures for context poisoning focus on preventing bad or misleading information from entering an LLM’s prompt, context window, or retrieval pipeline. Key steps include:

• Check the context constantly: Monitor the conversation or retrieved text for anything suspicious or harmful, not just the starting prompt.

• Use trusted sources: Score or label documents based on credibility so the system prefers reliable information and ignores low scored data.

• Spot unusual data: Use tools that detect odd, out-of-place, or manipulated content, and remove it before the model uses it.

• Filter inputs and outputs: Add guardrails so harmful or misleading text can’t easily enter the system or be repeated by the model.

• Keep the model updated with clean data: Regularly refresh the system with verified information to counter any bad data that slipped through.

• Human-in-the-loop: Have people review important outputs or compare them against known, trustworthy sources.

Simple user habits also help, resetting long chats, sharing only relevant information, breaking complex tasks into smaller steps, and maintaining clean notes outside the model.

Together, these measures create a layered defense that protects LLMs from context poisoning and keeps outputs accurate and trustworthy.

Without countermeasures as mentioned here, an agent might remember instructions, like ignore previous guidelinesor trivial facts that an attacker inserted, leading to harmful outputs.

Context distraction

Context distraction is when a context grows so long that the model overfocuses on the context, neglecting what it learned during training. In extreme cases, this resembles catastrophic forgetting; that is, the model effectively “forgets” its underlying knowledge and becomes overly attached to the information placed in front of it. Previous studies have shown that LLMs often lose focus when the prompt is extremely long.

The Gemini 2.5 agent, for example, supported a million-token window, but once its context grew beyond a certain point (on the order of 100,000 tokens in an experiment), it began to fixate on repeating its past actions instead of coming up with new solutions. In a sense, the agent became a prisoner of its extensive history. It kept looking at its long log of previous moves (the context) and mimicking them, rather than using its underlying training knowledge to devise fresh and novel strategies.

This is counterproductive. We want the model to use relevant context to help reasoning, not override its ability to think. Notably, even models with huge windows exhibit this context rot: Their performance degrades nonuniformly as more tokens are added. There appears to be an attention budget., Like humans with limited working memory, an LLM has a finite capacity to attend to tokens, and as that budget is stretched, its precision and focus drop.

As a mitigation, you can prevent context distraction using chunking, engineering the right information, regular context summarization, and evaluation and monitoring techniques to measure the accuracy of the response using scoring.

These methods keep the model grounded in both relevant context and its underlying training, reducing the risk of distraction and improving overall reasoning quality.

Context confusion

Context confusion is when superfluous content in the context is used by the model to generate a low-quality response.A prime example is giving an agent a large set of tools or API definitions that it might use. If many of those tools are unrelated to the current task, the model may still try to use them inappropriately, simply because they’re present in context. Experiments have found that providing more tools or documents can hurt performance if they’re not all needed. The agent starts making mistakes, like calling the wrong function or referencing irrelevant text.

In one case, a small Llama 3.1 8B model failed a task when given 46 tools to consider but succeeded when given only 19 tools. The extra tools created confusion, even though the context was within length limits. The underlying issue is that any information in the prompt will be attended to by the model. If it doesn’t know to ignore something, that something could influence its output in undesired ways. Irrelevant bits can “steal” some of the model’s attention and lead it astray (for instance, an irrelevant document might cause the agent to answer a different question than asked). Context confusion often manifests as the model producing a low-quality response that integrates unrelated context. Refer to the research paper: Less is More: Optimizing Function Calling for LLM Execution on Edge Devices.

It reminds us that more context isn’t always better, especially if it’s not curated for relevance.

Context clash

Context clash occurs when parts of the context contradict each other, causing internal inconsistencies that derail the model’s reasoning. A clash can happen if the agent accumulates multiple pieces of information that are in conflict.

For example, imagine an agent that fetched data from two sources: One says Flight A departs at 5 PM, and the other says Flight A departs at 6 PM. If both facts end up in the context, the poor model has no way to know which is correct; it may get confused or produce an incorrect or non-similar answer.

Context clash also frequently occurs in multiturn conversations where the model’s earlier attempts at answering are still lingering in the context along with later refined information.

A research study by Microsoft and Salesforce shows that if you break a complex query into multiple chatbot turns (adding details gradually), the final accuracy drops significantly, compared to giving all details in a single prompt. Why? Because the early turns contain partial or incorrect intermediate answers from the model, and those remain in the context. When the model later tries to answer with all info, its memory still includes those wrong attempts, which conflict with the corrected info and lead it off track. Essentially, the conversation’s context clashes with itself. The model may inadvertently use an outdated piece of context (from an earlier turn) that doesn’t apply after new info is added.

In agent systems, context clash is especially dangerous because an agent might combine outputs from different tools or subagents. If those outputs disagree, the aggregated context is inconsistent. The agent could then get stuck or produce nonsensical results trying to reconcile the contradictions. Preventing context clash involves ensuring the context is fresh and consistent,for instance, clearing or updating any outdated info and not mixing sources that haven’t been vetted for consistency.

Context leakage and knowledge conflicts

In systems where multiple agents or users share a memory store, there’s a risk of information bleeding over between contexts.

For example, if two separate users’ data embeddings reside in the same vector database without proper access control, an agent answering User A’s query might accidentally retrieve some of User B’s memory. This cross-context leak can expose private information or just create confusion in responses.

According to the OWASP Top 10 for LLM Applications, multitenant vector databases must guard against such leakage:

According to LLM08:2025 Vector and Embedding Weaknesses, one of the common risks is context leakage:

Another aspect is that an LLM might have trouble overriding its built-in knowledge with new info from memory. If the model was trained on some fact and the retrieved context says the opposite, the model can get confused about which to trust. Without proper design, the agent could mix up contexts or fail to update old knowledge with new evidence, leading to stale or incorrect answers.

Hallucinations and misinformation

While hallucination (the LLM making up plausible-sounding but false information) is a known problem even without long contexts, poor memory management can amplify it.

If the agent’s memory is lacking a crucial fact, the model may just fill in the gap with a guess, and if that guess then enters the context (poisoning it), the error persists.

The OWASP LLM security report (LLM09:2025 Misinformation) highlights misinformation as a core vulnerability: LLMs can produce confident but fabricated answers, and users may overtrust them. An agent with a bad or outdated long-term memory might confidently cite something that was true last year but is false now, unless its memory is kept up to date.

Overreliance on the AI’s output (by either the user or the agent itself in a loop) can make this worse. If no one ever checks the info in memory, the agent can accumulate falsehoods. This is why RAG is often used to reduce hallucinations: By retrieving an authoritative source, the model doesn’t have to invent facts. But if your retrieval pulls in the wrong document (say, one that contains misinformation) or if an early hallucination isn’t pruned, the system may propagate that misinformation throughout its actions.

The bottom line: Failing to manage memory can lead to incorrect and misleading outputs, which can be damaging, especially if the stakes are high (for example, bad advice in a finance or medical domain). An agent needs mechanisms to verify or correct its memory content, not just unconditionally trust whatever is in the context.

In summary, giving an AI agent an infinitely long memory or dumping every possible thing into its context is not a recipe for success.

Best practices for memory management in LLM applications

To avoid the pitfalls above, developers and researchers devised a number of best practices for managing context and memory in AI systems. These practices aim to keep the AI’s working context lean, relevant, and up to date.Here are some of the key strategies, along with examples of how they help.

RAG: Use targeted context

Much of RAG has already been covered in the earlier section, so this serves as a concise set of practical reminders:

• Use targeted retrieval, not bulk loading: Retrieve only the most relevant chunks instead of pushing entire documents or full conversation histories into the prompt.

• Treat RAG as just-in-time memory recall: Fetch context only when it’s needed, rather than carrying everything forward across turns.

• Prefer relevance-aware retrieval strategies: Approaches like top-k semantic search, Reciprocal Rank Fusion, or tool loadout filtering help reduce noise and improve grounding.

• Larger context windows don’t remove the need for RAG: Two highly relevant paragraphs are almost always more effective than 20 loosely related pages.

That said, RAG isn’t about adding more context; it’s about adding the right context.

Tool loadout

Tool loadout is about giving a model only the tools it actually needs for a task. The term comes from gaming: You pick a loadout that fits the situation. Too many tools slow you down; the wrong ones cause failure. LLMs behave the same way, according to the research paper Less is more. Once you pass ~30 tools, descriptions start overlapping and the model gets confused. Past ~100 tools, failure is almost guaranteed. This isn’t a context window problem, it’s context confusion.

A simple and effective fix is RAG-MCP. Instead of dumping every tool into the prompt, tool descriptions are stored in a vector database and only the most relevant ones are retrieved per request. In practice, this keeps the loadout small and focused, dramatically shortens prompts, and can improve tool selection accuracy by up to 3x.

Smaller models hit this wall even sooner. The research shows an 8B model failing with dozens of tools but succeeding once the loadout is trimmed. Dynamically selecting tools, sometimes with an LLM first, reasoning about what it thinks it needs, can boost performance by 44%, while also reducing power usage and latency. The takeaway is that most agents only need a few tools, but as your system grows, tool loadout and RAG-MCP become first-order design decisions.

Context pruning: Limit the chat history length

If a conversation goes on for many turns, the accumulated chat history can become too large to fit, leading to context overflow or becoming too distracting to the model.

Trimming means programmatically removing or shortening less important parts of the dialogue as it grows. One simple form is to drop the oldest turns of the conversation when you hit a certain limit, keeping only the latest N messages. More sophisticated pruning might remove irrelevant digressions or previous instructions that are no longer needed. The goal is to keep the context window uncluttered by old news.

For example, if the agent solved a subproblem 10 turns ago and we have since moved on, we might delete that portion of the history from the context (assuming it won’t be needed further). Many chat-based implementations do this: They maintain a rolling window of recent messages.

Trimming can be as simple as “forgetting” the earliest parts of a conversation once they’ve been summarized or are deemed irrelevant. By doing so, we reduce the risk of context overflow errors and also reduce context distraction, so the model won’t see and get sidetracked by old or off-topic content. This approach is very similar to how humans might not remember every word from an hour-long talk but will retain the highlights.

If you’re confused about context pruning, as highlighted by the author Drew Breunig here, usage of the Provence (`naver/provence-reranker-debertav3-v1`) model, a lightweight (1.75 GB), efficient, and accurate context pruner for question answering, can make a difference. It can trim large documents down to only the most relevant text for a given query. You can call it in specific intervals.

Here’s how we invoke the `provence-reranker` model in our code to prune the context:

We use the Provence reranker model (`naver/provence-reranker-debertav3-v1`) to score sentence relevance. Threshold-based filtering keeps sentences above the relevance threshold. Also, we introduce a fallback mechanism, where we return to the original context if pruning fails. Finally, statistics logging tracks reduction percentage in verbose mode.

Context summarization: Condense older information instead of dropping it entirely

Summarization is a companion to trimming. When the history or knowledge base becomes too large, you can employ the LLM to generate a brief summary of the important points and use that summary in place of the full content going forward, as we performed in our code above.

For example, if an AI assistant has had a 50-turn conversation, instead of sending all 50 turns to the model on turn 51 (which likely won’t fit), the system might take turns 1–40, have the model summarize them in a paragraph, and then only supply that summary plus the last 10 turns in the next prompt. This way, the model still knows what was discussed without needing every detail. Early chatbot users did this manually by asking, “Can you summarize what we’ve talked about so far?” and then continuing in a new session with the summary. Now it can be automated. Summarization not only saves context window space but can also reduce context confusion/distraction by stripping away extra detail and retaining just the salient facts.

Here’s how we use OpenAI models (you can use any LLMs) to condense context while preserving all relevant information, eliminating redundancy and duplication.

Importantly, when the context is summarized, the model is less likely to get overwhelmed by trivial details or past errors (assuming the summary is accurate).

However, summarization has to be done carefully. A bad summary might omit a crucial detail or even introduce an error. It’s essentially another prompt to the model (“summarize this”), so it can hallucinate or lose nuance. Best practice is to summarize incrementally and perhaps keep some canonical facts unsummarized.

Nonetheless, it has proven very useful. In the Gemini agent scenario, summarizing the context every ~100k tokens was a way to counteract the model’s tendency to repeat itself. The summary acts like a compressed memory of the conversation or data. As developers, we can implement this by having an agent periodically call a summarization function (maybe a smaller LLM or a dedicated routine) on the conversation history or a long document. The resulting summary replaces the original content in the prompt. This tactic is widely used to keep contexts within limits and distill the information.

Context quarantine: Isolate contexts when possible

This is more relevant in complex agent systems or multistep workflows. The idea of context segmentation is to split a big task into smaller, isolated tasks, each with its own context, so that you never accumulate one enormous context that contains everything. Each subagent or subtask works on a piece of the problem with a focused context, and then a higher-level agent, or supervisor or coordinator integrates the results.

Anthropic’s research strategy uses multiple subagents, each investigating a different aspect of a question, with their own context windows, and a lead agent that reads the distilled results from those subagents. This parallel, modular approach means that no single context window gets too bloated. It also reduces the chance of irrelevant information mixing, each thread stays on topic (no context confusion), and it doesn’t carry unnecessary baggage when answering its specific subquestion. In a sense, it’s like running separate threads of thought that only share their outcomes, not their entire thought process.

In multi-agent systems, this approach is essential. If Agent A is handling task A and Agent B is handling task B, there’s no reason for either agent to consume the other’s full context unless it’s truly required. Instead, agents can exchange only the necessary information. For example, Agent A can pass a consolidated summary of its findings to Agent B via a supervisor agent, while each subagent maintains its own dedicated context thread. This setup doesn’t require human-in-the-loop intervention; it relies on a supervisory agent with enabled tools with minimal and controlled context sharing.

Nonetheless, designing your system so that agents or tools operate with minimal necessary context overlap can greatly enhance clarity and performance. Think of it as microservices for AI, each component deals with its context, and you pass messages between them in a controlled way, instead of one monolithic context.These best practices are often used in combination. Also, this gives you the flexibility to trim trivial history, summarize important older messages or conversations, offload the detailed logs to Elasticsearch for long-term context, and use retrieval to bring back anything relevant when needed.

As mentioned here, the guiding principle is that context is a limited and precious resource. You want every token in the prompt to earn its keep, meaning it should contribute to the quality of the output. If something in memory is not pulling its weight (or worse, actively causing confusion), then it should be pruned, summarized, or kept out.

As developers, we can now program the context just like we program code, deciding what information to include, how to format it, and when to omit or update it. By following these practices, we can give LLM agents the much-needed context to perform tasks without falling victim to the failure modes described earlier. The result is agents that remember what they should, forget what they don’t need, and retrieve what they require just in time.

Conclusion

Memory isn’t something you add to an agent; it’s something you engineer. Short-term memory is the agent’s working scratch pad, and long-term memory is its durable knowledge store. RAG is the bridge between the two, turning a passive datastore, like Elasticsearch, into an active recall mechanism that can ground outputs and keep the agent current.

But memory is a double-edged sword. The moment you let context grow unchecked, you invite poisoning, distraction, confusion, and clashes, and in shared systems, even data leakage. That’s why the most important memory work isn’t “store more,” it’s “curate better”: Retrieve selectively, prune aggressively, summarize carefully, and avoid mixing unrelated contexts unless the task truly demands it.

In practice, good context engineering looks like good systems design: smaller, sufficient contexts, controlled interfaces between components, and a clear separation between raw and the distilled state you actually want the model to see. Done right, you don’t end up with an agent that remembers everything - you end up with an agent that remembers the right things, at the right time, for the right reason.

Model Context Protocol (MCP) is an open-source standard that enables applications and LLMs to communicate with external systems via MCP tools (programmatic capabilities), and LangGraph (an extension of LangChain) provides the orchestration framework for these agentic workflows.

We’ll implement an application that can search both internal knowledge (Elasticsearch stored data) and external sources (on the internet) to identify potential and known vulnerabilities related to a specific tool. The application will gather the information and generate a detailed summary of the findings.

Requirements

• Elasticsearch 9.2

• Python 3.1x

• OpenAI API Key

• Elasticsearch API Key

• Serper API Key

Elastic Agent Builder

Elastic Agent Builder is a set of AI-powered capabilities for developing and integrating agents that can interact with your Elasticsearch data. It provides a built-in agent that can be used for natural language conversations with your data or instance, and it also supports tool creation, Elastic APIs, A2A, and MCP. In this article, we will focus on using the MCP server for external access to the Elastic Agent Builder tools.

To know more about Agent Builder features, you can read this article.

Agent Builder MCP feature

The MCP server is available in the Agent Builder and can be accessed at:

The Agent Builder offers Built-in tools, and you can also create your custom tools.

Reference architecture

To get a complete overview of the elements used by an agentic application in an end-to-end workflow, let’s look at the following diagram:

Elasticsearch is at the center of this architecture, functioning as a vector store, providing the embeddings generation model, and also serving the MCP server to access the data via tools. To better explain the workflow, let’s look at the ingestion and the Agent Builder layer separately.

Here, the first element is the data that will be stored in Elasticsearch. The data passes through an ingest pipeline, where it is processed by the Elasticsearch ELSER model to generate embeddings and then stored in Elasticsearch.

Elastic Agent Builder layer

On this layer, the Agent Builder plays a central role by exposing the tools needed to interact with the Elasticsearch data. It manages the tools that operate over Elasticsearch indices and makes them available for consumption. Then LangChain handles the orchestration via the MCP client.

This architecture allows Agent Builder to work as one of many MCP servers available to the client so that the Elasticsearch agent builder can combine with other MCPs. This way, the MCP client can ask cross-source questions and then combine the answers.

Use case: Security vulnerability agent

The security vulnerability agent identifies potential risks based on a user’s question by combining three complementary layers:

First, it performs a semantic search with embeddings over an internal knowledge base of past incidents, configurations, and known vulnerabilities to retrieve relevant historical evidence.

Second, it searches the internet for newly published recommendations or threat intelligence that may not yet exist internally.

Finally, an LLM correlates and prioritizes both internal and external findings, evaluates their relevance to the user’s specific environment, and produces a clear explanation along with potential mitigation steps.

Developing the application

The application’s code can be found in the attached notebook.

You can see the setup for the Python application below:

We need to access Agent Builder and create one agent specialized in security queries and one tool to perform semantic search. You need to have the Agent Builder enabled for the next step. Once it’s on, we’ll use the tools API to create a tool that will perform a semantic search.

Configure your tools following the best practices defined by Elastic for developing Tools. Once created, this tool will be ready to use in the Kibana UI.

With the tool created, we can start writing the code for the ingestion workflow:

Ingest pipeline

To define the data structure, we need to have a dataset prepared for ingestion. Below is a sample document for this example:

For this type of document, we will use the following index mappings:

We are creating a semantic_text field to perform semantic search using the information from the fields marked with the copy_to property.

With that mapping definition, we can ingest the data using the bulk API.

LangChain MCP client

Here we’re going to create an MCP client using LangChain to consume the Agent Builder tools and build a workflow with LangGraph to orchestrate the client execution. The first step is to connect to the MCP server:

Next, we create an agent that selects the appropriate tool based on the user input:

We’ll use the GPT-5.2 model, which represents OpenAI’s state-of-the-art for agent management tasks. We configure it with low reasoning effort to achieve faster responses compared to the medium or high settings, while still delivering high-quality results by leveraging the full capabilities of the GPT-5 family. You can read more about the GPT 5.2 here.

Now that the initial setup is done, the next step is to define a workflow capable of making decisions, running tool calls, and summarizing results.

For this, we use LangGraph. We won’t cover LangGraph in depth here; this article provides a detailed overview of its functionality.

The following image shows a high-level view of the LangGraph application.

We need to define the application state:

To better understand how the workflow operates, here is a brief description of each function. For full implementation details, refer to the accompanying notebook.

• call_agent_builder_semantic_search: Queries internal documentation using the Agent Builder MCP server and also stores the retrieved messages in the state.

• decide_internet_search: Analyzes the internal results and determines whether an external search is required.

• perform_internet_search: Runs an external search using the Serper API when needed.

• generate_response: Correlates internal and external findings and produces a final, actionable cybersecurity analysis for the user.

With the workflow defined, we can now send a query:

In this example, we want to evaluate whether this specific version of Express is affected by known vulnerabilities.

Research results

See the complete response in this file.

This response clearly correlates internal and internet findings and provides actionable mitigation steps. It successfully highlights the severity of the vulnerability and offers a structured, security-oriented summary.

Extensions and future enhancements

This architecture is modular and allows us to extend its capabilities by replacing, improving, or adding components to the existing list. We could add another agent, consumed by the same MCP client. We can also use an automated ingestion workflow with tools such as Logstash, Kafka, or Elastic self-managed connectors. Feel free to change the LLM, the MCP client framework, or the embeddings model or add more tools depending on your needs.

Conclusion

This reference architecture shows a practical way to combine Elasticsearch, the Agent Builder, and MCP to build an AI-driven application. Its structure keeps each part independent, which makes the system easy to implement, maintain, and extend.

You can start with a simple setup (like the security use case in this article) and scale it by adding new tools, data sources, or agents as your needs grow. Overall, it provides a straightforward path for building flexible and reliable agentic workflows on top of Elasticsearch.

This article will show you how to build an AI agent for HR using GPT-OSS and Elastic Agent Builder. The agent can answer your questions without sending data to OpenAI, Anthropic, or any external service.

We’ll use LM Studio to serve GPT-OSS locally and connect it to Elastic Agent Builder.

By the end of this article, you’ll have a custom AI agent that can answer natural language questions about your employee data while maintaining full control over your information and model.

Prerequisites

For this article, you need:

• Elastic Cloud hosted 9.2, serverless or local deployment

• Machine with 32GB RAM recommended (minimum 16GB for GPT-OSS 20B)

• LM Studio installed

• Docker Desktop Installed

Why use GPT-OSS?

With a local LLM you have the control to deploy it in your own infrastructure and fine-tune it to fit your own needs. All this while maintaining control over the data that you share with the model, and of course, you don’t have to pay a license fee to an external provider.

OpenAI released GPT-OSS on August 5, 2025, as part of their commitment to the open model ecosystem.

The 20B parameter model offers:

• Tool use capabilities

• Efficient inference

• OpenAI SDK compatible

• Compatible with agentic workflows

Benchmark comparison:

Solution architecture

The architecture runs entirely on your local machine. Elastic (running in Docker) communicates directly with your local LLM through LM Studio, and the Elastic Agent Builder uses this connection to create custom AI agents that can query your employee data.

For more details, refer to this documentation.

Building an AI agent for HR: Steps

We’ll divide the implementation into 5 steps:

1. Configure LM studio with a local model

2. Deploy Local Elastic with Docker

3. Create the OpenAI connector in Elastic

4. Upload employee data to Elasticsearch

5. Build and test your AI Agent

Step 1: Configure LM Studio with GPT-OSS 20B

LM Studio is a user-friendly application that allows you to run large language models locally on your computer. It provides an OpenAI-compatible API server, making it easy to integrate with tools like Elastic without a complex setup process. For more details, refer to the LM Studio Docs.

First, download and install LM Studio from the official website. Once installed, open the application.

In the LM Studio interface:

1. Go to the search tab and search for “GPT-OSS”

2. Select the openai/gpt-oss-20b from OpenAI

3. Click download

The size of this model should be approximately 12.10GB. The download may take a few minutes, depending on your internet connection.

Once the model is downloaded:

1. Go to the local server tab

2. Select the openai/gpt-oss-20b

3. Use the default port 1234

4. On the right panel, go to Load and set the Context Length to 40K or higher

5. Click start server

You should see this if the server is running.

Step 2: Deploy Local Elastic with Docker

Now we’ll set up Elasticsearch and Kibana locally using Docker. Elastic provides a convenient script that handles the entire setup process. For more details refer to the official documentation.

Run the start-local script

Execute the following command in your terminal:

This script will:

• Download and configure Elasticsearch and Kibana

• Start both services using Docker Compose

• Automatically activate a 30-day Platinum trial license

Expected output

Just wait for the following message and save the password and API key shown; you’ll need them to access Kibana:

Access Kibana

Open your browser and navigate to:

Log in using the credentials obtained in the terminal output.

Enable Agent Builder

Once logged in to Kibana, navigate to Management > AI > Agent Builder and activate the Agent Builder.

Step 3: Create the OpenAI connector in Elastic

Now we’ll configure Elastic to use your local LLM.

Access Connectors

1. In Kibana

2. Go to Project Settings > Management

3. Under Alerts and Insights, select Connectors

4. Click Create Connector

Configure the connector

Select OpenAI from the list of connectors. LM Studio uses the OpenAI SDK, making it compatible.

Fill in the fields with these values:

• Connector name: LM Studio - GPT-OSS 20B

• Select an OpenAI provider: Other (OpenAI Compatible Service)

• URL: http://host.docker.internal:1234/v1/chat/completions

• Default model: openai/gpt-oss-20b

• API Key: testkey-123 (any text works, because LM Studio Server doesn't require authentication.)

To finish the configuration, click Save & test.

Important: Toggle ON the “Enable native function calling”; this is required for the Agent Builder to work properly. If you don’t enable this, you’ll get a No tool calls found in the response error.

Test the connection

Elastic should automatically test the connection. If everything is configured correctly, you’ll see a success message like this:

Response:

Step 4: Upload employee data to Elasticsearch

Now we’ll upload the HR employee dataset to demonstrate how the agent works with sensitive data. I generated a fictional dataset with this structure.

Dataset structure

Create the index with mappings

First, create the index with proper mappings. Note that we’re using semantic_text fields for some key fields; this enables semantic search capabilities for our index.

Index with Bulk API

Copy and paste the dataset into your Dev Tools in Kibana and execute it:

Verify the data

Run a query to verify:

Step 5: Build and test your AI agent

With everything configured, it’s time to build a custom AI agent using Elastic Agent Builder. For more details refer to the Elastic documentation.

Add the connector

Before we can create our new agent, we have to set our Agent builder to use our custom connector called LM Studio - GPT-OSS 20B because the default one is the Elastic Managed LLM. For that, we need to go to Project Setting > Management > GenAI Settings; now we select the one we created and click Save.

Access Agent Builder

1. Go to Agents

2. Click on Create a new agent

Configure the agent

To create a new agent, the required fields are the Agent ID, Display Name, and Display Instructions.

But there are more customization options, like the Custom Instructions that guide how your agent is going to behave and interact with your tools, similar to a system prompt, but for our custom agent. Labels help organize your agents, avatar color, and avatar symbol.

The ones that I chose for our agent based on the dataset are: Agent ID: hr_assistant

Custom instructions:

Labels: Human Resources and GPT-OSS

Display name: HR Analytics Assistant

Display description:

With all the data in there, we can click on Save our new agent.

Test the agent

Now you can ask natural language questions about your employee data, and GPT-OSS 20B will understand the intent and generate an appropriate response.

Prompt:

Answer:

The Agent process was:

1. Understand your question using the GPT-OSS connector

2. Generate the appropriate Elasticsearch query (using the built-in tools or custom ES|QL)

3. Retrieve matching employee records

4. Present results in natural language with proper formatting

Unlike traditional lexical search, the agent powered by GPT-OSS understands intent and context, making it easier to find information without knowing exact field names or query syntax. For more details on the agent's thinking process, refer to this article.

Conclusion

In this article, we built a custom AI agent using Elastic’s Agent Builder to connect to the OpenAI GPT-OSS model running locally. By deploying both Elastic and the LLM on your local machine, this architecture allows you to leverage generative AI capabilities while maintaining full control over your data, all without sending information to external services.

We used GPT-OSS 20B as an experiment, but the officially recommended models for Elastic Agent Builder are referenced here. If you need more advanced reasoning capabilities, there's also the 120B parameter variant that performs better for complex scenarios, though it requires a higher-spec machine to run locally. For more details, refer to the official OpenAI documentation.

In this blog post we’ll walk through all the steps to use Elastic Agent Builder to create an agent. Then we’ll walk through the process of running an example Python app that uses the Microsoft Agent Framework to orchestrate your Elastic agent.

Create an Elastic Serverless project

To use Agent Builder you need an Elastic deployment or an Elastic serverless project, so let’s begin by creating an Elastic serverless project. Go to Elastic Cloud and create a new Elasticsearch Serverless project.

Create an index and add data

Now that we’ve got an Elastic project, let’s create an index, which is what Elasticsearch uses to store data. Open Developer Tools in Elastic Cloud where we can run a command to create an index.

Copy the following PUT command which creates an index named my-docs containing a mixture of fields, and our content leveraging semantic search.

Paste the PUT command into the input area of the Developer Tools console. Hover your mouse over the command in the console and then click the Run button to execute the command.

The next step is to add some data to the my-docs index that you just created. Copy and paste the following command into the Develop Tools console.

Click the command’s Run button to execute the command which will add a document to the my-docs index.

As you can see, the command above adds a document named greetings.md that includes the contents of different potential types of greeting responses.

Now that we’ve got some data in an Elastic index, let’s get a confirmation of what data we have to work with. Using the power of the built-in Elastic AI Agent that is enabled by default in Agent Builder, you can now have a chat about your data. Select Agents in the navigation menu.

Then simply ask, “What data do I have?”

The default Elastic AI Agent provides a nice summary of the data currently stored in Elastic.

Create a tool

The next step on this walkthrough journey is to create an agent that can utilize the data stored in Elastic.

As you’ve seen the default agent in Elastic Agent builder is already useful for chatting with your data but to really give agents custom powers, they need access to tools via the Model Context Protocol (MCP). Agent Builder has fully featured tool creation and management functionality that you can use to quickly create custom MCP tools that are hosted right in the same scalable Elastic project as your data.

Let’s create a tool in Elastic Agent Builder that can access the data now stored in Elastic. Click + New to start a new chat.

Then click on Manage tools.

Click the + New tool button.

In the Create Tool form, select the ES|QL as the tool Type and enter the following values.

For Tool ID:

For Description:

For Configuration enter the following query into the ES|QL Query text area:

Your completed Create a new tool form should look like the following completed form. Click Save to create the tool.

Create an Agent and assign it a tool

Ah! There’s that feeling of having a new tool and being ready to use it. Agents need tools to give them special abilities beyond what general LLMs can provide and we’ve now got a brand new tool. Let’s create an agent that can put our tool to good use. Select Agents in the navigation menu.

Click Create a new agent.

Based on the name of the tool and the data it’s accessing, you’ve probably already guessed that we’re going to be creating a greeting agent and you’re right! Let’s create a Hello World agent right now.

In the New Agent form, enter the following values.

For Agent ID enter the text:

In the Custom Instructions text area enter the following instructions:

For Display name enter the text:

For the Display description enter the text:

Your completed New Agent form should look like the following completed form. The next step is to assign the agent the tool we created in the previous step. Click the Tools tab.

Select only the example.get_greetings tool that we created previously. Unselect all the other available tools. This will configure the agent being created to only have access to the tool we’ve created.

Click Save to create the agent.

You’ll be taken to the Agents list where you can see that the new HelloWorld Agent has been created.We can quickly test out our new agent right inside Agent Builder. Select Agents in the navigation menu.

Select the HelloWorld Agent from the Agent Chat agent selector. Enter the prompt “hello world” and you should get back the Hello World text from the greetings.md document stored in the my-docs Elastic index.

Well done. Now that we know our agent is working as expected, let’s explore the immediate development benefit that you get with tools and agents created in Agent Builder. Any tools you create in Agent Builder are usable via MCP by any agent-building platform that supports MCP. Also, any agents you create in Agent Builder are available for use in any agent-building platform that supports the AgentToAgent (A2A) protocol.

Microsoft Agent Framework

If you’re interested in trying out new Agent development tools, then there’s a recently announced open-source development kit called the Microsoft Agent Framework that you should definitely try out for yourself. The Agent Framework allows you to use the A2A protocol to orchestrate agentic apps that can combine multiple agents running on different hosts to enable solutions that aren’t possible with only a generic GenAI Large Language Model. The Agent Framework is available in Python and C#. Let’s see how we can use the Python-based Agent Framework to call the custom Elastic Agent we just created.

Getting started with the Agent Framework in Python

Let’s run some code! On your local computer open Visual Studio Code and open a new terminal.

In the open terminal, clone the Elastic Search Labs source code repository which contains the Elastic Agent Builder A2A example app.

In the terminal, cd to change directory to elasticsearch-labs.

In the terminal, enter the following command to open the current folder in the Visual Studio Code editor.

In the Visual Studio File Explorer, expand the supporting-blog-content and agent-builder-a2a-agent-framework folders and then open the file named elastic_agent_builder_a2a.py in the text editor.

Here’s the contents of elastic_agent_builder_a2a.py that you should see in your text editor:

The code within the main() method demonstrates how to control your Elastic Agent Builder agent using the Agent Framework. It creates an http_client using a URL and API key for the agent which you’ll provide from your Elastic project. Then the Agent Framework’s A2ACardResolver is called with that http_client to get your agent’s A2A agent card based on the relative_card_path of “/helloworld_agent.json” to reference your agent’s Agent ID which is “helloworld_agent”. The code then uses the Agent Framework to invoke your agent with the A2A agent card. The final part of the main() method prompts the user of the app for input of a “greeting” and then sends the user input as a prompt to your agent. Based on the instructions and tools specified when you created your agent, the agent’s response is displayed to the app user.

Setting your agent URL and API Key as environment variables

Make a copy of the file env.example and name the new file .env Edit the newly created .env file to set the values of the environment variables to use specific values copied from your Elastic project.

First we’ll replace <YOUR-ELASTIC-AGENT-BUILDER-URL> with the Agent URL path that you can copy from your Elastic project’s Agent Builder - Tools page. Back in Elastic Agent Builder click Agents in the navigation menu.

Select Manage tools.

Click the MCP Server dropdown at the top of the Tools page. Select Copy MCP Server URL.

Back in Visual Studio Code, within the .env file, find where the placeholder text “<YOUR-ELASTIC-AGENT-BUILDER-URL>” appears and paste in the copied MCP Server URL to replace the placeholder text. Now edit the pasted MCP Server URL. Delete the text “mcp” at the end of the URL and replace it with the text “a2a”. The edited URL should look something like this:

The next placeholder text to replace in the .env file is <YOUR-ELASTIC-API-KEY>. We’ll replace it with an actual API Key from your Elastic project. Back in your Elastic project, click Elasticsearch in the navigation menu.

Click Create API key to create a new API key.

Enter a Name for the API key and click Create API key.

Click the copy button to copy the API key.

Back in Visual Studio Code, within the .env file , find where the placeholder text “<YOUR-ELASTIC-API-KEY>” appears and paste in the copied API Keyvalueto replace the placeholder text.

Now we can save the changes we’ve made to the .env file. The edited file should look something like this:

Run the example app

It’s time to run the code. To do so, open a new terminal in Visual Studio Code. Click the Terminal top level menu and select New Terminal.

In the new terminal, cd to change directory to the directory containing the agent-builder-a2a-agent-framework example app.

In the terminal, create a Python virtual environment by running the following code.

Activate the virtual environment by running the following command (based on your operating system) in the terminal window:

• If you’re running MacOS or Linux, the command to activate the virtual environment is:

• If you’re on Windows, the command to activate the virtual environment is:

The code in the elastic_agent_builder_a2a.py file is powered by the Microsoft Agent Framework and we still need to install it, so let's do that now. Run the following pip command to install the Python based Agent Framework along with its necessary Python packages:

Hurray! Everything is now in its right place. It’s time for the good feeling fireworks…let’s run it. Run the example code by entering the following command into the terminal:

You should see the agent framework connect to the Elastic Agent. When prompted for a greeting, enter “hello world”. You should see the HelloWorld Agent’s response → Hello World! 🌎

Top-notch work!

Building agents and connecting them to tools in Agent Builder gets you immediate operability with the latest agent development platforms like the Microsoft Agent Framework. You now know how to create an Elastic agent and put it to use as a scalable relevant data source, ready to provide custom context to all the AI apps you’ll be building next.

Try Elastic for free and build some agents today!

Large codebases and their maintainers are like organisms, and any change can make them sick: Breaking builds, unit tests, etc. Maintainers, like antibodies, quickly jump in to restore health by removing the resulting bugs, a process that takes time and energy.

These codebases are built on the shoulders of hundreds of dependencies. We keep them all up to date on the products we host or distribute. This is aligned with our security standards, but comes with significant work volume generated from high rates of update-fix cycles. Each update is a potential germ that can break the build, getting the organism “sick”.

Traditional automation to deal with a big problem: Keeping dependencies up-to-date

This post is about automation. In this day and age, there is an important distinction to make between:

• Traditional automation: Steps performed by software encoding algorithms, driving the process being automated. It is deterministic, and its functionality is limited to what the developer intended the software to do.

• Generative AI (gen AI) automation: Steps performed by Large Language Models technology from inputs in natural language and driven by prompts. Its results are usually non-deterministic and require human supervision.

Back to the codebase-maintainers as organisms analogy, let’s talk about choosing our experiment subject. And yes, this post is about an experiment. An experiment that was so successful that it started helping our teams before becoming a fully polished internal feature.

We maintain a considerably large monorepo. We make sure dependencies are up-to-date and free of known vulnerabilities.

With about 500 actively updated dependencies for our core services, checking and bumping dependency versions is a non-trivial challenge. If performed manually, capable of taking away hundreds of engineering hours a week.

We adopted the dependency updates management system provided by our internal Engineering Productivity team as traditional automation. It is based on Renovate.

Renovate is a simple but effective bot. In a nutshell, it does two things:

1. Compares our repository dependency catalogs with the upstream packages repositories.

2. Opens Pull Requests when newer versions are found.

In about six months of operation, Renovate authored pull -requests have already bumped 41% of our dependencies, becoming one of our most prolific contributors:

Self-healing Pull Requests: Fix-Approve-Merge

Our integration with Renovate was successful. So much so that it raised the bar enough to swamp the team with PR reviews. Of course, it can be set up to throttle down, but we really want to keep everything up-to-date. It made us aware that such a standard came with an effort price that could have a detrimental impact on project progression time.

Renovate PRs, as any other in our repository, go through a build pipeline that compiles code, runs unit tests (UTs) and integration tests (ITs), builds Docker images, and publishes them. All this is done using the Buildkite platform with Gradle build steps as demonstrated in the following screenshot:

Renovate is set up to automatically merge Pull Requests when ITs pass successfully.

One could expect that most dependency bumps would just go through, especially for patch versions. So, where is the toil coming from? Well, we live in a world where dependency maintainers introduce breaking changes in patch or minor releases, or even worse, where they introduce subtle runtime changes that are not reflected in the interfaces at all (e.g, ZooKeeper’s 3.8.3->3.8.4 new ACL constraints in existence checks operations - ZOOKEEPER-2590 and downstream consequences in Apache Curator clients CURATOR-715). Here is where the build breaks and humans are required.

Really, humans?

We noticed that these broken PRs were the real bottleneck for automatic updates. Broken as in not compiling at all, failing UTs or ITs. Engineers on our team needed to chime in on these cases invariably. This is an unplanned interruption of work, bringing frequent and production-killing context switches.

These fixes:

• Are usually self-contained: They don’t require software rearchitecture, just adjustments to the changes in the updated libraries. Nor do they require deep creative work.

• They provide fast feedback loops: Edit-compile-test.

Can you think of an emergent technology targeting repetitive self-contained coding tasks?

What if each broken PR to review came with proposed code changes fixing it?

This is exactly the approach we decided to try in an experimentation week. And it worked!

But, how?

The idea is simple: follow the natural way of working with your workmates. Let AI chime in and contribute to PR branches. That is, integrate AI agents as part of PRs’ workflow.

This is also an approach pursued in the coding agents industry, with examples like GitHub Copilot Coding Agent or Claude Code GitHub actions.

Both off-the-shelf solutions make it easy to get the agent to interact with the code, but are less flexible.

We wanted the agent to act on the build errors in a targeted way: This unit test failed with this error for this module, fix that concrete problem, and add a commit to the working branch when you have fixed it.

For that, we needed to:

• Be able to feed the AI agent with the concrete error messages and failed build tasks. Including those depending on internal Elastic Cloud services.

• Give it the agency to run and iterate over failed Gradle steps until a solution is found or it desists.

As in most systems, the key to human Software Engineers’ productivity is fast iteration cycles: Edit-Compile-Test, and that’s what we decided with code editing AI agents.

For that, we just expanded the set of build steps we described at the beginning of this post with another. A replica of the Gradle build steps with the twist of being controlled by a coding agent with a quite specific prompt:

claude-fix-build.sh is the script that invokes the agent through several steps:

1. Run a pre-hook command, grabbing the credentials we need to interact with Claude and GitHub.

2. Clone the GitHub target repository.

3. Get ready to run Gradle build steps. That is preparing the environment for Gradle to be able to build the working code. To this point, this script is just a copy of the scripts we were already using for the regular build steps in the Buildkite workers: publishPlatformIndependent-amd64 and publishPlatformIndependent-arm64.

4. Use Buildkite CLI to try to fetch the previous step, the building step that failed, thus triggering the fix log. As we’ll see soon, this is a performance booster: The AI agent will analyze the file, deduce which build steps failed, and iterate over fixing loops for them.

5. Install and configure the Claude Code agent CLI tool:

6. Prepare the agent actions log file and replicate its content on stdout (more about this below):

7. Set the prompt, the soul of this integration (we are skipping the details here because they deserve their own section in the post: The prompt), along with the repository CLAUDE.md file.

8. The original motivation for this feature is to help us fix Pull Requests bumping dependency versions. Pull Requests created by the Renovate version management bot. Our Renovate set-up rebases its PRs branches as it detects changes in upstream main branches. We observed that this behaviour interrupted the agent’s work: More often than not, it was verifying that it fixes with a long-running Gradle task just to have the build canceled by the latest change in the main branch. To avoid this situation, we leveraged the Renovate “stop updating labels” feature so it would stop pushing changes to branches of PRs with a pre-configured label. In our case, stop-updating . This is the magical spot where you can see traditional automation shaking hands with generative AI automation to reach a common goal. Our integration is telling Renovate: “Hey, I am in charge now”.

9. With the next step, this integration is going to start pushing AI-generated commits to a branch candidate, merging a branch that is likely set to be auto-merged upon successful builds. Our team has a core principle: Never commit AI work without human supervision. Therefore, the script makes sure that GitHub PR auto-merge is disabled:

10. Finally! The agent is invoked with the configurations needed, and the prompt is prepared from the previous steps. The --allowedTools parameter deterministically constrains which commands and actions Claude can use and take, respectively. This is key for safety and was set as part of the configuration generated in step (5). As we’ll see in the prompt description and analysis, we ask Claude to explain its actions as they happen, appending them to the file created in step (6), claude-actions.log . This is crucial for real-time monitoring and post-finalization reporting.

11. When the agent finishes its work, some post-processing steps are taken:

a. Upload the actions log.

b. Determine the script return code depending on whether it has found a solution to the broken build.

c. Add success report labels to the pull request. 

One important point about this flow is that the build pipeline is set up to be restarted when new commits are pushed. So, after the agent pushes its changes, the pipeline starts over, and the agent kicks in again only and only if the new iteration fails.Commits control the pipeline flow, pipeline build steps control the invocation of the agent, and this latter commits fixes when it can find them. As a result, closing the circle of a user experience that can be described as human-supervised AI autonomous contribution.

This approach to AI agent-based coding is to editors with AI batteries (VSCode+Copilot, Cursor, Windsurf…) what a real autonomous car is to a car with cruise control and lane-keep assist.

“This isn't about replacing our work with generated code, nor is it about having an AI buddy making suggestions beside us. Instead, it's about an AI buddy contributing to our codebase through GitHub, offering code change suggestions in an experience akin to open-source collaboration online.”

The prompt

Though it comes with the disadvantages of undetermined behaviour and ambiguity, the big win behind generative AI is that the instructions are self-explanatory. This is the prompt we are currently using (italic black font) with annotations adding context for this post (regular, magenta):

The build is failing, you might find a Gradle log to analyze under /tmp/previous_step_artifacts, but if not, you will have to run the build commands to evaluate what is failing. The reader might recall that on step (4) of the integration script we fetched the Gradle logs from Buildkite so Claude could analyze them. This is where we tell it to look at them and take action.

These commands are './gradlew "--max-workers=$MAX_WORKERS" --console=plain publishForPlatform' and

'./gradlew "--max-workers=$MAX_WORKERS" --console=plain publishPlatformIndependent' but you don't need to run them as a first step if the log files under /tmp/previous_step_artifacts exist and contain the necessary information to analyze the failure. …to fall back to building everything from scratch in those cases where those logs are not available.

In any case, you must find which Gradle subtasks are failing and fix them so that the build succeeds. This gives Claude its goal: It must make sure that the build succeeds, applying whatever changes are necessary to the source (always obeying the constraints set below) code and iterating on the execution of global or local subtasks.

Please:

- Analyze their output and apply the necessary fixes to make them succeed.

This integration is our AI contributor buddy, our team treats it as a new hire. A capable one which still needs to learn “our ways”. The code style, preferred techniques, pitfalls to avoid, etc… In a nutshell, what we learn as we contribute is encoded in an ever evolving file of recommendations in the Cloud repository: CLAUDE.MD.Claude code looks for this file by default but we wanted to make it explicit that it must follow its recommendations:

- Follow the recommendations under the CLAUDE.md file within the working directory.

- If concrete Gradle subtasks are failing, iterate over them before attempting execution of the global ones. Boom! Just like this, you can follow the agent's “thinking” and actions in real-time from Buildkite. This is how the contents appended to claude-actions.log are generated.

- Log each action you take in the file /tmp/claude-actions.log as they happen in realtime. Each entry should have a prefix with the current timestamp in the format "Claude Action [YYYY-MM-DD HH:MM:SS]: " and a description of the action taken. Run commands and their outputs should be logged too.

- Log in the same file, using the same format, the plans you are going to follow and their outcomes as you finish them.

- Always include "--max-workers=$MAX_WORKERS" option in invocations to ./gradlew command.Used by the post-processing step in the integration script, step (11):

- If you get the build fixed, please add a last line to /tmp/claude-actions.log with the string "SUCCESSFUL FIX". Otherwise, add "FAILED FIX" at the end of the file.

Next, the instructions telling Claude to commit its fix changes, only they are successful: - If you succeed and added the "SUCCESSFUL FIX" line, please commit the changes that fixed the problems and push to the source branch in the same Git repository. The branch name is given by the BUILDKITE_BRANCH environment variable. Your commit messages should be prefixed with the "Claude fix: " string.

- To push the changes, you must use Github token authentication like in the following example git push https://token:$GITHUB_TOKEN@github.com/elastic/cloud.git HEAD:<BRANCH_NAME> taking into account that the token is stored in the GITHUB_TOKEN environment variable.

- Separate each fix into a different commit, so that the history is clear and understandable.

- If Git pushes fail due authentication issues, retry again after 1 minute. If still failing, then after 5 minutes and a last attempt after 10 minutes.

- Before pushing to Git: If and only if there are changes to push because the fix was successful, add the label "claude-fix-success" to the PR.

- Do not change what is not strictly necessary to fix the build. Humans tend to be indolent, AI even more. We had to introduce this last point as we found that Claude tended to fix the problems of a version bump by, well, … removing the version bump:

- NEVER downgrade versions of dependencies as declared in the version catalogs of elastic/cloud master branches such as gradle/libs.versions.toml, prefer failure to version downgrades relative to the master branch.

Results and lessons learnt

During its first month of operation, and limited to only 45% of dependencies, Self Healing PRs became one of our Cloud GitHub repository’s top contributors. The plugin successfully fixed a total of 24 initially broken PRs, with the Claude author making 22 commits between July 22nd and August 22nd.

Our estimates indicate that, during this period, its contributions saved 20 days of active development work for our team. That’s a considerable amount of reduced toil, repetitive low-value work for engineering that is key for the codebase organism but invisible for its performance metrics.

Even when it fails to completely fix problems, it nudges things forward, hinting at the way to the solution or trimming the search space.

We have also learnt that tuning AI agents' behaviour marks the difference between failure and success. Teaching it the team’s skills through contributions to CLAUDE.md repository files made it stop following undesired coding practices, but, more importantly, made it diligent. We have learnt that there is nothing as lazy as an uneducated coding agent. This teaching process never ends, but each added hint and rule translates into a saved day of work.

“Trust but verify”

I started this post highlighting the differences between traditional and generative AI automation. Underterminism in the latter group means that you can never blindly trust the changes proposed by this automation. Reviews are imperative, and a critical attitude towards the proposed changes is extremely important for the health of this axolotl-like code organism. The alternative is exposing ourselves to devilish incidents and bugs behind seemingly perfect code with alien ways of being wrong.

Conclusion

We have seen how CI and AI can work together to satisfy the ever-expanding demands of infosec quality standards.

This area of intersection between tools and needs made us design an application that behaves pretty much like a human, adding contributions in a collaborative code repository, and that can easily control traditional automation, avoiding bot tug-of-wars.

This is just the beginning. With this tool at our hand, we are starting to explore alternative direct applications.

For example, by enabling it on all pull requests, we can just open Pull Requests with incomplete changes, leaving behind less creative tasks such as API specs regeneration or linting; expecting the integration to add the necessary commits to finish these steps that are as important as boring for software engineers.

We have observed with awe how it has worked around transient problems in ancillary build services, filling the gaps when necessary with proactive execution of approved tools.

We expect this to go well beyond helping with automatic updates.

Who knows? Even to the extreme of reverse axolotl PRs: Humans writing interfaces and their unit tests, and letting Self Healing PRs come up with the rest.

The success of this pilot has traced a plan where we activate the integration for all Renovate PRs in the Cloud repository and possibly expand to all pull requests regardless of their origin.