Ask Elastic Agent Builder why it's slow: Natural-language trace analysis

Agent Builder is available now GA. Get started with an Elastic Cloud Trial, and check out the documentation for Agent Builder here.

Ask Elastic Agent Builder how many tokens your agents burned today, and it writes the Elasticsearch Query Language (ES|QL) and runs it against your OpenTelemetry (OTel) trace data. Then it answers in the chat UI. The same holds for your other agent performance questions: Which tool fails most often? Which conversation turns are slowest? What have users actually been asking? Elastic Agent Builder is designed for this purpose. It allows users to ship agents grounded in your data in minutes. 

If tracing is on, the agent-builder-traces skill is already loaded on every agent in your space and there’s nothing to install. The first post in our series covered enabling OTel tracing and the out-of-the-box (OOTB) dashboards, along with threshold alerts. This post is about asking.

What is the agent-builder-traces skill?

agent-builder-traces is a built-in Agent Builder skill. It takes a natural-language question, generates an ES|QL query from it, executes that query against your trace index, and returns a plain-language summary. The index it targets is traces-agent_builder.otel-<space-id>, where <space-id> is the Kibana space that you’re working in. Using the exact space-scoped index pattern, rather than a wildcard, keeps data from other spaces out of your results.

Under the hood, the skill uses a single inline tool: agent-builder-traces.generate_esql. Instead of calling this tool directly, you ask a question and the agent calls the tool on your behalf, passing your question as the prompt for ES|QL generation. The tool resolves the current space's trace index, builds a query using the default model, executes it against Elasticsearch, and returns the result.

Which privacy settings control what your traces capture?

Several fields containing sensitive information are off by default. You can enable them in Gen AI Settings (Stack Management > AI Assistants), under the "Agent Builder Traces" section. Expand the Advanced privacy settings to find them.

Why is my trace analysis query returning empty rows?

If the skill returns empty rows or tells you that a field is unavailable, check two things: whether the relevant setting is enabled in your configuration, and whether your query time window overlaps with any recorded spans. The skill reports exactly what the query returned; it doesn’t fabricate content when fields are empty..

What agent performance questions can you ask?

Examples of questions that you can ask include:

How many tokens have my agents used, by model?

Ask: How many input and output tokens have my agents used in the last 24 hours, broken down by model?

ES|QL: 

FROM traces-agent_builder.otel-<space-id>
| WHERE span.name LIKE "chat *" AND @timestamp >= NOW() - 24 hours
| STATS
    input_tokens  = SUM(TO_LONG(attributes.gen_ai.usage.input_tokens)),
    output_tokens = SUM(TO_LONG(attributes.gen_ai.usage.output_tokens))
  BY attributes.gen_ai.request.model
| SORT input_tokens DESC

The TO_LONG() cast around each token field is required. These fields can surface as mixed integer and long types across index generations, and ES|QL’s SUM() needs an explicit numeric conversion before it can aggregate them. If you write your own queries against this index and hit unexpected type errors, this is usually why.

Which tools are failing most often?

Ask: What is the error rate for each tool over the last 7 days?

ES|QL: 

FROM traces-agent_builder.otel-default
| WHERE @timestamp >= NOW() - 7 days
| WHERE span.name LIKE "execute_tool *"
| STATS total_calls = COUNT(*), error_count = COUNT(*) WHERE status.code == "Error" BY tool_name = attributes.gen_ai.tool.name
| EVAL error_rate_pct = ROUND(error_count * 100.0 / total_calls, 2)
| SORT error_rate_pct DESC
| LIMIT 100

The tool error rate query aggregates across spans named ‘execute_tool’ and compares status.code == "Error" counts to total counts per tool name. The result shows which tools are failing most often. 

Which conversation turns are slowest?

Ask: Show me the 10 slowest conversation turns in the last hour.

ES|QL: 

FROM traces-agent_builder.otel-default
| WHERE @timestamp >= NOW() - 1 hour
| WHERE span.name LIKE "invoke_agent *" AND attributes.elastic.inference.span.kind == "CHAIN"
| EVAL duration_seconds = duration / 1000000000.0
| KEEP attributes.gen_ai.conversation.id, attributes.gen_ai.agent.id, @timestamp, duration_seconds
| SORT duration_seconds DESC
| LIMIT 10

The skill queries spans where span.name LIKE "invoke_agent *" and attributes.elastic.inference.span.kind == "CHAIN". These correspond to individual conversation turns, from the moment a user sends a message to when the agent returns a response. Durations are stored in nanoseconds, so the skill converts to seconds before sorting.

What have users been asking my agents?

Ask: What questions have users been asking in the last 30 minutes?

ES|QL: 

FROM traces-agent_builder.otel-default
| WHERE @timestamp >= NOW() - 30 min
| WHERE span.name LIKE "chat *"
| WHERE attributes.gen_ai.input.messages IS NOT NULL
| SORT @timestamp DESC
| KEEP @timestamp, attributes.gen_ai.conversation.id, attributes.gen_ai.input.messages
| LIMIT 100

The recent-prompts query returns data only when agentBuilder:tracing:includeUserPrompts is set to true. When the setting is off, the attributes.gen_ai.input.messages field is empty and the skill will suggest that you check your Include User Prompts privacy setting.

When should you use Discover or Kibana Lens instead?

The skill targets one index pattern: traces-agent_builder.otel-<space-id>. Two scenarios fall outside that boundary.

Can the skill query my own application indices?

If you want to run ad hoc questions against data that you've indexed yourself, use a general data exploration skill or write ES|QL directly in Discover. The agent-builder-traces skill won’t query outside the Agent Builder traces index.

Can the skill build or edit dashboards?

The skill focuses on ad hoc queries and generating summary text. If your goal is to build a permanent visualization from your trace data, use Lens or the dashboard-management skill instead. We covered that specific process in depth in the first post in our series.

The skill sits alongside two other ways of watching your agents.

How to build evaluation pipelines from agent trace data

Spans in traces-agent_builder.otel-<space-id> capture real execution data from every conversation turn and LLM call running in your space. That record is the raw material for evaluation pipelines: automated checks on response quality and latency budgets per agent type, along with regression detection when you update a system prompt. The next post in our series covers how to build those eval loops from trace data that Agent Builder is already collecting.