Create alerts on Elastic Agent Builder trace data
Elastic Agent Builder collects execution traces into a data stream in your Elasticsearch cluster. These traces record token usage, errors, latency, and tool calls, so you can create Kibana alerting rules that notify you when something needs your attention. For example, you can alert on a conversation that uses too many tokens, total token usage that exceeds a budget, an agent error rate that spikes, or a tool that fails repeatedly.
Because traces are stored in a regular data stream, you can use the same Kibana rule types, check schedules, and connectors as any other alerting workflow.
Before you create a rule, make sure that:
- Trace collection is on. Elastic Agent Builder must be writing traces to the
traces-agent_builder.otel-<space-id>data stream in the space where you create the rule. Collection is on by default. See Collect agent traces. - You can read the trace data. The rule queries
traces-agent_builder.otel-*, so you need read access to those data streams. See Read trace data. - You can use Kibana alerting. You need privileges to create and manage rules, plus a connector to send notifications such as Slack, email, or PagerDuty. See Set up alerting.
Alert on the trace data with an Elasticsearch query rule that runs an ES|QL query on a schedule and runs an action when the query returns matches.
For a simple count-based threshold, you can use an index threshold rule instead. It cannot sum the token fields, so use the Elasticsearch query rule with ES|QL for token-based alerts.
With ES|QL, the query targets the data stream directly in its FROM command, so you do not need a data view. The alert condition lives in the query, usually in a WHERE clause that compares a value to a threshold. Query one space at a time and avoid wildcards, so you do not mix data from different spaces.
In Kibana, go to Stack Management → Rules and click Create rule.
Select the Elasticsearch query rule type, then name the rule.
For the query language, select ES|QL.
Enter your ES|QL query against the trace data stream for your space, for example
FROM traces-agent_builder.otel-<space-id>. The query defines the condition, including the threshold. See Example alerts.Set the alert grouping:
- Select a time field: the field used to filter results by the rule's time window, for example
@timestamp. - Select alert group: select Create an alert for each row to raise one alert per matching row, for example per conversation over the threshold. Select Create an alert if matches are found to raise a single alert when the query returns any rows.
- Select a time field: the field used to filter results by the rule's time window, for example
Set the time window to define how far back the query searches, for example the last hour.
Set the check interval to define how often the rule runs. Keep it smaller than the time window to avoid gaps in detection.
NoteES|QL rules do not offer the Exclude matches from previous run option. If the check interval is smaller than the time window, a row that keeps matching can alert more than once. Choose the time window, check interval, and query so that a condition alerts as often as you want.
Click Test query to confirm the query is valid. For an ES|QL query, the matching rows appear in a table.
Add an action, select a connector, then set the action frequency. See Add actions.
Click Save.
After you save the rule, it appears on the Rules page, where you can confirm that it runs on schedule and check its status.
Each example gives an ES|QL query and the rule settings to use with it. Adjust the fields, thresholds, and time windows to your environment.
These queries are starting points, not tested rules. Run each one with Test query on your own data before you rely on it. Replace default in traces-agent_builder.otel-default with your space id. The rule applies its own time window through the Time field you select, so the queries do not include a @timestamp filter. If you run a query outside a rule, make sure a time range applies, through the Kibana time picker or a @timestamp filter in the query, so it does not scan the whole data stream.
By default, Elastic Agent Builder anonymizes custom tool names, so any query that groups by tool name buckets all custom tools under execute_tool custom. Built-in tools keep their names. To group by individual custom tool names, turn on Include real tool and agent names in traces (agentBuilder:tracing:includeRealNames).
Alert when a single conversation uses more than a set number of tokens. Sum the input and output tokens on the conversation's chat spans and group by conversation.
FROM traces-agent_builder.otel-default
| WHERE `span.name` LIKE "chat *"
| STATS input_tokens = SUM(TO_LONG(attributes.gen_ai.usage.input_tokens)),
output_tokens = SUM(TO_LONG(attributes.gen_ai.usage.output_tokens))
BY attributes.gen_ai.conversation.id
| EVAL total_tokens = input_tokens + output_tokens
| WHERE total_tokens > 256000
Rule settings:
- Alert group: Create an alert for each row, so you get one alert per conversation over the threshold.
- Time window: the period to evaluate, for example the last 24 hours.
256,000 is an example token threshold, not a hard limit. Elastic Agent Builder compacts long conversations, so a conversation can pass this value without failing. By default, attributes.gen_ai.conversation.id is a stable hash, which is enough to group and count conversations. To include the real conversation ID in alerts, turn on the Include real conversation and workflow IDs privacy setting (agentBuilder:tracing:includeRealIds).
Alert when total token usage across all conversations goes over a budget for the period. This is the same sum without grouping by conversation.
FROM traces-agent_builder.otel-default
| WHERE `span.name` LIKE "chat *"
| STATS input_tokens = SUM(TO_LONG(attributes.gen_ai.usage.input_tokens)),
output_tokens = SUM(TO_LONG(attributes.gen_ai.usage.output_tokens))
| EVAL total_tokens = input_tokens + output_tokens
| WHERE total_tokens > 5000000
Rule settings:
- Alert group: Create an alert if matches are found, so you get a single alert for the period.
- Time window: the budget period, for example the last 30 days.
Set the threshold to your budget. 5,000,000 is a placeholder.
Alert when an agent's error rate goes above a threshold. Count each agent's executions and its errors, then compare the ratio.
FROM traces-agent_builder.otel-default
| WHERE `span.name` LIKE "invoke_agent *" AND attributes.elastic.inference.span.kind == "AGENT"
| STATS executions = COUNT(*), errors = COUNT(*) WHERE status.code == "Error"
BY attributes.gen_ai.agent.id
| EVAL error_rate = TO_DOUBLE(errors) / executions
| WHERE executions >= 20 AND error_rate > 0.1
Rule settings:
- Alert group: Create an alert for each row, so you get one alert per agent.
- Time window: the period to evaluate, for example the last hour.
The executions >= 20 guard avoids noisy alerts when an agent has run only a few times. error_rate > 0.1 alerts when more than 10 percent of executions fail. Like conversation IDs, attributes.gen_ai.agent.id is a stable hash by default, so custom agents group by hash unless you turn on agentBuilder:tracing:includeRealIds.
This alert counts agent executions that fail outright. An error that the agent handles and still returns as a completed response does not set status.code to Error, so it is not counted here.
Alert when a tool records more than a set number of errors. Count execute_tool spans that have an error status and group by the tool's span name.
FROM traces-agent_builder.otel-default
| WHERE `span.name` LIKE "execute_tool *" AND status.code == "Error"
| STATS failures = COUNT(*) BY `span.name`
| WHERE failures > 5
Rule settings:
- Alert group: Create an alert for each row, so you get one alert per tool.
- Time window: the period to evaluate, for example the last hour.
Each alert identifies the tool by its span name, for example execute_tool <toolId>.
status.code == "Error" on execute_tool spans is set only for parameter and schema validation errors, such as invalid arguments. Errors that a tool catches and returns as a result do not set this status, so this alert can miss some tool failures.
- Collect traces: turn on trace collection and learn about the data streams, privacy settings, and access model.
- Agent Builder traces dashboard: the prebuilt overview dashboard and the full span and attribute reference.
- Monitor usage and costs: how Elastic Agent Builder counts tokens and how usage maps to cost.
- Create and manage rules: manage, snooze, and troubleshoot Kibana alerting rules.
- Elasticsearch query rule: full reference for the rule type used on this page.