Nine ES|QL queries that show where your Foundry agent’s tokens go
The spans your Foundry agent already sends to Agent 365 can go to Elastic as well, where one Kibana dashboard shows cost by model, which tool calls run slow, which spans fail, and who is spending the most.
Build and observe AI agents where your data already lives. Begin with Elastic Agent Builder. You can also start a free cloud trial or try Elastic on your local machine today.
A Foundry hosted agent’s OpenTelemetry spans already tell you what it costs. Nine ES|QL queries and an importable Kibana dashboard put token cost by model, tool latency, failed spans and spend per caller on one screen. The same spans feed Agent 365, which uses them for a different job: establishing who the agent is and what it may do.
In the sample data, one tool sits at a 10-second p95 and drags the agent’s own p95 to 13.1 seconds, and 79 spans failed across five error types. There is a sample agent container, a generator that produces 605 runs and about 3,000 spans, and an ES|QL rule that fires when the agent calls a tool it has never called before.
This post picks up from AI agent observability for Microsoft Foundry: two env vars, no collector, which gets Foundry spans into Elastic in the first place.
Prerequisites: Get Foundry agent spans into Elastic
This post picks up where AI agent observability for Microsoft Foundry: two env vars, no collector leaves off. In that post, we cover getting a Foundry agent's spans into the Elastic managed OTLP endpoint and reading a single run as an application performance monitoring (APM) waterfall. You need the endpoint URL and API key from it before anything in the following sections will work. Spans land in the traces-generic.otel-default data stream.
The waterfall is the right view for a single run. Questions, like Which model costs the most this week? or Which caller is driving the spend?, or whether a particular tool is getting slower, need aggregation across every run, and that’s what a dashboard is for. In this post, we build that dashboard and add the Agent 365 context that the earlier post didn’t cover.
Where do Microsoft Foundry agent traces go?
Foundry agents produce traces from three sources and can send them to several destinations, including Agent 365, and only some of them reach Elastic:
Foundry generates server-side traces for prompt agents and hosted agents, as well as workflows. Foundry stores them in the Application Insights resource connected to your project, and by default, nothing sends them to Elastic.
Client-side traces come from the application that calls an agent. They record your calling code and its requests to Foundry, and they stop at the edge of the agent.
In-container traces come from a hosted agent, where the container runs your own orchestration code. Instrumenting that code records every model call and tool call that the agent makes. The container's telemetry pipeline can send these spans to Elastic over OTLP and, when Agent 365 export is on, to Agent 365, as well. These are separate exporters that share one pipeline, not a single generic OTLP exporter. We use this path in the rest of this post.
Agent 365 is a destination rather than a source. It’s a Microsoft control plane for agent identity, governance, and security. It ingests spans for the agent inventory in the Microsoft 365 admin center and for Microsoft Purview, as well as for Microsoft Defender. Agent 365 isn’t a general-purpose store of complete raw spans. It supports queries over processed agent activity records, through surfaces such as Microsoft Defender advanced hunting.
What Agent 365 keeps and what Elastic keeps
The two destinations do different jobs, and the same span feeds both:
| Agent 365 | Elastic |
|---|---|---|
Questions that it answers | Which agents exist? Who owns them? What are they allowed to do? | How are the agents behaving in terms of cost, latency, failures, and usage? |
Strength | Identity, governance, compliance, and security views in Microsoft 365, Purview, and Defender. | Ad hoc ES|QL over every span that the agent emits, with no limit on span types. You control how long the spans are kept and who can read them, down to field-level security on prompt content. Dashboards, alert rules, and detections, such as the new-tool query, below, all run on the same data. You can also correlate agent traces with the logs, metrics, and application performance monitoring (APM) data from the rest of your stack. |
Spans it keeps | Only | Everything that the agent emits, including span types that Agent 365 ignores, such as |
The two destinations keep different spans, but the identity fields they share are what let you line them up. Filter in Elastic on gen_ai.agent.id and microsoft.tenant.id to isolate one agent instance in one tenant, and then use gen_ai.conversation.id and the trace ID to follow a single run from the admin center into Elastic.
Send Foundry hosted agent traces to Elastic and Agent 365
A Foundry hosted agent runs inside the Foundry agent server host, and that host already does most of the work. In the versions I tested (azure-ai-agentserver-core 2.2.0 and microsoft-opentelemetry 1.3.9), it configures the Microsoft OTel distro for you. It exports to any OTLP endpoint named in OTEL_EXPORTER_OTLP_ENDPOINT, over HTTP/protobuf by default. It stamps the agent ID, blueprint ID, tenant ID, and session ID on every span from the FOUNDRY_AGENT_* environment variables. It also turns on Agent 365 export when Foundry enables it for the agent. Foundry controls Agent 365 export through the account-level a365LoggingEnabled setting. When it’s true, the supported hosting libraries turn export on automatically. So the container doesn’t call use_microsoft_opentelemetry() itself.
The container does describe each step of the run with the distro's scopes. InvokeAgentScope, InferenceScope, and ExecuteToolScope emit the invoke_agent, chat, and execute_tool spans in the shape that Agent 365 accepts and the dashboard queries read. This agent handles one invocation endpoint. Each request runs inside a root invoke_agent span and calls one tool. It also makes one model call through the Foundry client. The code below describes the agent's identity once in AgentDetails and then wraps the tool call and the model call in scopes that emit the execute_tool and chat spans. Content capture stays off unless you set CAPTURE_MESSAGE_CONTENT.
import asyncio
import os
import uuid
from urllib.parse import urlparse
# The scopes below do nothing unless observability is switched on. The agent server host
# only does that itself when Foundry enables Agent 365 export, so opt in here to get spans
# in Elastic either way.
os.environ.setdefault("ENABLE_OBSERVABILITY", "true")
from azure.ai.agentserver.invocations import InvocationAgentServerHost
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
from microsoft.opentelemetry.a365.core import (
AgentDetails,
CallerDetails,
Channel,
ExecuteToolScope,
InferenceCallDetails,
InferenceOperationType,
InferenceScope,
InvokeAgentScope,
InvokeAgentScopeDetails,
Request,
ServiceEndpoint,
ToolCallDetails,
ToolType,
UserDetails,
)
from starlette.responses import JSONResponse
# The agent server host turns on OTLP export when OTEL_EXPORTER_OTLP_ENDPOINT is set.
app = InvocationAgentServerHost()
CAPTURE_CONTENT = os.environ.get("CAPTURE_MESSAGE_CONTENT", "false").lower() == "true"
MODEL = os.environ["FOUNDRY_MODEL_NAME"]
PROJECT_HOST = ServiceEndpoint(hostname=urlparse(os.environ["FOUNDRY_PROJECT_ENDPOINT"]).hostname)
# The instance ID goes in agent_id. Using the blueprint ID there makes Agent 365 reject
# the export with a 403 agent ID mismatch.
AGENT = AgentDetails(
agent_id=os.environ["FOUNDRY_AGENT_INSTANCE_CLIENT_ID"],
agent_name=os.environ.get("FOUNDRY_AGENT_NAME", "support-agent"),
agent_blueprint_id=os.environ["FOUNDRY_AGENT_BLUEPRINT_CLIENT_ID"],
tenant_id=os.environ["FOUNDRY_AGENT_TENANT_ID"],
)
project = AIProjectClient(
endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
credential=DefaultAzureCredential(),
)
openai = project.get_openai_client()
def lookup_order(request: Request, order_id: str) -> str:
"""Stand-in for a real tool call, such as a database or MCP lookup."""
arguments = {"order_id": order_id}
details = ToolCallDetails(
tool_name="lookup_order",
arguments=arguments if CAPTURE_CONTENT else None,
tool_call_id=str(uuid.uuid4()),
tool_type=ToolType.FUNCTION.value,
# Replace with the host your real tool calls.
endpoint=ServiceEndpoint(hostname="orders.example.com"),
)
with ExecuteToolScope.start(request=request, details=details, agent_details=AGENT) as scope:
result = f"Order {order_id} shipped and arrives Thursday."
if CAPTURE_CONTENT:
scope.record_response(result)
return result
def ask_model(request: Request, prompt: str) -> str:
"""One model call, recorded as a chat span with token counts."""
details = InferenceCallDetails(
operationName=InferenceOperationType.CHAT,
model=MODEL,
providerName="microsoft.foundry",
endpoint=PROJECT_HOST,
)
with InferenceScope.start(request=request, details=details, agent_details=AGENT) as scope:
response = openai.responses.create(model=MODEL, input=prompt)
if CAPTURE_CONTENT:
scope.record_input_messages([prompt])
scope.record_output_messages([response.output_text])
scope.record_input_tokens(response.usage.input_tokens)
scope.record_output_tokens(response.usage.output_tokens)
return response.output_text
@app.invoke_handler
async def handle(request):
body = await request.json()
question = body.get("question", "")
# Content=None keeps the question out of the span. The conversation ID groups the runs
# in one thread. The host adds Foundry's session ID to every span for you.
agent_request = Request(
content=question if CAPTURE_CONTENT else None,
conversation_id=body.get("conversation_id"),
channel=Channel(name=body.get("channel", "web")),
)
caller = CallerDetails(
user_details=UserDetails(user_id=body.get("user_id"), user_client_ip=request.client.host)
)
# One root span per request keeps the model and tool calls in a single trace.
with InvokeAgentScope.start(
request=agent_request,
scope_details=InvokeAgentScopeDetails(endpoint=PROJECT_HOST),
agent_details=AGENT,
caller_details=caller,
) as scope:
order_status = lookup_order(agent_request, body.get("order_id", "unknown"))
# The OpenAI client is synchronous, so run it off the event loop.
answer = await asyncio.to_thread(
ask_model,
agent_request,
"You are a support agent. Answer in two sentences.\n"
f"Order status: {order_status}\n"
f"Customer question: {question}",
)
if CAPTURE_CONTENT:
scope.record_output_messages([answer])
return JSONResponse({"answer": answer})
if __name__ == "__main__":
app.run()Install the packages in your container image, and set the OTLP environment variables on the hosted agent. Foundry injects the FOUNDRY_* variables into every hosted agent container at startup. They’re reserved and read-only, so your code should read them but should never set or override them:
pip install azure-ai-agentserver-invocations azure-ai-projects azure-identity \
microsoft-opentelemetry
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<your-motlp-endpoint>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey <your-api-key>"Deploy the container as a hosted agent by following the Foundry hosted agents documentation, and then send it a few requests with an order_id, a question, a user_id, and a conversation_id in the JSON body.
Details that quietly stop Foundry agent spans arriving
A few details in that file matter more than they look:
Switch observability on. The scopes do nothing unless
ENABLE_OBSERVABILITYistrue. The host sets that up itself only when Foundry enables Agent 365 export, so the file sets it to get spans in Elastic either way. Without it, the agent runs normally and emits noinvoke_agent,chat, orexecute_toolspans.Use the instance ID, not the blueprint ID.
gen_ai.agent.idis the agent instance's app ID. A blueprint ID there makes Agent 365 reject the export with a 403 agent ID mismatch. Keep both, as the sample does: the instance ID onAgentDetails.agent_idand the blueprint ID onagent_blueprint_id, so Elastic can roll instances up by blueprint.Leave
gen_ai.agent.typealone. Microsoft reserves values such asFoundryfor its own use, and Agent 365 classifies Microsoft Entra–registered agents itself.The host sets the service name.
service.namecomes fromFOUNDRY_AGENT_NAME, whichFoundryinjects, not fromOTEL_SERVICE_NAME. Without it, spans arrive as azure.ai.agentserver.The operation name is
Chat. The distro writesgen_ai.operation.namefor model calls asChat, with a capital C, and Elastic compares keyword values case-sensitively. Every query below wraps the field inTO_LOWER()so it matches both spellings.Token counts are numbers.
gen_ai.usage.input_tokensandgen_ai.usage.output_tokensarrive over OTLP as integers, so the queries can add them without conversion. Agent 365's own wire format sends everything as strings, but the distro handles that on its side.
If you build on Microsoft Agent Framework, Semantic Kernel, LangChain, LangGraph, or the OpenAI Agents SDK, the distro can instrument them for you with the instrumentation_options argument, so you can drop the manual scopes. Auto-instrumentation records standard OTel attributes only, so Agent 365 still needs the Microsoft identity attributes from BaggageBuilder or from the FOUNDRY_AGENT_* variables that the host reads.
The host also sends metrics over the same OTLP settings. This post builds panels on traces only. Check the metric names in your own data before you build panels on them.
To confirm that the data arrived, open Discover in ES|QL mode and run:
FROM traces-generic.otel-*
| WHERE attributes.gen_ai.operation.name IS NOT NULL
| STATS spans = COUNT(*) BY operation = attributes.gen_ai.operation.nameYou should see rows for invoke_agent and Chat, along with execute_tool. If a row is missing, check ENABLE_OBSERVABILITY. If everything is missing, check that the API key has write privileges on the OTLP endpoint.
Decide what prompt and response content Elastic receives
The scopes can record the prompt and the response, in addition to each tool call's arguments and results. Content capture is off in the sample, and none of the dashboard queries need it. But Agent 365 only consumes spans that carry message content and tool arguments and results, and it filters out ineligible spans that lack them. You can leave Agent 365 export on with content off, but Agent 365 won’t use those spans. If you turn content on so that it can, the same spans then carry that content to Elastic.
You have three ways to handle it:
Option | Agent 365 | Elastic | What it takes |
Keep content off | Will not consume the spans | Gets everything the dashboard needs | Nothing |
Content on, restricted in Elastic | Consumes the spans |
| Field-level security |
Content on, stripped before Elastic | Consumes the spans | No content stored | An OpenTelemetry Collector with an attributes processor between the agent and Elastic |
The sample reads the CAPTURE_MESSAGE_CONTENT environment variable and defaults to off. Make this choice with your privacy owners before the first production deploy, not after.
Authentication and licensing for Agent 365 export
The Elastic side needs one thing: an API key with write access to the managed OTLP (mOTLP) endpoint. Keep it out of the container image. Store it in Azure Key Vault, and supply it to the hosted agent as a secret.
The Agent 365 side has more moving parts, all documented on Microsoft Learn:
At least one user in the tenant needs an assigned Microsoft 365 E7 or Microsoft Agent 365 license. Without one, the HTTP export returns 200 OK, but the response reports the spans as rejected, with the reason tenant_not_licensed. A 403 Forbidden means something else; that is, a required application role or delegated scope is missing, or the agent ID in the URL doesn’t match the authenticated identity.
A registered Agent 365 agent instance exports on the service-to-service route with no extra permission or admin consent. The delegated route and unregistered identities (such as a standard app registration) need the
Agent365.Observability.OtelWritepermission with admin consent.For service-to-service export, any identity that isn’t registered as an Agent 365 agent instance, whether an app registration or a managed identity, or even an agent ID, needs the Agent365.Observability.OtelWrite app role and admin consent.
What are the Agent 365 observability attributes used for?
Agent 365 makes more attributes mandatory than a plain generative AI (GenAI) trace carries. They also make useful dashboard dimensions, because they say who, where, and in which conversation:
Attribute | Use in Elastic |
|---|---|
| Filter or split the dashboard by tenant. |
| The agent instance's app ID. Join key to Agent 365 admin records, and the grouping key for the new-tool detection below. |
| Filter the dashboard by agent. |
| Roll up instances by blueprint. |
| Group runs into conversations. |
| The sessions and channel panels. Channel values are short lowercase tokens, such as |
| The Entra object ID of the caller. Token spend and activity per caller. |
| Compare function tools with Model Context Protocol (MCP) servers. Add |
Messages, tool arguments, tool results | Content. (See the decision above.) |
Microsoft's attribute reference lists every attribute and notes which are mandatory per span type. The sample sets the identity, conversation, channel, caller, tool type, tool call ID, model, and provider attributes. Agent 365 also expects server and client addresses, in addition to message content, and you should check the full list against what your own agent emits before you rely on the Agent 365 side.
Nine ES|QL queries for Foundry agent cost, latency and failures
Each query below feeds one dashboard panel. The field names come from the OTel GenAI semantic conventions and the Agent 365 attributes. Microsoft, in collaboration with Outshift by Cisco, extended the conventions for multi-agent systems with span types, such as invoke_agent, plan, invoke_workflow, execute_tool, and memory operations. Microsoft's agent tracing overview describes them and notes that the OTel GenAI conventions still have Development status upstream and might change. Pin your SDK versions, and check field names against a live trace if a query comes back empty.
Token usage by model over time
Use a stacked area chart. The query below adds up input and output tokens for each hour and splits the total by model, which produces the chart that follows it. This is the panel that makes a runaway agent obvious. A loop that retries the same tool call shows up as a spike in one model’s band.
FROM traces-generic.otel-*
| WHERE TO_LOWER(attributes.gen_ai.operation.name) == "chat"
| STATS tokens = SUM(attributes.gen_ai.usage.input_tokens + attributes.gen_ai.usage.output_tokens)
BY hour = BUCKET(@timestamp, 1 hour), model = attributes.gen_ai.request.model
| SORT hourEstimated token cost by model
Use a horizontal bar chart. The query multiplies token counts by a per-million token rate. The rates below are placeholders, so replace them with the prices on your own Azure agreement. Any model without a matching CASE row returns a null cost, so add a row for every deployment that you run. If your Foundry deployment names differ from the underlying model names, group by attributes.gen_ai.response.model instead, which carries the model that actually answered.
FROM traces-generic.otel-*
| WHERE TO_LOWER(attributes.gen_ai.operation.name) == "chat"
| STATS input_tokens = SUM(attributes.gen_ai.usage.input_tokens),
output_tokens = SUM(attributes.gen_ai.usage.output_tokens)
BY model = attributes.gen_ai.request.model
| EVAL cost_usd = CASE(
model == "gpt-4o", input_tokens / 1000000.0 * 2.50 + output_tokens / 1000000.0 * 10.00,
model == "gpt-4o-mini", input_tokens / 1000000.0 * 0.15 + output_tokens / 1000000.0 * 0.60,
model == "gpt-4.1", input_tokens / 1000000.0 * 2.00 + output_tokens / 1000000.0 * 8.00,
NULL)
| KEEP model, cost_usd
| SORT cost_usd DESC NULLS LASTThe most token-heavy agent runs
Use a data table. Summing the chat spans per trace ID gives you the cost of a whole agent run instead of a single model call, and the top row is usually the first one worth opening. Failed model calls carry no token counts, so NULLS LAST prevents runs that are made up only of failures from crowding the top of the table.
FROM traces-generic.otel-*
| WHERE TO_LOWER(attributes.gen_ai.operation.name) == "chat"
| STATS input_tokens = SUM(attributes.gen_ai.usage.input_tokens),
output_tokens = SUM(attributes.gen_ai.usage.output_tokens),
chat_calls = COUNT(*)
BY trace.id
| SORT input_tokens DESC NULLS LAST
| LIMIT 10Latency by operation type
Use a grouped bar chart. Comparing p50 to p95 for each operation type shows whether slowness lives in the model calls or the tool calls, or possibly in the orchestration around them. OTel stores span duration in nanoseconds, so the query converts to milliseconds.
FROM traces-generic.otel-*
| WHERE TO_LOWER(attributes.gen_ai.operation.name) IN ("invoke_agent", "chat", "execute_tool")
| EVAL duration_ms = duration / 1000000.0
| STATS p50 = PERCENTILE(duration_ms, 50),
p95 = PERCENTILE(duration_ms, 95)
BY operation = TO_LOWER(attributes.gen_ai.operation.name)Slowest tool calls by p95
Use a horizontal bar chart. Ranking by p95 instead of average keeps a tool that’s usually fast, but occasionally takes 10 seconds, from hiding behind a healthy mean.
FROM traces-generic.otel-*
| WHERE TO_LOWER(attributes.gen_ai.operation.name) == "execute_tool"
| EVAL duration_ms = duration / 1000000.0
| STATS p95_ms = PERCENTILE(duration_ms, 95), calls = COUNT(*)
BY tool = attributes.gen_ai.tool.name
| SORT p95_ms DESC
| LIMIT 10Failed spans by operation and error type
Use a data table. Grouping by operation and error type separates provider rate limits and content filter blocks from tool timeouts, which have different owners and different fixes.
FROM traces-generic.otel-*
| WHERE attributes.gen_ai.operation.name IS NOT NULL
AND status.code == "Error"
| STATS failures = COUNT(*)
BY operation = TO_LOWER(attributes.gen_ai.operation.name), error_type = attributes.error.type
| SORT failures DESCAgent runs by channel: Teams, Outlook, and web
Use a horizontal bar chart. Agent 365 records the surface each run came from, such as Teams, Outlook, or the web. Comparing channels shows where the agent is actually used and which channel to look at first when something breaks.
FROM traces-generic.otel-*
| WHERE TO_LOWER(attributes.gen_ai.operation.name) == "invoke_agent"
| STATS runs = COUNT(*) BY channel = attributes.microsoft.channel.name
| SORT runs DESCToken spend by caller
Use a data table. Tokens live on the chat spans, and the caller lives on the root invoke_agent span, so the query first totals tokens per trace and picks up the caller from the same trace. It then totals again per caller. The caller is the Entra object ID in user.id. Resolve it to a name in Entra, or enrich it in Elastic.
FROM traces-generic.otel-*
| WHERE TO_LOWER(attributes.gen_ai.operation.name) IN ("invoke_agent", "chat")
| STATS run_tokens = SUM(attributes.gen_ai.usage.input_tokens + attributes.gen_ai.usage.output_tokens),
caller = MAX(attributes.user.id)
BY trace.id
| WHERE caller IS NOT NULL
| STATS runs = COUNT(*), tokens = SUM(run_tokens) BY caller
| SORT tokens DESC NULLS LAST
| LIMIT 10Busiest sessions and conversations
Use a data table. A session is one visit, and a conversation is a thread inside it. Sessions with many runs and several conversations are the ones to open when you want to see how people actually use the agent.
FROM traces-generic.otel-*
| WHERE TO_LOWER(attributes.gen_ai.operation.name) == "invoke_agent"
| STATS runs = COUNT(*),
conversations = COUNT_DISTINCT(attributes.gen_ai.conversation.id)
BY session = attributes.microsoft.session.id
| SORT runs DESC
| LIMIT 10Build the Foundry agent dashboard in Kibana
You can import a finished dashboard or build it panel by panel, with the following steps.
Import the finished Kibana dashboard
foundry-agent-traces-dashboard.ndjson is an export of the dashboard with all nine panels and five filters for agent, tenant, channel, conversation, and session. To bring it into your own Kibana:
Open Stack Management and then Saved Objects. On Elastic Cloud Serverless, open Project settings, Management, and then Saved objects.
Select Import, choose the
.ndjsonfile, and keep the default options.Open Dashboards, and select Foundry agent traces. The time picker starts at the last 24 hours.
The panels read traces-generic.otel-* and aren’t filtered to one agent, so they cover every agent that writes to the data stream. To narrow them down, use the Agent and Tenant filters at the top.
You can also import the file with the saved objects API:
curl -X POST "https://<your-kibana-url>/api/saved_objects/_import?overwrite=true" \
-H "Authorization: ApiKey <your-api-key>" \
-H "kbn-xsrf: true" \
-F file=@foundry-agent-traces-dashboard.ndjsonBuild the dashboard panel by panel
To manually create the dashboard:
In Kibana, open Dashboards and create a new dashboard.
Select Add panel, and then choose ES|QL.
Paste one of the queries, and run it. Kibana suggests chart types based on the result shape, so pick the one listed above for that query.
Save the panel to the dashboard, and repeat for the others.
The dashboard time picker applies to every ES|QL panel through @timestamp, so there’s nothing to add to the queries for time filtering. To filter by agent, tenant, channel, conversation, or session, add an ES|QL control for the field, as the imported dashboard does.
Note that the sample lookup_order tool returns instantly, so your own tools panel will show near-zero latency until you wrap a real call.
Generate 605 sample agent runs to fill the dashboard
If you don't have a busy agent yet, generate_sample_traces.py sends 24 hours of realistic traces over OTLP, plus a week of earlier history. It emits the same invoke_agent, Chat, and execute_tool spans and the same Agent 365 attributes as the container above, as 605 agent runs and about 3,000 spans under the service name support-agent-sample.
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<your-motlp-endpoint>"
export ELASTIC_API_KEY="<your-api-key>"
python generate_sample_traces.pyThe script covers the 24 whole hours before the current hour, so the dashboard's default time range shows all of it. Every panel in the preview above comes from this data, and each number in the next section can be reproduced by running the script and the queries.
What the sample data shows: A token spike, a slow tool, and 79 failures
The preview at the top of this post comes from the sample data above. It includes four tools (web_search, mcp_ticket_lookup, code_interpreter, file_search) where the minimal container has one, so your own tools panel will list whatever your agent calls. Kibana draws the time axis in your browser's time zone, so the clock times on your chart will differ from the preview. Four things stand out:
The token chart has a sharp spike about nine hours before the end of the window, concentrated in the
gpt-4oband, at roughly twice the usual hourly total. The token-heavy runs table shows why: four runs in that hour used between 12,060 and 41,200 input tokens each, against a few thousand for a typical run, and the top one, trace4f2a91c0, made 14 model calls. Together, those four runs account for about 110,000 tokens, which is most of the spike. Paste the top trace ID into Elastic APM to open the waterfall. If it shows the agent calling the same tool over and over, you’ve found a loop, and loops return HTTP 200 and never raise an error.The tools panel puts
web_searchat a p95 of about 10 seconds, well ahead of everything else, and the latency panel shows the cost of that:invoke_agenthas a p95 of 13.1 seconds against a p50 of 4.6. Any run that touchesweb_searchpays for it.The failed spans table splits the 79 failures into rate limits on
chat(37), tool timeouts (21), and content filter blocks (12), plus a smaller group of connection errors (6) and token limit hits (3). The rate limit failures point at a quota problem on the model deployment, and the timeouts point back at the same slow tool. The 10-second p95 onweb_searchis mostly those timeouts, so fixing them moves the number the most.The caller and channel panels say who’s behind the numbers. Of the 605 runs, 336 came through Teams, 165 through the web, and 104 through Outlook. The top caller accounts for about 253,000 tokens over 63 runs, so a handful of callers drive most of the spend. When a spike in the token chart lines up with one caller, you know whose usage to look at and whom to ask.
Catch a Foundry agent calling a tool it has never used before
An agent that suddenly calls a tool it has never called before is worth a look. It might be a new feature, or it could be a prompt injection steering the agent toward something that it shouldn’t touch. This query finds agent and tool pairs seen for the first time in the last day, against a 30-day lookback.
FROM traces-generic.otel-*
| WHERE TO_LOWER(attributes.gen_ai.operation.name) == "execute_tool" AND @timestamp >= NOW() - 30 days
| STATS first_seen = MIN(@timestamp), calls = COUNT(*)
BY agent_id = attributes.gen_ai.agent.id, tool = attributes.gen_ai.tool.name
| WHERE first_seen > NOW() - 24 hoursThe sample data includes one: export_customer_data appears in the last hours of the window, for an agent that has otherwise used only its four regular tools. To run it as a detection, open Elastic Security, create a new ES|QL rule, paste the query, and schedule it hourly. It also works as a plain Kibana alert, if you don't use Elastic Security. Expect noise after every deploy that adds a tool, so tune the lookback and suppress known tools once the rule has run for a week.
Can you get traces from Foundry prompt agents or Copilot Studio?
This dashboard approach covers Foundry hosted agents, where you own the container and can send its spans anywhere. Prompt agents run entirely inside Foundry and have no container to instrument, so their server-side traces stay in Application Insights. Microsoft Copilot Studio and Microsoft 365 agents likewise run on Microsoft's side, with no container of yours to add an exporter to. Nothing here sends any of those traces to Elastic. Bridging them might be possible; for example, by exporting Application Insights data to Azure Event Hubs and reading it with Elastic's Azure integration, but that’s a different pipeline with a different schema (and a topic for a follow-up discussion).
Production checklist: Sampling, version pinning, and content capture
Decide what message content Elastic receives before launch, not after.
Sampling changes the numbers. These panels sum the spans that arrive, so if you use tail sampling to keep only 10 percent of routine traces, token and cost totals will undercount by roughly that factor. Keep error and slow traces, as the earlier post recommends, and either send agent traces unsampled or scale the cost panel by the sampling rate.
Pin your versions. The sample was tested with azure-ai-agentserver-core 2.2.0, azure-ai-agentserver-invocations 1.2.0, azure-ai-projects 2.8.0, and microsoft-opentelemetry 1.3.9. The distro and the GenAI conventions are both moving, and the Chat casing above is the kind of detail that can change between releases. The TO_LOWER() in each query keeps the dashboard working, if it does.
Alert on runaway token usage before the invoice arrives
Start with the token-heavy runs query. Save it as an ES|QL alert rule that fires when any single trace passes a token threshold set a little above your normal p95 per run, and a looping agent notifies you the same hour instead of appearing on the monthly invoice. Then add the new-tool detection and decide what content your Elastic project should hold before Agent 365 export goes on.
Start a free trial on Elastic Cloud to get a managed OTLP endpoint, or read the mOTLP endpoint docs for the other deployment options.
More resources: AI agent observability for Microsoft Foundry | Microsoft OpenTelemetry Distro | Agent 365 observability attribute reference | Agent 365 observability concepts | Agent tracing overview | Add client-side tracing to Foundry agents | OpenTelemetry GenAI semantic conventions | ES|QL reference
How helpful was this content?



.jpg)
