AI agent observability for Microsoft Foundry in Elastic

Instrument your Microsoft Foundry agent with the OTel SDK, set two environment variables, and every LLM call, tool execution and agent handoff lands in Kibana as a queryable trace with token counts on every span. No OTel Collector sits in between, and nothing gets rewritten into a proprietary schema on the way in. Normal APM assumes a successful response is a correct one. AI agent observability can't, because an agent run can return HTTP 200 and still answer wrong, having burned 40,000 tokens to get there, so what you need is the whole decision tree. Code below for Agent Framework, LangGraph, and custom Node.js containers.

Why does AI agent observability need a different model?

Standard application monitoring answers: "Did this request succeed? How long did the database query take?" These questions work because traditional software is deterministic: same input, same output, errors have error codes.

AI agents break that model. A single user request to a Foundry hosted agent might trigger ten LLM calls, five tool executions, two file searches, and an MCP server call, each with its own latency and potential failure mode. The agent can return HTTP 200 and still produce a wrong answer. It can silently loop on the same tool call until it hits a token limit. It can hand off a task to a specialist agent and lose context in transit, and none of that shows up in your error rate or P99 latency.

The questions you actually need to answer in production are different:

Why did this run consume 40,000 tokens when the average is 3,000?
Which tool call is responsible for the long tail?
When the orchestrator handed off to the search agent, did the trace context survive? For those, you need hierarchical trace data that maps the agent's decision tree, not just its I/O boundary.

Foundry Agent Service has strong built-in observability. For Prompt agents, server-side tracing is zero-config: connect an Application Insights resource to your project and Foundry automatically captures inputs, outputs, tool calls, token usage, and latency with no code changes required. For Hosted agents, you add client-side instrumentation to your container code, which is what this post focuses on.

What Foundry's built-in tracing doesn't give you is the ability to run arbitrary queries over the raw OTel data or correlate agent traces with infrastructure telemetry from the rest of your stack. That's where routing those same traces to Elastic adds value. The Instrument → Debug → Evaluate → Optimize loop that Foundry supports in its portal is even more powerful when you can drive it from ES|QL queries against the full trace corpus.

A note on Application Insights: it receives OTel traces but converts them into its own schema on ingestion. Elastic stores the data as-is (resource attributes, semantic convention fields, everything preserved) so you can query exactly what was emitted, without a translation layer between you and the data.

LLM tracing with the OpenTelemetry GenAI conventions

Foundry uses OpenTelemetry's GenAI semantic conventions to structure its traces. Three core span types cover most agent workloads.

Stability note: All gen_ai.* span names and attributes currently carry a Development stability badge in the OpenTelemetry registry; they are pre-1.0 and have already changed once (for example, gen_ai.system was renamed to gen_ai.provider.name). Pin your SDK versions and expect attribute strings to shift before these conventions reach stable status.

invoke_agent wraps the entire agent execution. Every run gets one of these as the root span.

chat is a single LLM API call. It carries gen_ai.provider.name (the provider: openai, anthropic, aws.bedrock), gen_ai.request.model, and gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on every single call, which is what makes per-call token cost attribution possible.

execute_tool is a tool or function invocation triggered by the model, nested under the chat span that requested it.

For multi-agent systems, Microsoft (in collaboration with Cisco Outshift) extended these conventions with additional span types now integrated into Foundry, Agent Framework, LangChain, LangGraph, and the OpenAI Agents SDK:

execute_task captures task planning and how work is decomposed and distributed across agents.
agent_to_agent_interaction (child of invoke_agent) traces direct communication between agents.
agent_planning logs an agent's internal planning steps
agent.state.management covers context and memory operations

The resulting trace for a multi-step Foundry agent looks like this:

[invoke_agent: research-agent]             ← root: the whole task
  [agent_planning]                         ← agent decides its approach
  [chat: azure]                            ← first LLM call
  [execute_tool: file_search]              ← Foundry built-in tool call
  [chat: azure]                            ← LLM call to reason over results
  [agent_to_agent_interaction: summarizer] ← handoff to specialist agent
    [invoke_agent: summarizer]             ← nested agent execution
      [chat: azure]
  [chat: azure]                            ← final synthesis call

In Elastic, the agent trace renders as a waterfall. You see how long each step took, which one errored, and where the tokens went, including across the agent handoff boundary. That's the data model we're getting into Elastic.

How to instrument a Foundry Hosted agent to emit OTel traces

Foundry Hosted agents run your code in a container managed by Foundry. Because you own the container, you control the instrumentation. Which path you take depends on the framework you're using.

Framework	Language	Key package	Instrumentation method
Agent Framework	Python	`azure-ai-projects`	`AIProjectInstrumentor().instrument()`
LangGraph	Python	`langchain-azure-ai`	`AzureAIOpenTelemetryTracer` callback
Custom container	TypeScript / Node.js	`@opentelemetry/sdk-node`	Manual `startActiveSpan`

Trace Agent Framework agents with the Azure AI Projects SDK (Python)

Agent Framework is Microsoft's framework for building Hosted agents on Foundry. It calls the Foundry Responses API for model inference and tool orchestration. To route traces to Elastic, configure the OTel SDK at container startup and add the OTLP exporter pointing at your Elastic endpoint.

Install the packages:

pip install azure-ai-projects azure-identity opentelemetry-sdk azure-core-tracing-opentelemetry opentelemetry-exporter-otlp-proto-http

Configure at startup in your agent container:

import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.telemetry import AIProjectInstrumentor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# OTLPSpanExporter reads OTEL_EXPORTER_OTLP_ENDPOINT and
# OTEL_EXPORTER_OTLP_HEADERS automatically, no need to pass them explicitly
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

# Tracing is off by default — opt in explicitly (experimental preview as of 2026)
# Set AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING=true in your container environment
AIProjectInstrumentor().instrument()

client = AIProjectClient(
    endpoint=os.getenv("AZURE_AI_FOUNDRY_PROJECT_ENDPOINT"),
    credential=DefaultAzureCredential(),
)

Every model call, tool invocation, and agent handoff made through the Responses API now produces a structured OTel span with token usage, model identity, and tool metadata. Note that GenAI tracing in the Azure AI Projects SDK is an experimental preview — you must set AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING=true in your container environment and call AIProjectInstrumentor().instrument() before any agent runs, or no spans are produced.

Trace LangGraph agents with AzureAIOpenTelemetryTracer (Python)

LangGraph is a supported framework for Foundry Hosted agents. Microsoft's langchain-azure-ai package provides an OTel-compliant tracer for LangGraph that emits spans for graph steps, tool invocations, and model calls. Configure the OTLP exporter the same way, then attach the tracer as a callback on each invocation:

pip install langchain-azure-ai langgraph langchain langchain-openai opentelemetry-sdk opentelemetry-exporter-otlp-proto-http azure-identity

from langchain_azure_ai.callbacks.tracers import AzureAIOpenTelemetryTracer
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# OTLPSpanExporter reads OTEL_EXPORTER_OTLP_ENDPOINT and
# OTEL_EXPORTER_OTLP_HEADERS automatically, no need to pass them explicitly
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

# AzureAIOpenTelemetryTracer is Microsoft's supported tracer for LangChain and LangGraph.
# Pass it as a callback when invoking your graph — it uses the active TracerProvider above.
azure_tracer = AzureAIOpenTelemetryTracer(name="my-langgraph-agent")

# app is your compiled LangGraph workflow (e.g. workflow.compile())
config = {"callbacks": [azure_tracer]}
result = app.invoke({"messages": [...]}, config=config)

Trace custom Node.js agents with the OpenTelemetry SDK (TypeScript)

For custom agent architectures in Node.js, wrap your orchestration logic directly with the OTel SDK. The startActiveSpan API handles parent-child nesting automatically. Any span created inside the callback is a child of the current span, giving you the decision tree hierarchy without explicit parent references.

import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { trace, SpanKind, SpanStatusCode } from "@opentelemetry/api";

// OTLPTraceExporter reads OTEL_EXPORTER_OTLP_ENDPOINT and
// OTEL_EXPORTER_OTLP_HEADERS automatically, no need to pass them explicitly
const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter(),
  serviceName: "my-foundry-agent",
});
sdk.start();

const tracer = trace.getTracer("foundry-agent", "1.0.0");

// Wrap your agent's entry point: everything inside becomes a child span
async function runAgent(userMessage: string) {
  return tracer.startActiveSpan(
    "invoke_agent my-agent",
    {
      kind: SpanKind.INTERNAL,
      attributes: {
        "gen_ai.operation.name": "invoke_agent",
        "gen_ai.agent.name": "my-agent",
        "gen_ai.provider.name": "azure",
      },
    },
    async (span) => {
      try {
        const result = await agentLoop(userMessage);
        span.setStatus({ code: SpanStatusCode.OK });
        return result;
      } catch (err) {
        span.recordException(err as Error);
        span.setStatus({ code: SpanStatusCode.ERROR });
        throw err;
      } finally {
        span.end();
      }
    }
  );
}

// Record token usage on every LLM call
async function callAzureOpenAI(messages: Message[]) {
  return tracer.startActiveSpan(
    "chat azure",
    {
      kind: SpanKind.CLIENT,
      attributes: {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": "azure",
        "gen_ai.request.model": "gpt-4o",
      },
    },
    async (span) => {
      const response = await client.chat.completions.create({ model: "gpt-4o", messages });
      span.setAttributes({
        "gen_ai.usage.input_tokens": response.usage.prompt_tokens,
        "gen_ai.usage.output_tokens": response.usage.completion_tokens,
        "gen_ai.response.model": response.model,
      });
      span.end();
      return response;
    }
  );
}

If your Node.js agent calls the OpenAI SDK directly, OpenTelemetry JS also ships an instrumentation-openai auto-instrumentation package as an alternative to the manual spans above (it works with Azure OpenAI clients too). Check its semantic-convention version against what's shown here before relying on it; third-party instrumentations for LangChain and others also exist but vary in how current they are.

How to get your Elastic managed OTLP endpoint and API key

The managed OTLP endpoint (mOTLP) is generally available on both Elastic Cloud Serverless and Elastic Cloud Hosted. It is not available for self-managed, ECE, or ECK deployments. For those, use the EDOT Collector as a gateway instead.

Serverless: Log in to Elastic Cloud → find your project → Manage → Application endpoints, cluster and component IDs → Ingest. Copy the endpoint value. Alternatively, go to Add data → Applications → OpenTelemetry inside your project, which also generates a pre-configured API key.

Elastic Cloud Hosted: Log in → Hosted deployments → Manage → Application endpoints → Managed OTLP. Copy the public endpoint value.

The API key must have event:write privilege on the apm application. The Elastic Cloud quickstart wizard generates this for you. Set two environment variables in your Foundry Hosted agent container:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://<your-motlp-endpoint>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey <your-api-key>"

Note the header format: ApiKey <key>, not Bearer. And the env var uses = as the separator between header name and value, not :.

Traces sent to the endpoint land in the traces-generic.otel-default data stream by default. In Kibana, find them under Observability → APM → Traces or query them directly with ES|QL against traces-generic.otel-*. No OTel Collector required. No schema translation.

Auto-instrument Foundry agents on AKS with the OpenTelemetry Operator

Foundry Hosted agents support bring-your-own VNet and can run container workloads on AKS. If you're in that configuration, you can skip the SDK-level OTLP configuration entirely. Add this annotation to your pod spec and the OpenTelemetry Operator injects the SDK automatically:

annotations:
  instrumentation.opentelemetry.io/inject-python: "true"

You still set OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS to point at Elastic. The SDK wiring is handled for you.

For agents running in Azure Container Apps, the platform's built-in managed OTel collector routes to Application Insights by default. To also get traces into Elastic, use the SDK-level configuration from the examples above alongside Foundry's built-in observability, or run a sidecar OTel Collector with an OTLP exporter configured for your Elastic endpoint.

How to keep a multi-agent handoff in one trace

If your architecture involves an orchestrator delegating to specialist agents, you want all of that to appear as one connected trace, not N disconnected fragments.

OTel handles this via the W3C TraceContext standard (traceparent and tracestate headers). For HTTP-based inter-agent calls, the SDK propagates these automatically. For queue-based handoffs (Service Bus, Event Hubs), you carry the context in the message itself:

from opentelemetry import propagate, context

# Sending agent: inject the active trace context into the message
carrier = {}
propagate.inject(carrier)

await queue.send_message({
    "payload": task_data,
    "trace_context": carrier,  # {"traceparent": "00-abc123...", "tracestate": "..."}
})

# Receiving agent: restore the trace context before doing any work
incoming_ctx = propagate.extract(message["trace_context"])
with context.use_context(incoming_ctx):
    await process_task(message["payload"])

With this in place, the entire work item appears as one trace in Elastic, from orchestrator through every worker agent, across process boundaries. The waterfall shows exactly where time was spent at each tier.

What Foundry agent traces look like in Elastic APM

Traces arrive as a hierarchical waterfall in the APM UI. For each agent invocation, you get the full span tree: every LLM call, tool execution, and sub-agent call nested under the root invoke_agent span. Every chat span carries gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, so you can see at a glance which model invocations are expensive and which are routine. Errors surface as specific failed spans with full stack traces, not a red line on a latency graph.

You can also query the trace data directly with ES|QL. To find every agent run in the past hour that consumed more than 20,000 input tokens:

FROM traces-generic.otel-default
| WHERE attributes.gen_ai.operation.name == "invoke_agent"
  AND @timestamp > NOW() - 1 hour
| STATS total_input_tokens = SUM(attributes.gen_ai.usage.input_tokens)
    BY trace.id, attributes.gen_ai.agent.name
| WHERE total_input_tokens > 20000
| SORT total_input_tokens DESC

Combining trace structure with token accounting in a single query is what OTel native storage makes possible.

How to sample agent traces without losing errors

For tail-based sampling, add a local OTel Collector as an intermediate hop before the Elastic endpoint. A policy that keeps all error traces, all slow traces, and samples down routine successes is a reasonable starting point:

# Run this Collector between your Foundry Hosted agent and the Elastic endpoint
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 5000 }
      - name: sample-routine
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  otlp/elastic:
    endpoint: "${OTEL_EXPORTER_OTLP_ENDPOINT}"
    headers:
      Authorization: "ApiKey ${ELASTIC_API_KEY}"
    sending_queue:
      enabled: true
      sizer: bytes
      queue_size: 50_000_000
      block_on_overflow: true

If you're sending directly from the SDK without a Collector, start at 100% sampling. Agent trace volume is usually smaller than expected. Evaluate storage costs after a week and tune from there. For most Foundry Hosted agent workloads, the direct SDK path is the right starting point.

How to use agent traces for evaluation and optimization

The Build session demo framed tracing as a four-step production loop: instrument, debug, evaluate, optimize. Debugging is the obvious first payoff: when an agent run fails or produces a wrong answer, the trace shows exactly which span introduced the problem. But the evaluate and optimize steps are where the loop compounds.

With traces flowing into Elastic, you can identify which agent runs produced incorrect or low-quality outputs, then pull those traces directly into an evaluation workflow in Foundry. The trace gives you the full context (the prompt, the tool calls, the model's reasoning path) that an eval needs to score quality and catch regressions. From there, optimization targets are concrete: this tool call is slow, this model is expensive for its output quality, this planning step runs unnecessarily on every request.

Foundry's agent optimizer can use trace data to improve agent instructions automatically. Elastic gives you the query layer to find the traces worth optimizing in the first place.

What you need to start tracing Foundry agents

You need four things: an Elastic Cloud Serverless project (the managed OTLP endpoint is included), your endpoint URL and API key from Project Management → Edit alias, instrumentation that matches your stack (AIProjectInstrumentor for Agent Framework, AzureAIOpenTelemetryTracer from langchain-azure-ai for LangGraph, or the OTel SDK directly for custom container code), and one agent run to verify the trace appears in Kibana's APM view.

The first trace that shows you exactly which tool call caused that 45-second timeout makes the setup worth it.

AI agent observability for Microsoft Foundry: two env vars, no collector