Both features follow the OTel GenAI semantic conventions and work with any OTel-instrumented provider. If your framework already emits OTel GenAI span attributes, there is nothing to change.

How OTel GenAI spans are structured

A GenAI span stores everything as span attributes. A typical chat span includes:

• gen_ai.provider.name: the provider (openai, anthropic, aws.bedrock, etc.); gen_ai.system is supported as a fallback for older instrumentation.
• gen_ai.operation.name: the operation type (chat, embeddings, etc.).
• gen_ai.request.model: the model being called.
• gen_ai.usage.input_tokens: tokens consumed by the prompt.
• gen_ai.usage.output_tokens: tokens generated in the response.
• gen_ai.input.messages, gen_ai.output.messages: conversation messages.
• gen_ai.system_instructions: the system prompt.

Both features read from these attributes:

| Feature | What it shows | Where it appears | Attributes it reads | | --- | --- | --- | --- | | GenAI tab | Details (operation type, request model, provider, input and output token counts, response model, response ID) and Conversation (system prompt, user messages, model response) | Span flyout in the APM trace view, and the span flyout in Discover | Appears with any gen_ai.* attribute. Conversation needs gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages | | Token count badges | Input and output token counts for each GenAI span | Every GenAI span row in the trace waterfall | gen_ai.usage.input_tokens, gen_ai.usage.output_tokens |

How to read LLM prompts and responses in the GenAI tab

When any gen_ai.* attribute is present on a span, the span flyout shows a dedicated GenAI tab next to Metadata.

The Details section shows model metadata from the span attributes: operation type, request model, provider, input and output token counts, response model, and response ID. The Conversation section shows the full exchange, populated from gen_ai.system_instructions (system prompt), gen_ai.input.messages (user messages), and gen_ai.output.messages (model response), each with a copy button so you can pull the exact prompt or response out of the trace without scraping text from a formatted table.

All raw span attributes remain accessible on the Metadata tab.

The GenAI tab is also available in the span flyout in Discover, so you can inspect LLM prompts and responses directly alongside your log and trace data without switching to the APM view.

What instrumentation does LLM tracing require?

No Kibana-side configuration is needed. The GenAI tab appears automatically when any gen_ai.* attribute is present on a span. Full Conversation support requires the OTel GenAI v1.37.0 span-attribute model: gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions.

Frameworks that emit the older span-events model (gen_ai.user.message, gen_ai.assistant.message, gen_ai.choice) will show the Details metadata section but will not populate the Conversation section. For a current list of compatible instrumentations, see the OTel GenAI semantic conventions.

To verify, open the span in Discover, check that gen_ai.input.messages and gen_ai.output.messages are present, and confirm the Conversation section renders.

If your application already sends APM data to Elastic from a GenAI workload, open any GenAI span in the trace view and check for the GenAI tab.

LLM token usage in the trace waterfall

Token count badges now appear on each GenAI span row in the waterfall, so you can scan the full trace without drilling in. In agentic traces with ten or twenty LLM calls, this lets you identify which span is driving token consumption before opening any span.

Each row shows input and output token count badges sourced from gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. The row label is the span name, which instrumentation frameworks typically set to something like chat gpt-4o-mini.

Waterfall-level token counts are most useful in agentic traces where a chain of LLM calls uses different models or the same model with varying context sizes.

Which LLM providers does Elastic APM support for GenAI tracing?

Elastic APM's LLM tracing works with any OTel-instrumented provider: the GenAI tab and waterfall token counts use the same OTel attribute schema regardless of which provider your application uses. Provider is read from gen_ai.provider.name, falling back to gen_ai.system for older instrumentation.

The OTel GenAI semantic conventions that enable this provider detection are currently in a Development lifecycle. Check the release notes before upgrading instrumentation.

How to enable LLM tracing in Elastic APM

Availability: Both features are available as a Technical Preview on Elastic Serverless and will be available as a Technical Preview in Elastic Stack 9.6.

To try these features:

1. Instrument your GenAI application with an OTel SDK that follows the OTel GenAI semantic conventions (v1.37.0 or later for full Conversation support).
2. Send traces to Elastic Observability using OTLP, the Elastic APM agent, or an EDOT SDK.
3. Open the APM section in Kibana, navigate to a service that makes LLM calls, and open the trace waterfall for any transaction.

The GenAI tab appears on any span with at least one gen_ai.* attribute set; token count badges appear when gen_ai.usage.input_tokens or gen_ai.usage.output_tokens are present.

If you don't have a GenAI application to test with, the otel-genai-chat-app repository is a minimal OpenAI chat app pre-instrumented with EDOT. Set OPENAI_API_KEY and follow the EDOT commands in the README to send traces to Elastic and see both features in action.

What's next for LLM observability in Elastic APM

We're exploring cost estimation per span (estimated spend based on model pricing and token counts, surfaced in the waterfall) and tool call rendering (structured display of tool/function call inputs and outputs for agentic spans).

If you are building GenAI applications and want early access or to share feedback, reach out through the Elastic community forums or open an issue in the kibana repository.

Here's the map before this rebuild, and after:

| Before | After | |---|---| | | |

Why the APM service map moved from Cytoscape.js to React Flow graph visualization

The payoff is concrete: React Flow renders nodes as real React components instead of a canvas draw loop, so panning and zooming stay smooth on large topologies, our Elastic UI (EUI) nodes and badges render natively (no more re-rendering the full canvas on every update), and DOM-based rendering makes nodes naturally visible to screen readers, unlike canvas-based rendering.

It's lighter, too: the graph library dropped from 172.4 KiB (cytoscape.js) to 69 KiB (@xyflow/react), about 103 KiB less JavaScript to ship, #248470.

How much faster is the APM service map with React Flow?

Before committing to the APM service map migration from Cytoscape.js to React Flow, we benchmarked both libraries at 100, 200, and 500 services (a synthetic chain topology, measured via Lighthouse and component-level timings averaged over multiple runs, #248470):

| Services | Cytoscape.js render | React Flow render | Faster by | |---|---|---|---| | 100 | 61.6 ms | 15.0 ms | ~76% | | 200 | 102.5 ms | 28.1 ms | ~73% | | 500 | 392.5 ms | 64.1 ms | ~84% |

At 500 services, React Flow draws the map in ~64 ms versus Cytoscape.js's ~393 ms, roughly 6x faster. The underlying graph layout step alone got ~70-78% faster. Total main-thread blocking time dropped ~20%, and peak memory was slightly lower despite rendering everything as real DOM. Those results and more are available in the benchmark results comment.

Keyboard navigation and accessibility in the APM service map

The map now announces your position in the topology as you navigate, with live screen-reader context for every interaction (#251444).

• Spatial arrow-key navigation: arrow keys navigate based on visual proximity rather than logical document order. Focus moves to the nearest service in the visual direction you press, even through complex serpentine folds.

• Direct shortcuts: launch flyouts with Enter or Space, and dismiss them with Escape.

• Live announcements: every interaction is backed by a screen-reader announcement, such as "Selected connection from A to B."

  

How the APM service map layout algorithm works

The APM service map uses Dagre for hierarchical graph layout, then applies serpentine folding to prevent long dependency chains from producing unreadable high-aspect-ratio strips:

1. Dagre hierarchical layout: we use Dagre, a graph layout engine, with a direction toggle between horizontal and vertical that you can change from the options panel. If Dagre fails to compute a layout, the map falls back to a deterministic grid so it stays interactive and error-free.

   

2. Serpentine folding for long chains: long dependency pipelines produce thin, unreadable strips that require heavy zooming. When the aspect ratio becomes too extreme, we wrap ranks into stacked bands that snake back and forth, so "fit view" can zoom in much tighter on the actual services. If the topology is already compact or has too many cross-band edges, we skip folding. (#272900).

   

Service dependency mapping and Kibana dashboard embedding under the hood

The visual refresh is the easy part to see. Some of the architecture changes that underpin the rebuilt map are:

Unified resource nodes for cleaner service dependency mapping

We now group external dependencies into unified resource nodes, reducing visual noise. We also fixed message-queue span grouping so those patterns no longer create orphaned nodes (#252713).

Service map flyout charts powered by ES|QL and Lens

The service flyout's infrastructure and RED metrics use ES|QL and Kibana's core Lens visualization engine, giving the map the same charting experience used across Kibana (#273713).

Embedding the service map on any Kibana dashboard

Adding a service map to a dashboard takes one click. Open the "Copy to dashboard" menu from any map view in APM, and the panel carries your current environment, service filter, KQL query, and filter chips directly across. Relative time ranges like now-15m transfer as-is rather than freezing to an absolute timestamp, so the panel stays live on the dashboard (#272277).

Once embedded, the panel adapts to where it lives. It shows or hides controls based on view mode, and respects global time settings without overwriting your relative ranges (#274551).

What's next for the APM service map

The service map rebuild is a foundation, not a finish line. Beyond what's available in 9.5, some of what's coming next is already on the APM service map roadmap.

Contributors

I led this migration and built these features alongside a great team. Thanks to Samuel Brito, Gonçalo Rica Pais da Silva, Irene Blanco Fabregat, Carlos Crespo, Miriam Aparicio Garcia, Sandra G, and Nathan Smith for the engineering, and to Karolina Kurstak and Roshan Gonsalkorale for the design and product work.

From a checkout RED metric alert to a failing shipping dependency on the APM service map (OpenTelemetry Demo).

Where the APM service map is available

This is available in Elastic Observability serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

Prerequisites for APM service map dependency analysis

You need trace data from services instrumented with any method Elastic APM supports.

• Application instrumentation: one of the following:
• Elastic APM agents for Java, .NET, Node.js, Python, PHP, Ruby, Go, and other supported languages
• Elastic Distributions of OpenTelemetry (EDOT) language SDKs
• OpenTelemetry SDKs sending OTLP via the EDOT Collector, Elastic Agent, APM Server, or the Managed OTLP endpoint. If you run a custom upstream Collector pipeline, include both the elasticapm connector and the elasticapm processor. Those components ship with the EDOT Collector (or a custom EDOT-like build); they are not part of the standard OpenTelemetry Collector Contrib distribution. For wiring details, see the upstream collector setup.
• Service map data: distributed traces that link the services in your architecture. The map draws connections from span parent-child relationships across instrumented services.
• Backend: Elastic Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

Dependency analysis walkthrough: from APM alert to shipping dependency

Step 1: APM service map on the alert detail page

When you receive a notification for a RED metric threshold breach on the checkout service, open the alert detail page.

The upgraded service map is embedded on the page, so you can start analysing dependencies the moment you land on the alert. You see checkout in context with its upstream and downstream connections without navigating away.

In our example, failed transactions have increased on checkout. The map is already scoped to the alert time range:

Step 2: Spotting a failing dependency on the service map

When we scan through the downstream dependencies, we can see some indicators that there is a problem with the checkout and shipping services. When we open the shipping service fly-out to view more details, we can see the transactions for /get-quote have an elevated failure rate:

Step 3: Drilling into failing transactions to identify RED metric drivers

Click on the /get-quote transaction to drill into these transactions. We can see a release marker for version 2.3.0 that seems to correlate to increase in failures. To validate this is the main contributing factor, we want to analyse these transactions and check a few dimensions to isolate the behaviour to the release:

Step 4: Comparing RED metrics by dimension in Traces in Discover

Using Traces in Discover, we can use the breakdown feature to compare various attributes to confirm the RED metric change is due to the release. As we break down by service.version, k8s.deployment.name and k8s.pod.name, we can see the failure rate is highly elevated for just this Kubernetes deployment.

These field names match the OpenTelemetry semantic conventions used by the demo (EDOT / OTel instrumentation). If you instrument with classic Elastic APM agents, use the ECS-style kubernetes.* equivalents instead (see ECS ↔ OTel field alignment).

Given we have clear indicators that this problem is almost certainly due to a problem caused by this Kubernetes deployment, we would next investigate the changes this deployment made so we can identify a mitigation strategy to return the system to a healthy status, most likely a rollback of the Kubernetes deployment.

Dependency analysis with the service map on custom dashboards

The APM service map can also be added as a panel to custom Kibana dashboards. If you have a custom Dashboard you would like to surface for a debugging problem with your service, you can attach it to your alert rule so users can use this to complement the default alert detail view.

When you receive the notification and land on the alert detail page, you can click Related Dashboards and open your custom dashboard. From here, you can use the same Service Map panel to quickly analyse dependencies just like the alert detail page:

What's new in the APM service map

Below is a list of the upgrades to the service map:

Service map embedded on APM alert pages and dashboards

• All APM alert detail pages will have the service map embedded to facilitate faster dependency analysis.
• There is a new Service Map Dashboard panel that you can add to any Dashboard.

Full-screen service map for large architectures

Open the map in full screen from the APM UI or from a dashboard panel. Full screen gives you more room on large architectures and works the same whether you opened the map from an alert, a service, or a dashboard.

Service map controls: search, filters, and orientation

On the service map in the APM UI and on dashboard embeds, the map toolbar adds controls to move faster on busy environments:

• Search — find services by text string

• Orientation — switch layout direction when dense graphs are hard to read

• Filters — narrow the map by alert status, whether a service has dependencies, anomaly status, and SLO status

Service map legend: node shapes, connections, and anomaly scores

A built-in legend explains node shapes, connection styles, and health indicators on the map. You spend less time decoding colours and icons when you are triaging under pressure.

The legend covers node shapes (instrumented services, databases and messaging, grouped resources), connection styles (one-way and two-way requests), and anomaly score colour rings from low through critical:

Minimap for navigating large service maps

A minimap helps you orient yourself on large maps. It also highlights anomalous services at a glance, so you can spot outliers without panning across the full graph.

RED metrics on connections with drill-through to Traces in Discover

Select a connection between two services to view RED metrics for the requests between them: rate, errors, and duration for that specific dependency edge.

From there, open Traces in Discover in one click to analyse those requests with full query flexibility. That is how we confirmed shipping was failing the requests checkout sent to it in the walkthrough above.

Adding the service map to a dashboard from the APM UI

From the service map in the APM UI, add the current map view to a dashboard in one click. You do not need to rebuild filters or time range settings manually when you want the same map on a team dashboard.

Service fly-out: preview RED metrics, anomalies and SLOs from the map

Wherever you view the service map in Kibana, you can open a service fly-out to preview a service without leaving the map.

The fly-out shows:

• RED metrics for the service
• Anomaly status — whether machine learning has flagged unusual behaviour
• SLO status — whether the service is meeting its objectives
• Transaction breakdown — how rate, errors, and duration split across transaction types

Use it to sanity-check a node before you open the full service page or follow a connection into Traces in Discover.

Summary: From APM alert to root cause with the service map

From a RED metric alert on checkout, the embedded service map showed shipping as the failing downstream dependency. Map controls, connection RED metrics, dashboard embeds, and the service fly-out are available on every map view in Kibana, so you can start dependency analysis wherever you already work.

Further Reading

You can see how we implemented these changes in the following post from Jenny Pavlova, who led the technical implementation:

6x faster at 500 services: how we rebuilt the Kibana APM service map from canvas to React DOM

Related Posts

See other recent improvements we've made for observing instrumented services in the blogs below:

• Four clicks from alert to root cause: how Elastic Observability links APM services to Kubernetes infrastructure
• Your SLO is on fire; here's how to find the arsonist in Elastic Observability
• Three clicks from alert to error log: breaking down RED metrics by any span attribute in Elastic Observability