Both features follow the OTel GenAI semantic conventions and work with any OTel-instrumented provider. If your framework already emits OTel GenAI span attributes, there is nothing to change.

How OTel GenAI spans are structured

A GenAI span stores everything as span attributes. A typical chat span includes:

• gen_ai.provider.name: the provider (openai, anthropic, aws.bedrock, etc.); gen_ai.system is supported as a fallback for older instrumentation.
• gen_ai.operation.name: the operation type (chat, embeddings, etc.).
• gen_ai.request.model: the model being called.
• gen_ai.usage.input_tokens: tokens consumed by the prompt.
• gen_ai.usage.output_tokens: tokens generated in the response.
• gen_ai.input.messages, gen_ai.output.messages: conversation messages.
• gen_ai.system_instructions: the system prompt.

Both features read from these attributes:

| Feature | What it shows | Where it appears | Attributes it reads | | --- | --- | --- | --- | | GenAI tab | Details (operation type, request model, provider, input and output token counts, response model, response ID) and Conversation (system prompt, user messages, model response) | Span flyout in the APM trace view, and the span flyout in Discover | Appears with any gen_ai.* attribute. Conversation needs gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages | | Token count badges | Input and output token counts for each GenAI span | Every GenAI span row in the trace waterfall | gen_ai.usage.input_tokens, gen_ai.usage.output_tokens |

How to read LLM prompts and responses in the GenAI tab

When any gen_ai.* attribute is present on a span, the span flyout shows a dedicated GenAI tab next to Metadata.

The Details section shows model metadata from the span attributes: operation type, request model, provider, input and output token counts, response model, and response ID. The Conversation section shows the full exchange, populated from gen_ai.system_instructions (system prompt), gen_ai.input.messages (user messages), and gen_ai.output.messages (model response), each with a copy button so you can pull the exact prompt or response out of the trace without scraping text from a formatted table.

All raw span attributes remain accessible on the Metadata tab.

The GenAI tab is also available in the span flyout in Discover, so you can inspect LLM prompts and responses directly alongside your log and trace data without switching to the APM view.

What instrumentation does LLM tracing require?

No Kibana-side configuration is needed. The GenAI tab appears automatically when any gen_ai.* attribute is present on a span. Full Conversation support requires the OTel GenAI v1.37.0 span-attribute model: gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions.

Frameworks that emit the older span-events model (gen_ai.user.message, gen_ai.assistant.message, gen_ai.choice) will show the Details metadata section but will not populate the Conversation section. For a current list of compatible instrumentations, see the OTel GenAI semantic conventions.

To verify, open the span in Discover, check that gen_ai.input.messages and gen_ai.output.messages are present, and confirm the Conversation section renders.

If your application already sends APM data to Elastic from a GenAI workload, open any GenAI span in the trace view and check for the GenAI tab.

LLM token usage in the trace waterfall

Token count badges now appear on each GenAI span row in the waterfall, so you can scan the full trace without drilling in. In agentic traces with ten or twenty LLM calls, this lets you identify which span is driving token consumption before opening any span.

Each row shows input and output token count badges sourced from gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. The row label is the span name, which instrumentation frameworks typically set to something like chat gpt-4o-mini.

Waterfall-level token counts are most useful in agentic traces where a chain of LLM calls uses different models or the same model with varying context sizes.

Which LLM providers does Elastic APM support for GenAI tracing?

Elastic APM's LLM tracing works with any OTel-instrumented provider: the GenAI tab and waterfall token counts use the same OTel attribute schema regardless of which provider your application uses. Provider is read from gen_ai.provider.name, falling back to gen_ai.system for older instrumentation.

The OTel GenAI semantic conventions that enable this provider detection are currently in a Development lifecycle. Check the release notes before upgrading instrumentation.

How to enable LLM tracing in Elastic APM

Availability: Both features are available as a Technical Preview on Elastic Serverless and will be available as a Technical Preview in Elastic Stack 9.6.

To try these features:

1. Instrument your GenAI application with an OTel SDK that follows the OTel GenAI semantic conventions (v1.37.0 or later for full Conversation support).
2. Send traces to Elastic Observability using OTLP, the Elastic APM agent, or an EDOT SDK.
3. Open the APM section in Kibana, navigate to a service that makes LLM calls, and open the trace waterfall for any transaction.

The GenAI tab appears on any span with at least one gen_ai.* attribute set; token count badges appear when gen_ai.usage.input_tokens or gen_ai.usage.output_tokens are present.

If you don't have a GenAI application to test with, the otel-genai-chat-app repository is a minimal OpenAI chat app pre-instrumented with EDOT. Set OPENAI_API_KEY and follow the EDOT commands in the README to send traces to Elastic and see both features in action.

What's next for LLM observability in Elastic APM

We're exploring cost estimation per span (estimated spend based on model pricing and token counts, surfaced in the waterfall) and tool call rendering (structured display of tool/function call inputs and outputs for agentic spans).

If you are building GenAI applications and want early access or to share feedback, reach out through the Elastic community forums or open an issue in the kibana repository.

The challenge deepens when something goes wrong. A crash on the device, a slow screen, or an error response might originate in the Android app itself, in a backend service, or somewhere in the network path between them. Debugging these problems without a connected, E2E view from the mobile client to the backend is time-consuming and frustrating. And without a standard instrumentation format, mobile teams often end up maintaining separate tooling that doesn't integrate with what the backend and infrastructure teams already use.

OpenTelemetry offers a way out: a unified, open-standard instrumentation model that works across platforms and languages, backed by a large community. The Elastic Distribution of OpenTelemetry Android, or EDOT Android, is an APM agent for native Android applications built on top of OpenTelemetry. It gives Android teams a practical path to observe mobile app behavior in Elastic, providing them with distributed tracing, crash reporting, session tracking, disk buffering, and automatic instrumentation, with as little code as possible while staying grounded in open standards.

To see what it all looks like, we will instrument a demo Android weather application end to end. You will run Elasticsearch, Kibana, and the Elastic Agent locally. The Elastic Agent provides the OTLP endpoint that receives telemetry from the Android app and backend. You will then generate distributed traces, custom spans, logs, and Android crashes from the app, and explore the results in Kibana using the Android OpenTelemetry dashboards.

This article focuses on a hands-on experiment to explore the E2E experience of observing Android apps with Elastic, using the EDOT Android agent. For more specific details on the EDOT Android agent, such as a list of supported features and a setup guide for your own Android project, take a look at EDOT Android docs.

Setting up EDOT Android with Elasticsearch and Kibana

We will use the EDOT Android demo application. The demo is intentionally small but covers the main workflows you need when evaluating mobile observability with Elastic.

The demo has two main components: an Android app, and a Spring Boot backend. Additionally, you'll need an Elastic Stack environment up and running; we'll explain more about how to get one later in this guide.

Prerequisites

• Java 17 or higher.
• Docker.
• Android Studio.
• An Android emulator.
• On Windows, use Windows Subsystem for Linux (WSL) to run the demo scripts.

Step 1: Clone the demo app's repository

We'll start by cloning the EDOT Android demo application:

git clone git@github.com:elastic/android-agent-demo.git

Step 2: Start the Elastic Stack

The demo uses start-local to run Elasticsearch, Kibana, and the Elastic Agent with a single command. In this setup, the Elastic Agent provides the OTLP endpoint that receives telemetry from the application and backend. Run this from the directory where you want the local Elastic files to be created:

curl -fsSL https://elastic.co/start-local | sh -s -- --edot

For more information on this step, take a look at the demo app's instructions.

Step 3: Start the local backend

The demo backend is a Spring Boot service instrumented with the EDOT Java agent. It handles the app's weather requests and calls the Open-Meteo public API for weather data.

./backend-manager start

For more information on managing the backend service, take a look at the demo app's instructions.

Step 4: Launch the Android application

Use Android Studio to open up the EDOT Android demo application repo and run the application in your emulator. More info on how to run Android apps from Android Studio here.

Generating distributed traces, errors and crashes from an Android app

The Android app has two screens: a city selector and a weather display screen that shows the current weather for the selected city on the previous screen. It includes two intentional failure paths: the first one is reached by selecting New York, which causes the backend to reject the request (the demo backend only supports European cities), and tapping the floating crash button intentionally crashes the app so you can review crash reporting in Kibana after relaunch. We'll take a look at those use cases in more detail below.

Tracing a successful request end to end

In the EDOT Android demo app, selecting "Paris" as the city triggers a successful backend request on the second screen, for which a span will be automatically generated using EDOT Android's OkHttp auto-instrumentation, which supports all OkHttp-generated HTTP requests and tools using it, such as Retrofit. Aside from the Android HTTP span, the successful city request continues e2e and creates a backend HTTP client span to Open-Meteo.

How backend errors appear in the Android distributed trace

The demo backend only supports European cities, so selecting "New York" causes it to fail, which in turn automatically creates an error associated with our Android app's HTTP span. This is done automatically. We'll see later how to find and inspect these issues from Kibana.

How EDOT Android captures and reports app crashes

The crash button creates a crash event that appears in Kibana after the app is reopened. This event contains session information that will help us narrow down its root cause from Kibana, as we'll see later.

Note: EDOT Android automatically attaches Android session context to spans and logs. That means that any span or log created before the crash can be reviewed together with the crash event and nearby spans from the same session, giving you a complete picture of what the user was doing. This even applies to manually created spans and logs.

Visualizing Android application monitoring data in Kibana

To see the whole story from our Android app in a single place, we'll install Kibana's Android OpenTelemetry Assets package by following the steps below.

1. In Kibana, search for "Android OpenTelemetry Assets" in the global search field.
2. Open it and click Install to add the Android dashboards to your Kibana instance.

Exploring Android application monitoring dashboards in Kibana

Once the content package is installed, open the [Android OTel] Application Overview dashboard:

1. In Kibana, search for Dashboards in the global search field or in Kibana's menu.
2. In Dashboards, search for Android OTel and open the "[Android OTel] Application Overview" dashboard.
3. Select your application from the Applications panel at the top of the dashboard.

The dashboard provides a set of metric panels for an overview of your app's health, performance, and RUM, as well as a set of panels that can be further explored either in Discover or the Exception dashboard, as explained below.

How to inspect the distributed tracing waterfall in Kibana

From the Application Overview dashboard, go to one of the span tables (either All spans or Failed spans) and click its Explore in Discover button.

In Discover, click the expand icon on the left side of any span row to open its details panel. The trace waterfall UI appears inside, showing the full span hierarchy and timing for that trace. You can expand the waterfall to full screen and drill down from there.

Analyzing failed spans and backend errors in Kibana

While you can find all kinds of spans in the dashboard's All spans panel, you can narrow them down to failed ones only by exploring the Failed spans panel instead.

For the New York path, find a failed span and expand it. The trace waterfall highlights the backend error, and the exception details show the intentional backend rule that only supports European cities.

Reviewing crash details and stacktraces in the exception dashboard

Crash reporting is provided by the crash automatic instrumentation plugin. When an unhandled exception crashes the app, EDOT Android stores the crash event on disk. The event is exported the next time the app starts. Disk buffering ensures the crash event is not lost even if the network was unavailable at the time of the crash.

In the Application Overview dashboard, scroll to the Crashes section. You will see crash groups listed by a computed stacktrace group ID. Select a group and click View crash details to open the Exception Details dashboard.

The Exception Details dashboard shows a set of metrics to better understand the impact of the selected crash, as well as its full stacktrace. Crash events are grouped based on their stacktrace, which helps ensure that the same crash is counted and aggregated in this dashboard to better understand a single crash's impact on your application.

For this demo, the stacktrace points to the intentional crash in MainActivity. The nearby session events should also include the custom Crash button click log created just before the crash, which helps explain how the crash was triggered. We'll take a look at how to inspect a session to get an idea of the user's journey within your application that led them to a crash.

Using sessions to understand user flow in EDOT Android

Mobile troubleshooting often starts with a single bad outcome (a crash, an error, a slow UX), but the useful question is what happened before that outcome. EDOT Android helps answer that by attaching session.id to every span and log emitted by the application, even for manually created ones.

A session is meant to cover a single user interaction with your application. A new one is created when there is no previous active session or when the previous session has expired. Sessions expire after 30 minutes of inactivity. If the app stays active, a session can last up to 4 hours.

This lets you query all the telemetry from a single session and review it in order. After finding a crash group, drill into one affected session from the Top affected sessions panel and review the event timeline. You can see the custom logs, app startup spans, HTTP request spans, and crash data together in one investigation path.

The Event timeline panel on the Application Overview dashboard is also useful here: select a session from the dashboard's top filters, and the timeline shows the full sequence of spans and logs in that session chronologically.

What EDOT Android adds as a mobile APM for production apps

The demo uses a local stack and simple code, but the same agent features apply to production apps.

Disk buffering stores telemetry locally before export. This reduces data loss when the device has poor connectivity or the app is temporarily offline.

Automatic instrumentation creates telemetry for supported targets without adding code around every call. Today that includes OkHttp, crash reporting, and an adapter for OpenTelemetry Android instrumentation.

Manual instrumentation lets you add spans, logs, and metrics for app-specific workflows. This is useful for screen loading times, checkout flows, login steps, feature usage, or any area where framework-level telemetry alone is not enough.

Central configuration can remotely adjust selected EDOT Android behavior through Kibana when the OpAMP endpoint is configured. At the time of writing, central configuration for EDOT Android is in preview and supports settings such as recording and session sample rate.

Distributed tracing connects Android app requests to backend service spans so you can trace the full path of any user action, from the tap on the screen to the database query on the server. EDOT Android ensures that your application's telemetry timestamps are in sync with the coordinated universal time. This ensures a proper trace waterfall hierarchy later on in Kibana, where different components are properly coordinated in time.

Clean up the demo

When you are finished, stop the backend in case you're planning to restart it later, or uninstall it otherwise:

./backend-manager stop
# ./backend-manager uninstall

Then stop or uninstall the local Elastic Stack:

cd elastic-start-local
./stop.sh
# ./uninstall.sh

Getting started with EDOT Android in your own app

EDOT Android gives native Android teams an OpenTelemetry-based path for mobile APM in Elastic. With a small Gradle setup and one early initialization call, you get automatic HTTP spans, crash reporting, session tracking, and direct access to the OpenTelemetry SDK for custom telemetry, and you can see it all tied together in Kibana's Android dashboards.

Observability is a crucial part of modern mobile development. Crashes, slow screens, and backend errors all impact real users, and the sooner you can identify root causes across the full request path, from the device to the database, the better. The demo app is a good first step because it exercises the complete workflow without requiring a production deployment. After that, the same setup model applies to your own app with production endpoints, API key authentication, and custom spans and logs tailored to your use cases.

Developer resources:

• EDOT Android demo application
• EDOT Android documentation
• EDOT Android getting started guide
• EDOT Android automatic instrumentation
• EDOT Android manual instrumentation
• EDOT Android configuration
• EDOT Android troubleshooting
• Android OpenTelemetry Assets dashboard docs

Don't have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out mobile observability with EDOT Android as described in this guide. We'd love to hear about your experience gaining visibility into your Android application stack with Elastic.

This walkthrough shows the full path using the OpenTelemetry Demo (Astronomy Shop), where a failing shipping dependency drives every checkout SLI miss. You can reproduce it by running the demo against Elastic Observability, defining an APM availability SLO on checkout, and introducing a failing dependency.

Availability

SLO burn rate analysis is available in Elastic Observability serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

Prerequisites for SLO burn rate alerting in Elastic Observability

You need an instrumented service and an SLO defined on its APM data in Elastic Observability.

• Application instrumentation: one of the following:
• Elastic APM agents for Java, .NET, Node.js, Python, PHP, Ruby, Go, and other supported languages
• Elastic Distributions of OpenTelemetry (EDOT) language SDKs
• OpenTelemetry SDKs sending OTLP via the EDOT Collector, Elastic Agent, APM Server, or the Managed OTLP endpoint. If you run a custom upstream Collector pipeline, include both the elasticapm connector and the elasticapm processor. Those components ship with the EDOT Collector (or a custom EDOT-like build); they are not part of the standard OpenTelemetry Collector Contrib distribution. For wiring details, see the upstream collector setup.
• SLO definition: an SLO based on APM latency or APM availability for the service you want to protect. Elastic creates a default burn rate alert rule when you save the SLO.
• Backend: Elastic Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

SLO burn rate analysis walkthrough: alert to failing dependency

Step 1: Review SLO burn rate on the alert detail page

Open the SLO burn rate alert detail page from the notification.

The chart shows when burn rate rose and how much error budget remains for the rolling period. From there, open the SLI linked to the alert and decide whether you are looking at a short spike or sustained degradation before you dig into events.

In this example, the availability SLO for checkout is burning budget quickly, and the rise is recent, so the change likely landed in the last few hours rather than as long-running drift:

Step 2: Check the SLI error rate driving SLO error budget consumption

On the SLI view, check the error rate that feeds the SLO.

You can look at the SLI as RED metrics in the APM UI for a quick service-level picture, or open Traces in Discover when you want to filter, compare, or break down the spans that count toward the SLO.

Here, error rate on checkout has climbed, which matches the jump in availability SLO burn rate:

Step 3: Compare good and bad event spans for the SLI

Open the events for the SLI and switch between spans for good events and bad events.

Differences between those sets usually show up before you open individual traces. In this case, bad events share a pattern that good events do not, which narrows where to look next:

Step 4: Find the failing span in the trace waterfall

Open a few sample spans for bad events and view each trace in the waterfall.

That puts the failing span in the request path so you can see which hop is causing the SLI miss. In this example, the failing span is a downstream call, not application logic inside checkout itself:

Step 5: Verify which dependency is burning your SLO error budget

Open the service that owns the bad span and check its dependencies.

Here the waterfall points to shipping and a failing call to quote-old. Requests to that dependency fail on every attempt, so the SLO burn is driven by an external dependency rather than checkout code:

SLO burn rate investigation: from alert to root cause

From an SLO burn rate alert in Elastic Observability, you can see when burn rate rose, open the linked SLI, compare good and bad event spans, inspect bad spans in the trace waterfall, and verify failing dependencies to find what is burning error budget.

Here's the map before this rebuild, and after:

| Before | After | |---|---| | | |

Why the APM service map moved from Cytoscape.js to React Flow graph visualization

The payoff is concrete: React Flow renders nodes as real React components instead of a canvas draw loop, so panning and zooming stay smooth on large topologies, our Elastic UI (EUI) nodes and badges render natively (no more re-rendering the full canvas on every update), and DOM-based rendering makes nodes naturally visible to screen readers, unlike canvas-based rendering.

It's lighter, too: the graph library dropped from 172.4 KiB (cytoscape.js) to 69 KiB (@xyflow/react), about 103 KiB less JavaScript to ship, #248470.

How much faster is the APM service map with React Flow?

Before committing to the APM service map migration from Cytoscape.js to React Flow, we benchmarked both libraries at 100, 200, and 500 services (a synthetic chain topology, measured via Lighthouse and component-level timings averaged over multiple runs, #248470):

| Services | Cytoscape.js render | React Flow render | Faster by | |---|---|---|---| | 100 | 61.6 ms | 15.0 ms | ~76% | | 200 | 102.5 ms | 28.1 ms | ~73% | | 500 | 392.5 ms | 64.1 ms | ~84% |

At 500 services, React Flow draws the map in ~64 ms versus Cytoscape.js's ~393 ms, roughly 6x faster. The underlying graph layout step alone got ~70-78% faster. Total main-thread blocking time dropped ~20%, and peak memory was slightly lower despite rendering everything as real DOM. Those results and more are available in the benchmark results comment.

Keyboard navigation and accessibility in the APM service map

The map now announces your position in the topology as you navigate, with live screen-reader context for every interaction (#251444).

• Spatial arrow-key navigation: arrow keys navigate based on visual proximity rather than logical document order. Focus moves to the nearest service in the visual direction you press, even through complex serpentine folds.

• Direct shortcuts: launch flyouts with Enter or Space, and dismiss them with Escape.

• Live announcements: every interaction is backed by a screen-reader announcement, such as "Selected connection from A to B."

  

How the APM service map layout algorithm works

The APM service map uses Dagre for hierarchical graph layout, then applies serpentine folding to prevent long dependency chains from producing unreadable high-aspect-ratio strips:

1. Dagre hierarchical layout: we use Dagre, a graph layout engine, with a direction toggle between horizontal and vertical that you can change from the options panel. If Dagre fails to compute a layout, the map falls back to a deterministic grid so it stays interactive and error-free.

   

2. Serpentine folding for long chains: long dependency pipelines produce thin, unreadable strips that require heavy zooming. When the aspect ratio becomes too extreme, we wrap ranks into stacked bands that snake back and forth, so "fit view" can zoom in much tighter on the actual services. If the topology is already compact or has too many cross-band edges, we skip folding. (#272900).

   

Service dependency mapping and Kibana dashboard embedding under the hood

The visual refresh is the easy part to see. Some of the architecture changes that underpin the rebuilt map are:

Unified resource nodes for cleaner service dependency mapping

We now group external dependencies into unified resource nodes, reducing visual noise. We also fixed message-queue span grouping so those patterns no longer create orphaned nodes (#252713).

Service map flyout charts powered by ES|QL and Lens

The service flyout's infrastructure and RED metrics use ES|QL and Kibana's core Lens visualization engine, giving the map the same charting experience used across Kibana (#273713).

Embedding the service map on any Kibana dashboard

Adding a service map to a dashboard takes one click. Open the "Copy to dashboard" menu from any map view in APM, and the panel carries your current environment, service filter, KQL query, and filter chips directly across. Relative time ranges like now-15m transfer as-is rather than freezing to an absolute timestamp, so the panel stays live on the dashboard (#272277).

Once embedded, the panel adapts to where it lives. It shows or hides controls based on view mode, and respects global time settings without overwriting your relative ranges (#274551).

What's next for the APM service map

The service map rebuild is a foundation, not a finish line. Beyond what's available in 9.5, some of what's coming next is already on the APM service map roadmap.

Contributors

I led this migration and built these features alongside a great team. Thanks to Samuel Brito, Gonçalo Rica Pais da Silva, Irene Blanco Fabregat, Carlos Crespo, Miriam Aparicio Garcia, Sandra G, and Nathan Smith for the engineering, and to Karolina Kurstak and Roshan Gonsalkorale for the design and product work.

Availability

This is available in serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

What you need for RED metrics breakdown in Elastic Observability

You need trace data from a service instrumented with any method Elastic APM supports.

• Application instrumentation: one of the following:
• Elastic APM agents for Java, .NET, Node.js, Python, PHP, Ruby, Go, and other supported languages.
• OpenTelemetry SDKs sending OTLP via Elastic Agent or an upstream OpenTelemetry Collector with the elasticapm connector under connectors (not processors).
• Useful attributes: breakdown works best when spans include the dimensions you want to compare (k8s.pod.name, k8s.deployment.name, service.version, and others). You can also declare custom attributes on spans: add labels to transactions and spans with Elastic APM agents, or set OpenTelemetry attributes on spans and resources with OpenTelemetry SDKs. Those custom fields work as breakdown dimensions too.
• Backend: Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

How to go from a RED metrics alert to the root-cause error log

Step 1: Review RED metrics on the alert detail page

When you receive a notification for a RED metric threshold breach, if you open the alert detail page, you can review the symptoms for the impacted service on one page.

In our example, failed transactions have clearly increased for the cart service, so we want to understand what is driving that RED metric change:

Step 2: Break down RED metrics by span attributes in Discover

To investigate why a RED metric changed, open Traces in Discover and use the new breakdown feature to split RED metrics by any attribute on your spans. In our example, we're going to check Kubernetes attributes and service version, but you could break down by any span attribute you send (e.g. cloud.region, cloud.availability_zone, or container.id).

Each breakdown shows which attribute values moved the metric, so you can see whether the problem is isolated to one pod, deployment, version, or whatever dimension you split on.

In our example, error rate clusters on a single Kubernetes deployment, which points the investigation at a release. We will break down by service.version to validate our hypothesis:

Step 3: Open the trace waterfall and read the linked error log

Once trace breakdown has identified a specific service version as the likely cause, we can filter by that service.version and look at sample failing spans to see if they explain why the version is causing failures.

Open the trace waterfall for one failing span and follow through to the linked error log.

In our example, the error log points to bad configuration that could be causing the issue. Either way, we have narrowed the investigation to a solid hypothesis we can act on in just a few clicks:

From RED metrics alert to error log in Elastic Observability

From a RED metric alert, you can review the symptomatic service, break down Traces in Discover by any attribute on your spans, and open a failing span's trace waterfall to reach the error log in just a few clicks.

From a checkout RED metric alert to a failing shipping dependency on the APM service map (OpenTelemetry Demo).

Where the APM service map is available

This is available in Elastic Observability serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

Prerequisites for APM service map dependency analysis

You need trace data from services instrumented with any method Elastic APM supports.

• Application instrumentation: one of the following:
• Elastic APM agents for Java, .NET, Node.js, Python, PHP, Ruby, Go, and other supported languages
• Elastic Distributions of OpenTelemetry (EDOT) language SDKs
• OpenTelemetry SDKs sending OTLP via the EDOT Collector, Elastic Agent, APM Server, or the Managed OTLP endpoint. If you run a custom upstream Collector pipeline, include both the elasticapm connector and the elasticapm processor. Those components ship with the EDOT Collector (or a custom EDOT-like build); they are not part of the standard OpenTelemetry Collector Contrib distribution. For wiring details, see the upstream collector setup.
• Service map data: distributed traces that link the services in your architecture. The map draws connections from span parent-child relationships across instrumented services.
• Backend: Elastic Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

Dependency analysis walkthrough: from APM alert to shipping dependency

Step 1: APM service map on the alert detail page

When you receive a notification for a RED metric threshold breach on the checkout service, open the alert detail page.

The upgraded service map is embedded on the page, so you can start analysing dependencies the moment you land on the alert. You see checkout in context with its upstream and downstream connections without navigating away.

In our example, failed transactions have increased on checkout. The map is already scoped to the alert time range:

Step 2: Spotting a failing dependency on the service map

When we scan through the downstream dependencies, we can see some indicators that there is a problem with the checkout and shipping services. When we open the shipping service fly-out to view more details, we can see the transactions for /get-quote have an elevated failure rate:

Step 3: Drilling into failing transactions to identify RED metric drivers

Click on the /get-quote transaction to drill into these transactions. We can see a release marker for version 2.3.0 that seems to correlate to increase in failures. To validate this is the main contributing factor, we want to analyse these transactions and check a few dimensions to isolate the behaviour to the release:

Step 4: Comparing RED metrics by dimension in Traces in Discover

Using Traces in Discover, we can use the breakdown feature to compare various attributes to confirm the RED metric change is due to the release. As we break down by service.version, k8s.deployment.name and k8s.pod.name, we can see the failure rate is highly elevated for just this Kubernetes deployment.

These field names match the OpenTelemetry semantic conventions used by the demo (EDOT / OTel instrumentation). If you instrument with classic Elastic APM agents, use the ECS-style kubernetes.* equivalents instead (see ECS ↔ OTel field alignment).

Given we have clear indicators that this problem is almost certainly due to a problem caused by this Kubernetes deployment, we would next investigate the changes this deployment made so we can identify a mitigation strategy to return the system to a healthy status, most likely a rollback of the Kubernetes deployment.

Dependency analysis with the service map on custom dashboards

The APM service map can also be added as a panel to custom Kibana dashboards. If you have a custom Dashboard you would like to surface for a debugging problem with your service, you can attach it to your alert rule so users can use this to complement the default alert detail view.

When you receive the notification and land on the alert detail page, you can click Related Dashboards and open your custom dashboard. From here, you can use the same Service Map panel to quickly analyse dependencies just like the alert detail page:

What's new in the APM service map

Below is a list of the upgrades to the service map:

Service map embedded on APM alert pages and dashboards

• All APM alert detail pages will have the service map embedded to facilitate faster dependency analysis.
• There is a new Service Map Dashboard panel that you can add to any Dashboard.

Full-screen service map for large architectures

Open the map in full screen from the APM UI or from a dashboard panel. Full screen gives you more room on large architectures and works the same whether you opened the map from an alert, a service, or a dashboard.

Service map controls: search, filters, and orientation

On the service map in the APM UI and on dashboard embeds, the map toolbar adds controls to move faster on busy environments:

• Search — find services by text string

• Orientation — switch layout direction when dense graphs are hard to read

• Filters — narrow the map by alert status, whether a service has dependencies, anomaly status, and SLO status

Service map legend: node shapes, connections, and anomaly scores

A built-in legend explains node shapes, connection styles, and health indicators on the map. You spend less time decoding colours and icons when you are triaging under pressure.

The legend covers node shapes (instrumented services, databases and messaging, grouped resources), connection styles (one-way and two-way requests), and anomaly score colour rings from low through critical:

Minimap for navigating large service maps

A minimap helps you orient yourself on large maps. It also highlights anomalous services at a glance, so you can spot outliers without panning across the full graph.

RED metrics on connections with drill-through to Traces in Discover

Select a connection between two services to view RED metrics for the requests between them: rate, errors, and duration for that specific dependency edge.

From there, open Traces in Discover in one click to analyse those requests with full query flexibility. That is how we confirmed shipping was failing the requests checkout sent to it in the walkthrough above.

Adding the service map to a dashboard from the APM UI

From the service map in the APM UI, add the current map view to a dashboard in one click. You do not need to rebuild filters or time range settings manually when you want the same map on a team dashboard.

Service fly-out: preview RED metrics, anomalies and SLOs from the map

Wherever you view the service map in Kibana, you can open a service fly-out to preview a service without leaving the map.

The fly-out shows:

• RED metrics for the service
• Anomaly status — whether machine learning has flagged unusual behaviour
• SLO status — whether the service is meeting its objectives
• Transaction breakdown — how rate, errors, and duration split across transaction types

Use it to sanity-check a node before you open the full service page or follow a connection into Traces in Discover.

Summary: From APM alert to root cause with the service map

From a RED metric alert on checkout, the embedded service map showed shipping as the failing downstream dependency. Map controls, connection RED metrics, dashboard embeds, and the service fly-out are available on every map view in Kibana, so you can start dependency analysis wherever you already work.

Further Reading

You can see how we implemented these changes in the following post from Jenny Pavlova, who led the technical implementation:

6x faster at 500 services: how we rebuilt the Kibana APM service map from canvas to React DOM

Related Posts

See other recent improvements we've made for observing instrumented services in the blogs below:

• Four clicks from alert to root cause: how Elastic Observability links APM services to Kubernetes infrastructure
• Your SLO is on fire; here's how to find the arsonist in Elastic Observability
• Three clicks from alert to error log: breaking down RED metrics by any span attribute in Elastic Observability

Availability

This is available in Elastic Observability serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

Prerequisites for linking APM services to Kubernetes infrastructure

You need application traces and Kubernetes infrastructure metrics in the same Elastic Observability project.

• Application instrumentation: EDOT-instrumented services sending traces via the EDOT Collector or an upstream OpenTelemetry Collector with both the elasticapm connector and the elasticapm processor. The EDOT Collector includes both by default; for a custom upstream pipeline, see the upstream collector setup.
• Kubernetes observation: the cluster observed via OpenTelemetry with host and Kubernetes metrics from the EDOT Collector. See the Kubernetes quickstarts and Kubernetes observability with EDOT for setup.
• Backend: Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

APM alert triage: from notification to problematic pod

Step 1: Confirm the service degradation on the APM alert detail page

The redesigned alert detail page in Elastic Observability shows the impacted service, environment, endpoint and RED metrics in one view. Open it from the alert notification.

You can clearly see which service is impacted, which environment it runs in, what endpoint is being affected and easily look for correlations in their RED metrics. In this case, we can immediately rule out a spike in traffic as the throughput is clearly stable.

Step 2: Rule out service dependencies with the embedded service map

The newly embedded service map preview on the alert detail page shows the health and RED metrics of every dependent service, so you can rule out upstream causes without navigating away. In this case, we have been able to quickly rule out problems with other services causing the symptom with the symptomatic service:

Step 3: Review Kubernetes infrastructure metrics per pod, container and host

After ruling out service dependencies, open the service's updated Infrastructure tab in Elastic Observability to check for infrastructure-level patterns.

Step 4: Compare per-instance metric trends to find the root cause

The Infrastructure tab shows the average metric values over the specified time period. To really understand whether there is a problem with the infrastructure, we need to compare the pod, container and host metrics over time. This allows us to easily spot differences between different entities that may correlate with when the service started showing symptoms. In our example, we can clearly see a difference between some of the metrics between the pods that correlates with when the service symptoms began. So we know there is something going on with the infrastructure that needs investigating:

Summary: from APM alert to root cause in four clicks

In just a few clicks from an alert in Elastic Observability, you can rule out healthy dependent services without leaving the alert detail page, then compare per-pod infrastructure metrics to see which instance correlates with when the symptoms started.

What is Tail Sampling?

Distributed tracing is useful for debugging, but at production scale it comes with processing overhead and storage costs, at which point sampling becomes a natural way to maintain the value of tracing while keeping costs under control. Tail-based sampling, or tail sampling, is a technique that makes a sampling decision conditionally at a later stage, so that high-value traces like errors or slow transactions are more likely to be sampled. The opposite is head sampling, which makes the decision at the start of a trace, before any such information is available.

How does the tail sampling processor work?

The tail sampling processor buffers 100% of incoming traces (or spans, used interchangeably), then forwards the sampled subset after applying the sampling policies. Buffering is a major source of memory usage, and it scales proportionally to the volume of spans, a well known pain point in the community.

Memory usage is bounded by configuration parameters like decision_wait and num_traces. Setting decision_wait to 1 minute means a sampling decision is made for a trace after 1 minute, during which all spans for that trace are expected to have arrived. If a trace is slower than 1 minute, the decision is made with some spans missing.

As a side note, scaling out the tail sampling setup involves using the loadbalancingexporter to satisfy the requirement that all spans for a trace must be routed to the same collector. This introduces some operational complexity and potentially data loss during collector restarts. But this post focuses on the memory usage of a single tail sampling processor instance, regardless of horizontal scaling.

Why does tail sampling cause memory pressure?

These parameters introduce a tradeoff between data loss and memory usage, and they require assumptions about the shape of traces: how slow they can be, how many spans they contain, how large each span is. These assumptions can become stale as instrumentation evolves.

How much data loss is acceptable to limit memory usage, and can the tradeoff be improved? The following two contributions aim to give operators more flexibility.

How span-ingest reduces tail sampling memory by releasing spans early

sampling_strategy is a new configuration option added to the tail sampling processor in v0.149.0.

sampling_strategy defaults to trace-complete, which matches the original behavior: sampling policies are only evaluated when decision_wait has elapsed, at which point the trace is considered complete. (There is a similar config, decision_wait_after_root_received, for optimization, but it is excluded from this discussion for simplicity.) This means all spans are buffered in memory for roughly decision_wait before being released, regardless of whether a decision could have been made earlier. For example, health check spans that should always be dropped are still held in memory until policy evaluation time.

Alternatively, sampling_strategy can be set to span-ingest, where spans are evaluated individually at ingest time. This allows terminal decisions, specifically drop or sampled, to be made earlier, freeing memory by dropping or exporting all spans buffered so far for that trace before decision_wait elapses. In the health check example, a policy can be configured to drop the entire trace as soon as the root span belongs to a health check. It is worth noting that an unsampled decision, unlike an explicit drop, is not terminal, as it can be overruled by a sampled or drop decision from another span in the same trace, so unsampled traces cannot be released early.

Switching from trace-complete to span-ingest will require policy adjustments, as policies can no longer assume all spans are available at evaluation time. Moreover, not all policy types are supported with the span-ingest strategy.

Disk-backed tail sampling storage with Pebble

Even with span-ingest, all spans are still buffered in memory. As decision_wait is increased to accommodate slow traces and num_traces is increased to limit data loss, the collector will eventually hit its memory limit and get OOM killed, resulting in further data loss.

What if traces were buffered on disk instead, where there is an order of magnitude more capacity? The main drawback is performance: disk throughput and latency, even with SSDs, are at least an order of magnitude slower than memory, so disk writes need to be efficient. For this reason, Pebble, an LSM database, was chosen as the storage backend for its fast write performance. Read performance is less of a concern, as reads only happen for the sampled subset of traces when sampling_strategy is set to span-ingest.

The implementation introduces a TailStorage interface for trace storage operations, and a new tail_storage option in v0.150.0 (behind feature gate processor.tailsamplingprocessor.tailstorageextension) to configure the storage backend. The default in-memory behavior is unchanged, but it is now possible to swap in a different storage backend, like the new pebbletailstorageextension contributed to Collector Contrib.

Tail sampling memory benchmarks: trace-complete vs span-ingest with Pebble

Benchmark setup: OpenTelemetry Demo with fan-out collectors

The following benchmarks were produced by running OpenTelemetry Demo with increased load against a pipe collector, which receives all spans and fans them out to two identical collectors under observation (CUO-A and CUO-B), differing only in their tail sampling configuration. Measurements include pipe collector throughput, spans received, spans sent (sampled), CPU usage, and memory usage.

Benchmark setup diagram

                      demo ns
     +----------------------------------------+
     |  opentelemetry-demo                     |
     |    loadgenerator (locust)               |
     |    services: frontend, cart, ...        |
     |    demo-collector                       |
     +----------------------------------------+
                          |  OTLP/gRPC
                          v
                     chamber ns
     +----------------------------------------+
     |             pipe-collector             |
     |         receive once, fan out          |
     |      exporters: [otlp/a, otlp/b]       |
     +----------------------------------------+
              | OTLP                  | OTLP
              v                       v
     +----------------+      +----------------+
     |     CUO-A      |      |     CUO-B      |
     | tail_sampling  |      | tail_sampling  |
     |   (config A)   |      |   (config B)   |
     +----------------+      +----------------+

Tail sampling processor configurations

CUO-A

config:
  processors:
    tail_sampling:
      sampling_strategy: trace-complete
      decision_wait: 5m
      num_traces: 5000000
      block_on_overflow: true
      decision_cache:
        sampled_cache_size: 10000
        non_sampled_cache_size: 200000
      policies:
        - name: root_1pct
          type: and
          and:
            and_sub_policy:
              - name: root_span_only
                type: ottl_condition
                ottl_condition:
                  error_mode: ignore
                  span:
                    - "IsRootSpan()"
              - name: root_probabilistic
                type: probabilistic
                probabilistic:
                  sampling_percentage: 1.0

CUO-B

CUO-B uses the same tail sampling processor configuration as CUO-A, except it sets sampling_strategy: span-ingest and tail_storage: pebble_tail_storage/main, along with its corresponding pebbletailstorageextension configuration.

extensions:
  pebble_tail_storage/main:
    directory: /var/lib/otelcol/pebble

Memory, CPU and throughput results

The following tables compare trace-complete (CUO-A) against span-ingest with Pebble disk storage (CUO-B) across memory, CPU and throughput. The process RSS, Go heap allocation, and per-process CPU measurements come from OpenTelemetry Collector internal process and runtime metrics, while container working set and container CPU come from Kubernetes cgroup metrics scraped by kubelet/cAdvisor.

Memory (peak over the window)

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | process RSS | 916.4 MiB | 442.7 MiB | -51.7% | | Go heap alloc | 699.3 MiB | 241.7 MiB | -65.4% | | container working set | 763.0 MiB | 282.9 MiB | -62.9% |

CPU (total over the window)

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | per-process CPU | 11.9 core-s | 22.7 core-s | +90.7% | | container CPU | 11.9 core-s | 22.7 core-s | +90.1% |

Throughput (total over the window)

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | spans received | 257,804 | 257,804 | 0.0% | | spans sent | 2,477 | 2,477 | 0.0% |

Tail sampling

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | traces in memory peak | 29,284 | 29,245 | -0.1% | | traces sampled by root_1pct policy | 496 | 496 | 0.0% |

• cuo-a = trace-complete, cuo-b = span-ingest-pebble
• Window: 15m 39s (t+0:00 start, t+10:06 drain start, t+15:39 drain end)

The results show a significant memory reduction when using pebbletailstorageextension with span-ingest, at the cost of increased CPU usage from event serialization and database overhead.

What's next for OpenTelemetry tail sampling

Both sampling_strategy and pebbletailstorageextension are still in their early stages at the time of writing. Feedback and contributions are welcome in the OpenTelemetry Collector Contrib repo. Stay tuned for more improvements.

This agent fans out to five ES|QL queries over your existing traces-* data and answers Red, Yellow or Green with the root cause attached, in one response, without switching dashboards or correlating anything by hand. It runs on Elastic Agent Builder, tested on Elasticsearch and Kibana 9.3, and one deployment covers every service in your fleet with no per-service setup.

Here's how it's built

Is my service healthy? One question, one agent, one answer

There is a question every engineer dreads during an incident: "Is the service healthy?"

It sounds simple. It is not. Answering it properly means switching between dashboards, firing off multiple queries, correlating latency spikes with error logs, and checking whether downstream dependencies are contributing to the problem, all under pressure and often at an inconvenient hour.

What if that entire investigation collapsed into a single conversation?

That is what we set out to build: an AI agent that behaves like a knowledgeable SRE sitting beside you. It knows which questions to ask, knows how to query your APM data, and returns a clear Red, Yellow, or Green health verdict with context and recommendations. We call it the APM Service Health Monitor, and it runs entirely on Elastic.

Why APM health is a derived signal, not a single metric

Here is the thing about APM data: you already have everything you need. Latency percentiles, error rates, throughput, and span-level dependency traces are all sitting in your traces-* indices, indexed and ready.

The missing piece is not data. It is the reasoning layer that connects those signals into a coherent picture.

A p95 latency of 890 ms means very little on its own. But a p95 latency of 890 ms that runs 38% above the 24-hour baseline, alongside a 6% error rate on your postgres-primary dependency? That is a story. That is a Red, and it tells an engineer exactly where to look.

The APM Service Health Monitor encodes that reasoning (threshold logic, trend comparison, and dependency blast-radius analysis) into an Elastic AI agent that runs it on demand, for any service, anytime.

APM Service Health Monitor agent at the centre, fanning out to five ES|QL tools querying traces-*

How does the APM Service Health Monitor architecture work?

The agent is registered in Elastic Agent Builder as a chat-type agent wired to five ES|QL-powered tools. Each tool does one thing precisely. The agent orchestrates, compares, and reasons across all five.

POST kbn:/api/agent_builder/agents
{
  "id": "apm_service_health_agent",
  "type": "chat",
  "name": "APM Service Health Monitor",
  ...
}

The APM Service Health Monitor sits at the center, fanning out to five ES|QL tools that query traces-*.

Here is how the tool layer is composed.

Tool 1: apm_metrics_overview_tool, the 24-hour baseline

Every health assessment starts with a baseline. This tool computes the aggregate picture for a service over the last 24 hours (average latency, p95 and p99 latency, error rate, and throughput) in a single ES|QL query:

FROM traces-*
| WHERE service.name == ?service
| WHERE transaction.type == "request"
| WHERE @timestamp >= NOW() - 24 hours
| EVAL is_error = CASE(event.outcome == "failure", 1, 0)
| STATS
    avg_latency_ms  = AVG(transaction.duration.us / 1000),
    p95_latency_ms  = PERCENTILE(transaction.duration.us / 1000, 95),
    p99_latency_ms  = PERCENTILE(transaction.duration.us / 1000, 99),
    error_rate      = 100.0 * SUM(is_error) / COUNT(*),
    throughput_rps  = COUNT(*) / (24*3600)

This snapshot becomes the anchor. Every trend tool that follows compares its latest readings back to these numbers, so the agent always has a reference point, not just a raw value.

Tool 2: apm_latency_trend_tool, performance over time

Averages hide inflection points. The latency trend tool buckets p95, p75, and average latency into 5-minute intervals across the last 24 hours, giving the agent a time-series view of performance:

| STATS
    avg_latency_ms = AVG(transaction.duration.us / 1000),
    p95_latency_ms = PERCENTILE(transaction.duration.us / 1000, 95),
    p75_latency_ms = PERCENTILE(transaction.duration.us / 1000, 75)
  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)
| SORT time_bucket ASC

The agent takes the most recent 5-minute bucket's p95_latency_ms and compares it against the p95_latency_ms from apm_metrics_overview_tool, the 24-hour rolling average. That percentage difference is what gets scored Green, Yellow, or Red.

Tool 3: apm_error_trend_tool, failure shape detection

Error rate is directional. A gradual rise from 0.5% to 1.2% tells a different story than a sudden spike to 8%. The error trend tool captures this shape with 5-minute buckets:

| EVAL is_error = CASE(event.outcome == "failure", 1, 0)
| STATS
    total_requests = COUNT(*),
    error_count    = SUM(is_error),
    error_rate     = 100.0 * SUM(is_error) / COUNT(*)
  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)

The agent reads the most recent bucket's error_rate from this trend and scores it directly against the fixed thresholds (below 1% Green, 1 to 5% Yellow, above 5% Red).

Tool 4: apm_throughput_trend_tool, traffic as a health signal

Throughput is underrated as a health indicator. A 40% drop in requests per second is itself an incident. It can mean a deployment regression, a misconfigured load balancer, or a silent upstream failure.

| STATS requests_count = COUNT(*)
  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)
| SORT time_bucket ASC

The agent normalizes each 5-minute bucket's request count to a per-second rate and compares it against throughput_rps from apm_metrics_overview_tool.. The same 10% and 30% drift thresholds apply. A service that is suspiciously quiet is flagged just as quickly as one that is on fire.

Tool 5: apm_dependency_health_tool, the blast-radius view

The apm_dependency_health_tool is often the one that finds what the others cannot surface directly. A service can look internally healthy while a downstream database or external API silently accumulates failures. This tool maps every dependency via span.destination.service.resource and computes its error rate from span data:

FROM traces-*
| WHERE processor.event == "span"
  AND span.destination.service.resource IS NOT NULL
| EVAL is_error = CASE(event.outcome == "failure", 1, 0)
| STATS
    total_calls  = COUNT(*),
    failed_calls = SUM(is_error),
    error_rate   = 100.0 * SUM(is_error) / COUNT(*)
  BY dependency_name = span.destination.service.resource
| SORT error_rate DESC

Unlike the other four tools, dependency health is scored independently. Each dependency's error_rate is compared directly against fixed thresholds (1% or below Green, above 1% Yellow, above 5% Red), with no baseline comparison needed.

How the health scoring works

Once all five tools return their data, the agent computes a health state for each metric in a single reasoning step. There is no polling loop and no intermediate storage. Here is exactly what gets compared to what:

| Metric | Tool providing the value | Compared against | How scored | | :---- | :---- | :---- | :---- | | Latency (p95) | apm_latency_trend_tool | 24h p95 from apm_metrics_overview_tool | < 10% Green · 10–30% Yellow · > 30% Red| | Error rate | apm_error_trend_tool | Fixed thresholds + 24h error_rate from overview | < 1% Green · 1–5% Yellow · > 5% Red | | Throughput | apm_throughput_trend_tool | 24h throughput_rps from apm_metrics_overview_tool | < 10% Green · 10–30% Yellow · > 30% Red| | Each dependency | apm_dependency_health_tool | Fixed thresholds only (no baseline) | All ≤1% Green · Any >1% Yellow · Any >5% Red |

The overall verdict follows a single rule: the worst individual metric wins.

Any metric is Red                → Overall = 🔴 Red
Any metric is Yellow (no Red)   → Overall = 🟡 Yellow
All metrics are Green            → Overall = 🟢 Green

The Red/Yellow/Green scoring logic lives in the agent's natural-language instructions. It is readable, auditable, and adjustable without touching code or redeploying infrastructure.

Four metric columns (Latency, Error Rate, Throughput, Dependencies) converging into Red/Yellow/Green outcome states

Deploying the APM Service Health Monitor in Agent Builder

Registration follows Elastic's standard Agent Builder API pattern, tested against Elasticsearch 9.3 and Kibana 9.3. Register the tools first, then wire the agent to all five tool IDs:

# Step 1: Register the five ES|QL tools
POST kbn:/api/agent_builder/tools   # apm_metrics_overview_tool
POST kbn:/api/agent_builder/tools   # apm_latency_trend_tool
POST kbn:/api/agent_builder/tools   # apm_error_trend_tool
POST kbn:/api/agent_builder/tools   # apm_throughput_trend_tool
POST kbn:/api/agent_builder/tools   # apm_dependency_health_tool

# Step 2: Register the agent with tools wired in
POST kbn:/api/agent_builder/agents  # apm_service_health_agent

# Step 3: Verify the agent is live
GET kbn:/api/agent_builder/agents/apm_service_health_agent

Each tool is parameterized with a service binding, so the same agent serves every service in your fleet with no per-service configuration. The queries target traces-* with cross-cluster wildcard support, so a single deployment covers multi-cluster environments out of the box.

Try it: ask the agent about your service

Once deployed, open the agent in Kibana and type this:

What is the health of my checkout-service in the last 24 hours?

The agent fans out to all five tools, computes the metric states, applies the health logic in one pass, and responds with a structured report.

Kibana chat card: checkout-service · Last 24h · per-metric table with Red/Yellow/Green status pills · Overall: 🔴 Red

The agent identified the root cause without any manual correlation: postgres-primary is failing 6.1% of its calls, and that is cascading directly into the p95 latency spike. No dashboard pivoting, no manual ES|QL. One prompt, full situational awareness.

You can continue the conversation with follow-up questions in the same session:

• Which other services depend on postgres-primary?
• How does this compare to yesterday's health?
• Show me the error trend for the last 6 hours only.

The agent invokes the appropriate tools for each follow-up, keeping the full context of the original health assessment in view.

Why Agent Builder and ES|QL are the right stack for this

A few deliberate choices made this design work cleanly.

ES|QL as the query layer

ES|QL's pipe-based syntax makes each tool query readable, testable, and independently verifiable. The DATE_TRUNC bucketing for trends, the PERCENTILE aggregations for latency, and the span.destination.service.resource grouping for dependencies are precise, auditable queries. Run any of them directly in Kibana Dev Tools and you will see exactly what the agent sees.

Narrow, stateless tools

Each tool does one thing and returns structured data. The agent provides the orchestration and reasoning. Adding a new metric dimension means registering one new tool and updating the agent instructions. Nothing else changes.

Instructions as a runbook

The agent's health logic is expressed in natural language inside its configuration. Anyone on your team can read it, adjust it without a code deploy, and audit it in full from the Agent Builder UI in Kibana.

What's next for the APM Service Health Monitor

The APM Service Health Monitor is a foundation, not a ceiling. Natural extensions include:

• Alert-triggered health checks: wire the agent to fire automatically when an anomaly detection rule triggers, attaching the health summary directly to the alert notification.

• Deployment correlation: integrate change event data so the agent can identify whether a Red status started after a specific deployment.

• SLO-aware thresholds: replace fixed percentage thresholds with per-service error budget consumption, so Red, Yellow, and Green reflect actual business impact against defined SLOs.

• Cross-service traversal: extend the dependency tool to recursively assess upstream and downstream services, building a full topology view from a single query.

Requirements and deployment

Requirements:

• Elasticsearch 9.3 and Kibana 9.3 (tested version).
• Agent Builder enabled in Kibana.
• APM data flowing into traces-* indices via any Elastic APM agent.
• Elastic Enterprise license is required to utilize Agent Builder and ES|QL cross-cluster search.

The full agent and tool definitions are available in this repository. Once your traces are indexed, all five ES|QL tools work against your data with no modification, parameterized by service name at query time, covering your entire service fleet from a single deployment.

The next time someone asks "is the service healthy?", you will have a precise, data-backed answer before the question finishes echoing in the incident channel.

Questions, extensions, or feedback? Join the conversation in the Elastic community forums.

Full deployment reference

Everything you need to deploy the APM Service Health Monitor, in execution order. Run the five tool registrations first, then the agent.

Tool 1: Dependency health

POST kbn:/api/agent_builder/tools
{
  "id": "apm_dependency_health_tool",
  "type": "esql",
  "description": "Evaluates the health of external dependencies (DBs, APIs, caches)...",
  "configuration": {
    "query": "FROM traces-*\n| WHERE service.name == ?service\n| WHERE processor.event == \"span\"\n  AND span.destination.service.resource IS NOT NULL\n| WHERE @timestamp >= NOW() - 24 hours\n| EVAL is_error = CASE(event.outcome == \"failure\", 1, 0)\n| STATS\n    total_calls = COUNT(*),\n    failed_calls = SUM(is_error),\n    error_rate = 100.0 * SUM(is_error)/COUNT(*)\n  BY dependency_name = span.destination.service.resource\n| SORT error_rate DESC | LIMIT 100"
  }
}

Tool 2: Error trend

POST kbn:/api/agent_builder/tools
{
  "id": "apm_error_trend_tool",
  "type": "esql",
  "description": "Tracks 5-minute bucketed error rates for the service over 24 hours...",
  "configuration": {
    "query": "FROM traces-*\n| WHERE service.name == ?service\n| WHERE transaction.type == \"request\"\n| WHERE @timestamp >= NOW() - 24 hours\n| EVAL is_error = CASE(event.outcome == \"failure\", 1, 0)\n| STATS\n    total_requests = COUNT(*),\n    error_count = SUM(is_error),\n    error_rate = 100.0 * SUM(is_error)/COUNT(*)\n  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)\n| SORT time_bucket ASC | LIMIT 1000"
  }
}

Tool 3: Latency trend

POST kbn:/api/agent_builder/tools
{
  "id": "apm_latency_trend_tool",
  "type": "esql",
  "description": "Provides 5-minute bucketed latency trends (avg, p95, p75) for last 24 hours...",
  "configuration": {
    "query": "FROM traces-*\n| WHERE service.name == ?service\n| WHERE transaction.type == \"request\"\n| WHERE @timestamp >= NOW() - 24 hours\n| STATS\n    avg_latency_ms = AVG(transaction.duration.us / 1000),\n    p95_latency_ms = PERCENTILE(transaction.duration.us / 1000, 95),\n    p75_latency_ms = PERCENTILE(transaction.duration.us / 1000, 75)\n  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)\n| SORT time_bucket ASC | LIMIT 1000"
  }
}

Tool 4: Metrics overview

POST kbn:/api/agent_builder/tools
{
  "id": "apm_metrics_overview_tool",
  "type": "esql",
  "description": "Aggregates key service metrics over last 24 hours: avg latency, p95/p99, error rate, throughput...",
  "configuration": {
    "query": "FROM traces-*\n| WHERE service.name == ?service\n| WHERE transaction.type == \"request\"\n| WHERE @timestamp >= NOW() - 24 hours\n| EVAL is_error = CASE(event.outcome == \"failure\", 1, 0)\n| STATS\n    avg_latency_ms = AVG(transaction.duration.us / 1000),\n    p95_latency_ms = PERCENTILE(transaction.duration.us / 1000, 95),\n    p99_latency_ms = PERCENTILE(transaction.duration.us / 1000, 99),\n    error_rate = 100.0 * SUM(is_error) / COUNT(*),\n    throughput_rps = COUNT(*) / (24*3600)"
  }
}

Tool 5: Throughput trend

POST kbn:/api/agent_builder/tools
{
  "id": "apm_throughput_trend_tool",
  "type": "esql",
  "description": "Provides service throughput trends in 5-minute intervals over 24 hours...",
  "configuration": {
    "query": "FROM traces-*\n| WHERE service.name == ?service\n| WHERE transaction.type == \"request\"\n| WHERE @timestamp >= NOW() - 24 hours\n| STATS requests_count = COUNT(*)\n  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)\n| SORT time_bucket ASC | LIMIT 1000"
  }
}

Agent: APM service health monitor

POST kbn:/api/agent_builder/agents
{
  "id": "apm_service_health_agent",
  "name": "APM Service Health Monitor",
  "description": "Provides Red/Yellow/Green health status and trends for services over 24 hours.",
  "labels": ["apm", "health", "service", "monitoring"],
  "avatar_color": "#4CAF50",
  "configuration": {
    "instructions": "You are the Service Health Agent...",
    "tools": [{
      "tool_ids": [
        "apm_dependency_health_tool",
        "apm_error_trend_tool",
        "apm_latency_trend_tool",
        "apm_metrics_overview_tool",
        "apm_throughput_trend_tool"
      ]
    }]
  }
}

Prerequisites for Elastic Agent Builder MCP tools

• An Elastic Cloud deployment with Elastic Stack 9.3+ (or Elastic Cloud Serverless) with Agent Builder enabled.
• An APM-ingested service. If your cluster does not already have APM data, the companion notebook includes instructions to generate synthetic traffic using elastic/apm-integration-testing with the opbeans-node demo app.
• An MCP-compatible client: Claude Code, Cursor, or VS Code with an MCP extension.
• Basic familiarity with ES|QL syntax.
• Node.js 18+ (for the mcp-remote bridge).

If you are new to MCP or need to set up the Elastic MCP server for the first time, check out Connect Agent Builder tools to any AI agent with Elastic MCP server for the full setup walkthrough. This article assumes the MCP server is already configured.

The problem: why developers fly blind

A developer is about to merge a pull request. The change looks simple: increasing the timeout for the downstream recommendations service call from 2 seconds to 5 seconds. But before hitting the merge button, a question lingers: is the service healthy enough to absorb this change right now?

To answer that question today, the developer has two options:

1. Check dashboards manually. Open the APM UI, look at error rates, scan latency charts, find the SLO page, and look for recent deploys. This takes 10-15 minutes and requires knowing what to look for and how to interpret it.
2. Ask an SRE. Ping the platform team on Slack: "Hey, is checkout healthy? I want to merge something." This creates an interruption, adds latency to the decision, and doesn't scale.

The core problem is not the data. Elastic already collects everything: traces, metrics, error logs, deploy markers, and SLO budgets. The problem is that correlating multiple signals requires mental overhead and domain knowledge that most developers don't have.

An SRE knows that a p99 spike after a deploy is normal for 5 minutes, that an error rate under 0.5% is acceptable during a release window, and that merging when the SLO budget is below 20% is risky. That knowledge lives in runbooks, tribal memory, and experience.

What if the platform engineer could encode that knowledge into tools that any developer can query from their editor?

How MCP tools in Elastic Agent Builder encode your runbook

The key insight is this: a tool is not just a query; it is a query plus interpretation. A dashboard shows you a p99 of 450ms. A well-designed tool tells you "p99 is 450ms, which is within normal range for this service, and has been stable since the last deploy 2 hours ago."

The difference is that the tool description carries the domain knowledge. When a platform engineer creates a tool in Agent Builder, they write descriptions like: "Error rate above 1% typically indicates a regression. If this coincides with a recent deploy, the deploy is the likely cause." That description becomes part of the context the AI agent uses when reasoning across multiple tool results.

This is what we mean by contextual AI: the AI agent does not just fetch data; it reasons over it using the interpretation rules that the platform team encoded.

Here is the architecture:

The platform engineer authors the tools once. Every developer on the team benefits from their own editor, without needing to learn ES|QL or understand APM data models.

Setting up the Elastic Agent Builder sample environment

The full end-to-end setup (traffic generation with opbeans-node, deploy annotations, SLO creation, and the three Agent Builder tools) is available as a runnable notebook at this repository: notebook.ipynb. The sections below focus on the ES|QL queries and tool descriptions: the why behind each tool, not the mechanics of posting them.

Building Tool 1: get_endpoint_health

This tool answers the question: "How is this endpoint performing right now?" It returns error rate, latency percentiles (p50, p95, p99), and throughput for a given service and endpoint within a time window.

Here is the full tool configuration as created in Agent Builder:

{
  "id": "get_endpoint_health",
  "type": "esql",
  "description": "Returns the current health of a service endpoint: error rate, latency percentiles (p50/p95/p99), and throughput. Use this tool to assess whether a service is healthy before making changes. Interpretation guide: error rate below 0.5% is healthy, 0.5-1% is elevated (check for recent deploys), above 1% indicates a problem. For latency, compare p99 against the service baseline: checkout is typically under 500ms, product-search under 200ms. A sudden p99 spike within 15 minutes of a deploy suggests the deploy caused a regression.",
  "tags": ["apm", "reliability", "health"],
  "configuration": {
    "query": "FROM traces-apm-* | WHERE service.name == ?serviceName AND @timestamp >= NOW() - ?timeWindow AND transaction.duration.us IS NOT NULL | STATS total_transactions = COUNT(*), error_count = SUM(CASE(event.outcome == \"failure\", 1, 0)), p50_latency_ms = PERCENTILE(transaction.duration.us, 50) / 1000, p95_latency_ms = PERCENTILE(transaction.duration.us, 95) / 1000, p99_latency_ms = PERCENTILE(transaction.duration.us, 99) / 1000 BY service.name | EVAL error_rate_pct = ROUND(error_count / total_transactions * 100, 2) | EVAL throughput_per_min = ROUND(total_transactions / ?windowMinutes, 1)",
    "params": {
      "serviceName": {
        "type": "keyword",
        "description": "The APM service name to check (e.g., opbeans-node)"
      },
      "timeWindow": {
        "type": "keyword",
        "description": "Time window to analyze, in ES|QL duration format (e.g., 30 minutes, 1 hour, 6 hours)"
      },
      "windowMinutes": {
        "type": "integer",
        "description": "Time window in minutes, used to calculate throughput per minute"
      }
    }
  }
}

The query uses the traces-apm-* data stream, which contains raw transaction data. We filter with transaction.duration.us IS NOT NULL to select only transaction events (excluding spans). Using traces-apm-* is more portable than the pre-aggregated metrics-apm.transaction.1m-* stream, which only populates after sustained traffic.

Notice the description field. It is not just "returns health metrics." It includes interpretation rules: what error rate thresholds mean, what latency baselines look like, and how to correlate spikes with deploys. This is the runbook encoded in the tool.

Building Tool 2: get_recent_deploys

This tool answers: "What has been deployed recently?" Deploy history is a critical context because most production issues correlate with code changes. The agent needs this to reason about whether current metrics are normal or reflect a recent deployment.

Deploy annotations are stored in the observability-annotations index. Here is the full tool configuration:

{
  "id": "get_recent_deploys",
  "type": "esql",
  "description": "Returns the deployment history for a service over the last 24 hours, including version numbers, timestamps, and deploy messages. Use this tool to understand the deployment timeline when assessing service health. Key patterns: if a deploy happened within the last 15 minutes, elevated error rates or latency may be expected (warm-up period). If metrics degraded immediately after a deploy, the deploy is the likely cause. Multiple deploys in a short window (under 2 hours) increase risk because it becomes harder to isolate which change caused an issue.",
  "tags": ["apm", "deploys", "change-tracking"],
  "configuration": {
    "query": "FROM observability-annotations | WHERE service.name == ?serviceName AND @timestamp >= NOW() - 24 hours | SORT @timestamp DESC | KEEP @timestamp, service.version, service.environment, message | LIMIT 10",
    "params": {
      "serviceName": {
        "type": "keyword",
        "description": "The APM service name to check deploy history for"
      }
    }
  }
}

Again, the description encodes domain knowledge: the 15-minute warm-up window, the correlation between deploys and metric changes, and the risk of multiple rapid deploys. This is how a platform engineer transfers their intuition into something an AI agent can reason with.

Building Tool 3: get_slo_status

This tool answers: "How much error budget do we have left?" SLO budget is the platform team's quantified way of expressing risk tolerance. If the budget is nearly spent, even a small change could cause a violation.

Unlike the previous tools that query APM data, this one queries the internal SLO indices where Elastic stores pre-computed SLI data. The query calculates the current burn rate, that is, how fast the service is consuming error budget relative to the allowed threshold:

{
  "id": "get_slo_status",
  "type": "esql",
  "description": "Returns the current SLO burn rate for a service over the last hour. The response includes: SLI value (current performance), error budget target, and burn rate percentage. The burn rate tells you how fast the service is consuming error budget relative to the allowed threshold. Interpretation: a burn rate below 100% means the service is consuming budget slower than the limit (sustainable). Between 100-200%, the service is burning budget faster than planned (proceed with caution). Above 200%, the service is burning budget at double the allowed rate (delay non-critical changes). Above 500%, investigate immediately. Note: this measures the current burn rate over the last hour, not cumulative budget consumption over the full SLO window. A temporarily high burn rate does not mean the overall budget is exhausted.",
  "tags": ["slo", "reliability", "budget"],
  "configuration": {
    "query": "FROM .slo-observability.sli-v* | WHERE slo.id == ?sloId AND @timestamp >= NOW() - 1 hour | STATS sli_value = AVG(slo.numerator) / AVG(slo.denominator) BY slo.id, slo.name | EVAL error_budget_target = 0.995 | EVAL burn_rate_pct = ROUND((1 - sli_value) / (1 - error_budget_target) * 100, 1)",
    "params": {
      "sloId": {
        "type": "keyword",
        "description": "The SLO identifier. Use the SLO ID for the service you are evaluating."
      }
    }
  }
}

Note on the SLI index: the version suffix in .slo-observability.sli-v* depends on your Stack release (e.g., v3.6 in Stack 9.3). Verify with GET _cat/indices/.slo-observability.*?v and adjust the pattern if your cluster uses a different version.

The burn rate interpretation rules in the description are the most valuable part. A raw number like "burn rate 85%" means nothing to a developer without context. The tool description translates that into actionable guidance: "below 100% means sustainable, above 200% means delay non-critical changes."

Connecting to your editor via MCP

With all three tools created in Agent Builder, they are automatically available through the MCP server endpoint. Configure your MCP client to connect.

Claude Code configuration

Add the Elastic MCP server to your Claude Code settings:

{
  "mcpServers": {
    "elastic-agent-builder": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://your-kibana-url/api/agent_builder/mcp",
        "--header",
        "Authorization: ApiKey your-base64-api-key"
      ]
    }
  }
}

Cursor configuration

For Cursor, add the server in Settings > MCP Servers:

{
  "mcpServers": {
    "elastic-agent-builder": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://your-kibana-url/api/agent_builder/mcp",
        "--header",
        "Authorization: ApiKey your-base64-api-key"
      ]
    }
  }
}

Once connected, your editor's AI agent will discover all three tools automatically. You can verify by asking: "What Elastic tools do you have available?" The agent should list get_endpoint_health, get_recent_deploys, and get_slo_status.

API key permissions: the API key needs the feature_agentBuilder.read Kibana privilege and read access to the relevant indices (traces-apm.*, observability-annotations, .slo-observability.*). For production use, set the key expiry to 30-90 days and follow the principle of least privilege.

The scenario: "Is it safe to merge this PR?"

A developer on the team has a pull request that increases the timeout for the downstream recommendations service call from 2 seconds to 5 seconds in opbeans-node. Before merging, they ask the agent:

Developer: "I'm about to merge PR #42, which increases the recommendations service timeout from 2s to 5s in opbeans-node. Is it safe to merge right now?"

The agent begins its multi-signal reasoning chain. Here is what happens.

Step 1: the agent calls get_endpoint_health

The agent checks the current health of the service:

Step 2: the agent calls get_recent_deploys

Next, it checks for recent deployments:

Step 3: the agent calls get_slo_status

The agent's response

After correlating all three results, the agent produces a recommendation:

The agent pulled the current p99, checked recent deploys, and read the SLO burn rate. It combined those signals with the timeout change in the PR, flagged the merge as risky, and recommended next steps.

Conclusion: when to use MCP tools instead of pinging an SRE

With Elasticsearch, Agent Builder, and MCP, a developer can answer questions like "is it safe to merge this PR?" from inside their editor, in seconds, without pinging an SRE. Elasticsearch holds the signals: traces, deploy markers, and SLO budgets. Agent Builder is where the platform team encodes how to read those signals: the thresholds, the warm-up windows, the correlation rules. MCP is what carries those tools into the developer's editor.

The query pulls the data. The description tells the agent how to read it. The platform engineer writes the runbook once, and every developer on the team gets to use it.

Next steps: extend Elastic Agent Builder MCP tools to CI/CD

• Explore the Elastic Agent Builder documentation for more tool types and configuration options.
• See Agentic CI/CD: Kubernetes Deployment Gates with Elastic MCP Server for extending this pattern into your CI/CD pipeline.
• Check out Agent Skills for Elastic Observability for a complementary approach using pre-packaged observability skills.

The processing pipeline is straightforward: The profiler samples every CPU core on the system at a fixed rate (19Hz by default), unwinds execution stacks, symbolizes the resulting stacktraces and ships the profiles to a backend like Elasticsearch.

And then… the user has to figure out what to do with them.

That last step is where continuous profiling has historically faced adoption challenges, as the path from "profiling is on" to "profiling is useful" is steeper than it should be.

Four barriers to adoption

• Storage cost: Full stacktraces, even after deduplication and clever storage schemas, are expensive to store at fleet scale. That cost makes continuous profiling an opt-in feature in practice: a lot of potential users never enable it, and the ones who do, tend to enable it only on a subset of hosts.

• Query friction: A normalized stacktrace schema is optimized for ingestion and storage but complicates ad-hoc questions. "How much CPU time does my service spend in TLS?" is a simple question that may require intricate ES|QL or custom code in order to be answered.

• AI-hostile data: Normalized stacktrace data (typically involving multiple levels of indirection) resists straightforward algorithmic analysis. LLMs in particular struggle with it and necessitate further data transformations into representations more amenable to LLM processing.

• UX barrier: Flamegraphs are extremely useful when you know how to read them but intimidating when you don't.

These four barriers compound: storage cost limits coverage, the UX barrier limits who benefits from coverage, query friction limits what questions users can ask and the AI-hostile data representation limits what the system can do when users don't know what questions to ask.

How profiling-derived metrics work: classify at the edge

The core idea is simple: instead of sending full stacktraces all the way to a backend and asking the user to make sense of them there, we classify and count at the edge, inside an OpenTelemetry Collector pipeline, and emit ordinary OpenTelemetry time-series counters. The profiling logic itself doesn't change; it's still the OpenTelemetry eBPF profiler running inside the OpenTelemetry Collector. All the new work happens in a stateless connector inside the Collector: the connector inspects each stacktrace produced by the profiler, classifies its frames into one or more categories and increments counters.

We've released profilingmetricsconnector as part of Elastic's opentelemetry-collector-components repository. It sits between the OpenTelemetry eBPF profiler receiver and any metrics exporter, and turns symbolized stacktraces into named, aggregated counters with attributes.

Because the profilingmetricsconnector lives inside the standard OpenTelemetry Collector pipeline, every metric it produces flows through the same processors as the rest of your telemetry. In the following example, the resourcedetectionprocessor enriches each counter with host-derived attributes.

connectors:
  profilingmetrics:
    flush_interval: 30s

receivers:
  profiling: {}

exporters:
  elasticsearch:
    endpoints:
      - # ENDPOINT
    api_key: # API_KEY
    mapping:
      mode: otel

processors:
  resourcedetection:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: false
        host.arch:
          enabled: true
        os.description:
          enabled: true
        os.type:
          enabled: true

service:
  pipelines:
    profiles:
      receivers: [ profiling ]
      exporters: [ profilingmetrics ]
    metrics:
      receivers: [ profilingmetrics ]
      processors: [resourcedetection]
      exporters: [ elasticsearch ]

Profiling-derived CPU metrics: what gets emitted

The connector ships with a set of pre-baked counters built from useful classification rules. Each metric is a count of stacktrace samples whose leaf frame matched a particular category, with the frequency value standing in for CPU consumption.

| Metric | Classifies | Attached metadata | |---|---|---| | kernel.count | Kernel leaf frames | syscall, category (disk/rw, ipc/rw, network/{tcp,udp,other}/rw, memory, synchronization, …) | | native.count | Native C/C++/Rust leaf frames | shared library name (libcrypto, libclrjit, libsystemd, …) | | hotspot.count, go.count, python.count, … | Runtime-specific leaf frames | runtime-specific attributes |

The kernel categorization is worth a closer look as a modern Linux kernel has more than 400 system calls. However, most of what shows up in CPU stacktraces falls into a handful of subsystems: filesystem read/write, network read/write, memory management, scheduling, synchronization. Some syscalls (e.g. read, write) are ambiguous on their own and only become specific when one examines more frames down the stack: ext4_file_read_iter points to filesystem, tcp_v4_rcv to network. The connector handles this disambiguation as part of frame iteration.

Native frames typically lack symbolic information beyond shared library names, but those names are still informative: libssl and libcrypto mean cryptographic work as part of OpenSSL or one of its variants; libz means compression; libclrjit means the .NET JIT is busy. We don't need to enumerate libraries statically as the connector dynamically generates shlib_name attribute values using the trimmed library name (e.g. libssl not libssl.so.3) for clean cardinality.

Currently, for each stacktrace, the connector computes a Self CPU count (the leaf frame matched the category) corresponding to exclusive CPU usage. A complication exists for fine-grained kernel categories like network/tcp/write where the actual leaf frame is usually a device-driver call that we can't meaningfully match. We deal with that by trying to match frames further up the stack (e.g. tcp_sendmsg is enough to correctly classify the sample).

Users can also add their own categories by specifying a frame pattern (e.g. a function or package) and the connector will generate counters for them.

Benefits of profiling-derived metrics for observability

This shift looks small from the outside — "we're emitting counters" — but it changes almost everything about how profiling fits into an observability stack.

• Orders of magnitude less storage: A counter aggregated over a 5-second (or 30-second or one-minute) window is dramatically cheaper than the full stacktraces it distills. The pre-aggregation interval is configurable with the trade-off being time resolution rather than categorization fidelity. For most "where is my CPU being spent?" questions, 30 seconds is plenty.

• On by default: Because the storage cost is now in line with regular metrics, profiling-derived metrics can be on for everyone, on every host, from the moment the profiler is deployed. Users get a CPU breakdown by runtime, syscall, kernel category and shared library on day one.

• Standard dashboards: These are ordinary OpenTelemetry time-series counters and can be visualized ad-hoc using stacked bar graphs, pie charts, top-N panels or any other visualization Kibana supports out of the box. The same Lens and TSDB-backed views for application metrics work here.

• AI and query-friendly: Standard time-series data is trivially consumable by ES|QL, ML jobs, anomaly detectors and by LLMs. "Show me the top services by network/udp/write time, filtered to the payments namespace, over the last six hours" is one query that is not only simple for the system to answer but also simple for an LLM to generate.

• Cross-signal correlation: Because the metrics flow through the standard OpenTelemetry Collector pipeline, they pick up the same resource attributes (e.g. service.name, k8s.pod.name, host.name, deployment.environment) that logs, other metrics and traces already carry.

• Instant value, with a path to more detail: A user who just wants to know "what's burning my CPU?" gets a meaningful answer without ever opening a flamegraph. A user who wants to dig deeper still has the full eBPF profiler underneath, ready to hand back complete stacktraces when they're warranted.

User-programmable profiling and adaptive sampling

The longer-term direction is for the profiler to stop being something users consume and start being something they program. User-defined metrics are the first step in this direction, complemented by on-demand (full) profiling and adaptive sampling.

Profiling-derived metrics or other signals can act as a trigger for on-demand profiling where the system enables full profiling on a specific host or service to capture complete stacktraces. In that way, the full profiling processing and storage cost is paid only when it matters.

We can apply the same idea to the sampling rate. 19Hz is a sensible baseline for steady state but when the metrics signal an interesting event or an anomaly, the system can automatically ramp to 100Hz or higher to capture high-fidelity data for the time window during which it's relevant. It can then ramp down to baseline.

How profiling-derived metrics enable self-driving observability

Most observability stacks today use an open-loop model: the profiler emits data with a fixed configuration. Then a human looks at flamegraphs and dashboards, potentially correlates with logs, other metrics and traces, forms a hypothesis and triggers a deeper investigation. Every link in this chain requires a human decision. Nothing feeds back into the profiler at speed and the system cannot act on its own observations.

Profiling-derived metrics close that loop.

1. A "significant host events" metric, an anomaly on network/udp/write or a spike in native.count/libz: something crosses a threshold.

2. The profiler adjusts in response: sampling rate increases, full profiling turns on for the affected hosts.

3. The richer data is correlated against logs, traces, and other metrics by an LLM, by a human or both. The same resource attributes that make cross-signal correlation easy for the user make it easy for the system.

4. A root cause is identified. A remediation is suggested or applied. The metric returns to baseline and the loop continues.

This is what we mean when we talk about self-driving observability. The profiler is no longer just an instrument that someone wields. It is the sensory organ of an autonomous feedback loop: a system that observes itself, decides what to look at more closely and adjusts its own configuration in response to what it sees.

What's next: inclusive CPU, off-CPU metrics, and runtime-specific profiling

Any piece of data visible in a stacktrace can be a metric source and several extensions are already on the roadmap.

• Inclusive-CPU metrics: Today's pre-baked counters attribute CPU at the leaf frame (exclusive-CPU). Inclusive-CPU metrics will attribute the entire call chain which is useful when you care about the total cost of a function call — the function plus everything it transitively calls — not just the work done directly in its own body.

• Runtime-specific metrics: GC time per runtime, JSON/Protobuf serialization, RPC frameworks, FFI boundaries. The kinds of questions every team eventually asks about their language runtime, answered by default.

• Off-CPU metrics: On-CPU profiling tells you where you're spending CPU but Off-CPU profiling tells you where you're not (e.g. blocked on I/O, locks). The same classification logic applies, with the only change being the source signal.

Profiling-derived metrics are an active area of work within Elastic and the profilingmetricsconnector is the place to start if you want to play with this today. A ready-made Kibana integration ships dashboards for all the metrics described above.

If you're already using Elastic's continuous profiling, expect these metrics to show up as first-class citizens in the Elastic stack. If you're not, this is a very low-friction way in as no flamegraph expertise is required and storage cost is minimal.

The flamegraph isn't going anywhere, but for the first time, it isn't the only way profiling yields results.

Prerequisites

• Elastic Cloud hosted (9.3+) or serverless

• Claude Desktop

• An MCP server instrumented with the Elastic Distribution of OpenTelemetry (EDOT). We cover how to instrument one below.

The observability gap in MCP servers

MCP servers ship with no built-in observability, which means tool-call latency, errors, and performance baselines are invisible to developers. MCP (Model Context Protocol) servers are increasingly used as infrastructure for AI-powered applications, giving AI models access to databases, APIs, internal tools, and business data. The MCP SDK does not instrument any of it.

The gap shows up in three concrete ways:

• When a tool call takes 3 seconds, you don't know if the bottleneck is in your business logic, a downstream API, or the data layer.
• When a tool call fails, you get the error message but no context about what the server was doing before it failed.
• When you add a new tool, you have no baseline to compare its performance against.

These are the same problems any backend service faces. The answer is the same one backend developers have used for years: distributed tracing with OpenTelemetry.

MCP servers are standard programmatic processes. There is nothing special about them from an instrumentation perspective. You add the OTel SDK, define spans around your tool handlers, and ship traces to your backend. The only new part is knowing which span names and attributes to use so your traces are meaningful and consistent.

What we built

For this article, we use the @modelcontextprotocol/server-everything package, the official reference MCP server published by Anthropic. It ships with a set of tools that cover the common patterns you will find in real-world MCP servers: simple request/response, parameterized calls, long-running operations, and calls that return structured data.

The server exposes several tools. In this article we use three of them:

• echo: receives a string and returns it unchanged. A minimal request/response tool, useful for verifying that the instrumentation pipeline works end to end.

• get-sum: receives two numbers and returns their sum. Represents a parameterized tool with simple business logic.

• trigger-long-running-operation: starts a multi-step operation that takes several seconds to complete. Simulates tools that call downstream APIs or run expensive computations.

We instrumented it with EDOT Node.js (@elastic/opentelemetry-node), which is Elastic's distribution of the OpenTelemetry SDK. EDOT replaces the five or six individual OTel packages you would otherwise install and adds the elasticapm connector that the Kibana APM UI needs to build its service maps, transaction groupings, and latency charts. Without that connector, raw OTLP data arrives in Elasticsearch but the APM views have nothing to build from.

The architecture looks like this:

This is the loop: Claude executes tools, generates telemetry, and then uses a second MCP to analyze that telemetry. The observability data becomes something the AI can reason about, not just something that sits in a dashboard waiting for a human to check it.

Both MCP servers are active simultaneously in Claude Desktop. The instrumented MCP generates telemetry. The Agent Builder MCP lets us query it.

To connect Claude Desktop to both servers, the claude_desktop_config.json looks like this:

{
  "mcpServers": {
    "everything": {
      "command": "node",
      "args": [
        "--import",
        "/path/to/node_modules/@elastic/opentelemetry-node/import.mjs",
        "/path/to/everything/dist/index.js",
        "stdio"
      ],
      "env": {
        "OTEL_SERVICE_NAME": "everything-mcp-server",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "https://<your-otlp-endpoint>",
        "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=ApiKey <your-api-key>",
        "OTEL_LOG_LEVEL": "none"
      }
    },
    "elastic-agent-builder": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://<your-kibana-url>/api/agent_builder/mcp",
        "--header",
        "Authorization:ApiKey <your-api-key>"
      ]
    }
  }
}

The --import /path/to/@elastic/opentelemetry-node/import.mjs flag is all it takes for zero-code auto-instrumentation. But auto-instrumentation only captures HTTP calls, database queries, and other Node.js instrumented libraries. MCP tool calls are application logic, and application logic needs manual spans.

What traces MCP tool calls generate

The OpenTelemetry specification includes official semantic conventions for MCP. Following them means your traces are consistent, searchable by name across tools and teams, and compatible with any OTel-aware backend, including Elastic APM.

Span naming follows the pattern {mcp.method.name} {target}. For a tool call, this becomes tools/call echo or tools/call get-sum. This is what you will see as the transaction name in Kibana APM.

Key attributes on each span:

| Attribute | Value | Purpose | | ----------------------- | ---------------- | ------------------------- | | mcp.method.name | tools/call | The MCP protocol method | | gen_ai.tool.name | echo | The specific tool invoked | | gen_ai.operation.name | execute_tool | GenAI semantic convention | | error.type | error class name | Set only on failure |

The wrapper pattern that creates these spans looks like this:

const tracer = trace.getTracer('everything-mcp-server', '1.0.0');

function withToolSpan(toolName, fn) {
  return tracer.startActiveSpan(`tools/call ${toolName}`, (span) => {
    span.setAttribute('mcp.method.name', 'tools/call');
    span.setAttribute('gen_ai.tool.name', toolName);
    span.setAttribute('gen_ai.operation.name', 'execute_tool');

    try {
      const result = fn();
      span.end();
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      span.setAttribute('error.type', err.constructor.name);
      span.end();
      throw err;
    }
  });
}

Each tool handler wraps its logic in withToolSpan. The result is a named span in Elastic APM for every tool invocation, with duration, status, and error details attached.

Security note: The OTel spec defines two optional attributes for tool calls: gen_ai.tool.call.arguments and gen_ai.tool.call.result_. Both are flagged as potentially containing sensitive data. The_ get-env tool in the everything server is a good example of why this matters: it returns all environment variables, which may include API keys and credentials. Capture these attributes only if you have confirmed the data is safe to store in your observability backend, and consider masking or filtering at the SDK level before export.

Kibana APM: exploring MCP server performance

After starting Claude Desktop with both MCPs configured and triggering a few tool calls, the everything-mcp-server service appears in Kibana under Observability > Applications > Services Inventory:

Transactions view

Kibana groups traces by transaction name. Because we follow the semantic conventions, each tool gets its own row: tools/call echo, tools/call get-sum, tools/call trigger-long-running-operation. You can immediately see latency, throughput, and error rate per tool without any configuration.

This is where the value of consistent span naming becomes concrete. If you have five different developers adding tools to an MCP server and everyone follows tools/call {toolName}, the APM UI stays organized automatically.

Trace waterfall

Clicking on a specific trace shows the waterfall view. For a single tool call, the waterfall is straightforward: one span covering the full execution. If your tool handler makes downstream HTTP requests or database queries that are auto-instrumented, those appear as child spans. You can see exactly how much time was spent in business logic versus waiting for external calls.

Latency distribution

The latency chart shows p50, p95, and p99 distribution across all executions of a given tool. This makes it easy to distinguish between tools that are consistently fast and those that have occasional outliers. The trigger-long-running-operation tool, for example, shows a wide distribution depending on how many steps were requested: a useful baseline for understanding expected execution time ranges before setting alerts.

Error tracking

Failed tool calls appear in the Errors panel with their full stack trace, the span attributes attached at the time of failure, and a count of how many times the error has occurred. If you record the exception with span.recordException(err), Kibana links the error directly to the trace that produced it.

Closing the loop with the Agent Builder MCP

The Elastic Agent Builder MCP lets Claude query its own trace data from Elasticsearch in the same chat session that produced the traces. The Agent Builder MCP can query any Elasticsearch index the API key has access to, and APM traces are stored under .ds-traces-apm.otel-default-*. Granting the Agent Builder API key read access to those indices is what closes the loop: the agent that executed the tool calls can now reason about how they performed.

Here is what this looks like in practice. To generate traces, let's ask in Claude Desktop: "Use the echo tool to say hello, then use get-sum to add 1337 and 42, then run a long-running operation with 3 steps."

Claude executes three tool calls on the instrumented MCP. Three spans land in Elastic APM.

Now, without leaving the chat, let's try querying the traces by asking: "Search the APM trace data from the last 10 minutes. What tool calls were made, and how long did each one take?"

We can confirm the information against the services data:

Claude uses the Agent Builder MCP to run a query against the traces index. It returns the tool names, durations, and status from the actual trace data, then synthesizes an answer in natural language.

You can go further by asking:

• "Which of those tool calls had the highest p95 latency?"

• "Did any tool calls fail? If so, what was the error message?"

• "Compare the latency of the echo tool vs get-sum across all calls in the last hour."

Each of these questions translates into an ES|QL query via the Agent Builder's platform.core.execute_esql tool, run against the APM trace indices.

Why Elastic for MCP observability

The Agent Builder closes the loop: this is the part that is specific to the Elastic ecosystem. Because APM data lives in Elasticsearch, and Elasticsearch is queryable via the Agent Builder MCP, you can bring your AI agent's own observability data back into the conversation. Your AI can reflect on its own performance and spot anomalies.

APM UI built for distributed tracing: Kibana's APM interface is designed for exactly this kind of data: named transactions, trace waterfalls, latency percentiles, error tracking with stack traces, and service maps.

Managed OTLP endpoint: Elastic APM accepts OTLP directly since Elastic 8.x. You point OTEL_EXPORTER_OTLP_ENDPOINT at your APM server and it works.

EDOT simplifies the setup: the elasticapm connector is included, which means the APM UI views work without any additional configuration.

Conclusion

MCP servers do not need special observability tooling. They are programmatic processes, and OpenTelemetry is the right instrument for processes. The OTel MCP semantic conventions are stable and give you a consistent naming scheme that scales across tools and teams.

What makes the Elastic setup interesting is not the instrumentation itself. It is the second MCP. When your observability data lives in Elasticsearch, you can query it from the same AI session that generated it. That feedback loop is new, and it opens up use cases that dashboards alone cannot cover: real-time anomaly questions, automated triage, and AI-assisted incident investigation from the chat interface your team is already using.

Next steps

• Set up EDOT Node.js

• Configure the Elastic Agent Builder MCP

• OTel semantic conventions for MCP

• Managed OTLP endpoint on Elastic Cloud (now GA)

• Instrumenting Node.js applications with EDOT

Frequently asked questions

How do I add tracing to an MCP server? Use the OpenTelemetry SDK and follow the official OTel MCP semantic conventions. With the Elastic Distribution of OpenTelemetry (EDOT) for Node.js, a single --import flag enables auto-instrumentation for HTTP and database calls. Tool-call spans need to be added manually using a wrapper that sets mcp.method.name, gen_ai.tool.name, and gen_ai.operation.name.

Why are my MCP tool calls slow and how do I find the bottleneck? Without tracing, an MCP tool call is a black box: you see the result but not where the time went. Instrument the server with OpenTelemetry, ship traces to Elastic APM, and use the trace waterfall view in Kibana to see exactly how much time was spent in business logic versus downstream HTTP or database calls.

Can I send MCP server traces to Elastic APM without a custom collector? Yes. Elastic APM accepts OTLP directly. Set OTEL_EXPORTER_OTLP_ENDPOINT to your APM endpoint and OTEL_EXPORTER_OTLP_HEADERS with an API key, and traces flow in. EDOT bundles the elasticapm connector that the Kibana APM UI needs for service maps and transaction grouping.

What span names and attributes should I use for MCP tool calls? Follow the OpenTelemetry MCP semantic conventions: name spans {mcp.method.name} {target} (for example, tools/call echo), and set mcp.method.name, gen_ai.tool.name, and gen_ai.operation.name=execute_tool. On failure, set error.type to the error class name. Consistent naming means the Elastic APM transactions view groups your tool calls automatically.

How is this different from sending MCP traces to Datadog or Grafana? Any OTel-compatible backend can receive the traces. The Elastic-specific part is the Agent Builder MCP, which lets the same AI agent that generated the traces query them back from Elasticsearch in natural language. That feedback loop, where the AI reasons about its own tool-call performance, is not available with backends that do not expose their data through an MCP server.

Should I capture MCP tool call arguments and results in my traces? The OTel spec defines gen_ai.tool.call.arguments and gen_ai.tool.call.result as optional and warns they may contain sensitive data. Tools like get-env, which returns environment variables, illustrate the risk: API keys and credentials can land in your observability backend. Capture these only when the data is safe to store, and consider masking at the SDK level before export.

Does this work for MCP servers written in languages other than Node.js? The OpenTelemetry MCP semantic conventions are language-agnostic. EDOT is available for Node.js, Java, Python, .NET, and other languages, and any of them can send OTLP to Elastic APM. The wrapper pattern shown in this post translates directly: open a span around the tool handler, set the standard attributes, record exceptions on failure.

• Elasticsearch 9.3+ (or Elastic Cloud Serverless)

• Elasticsearch API KEY and Kibana URL

• An application instrumented with Elastic APM: the RUM agent for frontend interactions (populates traces-apm-*) and the APM agent for backend errors (populates logs-apm.error-*

• Cursor (version 2.6+) installed

The problem with two worlds

Application logs and code are two separate worlds that don't talk to each other. If you want to apply log insights into the application you have to analyze the logs, and then come back to the editor and apply your findings.

The Model Context Protocol (MCP) changes this. MCP is an open standard that lets AI clients like Cursor connect to external tools and data sources through a standardized interface. Instead of your IDE only knowing about your local code, it can also talk to your Elasticsearch cluster, query your APM data, and reason about production behavior alongside your source files.

Elastic ships a built-in MCP server as part of Agent Builder. You define tools in Kibana, expose them via the MCP endpoint, and any MCP-compatible client can call them. Cursor supports MCP natively, which means you can set this up in minutes.

What we're building

We're working with an eCommerce search app instrumented with Elastic APM. The RUM JS agent tracks filter click interactions from the browser, stored in traces-apm-default. The Node.js APM agent captures backend errors, stored in logs-apm.error-default.

Two situations come up during development:

• Use case 1: The product team wants to simplify the search page. There are six filters but we don't know which ones users actually click. We need usage data to decide which to keep.

• Use case 2: Users report intermittent 500 errors on search. The errors are not constant and started two days ago. We need the error details to find the root cause.

To bring that data into Cursor, we'll build two Agent Builder tools in Kibana and connect them via the Elastic Agent Builder MCP Server:

• get_filter_usage: queries traces-apm-default for filter click events and returns a usage breakdown by filter name

• get_recent_errors: queries logs-apm.error-default for the most recent error groups for a given service, including the exception message and stack trace culprit

For a deeper look at the overall architecture, see the Agent Builder reference guide.  

Setting up the Elastic MCP Server  

Step 1: Create the Agent Builder tools

We create both tools via the Kibana Agent Builder API. Each tool is an ES|QL query with a name and description that Cursor uses to decide when to call it. The full implementation of the tools is in the following notebook.

Tool 1: get_filter_usage

The product team needs to know which filters users actually click before deciding which ones to remove. The query reads RUM interaction events from traces-apm-default and groups them by filter name:

    {
      "id": "get_filter_usage",
      "type": "esql",
      "description": "Returns the usage count for each search filter in the ecommerce-search-ui service, sorted by most used first.",
      "configuration": {
        "query": "FROM traces-apm-default | WHERE service.name == \"ecommerce-search-ui\" | WHERE transaction.type == \"user-interaction\" | WHERE labels.filter_name IS NOT NULL | STATS count = COUNT(*) BY labels.filter_name | SORT count DESC"
      }
    }

Tool 2: get_recent_errors

For the error debugging use case, we need to surface the most frequent recent errors for a service, along with where in the code they originate. STATS ... BY groups errors by their fingerprint (grouping_key), surfaces the exception message and the line of code that caused it (culprit), and ranks by frequency:

    {
      "id": "get_recent_errors",
      "type": "esql",
      "description": "Returns the most frequent error groups for ecommerce-search-ui, ranked by occurrence count, with the exception message and code location.",
      "configuration": {
        "query": "FROM logs-apm.error-default | WHERE service.name == \"ecommerce-search-ui\" | WHERE processor.name == \"error\" | STATS count = COUNT(*) BY error.grouping_key, error.exception.0.message, error.culprit | SORT count DESC | LIMIT 5"
      }
    }

Both tools are created with POST /api/agent_builder/tools. You can learn more about the Kibana API endpoints for Elastic Agent Builder here.

Step 2: Connect to Cursor

Open ~/.cursor/mcp.json and add the Elastic server. For detailed information, see the Cursor documentation. The Agent Builder MCP endpoint uses Server-Sent Events (SSE) transport, so we connect via mcp-remote, a lightweight bridge that Cursor invokes as a local process:

    {
      "mcpServers": {
        "elastic-agent-builder": {
          "command": "npx",
          "args": [
            "-y",
            "mcp-remote",
            "https://YOUR_KIBANA_URL/api/agent_builder/mcp",
            "--header",
            "Authorization: ApiKey YOUR_API_KEY"
          ]
        }
      }
    }

Replace YOUR_KIBANA_URL and YOUR_API_KEY with your values.

Restart Cursor, open the Agent panel, and confirm that get_filter_usage and get_recent_errors appear in the available tools list. 

Use case 1: Data-driven UI optimization

The eCommerce search page has six filters: category, manufacturer, price range, customer gender, day of week, and region. The product team wants to simplify the UI by removing filters that users don't use as much. Rather than guessing, we ask Cursor to check.

When you type a prompt in Cursor's Agent panel, the model sees the name and description of every connected MCP tool. It matches your intent to the best-fitting tool and calls it automatically. This is why the description field we set in Step 1 matters: it's what the model reads to decide which tool answers your question. If you are interested in learning more about Cursor’s MCP tools management, read the following documentation.

Open a Cursor chat and ask: "Show me how often each search filter is used." Cursor calls the tool and returns something like:

The category and manufacturer filters get most of the clicks. The bottom three filters (customer_gender, day_of_week, region) are rarely used.

Ask Cursor to act on this: "Based on this data, simplify the SearchFilters component. Keep the top 3 filters visible, collapse the others under a 'More filters' toggle."

Cursor opens src/components/SearchFilters.jsx, reads the current implementation, and proposes the change.

Before: 

After: 

The entire loop took one chat prompt. The decision was backed by production data, not a team discussion about what users probably care about.

Use case 2: Production error debugging

A bug report comes in: intermittent 500 errors on the search endpoint. The errors started appearing two days ago but they're not constant. The developer opens Cursor and asks: "Show me what errors ecommerce-search-ui is throwing."

Cursor calls the tool and returns the error groups:

The error message is explicit: category is a text field and can't be used in terms of aggregation. The correct field is category.keyword. 

With APM data available alongside your code, the debugging session becomes a conversation: you describe the symptom, the agent pulls the relevant logs, and you work through what's happening together. You can ask follow-up questions, check whether the error correlates with a recent deployment, or ask which endpoints are most affected, all within the same context where you'll make the fix. If you want to go further, Elastic also provides pre-built observability tools in Agent Builder that you can use alongside custom tools like the ones we created here. For a complementary approach to AI-driven observability, see how to monitor web AI agents with OpenLIT and Elastic.

Conclusion

What we covered:

• How to create Agent Builder tools in Kibana that wrap APM data queries

• How to connect the Elastic Agent Builder MCP Server to Cursor in three lines of JSON

• Using production telemetry to make a UI decision backed by real usage data

• Debugging a production error from the same window where you fix it

These two use cases are a starting point. The same pattern works for any data you have in Elasticsearch: performance metrics, A/B test results, audit logs, feature flag usage, user session data. Define the Agent Builder tool, connect it via MCP, and it becomes part of your development context in Cursor. For other examples of what's possible, see automating synthetic monitoring with MCP and agentic CI/CD deployment gates.

Next steps

• Elastic Agent Builder MCP server documentation

• Model Context Protocol specification

• Elastic Agent Builder overview

• Distributed components: A single request might hit an API gateway, retrieve data from a feature store, evaluate a predictive model in a Python inference service, query a vector database, and call an external LLM.
• Non-determinism: AI agents make autonomous decisions and tool calls. If an agent fails, you need a full trace to understand its reasoning loop and what external tools it tried to invoke.
• Context dependence: You don't just care that an error happened; you need to know what model version was running, what hyperparameters were used, what the input data looked like, what was the commit that made that change. Many of these attributes are custom to your app, and you need an Observability environment that has the flexibility of creating new parameters on the fly and use them to find and fix issues.

On top of that, with the increased use of AI agents to generate code and make autonomous decisions, Observability becomes key to understanding what is working and what is not. It creates a critical feedback loop to quickly fix problems. More than ever, ML and AI applications need to adopt the best practices of mature software engineering systems to succeed.

This guide shows how to use OpenTelemetry and Elastic to correlate traces, logs, and metrics to track runs, compare model behavior, and trace requests across Python and Go services with one shared context.

Problem context: why AI systems are harder to debug

Traditional services already have distributed failure modes, but ML and AI systems add more moving parts:

• notebook experiments and ad hoc jobs
• batch training and evaluation pipelines
• online inference services
• external API calls, including LLM providers
• changing model versions and hyperparameters

When one prediction path gets slower or starts failing, plain isolated logs do not answer enough questions. You need to correlate:

• what ran (run ID, model version, parameters)
• where time was spent (pipeline stage latencies)
• what was the result (model stats, predictions, API calls, compare with other runs)
• what changed (code, data, dependencies)

In a future blog post, we'll show you how to set up automatic RCA and remediations with Elastic Workflows and our AI integrations. But as a first step, ML and AI pipelines need a robust Observability framework, which is very easy to set up with OpenTelemetry and Elastic.

Solution overview

OpenTelemetry gives you a standard way to emit traces, metrics, and logs. Elastic provides full OpenTelemetry ingestion, giving you a single place to store and query that telemetry. Kibana's UI is fully integrated with OpenTelemetry, allowing you to explore your services, service dependencies, service latencies, spans, and metrics out-of-the-box.

You can start with two deployment options:

• Cloud: send OpenTelemetry data directly to Elastic Cloud Managed OTLP Endpoint (mOTLP docs), without the overhead of managing collectors
• Local: run Elastic and the EDOT Collector with start-local, the EDOT Collector will be automatically listening for OTLP data in localhost:4317

Both options let you keep your application code unchanged for the initial implementation.

Step 1: zero-code baseline for Python services

Start by just installing the Elastic Distribution of OpenTelemetry Python (EDOT Python) package and using the opentelemetry-instrument wrapper to run your script. By simply running your script with this wrapper—without modifying your application code—your Python services begin emitting standard telemetry right away. This includes any logs exported via logging, alongside metrics and traces for auto-instrumented libraries. This data can be routed directly to Elastic's managed OTLP endpoint or a local EDOT collector.

pip install elastic-opentelemetry
edot-bootstrap --action=install

Export the OpenTelemetry environment variables, then run opentelemetry-instrument on your script to enable auto-instrumentation.

export OTEL_EXPORTER_OTLP_ENDPOINT="https://<motlp-endpoint>" # No need when using start-local with EDOT
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey <key>" # No need when using start-local with EDOT
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=prod,service.version=1.0.0" # Set the environment and version for your app
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
export ELASTIC_OTEL_SYSTEM_METRICS_ENABLED=true
export OTEL_METRIC_EXPORT_INTERVAL=5000 # Choose the interval for your application metrics

opentelemetry-instrument --service_name=<pipeline-name> python3 <your_python_script>.py # Set your chosen name for your service

With this baseline, you can quickly get:

• Centralized logs with trace context. Any logs exported via logging will be searchable in Elastic and Kibana, with the ability to perform full-text search on your logs
• Set alerting on log errors
• Process and system metrics. System and process metrics from the execution will be automatically exported to Elastic. You can visualize them, and analyse memory usage (leaks, OOM errors), CPU utilization (Bottlenecks / Spikes), thread counts, disk I/O bottlenecks or network I/O saturation.
• Set alerting on metrics
• Spans for auto instrumented libraries
• Service latency baselines and error trends
• Set manual or Anomaly detection alerting on error rates, latencies or throughput
• Correlate logs, metrics, and traces in a single shared context to quickly find the root cause of issues, using OpenTelemetry for instrumentation and Elastic for analysis.

Once ingested, Kibana immediately populates out-of-the-box dashboards. You can explore full-text searchable logs, monitor system and process metrics, investigate auto-instrumented trace waterfalls, map out your ML dependencies with service maps, and easily set up alerts for latency spikes, memory or CPU usage or log errors.

For LLM-specific observability, OpenTelemetry provides official Semantic Conventions for Generative AI to standardize how you track token usage, model names, and prompts. These semantic conventions are still in development and not stable yet. Some instrumentations for the most used libraries in this space are being developed as part of the OpenTelemetry Python Contrib repository. Alternatively you can implement these conventions manually in your custom spans. LLM related OpenTelemetry logs, metrics and traces sent to Elastic will be in context and automatically correlated with the rest of your application or stack of applications.

Step 2: add ML-specific context with custom spans and log fields

Auto-instrumentation is a starting point. For ML and AI Ops, add explicit spans around business stages and attach run metadata. Elastic's schema flexibility and dynamic mappings make it a perfect fit for custom attributes or metrics that are exclusive to your pipelines or specific experiments. There is no need to know what the data will look like before writing it. You have the flexibility of creating new parameters on the fly, Elastic maps them automatically, and you can track them instantly.

Add custom fields and metric-like values as structured log fields so you can chart and alert on them later:

logger.info("training metrics", extra={
    "ml.run_id": run_id,
    "ml.training_accuracy": train_accuracy,
    "ml.validation_accuracy": val_accuracy,
    "ml.drift_detected": drift_detected,
})

Because Elastic handles dynamic mapping, any custom metrics or attributes you log, like model ids, training accuracy or drift detection, are instantly indexed and available to search in Discover or visualize via Dashboards.

This makes dashboards and rules practical:

• alert when ml.validation_accuracy < 0.8
• alert when ml.drift_detected == true
• compare stage latency by ml.model_version

You can use these custom attributes to build targeted visualizations, and trigger alerts when ML-specific metrics like validation accuracy drop below a critical threshold.

Adding custom spans allows you to break down the specific stages of your ML pipelines, such as data loading and model training, wrapping them in their own measurable execution blocks, and analyze average latency or error rates for specific pipeline stages.

from opentelemetry import trace

tracer = trace.get_tracer("ml.pipeline")

with tracer.start_as_current_span("load_data") as span:
    span.set_attribute("ml.run_id", run_id)
    span.set_attribute("ml.dataset", dataset_source)
    load_data()

with tracer.start_as_current_span("train_model") as span:
    span.set_attribute("ml.model_version", model_version)
    span.set_attribute("ml.learning_rate", learning_rate)
    train_model()

Custom spans will be reflected in the APM UI alongside your traces. So you can explore their latency, impact in total execution, stack traces, error rates.

Step 3: trace across Python and Go in production

Real inference paths often cross service boundaries. For example:

In a production environment, a user request might pass through a Go-based API before hitting your Python ML inference service. OpenTelemetry ensures tracing context is preserved seamlessly across these boundaries.

In our example, we have a simple Go HTTP service that acts as the entry point and demonstrates OpenTelemetry instrumentation in Go. This REST API service stores and retrieves ML predictions by querying Elasticsearch based on data IDs from the source dataset. All of its endpoints are natively instrumented with OTel spans.

The full request lifecycle looks like this:

1. The Go API receives the client request.
2. It searches Elasticsearch for an existing prediction or calls the Python model service to run inference.
3. The Python service loads features, runs the model, and returns predictions.

When both services use OpenTelemetry, trace context is propagated automatically through headers. In Elastic, you can inspect one end-to-end trace and locate latency or errors by service and span.

The resulting distributed trace in Elastic pieces the entire journey together. You can see the exact breakdown of time spent in the Go API versus the Python model, and correlate logs from both services in a single unified view.

Validation checklist

After instrumentation, validate with a short runbook:

1. Confirm logs, metrics, and traces arrive for each service.
2. Verify your custom attributes (e.g. run_id, model_version, llm_ground_truth_score) are present in traces and logs.
3. Compare p95 latency per stage (load_data, train_model, predict).
4. Trigger a controlled failure and confirm error traces include stack context.
5. Test one rule for errors, one rule for latency spikes, and one rule for model-quality fields. Set up a connector and attach it to the rule to reach you in Slack, email, or trigger an auto-remediation workflow.

Conclusion and next steps

OpenTelemetry gives ML and AI teams a unified telemetry layer, while Elastic makes that data instantly queryable and actionable across your entire lifecycle—from notebook experiments to production inference. By starting with zero-code instrumentation and incrementally adding ML-specific attributes and cross-language tracing, your team can easily adopt the Observability best practices of mature software engineering systems and succeed in the new era of complex AI operations.

Try this setup in Elastic Cloud, and use mOTLP for a managed ingest path. If you want a local sandbox first, start with Elastic start-local + EDOT Collector.

As a core contributor, Elastic is proud to have accelerated this effort by previously donating its Universal Profiling™ eBPF-based continuous profiling agent to OpenTelemetry. This production-grade agent enables whole-system visibility across all applications, covering a multitude of programming languages and runtimes including third-party libraries and kernel operations with minimal overhead. It allows SREs and developers to quickly identify performance bottlenecks, maximize resource utilization, and optimize cloud spend.

Additionally, over the last two years, Elastic has been heavily contributing to the OpenTelemetry Collector, Semantic Conventions and Profiling Special Interest Groups (SIGs) to lay the technical foundation for the promotion of Profiles to Alpha.

This Alpha milestone not only boosts the standardization of continuous profiling but also accelerates the practical adoption of profiling as the fourth key signal in observability. Customers now have a vendor-agnostic way of collecting profiling data and enabling correlation with existing signals, like logs, metrics and traces, unveiling new potential for observability insights and a more efficient troubleshooting experience.

What is continuous profiling?

Profiling is a technique used to understand the behavior of a software application by collecting information about its execution. This includes tracking the duration of function calls, memory usage, CPU usage, and other system resources.

However, traditional profiling solutions have significant drawbacks limiting adoption in production environments:

• Significant cost and performance overhead due to code instrumentation
• Disruptive service restarts
• Inability to get visibility into third-party libraries

Unlike traditional profiling, which is often done only in a specific development phase or under controlled test conditions, continuous profiling runs in the background with minimal overhead, eliminating the need for service restarts or manual intervention. This provides real-time, actionable insights without replicating issues in separate environments. SREs, DevOps, and developers can see how code affects performance and cost, making code and infrastructure improvements easier.

Elastic's contribution: Powering the Alpha

The Elastic-donated profiler now forms the reference eBPF-based profiler implementation within OpenTelemetry: opentelemetry-ebpf-profiler. With the Alpha release, the eBPF profiler operates as an OpenTelemetry Collector receiver and contains numerous improvements such as automatic Go symbolization and support for new language runtimes. Operating as an OpenTelemetry Collector receiver enables the profiler to seamlessly leverage existing OpenTelemetry processing and filtering pipelines.

For example, the k8sattributesprocessor can use the container.id resource attribute to automatically enrich every profile with its corresponding Kubernetes context. This means you don't just see a raw stack trace; you see exactly which namespace, pod, and deployment produced it.

receivers:
  # Profiling receiver
  profiling: {}

processors:
  k8sattributes:
    passthrough: false 
    pod_association:
      - sources:
          - from: resource_attribute
            name: container.id
    extract:
      metadata:
        - "k8s.namespace.name"
        - "k8s.deployment.name"
        - "k8s.replicaset.name"
        - "k8s.statefulset.name"
        - "k8s.daemonset.name"
        - "k8s.node.name"
        - "k8s.pod.name"
        - "k8s.pod.ip"
        - "k8s.pod.uid"

Besides improvements to the eBPF profiler, Elastic has made significant contributions to:

• Correlating profiles with the information produced by OpenTelemetry eBPF instrumentation (OBI), a powerful auto-instrumentation tool that can enable distributed tracing.
• Process Context Sharing OTEP which is designed to bridge the gap between application SDKs and the profiler. This mechanism will allow OpenTelemetry SDKs to "publish" their resource attributes (like service.name) into a small, standardized memory region. Because this data is stored in the process's own memory map, the eBPF Profiler can instantly discover and associate it with its corresponding Profile.
• Semantic conventions and integration of OpenTelemetry Profiles with Google's pprof format (transparent conversion)
• OpenTelemetry Collector processing pipelines, allowing it to better integrate with the profiling receiver

Elastic's Next-Generation Profiling Development

Elastic remains deeply committed to OpenTelemetry's vision and is pushing the boundaries of what is possible with profiling data. We are dedicating a team of profiling domain experts to co-maintain and advance profiling capabilities within OpenTelemetry, while simultaneously working on groundbreaking features built on this new open standard.

Exciting areas of internal profiling-specific development include:

• OpenTelemetry Profiles derived Metrics: We are developing innovative ways to automatically generate actionable performance metrics directly from the raw OTel Profiles data, providing a new dimension for infrastructure modeling and alerting.
• Rapid Integration with the Elastic Stack: We are making swift progress on first-class support for OTLP Profiles within the Elastic Stack, ensuring seamless ingestion (the ebpf-profiler receiver is already integrated with the Elastic Distributions of OpenTelemetry (EDOT) collector), storage, and visualization of this new signal alongside your existing logs, metrics and traces.
• AI-Powered Workflows: We are leveraging the deep insights provided by continuous profiling data to power new AI-driven workflows, enabling automatic root-cause analysis, anomaly detection, and intelligent optimization suggestions for both code and infrastructure.

While the Alpha release marks a significant milestone, it is just the beginning. We encourage the community to start testing early preview versions of the OTel Profiles integration and contribute to the ongoing profiling work. To get started with an actual, local deployment, you can use the OpenTelemetry eBPF profiler in combination with a self-hosted Elastic Observability Stack or devfiler, a standalone desktop application that acts as an OpenTelemetry Profiles compliant backend aimed at experimentation and development.

This article explains why Nginx tracing is important in an application context, and provides a practical guide to enable the Nginx Otel tracing module exporting spans directly to Elastic APM.

Why Nginx Tracing Matters for Modern Observability

Instrumenting only backend services gives you only half the picture.
Nginx sees:

• every incoming request
• client trace context
• TLS negotiation
• upstream errors (502, 504)
• edge-layer latency
• routing decisions

If Nginx is not in your traces, your distributed trace is incomplete.

By adding OpenTelemetry tracing at this ingress layer, you unlock:

1. Full trace continuity : From browser → Nginx → backend → database.

2. Accurate latency attribution : Edge delays vs. backend delays are clearly separated which unlock Elastic APM Latency anomaly detection for proactive detection.

3. Error root-cause clarity : Nginx errors appear as spans instead of backend “mystery gaps”.

4. Complete service topology : Your APM service map finally shows the real architecture.

Integrating Nginx with OpenTelemetry on Debian

This guide provides a comprehensive overview of why, how to install and configure the Nginx OpenTelemetry module on a Debian-based system. The configuration examples are tailored to send telemetry data directly to an Elastic APM endpoint whether it's an EDOT Collector or mOtel in case of our serverless, enabling end-to-end distributed tracing.

Installation on Debian

The Nginx OTEL module is not included in the standard Nginx packages. It must be installed along with a working nginx configuration.

Prerequisites

First, install the necessary tools for compiling software and the Nginx development dependencies.

sudo apt update
sudo apt install -y apt install nginx-module-otel

Load the Module in Nginx

Edit your main /etc/nginx/nginx.conf file to load the new module. This directive must be at the top level, before the http block.

# /etc/nginx/nginx.conf

load_module modules/ngx_otel_module.so;

events {
    # ...
}

http {
    # ...
}

Now, test your configuration and restart Nginx.

sudo nginx -t
sudo systemctl restart nginx

Configuration

Configuration is split between the main nginx.conf file (for global settings) and your site-specific server block files.

Global Configuration (/etc/nginx/nginx.conf)

This configuration sets up the destination for your telemetry data and defines global variables used for CORS and tracing. These settings are placed inside the http block.

http {
    ...

    # --- OpenTelemetry Exporter Configuration ---
    # Defines where Nginx will send its telemetry data directly to Elastic APM or EDOT.
    otel_exporter {
        endpoint https://<ELASTIC_URL>:443;
        header Authorization "Bearer <TOKEN>";
    }

    # --- OpenTelemetry Service Metadata ---
    # These attributes identify Nginx as a unique service in the APM UI.
    otel_service_name nginx;
    otel_resource_attr service.version 1.28.0;
    otel_resource_attr deployment.environment production;
    otel_trace_context propagate; # Needed to propagate the RUM traces to the backend

    # --- Helper Variables for Tracing and CORS ---
    # Creates the $trace_flags variable needed to build the outgoing traceparent header.
    map $otel_parent_sampled $trace_flags {
        default "00"; # Not sampled
        "1"     "01"; # Sampled
    }

    # Creates the $cors_origin variable for secure, multi-origin CORS handling.
    map $http_origin $cors_origin {
        default "";
        "http://<URL_ORIGIN_1>/" $http_origin; # Add your Origin here to allow CORS
        "https://<URL_ORIGIN_2>/" $http_origin; # Add your others Origin here to allow CORS
    }
...
}

Server Block Configuration (/etc/nginx/conf.d/site.conf)

This configuration enables tracing for a specific site, handles CORS preflight requests, and propagates the trace context to the backend service.

server {
    listen 443 ssl;
    server_name <WEBSITE_URL>;

    # --- OpenTelemetry Module Activation ---
    # Enable tracing for this server block.
    otel_trace on;
    otel_trace_context propagate;

    location / {
        # --- CORS Preflight (OPTIONS) Handling ---
        # Intercepts preflight requests and returns the correct CORS headers,
        # allowing the browser to proceed with the actual request.
        if ($request_method = 'OPTIONS') {
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
            add_header 'Access-Control-Allow-Headers' 'Content-Type, traceparent, tracestate' always;
            add_header 'Access-Control-Max-Age' 86400;
            add_header 'Access-Control-Allow-Origin' "$cors_origin" always;
            return 204;
        }

        # --- OpenTelemetry Trace Context Propagation ---
        # Manually constructs the W3C traceparent header and passes the tracestate
        # header to the backend, linking this trace to the upstream service.
        proxy_set_header traceparent      "00-$otel_trace_id-$otel_span_id-$trace_flags";
        proxy_set_header tracestate       $http_tracestate;

        # --- Standard Proxy Headers ---
        proxy_set_header Host             $host;
        proxy_set_header X-Real-IP        $remote_addr;
        proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # --- Forward to Backend ---
        # Passes the request to the actual application (eg. localhost in this example).
        proxy_pass http://<BACKEND_URL>:8080;
    }
}

Test your configuration and restart Nginx.

sudo nginx -t
sudo systemctl restart nginx

Conclusion: Turning Nginx into a First-Class Observability Signal

By enabling OpenTelemetry tracing directly in Nginx and exporting spans to Elastic APM (via EDOT or Elastic’s managed OTLP endpoint), you bring your ingress layer into the same observability model as the rest of your stack. The result is:

• true end-to-end trace continuity from the browser to backend services
• clear separation between edge latency and application latency
• immediate visibility into gateway-level failures and retries
• accurate service maps that reflect real production traffic

Most importantly, this approach aligns Nginx with modern observability standards. It avoids proprietary instrumentation, fits naturally into OpenTelemetry-based architectures, and scales consistently across hybrid and cloud-native environments.

Try it out!

Once Nginx tracing is in place, several natural extensions can further improve your observability posture:

• correlate Nginx traces with application logs and metrics using Elastic’s unified observability
• add Real User Monitoring (RUM) to close the loop from frontend to backend
• introduce sampling and tail-based decisions at the collector level for cost control
• use Elastic APM service maps and anomaly detection to proactively detect edge-related issues

Instrumenting Nginx is often the missing link in distributed tracing strategies. With OpenTelemetry and Elastic, that gap can now be closed in a clean, standards-based, and production-ready way.

If you want to experiment with this setup quickly, Elastic Serverless provides the fastest way to get started. Sign up and try it out in just a few minutes using our trial environment available at https://cloud.elastic.co/ .

The challenge is clear: How do you adopt OTel’s unified observability without abandoning your proven ECS-based logging?

In this guide, we’ll demonstrate how to bridge this gap using the Elastic Distribution of OpenTelemetry (EDOT). We will first show you how to leverage the EDOT Collector to ingest your logs into Elasticsearch, ensuring a seamless transition that unlocks the full power of OTel’s distributed tracing without breaking your current workflows.

Once the data is flowing, we will explore how Elasticsearch's underlying mapping architecture to allow that your existing filters and visualizations remain fully functional through two key features:

• Field Aliases: We’ll explain how Elastic uses aliases to ensure that legacy dashboards looking for log.level (ECS) still work perfectly, even as your new telemetry arrives as severity_text (OTel).

• Passthrough Fields: We’ll show how Elastic’s native OTel mapping structures use passthrough fields to handle OTel attributes. This ensures your data remains searchable and performant without the need for complex, manual schema migrations.

By combining EDOT for ingestion with these intelligent mapping structures, you can maintain your existing Java ECS logging while evolving toward a unified, OTel-native future.

The ECS Foundation

We begin with a Java application using Log4j2 and the ecs-java-plugin. This setup generates structured JSON logs in the Elastic Common Schema (ECS) that Elastic handles natively leveraging the ECS logging plugins that easily integrate with common logging libraries across various programming languages.

The following provides a Log4j2 Configuration Extract and this setup assumes prior configuration of Log4j2 dependencies to include the required ECS plugin libraries:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="DEBUG">
    <Appenders>
        <Console name="LogToConsole" target="SYSTEM_OUT">
            <EcsLayout serviceName="logger-app" serviceVersion="v1.0.0"/>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="LogToConsole"/>
        </Root>
    </Loggers>
</Configuration>

Note: <EcsLayout serviceName="logger-app" serviceVersion="v1.0.0"/> we will come back to this setting later in the blog article, as with Kubernets deployments these values can be automatically populated by the EDOT Collector and the setting could be simplified to <EcsLayout/>

Introducing the Elastic Distribution of OpenTelemetry (EDOT)

The Elastic Distribution of OpenTelemetry (EDOT) is more than just a repackaging; it is a curated set of OTel components (Collector and SDKs) optimized for Elastic Observability. Released in v8.15, it allows you to collect traces, metrics, and logs using standard OTel receivers while benefiting from Elastic-contributed enhancements like powerful log parsing and Kubernetes metadata enrichment.

EDOT's Primary Benefits:

Deliver Enhanced Features Earlier: Provides features not yet available in "vanilla" OTel components, which Elastic continuously contributes upstream.

Enhanced OTel Support: Offers enterprise-grade support and maintenance for fixes outside of standard OTel release cycles.

The question then becomes: How can users transition their ingestion architecture to an OTel-native approach while maintaining the ability to collect logs in ECS format?

This involves replacing classic collection and instrumentation components (like Elastic Agent and the Elastic APM Java Agent). Let us show you how this can be done step by step replacing it with the full suite of components provided by EDOT. A comprehensive view of the EDOT architecture components in Kubernets is shown below.

In a Kubernetes environment, EDOT components are typically installed via an OTel Operator and HELM chart. The main components are:

• EDOT Collector Cluster: deployment used to collect cluster-wide metrics.
• EDOT Collector Daemon: daemonset used to collect node metrics, logs, and application telemetry data.
• EDOT Collector Gateway: performs pre-processing, aggregation, and ingestion of data into Elastic.

Elastic provides a curated configuration file for all the EDOT components available as part of the the OpenTelemetry Operator using the opentelemetry-kube-stack Helm chart. Downloadable from here.

Achieving Correlation: SDK + Logging Context

To link a log line to a specific trace, the EDOT Java SDK performs a "handshake" with your logging library. When a trace is active, the SDK extracts the trace_id and span_id and injects them into the Mapped Diagnostic Context (MDC) of Log4j2. Even though your logs are in ECS format, they now carry the OTel DNA required for correlation. While the EDOT SDK can collect logs directly, a generally more resilient approach is to stick to file collection. This is important because if the OTel Collector is down, logs written to a file are buffered locally on the disk, preventing the data loss that can occur if the SDK's in-memory queue reaches its limit and starts discarding new logs. For an in-depth discussion on this topic we refer to the OpenTelemetry Documentation.

Zero-Code Instrumentation

The EDOT Java SDK is a customized version of the OpenTelemetry Java Agent. In Kubernetes, zero-code Java autoinstrumentation is supported by adding an annotation in the pod template configuration in the deployment manifest:

apiVersion: apps/v1
kind: Deployment
...
spec:
  ..
  template:
    metadata:
      # Auto-Instrumentation
      annotations:
        instrumentation.opentelemetry.io/inject-java: "opentelemetry-operator-system/elastic-instrumentation"

Collecting and Processing Logs with the EDOT Collector

This is the most critical step. Our logs are now JSON, they are in the console output, and they contain trace IDs. Now, we need the EDOT Collector to pick them up and map them to the OpenTelemetry Log Data Model.

EDOT Collector Configuration: Dynamic Workload Discovery and filelog receiver

Applications running on containers become moving targets for monitoring systems. To handle this, we rely on Dynamic workload discovery on Kubernetes. This allows the EDOT Collector to track pod lifecycles and dynamically attach log collection configurations based on specific annotations relying on the k8s_observer and the receiver_creator component.

In our example, we have a Deployment with a Pod consisting of one container. We use Kubernetes annotations to:

1. Enable auto-instrumentation (Java).

2. Enable log collection for this pod.

3. Instruct the collector to parse the output as JSON immediately (json-parser configuration).

4. Add custom attributes (e.g. identify the Application souce code)

Deployment Manifest Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: logger-app-deployment
  labels:
    app: logger-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logger-app
  template:
    metadata:
      annotations:
        # 1. Turn on Auto-Instrumentation
        instrumentation.opentelemetry.io/inject-java: "opentelemetry-operator-system/elastic-instrumentation"
        # 2. Enable Log Collection for this pod
        io.opentelemetry.discovery.logs/enabled: "true"
        # 3. Provide the parsing "hint" (Treat logs as JSON)
        io.opentelemetry.discovery.logs.ecs-log-producer/config: |
            operators:
            - type: container
              id: container-parser
            - type: json_parser
              id: json-parser
         # 4. Identify this application as Java (To allow for user interface rendering in Kibana)
        resource.opentelemetry.io/telemetry.sdk.language: "java"
      ...

This setup provides a bare-minimum configuration for ingesting ECS library logs. Crucially, it decouples log collection from application logic. Developers simply need to provide a hint via annotations that their logs are in JSON format (structurally guaranteed by the ECS libraries). We then define the standardized enrichment and processing rules centrally at the processor level in the (Daemon) EDOT Collector.

This centralization ensures consistency across the platform: if we need to update our standard formatting or enrichment strategies later, we apply the change once in the collector, and it automatically propagates to all services without developers needing to touch their manifests.

(Daemon) EDOT Collector Configuration

To enable this, we configure a Receiver Creator in the Daemon Collector. This component uses the k8s_observer extension to monitor the Kubernetes environment and automatically discover the target pods based on the annotations above.

daemon:
  ...
  config:
    ...
    extensions:
      extensions:
        k8s_observer:
          auth_type: serviceAccount
          node: ${env:K8S_NODE_NAME}
          observe_nodes: true
          observe_pods: true
          observe_services: true
          ...
    receivers:
        receiver_creator/logs:
          watch_observers: [k8s_observer]
          discovery:
            enabled: true
    ...
...

Finally, we reference the receiver_creator in the pipeline instead of a static filelog receiver and we make sure to include the k8s_observer extension:

daemon:
  ...
  config:
    ...
    service:
      extensions:
      - k8s_observer
      pipelines:
        # Pipeline for node-level logs
        logs/node:
          receivers:
            # - filelog             # We disable direct filelog receiver
            - receiver_creator/logs # Using the configured receiver_creator instead of filelog
          processors:
            - batch
            - k8sattributes
            - resourcedetection/system
          exporters:
            - otlp/gateway # Forward to the Gateway Collector for ingestion

The Transformation Layer

While the logs are structured, OTel sees them as generic attributes. We use the OpenTelemetry Transformation Language (OTTL) within a transform processor to "promote" ECS fields to top-level OTel fields. To finalize the pipeline, we use the transform processor, which allows us to modify and restructure telemetry signals using the OpenTelemetry Transformation Language (OTTL).

We use the processor to promote specific ECS fields into the top-level OpenTelemetry fields and renaming attributes according to OpenTelemetry Semantic Conventions:

• Promote the message attribute to the top-level Body field.
• Promote the log.level attribute to the OTel SeverityText field.
• Move the @timestamp attribute to the OTel Time field.
• Map trace_id and span_id to the right log context.

The following provides a sample transform configuration:

 processors:
    transform/ecs_handler:
      log_statements:
      - context: log
        conditions:
          - log.attributes["ecs.version"] != nil
        statements:
          # Map ECS fields to OTel Log Model
          - set(log.body, log.attributes["message"])
          - set(log.time, Time(log.attributes["@timestamp"], "%Y-%m-%dT%H:%M:%SZ"))
          - set(log.trace_id.string, log.attributes["trace_id"])
          - set(log.span_id.string, log.attributes["span_id"])
          - set(log.severity_text, log.attributes["log.level"])
          # Cleanup original keys to save space
          - delete_key(log.attributes, "message")
          - delete_key(log.attributes, "trace_id")
          - delete_key(log.attributes, "span_id")

          # Add here additional transformations as needed...

Note: When working with EDOT Collector and the OpenTelemetry Kube-Stack Helm Chart, resource attributes such as service.name and service.version are automatically populated based on a set of well-defined rules by the k8sattributes processor. Thus, on Kubernetes we do not need to extract those fields from the log content itself.

Make sure to use the newly created processor in the logs pipeline for the Daemon Collector:

service:
  pipelines:
    logs/node:
      receivers:
        - receiver_creator/logs
      processors:
        - batch
        - k8sattributes
        - resourcedetection/system
        - transform/ecs_handler          # Newly created transform processor
      exporters:
        - otlp/gateway

The Compatibility Layer: Bridging ECS and OTel

To bridge the gap between the Elastic Common Schema (ECS) and OpenTelemetry (OTel), Elastic provides a "compatibility layer" built directly into its Observability solution relying on existing index templates and mappings. This architecture allows you to send OTel-native data while still using your legacy ECS-based dashboards, saved searches, and other associated objects.

This "bridge" relies on two key features:

• Bridging ECS and OTel with Passthrough: OpenTelemetry (OTel) data often uses deeply nested structures (e.g., resource.attributes.*). Elasticsearch uses the Passthrough object type to "promote" these nested attributes to the top level when performing a search query. Any new metadata added by the OTel collector is automatically searchable without the user needing to know the full JSON path. This creates a "virtual flattening" layer and makes sure that all fields that match in name are automatically compatible, even though thery're stored in different namespaces (attributes/resource.attributes for OTel vs top-level for ECS). To learn more about fields and attributes alignment between ECS and Otel SemanticConvention refer to this page.

• Bridging with Field Aliases: Elastic relies on OTel mapping templates that include Field Aliases. These aliases link OTel semantic names back to their equivalent ECS fields at query to handle fields that do not align with Otel naming convention.

The Benefit: If you have an existing dashboard looking for message (ECS), but your data is now indexed as body.text (OTEL), an alias allows the dashboard to aggregate and visualize data from both sources simultaneously. This ensures that your existing filters and KQL queries also work flawlessly whether the data originated from a Filebeat agent or a modern OTel SDK Agent.

Some more details about field aliases and pass-through objects can be found here.

Here is an example of the provided mapping template:

{
  "mappings": {
    ...
    "properties": {
      "log": {
          "properties": {
            "level": {
              "type": "alias",
              "path": "severity_text"
            }
          }
        },
      "message": {
        "type": "alias",
        "path": "body.text"
      }
    ...
    }
  }
 }

This architectural approach provides three major advantages for teams in transition:

• Zero Reindexing: You don't have to rewrite or migrate old data. Aliases resolve at query time, meaning your old indices and new indices can coexist in the same visualization.

• Future-Proofing: As OTel becomes the primary standard (following the donation of ECS to the OTel project), Elastic is shifting its native UI to look for OTel fields first. These mappings ensure that your legacy ECS-native data still appears in OTel-native views.

• Unified Observability: It enables "Correlation by Default." Because the aliases link trace_id (OTel) and trace.id (ECS), you can jump from a legacy log to a modern OTel trace without losing context or breaking the drill-down path.

Sending data to Elasticsearch

If you are running Elastic Serverless or the latest Elastic Cloud Hosted (ECH) v9.2+, you now have access to a managed OTLP endpoint. This native functionality allows you to route telemetry directly from your Collector Gateway to Elasticsearch using the OTLP protocol.

Because we mapped our ECS fields to the OTel model in the collector, Elasticsearch recognizes the correlation immediately. You get the best of both worlds: Legacy Compatibility: Your old ECS-based dashboards still work (with minor tweaks). Modern Power: You can now click "View Trace" directly from a log entry in Kibana's Observability UI.

Conclusion

Transitioning to OpenTelemetry doesn't have to be a "big bang" migration. By using the EDOT SDK and Collector, you can: Protect your investment in ECS-based logging libraries. Centralize complexity by handling schema translation in the collector rather than the application. Enable full correlation between traces and logs with zero code changes.

For decades, PHP has powered everything from small business websites to large-scale SaaS platforms. Yet observability in PHP has often required manual setup, compilers, custom extensions, or changes to application code — challenges that limited adoption in production environments. This upcoming donation aims to change that, by making OpenTelemetry for PHP as easy to deploy as any other runtime.

What's coming

Once the contribution process is complete, EDOT PHP will become part of the OpenTelemetry project — providing a complete, production-ready distribution that's optimized for performance, simplicity, and scalability.

EDOT PHP introduces a new approach to PHP observability:

• Simple installation - installing OpenTelemetry for PHP will be as straightforward as installing a standard system package. From that point, the agent automatically detects and instruments PHP applications — no code changes, no manual setup.
• Automatic agent loading - works transparently in cloud and container environments without modifying application deployments.
• Zero configuration - ships as a single, self-contained binary; no need to install or compile any external extensions.
• Native C++ performance - a built-in serializer written in C++ reduces telemetry overhead by up to 5×.
• Automatic instrumentation - instruments popular frameworks and libraries out of the box.
• Inferred spans - reveals the behavior of even uninstrumented code paths, providing full trace coverage.
• Automatic root spans - ensures complete traces, even in legacy or partially instrumented applications.
• OpAMP readiness - while the OpenTelemetry community continues to standardize configuration schemas and management workflows, the implementation in EDOT PHP is fully prepared to support these upcoming specifications — ensuring seamless adoption once the OpAMP ecosystem matures.
• Asynchronous backend communication - telemetry data is exported to the OpenTelemetry Collector or backend asynchronously, without blocking the instrumented application. This ensures that span and metric exports do not add latency to user requests or impact response times, even under heavy load.

Together, these features make EDOT PHP the first truly zero-effort observability solution for PHP — from local testing to cloud-scale production systems.

The native C++ serializer and asynchronous export pipeline in EDOT PHP reduce average request time from 49 ms to 23 ms, more than 2× faster than the pure PHP implementation.

Building on the existing foundation

EDOT PHP doesn't replace the existing OpenTelemetry PHP SDK — it extends and strengthens it. It packages the SDK, automatic instrumentation, and native extension into a single, unified agent package that works seamlessly with existing OpenTelemetry specifications and APIs.

By contributing this work, Elastic helps the OpenTelemetry community accelerate PHP adoption, align implementations across languages, and make distributed tracing truly universal.

“This isn't a hand-off — it's a collaboration. We're contributing years of development to help OpenTelemetry for PHP evolve faster, run more efficiently, and reach more users in every environment.”

• Elastic Observability team

Ongoing improvements

Elastic continues to invest in advancing EDOT PHP ahead of its integration into OpenTelemetry. The team is currently focused on reducing resource usage and memory footprint, particularly in multi-worker server environments such as PHP-FPM or Apache prefork. These optimizations aim to make the agent more predictable and efficient under heavy load — ensuring that telemetry remains lightweight even in large-scale production deployments.

Beyond that, we're exploring further improvements that can enhance both performance and interoperability. Areas under investigation include smarter coordination in high-concurrency scenarios, better sharing of telemetry resources across workers, and future alignment with additional OpenTelemetry signals such as metrics and logs.

Together, these efforts will help make EDOT PHP not only faster, but also more adaptable and seamlessly integrated into diverse runtime architectures.

Why it matters

This contribution is about more than performance — it's about removing barriers. By making OpenTelemetry for PHP installable as a simple system package and automatically loaded into running applications, the project opens observability to every PHP developer, operator, and platform provider.

For the OpenTelemetry ecosystem, it fills one of the last major language gaps, extending visibility to a vast portion of the internet — all under open governance and community collaboration.

Looking ahead

In the months ahead, Elastic and the OpenTelemetry PHP SIG will work closely on the technical integration, documentation, and community onboarding process. Once the transition is complete, developers will gain a fully open, community-driven, and production-ready OpenTelemetry agent that “just works” — without friction, configuration, or code changes.

Together, we're building a future where observability just works — for every language, every framework, and every environment.

For more information:

EDOT documentation
Learn about OTLP Endpoint

Why OpenTelemetry?

OpenTelemetry provides a powerful set of capabilities that make it a compelling choice for open-source-focused users. Elastic is re-architecting its data ingest tools around OpenTelemetry to offer users vendor-agnostic flexibility, performance optimization through OTel's efficient data model for correlating telemetry, and enhanced flexibility and control over data pipelines. This move brings the benefits of open-source telemetry to Elastic users.

Elastic engineers are active contributors to the Otel project in several areas of the project. Demonstrating its commitment to open source, Elastic continues to make significant contributions to OpenTelemetry.

OpenTelemetry as the Core of Elastic's Data Ingestion

Elastic is transforming its data ingestion strategy by basing all ingestion mechanisms on the OpenTelemetry components. Elastic currently supports the following OTel based ingest architecture, which support OTel SDKs and Collectors from OTel or Elastic's Distribution of OpenTelemetry (EDOT).

This marks a fundamental shift, ensuring a more standardized and scalable telemetry pipeline. All the existing Elastic ingest components will become OTel based.

| | | | ---------------------------- | ----------------------------------------------------------------------------------------------- | | Beats | Beats architecture will be based on OTel. | | Elastic Agent | Agent architecture will be based on OTel to support both beats based inputs and OTel receivers. | | Integrations | Integrations catalogue will additionally include OTel based modules for ease of configuration. | | Fleet central management | Fleet will support monitoring of Elastic OTel collectors. |

Let's discuss how each component of Elastic's data ingestion platform will be based on an OpenTelemetry collector whilst still providing the same functionality to the user.

Beats

Elastic's traditional data shippers will be re-architected as OpenTelemetry Collectors, aligning with OTel's extensibility model. Current Beat architecture is essentially made up of a few stages in its pipeline, as shown in the diagram below. It consists of an Input, Processors for enrichments, Queuing of events and Output for batching and writing the data to a specific output.

Beatreceiver Concept

To ensure a smooth transition without major disruptions, a "beatsreceiver" concept is being implemented. These beatreceivers (like filebeatreceiver or metricbeatreceiver) act as dedicated Beat inputs integrated into the OpenTelemetry Collector as native receivers. They support all existing inputs and processors, guaranteeing that the final architecture accepts the user's current configuration and delivers the same functionality as today's Beats, all without introducing any breaking changes.

An OTel based Beats architecture will see the Input phase embedded as an OTel receiver (eg.  filebeatreceiver to represent the functionality of filebeat). This receiver would only be available as part of Elastic's distribution of OTel in support of our current user base and not a functionality that would be available upstream.

All the remaining components of the pipeline will be based on OTel. The new Beat will accept the same filebeat configuration (as an example) and will transform it to an OTel based configuration in order to avoid any deployment disruption. It should be noted that in this architecture the Beats will continue to only support ECS formatted data. In order to keep the Beat functionality inline with what exists today, the Elasticsearch exporter (as an example) will output ECS formatted data only.

The following diagram illustrates the beatreceiver concept by showing how a basic filebeat configuration is automatically translated into an OpenTelemetry-based configuration. This new configuration retains the original inputs and processors but leverages the native OpenTelemetry pipeline and exporter to achieve the same overall filebeat functionality. Existing filebeat configurations will be automatically converted, eliminating the need for manual adjustments or introducing breaking changes.

Elastic Agent

Elastic Agent is a unified agent for data collection, security, and observability. It can also be deployed in an OpenTelemetry only mode, enabling native OTel workflows. Elastic Agent is a supervisor that manages many other Beats as sub-processes in order to provide a more comprehensive data collection tool. It is capable of translating Agent Policy received from Fleet into configuration acceptable by the various sub-processes.

Expanding on the Beat receiver concept described above, the Elastic Agent, which currently can be deployed as an OTel collector (see blog), will be also modified to a much simpler OTel based architecture based on these receivers. As shown below, this architecture will streamline the components within the Elastic Agent and remove duplicated functionality such as queuing and output. Whilst supporting the current functionality, these changes will reduce the agent footprint and also present a reduction in number of connections opened to pipeline elements egress of the agent (such as Elasticsearch clusters, Logstash or Kafka brokers).

By moving to an OTel based architecture Elastic Agent is now able to operate as a truly hybrid Elastic Agent which provides not only the Beat functionality but also allows our users to create OTel native pipelines and take advantage of plethora of functionality available as part of the open source project.

Elastic's commitment to OpenTelemetry will deepen through increased contributions, resulting in OpenTelemetry receivers gradually superseding Beats receiver features. This evolution will eventually reduce the need for a distinct Beats receiver within the Elastic Agent architecture. The envisioned architecture will empower the Elastic Agent to transmit data in OTLP format as well, granting users the flexibility to select any OTLP-compatible backend, thereby upholding the principle of vendor neutrality.

Fleet & Integrations: Managing OpenTelemetry at Scale

Elastic's centralized management system will support OpenTelemetry-based configurations, making large-scale deployments easier to manage. Managing thousands of telemetry agents at scale presents a significant challenge. Elastic's Fleet & Integrations simplify this process by providing robust lifecycle management for these new OpenTelemetry-based Elastic agents.

Key Capabilities Offered:

• Scalability: Manage up to 100K+ agents across distributed environments.

• Automated Upgrades: Staged rollouts and automatic upgrades ensure minimal downtime.

• Monitoring & Diagnostics: Real-time status updates, failure detection, and diagnostic downloads improve system reliability.

• Policy-Based Configuration Management: Enables centralized control over agent configurations, improving consistency across deployments.

• Pre-Built Integrations: Elastic offers a catalog of 470+ pre-built integrations, allowing users to ingest data seamlessly from various sources. These will also include OTel based packages making configuration much more efficient across a large deployment.

The goal is for Fleet to also provide monitoring capabilities for native OTel collectors as well in a vendor agnostic fashion.

Conclusion

Elastic's adoption of OpenTelemetry marks a significant milestone in the evolution of open-source observability. By standardizing on OpenTelemetry, Elastic is ensuring that its data ingestion strategy remains open, scalable, and future-proof.

For open-source users, this shift means:

• Greater interoperability across observability tools.

• Enhanced flexibility in choosing telemetry backends.

• A stronger commitment to community-driven observability standards.

• Existing Beats and Elastic Agent users can seamlessly adopt OpenTelemetry without rearchitecting their pipelines.

• OpenTelemetry users can integrate with Elastic's observability stack without additional complexity.

Stay tuned for more updates as Elastic continues to expand its OpenTelemetry-based data collection capabilities! In the mean time here are some other references:

• Elastic Distributions of OpenTelemetry (EDOT) Now GA

• Dynamic workload discovery on Kubernetes now supported with EDOT Collector

• Introducing the OTTL Playground for OpenTelemetry

Luckily, Elastic has been one of the leading solutions in observability space and a contributor to the OpenTelemetry project. Elastic’s unique position and its extensive observability capabilities allows end-users to monitor applications built with object-oriented programming languages & Framework in a variety of ways.

In this blog we will explore using Elastic APM to investigate C++ traces with the OpenTelemetry client. We will be providing a comprehensive guide on how to implement the OpenTelemetry client for C++ applications and connecting to Elastic APM solutions. While OTel has its libraries, and this blog reviews how to use the OTel CPP library, Elastic also has its own Elastic Distributions of OpenTelemetry, which were developed to provide commercial support, and are completely upstreamed regularly.

Here are some resources to help get you started:

• Use OpenTelemetry with APM

• The OpenTelemetry C++ Client

• OpenTelemetry C++ Docs

Step by Step Guide

Prerequisites

• ### Environment

Choosing an environment is quite important as there is limited support for the OTEL client. We have experimented with using multiple Operating Systems and here are the suggestions:

• Ubuntu 22.04

• Debian 11 Bullseye

• For this guide we are focusing on Ubuntu 22.04.

• Machine: 2 vCPU, 4GB is sufficient.

• Image: Ubuntu 22.04 LTS (x86_64).

• Disk: ~30 GB is enough.

Implementation method 

We have experimented with multiple methods but we found that the most suitable approach is to use a package manager. After extensive testing, It appears that trying to run otel-cpp client could be quite challenging to the users. If practitioners desire to build with tools such as CMake and Bazel that is a viable solution. With that, as we tested both methods it became obvious that we were spending most of our time and effort fixing compatibility and dependencies’ issues for the OS Vs. Focusing on sending data to our APM. Hence we decided to move to a different method.

The main issues that we kept running into as we test are:

• Compatibility of packages.

• Availability of packages.

• Dependencies of libraries and packages.

In this guide we will use vcpkg since it allows us to bring in all the dependencies required to run the Opentelemetry C++ client.

Installing required OS tools

Update package lists

    sudo apt-get update

Install build essentials, cmake, git, and sqlite dev library

    sudo apt-get install -y build-essential cmake git curl zip unzip sqlite3 libsqlite3-dev

sqlite3 and libsqlite3-dev allow us to build/run SQLite queries in our C++ code.

Set Up vcpkg

vcpkg is the C++ package manager that we’ll use to install opentelemetry-cpp client.

    # Clone vcpkg
    cd ~
    git clone https://github.com/microsoft/vcpkg.git

    # Bootstrap
    cd ~/vcpkg
    ./bootstrap-vcpkg.sh

Install OpenTelemetry C++ with OTLP gRPC

In this guide we focus on trace export to Elastic. At time of writing, vcpkg’s opentelemetry-cpp

version 1.18.0 fully supports traces but has limited direct metrics exporting.

Install the package

    cd ~/vcpkg
    ./vcpkg install opentelemetry-cpp[otlp-grpc]:x64-linux

Note

Sometimes when installing opentelemetry-cpp on linux it doesn't install all the required packages. As a workaround if you run into that case, try running again but pass a flag to allow-unsupported:

    ./vcpkg install opentelemetry-cpp[*]:x64-linux --allow-unsupported

Verify

    ./vcpkg list | grep opentelemetry-cpp

The output thould be something like this:

opentelemetry-cpp:x64-linux 1.18.0

Create the C++ Project with Database Spans

We’ll build a sample in ~/otel-app that:

• Uses SQLite to do basic CREATE/INSERT/SELECT queries. This is helpful to showcase capturing transactions for apps that use databases on Elastic APM.

• Generate random traces to showcase how they are captured on Elastic APM.

This app is going to generate random queries where some will contain database transactions and some are just application traces. Each query is contained in a child span, so they appear in APM as separate database transactions.

# Below is the structure of our project

    otel-app/
    ├── main.cpp
    └── CMakeLists.txt

Create App Project

    cd ~
    mkdir otel-app
    cd otel-app

Inside this project we will create two files

• main.cpp

• CMakeLists.txt

Keep in mind that main.cpp is where you are going to pass the otel exporters that are going to send data to the Elastic cluster. So for your tech stack it would be your application's source code.

Sample application code

    main.cpp
    // Below we declare required libraries that we will be using to ship
    // traces to Elastic APM
    #include <opentelemetry/exporters/otlp/otlp_grpc_exporter.h>
    #include <opentelemetry/sdk/trace/tracer_provider.h>
    #include <opentelemetry/sdk/trace/simple_processor.h>
    #include <opentelemetry/trace/provider.h>

    #include <sqlite3.h>
    #include <chrono>
    #include <iostream>
    #include <thread>
    #include <cstdlib>&nbsp; // for rand(), srand()
    #include <ctime>&nbsp; &nbsp; // for time()

    // Namespace aliases
    namespace trace_api = opentelemetry::trace;
    namespace sdktrace&nbsp; = opentelemetry::sdk::trace;
    namespace otlp&nbsp; &nbsp; &nbsp; = opentelemetry::exporter::otlp;

    // Below we are using a helper function to run SQLITE statement inside&nbsp;
    // child span
    bool ExecuteSql(sqlite3 *db, const std::string &sql,
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;trace_api::Tracer &tracer,
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;const std::string &span_name)
    {
    &nbsp;&nbsp;// Starting the child span
    &nbsp;&nbsp;auto db_span = tracer.StartSpan(span_name);
    &nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;auto scope = tracer.WithActiveSpan(db_span);

    &nbsp;&nbsp;&nbsp;&nbsp;// Here we mark Database attributes for clarity in APM
    &nbsp;&nbsp;&nbsp;&nbsp;db_span->SetAttribute("db.system", "sqlite");
    &nbsp;&nbsp;&nbsp;&nbsp;db_span->SetAttribute("db.statement", sql);

    &nbsp;&nbsp;&nbsp;&nbsp;char *errMsg = nullptr;
    &nbsp;&nbsp;&nbsp;&nbsp;int rc = sqlite3_exec(db, sql.c_str(), nullptr, nullptr, &errMsg);
    &nbsp;&nbsp;&nbsp;&nbsp;if (rc != SQLITE_OK)
    &nbsp;&nbsp;&nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;db_span->AddEvent("SQLite error: " + std::string(errMsg ? errMsg : "unknown"));
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sqlite3_free(errMsg);
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;db_span->End();
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return false;
    &nbsp;&nbsp;&nbsp;&nbsp;}
    &nbsp;&nbsp;&nbsp;&nbsp;db_span->AddEvent("Query OK");
    &nbsp;&nbsp;}
    &nbsp;&nbsp;db_span->End();
    &nbsp;&nbsp;return true;
    }

    /**
    &nbsp;* DoNonDbWork - Simulate some other operation
    &nbsp;*/
    void DoNonDbWork(trace_api::Tracer &tracer, const std::string &span_name)
    {
    &nbsp;&nbsp;auto child_span = tracer.StartSpan(span_name);
    &nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;auto scope = tracer.WithActiveSpan(child_span);
    &nbsp;&nbsp;&nbsp;&nbsp;// Just sleep or do some "fake" work
    &nbsp;&nbsp;&nbsp;&nbsp;std::cout << "[TRACE] Doing non-DB work for " << span_name << "...\n";
    &nbsp;&nbsp;&nbsp;&nbsp;std::this_thread::sleep_for(std::chrono::milliseconds(200 + rand() % 300));
    &nbsp;&nbsp;&nbsp;&nbsp;child_span->AddEvent("Finished non-DB work");
    &nbsp;&nbsp;}
    &nbsp;&nbsp;child_span->End();
    }

    int main()
    {
    &nbsp;&nbsp;// Seed random generator for example
    &nbsp;&nbsp;srand(static_cast<unsigned>(time(nullptr)));

    &nbsp;&nbsp;// 1) Create OTLP exporter for traces
    &nbsp;&nbsp;otlp::OtlpGrpcExporterOptions opts;
    &nbsp;&nbsp;auto exporter = std::make_unique<otlp::OtlpGrpcExporter>(opts);

    &nbsp;&nbsp;// 2) Simple Span Processor
    &nbsp;&nbsp;auto processor = std::make_unique<sdktrace::SimpleSpanProcessor>(std::move(exporter));

    &nbsp;&nbsp;// 3) Tracer Provider
    &nbsp;&nbsp;auto sdk_tracer_provider = std::make_shared<sdktrace::TracerProvider>(std::move(processor));
    &nbsp;&nbsp;auto tracer = sdk_tracer_provider->GetTracer("my-cpp-multi-app");

    &nbsp;&nbsp;// Prepare an in-memory SQLite DB (for random DB usage)
    &nbsp;&nbsp;sqlite3 *db = nullptr;
    &nbsp;&nbsp;int rc = sqlite3_open(":memory:", &db);
    &nbsp;&nbsp;if (rc == SQLITE_OK)
    &nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;// Create a table so we can do inserts/reads
    &nbsp;&nbsp;&nbsp;&nbsp;ExecuteSql(db, "CREATE TABLE IF NOT EXISTS items (id INTEGER PRIMARY KEY, info TEXT);",
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;*tracer.get(), "db_create_table");
    &nbsp;&nbsp;}

    &nbsp;&nbsp;// Create the following loop to generate multiple transactions
    &nbsp;&nbsp;int num_transactions = 5;&nbsp; // Change this variable to the desired number of transaction
    &nbsp;&nbsp;for (int i = 1; i <= num_transactions; i++)
    &nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;// Each iteration is a top-level transaction
    &nbsp;&nbsp;&nbsp;&nbsp;std::string transaction_name = "transaction_" + std::to_string(i);
    &nbsp;&nbsp;&nbsp;&nbsp;auto parent_span = tracer->StartSpan(transaction_name);
    &nbsp;&nbsp;&nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;auto scope = tracer->WithActiveSpan(parent_span);

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;std::cout << "\n=== Starting " << transaction_name << " ===\n";

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Randomly select whether a transaction will interact with the database or not.
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;bool doDb = (rand() % 2 == 0); // 50% chance

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (doDb && db)
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Insert random data
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;std::string insert_sql = "INSERT INTO items (info) VALUES ('Item " + std::to_string(i) + "');";
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ExecuteSql(db, insert_sql, *tracer.get(), "db_insert_item");

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Select from DB
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ExecuteSql(db, "SELECT * FROM items;", *tracer.get(), "db_select_items");
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Do some random non-DB tasks
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DoNonDbWork(*tracer.get(), "non_db_task_1");
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DoNonDbWork(*tracer.get(), "non_db_task_2");
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Sleep a little to simulate transaction time
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;std::this_thread::sleep_for(std::chrono::milliseconds(200));
    &nbsp;&nbsp;&nbsp;&nbsp;}
    &nbsp;&nbsp;&nbsp;&nbsp;parent_span->End();
    &nbsp;&nbsp;}

    &nbsp;&nbsp;// Close DB
    &nbsp;&nbsp;sqlite3_close(db);

    &nbsp;&nbsp;// Extra sleep to ensure final flush
    &nbsp;&nbsp;std::cout << "\n[INFO] Sleeping 5 seconds to allow flush...\n";
    &nbsp;&nbsp;std::this_thread::sleep_for(std::chrono::seconds(5));
    &nbsp;&nbsp;std::cout << "[INFO] Exiting.\n";
    &nbsp;&nbsp;return 0;
    }

What does the code do?

We create 5 top-level “transaction_i” spans.

For each transaction, we randomly choose to do DB or non-DB work

- If DB: Insert a row, then select. Each is a child span.

- If non-DB: We do two “fake tasks” (child spans).

Once we finish, we close the database connection and wait 5 seconds for data flush.

Sample instruction file

CMakeLists.txt : This file contains instructions describing the source files and targets.

    cmake_minimum_required(VERSION 3.10)
    project(OtelApp VERSION 1.0)

    set(CMAKE_CXX_STANDARD 11)
    set(CMAKE_CXX_STANDARD_REQUIRED ON)

    # Here we are pointing to use the vcpkg toolchain
    set(CMAKE_TOOLCHAIN_FILE "PATH-TO/vcpkg.cmake" CACHE STRING "Vcpkg toolchain file")

    find_package(opentelemetry-cpp CONFIG REQUIRED)

    add_executable(otel_app main.cpp)

    # Below we are linking the OTLP gRPC exporter, trace library, and sqlite3
    target_link_libraries(otel_app PRIVATE
        opentelemetry-cpp::otlp_grpc_exporter
        opentelemetry-cpp::trace
        sqlite3
    )

Declare Environmental Variables

Here we are going to export our Elastic Cloud endpoints as environmental variables

You can get that information by doing the following:

1. Login into your elastic cloud

2. Go into your deployment

3. On the Left hand side, click on the hamburger menu and scroll down to “Integrations”

4. Go on the search bar inside the integration and type “APM”

1. Click on the APM integration

2. Scroll down and click on the OpenTelemetry Option on the far left side

1. You should be able to see values similar to the screenshot below. Once you copy the values to export, click on launch APM.

As you copy the required values, go ahead and export them.

    export OTEL_EXPORTER_OTLP_ENDPOINT="APM-ENDPOINT"
    export OTEL_EXPORTER_OTLP_HEADERS="KEY"
    export OTEL_RESOURCE_ATTRIBUTES="service.name=my-app,service.version=1.0.0,deployment.environment=dev"

Note that the elastic OTEL_EXPORTER_OTLP_HEADERS value usually starts with “Authorization=Bearer” make sure that you convert the upper case “A” in authorization to a lower case “a”. This is due to the fact that the otel header exporter expects a lower case “a” for authorization.

Build and Run

Once we create the two files we then move to building the application.

cd ~/otel-app
mkdir -p build
cd build

cmake -DCMAKE_TOOLCHAIN_FILE=~/vcpkg/scripts/buildsystems/vcpkg.cmake \
      -DCMAKE_PREFIX_PATH=~/vcpkg/installed/x64-linux/share \
      ..
make

Once make is successful run the the application

./otel-app

You should be able to see the script execute with a similar console output

    Console outcome:
    === Starting transaction_1 ===
    [TRACE] Doing non-DB work for non_db_task_1...
    [TRACE] Doing non-DB work for non_db_task_2...

    === Starting transaction_2 ===
    [TRACE] Doing DB work for doDb_task_1...
    [TRACE] Doing DB work for doDb_task_2...

    === Starting transaction_3 ===
    [TRACE] Doing non-DB work for non_db_task_1...
    [TRACE] Doing non-DB work for non_db_task_2...

    === Starting transaction_4 ===
    [TRACE] Doing non-DB work for non_db_task_1...
    [TRACE] Doing non-DB work for non_db_task_2...

    === Starting transaction_5 ===
    [TRACE] Doing non-DB work for non_db_task_1...
    [TRACE] Doing non-DB work for non_db_task_2...

    [INFO] Sleeping 5 seconds to allow flush...
    [INFO] Exiting.

Once the script executes you should be able to observe those traces on Elastic APM similar to the screenshots below.

Observe in Elastic APM

Go to Elastic Cloud, open your deployment, and navigate to Observability > APM.

Look for the app name in the service list (as defined by OTEL_RESOURCE_ATTRIBUTES).

Inside that service’s Traces tab, you’ll find multiple transactions like “transaction_1”,

“transaction_2”, etc.

Expanding each transaction shows child spans:

- Possibly db_insert_item and db_select_items if random DB path was taken.

- Otherwise, non_db_task_1 and non_db_task_2.

You can see how some transactions do DB calls, some do not, each with different spans.

This variety demonstrates how your real application might produce multiple different

“routes” or “operations.”

Service Map

If everything runs correctly, you should be able to view your services and see service maps for your application.

Services

My Elastic App

App Transactions

Dependencies

Logs

Navigate to your logs window/Discover to see the incoming application logs

Patterns

Log pattern analysis helps you to find patterns in unstructured log messages and makes it easier to examine your data.

Final Recap

Here is a quick summary of what we did:

• Provisioned an Ubuntu 22.04 machine.

• Installed build tools for SQLite, dev libs, and vcpkg.

• Installed the client for opentelemetry-cpp via vcpkg.

• Created a minimal C++ project that executes app traces and captures database operations.

• Connected database sqlite3 in CMakeLists.txt.

• Exported the Elastic OTLP endpoint & token as environment variables (with a lowercase authorization=Bearer key!).

• Ran the application and observed DB interactions and app traces in Elastic APM.

• Observed application logs and patterns on Elastic logs and Discover.

FAQ & Common Issues

• Getting “Could not find package configuration file provided by opentelemetry-cpp”?

Make sure you pass

-DCMAKE_TOOLCHAIN_FILE=... and -DCMAKE_PREFIX_PATH=... 

to cmake, or embed them in CMakeLists.txt.

• Crash: “validate_metadata: INTERNAL:Illegal header key”?

Use all-lowercase in

OTEL_EXPORTER_OTLP_HEADERS, e.g. authorization=Bearer \<token>.

• Missing otlp_grpc_metrics_exporter.h?

Your vcpkg version of opentelemetry-cpp (1.18.0) lacks a direct metrics exporter for OTLP. For metrics, either upgrade the library or consider an OpenTelemetry Collector approach.

• No data in Elastic APM?

Double-check your endpoint URL, Bearer token, firewall rules, or service name in the APM

Additional Resources:

• Sign up for Elastic Cloud free trial
• More Elastic OpenTelemetry Topics
• Introducing Elastic Distributions of OpenTelemetry
• Introducing Elastic Distribution of OpenTelemetry Collector
• Instrumenting your OpenAI- powered Python, Node.js, and Java Applications with EDOT

Elastic provides the ability to ingest OpenTelemetry data with native OTel SDKs, the off the shelf OTel collector, or even Elastic’s Distributions of OpenTelemetry (EDOT). EDOT enables you to bring in logs, metrics and traces for your GenAI application and for K8s. However you will also generally need libraries to help trace specific components in your application. In tracing GenAI applications you can pick from a large set of libraries.

• OpenTelemetry OpenAI Instrumentation-v2 - allows tracing LLM requests and logging of messages made by the OpenAI Python API library. (note v2 is built by OpenTelemetry, the non v2 version is from a specific vendor and not OpenTelemetry)

• OpenTelemetry VertexAI Instrumentation - allows tracing LLM requests and logging of messages made by the VertexAI Python API library

• Langtrace - commercially available library which supports all LLMs in one library, and all traces are also OTel native.

• Elastic’s EDOT - which recently added tracing. See blog.

As you can see OpenTelemetry is the defacto mechanism that is converging to collect and ingest. OpenTelemetry is growing its support for this but it is also early days.

In this blog, we will walk through how to, with minimal code, observe a RAG based chatbot application with tracing using Langtrace. We previously covered Langtrace in a blog to highlight tracing Langchain.

In this blog we used langtrace OpenAI, Amazon Bedrock, Cohere, and others in one library.

Pre-requisites:

In order to follow along, these few pre-requisites are needed

• An Elastic Cloud account — sign up now, and become familiar with Elastic’s OpenTelemetry configuration. With Serverless no version required. With regular cloud minimally 8.17

• Git clone the RAG based Chatbot application and go through the tutorial on how to bring it up and become more familiar.

• An account on your favorite LLM (OpenAI, AzureOpen AI, etc), with API keys

• Be familiar with EDOT to understand how we bring in logs, metrics, and traces from the application through the OTel Collector

• Kubernetes cluster - I’ll be using Amazon EKS

• Look at Langtrace documentation also.

Application OpenTelemetry output in Elastic

Chatbot-rag-app

The first item that you will need to get up and running is the ChatBotApp, and once up you should see the following:

As you select some of the questions you will set a response based on the index that was created in Elasticsearch when the app initializes. Additionally there will be queries that are made to LLMs.

Traces, logs, and metrics from EDOT in Elastic

Once you have OTel Collector with EDOT configuration on your K8s cluster, and Elastic Cloud up and running you should see the following:

Logs:

In Discover you will see logs from the Chatbotapp, and be able to analyze the application logs, any specific log patterns (saves you time in analysis), and view logs from K8s.

Traces:

In Elastic Observability APM, you can also see tha chatbot details, which include transactions, dependencies, logs, errors, etc.

When you look at traces, you will be able to see the chatbot interactions in the trace.

1. You will see the end to end http call

2. Individual calls to elasticsearch

3. Specific calls such as invoke actions, and calls to the LLM

You can also get individual details of the traces, and look at related logs, and metrics related to that trace,

Metrics:

In addition to logs, and traces, any instrumented metrics will also get ingested into Elastic.

Setting it all up

In order to properly set up the Chatbot-app on K8s with telemetry sent over to Elastic, a few things must be set up:

1. Git clone the chatbot-rag-app, and modify one of the python files.

2. Next create a docker container that can be used in Kubernetes. The Docker build here in the Chatbot-app is good to use.

3. Collect all needed env variables. In this example we are using OpenAI, but the files can be modified for any of the LLMs. Hence you will have to get a few environmental variables loaded into the cluster. In the github repo there is a env.example for docker. You can pick and chose what is needed or not needed and adjust appropriately in the K8s file below.

4. Set up your K8s Cluster, and then install the OpenTelemetry collector with the appropriate yaml file and credentials. This will help collect K8s cluster logs and metrics also.

5. Utilize the two yaml files listed below to ensure you can run it on Kubernetes.

• Init-index-job.yaml - Initiates the index in elasticsearch with the local corporate information

• k8s-deployment-chatbot-rag-app.yaml - initializes the application frontend and backend.

1. Open the app on the load balancer URL against the chatbot-app service in K8s

2. Go to Elasticsearch and look at Discover for logs, go to APM and look for your chatbot-app and review the traces, and finally.

Modify the code for tracing with Langtrace

Once you curl the app and untar, go to the chatbot-rag-app directory:

curl https://codeload.github.com/elastic/elasticsearch-labs/tar.gz/main | 
tar -xz --strip=2 elasticsearch-labs-main/example-apps/chatbot-rag-app
cd elasticsearch-labs-main/example-apps/chatbot-rag-app

Next open the app.py file in the api directory and add the following

from opentelemetry.instrumentation.flask import FlaskInstrumentor

from langtrace_python_sdk import langtrace

langtrace.init(batch=False)

FlaskInstrumentor().instrument_app(app)

into the code:

import os
import sys
from uuid import uuid4

from chat import ask_question
from flask import Flask, Response, jsonify, request
from flask_cors import CORS

from opentelemetry.instrumentation.flask import FlaskInstrumentor

from langtrace_python_sdk import langtrace

langtrace.init(batch=False)

app = Flask(__name__, static_folder="../frontend/build", static_url_path="/")
CORS(app)

FlaskInstrumentor().instrument_app(app)

@app.route("/")

See the items in BOLD which will add in the langtrace library, and the opentelemetry flask instrumentation. This combination will provide and end to end trace for the https call all the way down to the calls to Elasticsearch, and to OpenAI (or other LLMs).

Create the docker container

Use the Dockerfile that is in the chatbot-rag-app directory as is and add the following line:

RUN pip3 install --no-cache-dir langtrace-python-sdk

into the Dockerfile:

COPY requirements.txt ./requirements.txt
RUN pip3 install -r ./requirements.txt
RUN pip3 install --no-cache-dir langtrace-python-sdk
COPY api ./api
COPY data ./data

EXPOSE 4000

This enables the langtrace-python-sdk to be installed into the docker container so the langtrace libraries can be used properly.

Collecting the proper env variables:

First collect the env variables from Elastic:

Envs for index initialization in Elastic:

ELASTICSEARCH_URL=https://aws.us-west-2.aws.found.io
ELASTICSEARCH_USER=elastic
ELASTICSEARCH_PASSWORD=elastic

# The name of the Elasticsearch indexes
ES_INDEX=workplace-app-docs
ES_INDEX_CHAT_HISTORY=workplace-app-docs-chat-history

The ELASTICSEARCH_URL can be found in cloud.elastic.co when you bring up your instance. The user and password, you will need to setup in Elastic.

Envs for sending the OTel instrumentation you will need the following:

OTEL_EXPORTER_OTLP_ENDPOINT="https://123456789.apm.us-west-2.aws.cloud.es.io:443"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer xxxxx"

These credentials are found in Elastic under APM integration and under OpenTelemetry

Envs for LLMs

In this example we’re using OpenAI, hence only three variables are needed.

LLM_TYPE=openai
OPENAI_API_KEY=XXXX
CHAT_MODEL=gpt-4o-mini

All these variables will be needed in the Kubernetes yamls in the next step

Setup K8s cluster and load up OTel Collector with EDOT

This step is outlined in the following Blog. It’s a simple three step process.

This step will bring in all the K8s cluster logs and metrics and setup the OTel collector.

Setup secrets, initialize indices, and start the app

Now that the cluster is up, and you have your environmental variables, you will need to

1. Install and run the k8s-deployments.yaml with the variables

2. Initialize the index

Essentially run the following:

kubectl create -f k8s-deployment.yaml
kubectl create -f init-index-job.yaml

Here are the two yamls you should use. Also found here

k8s-deployment.yaml

apiVersion: v1
kind: Secret
metadata:
  name: genai-chatbot-langtrace-secrets
type: Opaque
stringData:
  OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer%20xxxx"
  OTEL_EXPORTER_OTLP_ENDPOINT: "https://1234567.apm.us-west-2.aws.cloud.es.io:443"
 ELASTICSEARCH_URL: "YOUR_ELASTIC_SEARCH_URL"
  ELASTICSEARCH_USER: "elastic"
  ELASTICSEARCH_PASSWORD: "elastic"
  OPENAI_API_KEY: "XXXXXXX"  

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: genai-chatbot-langtrace
spec:
  replicas: 2
  selector:
    matchLabels:
      app: genai-chatbot-langtrace
  template:
    metadata:
      labels:
        app: genai-chatbot-langtrace
    spec:
      containers:
      - name: genai-chatbot-langtrace
        image:65765.amazonaws.com/genai-chatbot-langtrace2:latest
        ports:
        - containerPort: 4000
        env:
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: OTEL_SDK_DISABLED
          value: "false"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=genai-chatbot-langtrace,service.version=0.0.1,deployment.environment=dev"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "http/protobuf"
        envFrom:
        - secretRef:
            name: genai-chatbot-langtrace-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

---
apiVersion: v1
kind: Service
metadata:
  name: genai-chatbot-langtrace-service
spec:
  selector:
    app: genai-chatbot-langtrace
  ports:
  - port: 80
    targetPort: 4000
  type: LoadBalancer

Init-index-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: init-elasticsearch-index-test
spec:
  template:
    spec:
      containers:
      - name: init-index
#update your image location for chatbot rag app
        image: your-image-location:latest
        workingDir: /app/api
        command: ["python3", "-m", "flask", "--app", "app", "create-index"]
        env:
        - name: FLASK_APP
          value: "app"
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: ES_INDEX
          value: "workplace-app-docs"
        - name: ES_INDEX_CHAT_HISTORY
          value: "workplace-app-docs-chat-history"
        - name: ELASTICSEARCH_URL
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_URL
        - name: ELASTICSEARCH_USER
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_USER
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_PASSWORD
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
      restartPolicy: Never
  backoffLimit: 4

Open App with LoadBalancer URL

Run the kubectl get services command and get the URL for the chatbot app

% kubectl get services
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                                                                     AGE
chatbot-langtrace-service            LoadBalancer   10.100.130.44    xxxxxxxxx-1515488226.us-west-2.elb.amazonaws.com   80:30748/TCP                                                                6d23h

Play with app and review telemetry in Elastic

Once you go to the URL, you should see all the screens we described earlier in the beginning of this blog.

Conclusion

With Elastic's Chatbot-rag-app you have an example of how to build out a OpenAI driven RAG based chat application. However, you still need to understand how well it performs, whether its working properly, etc. Using OTel, Elastic’s EDOT and Langtrace gives you the ability to achieve this. Additionally, you will generally run this application on Kubernetes. Hopefully this blog provides the outline of how to achieve this.

Here are the other Tracing blogs:

App Observability with LLM (Tracing)-

• Observing LangChain with Langtrace and OpenTelemetry

• Observing LangChain with OpenLit Tracing

• Instrumenting LangChain with OpenTelemetry

LLM Observability -

• Elevate LLM Observability with GCP Vertex AI Integration

• LLM Observability on AWS Bedrock

• LLM Observability for Azure OpenAI

• LLM Observability for Azure OpenAI v2

This app is also now insturmented with EDOT, and you can visualize the Chatbot's traces to OpenAI, as well as relevant logs, and metrics from the application. By running the app as instructed in the github repo with Docker you can see these traces on a local stack. But how about running it against serverless, Elastic cloud or even with Kubernetes?

In this blog we will walk through how to set up Elastic's RAG Based Chatbot application with Elastic cloud and Kubernetes.

Prerequisites:

In order to follow along, these few pre-requisites are needed

• An Elastic Cloud account — sign up now, and become familiar with Elastic's OpenTelemetry configuration. With Serverless no version required. With regular cloud minimally 8.17

• Git clone the RAG based Chatbot application and go through the tutorial on how to bring it up and become more familiar and how to bring up the application using Docker.

• An account on OpenAI with API keys

• Kubernetes cluster to run the RAG based Chatbot app

• The instructions in this blog are also found in observability-examples in github.

Application OpenTelemetry output in Elastic

Chatbot-rag-app

The first item that you will need to get up and running is the ChatBotApp, and once up you should see the following:

As you select some of the questions you will set a response based on the index that was created in Elasticsearch when the app initializes. Additionally there will be queries that are made to LLMs.

Traces, logs, and metrics from EDOT in Elastic

Once you have the application running on your K8s cluster or with Docker, and Elastic Cloud up and running you should see the following:

Logs:

In Discover you will see logs from the Chatbotapp, and be able to analyze the application logs, any specific log patterns, which saves you time in analysis.

Traces:

In Elastic Observability APM, you can also see tha chatbot details, which include transactions, dependencies, logs, errors, etc.

When you look at traces, you will be able to see the chatbot interactions in the trace.

1. You will see the end to end http call

2. Individual calls to elasticsearch

3. Specific calls such as invoke actions, and calls to the LLM

You can also get individual details of the traces, and look at related logs, and metrics related to that trace,

Metrics:

In addition to logs, and traces, any instrumented metrics will also get ingested into Elastic.

Setting it all up with Docker

In order to properly set up the Chatbot-app on Docker with telemetry sent over to Elastic, a few things must be set up:

1. Git clone the chatbot-rag-app

2. Modify the env file as noted in the github README with the following exception:

Use your Elastic cloud's OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADER instead.

You can find these in the Elastic Cloud under integrations->APM

Envs for sending the OTel instrumentation you will need the following:

OTEL_EXPORTER_OTLP_ENDPOINT="https://123456789.apm.us-west-2.aws.cloud.es.io:443"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20xxxxx"

Notice the %20 in the headers. This will be needed to account for the space in credentials.

1. Set the following to false - OTEL_SDK_DISABLED=false

2. Set the envs for LLMs

In this example we're using OpenAI, hence only three variables are needed.

LLM_TYPE=openai
OPENAI_API_KEY=XXXX
CHAT_MODEL=gpt-4o-mini

1. Run the docker container as noted

docker compose up --build --force-recreate

1. Play with the app at localhost:4000

2. Then log into Elastic cloud and see the output as shown previously.

Run chatbot-rag-app on Kubernetes

In order to set this up, you can follow the following repo on Observability-examples which has the Kubernetes yaml files being used. These will also point to Elastic Cloud.

1. Set up the Kubernetes Cluster (we're using EKS)

2. Get the appropriate ENV variables:

• Find the OTEL_EXPORTER_OTLP_ENDPOINT/HEADER variables as noted in the pervious for Docker.

• Get your OpenAI Key

• Elasticsearch URL, and username and password.

1. Follow the instructions in the following github repo in observability examples to run two Kubernetes yaml files.

Essentially you need only replace the secret variables in k8s-deployment.yaml, and run

kubectl create -f k8s-deployment.yaml
kubectl create -f init-index-job.yaml

The app needs to be running first, then we use the app to initialize Elasticsearch with indices for the app.

Init-index-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: init-elasticsearch-index-test
spec:
  template:
    spec:
      containers:
      - name: init-index
        image: ghcr.io/elastic/elasticsearch-labs/chatbot-rag-app:latest
        workingDir: /app/api
        command: ["python3", "-m", "flask", "--app", "app", "create-index"]
        env:
        - name: FLASK_APP
          value: "app"
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: ES_INDEX
          value: "workplace-app-docs"
        - name: ES_INDEX_CHAT_HISTORY
          value: "workplace-app-docs-chat-history"
        - name: ELASTICSEARCH_URL
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_URL
        - name: ELASTICSEARCH_USER
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_USER
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_PASSWORD
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
      restartPolicy: Never
  backoffLimit: 4

k8s-deployment.yaml

apiVersion: v1
kind: Secret
metadata:
  name: chatbot-regular-secrets
type: Opaque
stringData:
  ELASTICSEARCH_URL: "https://yourelasticcloud.es.us-west-2.aws.found.io"
  ELASTICSEARCH_USER: "elastic"
  ELASTICSEARCH_PASSWORD: "elastic"
  OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer%20xxxx"
  OTEL_EXPORTER_OTLP_ENDPOINT: "https://12345.apm.us-west-2.aws.cloud.es.io:443"
  OPENAI_API_KEY: "YYYYYYYY"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chatbot-regular
spec:
  replicas: 2
  selector:
    matchLabels:
      app: chatbot-regular
  template:
    metadata:
      labels:
        app: chatbot-regular
    spec:
      containers:
      - name: chatbot-regular
        image: ghcr.io/elastic/elasticsearch-labs/chatbot-rag-app:latest
        ports:
        - containerPort: 4000
        env:
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=chatbot-regular,service.version=0.0.1,deployment.environment=dev"
        - name: OTEL_SDK_DISABLED
          value: "false"
        - name: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT
          value: "true"
        - name: OTEL_EXPERIMENTAL_RESOURCE_DETECTORS
          value: "process_runtime,os,otel,telemetry_distro"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "http/protobuf"
        - name: OTEL_METRIC_EXPORT_INTERVAL
          value: "3000"
        - name: OTEL_BSP_SCHEDULE_DELAY
          value: "3000"
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

---
apiVersion: v1
kind: Service
metadata:
  name: chatbot-regular-service
spec:
  selector:
    app: chatbot-regular
  ports:
  - port: 80
    targetPort: 4000
  type: LoadBalancer

Open App with LoadBalancer URL

Run the kubectl get services command and get the URL for the chatbot app

% kubectl get services
NAME                                 TYPE           CLUSTER-IP    EXTERNAL-IP                                                               PORT(S)                                                                     AGE
chatbot-regular-service            LoadBalancer   10.100.130.44    xxxxxxxxx-1515488226.us-west-2.elb.amazonaws.com   80:30748/TCP                                                                6d23h

1. Play with app and review telemetry in Elastic

2. Once you go to the URL, you should see all the screens we described earlier in the beginning of this blog.

Conclusion

With Elastic's Chatbot-rag-app you have an example of how to build out a OpenAI driven RAG based chat application. However, you still need to understand how well it performs, whether its working properly, etc. Using OTel and Elastic’s EDOT gives you the ability to achieve this. Additionally, you will generally run this application on Kubernetes. Hopefully this blog provides the outline of how to achieve this. Here are the other Tracing blogs:

App Observability with LLM (Tracing)-

• Observing LangChain with Langtrace and OpenTelemetry

• Observing LangChain with OpenLit Tracing

• Instrumenting LangChain with OpenTelemetry

LLM Observability -

• Elevate LLM Observability with GCP Vertex AI Integration

• LLM Observability on AWS Bedrock

• LLM Observability for Azure OpenAI

• LLM Observability for Azure OpenAI v2

DevOps engineers continuously optimize software delivery, while SRE teams act as the stewards of application reliability, scalability, and top-tier performance. The challenge? These teams require a cutting-edge observability solution, one that encompasses full-stack insights, empowering them to rapidly manage, monitor, and rectify potential disruptions before they culminate into operational challenges.

Observability in our modern distributed software ecosystem goes beyond mere monitoring — it demands limitless data collection, precision in processing, and the correlation of this data into actionable insights. However, the road to achieving this holistic view is paved with obstacles, from navigating version incompatibilities to wrestling with restrictive proprietary code.

Enter OpenTelemetry (OTel), with the following benefits for those who adopt it:

• Escape vendor constraints with OTel, freeing yourself from vendor lock-in and ensuring top-notch observability.
• See the harmony of unified logs, metrics, and traces come together to provide a complete system view.
• Improve your application oversight through richer and enhanced instrumentations.
• Embrace the benefits of backward compatibility to protect your prior instrumentation investments.
• Embark on the OpenTelemetry journey with an easy learning curve, simplifying onboarding and scalability.
• Rely on a proven, future-ready standard to boost your confidence in every investment.

In this blog, we will explore how you can use automatic instrumentation in your Go application using Docker, without the need to refactor any part of your application code. We will use an application called Elastiflix, which helps highlight auto-instrumentation in a simple way.

Application, prerequisites, and config

The application that we use for this blog is called Elastiflix, a movie-streaming application. It consists of several micro-services written in .NET, NodeJS, Go, and Python.

Before we instrument our sample application, we will first need to understand how Elastic can receive the telemetry data.

All of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Prerequisites

• An Elastic Cloud account — sign up now.
• A clone of the Elastiflix demo application, or your own Go application
• Basic understanding of Docker — potentially install Docker Desktop
• Basic understanding of Go

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Step-by-step guide

Step 0. Log in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Run the Docker Image with auto-instrumentation

We are going to use automatic instrumentation with the Go service from the Elastiflix demo application.

We will be using the following service from Elastiflix:

Elastiflix/go-favorite

Per the OpenTelemetry Automatic Instrumentation for Go documentation, you will configure the application to be auto-instrumented using docker-compose.

As specified in the OTEL Go documentation, we will use environment variables and pass in the configuration values to enable it to connect with Elastic Observability’s APM server.

Because Elastic accepts OTLP natively, we just need to provide the Endpoint and authentication where the OTEL Exporter needs to send the data, as well as some other environment variables.

Getting Elastic Cloud variables You can copy the endpoints and token from Kibana under the path /app/apm/onboarding?agent=openTelemetry.

You will need to copy the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS

Update the docker-compose.yml file at the top of the Elastiflix repository, adding a go-auto service and updating the favorite-go one:

  favorite-go:
    build: go-favorite/.
    image: docker.elastic.co/demos/workshop/observability/elastiflix-go-favorite:${ELASTIC_VERSION}-${BUILD_NUMBER}
    depends_on:
      - redis
    networks:
      - app-network
    ports:
      - "5001:5000"
    environment:
      - REDIS_HOST=redis
      - TOGGLE_SERVICE_DELAY=${TOGGLE_SERVICE_DELAY:-0}
      - TOGGLE_CANARY_DELAY=${TOGGLE_CANARY_DELAY:-0}
      - TOGGLE_CANARY_FAILURE=${TOGGLE_CANARY_FAILURE:-0}
    volumes:
      - favorite-go:/app
  go-auto:
    image: otel/autoinstrumentation-go
    privileged: true
    pid: "host"
    networks:
      - app-network
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: "REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT"
      OTEL_EXPORTER_OTLP_HEADERS: "REPLACE WITH OTEL_EXPORTER_OTLP_HEADERS"
      OTEL_GO_AUTO_TARGET_EXE: "/app/main"
      OTEL_SERVICE_NAME: "go-favorite"
      OTEL_PROPAGATORS: "tracecontext,baggage"
    volumes:
      - favorite-go:/app
      - /proc:/host/proc

And, at the bottom of the file:

volumes:
  favorite-go:
networks:
  app-network:
    driver: bridge

Finally, in the configuration for the main node app, you will want to tell Elastiflix to call the Go favorites app by replacing the line:

environment:
  - API_ENDPOINT_FAVORITES=favorite-java:5000

with:

environment:
  - API_ENDPOINT_FAVORITES=favorite-go:5000

Step 3: Explore traces and logs in Elastic APM

Once you have this up and running, you can ping the endpoint for your instrumented service (in our case, this is /favorites), and you should see the app appear in Elastic APM, as shown below:

It will begin by tracking throughput and latency critical metrics for SREs to pay attention to.

Digging in, we can see an overview of all our Transactions.

And look at specific transactions:

This gives you complete visibility across metrics, and traces!

Summary

With this Dockerfile, you've transformed your simple Go application into one that's automatically instrumented with OpenTelemetry. This will aid greatly in understanding application performance, tracing errors, and gaining insights into how users interact with your software.

Remember, observability is a crucial aspect of modern application development, especially in distributed systems. With tools like OpenTelemetry, understanding complex systems becomes a tad bit easier.

In this blog, we discussed the following:

• How to auto-instrument Go with OpenTelemetry.
• Using standard commands in a Docker file, auto-instrumentation was done efficiently and without adding code in multiple places enabling manageability.
• Using OpenTelemetry and its support for multiple languages, DevOps and SRE teams can auto-instrument their applications with ease gaining immediate insights into the health of the entire application stack and reduce mean time to resolution (MTTR).

Since Elastic can support a mix of methods for ingesting data, whether it be using auto-instrumentation of open-source OpenTelemetry or manual instrumentation with its native APM agents, you can plan your migration to OTel by focusing on a few applications first and then using OpenTelemety across your applications later on in a manner that best fits your business needs.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry

• Python: Auto-instrumentation, Manual-instrumentation

• Java: Auto-instrumentation, Manual-instrumentation

• Node.js: Auto-instrumentation, Manual-instrumentation

• .NET: Auto-instrumentation, Manual-instrumentation

• Go: Auto-instrumentation Manual-instrumentation

• Best practices for instrumenting OpenTelemetry

  General configuration and use case resources:

• Independence with OpenTelemetry on Elastic

• Modern observability and security on Kubernetes with Elastic and OpenTelemetry

• 3 models for logging with OpenTelemetry and Elastic

• Adding free and open Elastic APM as part of your Elastic Observability deployment

• Capturing custom metrics through OpenTelemetry API in code with Elastic

• Future-proof your observability platform with OpenTelemetry and Elastic

• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all._

There are several options to trace for Langchain. Langtrace is one such option. Langtrace is an open-source observability software that lets you capture, debug and analyze traces and metrics from all your applications. Langtrace automatically captures traces from LLM APIs/inferences, Vector Databases, and LLM-based Frameworks. Langtrace stands out due to its seamless integration with popular LLM frameworks and its ability to provide deep insights into complex AI workflows without requiring extensive manual instrumentation.

Langtrace has an SDK, a lightweight library that can be installed and imported into your project to collect traces. The traces are OpenTelemetry-based and can be exported to Elastic without using a Langtrace API key.

OpenTelemetry (OTel) is now broadly accepted as the industry standard for tracing. As one of the major Cloud Native Computing Foundation (CNCF) projects, with as many commits as Kubernetes, it is gaining support from major ISVs and cloud providers delivering support for the framework. 

Hence, many LangChain-based applications will have multiple components beyond just LLM interactions. Using OpenTelemetry with LangChain is essential. 

This blog will cover how you can use Langtrace SDK to trace a simple LangChain Chat app connecting to Azure OpenAI, perform a search in DuckDuckGoSearch and export the output to Elastic.

Pre-requisites:

• An Elastic Cloud account — sign up now, and become familiar with Elastic’s OpenTelemetry configuration

• Have a LangChain app to instrument

• Be familiar with using OpenTelemetry’s Python SDK 

• An account on your favorite LLM (AzureOpen AI), with API keys

• The application we used in this blog, called langchainChat can be found in Github langhcainChat. It is built using Azure OpenAI and DuckDuckGo, but you can easily modify it for your LLM and search of choice.

App Overview and output in Elastic:

To showcase the combined power of Langtrace and Elastic, we created a simple LangChain app that performs the following steps:

1. Takes customer input on the command line. (Queries)

2. Sends these to the Azure OpenAI LLM via a LangChain.

3. Utilizes chain tools to perform a search using DuckDuckGo.

4. The LLM processes the search results and returns the relevant information to the user.

Here is a sample interaction:

Here is what the service view looks like after we ran a few queries.

As you can see, Elastic Observability’s APM recognizes the LangChain app and also shows the average latency, throughput, and transactions. Our average latency is 30s since it takes that log for humans to type the query (twice).

You can also select other tabs to see, dependencies, errors, metrics, and more. One interesting part of Elastic APM is the ability to use universal profiling (eBPF) output also analyzed for this service. Here is what our service’s dependency is (Azure OpenAI) with its average latency, throughput, and failed transactions:

We see Azure OpenAI is on average 4s to give us the results.

If we drill into transactions and look at the trace for our queries on Taylor Swift and Pittsburgh Steelers, we can see both queries and their corresponding spans.

In this trace:

1. The user makes a query

2. Azure OpenAI is called, but it uses a tool (DuckDuckGo) to obtain some results

3. Azure OpenAI reviews and returns a summary to the end user

4. Repeats for another query

We noticed that the other long span (other than Azure OpenAI) is Duckduckgo (~1000ms). We can individually look at the span and review the data:

Configuration:

How do we make all this show up in Elastic? Let's go over the steps:

OpenTelemetry Configuration

To leverage the full capabilities of OpenTelemetry with Langtrace and Elastic, we need to configure the SDK to generate traces and properly set up Elastic’s endpoint and authorization. Detailed instructions can be found in the OpenTelemetry Auto-Instrumentation setup documentation.

OpenTelemetry Environment variables:

For Elastic, you can set the following OpenTelemetry environment variables either in your Linux/Mac environment or directly in the code:

OTEL_EXPORTER_OTLP_ENDPOINT=12345.apm.us-west-2.aws.cloud.es.io:443
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20ZZZZZZZ"
OTEL_RESOURCE_ATTRIBUTES="service.name=langchainChat,service.version=1.0,deployment.environment=production"

In this setup:

• OTEL_EXPORTER_OTLP_ENDPOINT is configured to send traces to Elastic.

• OTEL_EXPORTER_OTLP_HEADERS provides the necessary authorization for the Elastic APM server.

• OTEL_RESOURCE_ATTRIBUTES define key attributes like the service name, version, and deployment environment.

These values can be easily obtained from Elastic’s APM configuration screen under the OpenTelemetry section.

Note: No agent is required; the OTLP trace messages are sent directly to Elastic’s APM server, simplifying the setup process.

Langtrace Library:

OpenTelemetry's auto-instrumentation can be extended to trace additional frameworks using instrumentation packages. For this blog post, you will need to install the Langtrace Python SDK:

pip install langtrace-python-sdk 

After installation, you can add the following code to your project:

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

from langtrace_python_sdk import langtrace, with_langtrace_root_span

Instrumentation:

Once the necessary libraries are installed and the environment variables are configured, you can use auto-instrumentation to trace your application. For example, run the following command to instrument your LangChain application with Elastic:

opentelemetry-instrument python langtrace-elastic-demo.py

The Langtrace OpenTelemetry library correctly captures the flow with minimal manual instrumentation, apart from integrating the OpenTelemetry library. Additionally, the LLM spans captured by Langtrace also include useful metadata such as token counts, model hyper-parameter settings etc. Note that the generated spans follow the OTEL GenAI semantics described here.

In summary, the instrumentation process involves:

1. Capturing customer input from the command line (Queries).

2. Sending these queries to the Azure OpenAI LLM via a LangChain.

3. Utilizing chain tools, such as DuckDuckGo, to perform searches.

4. The LLM processes the results and returns the relevant information to the user.

Conclusion

By combining the power of Langtrace with Elastic, developers can achieve unparalleled visibility into their LangChain applications, ensuring optimized performance and quicker debugging. This powerful combination simplifies the complex task of monitoring AI-driven systems, enabling you to focus on what truly matters—delivering value to your users. Throughout this blog,we've covered the following essential steps and concepts:

• How to manually instrument Langchain with OpenTelemetry

• How to properly initialize OpenTelemetry and add a custom span

• How to easily set the OTLP ENDPOINT and OTLP HEADERS with Elastic without the need for a collector

• How to view and analyze traces in Elastic Observability APM

These steps provide a clear and actionable guide for developers looking to integrate robust tracing capabilities into their LangChain applications.

We hope this guide makes understanding and implementing OpenTelemetry tracing for LangChain simple, ensuring seamless integration with Elastic.

Additional resources for OpenTelemetry with Elastic:

• Independence with OpenTelemetry on Elastic

• Modern observability and security on Kubernetes with Elastic and OpenTelemetry

• 3 models for logging with OpenTelemetry and Elastic

• Adding free and open Elastic APM as part of your Elastic Observability deployment

• Monitor OpenAI API and GPT models with OpenTelemetry and Elastic

• Futureproof your observability platform with OpenTelemetry and Elastic

• Instrumentation resources:

• Python: Auto-instrumentation, Manual instrumentation

• Java: Auto-instrumentation, Manual instrumentation 

• Node.js: Auto-instrumentation, Manual instrumentation

• .NET: Auto-instrumentation, Manual instrumentation

• Elastic APM - Langtrace AI Docs

Consider this scenario: You have a transaction labeled simply as "HTTP GET" with an average response time of 5ms. However, this generic label masks a variety of distinct operations – payment processing, user logins, and adding items to a cart. Does that 5ms average truly represent the performance of these diverse actions? Clearly not.

The other problem that happens is that span traces become all mixed up so that login spans and image serving spans all become part of the same bucket, this makes things like latency correlation analysis hard in Elastic.

We'll focus on a specific technique using the collector's attributes, and transform processors to extract meaningful information from HTTP URLs and use it to create more descriptive span names. This approach not only improves the accuracy of your metrics but also enhances your ability to quickly identify and troubleshoot performance issues across your microservices architecture.

By using these processors in combination, we can quickly address the issue of overly generic transaction names, creating more granular and informative identifiers that provide accurate visibility into your services' performance.

However, it's crucial to approach this technique with caution. While more detailed transaction names can significantly improve observability, they can also lead to an unexpected challenge: cardinality explosion. As we dive into the implementation details, we'll also discuss how to strike the right balance between granularity and manageability, ensuring that our solution enhances rather than overwhelms our observability stack.

In the following sections, we'll walk through the configuration step-by-step, explaining how each processor contributes to our goal, and highlighting best practices to avoid potential pitfalls like cardinality issues. Whether you're new to OpenTelemetry or looking to optimize your existing setup, this guide will help you unlock more meaningful insights from your telemetry data.

Prerequisites and configuration

If you plan on following this blog, here are some of the components and details we used to set up the configuration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• I am also using the OpenTelemetry demo in my environment, this is important to follow along with as this demo has the specific issue I want to address. You should clone the repository and follow the instructions here to get this up and running. I recommend using Kubernetes and I will be doing this in my AWS EKS (Elastic Kubernetes Service) environment.

The OpenTelemetry Demo

The OpenTelemetry Demo is a comprehensive, microservices-based application designed to showcase the capabilities and best practices of OpenTelemetry instrumentation. It simulates an e-commerce platform, incorporating various services such as frontend, cart, checkout, and payment processing. This demo serves as an excellent learning tool and reference implementation for developers and organizations looking to adopt OpenTelemetry.

The demo application generates traces, metrics, and logs across its interconnected services, demonstrating how OpenTelemetry can provide deep visibility into complex, distributed systems. It's particularly useful for experimenting with different collection, processing, and visualization techniques, making it an ideal playground for exploring observability concepts and tools like the OpenTelemetry Collector.

By using real-world scenarios and common architectural patterns, the OpenTelemetry Demo helps users understand how to effectively implement observability in their own applications and how to leverage the data for performance optimization and troubleshooting.

Once you have an Elastic Cloud instance and you fire up the OpenTelemetry demo, you should see something like this on the Elastic Service Map page:

Navigating to the traces page will give you the following set up.

As you can see there are some very broad transaction names here like HTTP GET and the averages will not be very accurate for specific business functions within your services as shown.

So let's fix that with the OpenTelemetry Collector.

The OpenTelemetry Collector

The OpenTelemetry Collector is a vital component in the OpenTelemetry ecosystem, serving as a vendor-agnostic way to receive, process, and export telemetry data. It acts as a centralized observability pipeline that can collect traces, metrics, and logs from various sources, then transform and route this data to multiple backend systems.

The collector's flexible architecture allows for easy configuration and extension through a wide range of receivers, processors, and exporters which you can explore over here. I have personally found navigating the 'contrib' archive incredibly useful for finding techniques that I didn't know existed. This makes the OpenTelemetry Collector an invaluable tool for organizations looking to standardize their observability data pipeline, reduce overhead, and seamlessly integrate with different monitoring and analysis platforms.

Let's go back to our problem, how do we change the transaction names that Elastic is using to something more useful so that our HTTP GET translates to something like payment-service/login? The first thing we do is we take the full http url and consider which parts of it relate to our transaction. Looking at the span details we see a url

my-otel-demo-frontendproxy:8080/api/recommendations?productIds=&sessionId=45a9f3a4-39d8-47ed-bf16-01e6e81c80bc&currencyCode=

Now obviously we wouldn't want to create transaction names that map to every single session id, that would lead to the cardinality explosion we talked about earlier, however, something like the first two parts of the url 'api/recommendations' looks like exactly the kind of thing we need.

The attributes processor

The OpenTelemetry collector gives us a useful tool here, the attributes processor can help us extract parts of the url to use later in our observability pipeline. To do this is very simple, we simply build a regex like this one below. Now I should mention that I did not generate this regex myself but I used an LLM to do this for me, never fear regex again!

attributes:
  actions:
    - key: http.url
      action: extract
      pattern: '^(?P<short_url>https?://[^/]+(?:/[^/]+)*)(?:/(?P<url_truncated_path>[^/?]+/[^/?]+))(?:\?|/?$)'

This configuration is doing some heavy lifting for us, so let's break it down:

• We're using the attributes processor, which is perfect for manipulating span attributes.
• We're targeting the http.url attribute of incoming spans.
• The extract action tells the processor to pull out specific parts of the URL using our regex pattern.

Now, about that regex - it's designed to extract two key pieces of information:

1. short_url: This captures the protocol, domain, and optionally the first path segment. For example, in "https://example.com/api/users/profile", it would grab "https://example.com/api".
2. url_truncated_path: This snags the next two path segments (if they exist). In our example, it would extract "users/profile".

Why is this useful? Well, it allows us to create more specific transaction names based on the URL structure, without including overly specific details that could lead to cardinality explosion. For instance, we avoid capturing unique IDs or query parameters that would create a new transaction name for every single request.

So, if we have a URL like "https://example.com/api/users/profile?id=123", our extracted url_truncated_path would be "users/profile". This gives us a nice balance - it's more specific than just "HTTP GET", but not so specific that we end up with thousands of unique transaction names.

Now it's worth mentioning here that if you don't have an attribute you want to use for naming your transactions it is worth looking at the options for your SDK or agent, as an example the Java automatic instrumentation Otel agent has the following options for capturing request and response headers. You can then subsequently use this data to name your transactions if the url is insufficient!

In the next steps, we'll see how to use this extracted information to create more meaningful span names, providing better granularity in our observability data without overwhelming our system. Remember, the goal is to enhance our visibility, not to drown in a sea of overly specific metrics!

The transform processor

Now that we've extracted the relevant parts of our URLs, it's time to put that information to good use. Enter the transform processor - our next powerful tool in the OpenTelemetry Collector pipeline.

The transform processor allows us to modify various aspects of our telemetry data, including span names. Here's the configuration we'll use:

transform:
  trace_statements:
    - context: span
      statements:
        - set(name, attributes["url_truncated_path"])

Let's break this down:

• We're using the transform processor, which gives us fine-grained control over our spans.
• We're focusing on trace_statements, as we want to modify our trace spans.
• The context: span tells the processor to apply these changes to each individual span.
• Our statement is where the magic happens: we're setting the span's name to the value of the url_truncated_path attribute we extracted earlier.

What does this mean in practice? Remember our previous example URL "https://example.com/api/users/profile?id=123"? Instead of a generic span name like "HTTP GET", we'll now have a much more informative name: "users/profile".

This transformation brings several benefits:

1. Improved Readability: At a glance, you can now see what part of your application is being accessed.
2. Better Aggregation: You can easily group and analyze similar requests, like all operations on user profiles.
3. Balanced Cardinality: We're specific enough to be useful, but not so specific that we create a new span name for every unique URL.

By combining the attribute extraction we did earlier with this transformation, we've created a powerful system for generating meaningful span names. This approach gives us deep insight into our application's behavior without the risk of cardinality explosion.

Putting it All Together

The resulting config for the OpenTelemetry collector is below remember this goes into the opentelemetry-demo/kubernetes/elastic-helm/configmap-deployment.yaml and is applied with kubectl apply -f configmap-deployment.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: elastic-otelcol-agent
  namespace: default
  labels:
    app.kubernetes.io/name: otelcol

data:
  relay: |
    connectors:
      spanmetrics: {}
    exporters:
      debug: {}
      otlp/elastic:
        endpoint: ${env:ELASTIC_APM_ENDPOINT}
        compression: none
        headers:
          Authorization: Bearer ${ELASTIC_APM_SECRET_TOKEN}
    extensions:
    processors:
      batch: {}
      resource:
        attributes:
          - key: deployment.environment
            value: "opentelemetry-demo"
            action: upsert
      attributes:
        actions:
          - key: http.url
            action: extract
            pattern: '^(?P<short_url>https?://[^/]+(?:/[^/]+)*)(?:/(?P<url_truncated_path>[^/?]+/[^/?]+))(?:\?|/?$)'
      transform:
        trace_statements:
          - context: span
            statements:
              - set(name, attributes["url_truncated_path"])
    receivers:
      httpcheck/frontendproxy:
        targets:
        - endpoint: http://example-frontendproxy:8080
      otlp:
        protocols:
          grpc:
            endpoint: ${env:MY_POD_IP}:4317
          http:
            cors:
              allowed_origins:
              - http://*
              - https://*
            endpoint: ${env:MY_POD_IP}:4318
    service:
      extensions:
      pipelines:
        logs:
          exporters:
          - debug
          - otlp/elastic
          processors:
          - batch
          - resource
          - attributes
          - transform
          receivers:
          - otlp
        metrics:
          exporters:
          - otlp/elastic
          - debug
          processors:
          - batch
          - resource
          receivers:
          - httpcheck/frontendproxy
          - otlp
          - spanmetrics
        traces:
          exporters:
          - otlp/elastic
          - debug
          - spanmetrics
          processors:
          - batch
          - resource
          - attributes
          - transform
          receivers:
          - otlp
      telemetry:
        metrics:
          address: ${env:MY_POD_IP}:8888

You'll notice that we tie everything together by adding our enrichment and transformations to the traces section in pipelines at the bottom of the collector config. This is the definition of our observability pipeline, bringing together all the pieces we've discussed to create more meaningful and actionable telemetry data.

By implementing this configuration, you're taking a significant step towards more insightful observability. You're not just collecting data; you're refining it to provide clear, actionable insights into your application's performance, check out the final result below!

Ready to Take Your Observability to the Next Level?

Implementing OpenTelemetry with Elastic Observability opens up a world of possibilities for understanding and optimizing your applications. But this is just the beginning! To further enhance your observability journey, check out these valuable resources:

1. Infrastructure Monitoring with OpenTelemetry in Elastic Observability
2. Explore More OpenTelemetry Content
3. Using the OTel Operator for Injecting Java Agents
4. What is OpenTelemetry?

We encourage you to dive deeper, experiment with these configurations, and see how they can transform your observability data. Remember, the key is to find the right balance between detail and manageability.

Have you implemented similar strategies in your observability pipeline? We'd love to hear about your experiences and insights. Share your thoughts in the comments below or reach out to us on our community forums.

Stay tuned for Part 2 of this series, where we will look at an advanced technique for collecting more data that can help you get even more granular by collecting Span names, baggage and data for metrics using a Java plugin all without code.

The Elastic Distributions of OpenTelemetry (EDOT) are composed of OpenTelemetry (OTel) project components, OTel Collector, and language SDKs,  which provide users with the necessary capabilities and out-of-the-box configurations, enabling quick and effortless infra and application monitoring.

While OTel components are feature-rich, enhancements through the community can take time. Additionally, support is left up to the community or individual users and organizations. Hence EDOT will bring the following to end users:

• Deliver enhanced features earlier than OTel: By providing features unavailable in the “vanilla” OpenTelemetry components, we can quickly meet customers’ requirements while still providing an OpenTelemetry native and vendor-agnostic instrumentation for their applications. Elastic will continuously upstream these enhanced features.

• Enhanced OTel support - By maintaining Elastic distributions, we can better support customers with enhancements and fixes outside of the OTel release cycles. In addition, Elastic support can troubleshoot issues on the EDOT.

EDOT currently includes the following tech preview components, which will  grow over time:

• Elastic Distribution of OpenTelemetry (EDOT) Collector

• Elastic Distribution of OpenTelemetry (EDOT) Java.

• Elastic Distribution of OpenTelemetry (EDOT) Python

• Elastic Distribution of OpenTelemetry (EDOT) NodeJS

• Elastic Distribution of OpenTelemetry (EDOT) .NET

• Elastic Distribution of OpenTelemetry (EDOT)  iOS and Android

Details and documentation for all EDOT are available in our public OpenTelemetry GitHub repository. 

Elastic Distribution of OpenTelemetry (EDOT) Collector

The EDOT Collector, recently released with the 8.15 release of Elastic Observability enhances Elastic’s existing OTel capabilities. The EDOT Collector can, in addition to service monitoring, forward application logs, infrastructure logs, and metrics using standard OpenTelemetry Collector receivers like file logs and host metrics receivers.

Additionally, users of the Elastic Distribution of the OpenTelemetry Collector benefit from container logs automatically enriched with Kubernetes metadata by leveraging the powerful container log parser that Elastic recently contributed. This OpenTelemetry-based enrichment enhances the context and value of the collected logs, providing deeper insights and more effective troubleshooting capabilities.

This new collector distribution ensures that exported data is fully compatible with the Elastic Platform, enhancing the overall observability experience. Elastic also ensures that Elastic-curated UIs can seamlessly handle both the Elastic Common Schema (ECS) and OpenTelemetry formats.

Elastic Distributions for Language SDKs

Elastic's APM agents have capabilities yet to be available in the OTel SDKs. EDOT brings these capabilities into the OTel language SDKs while maintaining seamless integration with Elastic Observability. Elastic will release OTel versions of all its APM agents, and continue to add additional language SDKs mirroring OTel.

Continued support for Native OTel components

EDOT does not preclude users from using native components. Users are still able to use:

• OpenTelemetry Vanilla Language SDKs: use standard OpenTelemetry code instrumentation for many popular programming languages sending OTLP traces to Elastic via APM server.

• Upstream Distribution of OpenTelemetry Collector (Contrib or Custom): Send traces using the OpenTelemetry Collector with OTLP receiver and OTLP exporter to Elastic via APM server.

Elastic is committed to contributing EDOT features or components upstream into the OpenTelemetry community, fostering a collaborative environment, and enhancing the overall OpenTelemetry ecosystem.

Extending our commitment to vendor-agnostic data collection

Elastic remains committed to supporting OpenTelemetry by being OTel first and building a vendor-agnostic framework. As OpenTelemetry constantly grows its support of SDKs and components,  Elastic will continue to refine and mirror EDOT to OpenTelemetry and push enhancements upstream. 

Over the past year, Elastic has been active in OTel through its donation of Elastic Common Schema (ECS), contributions to the native OpenTelemetry Collector and language SDKs, and a recent donation of its Universal Profiling agent to OpenTelemetry. 

EDOT  builds on our decision to fully adopt and recommend OpenTelemetry as the preferred solution for observing applications. With EDOT, Elastic customers can future-proof their investments and adopt OpenTelemetry, giving them vendor-neutral instrumentation with Elastic enterprise-grade support.

Our vision is that Elastic will work with the OpenTelemetry community to donate features through the standardization processes and contribute the code to implement those in the native OpenTelemetry components. In time, as OTel capabilities advance, and many of the Elastic-exclusive features transition into OpenTelemetry, we look forward to no longer having Elastic Distributions for OpenTelemetry.. In the meantime, we can deliver those capabilities via our OpenTelemetry distributions.

Introduction

Data pipelines, particularly ETL processes, form the backbone of modern data architectures. These pipelines are responsible for extracting raw data from various sources, transforming it into meaningful information, and loading it into data warehouses or data lakes for analysis and reporting.

In our organization, we have Python-based ETL scripts that play a pivotal role in exporting and processing data from Elasticsearch (ES) clusters and loading it into Google BigQuery (BQ). This processed data then feeds into DBT (Data Build Tool) models, which further refine the data and make it available for analytics and reporting. To see the full architecture and learn how we monitor our DBT pipelines with Elastic see Monitor your DBT pipelines with Elastic Observability. In this article we focus on the ETL scripts. Given the critical nature of these scripts, it is imperative to set up mechanisms to control and ensure the quality of the data they generate.

The strategies discussed here can be extended to any script or application that handles data processing or machine learning models, regardless of the programming language used as long as there exists a corresponding agent that supports OTEL instrumentation.

Motivation

Observability in data pipelines involves monitoring the entire lifecycle of data processing to ensure that everything works as expected. It includes:

1. Data Quality Control:

• Detecting anomalies in the data, such as unexpected drops in record counts.
• Verifying that data transformations are applied correctly and consistently.
• Ensuring the integrity and accuracy of the data loaded into the data warehouse.

1. Performance Monitoring:

• Tracking the execution time of ETL scripts to identify bottlenecks and optimize performance.
• Monitoring resource usage, such as memory and CPU consumption, to ensure efficient use of infrastructure.

1. Real-time Alerting:

• Setting up alerts for immediate notification of issues such as failed ETL jobs, data quality issues, or performance degradation.
• Identify the root case of such incidents
• Proactively addressing incidents to minimize downtime and impact on business operations

Issues such as failed ETL jobs, can even point to larger infrastructure or data source data quality issues.

Steps for Instrumentation

Here are the steps to automatically instrument your Python script for exporting OTEL traces, metrics, and logs.

Step 1: Import Required Libraries

We first need to install the following libraries.

pip install elastic-opentelemetry google-cloud-bigquery[opentelemetry]

You can also them to your project's requirements.txt file and install them with pip install -r requirements.txt.

Explanation of Dependencies

1. elastic-opentelemetry: This package is the Elastic Distribution for OpenTelemetry Python. Under the hood it will install the following packages:

   • opentelemetry-distro: This package is a convenience distribution of OpenTelemetry, which includes the OpenTelemetry SDK, APIs, and various instrumentation packages. It simplifies the setup and configuration of OpenTelemetry in your application.

   • opentelemetry-exporter-otlp: This package provides an exporter that sends telemetry data to the OpenTelemetry Collector or any other endpoint that supports the OpenTelemetry Protocol (OTLP). This includes traces, metrics, and logs.

   • opentelemetry-instrumentation-system-metrics: This package provides instrumentation for collecting system metrics, such as CPU usage, memory usage, and other system-level metrics.

2. google-cloud-bigquery[opentelemetry]: This package integrates Google Cloud BigQuery with OpenTelemetry, allowing you to trace and monitor BigQuery operations.

Step 2: Export OTEL Variables

Set the necessary OpenTelemetry (OTEL) variables by getting the configuration from APM OTEL from Elastic.

Go to APM -> Services -> Add data (top left corner).

In this section you will find the steps how to configure various APM agents. Navigate to OpenTelemetry to find the variables that you need to export.

Find OTLP Endpoint:

• Look for the section related to OpenTelemetry or OTLP configuration.

• The OTEL_EXPORTER_OTLP_ENDPOINT is typically provided as part of the setup instructions for integrating OpenTelemetry with Elastic APM. It might look something like https://<your-apm-server>/otlp.

  Obtain OTLP Headers:

• In the same section, you should find instructions or a field for OTLP headers. These headers are often used for authentication purposes.

• Copy the necessary headers provided by the interface. They might look like Authorization: Bearer <your-token>.

Note: Notice you need to replace the whitespace between Bearer and your token with %20 in the OTEL_EXPORTER_OTLP_HEADERS variable when using Python.

Alternatively you can use a different approach for authentication using API keys (see instructions). If you are using our serverless offering you will need to use this approach instead.

Set up the variables:

• Replace the placeholders in your script with the actual values obtained from the Elastic APM interface and execute it in your shell via the source command source env.sh.

Below is a script to set these variables:

#!/bin/bash
echo "--- :otel: Setting OTEL variables"
export OTEL_EXPORTER_OTLP_ENDPOINT='https://your-apm-server/otlp:443'
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=Bearer%20your-token'
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
export OTEL_PYTHON_LOG_CORRELATION=true
export ELASTIC_OTEL_SYSTEM_METRICS_ENABLED=true
export OTEL_METRIC_EXPORT_INTERVAL=5000
export OTEL_LOGS_EXPORTER="otlp,console"

With these variables set, we are ready for auto-instrumentation without needing to add anything to the code.

Explanation of Variables

• OTEL_EXPORTER_OTLP_ENDPOINT: This variable specifies the endpoint to which OTLP data (traces, metrics, logs) will be sent. Replace placeholder with your actual OTLP endpoint.

• OTEL_EXPORTER_OTLP_HEADERS: This variable specifies any headers required for authentication or other purposes when sending OTLP data. Replace placeholder with your actual OTLP headers.

• OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED: This variable enables auto-instrumentation for logging in Python, allowing logs to be automatically enriched with trace context.

• OTEL_PYTHON_LOG_CORRELATION: This variable enables log correlation, which includes trace context in log entries to correlate logs with traces.

• OTEL_METRIC_EXPORT_INTERVAL: This variable specifies the metric export interval in milliseconds, in this case 5s.

• OTEL_LOGS_EXPORTER: This variable specifies the exporter to use for logs. Setting it to "otlp" means that logs will be exported using the OTLP protocol. Adding "console" specifies that logs should be exported to both the OTLP endpoint and the console. In our case for better visibility on the infa side, we choose to export to console as well.

• ELASTIC_OTEL_SYSTEM_METRICS_ENABLED: It is needed to use this variable when using the Elastic distribution as by default it is set to false.

Note: OTEL_METRICS_EXPORTER and OTEL_TRACES_EXPORTER: This variables specify the exporter to use for metrics/traces, and are set to "otlp" by default, which means that metrics and traces will be exported using the OTLP protocol.

Running Python ETLs

We run Python ETLs with the following command:

OTEL_RESOURCE_ATTRIBUTES="service.name=x-ETL,service.version=1.0,deployment.environment=production" && opentelemetry-instrument python3 X_ETL.py 

Explanation of the Command

• OTEL_RESOURCE_ATTRIBUTES: This variable specifies additional resource attributes, such as service name, service version and deployment environment, that will be included in all telemetry data, you can customize these values per your needs. You can use a different service name for each script.

• opentelemetry-instrument: This command auto-instruments the specified Python script for OpenTelemetry. It sets up the necessary hooks to collect traces, metrics, and logs.

• python3 X_ETL.py: This runs the specified Python script (X_ETL.py).

Tracing

We export the traces via the default OTLP protocol.

Tracing is a key aspect of monitoring and understanding the performance of applications. Spans form the building blocks of tracing. They encapsulate detailed information about the execution of specific code paths. They record the start and end times of activities and can have hierarchical relationships with other spans, forming a parent/child structure.

Spans include essential attributes such as transaction IDs, parent IDs, start times, durations, names, types, subtypes, and actions. Additionally, spans may contain stack traces, which provide a detailed view of function calls, including attributes like function name, file path, and line number, which is especially useful for debugging. These attributes help us analyze the script's execution flow, identify performance issues, and enhance optimization efforts.

With the default instrumentation, the whole Python script would be a single span. In our case we have decided to manually add specific spans per the different phases of the Python process, to be able to measure their latency, throughput, error rate, etc individually. This is how we define spans manually:

from opentelemetry import trace

if __name__ == "__main__":

    tracer = trace.get_tracer("main")
    with tracer.start_as_current_span("initialization") as span:
            # Init code
            … 
    with tracer.start_as_current_span("search") as span:
            # Step 1 - Search code
            …
   with tracer.start_as_current_span("transform") as span:
           # Step 2 - Transform code
           …
   with tracer.start_as_current_span("load") as span:
           # Step 3 - Load code
           …

You can explore traces in the APM interface as shown below.

Metrics

We export metrics via the default OTLP protocol as well, such as CPU usage and memory. No extra code needs to be added in the script itself.

Note: Remember to set ELASTIC_OTEL_SYSTEM_METRICS_ENABLED to true.

Logging

We export logs via the default OTLP protocol as well.

For logging, we modify the logging calls to add extra fields using a dictionary structure (bq_fields) as shown below:

        job.result()  # Waits for table load to complete
        job_details = client.get_job(job.job_id)  # Get job details

        # Extract job information
        bq_fields = {
            # "slot_time_ms": job_details.slot_ms,
            "job_id": job_details.job_id,
            "job_type": job_details.job_type,
            "state": job_details.state,
            "path": job_details.path,
            "job_created": job_details.created.isoformat(),
            "job_ended": job_details.ended.isoformat(),
            "execution_time_ms": (
                job_details.ended - job_details.created
            ).total_seconds()
            * 1000,
            "bytes_processed": job_details.output_bytes,
            "rows_affected": job_details.output_rows,
            "destination_table": job_details.destination.table_id,
            "event": "BigQuery Load Job", # Custom event type
            "status": "success", # Status of the step (success/error)
            "category": category # ETL category tag 
        }

        logging.info("BigQuery load operation successful", extra=bq_fields)

This code shows how to extract BQ job stats, execution time, bytes processed, rows affected and destination table among them. You can add other metadata like we do such as custom event type, status, and category.

Any calls to logging (of all levels above the set threshold, in this case INFO logging.getLogger().setLevel(logging.INFO)) will create a log that will be exported to Elastic. This means that in Python scripts that already use logging there is no need to make any changes to export logs to Elastic.

For each of the log messages, you can go into the details view (click on the … when you hover over the log line and go into View details) to examine the metadata attached to the log message. You can also explore the logs in Discover.

Explanation of Logging Modification

• logging.info: This logs an informational message. The message "BigQuery load operation successful" is logged.

• extra=bq_fields: This adds additional context to the log entry using the bq_fields dictionary. This context can include details making the log entries more informative and easier to analyze. This data will be later used to set up alerts and data anomaly detection jobs.

Monitoring in Elastic's APM

As shown, we can examine traces, metrics, and logs in the APM interface. To make the most out of this data, we make use on top of nearly the whole suit of features in Elastic Observability alongside Elastic Analytic's ML capabilities.

Rules and Alerts

We can set up rules and alerts to detect anomalies, errors, and performance issues in our scripts.

The error count threshold rule is used to create a trigger when the number of errors in a service exceeds a defined threshold.

To create the rule go to Alerts and Insights -> Rules -> Create Rule -> Error count threshold, set the error count threshold, the service or environment you want to monitor (you can also set an error grouping key across services), how often to run the check, and choose a connector.

Next, we create a rule of type custom threshold on a given ETL logs data view (create one for your index) filtering on "labels.status: error" to get all the logs with status error from any of the steps of the ETL which have failed. The rule condition is set to document count > 0. In our case, in the last section of the rule config, we also set up Slack alerts every time the rule is activated. You can pick from a long list of connectors Elastic supports.

Then we can set up alerts for failures. We add status to the logs metadata as shown in the code sample below for each of the steps in the ETLs. It then becomes available in ES via labels.status.

logging.info(
            "Elasticsearch search operation successful",
            extra={
                "event": "Elasticsearch Search",
                "status": "success",
                "category": category,
                "index": index,
            },
        )

More Rules

We could also add rules to detect anomalies in the execution time of the different spans we define. This is done by selecting transaction/span -> Alerts and rules -> Custom threshold rule -> Latency. In the example below, we want to generate an alert whenever the search step takes more than 25s.

Alternatively, for finer-grained control, you can go with Alerts and rules -> Anomaly rule, set up an anomaly job, and pick a threshold severity level.

Anomaly detection job

In this example we set an anomaly detection job on the number of documents before transform.

We set up an Anomaly Detection jobs on the number of document before the transform using the Single metric job to detect any anomalies with the incoming data source.

In the last step, you can create alerting similarly to what we did before to receive alerts whenever there is an anomaly detected, by setting up a severity level threshold. Using the anomaly score which is assigned to every anomaly, every anomaly is characterized by a severity level.

Similarly to the previous example, we set up a Slack connector to receive alerts whenever an anomaly is detected.

You can go to your custom dashboard by going to Add Panel -> ML -> Anomaly Swim Lane -> Pick your job.

Similarly, we add jobs for the number of documents after the transform, and a Multi-Metric one on the execution_time_ms, bytes_processed and rows_affected similarly to how it was done in Monitor your DBT pipelines with Elastic Observability.

Custom Dashboard

Now that your logs, metrics, and traces are in Elastic, you can use the full potential of our Kibana dashboards to extract the most from them. We can create a custom dashboard like the following one: a pie chart based on labels.event (category field for every type of step in the ETLs), a chart for every type of step broken down by status, a timeline of steps broken down by status, BQ stats for the ETL, and anomaly detection swim lane panels for the various anomaly jobs.

Conclusion

Elastic’s APM, in combination with other Observability and ML features, provides a unified view of our data pipelines, allowing us to bring a lot of value with minimal code changes:

• Logging of new logs (no need to add custom logging) alongside their execution context
• Monitor the runtime behavior of our models
• Track data quality issues
• Identify and troubleshoot real-time incidents
• Optimize performance bottlenecks and resource usage
• Identify dependencies on other services and their latency
• Optimize data transformation processes
• Set up alerts on latency, data quality issues, error rates of transactions or CPU usage)

With these capabilities, we can ensure the resilience and reliability of our data pipelines, leading to more robust and accurate BI system and reporting.

In conclusion, setting up OpenTelemetry (OTEL) in Python for data pipeline observability has significantly improved our ability to monitor, detect, and resolve issues proactively. This has led to more reliable data transformations, better resource management, and enhanced overall performance of our data transformation, BI and Machine Learning systems.

There are several options to trace for LangChain. One is Langsmith, ideal for detailed tracing and a complete breakdown of requests to large language models (LLMs). However, it is specific to Langchain. OpenTelemetry (OTel) is now broadly accepted as the industry standard for tracing. As one of the major Cloud Native Computing Foundation (CNCF) projects, with as many commits as Kubernetes, it is gaining support from major ISVs and cloud providers delivering support for the framework. 

Hence, many LangChain-based applications will have multiple components beyond just LLM interactions. Using OpenTelemetry with LangChain is essential. OpenLLMetry is an available option for tracing Langchain apps in addition to Langsmith.

This blog will show how you can get LangChain tracing into Elastic using the OpenLLMetry library opentelemetry-instrumentation-langchain.

Pre-requisites:

• An Elastic Cloud account — sign up now, and become familiar with Elastic’s OpenTelemetry configuration

• Have a LangChain app to instrument

• Be familiar with using OpenTelemetry’s Python SDK 

• An account on your favorite LLM, with API keys

Overview

In highlighting tracing I created a simple LangChain app that does the following:

1. Takes customer input on the command line. (Queries)

2. Sends these to the Azure OpenAI LLM via a LangChain.

3. Chain tools are set to use the search with Tavily 

4. The LLM uses the output which returns the relevant information to the user.

As you can see Elastic Observability’s APM recognizes the LangChain App, and also shows the full trace (done with manual instrumentation):

As the above image shows:

1. The user makes a query
2. Azure OpenAI is called, but it uses a tool (Tavily) to obtain some results
3. Azure OpenAI reviews and returns a summary to the end user

The code was manually instrumented, but auto-instrument can also be used.

OpenTelemetry Configuration

In using OpenTelemetry, we need to configure the SDK to generate traces and configure Elastic’s endpoint and authorization. Instructions can be found in OpenTelemetry Auto-Instrumentation setup documentation.

OpenTelemetry Environment variables:

OpenTelemetry Environment variables for Elastic can be set as follows in linux (or in the code).

OTEL_EXPORTER_OTLP_ENDPOINT=12345.apm.us-west-2.aws.cloud.es.io:443
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20ZZZZZZZ"
OTEL_RESOURCE_ATTRIBUTES="service.name=langchainChat,service.version=1.0,deployment.environment=production"

As you can see OTEL_EXPORTER_OTLP_ENDPOINT is set to Elastic, and the corresponding authorization header is also provided. These can be easily obtained from Elastic’s APM configuration screen under OpenTelemetry

Note: No agent is needed, we simply send the OTLP trace messages directly to Elastic’s APM server. 

OpenLLMetry Library:

OpenTelemetry's auto-instrumentation can be extended to trace other frameworks via instrumentation packages.

First, you must install the following package: 

pip install opentelemetry-instrumentation-langchain

This library was developed by OpenLLMetry. 

Then you will need to add the following to the code.

from opentelemetry.instrumentation.langchain import LangchainInstrumentor
LangchainInstrumentor().instrument()

Instrumentation

Once the libraries are added, and the environment variables are set, you can use auto-instrumentation With auto-instrumentation, the following:

opentelemetry-instrument python tavilyAzureApp.py

The OpenLLMetry library does pull out the flow correctly with minimal manual instrumentation except for adding the OpenLLMetry library.

1. Takes customer input on the command line. (Queries)

2. Sends these to the Azure OpenAI LLM via a Lang chain.

3. Chain tools are set to use the search with Tavily 

4. The LLM uses the output which returns the relevant information to the user.

Manual-instrumentation

If you want to get more details out of the application, you will need to manually instrument. To get more traces follow my Python instrumentation guide. This guide will walk you through setting up the necessary OpenTelemetry bits, Additionally, you can also look at the documentation in OTel for instrumenting in Python.

Note that the env variables OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_ENDPOINT are set as noted in the section above. You can also set up the OTEL_RESOURCE_ATTRIBUTES.

Once you follow the steps in either guide and initiate the tracer, you will have to essentially just add the span where you want to get more details. In the example below, only one line of code is added for span initialization.

Look at the placement of with tracer.start_as_current_span("getting user query") as span: below

# Creates a tracer from the global tracer provider
tracer = trace.get_tracer("newsQuery")

async def chat_interface():
    print("Welcome to the AI Chat Interface!")
    print("Type 'quit' to exit the chat.")

    with tracer.start_as_current_span("getting user query") as span:
        while True:
            user_input = input("\nYou: ").strip()

            if user_input.lower() == 'quit':
                print("Thank you for chatting. Goodbye!")
                break

            print("AI: Thinking...")
            try:
                result = await chain.ainvoke({"query": user_input})
                print(f"AI: {result.content}")
            except Exception as e:
                print(f"An error occurred: {str(e)}")


if __name__ == "__main__":
    asyncio.run(chat_interface())

As you can see, with manual instrumentation, we get the following trace:

Which calls out when we enter our query function. async def chat_interface()

Conclusion

In this blog, we discussed the following:

• How to manually instrument LangChain with OpenTelemetry

• How to properly initialize OpenTelemetry and add a custom span

• How to easily set the OTLP ENDPOINT and OTLP HEADERS with Elastic without the need for a collector

• See traces in Elastic Observability APM

Hopefully, this provides an easy-to-understand walk-through of instrumenting LangChain with OpenTelemetry and how easy it is to send traces into Elastic.

Additional resources for OpenTelemetry with Elastic:

• Independence with OpenTelemetry on Elastic

• Modern observability and security on Kubernetes with Elastic and OpenTelemetry

• 3 models for logging with OpenTelemetry and Elastic

• Adding free and open Elastic APM as part of your Elastic Observability deployment

• Monitor OpenAI API and GPT models with OpenTelemetry and Elastic

• Futureproof your observability platform with OpenTelemetry and Elastic

• Instrumentation resources:

• Python: Auto-instrumentation, Manual instrumentation

• Java: Auto-instrumentation, Manual instrumentation 

• Node.js: Auto-instrumentation, Manual instrumentation

• .NET: Auto-instrumentation, Manual instrumentation

Also log into cloud.elastic.co to try out Elastic with a free trial.

In this second part, I show how to install any Java agent via the OpenTelemetry operator, using the Elastic Java agents as examples.

Installation and configuration recap

Part 1 of this series, Zero config OpenTelemetry auto-instrumentation for Kubernetes Java applications, details the installation and configuration of the OpenTelemetry operator and an Instrumentation resource. Here is an outline of the steps as a reminder:

1. Install cert-manager, eg kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml
2. Install the operator, eg kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
3. Create an Instrumentation resource
4. Add an annotation to either the deployment or the namespace
5. Deploy the application as normal

In that first part, steps 3, 4 & 5 were implemented for the OpenTelemetry Java agent. In this blog I’ll implement them for other agents, using the Elastic APM agents as examples. I assume that steps 1 & 2 outlined above have already been done, ie that the operator is now installed. I will continue using the banana namespace for the examples, so ensure that namespace exists (kubectl create namespace banana). As per part 1, if you use any of the example instrumentation definitions below, you’ll need to substitute my.apm.server.url and my-apm-secret-token with the values appropriate for your collector.

Using the Elastic Distribution for OpenTelemetry Java

From version 0.4.0, the Elastic Distribution for OpenTelemetry Java includes the agent jar at the path /javaagent.jar in the docker image - which is essentially all that is needed for a docker image to be usable by the OpenTelemetry operator for auto-instrumentation. This means the Instrumentation resource is straightforward to define, and as it’s a distribution of the OpenTelemetry Java agent, all the OpenTelemetry environment can apply:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: elastic-otel
  namespace: banana
spec:
  exporter:
    endpoint: https://my.apm.server.url
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  java:
    image: docker.elastic.co/observability/elastic-otel-javaagent:1.10.0
    env:
      - name: OTEL_EXPORTER_OTLP_HEADERS
        value: "Authorization=Bearer my-apm-secret-token"
      - name: ELASTIC_OTEL_INFERRED_SPANS_ENABLED
        value: "true"
      - name: ELASTIC_OTEL_SPAN_STACK_TRACE_MIN_DURATION
        value: "50"

I’ve included environment for switching on several features in the agent, including

1. ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED to switch on the inferred spans implementation feature described in this blog
2. Span stack traces are automatically captured if the span takes more than ELASTIC_OTEL_SPAN_STACK_TRACE_MIN_DURATION (default would be 5ms)

Adding in the annotation …

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "elastic-otel"

… to the pod yaml gets the application traced, and displayed in the Elastic APM UI, including the inferred child spans and stack traces

The additions from the features mentioned above are circled in red - inferred spans (for methodC and methodD) bottom left, and the stack trace top right. (Note that the pod included the OTEL_INSTRUMENTATION_METHODS_INCLUDE environment variable set to "test.Testing[methodB]" so that traces from methodB are shown; for pod configuration see the "Trying it" section in part 1)

Using the Elastic APM Java agent

From version 1.50.0, the Elastic APM Java agent includes the agent jar at the path /javaagent.jar in the docker image - which is essentially all that is needed for a docker image to be usable by the OpenTelemetry operator for auto-instrumentation. This means the Instrumentation resource is straightforward to define:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: elastic-apm
  namespace: banana
spec:
  java:
    image: docker.elastic.co/observability/apm-agent-java:1.55.4
    env:
      - name: ELASTIC_APM_SERVER_URL
        value: "https://my.apm.server.url"
      - name: ELASTIC_APM_SECRET_TOKEN
        value: "my-apm-secret-token"
      - name: ELASTIC_APM_LOG_LEVEL
        value: "INFO"
      - name: ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED
        value: "true"
      - name: ELASTIC_APM_LOG_SENDING
        value: "true"

I’ve included environment for switching on several features in the agent, including

• ELASTIC_APM_LOG_LEVEL set to the default value (INFO) which could easily be switched to DEBUG
• ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED to switch on the inferred spans implementation equivalent to the feature described in this blog
• ELASTIC_APM_LOG_SENDING which switches on sending logs to the APM UI, the logs are automatically correlated with transactions (for all common logging frameworks)

Adding in the annotation …

metadata:
  annotations:
     instrumentation.opentelemetry.io/inject-java: "elastic-apm"

… to the pod yaml gets the application traced, and displayed in the Elastic APM UI, including the inferred child spans

(Note that the pod included the ELASTIC_APM_TRACE_METHODS environment variable set to "test.Testing#methodB" so that traces from methodB are shown; for pod configuration see the "Trying it" section in part 1)

Using an extension with the OpenTelemetry Java agent

Setting up an Instrumentation resource for the OpenTelemetry Java agent is straightforward and was done in part 1 of this two part series - and you can see from the above examples it’s just a matter of deciding on the docker image URL you want to use. However if you want to include an extension in your deployment, this is a little more complex, but also supported by the operator. Basically the extensions you want to include with the agent need to be in docker images - or you have to build an image which includes the extensions that are not already in images. Then you declare the images and the directories the extensions are in, in the Instrumentation resource. As an example, I’ll show an Instrumentation which uses version 2.5.0 of the OpenTelemetry Java agent together with the inferred spans extension from the Elastic OpenTelemetry Java distribution. The distro image includes the extension at path /extensions/elastic-otel-agentextension.jar. The Instrumentation resource allows either directories or file paths to be specified, here I’ll list the directory:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: otel-plus-extension-instrumentation
  namespace: banana
spec:
  exporter:
    endpoint: https://my.apm.server.url
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.5.0
    extensions:
      - image: "docker.elastic.co/observability/elastic-otel-javaagent:1.10.0"
        dir: "/extensions"
    env:
      - name: OTEL_EXPORTER_OTLP_HEADERS
        value: "Authorization=Bearer my-apm-secret-token"
      - name: ELASTIC_OTEL_INFERRED_SPANS_ENABLED
        value: "true"

Note that you can have multiple image … dir pairs, ie include multiple extensions from different images. Note also if you are testing this specific configuration that the inferred spans extension included here will be contributed to the OpenTelemetry contrib repo at some point after this blog is published, after which the extension may no longer be present in a later version of the referred image (since it will be available from the contrib repo instead).

Next steps

Here I’ve shown how to use any agent with the OpenTelemetry Operator for Kubernetes, and configure that for your system. In particular the examples have showcased how to use the Elastic Java agents to auto-instrument Java applications running in your Kubernetes clusters, along with how to enable features, using Instrumentation resources. And you can set it up for either zero config for deployments, or for just one annotation which is generally a more flexible mechanism (you can have multiple Instrumentation resource definitions, and the deployment can select the appropriate one for its application).

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The mechanism to achieve zero-config auto-instrumentation of Java applications in Kubernetes is via the OpenTelemetry Operator for Kubernetes. This operator has many capabilities and the full documentation (and of course source) is available in the project itself. In this blog, I'll walk through installing, setting up and running zero-config auto-instrumentation of Java applications in Kubernetes using the OpenTelemetry Operator.

Installing the OpenTelemetry Operator

At the time of writing this blog, the OpenTelemetry Operator needs the certification manager to be installed, after which the operator can be installed. Installing from the web is straightforward. First install the cert-manager (the version to be installed will be specified in the OpenTelemetry Operator for Kubernetes documentation):

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml

Then when the cert managers are ready (kubectl get pods -n cert-manager)  …

NAMESPACE      NAME                                         READY
cert-manager   cert-manager-67c98b89c8-rnr5s                1/1
cert-manager   cert-manager-cainjector-5c5695d979-q9hxz     1/1
cert-manager   cert-manager-webhook-7f9f8648b9-8gxgs        1/1

… you can install the OpenTelemetry Operator:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

You can, of course, use a specific version of the operator instead of the latest. But here I’ve used the latest version.

An Instrumentation resource

Now you need to add just one further Kubernetes resource to enable auto-instrumentation: an Instrumentation resource. I am going to use the banana namespace for my examples, so I have first created that namespace (kubectl create namespace banana). The auto-instrumentation is specified and configured by these Instrumentation resources. Here is a basic one which will allow every Java pod in the banana namespace to be auto-instrumented with version 2.5.0 of the OpenTelemetry Java agent:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: banana-instr
  namespace: banana
spec:
  exporter:
    endpoint: "https://my.endpoint"
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.5.0
    env:
      - name: OTEL_EXPORTER_OTLP_HEADERS
        value: "Authorization=Bearer MyAuth"

Creating this resource (eg with kubectl apply -f banana-instr.yaml, assuming the above yaml was saved in file banana-instr.yaml) makes the banana-instr Instrumentation resource available for use. (Note you will need to change my.endpoint and MyAuth to values appropriate for your collector.) You can use this instrumentation immediately by adding an annotation to any deployment in the banana namespace:

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"

The banana-instr Instrumentation resource is not yet set to be applied by default to all pods in the banana namespace. Currently it's zero-config as far as the application is concerned, but it requires an annotation added to a pod or deployment. To make it fully zero-config for all pods in the banana namespace, we need to add that annotation to the namespace itself, ie editing the namespace (kubectl edit namespace banana) so it would then have contents similar to

apiVersion: v1
kind: Namespace
metadata:
  name: banana
  annotations:
    instrumentation.opentelemetry.io/inject-java: "banana-instr"
...

Now we have a namespace that is going to auto-instrument every Java application deployed in the banana namespace with the 2.5.0 OpenTelemetry Java agent!

Trying it

There is a simple example Java application at docker.elastic.co/demos/apm/k8s-webhook-test which just repeatedly calls the chain main->methodA->methodB->methodC->methodD with some sleeps in the calls. Running this (kubectl apply -f banana-app.yaml) using a very basic pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: banana-app
  namespace: banana
  labels:
    app: banana-app
spec:
  containers:
    - image: docker.elastic.co/demos/apm/k8s-webhook-test
      imagePullPolicy: Always
      name: banana-app
      env: 
      - name: OTEL_INSTRUMENTATION_METHODS_INCLUDE
        value: "test.Testing[methodB]"

results in the app being auto-instrumented with no configuration changes! The resulting app shows up in any APM UI, such as Elastic APM

As you can see, for this example I also added this env var to the pod yaml, OTEL_INSTRUMENTATION_METHODS_INCLUDE="test.Testing[methodB]" so that there were traces showing from methodB.

The technology behind the auto-instrumentation

To use the auto-instrumentation there is no specific need to understand the underlying mechanisms, but for those of you interested, here’s a quick outline.

1. The OpenTelemetry Operator for Kubernetes installs a mutating webhook, a standard Kubernetes component.
2. When deploying, Kubernetes first sends all definitions to the mutating webhook.
3. If the mutating webhook sees that the conditions for auto-instrumentation should be applied (ie
4. there is an Instrumentation resource for that namespace and
5. the correct annotation for that Instrumentation is applied to the definition in some way, either from the definition itself or from the namespace),
6. then the mutating webhook “mutates” the definition to include the environment defined by the Instrumentation resource.
7. The environment includes the explicit values defined in the env, as well as some implicit OpenTelemetry values (see the OpenTelemetry Operator for Kubernetes documentation for full details).
8. And most importantly, the operator
9. pulls the image defined in the Instrumentation resource,
10. extracts the file at the path /javaagent.jar from that image (using shell command cp)
11. inserts it into the pod at path /otel-auto-instrumentation-java/javaagent.jar
12. and adds the environment variable JAVA_TOOL_OPTIONS=-javaagent:/otel-auto-instrumentation-java/javaagent.jar.
13. The JVM automatically picks up that JAVA_TOOL_OPTIONS environment variable on startup and applies it to the JVM command-line.

Next steps

This walkthrough can be repeated in any Kubernetes cluster to demonstrate and experiment with auto-instrumentation (you will need to create the banana namespace first). In part 2 of this two part series, Using a custom agent with the OpenTelemetry Operator for Kubernetes, I show how to install any Java agent via the OpenTelemetry operator, using the Elastic Java agents as examples.

Background

Elastic is standardizing on OpenTelemetry (OTel) for observability and security data collection. As part of that effort, we are providing distributions of the OpenTelemetry Language SDKs. We have recently released alpha distributions for Java, .NET and Node.js. Our Android and iOS SDKs have been OpenTelemetry-based from the start. The Elastic Distribution for OpenTelemetry Python is the latest addition.

Design choices

We have chosen to provide a lean distribution that does not install all the instrumentations by default but that instead provides tools to do so. We leverage the opentelemetry-bootstrap tool provided by OpenTelemetry Python project to scan the packages installed in your environment and recognizes libraries we are able to instrument. This tool can just report the instrumentations available and optionally is able to install them as well. This allows you to avoid installing packages you are not going to need or instrument libraries you are not interested in tracing.

Getting started

To get started with Elastic Distribution for OpenTelemetry Python you need to install the package elastic-opentelemetry in your project environment. We'll use pip in our examples but you are free to use any python package and environment manager of your choice.

pip install elastic-opentelemetry

Once you have installed our distro you'll have also the opentelemetry-bootstrap command available. Running it:

opentelemetry-bootstrap

will list all available packages for your instrumentation, e.g. you can expect something like the following:

opentelemetry-instrumentation-asyncio==0.46b0
opentelemetry-instrumentation-dbapi==0.46b0
opentelemetry-instrumentation-logging==0.46b0
opentelemetry-instrumentation-sqlite3==0.46b0
opentelemetry-instrumentation-threading==0.46b0
opentelemetry-instrumentation-urllib==0.46b0
opentelemetry-instrumentation-wsgi==0.46b0
opentelemetry-instrumentation-grpc==0.46b0
opentelemetry-instrumentation-requests==0.46b0
opentelemetry-instrumentation-system-metrics==0.46b0
opentelemetry-instrumentation-urllib3==0.46b0

It also provides a command option to install the packages automatically

opentelemetry-bootstrap --action=install

It is advised to run this command every time you release a new version of your application so that you can install or just revise any instrumentation packages for your code.

Some environment variables are needed to provide the needed configuration for instrumenting your services. These mostly concern the destination of your traces but also for easily identifying your service. A service name is required to have your service distinguishable from the others. Then you need to provide the authorization headers for authentication with Elastic Observability cloud and the Elastic cloud endpoint where the data is sent.

The API Key you get from your Elastic cloud serverless project must be URL-encoded, you can do that with the following Python snippet:

from urllib.parse import quote
quote("ApiKey <your api key>)

Once you have all your configuration values you can export via environment variables as below:

export OTEL_RESOURCE_ATTRIBUTES=service.name=<service-name>
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=<url encoded apikey header value>"
export OTEL_EXPORTER_OTLP_ENDPOINT=<your elastic cloud url>

We are done with the configuration and the last piece of the puzzle is wrapping your service invocation with opentelemetry-instrument, the wrapper that provides zero-code instrumentation. Zero-code (or Automatic) instrumentation means that the distribution will set up the OpenTelemetry SDK and enable all the previously installed instrumentations for you. Unfortunately Zero-code instrumentation does not cover all libraries and some — web frameworks in particular — will require minimal manual configuration.

For a web service running with gunicorn it may look like:

opentelemetry-instrument gunicorn main:app

The result is an observable application using the industry-standard OpenTelemetry — offering high-quality instrumentation of many popular Python libraries, a portable API to avoid vendor lock-in and an active community.

Using Elastic Observability, some out-of-the-box benefits you can expect are: rich trace viewing, Service maps, integrated metrics and log analysis, and more.

What's next?

Elastic is committed to helping OpenTelemetry succeed and to helping our customers use OpenTelemetry effectively in their systems. Last year, we donated ECS and continue to work on integrating it with OpenTelemetry Semantic Conventions. More recently, we are working on donating our eBPF-based profiler to OpenTelemetry. We contribute to many of the language SDKs and other OpenTelemetry projects.

In the Python ecosystem we are active reviewers and contributors of both the opentelemetry-python and opentelemetry-python-contrib repositories.

The Elastic Distribution for OpenTelemetry Python is currently an alpha. Please try it out and let us know if it might work for you. Watch for the latest releases here. You can engage with us on the project issue tracker.

We are eager to know your use cases to help you succeed in your Observability journey.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Resources

• https://www.elastic.co/blog/elastic-opentelemetry-sdk-distributions
• https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-java-agent
• https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-dotnet-applications
• https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-node-js
• https://www.elastic.co/observability-labs/blog/manual-instrumentation-python-apps-opentelemetry
• https://www.elastic.co/observability-labs/blog/auto-instrumentation-python-applications-opentelemetry
• https://www.elastic.co/observability-labs/blog/opentelemetry-observability

Demo Application

For this blog post, we’ll be using the cpu-burner demo application to showcase the correlation capabilities of APM, tracing, and profiling in Elastic. This application was built to continuously execute several CPU-intensive tasks:

• It computes Fibonacci numbers using the naive, recursive algorithm.
• It hashes random data with the SHA-2 and SHA-3 hashing algorithms.
• It performs numerous large background allocations to stress the garbage collector.

The computations of the Fibonacci numbers and the hashing will each be visible as transactions in Elastic: They have been manually instrumented using the OpenTelemetry API.

Setting up Profiling and APM

First, we’ll need to set up the universal profiling host agent on the host where the demo application will run. Starting from version 8.14.0, correlation with APM data is supported and enabled out of the box for the profiler. There is no special configuration needed; we can just follow the standard setup guide. Note that at the time of writing, universal profiling only supports Linux. On Windows, you'll have to use a VM to try the demo. On macOS, you can use colima as docker engine and run the profiling host agent and the demo app in container images.

In addition, we’ll need to instrument our demo application with an APM agent. We can either use the classic Elastic APM agent or the Elastic OpenTelemetry Distribution.

Using the Classic Elastic APM Agent

Starting with version 1.50.0, the classic Elastic APM agent ships with the capability to correlate the traces it captures with the profiling data from universal profiling. We’ll just need to enable it explicitly via the universal_profiling_integration_enabled config option. Here is the standard command line for running the demo application with the setting enabled:

curl -o 'elastic-apm-agent.jar' -L 'https://oss.sonatype.org/service/local/artifact/maven/redirect?r=releases&g=co.elastic.apm&a=elastic-apm-agent&v=LATEST'
java -javaagent:elastic-apm-agent.jar \
-Delastic.apm.service_name=cpu-burner-elastic \
-Delastic.apm.secret_token=XXXXX \
-Delastic.apm.server_url=<elastic-apm-server-endpoint> \
-Delastic.apm.application_packages=co.elastic.demo \
-Delastic.apm.universal_profiling_integration_enabled=true \
-jar ./target/cpu-burner.jar

Using OpenTelemetry

The feature is also available as an OpenTelemetry SDK extension. This means you can use it as a plugin for the vanilla OpenTelemetry agent or add it to your OpenTelemetry SDK if you are not using an agent. In addition, the feature ships by default with the Elastic OpenTelemetry Distribution for Java and can be used via any of the possible usage methods. While the extension is currently Elastic-specific, we are already working with the various OpenTelemetry SIGs on standardizing the correlation mechanism, especially now after the eBPF profiling agent has been contributed.

For this demo, we’ll be using the Elastic OpenTelemetry Distro Java agent to run the extension:

curl -o 'elastic-otel-javaagent.jar' -L 'https://oss.sonatype.org/service/local/artifact/maven/redirect?r=releases&g=co.elastic.otel&a=elastic-otel-javaagent&v=LATEST'
java -javaagent:./elastic-otel-javaagent.jar \
-Dotel.exporter.otlp.endpoint=<elastic-cloud-OTLP-endpoint> \
"-Dotel.exporter.otlp.headers=Authorization=Bearer XXXX" \
-Dotel.service.name=cpu-burner-otel \
-Delastic.otel.universal.profiling.integration.enabled=true \
-jar ./target/cpu-burner.jar

Here, we explicitly enabled the profiling integration feature via the elastic.otel.universal.profiling.integration.enabled property. Note that with an upcoming release of the universal profiling feature, this won’t be necessary anymore! The OpenTelemetry extension will then automatically detect the presence of the profiler and enable the correlation feature based on that.

The demo repository also comes with a Dockerfile, so you can alternatively build and run the app in docker:

docker build -t cpu-burner .
docker run --rm -e OTEL_EXPORTER_OTLP_ENDPOINT=<elastic-cloud-OTLP-endpoint> -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer XXXX" cpu-burner

And that’s it for setup; we are now ready to inspect the correlated profiling data!

Analyzing Service CPU Usage

The first thing we can do now is head to the “Flamegraph” view in Universal Profiling and inspect flamegraphs filtered on APM services. Without the APM correlation, universal profiling is limited to filtering on infrastructure concepts, such as hosts, containers, and processes. Below is a screencast showing a flamegraph filtered on the service name of our demo application:

With this filter applied, we get a flamegraph aggregated over all instances of our service. If that is not desired, we could narrow down the filter, e.g. based on the host or container names. Note that the same service-level flamegraph view is also available on the “Universal Profiling” tab in the APM service UI.

The flamegraphs show exactly how the demo application is spending its CPU time, independently of whether it is covered by instrumentation or not. From left to right, we can first see the time spent in application tasks: We can identify the background allocations not covered by APM transactions as well as the SHA-computation and Fibonacci transactions. Interestingly, this application logic only covers roughly 60% of the total CPU time! The remaining time is spent mostly in the G1 garbage collector due to the high allocation rate of our application. The flamegraph shows all G1-related activities and the timing of the individual phases of concurrent tasks. We can easily identify those based on the native function names. This is made possible by universal profiling being capable of profiling and symbolizing the JVM’s C++ code in addition to the Java code.

Pinpointing Transaction Bottlenecks

While the service-level flamegraph already gives good insights on where our transactions consume the most CPU, this is mainly due to the simplicity of the demo application. In real-world applications, it can be much harder to pinpoint that certain stack frames come mostly from certain transactions. For this reason, the APM agent also correlates CPU profiling data from universal profiling on the transaction level.

We can navigate to the “Universal Profiling” tab on the transaction details page to get per-transaction flamegraphs:

For example, let’s have a look at the flamegraph of our transaction computing SHA-2 and SHA-3 hashes of randomly generated data:

Interestingly, the flamegraph uncovers some unexpected results: The transactions spend more time computing the random bytes to be hashed rather than on the hashing itself! So if this were a real-world application, a possible optimization could be to use a more performant random number generator.

In addition, we can see that the MessageDigest.update call for computing the hash values fans out into two different code paths: One is a call into the BouncyCastle cryptography library, the other one is a JVM stub routine, meaning that the JIT compiler has inserted special assembly code for a function.

The flamegraph shown in the screenshot displays the aggregated data for all “shaShenanigans” transactions in the given time filter. We can further filter this down using the transaction filter bar at the top. To make the best use of this, the demo application annotates the transactions with the hashing algorithm used via OpenTelemetry attributes:

public static void shaShenanigans(MessageDigest digest) {
    Span span = tracer.spanBuilder("shaShenanigans")
        .setAttribute("algorithm", digest.getAlgorithm())
        .startSpan();
    ...
    span.end()
}

So, let’s filter our flamegraph based on the used hashing algorithm:

Note that “SHA-256” is the name of the JVM built-in SHA-2 256-bit implementation. This now gives the following flamegraph:

We can see that the BouncyCastle stack frames are gone and MessageDigest.update spends all its time in the JVM stub routines. Therefore, the stub routine is likely hand-crafted assembly from the JVM maintainers for the SHA2 algorithm.

If we instead filter on “SHA3-256”, we get the following result:

Now, as expected, MessageDigest.update spends all its time in the BouncyCastle library for the SHA3 implementation. Note that the hashing here takes up more time in relation to the random data generation, showing that the SHA2 JVM stub routine is significantly faster than the BouncyCastle Java SHA3 implementation.

This filtering is not limited to custom attributes like those shown in this demo. You can filter on any transaction attributes, including latency, HTTP headers, and so on. For example, for typical HTTP applications, it allows analyzing the efficiency of the used JSON serializer based on the payload size. Note that while it is possible to filter on single transaction instances (e.g. based on trace.id), this is not recommended: To allow continuous profiling in production systems, the profiler by default runs with a low sampling rate of 20hz. This means that for typical real-world applications, this will not yield enough data when looking at a single transaction execution. Instead, we gain insights by monitoring multiple executions of a group of transactions over time and aggregating their samples, for example in a flamegraph.

Summary

A common reason for applications to degrade is overly high CPU usage. In this blog post, we showed how to combine universal profiling with APM to find the actual root cause in such cases: We explained how to analyze the CPU time using profiling flamegraphs on service and transaction levels. In addition, we further drilled down into data using custom filters. We used a simple demo application for this purpose, so go ahead and try it yourself with your own, real-world applications to uncover the actual power of the feature!

With today’s acceptance, we are officially contributing our continuous profiler technology to OpenTelemetry. We will also dedicate a team of profiling domain experts to co-maintain and advance the profiling capabilities within OTel.

We want to thank the OpenTelemetry community for the great and constructive cooperation on the donation proposal. We look forward to jointly establishing continuous profiling as an integral part of OpenTelemetry.

What is continuous profiling?

Profiling is a technique used to understand the behavior of a software application by collecting information about its execution. This includes tracking the duration of function calls, memory usage, CPU usage, and other system resources.

However, traditional profiling solutions have significant drawbacks limiting adoption in production environments:

• Significant cost and performance overhead due to code instrumentation
• Disruptive service restarts
• Inability to get visibility into third-party libraries

Unlike traditional profiling, which is often done only in a specific development phase or under controlled test conditions, continuous profiling runs in the background with minimal overhead. This provides real-time, actionable insights without replicating issues in separate environments. SREs, DevOps, and developers can see how code affects performance and cost, making code and infrastructure improvements easier.

Contribution of production-grade features

Elastic Universal Profiling is a whole-system, always-on, continuous profiling solution that eliminates the need for code instrumentation, recompilation, on-host debug symbols or service restarts. Leveraging eBPF, Elastic Universal Profiling profiles every line of code running on a machine, including application code, kernel, and third-party libraries. The solution measures code efficiency in three dimensions, CPU utilization, CO2, and cloud cost, to help organizations manage efficient services by minimizing computational waste.

The Elastic profiling agent facilitates identifying non-optimal code paths, uncovering "unknown unknowns", and provides comprehensive visibility into the runtime behavior of all applications. Elastic’s continuous profiling agent supports various runtimes and languages, such as C/C++, Rust, Zig, Go, Java, Python, Ruby, PHP, Node.js, V8, Perl, and .NET.

Additionally, organizations can meet sustainability objectives by minimizing computational wastage, ensuring seamless alignment with their strategic ESG goals.

Benefits to OpenTelemetry

This contribution not only boosts the standardization of continuous profiling for observability but also accelerates the practical adoption of profiling as the fourth key signal in OTel. Customers get a vendor-agnostic way of collecting profiling data and enabling correlation with existing signals, like tracing, metrics, and logs, opening new potential for observability insights and a more efficient troubleshooting experience. 

OTel-based continuous profiling unlocks the following possibilities for users:

• Improved customer experience: delivering consistent service quality and performance through continuous profiling ensures customers have an application that performs optimally, remains responsive, and is reliable.

• Maximize gross margins: Businesses can optimize their cloud spend and improve profitability by reducing the computational resources needed to run applications. Whole system continuous profiling identifies the most expensive functions (down to the lines of code) across diverse environments that may span multiple cloud providers. In the cloud context, every CPU cycle saved translates to money saved. 

• Minimize environmental impact: energy consumption associated with computing is a growing concern (source: MIT Energy Initiative ). More efficient code translates to lower energy consumption, reducing carbon (CO2) footprint. 

• Accelerate engineering workflows: continuous profiling provides detailed insights to help troubleshoot complex issues faster, guide development, and improve overall code quality.

• Improved vendor neutrality and increased efficiency: an OTel eBPF-based profiling agent removes the need to use proprietary APM agents and offers a more efficient way to collect profiling telemetry.

With these benefits, customers can now manage the overall application’s efficiency on the cloud while ensuring their engineering teams optimize it.

What comes next?

While the acceptance of Elastic’s donation of the profiling agent marks a significant milestone in the evolution of OTel’s eBPF-based continuous profiling capabilities, it represents the beginning of a broader journey. Moving forward, we will continue collaborating closely with the OTel Profiling and Collector SIGs to ensure seamless integration of the profiling agent within the broader OTel ecosystem. During this phase, users can test early preview versions of the OTel profiling integration by following the directions in the otel-profiling-agent repository.

Elastic remains deeply committed to OTel’s vision of enabling cross-signal correlation. We plan to further contribute to the community by sharing our innovative research and implementations, specifically those facilitating the correlation between profiling data and distributed traces, across several OTel language SDKs and the profiling agent.

We are excited about our growing relationship with OTel and the opportunity to donate our profiling agent in a way that benefits both the Elastic community and the broader OTel community. Learn more about Elastic’s OpenTelemetry support and learn how to contribute to the ongoing profiling work in the community.

Additional Resources

Additional details on Elastic’s Universal Profiling can be found in the FAQ.

For insights into observability, visit Observability labs where OTel specific articles are also available.

Elastic’s donation proposal for contributing its Universal Profiling™ agent has now been accepted by the OpenTelemetry community. Elastic’s Universal Profiling agent, the industry’s most comprehensive fleet-wide Universal Profiling solution, empowers users to quickly identify performance bottlenecks, reduce cloud spend, and minimize their carbon footprint. With the contribution of the Elastic Universal Profiling Agent to OpenTelemetry, all customers will benefit from its features and capabilities.

What do Elastic users need to know?

Elastic’s contribution of the continuous profiling agent will not change the existing set of Elastic’s continuous profiling features or how we ingest and store profiling data. 

Elastic will participate and closely collaborate with the OTel community to manage not only the addition of the continuous profiling agent to OTel but also work with and drive the OTel community’s Profiling Special Interest Group (SIG) in shaping OTel’s continuous profiling evolution. 

Elastic has facilitated the definition of the OTel Profiling Data Model, a crucial step toward standardizing profiling data. Moreover, the recent merge of the OpenTelemetry Enhancement Proposal (OTEP) introducing profiling support to the OpenTelemetry Protocol (OTLP) marked an additional milestone. 

Why is Elastic contributing its Profiling Agent to OTel?

This contribution not only accelerates the standardization of continuous profiling but also makes continuous profiling the 4th key signal in observability. This empowers everyone in the observability community to continuously profile with a standardized agent. The addition of Elastic’s continuous profiling agent will:

• Align efforts around a single standard poised for broad adoption by users.

• Drive better visibility and improvement of resource usage and cost management for operations.

• Enable vendors and the community to focus on richer features versus dealing with data transformation tasks.

• Enable continuous profiling to become the 4th key signal in Observability.

• Increase continuous profiling adoption and the continued evolution and convergence of observability and security domains.

Why is continuous profiling needed by organizations?

The contribution of Elastic’s continuous profiling agent now helps customers realize the following benefits of continuous profiling:

• Maximize gross margins: By reducing the computational resources needed to run applications, businesses can optimize their cloud spend and improve profitability. Whole-system continuous profiling is one way of identifying the most expensive applications (down to the lines of code) across diverse environments that may span multiple cloud providers. This principle aligns with the familiar adage, "A penny saved is a penny earned." In the cloud context, every CPU cycle saved translates to money saved. 

• Minimize environmental impact: Energy consumption associated with computing is a growing concern (source: MIT Energy Initiative). More efficient code translates to lower energy consumption, contributing to a reduction in carbon (CO2) footprint. 

• Accelerate engineering workflows: Continuous profiling provides detailed insights to help debug complex issues faster, guide development, and improve overall code quality.

With these benefits, customers can now not only manage the overall application’s efficiency on the cloud, but also ensure the application is optimally developed.

What is continuous profiling?

Elastic’s continuous profiling agent is a whole-system, always-on, continuous profiling solution that eliminates the need for run-time/bytecode instrumentation, recompilation, on-host debug symbols or service restarts.   

Profiling helps organizations run efficient services by minimizing computational wastage, thereby reducing operational costs. Leveraging eBPF, the Elastic profiling agent provides unprecedented visibility into the runtime behavior of all applications: it builds stack traces that go from the kernel, through userspace native code, all the way into code running in higher level runtimes, enabling you to identify performance regressions, reduce wasteful computations, and debug complex issues faster. 

To this end, it measures code efficiency in three dimensions: CPU utilization, CO2, and cloud cost. This approach resonates with the sustainability objectives of our customers –– ensuring that Elastic continuous profiling aligns seamlessly with their strategic ESG goals

Does Elastic support OpenTelemetry today?

Elastic supports OTel natively. Elastic users can send OTel data directly from applications or through the OTel collector into Elastic APM, which processes both OTel SemConv and ECS. With this native OTel support, all Elastic APM capabilities are available with OTel. See Elastic documentation to learn more about OTel integration.

Where can I learn more about Elastic’s Universal Profiling?

Elastic’s resources help you understand continuous profiling and how to use it in different scenarios:

• Elastic Universal Profiling home page

• Elastic Universal Profiling agent going open source under Apache 2

• Pinpointing performance issues with profiling

• Elastic releases Universal Profiling

• Whole system profiling with Universal Profiling

• Cost-effective applications with Universal Profiling

• Elastic documentation on Universal Profiling

Addressing this gap, Elastic developed the concept of inferred spans as a powerful enhancement to traditional instrumentation-based tracing as an extension for the OpenTelemetry Java SDK/Agent. We are in the process of contributing this back to OpenTelemetry, until then our extension can be seamlessly used with the existing OpenTelelemetry Java SDK (as described below).

Inferred spans are designed to augment the visibility provided by instrumentation-based traces, shedding light on latency sources within the application or libraries that were previously uninstrumented. This feature significantly expands the utility of distributed tracing, allowing for a more comprehensive understanding of system behavior and facilitating a deeper dive into performance optimization.

What is inferred spans?

Inferred spans is an observability technique that combines distributed tracing with profiling techniques to illuminate the darker, unobserved corners of your application — areas where standard instrumentation techniques fall short. The inferred spans feature interweaves information derived from profiling stacktraces with instrumentation-based tracing data, allowing for the generation of new spans based on the insights drawn from profiling data.

This feature proves invaluable when dealing with custom code or third-party libraries that significantly contribute to the request latency but lack built-in or external instrumentation support. Often, identifying or crafting specific instrumentation for these segments can range from challenging to outright unfeasible. Moreover, certain scenarios exist where implementing instrumentation is impractical due to the potential for substantial performance overhead. For instance, instrumenting application locking mechanisms, despite their critical role, is not viable because of their ubiquitous nature and the significant latency overhead the instrumentation can introduce to application requests. Still, ideally, such latency issues would be visible within your distributed traces.

Inferred spans ensures a deeper visibility into your application’s performance dynamics including the above-mentioned scenarios.

Inferred spans in action

To demonstrate the inferred spans feature we will use the Java implementation of the Elastiflix demo application. Elasticflix has an endpoint called favorites that does some Redis calls and also includes an artificial delay. First, we use the plain OpenTelemetry Java Agent to instrument our application:

java -javaagent:/path/to/otel-javaagent-<version>.jar \
-Dotel.service.name=my-service-name \
-Dotel.exporter.otlp.endpoint=https://<our-elastic-apm-endpoint> \
"-Dotel.exporter.otlp.headers=Authorization=Bearer SECRETTOKENHERE" \
-jar my-service-name.jar

With the OpenTelemetry Java Agent we get out-of-the-box instrumentation for HTTP entry points and calls to Redis for our Elastiflix application. The resulting traces contain spans for the POST /favorites entrypoint, as well as a few short spans for the calls to Redis.

As you can see in the trace above, it’s not clear where most of the time is spent within the POST /favorites request.

Let’s see how inferred spans can shed light into these areas. You can use the inferred spans feature either manually with your OpenTelemetry SDK (see section below), package it as a drop-in extension for the upstream OpenTelemetry Java agent, or just use Elastic’s distribution of the OpenTelemetry Java agent that comes with the inferred spans feature.

For convenience, we just download the agent jar of the Elastic distribution and extend the configuration to enable the inferred spans feature:

java -javaagent:/path/to/elastic-otel-javaagent-<version>.jar \
-Dotel.service.name=my-service-name \
-Dotel.exporter.otlp.endpoint=https://XX.apm.europe-west3.gcp.cloud.es.io:443 \
"-Dotel.exporter.otlp.headers=Authorization=Bearer SECRETTOKENHERE" \
-Delastic.otel.inferred.spans.enabled=true \
-jar my-service-name.jar

The only non-standard option here is elastic.otel.inferred.spans.enabled: The inferred spans Feature is currently opt-in and therefore needs to be enabled explicitly. Running the same application with the inferred spans feature enabled yields more comprehensive traces:

The inferred-spans (colored blue in the above screenshot) follow the naming pattern Class#method. With that, the inferred spans feature helps us pinpoint the exact methods that contribute the most to the overall latency of the request. Note that the parent-child relationship between the HTTP entry span, the Redis spans, and the inferred spans is reconstructed correctly, resulting in a fully functional trace structure.

Examining the handleDelay method within the Elastiflix application reveals the use of a straightforward sleep statement. Although the sleep method is not CPU-bound, the full duration of this delay is captured as inferred spans. This stems from employing the async-profiler's wall clock time profiling, as opposed to solely relying on CPU profiling. The ability of the inferred spans feature to reflect actual latency, including for I/O operations and other non-CPU-bound tasks, represents a significant advancement. It allows for diagnosing and resolving performance issues that extend beyond CPU limitations, offering a more nuanced view of system behavior.

Using inferred spans with your own OpenTelemetry SDK

OpenTelemetry is a highly extensible framework: Elastic embraces this extensibility by also publishing most extensions shipped with our OpenTelemetry Java Distro as standalone-extensions to the OpenTelemetry Java SDK.

As a result, if you do not want to use our distro (e.g., because you don’t need or want bytecode instrumentation in your project), you can still use our extensions, such as the extension for the inferred spans feature. All you need to do is set up the OpenTelemetry SDK in your code and add the inferred spans extension as a dependency:

<dependency>
    <groupId>co.elastic.otel</groupId>
    <artifactId>inferred-spans</artifactId>
    <version>{latest version}</version>
</dependency>

During your SDK setup, you’ll have to initialize and register the extension:

InferredSpansProcessor inferredSpans = InferredSpansProcessor.builder()
  .samplingInterval(Duration.ofMillis(10)) //the builder offers all config options
  .build();
SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
  .addSpanProcessor(inferredSpans)
.addSpanProcessor(BatchSpanProcessor.builder(OtlpGrpcSpanExporter.builder()
    .setEndpoint("https://<your-elastic-apm-endpoint>")
    .addHeader("Authorization", "Bearer <secrettoken>")
    .build()).build())
  .build();
inferredSpans.setTracerProvider(tracerProvider);

The inferred spans extension seamlessly integrates with the OpenTelemetry SDK Autoconfiguration mechanism. By incorporating the OpenTelemetry SDK and its extensions as dependencies within your application code — rather than through an external agent — you gain the flexibility to configure them using the same environment variables or JVM properties. Once the inferred spans extension is included in your classpath, activating it for autoconfigured SDKs becomes straightforward. Simply enable it using the elastic.otel.inferred.spans.enabled property, as previously described, to leverage the full capabilities of this feature with minimal setup.

How does inferred spans work?

The inferred spans feature leverages the capabilities of collecting wall clock time profiling data of the widely-used async-profiler, a low-overhead, popular production-time profiler in the Java ecosystem. It then transforms the profiling data into actionable spans as part of the distributed traces. But what mechanism allows for this transformation?

Essentially, the inferred spans extension engages with the lifecycle of span events, specifically when a span is either activated or deactivated across any thread via the OpenTelemetry context. Upon the activation of the initial span within a transaction, the extension commences a session of wall-clock profiling via the async-profiler, set to a predetermined duration. Concurrently, it logs the details of all span activations and deactivations, capturing their respective timestamps and the threads on which they occurred.

Following the completion of the profiling session, the extension processes the profiling data alongside the log of span events. By correlating the data, it reconstructs the inferred spans. It's important to note that, in certain complex scenarios, the correlation may assign an incorrect name to a span. To mitigate this and aid in accurate identification, the extension enriches the inferred spans with stacktrace segments under the code.stacktrace attribute, offering users clarity and insight into the precise methods implicated.

Inferred spans vs. correlation of traces with profiling data

In the wake of OpenTelemetry's recent announcement of the profiling signal, coupled with Elastic's commitment to donating the Universal Profiling Agent to OpenTelemetry, you might be wondering about how the inferred spans feature differentiates from merely correlating profiling data with distributed traces using span IDs and trace IDs. Rather than viewing these as competing functionalities, it's more accurate to consider them complementary.

The inferred spans feature and the correlation of tracing with profiling data both employ similar methodologies — melding tracing information with profiling data. However, they each shine in distinct areas. Inferred spans excels at identifying long-running methods that could escape notice with traditional CPU profiling, which is more adept at pinpointing CPU bottlenecks. A unique advantage of inferred spans is its ability to account for I/O time, capturing delays caused by operations like disk access that wouldn't typically be visible in CPU profiling flamegraphs.

However, the inferred spans feature has its limitations, notably in detecting latency issues arising from "death by a thousand cuts" — where a method, although not time-consuming per invocation, significantly impacts total latency due to being called numerous times across a request. While individual calls might not be captured as inferred spans due to their brevity, CPU-bound methods contributing to latency are unveiled through CPU profiling, as flamegraphs display the aggregate CPU time consumed by these methods.

An additional strength of the inferred spans feature lies in its data structure, offering a simplified tracing model that outlines typical parent-child relationships, execution order, and good latency estimates. This structure is achieved by integrating tracing data with span activation/deactivation events and profiling data, facilitating straightforward navigation and troubleshooting of latency issues within individual traces.

Correlating distributed tracing data with profiling data comes with a different set of advantages. Learn more about it in our related blog post, Beyond the trace: Pinpointing performance culprits with continuous profiling and distributed tracing correlation.

What about the performance overhead?

As mentioned before, the inferred spans functionality is based on the widely used async-profiler, known for its minimal impact on performance. However, the efficiency of profiling operations is not without its caveats, largely influenced by the specific configurations employed. A pivotal factor in this balancing act is the sampling interval — the longer the interval between samples, the lower the incurred overhead, albeit at the expense of potentially overlooking shorter methods that could be critical to the inferred spans feature discovery process.

Adjusting the probability-based trace sampling presents another way for optimization, directly influencing the overhead. For instance, setting trace sampling to 50% effectively halves the profiling load, making the inferred spans feature even more resource-efficient on average per request. This nuanced approach to tuning ensures that the inferred spans feature can be leveraged in real-world, production environments with a manageable performance footprint. When properly configured, this feature offers a potent, low-overhead solution for enhancing observability and diagnostic capabilities within production applications.

What’s next for inferred spans and OpenTelemetry?

This blog post outlined and introduced the inferred spans feature available as an extension for the OpenTelemetry Java SDK and built into the newly introduced Elastic OpenTelemetry Java Distro. Inferred spans allows users to troubleshoot latency issues in areas of code that are not explicitly instrumented while utilizing traditional tracing data.

The feature is currently merely a port of the existing feature from the proprietary Elastic APM Agent. With Elastic embracing OpenTelemetry, we plan on contributing this extension to the upstream OpenTelemetry project. For that, we also plan on migrating the extension to the latest async-profiler 3.x release. Try out inferred spans for yourself and see how it can help you diagnose performance problems in your applications.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic Universal Profiling™ Agent goes open source under Apache 2

At Elastic, open source is more than just a philosophy — it's our DNA. We believe the benefits of whole-system continuous profiling extend far beyond performance optimization. It's a win for businesses and the planet alike. For instance, since launching Elastic Universal Profiling in general availability (GA), we've observed a wide variety of use cases from customers.

These range from customers relying fully on Universal Profiling's differential flame graphs and topN functions for insights during release management to utilizing AI assistants for quickly optimizing expensive functions. This includes using profiling data to identify the optimal energy-efficient cloud region to run certain workloads. Additionally, customers are using insights that Universal Profiling provides to build evidence to challenge cloud provider bills. As it turns out, cloud providers' in-VM agents can consume a significant portion of the CPU time, which customers are billed for.

In a move that will empower the community to take advantage of continuous profiling's benefits, we're thrilled to announce that the Elastic Universal Profiling agent , a pioneering eBPF-based continuous profiling agent, is now open source under the Apache 2 license!

This move democratizes hyper-scaler efficiency for everyone , opening exciting new possibilities for the future of continuous profiling, as well as its role in observability and OpenTelemetry.

Implementation of the OpenTelemetry (OTel) Profiling protocol

Our commitment to open source goes beyond just the agent itself. We recently announced our intent to donate the agent to OpenTelemetry and have further solidified this goal by implementing the experimental OTel Profiling data model. This allows the open-sourced eBPF-based continuous profiling agent to communicate seamlessly with OpenTelemetry backends.

But that's not all! We've also launched an innovative feature that correlates profiling data with OpenTelemetry distributed traces. This powerful capability offers a deeper level of insight into application performance, enabling the identification of bottlenecks with greater precision. Upon donating the Profiling agent to OTel, Elastic will also contribute critical components that enable distributed trace correlation within the Elastic distribution of the OTel Java agent to the upstream OTel Java SDK. This underscores Elastic Observability's commitment to both open source and the support of open standards like OpenTelemetry while pushing the boundaries of what is possible in observability.

What does this mean for Elastic Universal Profiling customers?

We'd like to express our immense gratitude to all our customers who have been part of this journey, from the early stages of private beta to GA. Your feedback has been invaluable in shaping Universal Profiling into the powerful product it is today.

By open-sourcing the Universal Profiling agent and contributing it to OpenTelemetry, we're fostering a win-win situation for both you and the broader community. This move opens doors for innovation and collaboration, ultimately leading to a more robust and versatile whole-system continuous profiling solution for everyone.

Furthermore, we're actively working on exciting novel ways to integrate Universal Profiling seamlessly within Elastic Observability. Expect further announcements soon, outlining how you can unlock even greater value from your profiling data within a unified observability experience in a way that has never been done before.

The open-sourced agent is using the recently released (experimental) OTel Profiling signal. As a precaution, we recommend not using it in production environments.

Please continue using the official Elastic distribution of the Universal Profiling agent until the agent is formally accepted by OTel and the protocol reaches a stable phase. There's no need to take any action at this time, and we will ensure to have a smooth transition plan in place for you.

What does this mean for the OpenTelemetry community?

OpenTelemetry is adopting continuous profiling as a key signal. By open-sourcing the eBPF-based profiling agent and working towards donating it to OTel, Elastic is making it possible to accelerate the standardization of continuous profiling within OpenTelemetry. This move has a massive impact on the observability community, empowering everyone to continuously profile their systems with a standardized protocol.

This is particularly timely as Moore's Law slows down and cloud computing takes hold, making computational efficiency critical for businesses.

Here's how whole-system continuous profiling benefits you:

• Maximize gross margins: By reducing the computational resources needed to run applications, businesses can optimize their cloud spend and improve profitability. Whole-system continuous profiling is one way of identifying the most expensive applications (down to the lines of code) across diverse environments that may span multiple cloud providers. This principle aligns with the familiar adage, "a penny saved is a penny earned." In the cloud context, every CPU cycle saved translates to money saved.

• Minimize environmental impact: Energy consumption associated with computing is a growing concern (source: MIT Energy Initiative). More efficient code translates to lower energy consumption, contributing to a reduction in carbon footprint.

• Accelerate engineering workflows: Continuous profiling provides detailed insights to help debug complex issues faster, guide development, and improve overall code quality.

This is where Elastic Universal Profiling comes in — designed to help organizations run efficient services by minimizing computational wastage. To this end, it measures code efficiency in three dimensions: CPU utilization , CO** 2 , and cloud cost**.

Elastic's journey with continuous profiling began by joining forces with optimyze.cloud –– this became the foundation for Elastic Universal Profiling. We are excited to see this product evolve into its next growth phase in the open-source world.

Ready to give it a spin?

As Elastic Universal Profiling transitions into this new open source era, the potential for transformative impact on performance optimization, cost efficiency, and environmental sustainability is immense. Elastic's approach — balancing innovation with responsibility — paves the way for a future where technology not only powers our world but does so in a way that is sustainable and accessible to all.

Get started with the open source Elastic Universal Profiling agent today! Download it directly from GitHub and follow the instructions in the repository.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Traces provide the "what" and "where" — what happened and where in your system. Continuous profiling refines this understanding by pinpointing the "why" and validating your hypotheses about the "what." Just like a full-body MRI scan, Elastic's whole-system continuous profiling (powered by eBPF) uncovers unknown-unknowns in your system. This includes not just your code, but also third-party libraries and kernel activity triggered by your application transactions. This comprehensive visibility improves your mean-time-to-detection (MTTD) and mean-time-to-recovery (MTTR) KPIs.

[Related article: Why metrics, logs, and traces aren’t enough]

Bridging the disconnect between continuous profiling and OTel traces

Historically, continuous profiling signals have been largely disconnected from OpenTelemetry (OTel) traces. Here's the exciting news: we're bridging this gap! We're introducing native correlation between continuous profiling signals and OTel traces, starting with Java.

Imagine this: You're troubleshooting a performance issue and identify a slow trace. Whole-system continuous profiling steps in, acting like an MRI scan for your entire codebase and system. It narrows down the culprit to the specific lines of code hogging CPU time within the context of your distributed trace. This empowers you to answer the "why" question with minimal effort and confidence, all within the same troubleshooting context.

Furthermore, by correlating continuous profiling with distributed tracing, Elastic Observability customers can measure the cloud cost and CO₂ impact of every code change at the service and transaction level.

This milestone is significant, especially considering the recent developments in the OTel community. With OTel adopting profiling and Elastic donating the industry’s most advanced eBPF-based continuous profiling agent to OTel, we're set for a game-changer in observability — empowering OTel end users with a correlated system visibility that goes from a trace span in the userspace down to the kernel.

Furthermore, achieving this goal, especially with Java, presented significant challenges and demanded serious engineering R&D. This blog post will delve into these challenges, explore the approaches we considered in our proof-of-concepts, and explain how we arrived at a solution that can be easily extended to other OTel language agents. Most importantly, this solution correlates traces with profiling signals at the agent, not in the backend — to ensure optimal query performance and minimal reliance on vendor backend storage architectures.

Figuring out the active OTel trace and span

The primary technical challenge in this endeavor is essentially the following: whenever the profiler interrupts an OTel instrumented process to capture a stacktrace, we need to be able to efficiently determine the active span and trace ID (per-thread) and the service name (per-process).

For the purpose of this blog, we'll focus on the recently released Elastic distribution of the OTel Java instrumentation, but the approach that we ended up with generalizes to any language that can load and call into a native library. So, how do we get our hands on those IDs?

The OTel Java agent itself keeps track of the active span by storing a stack of spans in the OpenTelemetryContext, which itself is stored in a ThreadLocal variable. We originally considered reading these Java structures directly from BPF, but we eventually decided against that approach. There is no documented specification on how ThreadLocals are implemented, and reliably reading and following the JVM's internal data-structures would incur a high maintenance burden. Any minor update to the JVM could change details of the structure layouts. To add to this, we would also have to reverse engineer how each JVM version lays out Java class fields in memory, as well as how all the high-level Java types used in the context objects are actually implemented under the hood. This approach further wouldn't generalize to any non-JVM language and needs to be repeated for any language that we wish to support.

After we had convinced ourselves that reading Java ThreadLocal directly is not the answer, we decided to look for more portable alternatives instead. The option that we ultimately settled with is to load and call into a C++ library that is responsible for making the required information available via a known and defined interface whenever the span changes.

Other than with Java's ThreadLocals, the details on how a native shared library should expose per-process and per-thread data are well-defined in the System V ABI specification and the architecture specific ELF ABI documents.

Exposing per-process information

Exposing per-process data is easy: we simply declare a global variable . . .

void* elastic_tracecorr_process_storage_v1 = nullptr;

. . . and expose it via ELF symbols. When the user initializes the OTel library to set the service name, we allocate a buffer and populate it with data in a protocol that we defined for this purpose. Once the buffer is fully populated, we update the global pointer to point to the buffer.

On the profiling agent side, we already have code in place that detects libraries and executables loaded into any process's address space. We normally use this mechanism to detect and analyze high-level language interpreters (e.g., libpython, libjvm) when they are loaded, but it also turned out to be a perfect fit to detect the OTel trace correlation library. When the library is detected in a process, we scan the exports, resolve the symbol, and read the per-process information directly from the instrumented process’ memory.

Exposing per-thread information

With the easy part out of the way, let's get to the nitty-gritty portion: exposing per-thread information via thread-local storage (TLS). So, what exactly is TLS, and how does it work? At the most basic level, the idea is to have one instance of a variable for every thread. Semantically you can think of it like having a global Map\<ThreadID, T>, although that is not how it is implemented.

On Linux, there are two major options for thread locals: TSD and TLS.

Thread-specific data (TSD)

TSD is the older and probably more commonly known variant. It works by explicitly allocating a key via pthread_key_create — usually during process startup — and passing it to all threads that require access to the thread-local variable. The threads can then pass that key to the pthread_getspecific and pthread_setspecific functions to read and update the variable for the currently running thread.

TSD is simple, but for our purposes it has a range of drawbacks:

• The pthread_key_t structure is opaque and doesn't have a defined layout. Similar to the Java ThreadLocals, the underlying data-structures aren't defined by the ABI documents and different libc implementations (glibc, musl) will handle them differently.

• We cannot call a function like pthread_getspecific from BPF, so we'd have to reverse engineer and reimplement the logic. Logic may change between libc versions, and we’d have to detect the version and support all variants that may come up in the wild.

• TSD performance is not predictable and varies depending on how many thread local variables have been allocated in the process previously. This may not be a huge concern for Java specifically since spans are typically not swapped super rapidly, but it’d likely be quite noticeable for user-mode scheduling languages where the context might need to be swapped at every await point/coroutine yield.

None of this is strictly prohibitive, but a lot of this is annoying at the very least. Let’s see if we can do better!

Thread-local storage (TLS)

Starting with C11 and C++11, both languages support thread local variables directly via the _Thread_local and thread_local storage specifiers, respectively. Declaring a variable as per-thread is now a matter of simply adding the keyword:

thread_local void* elastic_tracecorr_tls_v1 = nullptr;

You might assume that the compiler simply inserts calls to the corresponding pthread function calls when variables declared with this are accessed, but this is not actually the case. The reality is surprisingly complicated, and it turns out that there are four different models of TLS that the compiler can choose to generate. For some of those models, there are further multiple dialects that can be used to implement them. The different models and dialects come with various portability versus performance trade-offs. If you are interested in the details, I suggest reading this blog article that does a great job at explaining them.

The TLS model and dialect are usually chosen by the compiler based on a somewhat opaque and complicated set of architecture-specific rules. Fortunately for us, both gcc and clang allow users to pick a particular one using the -ftls-model and -mtls-dialect arguments. The variant that we ended up picking for our purposes is -ftls-model=global-dynamic and -mtls-dialect=gnu2 (and desc on aarch64).

Let's take a look at the assembly that is being generated when accessing a thread_local variable under these settings. Our function:

void setThreadProfilingCorrelationBuffer(JNIEnv* jniEnv, jobject bytebuffer) {
  if (bytebuffer == nullptr) {
    elastic_tracecorr_tls_v1 = nullptr;
  } else {
    elastic_tracecorr_tls_v1 = jniEnv->GetDirectBufferAddress(bytebuffer);
  }
}

Is compiled to the following assembly code:

Both possible branches assign a value to our thread-local variable. Let’s focus at the right branch corresponding to the nullptr case to get rid of the noise from the GetDirectBufferAddress function call:

lea   rax, elastic_tracecorr_tls_v1_tlsdesc  ;; Load some pointer into rax.
call  qword ptr [rax]                        ;; Read & call function pointer at rax.
mov   qword ptr fs:[rax], 0                  ;; Assign 0 to the pointer returned by
                                             ;; the function that we just called.

The fs: portion of the mov instruction is the actual magic bit that makes the memory read per-thread. We’ll get to that later; let’s first look at the mysterious elastic_tracecorr_tls_v1_tlsdesc variable that the compiler emitted here. It’s an instance of the tlsdesc structure that is located somewhere in the .got.plt ELF section. The structure looks like this:

struct tlsdesc {
  // Function pointer used to retrieve the offset
  uint64_t (*resolver)(tlsdesc*);

  // TLS offset -- more on that later.
  uint64_t tp_offset;
}

The resolver field is initialized with nullptr and tp_offset with a per-executable offset. The first thread-local variable in an executable will usually have offset 0, the next one sizeof(first_var), and so on. At first glance this may appear to be similar to how TSD works, with the call to pthread_getspecific to resolve the actual offset, but there is a crucial difference. When the library is loaded, the resolver field is filled in with the address of __tls_get_addr by the loader (ld.so). __tls_get_addr is a relatively heavy function that allocates a TLS offset that is globally unique between all shared libraries in the process. It then proceeds by updating the tlsdesc structure itself, inserting the global offset and replacing the resolver function with a trivial one:

void* second_stage_resolver(tlsdesc* desc) {
  return tlsdesc->tp_offset;
}

In essence, this means that the first access to a tlsdesc based thread-local variable is rather expensive, but all subsequent ones are cheap. We further know that by the time that our C++ library starts publishing per-thread data, it must have gone through the initial resolving process already. Consequently, all that we need to do is to read the final offset from the process's memory and memorize it. We also refresh the offset every now and then to ensure that we really have the final offset, combating the unlikely but possible race condition that we read the offset before it was initialized. We can detect this case by comparing the resolver address against the address of the __tls_get_addr function exported by ld.so.

Determining the TLS offset from an external process

With that out of the way, the next question that arises is how to actually find the tlsdesc in memory so that we can read the offset. Intuitively one might expect that the dynamic symbol exported on the ELF file points to that descriptor, but that is not actually the case.

$ readelf --wide --dyn-syms elastic-jvmti-linux-x64.so | grep elastic_tracecorr_tls_v1
328: 0000000000000000     8 TLS   GLOBAL DEFAULT   19 elastic_tracecorr_tls_v1

The dynamic symbol instead contains an offset relative to the start of the .tls ELF section and points to the initial value that libc initializes the TLS value with when it is allocated. So how does ld.so find the tlsdesc to fill in the initial resolver? In addition to the dynamic symbol, the compiler also emits a relocation record for our symbol, and that one actually points to the descriptor structure that we are looking for.

$ readelf --relocs --wide elastic-jvmti-linux-x64.so | grep R_X86_64_TLSDESC
00000000000426e8  0000014800000024 R_X86_64_TLSDESC       0000000000000000
elastic_tracecorr_tls_v1 + 0

To read the final TLS offset, we thus simply have to:

• Wait for the event notifying us about a new shared library being loaded into a process

• Do some cheap heuristics to detect our C++ library, avoiding the more expensive analysis below from being executed for every unrelated library on the system

• Analyze the library on disk and scan ELF relocations for our per-thread variable to extract the tlsdesc address

• Rebase that address to match where our library was loaded in that particular process

• Read the offset from tlsdesc+8

Determining the TLS base

Now that we have the offset, how do we use that to actually read the data that the library puts there for us? This brings us back to the magic fs: portion of the mov instruction that we discussed earlier. In X86, most memory operands can optionally be supplied with a segment register that influences the address translation.

Segments are an archaic construct from the early days of 16-bit X86 where they were used to extend the address space. Essentially the architecture provides a range of segment registers that can be configured with different base addresses, thus allowing more than 16-bits worth of memory to be accessed. In times of 64-bit processors, this is hardly a concern anymore. In fact, X86-64 aka AMD64 got rid of all but two of those segment registers: fs and gs.

So why keep two of them? It turns out that they are quite useful for the use-case of thread-local data. Since every thread can be configured to have its own base address in these segment registers, we can use it to point to a block of data for this specific thread. That is precisely what libc implementations on Linux are doing with the fs segment. The offset that we snatched from the processes memory earlier is used as an address with the fs segment register, and the CPU automatically adds it to the per-thread base address.

To retrieve the base address pointed to by the fs segment register in the kernel, we need to read its destination from the kernel’s task_struct for the thread that we happened to interrupt with our profiling timer event. Getting the task struct is easy because we are blessed with the bpf_get_current_task BPF helper functions. BPF helpers are pretty much syscalls for BPF programs: we can just ask the Linux kernel to hand us the pointer.

Armed with the task pointer, we now have to read the thread.fsbase (X86-64) or thread.uw.tp_value (aarch64) field to get our desired base address that the user-mode process accesses via fs. This is where things get complicated one last time, at least if we wish to support older kernels without BTF support (we do!). The task_struct is huge and there are hundreds of fields that can be present or not depending on how the kernel is configured. Being a core primitive of the scheduler, it is also constantly subject to changes between different kernel versions. On modern Linux distributions, the kernel is typically nice enough to tell us the offset via BTF. On older ones, the situation is more complicated. Since hardcoding the offset is clearly not an option if we hope the code to be portable, we instead have to figure out the offset by ourselves.

We do this by consulting /proc/kallsyms, a file with mappings between kernel functions and their addresses, and then using BPF to dump the compiled code of a kernel function that rarely changes and uses the desired offset. We dynamically disassemble and analyze the function and extract the offset directly from the assembly. For X86-64 specifically, we dump the aout_dump_debugregs function that accesses thread->ptrace_bps, which has consistently been 16 bytes away from the fsbase field that we are interested in for all kernels that we have ever looked at.

Reading TLS data from kernel

With all the required offsets at our hands, we can now finally do what we set out to do in the first place: use them to enrich our stack traces with the OTel trace and span IDs that our C++ library prepared for us!

void maybe_add_otel_info(Trace* trace) {
  // Did user-mode insert a TLS offset for this process? Read it.
  TraceCorrProcInfo* proc = bpf_map_lookup_elem(&tracecorr_procs, &trace->pid);

  // No entry -> process doesn't have the C++ library loaded.
  if (!proc) return;

  // Load the fsbase offset from our global configuration map.
  u32 key = 0;
  SystemConfig* syscfg = bpf_map_lookup_elem(&system_config, &key);

  // Read the fsbase offset from the kernel's task struct.
  u8* fsbase;
  u8* task = (u8*)bpf_get_current_task();
  bpf_probe_read_kernel(&fsbase, sizeof(fsbase), task + syscfg->fsbase_offset);

  // Use the TLS offset to read the **pointer** to our TLS buffer.
  void* corr_buf_ptr;
  bpf_probe_read_user(
    &corr_buf_ptr,
    sizeof(corr_buf_ptr),
    fsbase + proc->tls_offset
  );

  // Read the information that our library prepared for us.
  TraceCorrelationBuf corr_buf;
  bpf_probe_read_user(&corr_buf, sizeof(corr_buf), corr_buf_ptr);

  // If the library reports that we are currently in a trace, store it into
  // the stack trace that will be reported to our user-land process.
  if (corr_buf.trace_present && corr_buf.valid) {
    trace->otel_trace_id.as_int.hi = corr_buf.trace_id.as_int.hi;
    trace->otel_trace_id.as_int.lo = corr_buf.trace_id.as_int.lo;
    trace->otel_span_id.as_int = corr_buf.span_id.as_int;
  }
}

Sending out the mappings

From this point on, everything further is pretty simple. The C++ library sets up a unix datagram socket during startup and communicates the socket path to the profiler via the per-process data block. The stacktraces annotated with the OTel trace and span IDs are sent from BPF to our user-mode profiler process via perf event buffers, which in turn sends the mappings between OTel span and trace and stack trace hashes to the C++ library. Our extensions to the OTel instrumentation framework then read those mappings and insert the stack trace hashes into the OTel trace.

This approach has a few major upsides compared to the perhaps more obvious alternative of sending out the OTel span and trace ID with the profiler’s stacktrace records. We want the stacktrace associations to be stored in the trace indices to allow filtering and aggregating stacktraces by the plethora of fields available on OTel traces. If we were to send out the trace IDs via the profiler's gRPC connection instead, we’d have to search for and update the corresponding OTel trace records in the profiling collector to insert the stack trace hashes.

This is not trivial: stacktraces are sent out rather frequently (every 5 seconds, as of writing) and the corresponding OTel trace might not have been sent and stored by the time the corresponding stack traces arrive in our cluster. We’d have to build a kind of delay queue and periodically retry updating the OTel trace documents, introducing avoidable database work and complexity in the collectors. With the approach of sending stacktrace mappings to the OTel instrumented process instead, the need for server-side merging vanishes entirely.

Trace correlation in action

With all the hard work out of the way, let’s take a look at what trace correlation looks like in action!

Future work: Supporting other languages

We have demonstrated that trace correlation can work nicely for Java, but we have no intention of stopping there. The general approach that we discussed previously should work for any language that can efficiently load and call into our C++ library and doesn’t do user-mode scheduling with coroutines. The problem with user-mode scheduling is that the logical thread can change at any await/yield point, requiring us to update the trace IDs in TLS. Many such coroutine environments like Rust’s Tokio provide the ability to register a callback for whenever the active task is swapped, so they can be supported easily. Other languages, however, do not provide that option.

One prominent example in that category is Go: goroutines are built on user-mode scheduling, but to our knowledge there’s no way to instrument the scheduler. Such languages will need solutions that don’t go via the generic TLS path. For Go specifically, we have already built a prototype that uses pprof labels that are associated with a specific Goroutine, having Go’s scheduler update them for us automatically.

Getting started

We hope this blog post has given you an overview of correlating profiling signals to distributed tracing, and its benefits for end-users.

To get started, download the Elastic distribution of the OTel agent, which contains the new trace correlation library. Additionally, you will need the latest version of Universal Profiling agent, bundled with Elastic Stack version 8.13.

Acknowledgment

We appreciate Trask Stalnaker, maintainer of the OTel Java agent, for his feedback on our approach and for reviewing the early draft of this blog post.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The Elastic AI Assistant significantly enhances the process, not only in identifying but also in resolving issues. This is further enhanced by Elastic’s new Service Level Objective (SLO) capability, allowing you to streamline your entire site reliability engineering (SRE) process from detecting potential issues to enhancing the overall customer experience.

In this blog, we will demonstrate how you, as an SRE, can detect issues in a service equipped with OpenTelemetry. We will explore problem identification using Elastic APM, Elastic’s AIOps capabilities, and the Elastic AI Assistant.

We will illustrate this using the OpenTelemetry demo, with a feature flag (cartService) that is activated.

Our walkthrough will encompass two scenarios:

1. When the SLO for cart service becomes noncompliant, we will analyze the error through Elastic APM. The Elastic AI Assistant will assist by providing a runbook and a GitHub issue to facilitate issue analysis.

2. Should the SLO for the cart service be noncompliant, we will examine the trace that indicates a high failure rate. We will employ AIOps for failure correlation and the AI Assistant to analyze logs and Kubernetes metrics directly from the Assistant.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up the configuration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).

• We used the OpenTelemetry Demo. Directions for using Elastic with OpenTelemetry Demo are here.

• Additionally you will need to connect your AI Assistant to your favorite LLM. We used Azure OpenAI GPT-4.

• We also ran the OpenTelemetry Demo on Kubernetes, specifically on GKE.

SLO noncompliance

Elastic APM recently released the SLO (Service Level Objectives) feature in 8.12. This feature enables setting measurable performance targets for services, such as availability, latency, traffic, errors, and saturation or define your own. Key components include:

• Defining and monitoring SLIs (Service Level Indicators)

• Monitoring error budgets indicating permissible performance shortfalls

• Alerting on burn rates showing error budget consumption

We set up two SLOs for cart service:

• Availability SLO , which monitors its availability by ensuring that transactions succeed. We set up the feature flag in the OpenTelemetry application, which generates an error for EmptyCart transactions 10% of the time.

• Latency SLO to ensure transactions are not going below a specific latency, which will reduce customer experiences.

Because of the OTel cartservice feature flag, the availability SLO is triggered, and within the SLO details, we see that over a seven-day period the availability is well below our target of 99.9, at 95.5. Additionally all the error budget that was available is also exhausted.

With SLO, you can easily identify when issues with customer experience occur, or when potential issues with services arise before they become potentially worse.

Scenario 1: Analyzing APM trace and logs with AI Assistant

Once the SLO is found as non-compliant, we can dive into cart service to investigate in Elastic APM. The following walks through the set of steps you can take in Elastic APM and how to use the AI Assistant to analyze the issue:

From the video, we can see that once in APM, we took the following steps.

1. Investigated the trace EmptyCart, which was experiencing larger than normal failure rates.

2. The trace showed a significant number of failures, which also resulted in slightly larger latency.

3. We used AIOps failure correlation to identify the potential component causing the failure, which correlated to a field value of FailedPrecondition.

4. While filtering on that value and reviewing the logs, we still couldn’t understand what this meant.

5. This is where you can use Elastic’s AI Assistant to further your understanding of the issue.

AI Assistant helped us analyze the following:

1. It helped us understand what the log message meant and that it was related to the Redis connection failure issue.

2. Because we couldn’t connect to Redis, we asked the AI Assistant to give us the metrics for the Redis Kubernetes pods.

3. We learned there were two pods for Redis from the logs over the last two hours.

4. However, we also learned that the memory of one seems to be increasing.

5. It seems that Redis restarted (hence the second pod), and with this information we could dive deeper into what happened to Redis.

You can see how quickly we could correlate a significant amount of information, logs, metrics, and traces through the AI Assistant and Elastic’s APM capabilities. We didn’t have to go through multiple screens to hunt down information.

Scenario 2: Analyzing APM error with AI Assistant

Once the SLO is found as noncompliant, we can dive into cart service to investigate in Elastic APM. The following walks through the set of steps you can take in Elastic APM and use the AI Assistant to analyze the issue:

From the video, we can see that once in APM, we took the following steps:

1. We noticed a specific error for the APM service.

2. We investigated this in the error tab, and while we see it’s an issue with connection to Redis, we still need more information.

3. The AI Assistant helps us understand the stacktrace and provides some potential causes for the error and ways to diagnose and resolve it.

4. We also asked it for a runbook, created by our SRE team, which gives us steps to work through this particular issue.

But as you can see, AI Assistant provides us not only with information about the error message but also how to diagnose it and potentially resolve it with an internal runbook.

Achieving operational excellence, optimal performance, and reliability

We’ve shown how an OpenTelemetry instrumented application (OTel demo) can be analyzed using Elastic’s features, especially the AI Assistant coupled with Elastic APM, AIOps, and the latest SLO features. Elastic significantly streamlines the process of identifying and resolving issues within your applications.

Through our detailed walkthrough of two distinct scenarios, we have seen how Elastic APM and the AI Assistant can efficiently analyze and address noncompliance with SLOs in a cart service. The ability to quickly correlate information, logs, metrics, and traces through these tools not only saves time but also enhances the overall effectiveness of the troubleshooting process.

The use of Elastic's AI Assistant in these scenarios underscores the value of integrating advanced AI capabilities into operational workflows. It goes beyond simple error analysis, offering insights into potential causes and providing actionable solutions, sometimes even with customized runbooks. This integration of technology fundamentally changes how SREs approach problem-solving, making the process more efficient and less reliant on manual investigation.

Overall, the advancements in Elastic’s APM, AIOps capabilities, and the AI Assistant, particularly in handling OpenTelemetry data, represent a significant step forward in operational excellence. These tools enable SREs to not only react swiftly to emerging issues but also proactively manage and optimize the performance and reliability of their services, thereby ensuring an enhanced customer experience.

Try it out

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on cloud? Start a free trial.

• Build better Service Level Objectives (SLOs) from logs and metrics
• Elastic Observability 8.12: GA for AI Assistant, SLO, and Mobile APM support
• Native Observability support in Elastic Observability
• Context-aware insights using the Elastic AI Assistant for Observability

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

What is APM?

Application performance monitoring lets you see where your applications spend their time, what they are doing, what other applications or services they are calling, and what errors or exceptions they are encountering.

In addition, APM also lets you see history and trends for key performance indicators, such as latency and throughput, as well as transaction and dependency information:

Whether you're setting up alerts for SLA breaches, trying to gauge the impact of your latest release, or deciding where to make the next improvement, APM can help with your root-cause analysis to help improve your users' experience and drive your mean time to resolution (MTTR) toward zero.

Logical architecture

Elastic APM relies on the APM Integration inside Elastic Agent, which forwards application trace and metric data from applications instrumented with APM agents to an Elastic Observability cluster. Elastic APM supports multiple agent flavors:

• Native Elastic APM Agents, available for multiple languages, including Java, .NET, Go, Ruby, Python, Node.js, PHP, and client-side JavaScript
• Code instrumented with OpenTelemetry
• Code instrumented with OpenTracing
• Code instrumented with Jaeger

In this blog, we'll provide a quick example of how to instrument code with the native Elastic APM Python agent, but the overall steps are similar for other languages.

Please note that there is a strong distinction between the Elastic APM Agent and the Elastic Agent. These are very different components, as you can see in the diagram above, so it's important not to confuse them.

Install the Elastic Agent

The first step is to install the Elastic Agent. You either need Fleet installed first, or you can install the Elastic Agent standalone. Install the Elastic Agent somewhere by following this guide. This will give you an APM Integration endpoint you can hit. Note that this step is not necessary in Elastic Cloud, as we host the APM Integration for you. Check Elastic Agent is up by running:

curl <ELASTIC_AGENT_HOSTNAME>:8200

Instrumenting sample code with an Elastic APM agent

The instructions for the various language agents differ based on the programming language, but at a high level they have a similar flow. First, you add the dependency for the agent in the language's native spec, then you configure the agent to let it know how to find the APM Integration.

You can try out any flavor you'd like, but I am going to walk through the Python instructions using this Python example that I created.

Get the sample code (or use your own)

To get started, I clone the GitHub repository then change to the directory:

git clone https://github.com/davidgeorgehope/PythonElasticAPMExample
cd PythonElasticAPMExample

How to add the dependency

Adding the Elastic APM Dependency is simple — check the app.py file from the github repo and you will notice the following lines of code.

import elasticapm
from elasticapm import Client

app = Flask(__name__)
app.config["ELASTIC_APM"] = {    "SERVICE_NAME": os.environ.get("APM_SERVICE_NAME", "flask-app"),    "SECRET_TOKEN": os.environ.get("APM_SECRET_TOKEN", ""),    "SERVER_URL": os.environ.get("APM_SERVER_URL", "http://localhost:8200"),}
elasticapm.instrumentation.control.instrument()
client = Client(app.config["ELASTIC_APM"])

The Python library for Flask is capable of auto detecting transactions, but you can also start transactions in code as per the following, as we have done in this example:

@app.route("/")
def hello():
    client.begin_transaction('demo-transaction')
    client.end_transaction('demo-transaction', 'success')

Configure the agent

The agents need to send application trace data to the APM Integration, and to do this it has to be reachable. I configured the Elastic Agent to listen on my local host's IP, so anything in my subnet can send data to it. As you can see from the code below, we use docker-compose.yml to pass in the config via environment variables. Please edit these variables for your own Elastic installation.

# docker-compose.yml
version: "3.9"
services:
  flask_app:
    build: .
    ports:
      - "5001:5001"
    environment:
      - PORT=5001
      - APM_SERVICE_NAME=flask-app
      - APM_SECRET_TOKEN=your_secret_token
      - APM_SERVER_URL=http://host.docker.internal:8200

Some commentary on the above:

• service_name: If you leave this out it will just default to the application's name, but you can override that here.
• secret_token: Secret tokens allow you to authorize requests to the APM Server, but they require that the APM Server is set up with SSL/TLS and that a secret token has been set up. We're not using HTTPS between the agents and the APM Server, so we'll comment this one out.
• server_url: This is how the agent can reach the APM Integration inside Elastic Agent. Replace this with the name or IP of your host running Elastic Agent.

Now that the Elastic APM side of the configuration is done, we simply follow the steps from the README to start up.

docker-compose up --build -d

The build step will take several minutes.

You can navigate to the running sample application by visiting http://localhost:5001. There's not a lot to the sample, but it does generate some APM data. To generate a bit of a load, you can reload them a few times or run a quick little script:

#!/bin/bash
# load_test.sh
url="http://localhost:5001"
for i in {1..1000}
do
  curl -s -o /dev/null $url
  sleep 1
done

This will just reload the pages every second.

Back in Kibana, navigate back to the APM app (hamburger icon, then select APM ) and you should see our new flask-app service (I let mine run so it shows a bit more history):

The Service Overview page provides an at-a-glance summary of the health of a service in one place. If you're a developer or an SRE, this is the page that will help you answer questions like:

• How did a new deployment impact performance?
• What are the top impacted transactions?
• How does performance correlate with underlying infrastructure?

This view provides a list of all of the applications that have sent application trace data to Elastic APM in the specified period of time (in this case, the last 15 minutes). There are also sparklines showing mini graphs of latency, throughput, and error rate. Clicking on flask-app takes us to the service overview page, which shows the various transactions within the service (recall that my script is hitting the / endpoint, as seen in the Transactions section). We get bigger graphs for Latency , Throughput , Errors , and Error Rates.

When you're instrumenting real applications, under real load, you'll see a lot more connectivity (and errors!)

Clicking on a transaction in the transaction view, in this case, our sample app's demo-transaction transaction, we can see exactly what operations were called:

This includes detailed information about calls to external services, such as database queries:

What's next?

Now that you've got your Elastic Observability cluster up and running and collecting out-of-the-box application trace data, explore the public APIs for the languages that your applications are using, which allow you to take your APM data to the next level. The APIs allow you to add custom metadata, define business transactions, create custom spans, and more. You can find the public API specs for the various APM agents (such as Java, Ruby, Python, and more) on the APM agent documentation pages.

If you'd like to learn more about Elastic APM, check out our webinar on Elastic APM in the shift to cloud native to see other ways that Elastic APM can help you in your ecosystem.

If you decide that you'd rather have us host your observability cluster, you can sign up for a free trial of the Elasticsearch Service on Elastic Cloud and change your agents to point to your new cluster.

Originally published May 5, 2021; updated April 6, 2023.

WARNING: This article shows information about the Android agent that is no longer accurate for versions 1.x. Please refer to its documentation to learn about its new APIs.

Elastic® APM for iOS and Android native apps is generally available in the stack release v8.12. The Elastic iOS and Android APM agents are open-source and have been developed on-top, i.e., as a distribution of the OpenTelemetry Swift and Android SDK/API, respectively.

Overview of the Mobile APM solution

The OpenTelemetry SDK/API for iOS and Android supports capabilities such as auto-instrumentation of HTTP requests, API for manual instrumentation, data model based on the OpenTelemetry semantic conventions, and buffering support. Additionally, the Elastic APM agent distributions also support an easier initialization process and novel features such as remote config and user session based sampling. The Elastic iOS and Android APM agents being distributions are maintained per Elastic’s standard support T&Cs.

There are curated or pre-built dashboards provided in Kibana® for monitoring, data analysis, and for troubleshooting purposes. The Service Overview view shown below provides relevant frontend KPIs such as crash rate, http requests, average app load time, and more, including the comparison view.

Further, the geographic distribution of user traffic is available on a map at a country and regional level. The service overview dashboard also shows trends of metrics such as throughput, latency, failed transaction rate, and distribution of traffic by device make-model, network connection type, and app version.

The Transactions view shown below highlights the performance of the different transaction groups, including the distributed trace end-to-end of individual transactions with links to associated spans, errors and crashes. Further, users can see at a glance the distribution of traffic by device make and model, app version, and OS version.

Tabular views such as the one highlighted below located at the bottom of Transactions tab makes it relatively easy to see how the device make and model, App version, etc., impacts latency and crash rate.

The Errors & Crashes view shown below can be used to analyze the different error and crash groups. The unsymbolicated (iOS) or obfuscated (Android) stacktrace of the individual error or crash instance is also available in this view.

The Service Map view shown below provides a visualization of the end-to-end service interdependencies, including any third-party APIs, proxy servers, and databases.

The comprehensive pre-built dashboards for observing the mobile frontend in Kibana provide visibility into the sources of errors, crashes, and bottlenecks to ease troubleshooting of issues in the production environment. The underlying Elasticsearch® Platform also supports the ability to query raw data, build custom metrics and custom dashboards, alerting, SLOs, and anomaly detection. Altogether the platform provides a comprehensive set of tools to expedite root cause analysis and remediation, thereby facilitating a high velocity of innovation.

Walkthrough of the debugging workflow for some error scenarios

Next, we will provide a walkthrough of the configuration details and the troubleshooting workflow for a couple of error scenarios in iOS and Android native apps.

Scenario 1

In this example, we will debug a crash in an asynchronous method using Apple’s crash report symbolication as well as breadcrumbs to deduce the cause of the crash.

Symbolication
In this scenario, users notice a spike in the crash occurrences of a particular crash group in the Errors & Crashes tab and decide to investigate further. A new crash comes in on the Crashes tab, and the developer follows these steps to symbolicate the crash report locally.

1. Copy the crash via the UI and paste it into a file with the following name format \<AppBinaryName>_\<DateTime>. For example, “opbeans-swift_2024-01-18-114211.ips`.

1. Apple provides detailed instructions on how to symbolicate this file locally either automatically through Xcode or manually using the command line.

Breadcrumbs
The second frame of the first thread shows that the crash is occuring in a Worker instance.

This instance is actually used in many places, and due to the asynchronous nature of this function, it’s not possible to determine immediately where this call is coming from. Nevertheless, we can utilize features of the Open Telemetry SDK to add more context to these crashes and then put the pieces together to find the site of the crash.

By adding “breadcrumbs” around this Worker instance, it is possible to track down which calls to the Worker are actually associated with this crash.

Example:
Create a logger provider in the Worker class as a public variable for ease of access, as shown below:

Create breadcrumbs everywhere the Worker.doWork() function is called:

Each of these breadcrumbs will use the same event name “worker_breadcrumb” so they can be consistently queried, and the differentiation will be done using the “ source ” attribute.

In this example, the Worker.doWork() function is being called from a CustomerRow struct (a table row which does work ‘onTapGesture’). If you were to call this method from multiple places in a CustomerRow struct, you may also add additional differentiations to the “ source ” attribute value, such as the associated function (e.g., “CustomerRow#onTapGesture”).

Now that the app is reporting these breadcrumbs, we can use Discover to query for them, as shown below:

Note: Event names sent by the agent are translated to event action in Elastic Common Schema (ECS), so ensure the query uses this field.

1. You can add a filter: event.action: “worker_breadcrumb” and it shows all events generated from this new breadcrumb.

2. You can also see the various sources: ProductRow, CustomerRow, CartRow, etc.

3. If you add error.type : crash to the query, you can see crashes alongside the breadcrumbs:

A crash and a breadcrumb next to each other in the timeline may come from completely different devices, so we need another differentiator. For each crash, we have metadata that contains the session.id associated with the crash, viewable from the Metadata tab. We can query using this session.id to ensure that the only data we are looking at in Discover is from a single user session (i.e., a single device) that resulted in the crash.

In Discover, we can now see the session event flow, on a single device, concerning the crash via the breadcrumbs, as shown below:

It looks like the last breadcrumb before the crash was from the “CustomerRow” breadcrumb. Now this gives the app developer a good place to start their root cause analysis or investigation.

Scenario 2

Note: This scenario requires the Elastic Android agent version “0.14.0” or higher.

An Android sample app has a form composed of two screens that are created using two fragments (FirstPage and SecondPage). In the first screen, the app makes a backend API call to get a key that identifies the form submission. This key is stored in memory in the app and must be available on the last screen where the form is sent; the key must be sent along with the form's data.

The problem
We start to see a spike in crash occurrences in Kibana (null pointer exception) in the Errors & Crashes tab that always seem to happen on the last screen of the form, when the users click on the "FINISH" button. Nevertheless, this is not always reproducible , so the root cause isn't clear just by looking at the crash’s stacktrace alone. Here’s what it looks like:

When we take a look at the code referenced in the stacktrace, this is what we can see:

This is the line where the crash happens, so it seems like the variable “formId” (which is a static String located in “FirstPage”) was null by the time this code was executed, causing a null pointer exception to be raised. This variable is set within the “FirstPage” fragment after the backend request is done to retrieve the id. The only way to get to the “SecondPage” is by passing through the “FirstPage.” So, the stacktrace alone doesn’t help much as the pages have to be opened in order, and the first one will always set the “formId” variable. Therefore, it doesn’t seem likely that the formId could be null in “SecondPage.”

Finding the root cause
Apart from taking a look at the crash’s stacktrace, it could also be useful to take a look at complementary data that would help put the pieces together and get a broader picture of what other things happened while our app was running when the crash happened. For this case, we know that the form ID must come from our backend service, so we could start by ruling out that there was an error with the backend call. We do this by checking the traces from the creation of our FirstPage fragment where the form ID request is executed, in the Transaction details view:

The “Created” spans represent the time it took to create the first fragment. The topmost one shows the Activity creation, followed by the NavHostFragment, followed by “FirstScreen.” Not long after its creation, we see that a GET HTTP request to our backend is made to retrieve our form ID and, according to the traces, the GET request was successful. We can therefore rule out that there is an issue with the backend communication for this problem.

Another option could be looking at the logs sent throughout the session in our app where the crash occurred (we could also take a look at all the logs coming from our app but they would be too many to analyze this one issue). To do so, we first copy one of the spans’ “session.id” values (any span would work since the same session ID will be available in all the data that was sent from our app during the time that the crash occurred) available in the span details flyout.

Note: The same session ID can also be found in the crash metadata.

Now that we have identified our session, we can open up the Logs Explorer view and take a look at all of our app’s logs within that same session, as shown below:

By looking at the logs, and adding a few fields to show the app’s lifecycle status and the error types, we see the log events that are automatically collected from our app. We can see the crash event at the top of the list as the latest one. We can also see our app’s lifecycle events, and if we keep scrolling through, we’ll get to some lifecycle events that are going to help find our root cause:

We can see there are a couple of lifecycle events that tell us that the app was restarted during the session. This is an important hint because it means that the Android OS killed our app at some point, which is common when an app stays in the background for a while. With this information, we could try to reproduce the issue by forcing the OS to kill our app in the background and then see how it behaves when reopened from the recently opened apps menu.

After giving it a try, we could reproduce the issue and we found that the static “formId” variable was lost when the app was restarted, causing it to be null when the SecondPage fragment requested it. We can now research best practices of passing arguments to Fragments so we can change our code to prevent relying on static fields and instead store and share values between screens, thus preventing this crash from happening again.

Bonus: For this scenario, it was enough for us to rely on the events that are sent automatically by the APM Agent; however, if those aren’t enough for other cases, we can always send custom events in the places where we want to track the state changes of our app via the OpenTelemetry event API, as shown in the the code snippet below:

Make the most of your Elastic APM Experience

In this post, we reviewed Elastic’s new Mobile APM solution available in 8.12. The new solution uses Elastic’s new iOS and Android APM agents that are open-source and have been developed on-top, i.e., as a distribution of the OpenTelemetry Swift and Android SDK/API, respectively.

We also reviewed configuration details and the troubleshooting workflow for two error scenarios in iOS and Android native apps.

• iOS scenario: Debug a crash in an asynchronous method using Apple’s crash report symbolication as well as breadcrumbs to deduce the cause of the crash.

• Android scenario: Analyze why users get a null pointer exception on the last screen when they click on the “FINISH” button of a form. Analyzing this is not always clear by looking at the crash’s stack trace and isn’t easily reproducible.

In both instances, we found the root cause of the crash using distributed traces from the mobile device as well as correlated logs. Hopefully this blog provided a review of how Elastic can help manage and monitor Mobile native apps.

Elastic invites SREs and developers to experience our Mobile APM solution firsthand and unlock new horizons in their data tasks. Try it today at https://ela.st/free-trial.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Why efficiency matters — for you and the environment

Data centers are estimated to consume ~3% of global electricity consumption, and their usage is expected to double by 2030.* The cost of a digital service is a close proxy to its computing efficiency, and thus, being more efficient is a win-win: less energy consumed, smaller bill.

In the same scenario, companies want the ability to scale to more users while spending less for each user and are effectively looking into methods of reducing their energy consumption.

In this spirit, Universal Profiling comes equipped with data and visualizations to help determine where efficiency improvement efforts are worth the most.

Energy efficiency measures how much a digital service consumes to produce an output given an input. It can be measured in multiple ways, and we at Elastic Observability chose CO₂ emissions and annualized CO₂ emissions (more details on them later).

Let’s take the example of an e-commerce website: the energy efficiency of the “search inventory” process could be calculated as the average CPU time needed to serve a user request. Once the baseline for this value is determined, changes to the software delivering the search process may result in more or less CPU time consumed for the same feature, resulting in less or more efficient code.

How to set up and configure wattage and CO2

You can find a “Settings” button in the top-right corner of the Universal Profiling views. From there, you can customize the coefficient used to calculate CO₂ emissions tied to profiling data.

The values set here will be used only when the profiles gathered from host agents are not already associated with publicly known data certified by cloud providers. For example, suppose you have a hybrid cloud deployment with a portion of your workload running on-premise and a portion running in GCP. In that case, the values set here will only be used to calculate the CO₂ emissions for the on-premise machines; we already use all the coefficients as declared by GCP to calculate the emissions of those machines.

Python vs. Go

Our first blog post implemented a solution to read PGN chess games, a text representation in Python. It showed how Universal Profiler can be leveraged to identify slow functions and help you rewrite your code faster and more efficiently. At the end of it, we were happy with the Python version. It is still used today to grab the monthly updates from the Lichess database and ingest them into Elasticsearch®. I always wanted a reason to work more with Go, and we rewrote Python to Go. We leveraged goroutines and channels to send data through message passing. You can see more about it in our GitHub repository.

Rewriting in Go also means switching from an interpreted language to a compiled one. As with everything in IT, this has benefits as well as disadvantages. One disadvantage is that we must ship debug symbols for the compiled binary. When we build the binary, we can use the symbtool program to ship the debug symbols. Without debug symbols, we see uninterpretable information as frames will be labeled with hexadecimal addresses in the flame graph rather than source code annotations.

First, make sure that your executable includes debug symbols. Go per default builds with debug symbols. You can check this by using file yourbinary. The important part is that it is not stripped.

file lichess
lichess: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, Go BuildID=gufIkqA61WnCh8haeW-2/lfn3ne3U_y8MGoFD4AvT/QJEykzbacbYEmEQpXH6U/MqVbk-402n1k3B8yPB6I, with debug_info, not stripped

Now we need to push the symbols using symbtool. You must create an Elasticsearch API key as the authentication method. In the Universal Profiler UI in Kibana®, an Add Data button in the top right corner will tell you exactly what to do. The command is like this. The -e is the part where you pass through the path of your executable file. In our case, this is lichess as above.

symbtool push-symbols executable -t "ApiKey" -u "elasticsearch-url" -e "lichess"

Now that debug symbols are available inside the cluster, we can run both implementations with the same file simultaneously and see what Universal Profiler can tell us about it.

Identifying CO2 and energy efficiency savings

Python is more frequently scheduled on the CPU. Thus, it runs more often on the hardware and contributes more to the machines’ resource usage.

We use the differential flame graph to identify and automatically calculate the difference in the following comparison. You need to filter on process.thread.name: “python3.11” in the baseline, and for the comparison, filter for lichess.

Looking at the impact of annualized CO₂ emissions, we see a decrease from 65.32kg of CO₂ from the Python solution to 16.78kg. That is a difference of 48.54kg CO₂ savings over a year.

If we take a step back, we’ll want to figure out why Python produces many more emissions. In the flamegraph view, we filter down to just showing Python, and we can click on the first frame called python3.11. A little popup tells us that it caused 32.95kg of emissions. That is nearly 50% of all emissions caused by the runtime. Our program itself caused the other ~32kg of CO₂. We immediately reduced 32kg of annual emissions by cutting out the Python interpreter with Go.

We can lock that box using a right click and click Show more information.

The Show more information link displays detailed information about the frame, like sample count, total CPU, core seconds, and dollar costs. We won’t go into more detail in this blog.

Reduce your carbon footprint today with Universal Profiling

This blog post demonstrates that rewriting your code base can reduce your carbon footprint immensely. Using Universal Profiler, you could do a quick PoC to showcase how much carbon resources can be spared.

Learn how you can get started with Elastic Universal Profiling today.

• Cluster for storing the data where three nodes, each 64GB RAM and 32 CPU cores, are running GCP on Elastic Cloud.
• The machine for sending the data is a GCP e2-standard-32, thus 128GB RAM and 32 CPU cores with a 500GB balanced disk to read the games from.
• The file used for the games is this Lichess database containing 96,909,211 games. The extracted file size is 211GB.

Source:

*https://media.ccc.de/v/camp2023-57070-energy_consumption_of_data_centers

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog, we’ll walk through a full example from scratch using a Java application. Apart from the Java code and using a JVM for the application, everything else works the same for the other languages supported by the Attacher.

Prerequisites

This walkthrough assumes that the following are already installed on the system: JDK 17, Docker, Kubernetes, and Helm.

The example application

While the application (shown below) is a Java application, it would be easily implemented in any language, as it is just a simple loop that every 2 seconds calls the method chain methodA->methodB->methodC->methodD, with methodC sleeping for 10 milliseconds and methodD sleeping for 200 milliseconds. The choice of application is just to be able to clearly display in the Elastic APM UI that the application is being monitored.

The Java application in full is shown here:

package test;

public class Testing implements Runnable {

  public static void main(String[] args) {
    new Thread(new Testing()).start();
  }

  public void run()
  {
    while(true) {
      try {Thread.sleep(2000);} catch (InterruptedException e) {}
      methodA();
    }
  }

  public void methodA() {methodB();}

  public void methodB() {methodC();}

  public void methodC() {
    System.out.println("methodC executed");
    try {Thread.sleep(10);} catch (InterruptedException e) {}
    methodD();
  }

  public void methodD() {
    System.out.println("methodD executed");
    try {Thread.sleep(200);} catch (InterruptedException e) {}
  }
}

We created a Docker image containing that simple Java application for you that can be pulled from the following Docker repository:

docker.elastic.co/demos/apm/k8s-webhook-test

Deploy the pod

First we need a deployment config. We’ll call the config file webhook-test.yaml, and the contents are pretty minimal — just pull the image and run that as a pod & container called webhook-test in the default namespace:

apiVersion: v1
kind: Pod
metadata:
  name: webhook-test
  labels:
    app: webhook-test
spec:
  containers:
    - image: docker.elastic.co/demos/apm/k8s-webhook-test
      imagePullPolicy: Always
      name: webhook-test

This can be deployed normally using kubectl:

kubectl apply -f webhook-test.yaml

The result is exactly as expected:

$ kubectl get pods
NAME           READY   STATUS    RESTARTS   AGE
webhook-test   1/1     Running   0          10s

$ kubectl logs webhook-test
methodC executed
methodD executed
methodC executed
methodD executed

So far, this is just setting up a standard Kubernetes application with no APM monitoring. Now we get to the interesting bit: adding in auto-instrumentation.

Install Elastic APM K8s Attacher

The first step is to install the Elastic APM K8s Attacher. This only needs to be done once for the cluster — once installed, it is always available. Before installation, we will define where the monitored data will go. As you will see later, we can decide or change this any time. For now, we’ll specify our own Elastic APM server, which is at https://myserver.somecloud:443 — we also have a secret token for authorization to that Elastic APM server, which has value MY_SECRET_TOKEN. (If you want to set up a quick test Elastic APM server, you can do so at https://cloud.elastic.co/).

There are two additional environment variables set for the application that are not generally needed but will help when we see the resulting UI content toward the end of the walkthrough (when the agent is auto-installed, these two variables tell the agent what name to give this application in the UI and what method to trace). Now we just need to define the custom yaml file to hold these. On installation, the custom yaml will be merged into the yaml for the Attacher:

apm:
  secret_token: MY_SECRET_TOKEN
  namespaces:
    - default
webhookConfig:
  agents:
    java:
      environment:
        ELASTIC_APM_SERVER_URL: "https://myserver.somecloud:443"
        ELASTIC_APM_TRACE_METHODS: "test.Testing#methodB"
        ELASTIC_APM_SERVICE_NAME: "webhook-test"

That custom.yaml file is all we need to install the attacher (note we’ve only specified the default namespace for agent auto-installation for now — this can be easily changed, as you’ll see later). Next we’ll add the Elastic charts to helm — this only needs to be done once, then all Elastic charts are available to helm. This is the usual helm add repo command, specifically:

helm repo add elastic https://helm.elastic.co

Now the Elastic charts are available for installation (helm search repo would show you all the available charts). We’re going to use “elastic-webhook” as the name to install into, resulting in the following installation command:

helm install elastic-webhook elastic/apm-attacher --namespace=elastic-apm --create-namespace --values custom.yaml

And that’s it, we now have the Elastic APM K8s Attacher installed and set to send data to the APM server defined in the custom.yaml file! (You can confirm installation with a helm list -A if you need.)

Auto-install the Java agent

The Elastic APM K8s Attacher is installed, but it doesn’t auto-install the APM application agents into every pod — that could lead to problems! Instead the Attacher is deliberately limited to auto-install agents into deployments defined a) by the namespaces listed in the custom.yaml, and b) to those deployments in those namespaces that have a specific annotation “co.elastic.apm/attach.”

So for now, restarting the webhook-test pod we created above won’t have any different effect on the pod, as it isn’t yet set to be monitored. What we need to do is add the annotation. Specifically, we need to add the annotation using the default agent configuration that was installed with the Attacher called “java” for the Java agent (we’ll see later how that agent configuration is altered — the default configuration installs the latest agent version and leaves everything else default for that version). So adding that annotation in to webhook-test yaml gives us the new yaml file contents (the additional config is shown labelled (1)):

apiVersion: v1
kind: Pod
metadata:
  name: webhook-test
  annotations: #(1)
    co.elastic.apm/attach: java #(1)
  labels:
    app: webhook-test
spec:
  containers:
    - image: docker.elastic.co/demos/apm/k8s-webhook-test
      imagePullPolicy: Always
      name: webhook-test

Applying this change gives us the application now monitored:

$ kubectl delete -f webhook-test.yaml
pod "webhook-test" deleted
$ kubectl apply -f webhook-test.yaml
pod/webhook-test created
$ kubectl logs webhook-test
… StartupInfo - Starting Elastic APM 1.45.0 …

And since the agent is now feeding data to our APM server, we can now see it in the UI:

Note that the agent identifies Testing.methodB method as a trace root because of the ELASTIC_APM_TRACE_METHODS environment variable set to test.Testing#methodB in the custom.yaml — this tells the agent to specifically trace that method. The time taken by that method will be available in the UI for each invocation, but we don’t see the sub-methods . . . currently. In the next section, we’ll see how easy it is to customize the Attacher, and in doing so we’ll see more detail about the method chain being executed in the application.

Customizing the agents

In your systems, you’ll likely have development, testing, and production environments. You’ll want to specify the version of the agent to use rather than just pull the latest version whatever that is, you’ll want to have debug on for some applications or instances, and you’ll want to have specific options set to specific values. This sounds like a lot of effort, but the attacher lets you enable these kinds of changes in a very simple way. In this section, we’ll add a configuration that specifies all these changes and we can see just how easy it is to configure and enable it.

We start at the custom.yaml file we defined above. This is the file that gets merged into the Attacher. Adding a new configuration with all the items listed in the last paragraph is easy — though first we need to decide a name for our new configuration. We’ll call it “java-interesting” here. The new custom.yaml in full is (the first part is just the same as before, the new config is simply appended):

apm:
  secret_token: MY_SECRET_TOKEN
  namespaces:
    - default
webhookConfig:
  agents:
    java:
      environment:
        ELASTIC_APM_SERVER_URL: "https://myserver.somecloud:443"
        ELASTIC_APM_TRACE_METHODS: "test.Testing#methodB"
        ELASTIC_APM_SERVICE_NAME: "webhook-test"
    java-interesting:
      image: docker.elastic.co/observability/apm-agent-java:1.55.4
      artifact: "/usr/agent/elastic-apm-agent.jar"
      environment:
        ELASTIC_APM_SERVER_URL: "https://myserver.somecloud:443"
        ELASTIC_APM_TRACE_METHODS: "test.Testing#methodB"
        ELASTIC_APM_SERVICE_NAME: "webhook-test"
        ELASTIC_APM_ENVIRONMENT: "testing"
        ELASTIC_APM_LOG_LEVEL: "debug"
        ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED: "true"
        JAVA_TOOL_OPTIONS: "-javaagent:/elastic/apm/agent/elastic-apm-agent.jar"

Breaking the additional config down, we have:

• The name of the new config java-interesting

• The APM Java agent image docker.elastic.co/observability/apm-agent-java

• With a specific version 1.43.0 instead of latest

• We need to specify the agent jar location (the attacher puts it here)

• artifact: "/usr/agent/elastic-apm-agent.jar"

• And then the environment variables

• ELASTIC_APM_SERVER_URL as before

• ELASTIC_APM_ENVIRONMENT set to testing, useful when looking in the UI

• ELASTIC_APM_LOG_LEVEL set to debug for more detailed agent output

• ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED turning this on (setting to true) will give us additional interesting information about the method chain being executed in the application

• And lastly we need to set JAVA_TOOL_OPTIONS to the enable starting the agent "-javaagent:/elastic/apm/agent/elastic-apm-agent.jar" — this is fundamentally how the attacher auto-attaches the Java agent

More configurations and details about configuration options are here for the Java agent, and other language agents are also available.

The application traced with the new configuration

And finally we just need to upgrade the attacher with the changed custom.yaml:

helm upgrade elastic-webhook elastic/apm-attacher --namespace=elastic-apm --create-namespace --values custom.yaml

This is the same command as the original install, but now using upgrade. That’s it — add config to the custom.yaml and upgrade the attacher, and it’s done! Simple.

Of course we still need to use the new config on an app. In this case, we’ll edit the existing webhook-test.yaml file, replacing java with java-interesting, so the annotation line is now:

co.elastic.apm/attach: java-interesting

Applying the new pod config and restarting the pod, you can see the logs now hold debug output:

$ kubectl delete -f webhook-test.yaml
pod "webhook-test" deleted
$ kubectl apply -f webhook-test.yaml
pod/webhook-test created
$ kubectl logs webhook-test
… StartupInfo - Starting Elastic APM 1.44.0 …
… DEBUG co.elastic.apm.agent. …
… DEBUG co.elastic.apm.agent. …

More interesting is the UI. Now that inferred spans is on, the full method chain is visible.

This gives the details for methodB (it takes 211 milliseconds because it calls methodC - 10ms - which calls methodD - 200ms). The times for methodC and methodD are inferred rather than recorded, (inferred rather than traced — if you needed accurate times you would instead add the methods to trace_methods and have them traced too).

Note on the ECK operator

The Elastic Cloud on Kubernetes operator allows you to install and manage a number of other Elastic components on Kubernetes. At the time of publication of this blog, the Elastic APM K8s Attacher is a separate component, and there is no conflict between these management mechanisms — they apply to different components and are independent of each other.

Try it yourself!

This walkthrough is easily repeated on your system, and you can make it more useful by replacing the example application with your own and the Docker registry with the one you use.

Learn more about real-time monitoring with Kubernetes and Elastic Observability.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Efficiency is important (again)

Before we jump into continuous profiling, let's start with the "Why should I care?" question. To do that, I'd like to talk a bit about efficiency and some large-scale trends happening in our industry that are making efficiency, specifically computational efficiency, important again. I say again because in the past, when memory and storage on a computer was very limited and you had to worry about every byte of code, efficiency was an important aspect of developing software.

The end of Moore’s Law

First, the Moore's Law era is drawing to a close. This was inevitable simply due to physical limits of how small you can make a transistor and the connections between them. For a long time, software developers had the luxury of not worrying about complexity and efficiency because the next generation of hardware would mitigate any negative cost or performance impact.

If you can't rely on an endless progression of ever faster hardware, you should be interested in computational efficiency.

The move to Software-as-a-Service

Another trend to consider is the shift from software vendors that sold customers software to run themselves to Software-as-a-Service businesses. A traditional software vendor didn't have to worry too much about the efficiency of their code. That issue largely fell to the customer to address; a new software version might dictate a hardware refresh to the latest and most performant. For a SaaS business, inefficient software usually degrades the customer’s experience and it certainly impacts the bottom line.

If you are a SaaS business in a competitive environment, you should be interested in computational efficiency.

Cloud migration

Next is the ongoing cloud migration to cloud computing. One of the benefits of cloud computing is the ease of scaling, both hardware and software. In the cloud, we are not constrained by the limits of our data centers or the next hardware purchase. Instead we simply spin up more cloud instances to mitigate performance problems. In addition to infrastructure scalability, microservices architectures, containerization, and the rise of Kubernetes and similar orchestration tools means that scaling services is simpler than ever. It's not uncommon to have thousands of instances of a service running in a cloud environment. This ease of scaling accounts for another trend, namely that many businesses are dealing with skyrocketing cloud computing costs.

If you are a business with ever increasing cloud costs, you should be interested in computational efficiency.

Our changing climate

Lastly, if none of those reasons pique your interest, let's consider a global problem that all of us should have in mind — namely, climate change. There are many things that need to be addressed to tackle climate change, but with our dependence on software in every part of our society, computational efficiency is certainly something we should be thinking about.

Thomas Dullien, distinguished engineer at Elastic and one of the founders of Optymize points out that if you can save 20% on 800 servers, and assume 300W power consumption for each server, that code change is worth 160 metric tons of CO₂ saved per year. That may seem like a drop in the bucket but if all businesses focus more on computational efficiency, it will make an impact. Also, let's not forget the financial benefits: those 160 metric tons of CO₂ savings also represent a significant annual cost savings.

If you live on planet Earth, you should be interested in computational efficiency.

Performance engineering

Who's job is it to worry about computational efficiency? Application developers usually pay at least some attention to efficiency as they develop their code. Profiling is a common approach for a developer to understand the performance of their code, and there is an entire portfolio of profiling tools available. Frequently, however, schedule pressures trump time spent on performance analysis and computational efficiency. In addition, performance problems may not become apparent until an application is running at scale in production and interacting (and competing) with everything else in that environment. Many profiling tools are not well suited to use in a production environment because they require code instrumentation and recompilation and add significant overhead.

When inefficient code makes it into production and begins to cause performance problems, the next line of defense is the Operations or SRE team. Their mission is to keep everything humming, and performance problems will certainly draw attention. Observability tools such as APM can shed light on these types of issues and lead the team to a specific application or service, but these tools have limits into the observability of the full system. Third-party libraries and operating system kernels functions remain hidden without a profiling solution in the production environment.

So, what can these teams do when there is a need to investigate a performance problem in production? That's where continuous profiling comes into the picture.

Continuous profiling

Continuous profiling is not a new idea. Google published a paper about it in 2010 and began implementing continuous profiling in its environments around that time. Facebook and Netflix followed suit not long afterward.

Typically, continuous profiling tools have been the domain of dedicated performance engineering or operating system engineering teams, which are usually only found at extremely large scale enterprises like the ones mentioned above. The key idea is to run profiling on every server, all of the time. That way, when your observability tools point you to a specific part of an application, but you need a more detailed view into exactly where that application is consuming CPU resources, the profiling data will be there, ready to use.

Another benefit of continuous profiling is that it provides a view of CPU intensive software across your entire environment — whether that is a very CPU intensive function or the aggregate of a relatively small function that is run thousands of times a second in your environment.

While profiling tools are not new, most of them have significant gaps. Let's look at a couple of the most significant ones.

• Limited visibility. Modern distributed applications are composed of a complex mix of building blocks, including custom software functions, third-party software libraries, networking software, operating system services, and more and more often, orchestration software such as Kubernetes. To fully understand what is happening in an application, you need visibility into each piece. However, even if a developer has the ability to profile their own code, everything else remains invisible. To make matters worse, most profiling tools require instrumenting the code, which adds overhead and therefore even your developers’ code is not profiled in production.
• Missing symbols in production. All of these pieces of code building blocks typically have descriptive names (some more intuitive than others) so that developers can understand and make sense of them. In a running program, these descriptive names are usually referred to as symbols. For a human being to make sense of the execution of a running application, these names are very important. Unfortunately, almost always, any software running in production has these human readable symbols stripped away for space efficiency since they are not needed by the CPU executing the software. Without all of the symbols, it makes it much more difficult to understand the full picture of what's happening in the application. To illustrate this, think of the last time you were in an SMS chat on your mobile device and you only had some of the people in the chat group in your address book while the rest simply appeared as phone numbers — this makes it very hard to tell who is saying what.

Elastic Universal Profiling: Continuous profiling for all

Our goal is to allow any business, large or small, to make computational efficiency a core consideration for all of the software that they run. Universal Profiling imposes very low overhead on your servers so it can be used in production and it provides visibility to everything running on every machine. It opens up the possibility of seeing the financial unit cost and CO₂ impact of every line of code running on every system in your business. How do we do that?

Whole-system visibility — SIMPLE

Universal Profiling is based on eBPF, which means that it imposes very low overhead (our goal is less than 1% CPU and less than 250MB of RAM) on your servers because it doesn't require code instrumentation. That low overhead means it can be run continuously, on every server, even in production.

eBPF also lets us deploy a single profiler agent on a host and peek inside the operating system to see every line of code executing on the CPU. That means we have visibility into all of those application building blocks described above — the operating system itself as well as containerization and orchestration frameworks without complex configuration.

All the symbols

A key part of Universal Profiling is our hosted symbolization service. This means that symbols are not required on your servers, which not only eliminates a need for recompiling software with symbols, but it also helps to reduce overhead by allowing the Universal Profiling agent to send very sparse data back to the Elasticsearch platform where it is enriched with all of the missing symbols. Since we maintain a repository of most popular third-party software libraries and Linux operating system symbols, the Universal Profiling UI can show you all the symbols.

Your favorite language, and then some

Universal Profiling is multilanguage. We support all of today’s popular programming languages, including Python, Go, Java (and any other JVM-based languages), Ruby, NodeJS, PHP, Perl, and of course, C and C++, which is critical since these languages still underly so many third-party libraries used by the other languages. In addition, we support profiling native code a.k.a. machine language.

Speaking of native code, all profiling tools are tied to a specific type of CPU. Most tools today only support the Intel x86 CPU architecture. Universal Profiling supports both x86 and ARM-based processors. With the expanding use of ARM-based servers, especially in cloud environments, Universal Profiling future-proofs your continuous profiling.

Many businesses today employ polyglot programming — that is, they use multiple languages to build an application — and Universal Profiling is the only profiler available that can build a holistic view across all of these languages. This will help you look for hotspots in the environment, leading you to "unknown unknowns" that warrant deeper performance analysis. That might be a simple interest rate calculation that should be efficient and lightweight but, surprisingly, isn't. Or perhaps it is a service that is reused much more frequently than originally expected, resulting in thousands of instances running across your environment every second, making it a prime target for efficiency improvement.

Visualize your impact

Elastic Universal Profiling has an intuitive UI that immediately shows you the impact of any given function, including the time it spends executing on the CPU and how much that costs both in dollars and in carbon emissions.

Finally, with the level of software complexity in most production environments, there's a good chance that making a code change will have unanticipated effects across the environment. That code change may be due to a new feature being rolled out or a change to improve efficiency. In either case, a differential view, before and after the change, will help you understand the impact.

Let's recap

Computational efficiency is an important topic, both from the perspective of the ultra-competitive business climate we all work in and from living through the challenges of our planet's changing climate. Improving efficiency can be a challenging endeavor, but we can't even begin to attempt to make improvements without knowing where to focus our efforts. Elastic Universal Profiling is here to provide every business with visibility into computational efficiency.

How will you use Elastic Universal Profiling in your business?

• If you are an application developer or part of the site reliability team, Universal Profiling will provide you with unprecedented visibility into your applications that will not only help you troubleshoot performance problems in production, but also understand the impact of new features and deliver an optimal user experience.
• If you are involved in cloud and infrastructure financial management and capacity planning, Universal Profiling will provide you with unprecedented visibility into the unit cost of every line of code that your business runs.
• If you are involved in your business’s ESG initiative, Universal Profiling will provide you with unprecedented visibility into your CO₂ emissions and open up new avenues for reducing your carbon footprint.

These are just a few examples. For more ideas, read how AppOmni benefits from Elastic Universal Profiling.

You can get started with Elastic Universal Profiling right now!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Using Elastic Observability and Azure Container Apps is a perfect combination for developers to deploy web apps that are auto-scaled with fully observable operations.

This blog post will show you how to deploy a simple Hello World web app to Azure Container Apps and then walk you through the steps to instrument the Hello World web app to enable observation of the application’s operations with Elastic Cloud.

Elastic Observability setup

We’ll start with setting up an Elastic Cloud deployment, which is where observability will take place for the web app we’ll be deploying.

From the Elastic Cloud console, select Create deployment.

Enter a deployment name and click Create deployment. It takes a few minutes for your deployment to be created. While waiting, you are prompted to save the admin credentials for your deployment, which provides you with superuser access to your Elastic® deployment. Keep these credentials safe as they are shown only once.

Elastic Observability requires an APM Server URL and an APM Secret token for an app to send observability data to Elastic Cloud. Once the deployment is created, we’ll copy the Elastic Observability server URL and secret token and store them somewhere safely for adding to our web app code in a later step.

To copy the APM Server URL and the APM Secret Token, go to Elastic Cloud . Then go to the Deployments page, which lists all of the deployments you have created. Select the deployment you want to use, which will open the deployment details page. In the Kibana row of links, click on Open to open Kibana® for your deployment.

Select Integrations from the top-level menu. Then click the APM tile.

On the APM Agents page, copy the secretToken and the serverUrl values and save them for use in a later step.

Now that we’ve completed the Elastic Cloud setup, the next step is to set up our account in Azure for deploying apps to the Container Apps service.

Azure Container Apps setup

First we’ll need an Azure account, so let’s create one by going to the Microsoft Azure portal and creating a new project. Click the Start free button and follow the steps to sign in or create a new account.

Deploy a Hello World web app to Container Apps

We’ll perform the process of deploying a C# Hello World web app to Container Apps using the handy Azure tool called Cloud Shell. To deploy the Hello World app, we’ll perform the following 12 steps:

1. From the Azure portal, click the Cloud Shell icon at the top of the portal to open Cloud Shell…

… and when the Cloud Shell first opens, select Bash as the shell type to use.

1. If you’re prompted that “You have no storage mounted,” then click the Create storage button to create a file store to be used for saving and editing files from Cloud Shell.

1. In Cloud Shell, clone a C# Hello World sample app repo from GitHub by entering the following command.

git clone https://github.com/elastic/observability-examples

1. Change directory to the location of the Hello World web app code.

cd observability-examples/azure/container-apps/helloworld

1. Define the environment variables that we’ll be using in the commands throughout this blog post.

RESOURCE_GROUP="helloworld-containerapps"
LOCATION="centralus"
ENVIRONMENT="env-helloworld-containerapps"
APP_NAME="elastic-helloworld"

1. Define a registry container name that is unique by running the following command.

ACR_NAME="helloworld"$RANDOM

1. Create an Azure resource group by running the following command.

az group create --name $RESOURCE_GROUP --location "$LOCATION"

1. Run the following command to create a registry container in Azure Container Registry.

az acr create --resource-group $RESOURCE_GROUP \
--name $ACR_NAME --sku Basic --admin-enable true

1. Build the app image and push it to Azure Container Registry by running the following command.

az acr build --registry $ACR_NAME --image $APP_NAME .

1. Register the Microsoft.OperationalInsights namespace as a provider by running the following command.

az provider register -n Microsoft.OperationalInsights --wait

1. Run the following command to create a Container App environment for deploying your app into.

az containerapp env create --name $ENVIRONMENT \
--resource-group $RESOURCE_GROUP --location "$LOCATION"

1. Create a new Container App by deploying the Hello World app’s image to Container Apps, using the following command.

az containerapp create \
  --name $APP_NAME \
  --resource-group $RESOURCE_GROUP \
  --environment $ENVIRONMENT \
  --image $ACR_NAME.azurecr.io/$APP_NAME \
  --target-port 3500 \
  --ingress 'external' \
  --registry-server $ACR_NAME.azurecr.io \
  --query properties.configuration.ingress.fqdn

This command will output the deployed Hello World app's fully qualified domain name (FQDN). Copy and paste the FQDN into a browser to see your running Hello World app.

Instrument the Hello World web app with Elastic Observability

With a web app successfully running in Container Apps, we’re now ready to add the minimal code necessary to enable observability for the Hello World app in Elastic Cloud. We’ll perform the following eight steps:

1. In Azure Cloud Shell, create a new file named Telemetry.cs by typing the following command.

touch Telemetry.cs

1. Open the Azure Cloud Shell file editor by typing the following command in Cloud Shell.

code .

1. In the Azure Cloud Shell editor, open the Telemetry.cs file and paste in the following code. Save the edited file in Cloud Shell by pressing the [Ctrl] + [s] keys on your keyboard (or if you’re on a macOS computer, use the [⌘] + [s] keys). This class file is used to create a tracer ActivitySource, which can generate trace Activity spans for observability.

using System.Diagnostics;

public static class Telemetry
{
    public static readonly ActivitySource activitySource = new("Helloworld");
}

1. In the Azure Cloud Shell editor, edit the file named Dockerfile to add the following Elastic OpenTelemetry environment variables. Replace the ELASTIC_APM_SERVER_URL text and the ELASTIC_APM_SECRET_TOKEN text with the APM Server URL and the APM Secret Token values that you copied and saved in an earlier step.

Save the edited file in Cloud Shell by pressing the [Ctrl] + [s] keys on your keyboard (or if you’re on a macOS computer, use the [⌘] + [s] keys).

The updated Dockerfile should look something like this:

FROM ${ARCH}mcr.microsoft.com/dotnet/aspnet:7.0. AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/sdk:8.0-preview AS build
ARG TARGETPLATFORM

WORKDIR /src
COPY ["helloworld.csproj", "./"]
RUN dotnet restore "./helloworld.csproj"
COPY . .
WORKDIR "/src/."
RUN dotnet build "helloworld.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "helloworld.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
EXPOSE 3500
ENV ASPNETCORE_URLS=http://+:3500

ENV OTEL_EXPORTER_OTLP_ENDPOINT='https://******.apm.us-east-2.aws.elastic-cloud.com:443'
ENV OTEL_EXPORTER_OTLP_HEADERS='Authorization=Bearer ***********'
ENV OTEL_LOG_LEVEL=info
ENV OTEL_METRICS_EXPORTER=otlp
ENV OTEL_RESOURCE_ATTRIBUTES=service.version=1.0,deployment.environment=production
ENV OTEL_SERVICE_NAME=helloworld
ENV OTEL_TRACES_EXPORTER=otlp

ENTRYPOINT ["dotnet", "helloworld.dll"]

1. In the Azure Cloud Shell editor, edit the helloworld.csproj file to add the Elastic APM and OpenTelemetry dependencies. The updated helloworld.csproj file should look something like this:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net7.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Elastic.Apm" Version="1.24.0" />
    <PackageReference Include="Elastic.Apm.NetCoreAll" Version="1.24.0" />
    <PackageReference Include="OpenTelemetry" Version="1.6.0" />
    <PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.6.0" />
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.6.0" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.6.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.5.0-beta.1" />
  </ItemGroup>

</Project>

1. In the Azure Cloud Shell editor, edit the Program.cs:

• Add a using statement at the top of the file to import System.Diagnostics, which is used to create Activities that are equivalent to “spans” in OpenTelemetry. Also import the OpenTelemetry.Resources and OpenTelemetry.Trace packages.

using System.Diagnostics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

• Update the “builder” initialization code block to include configuration to enable Elastic OpenTelemetry observability.

builder.Services.AddOpenTelemetry().WithTracing(builder => builder.AddOtlpExporter()
                    .AddSource("helloworld")
                    .AddAspNetCoreInstrumentation()
                    .AddOtlpExporter()
        .ConfigureResource(resource =>
            resource.AddService(
                serviceName: "helloworld"))
);
builder.Services.AddControllers();

• Replace the “Hello World!” HTML output string…

<h1>Hello World!</h1>

• …with the “Hello Elastic Observability” HTML output string.

<div style="text-align: center;">
  <h1 style="color: #005A9E; font-family:'Verdana'">
    Hello Elastic Observability - Azure Container Apps - C#
  </h1>
  <img
    src="https://elastichelloworld.blob.core.windows.net/elastic-helloworld/elastic-logo.png"
  />
</div>

• Add a telemetry trace span around the output response utilizing the Telemetry class’ ActivitySource.

using (Activity activity = Telemetry.activitySource.StartActivity("HelloSpan")!)
       {
           Console.Write("hello");
           await context.Response.WriteAsync(output);
       }

The updated Program.cs file should look something like this:

using System.Diagnostics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetry().WithTracing(builder => builder.AddOtlpExporter()
                    .AddSource("helloworld")
                    .AddAspNetCoreInstrumentation()
                    .AddOtlpExporter()
        .ConfigureResource(resource =>
            resource.AddService(
                serviceName: "helloworld"))
);
builder.Services.AddControllers();
var app = builder.Build();

string output =
"""
<div style="text-align: center;">
<h1 style="color: #005A9E; font-family:'Verdana'">
Hello Elastic Observability - Azure Container Apps - C#
</h1>
<img src="https://elastichelloworld.blob.core.windows.net/elastic-helloworld/elastic-logo.png">
</div>
""";

app.MapGet("/", async context =>
    {
        using (Activity activity = Telemetry.activitySource.StartActivity("HelloSpan")!)
            {
                Console.Write("hello");
                await context.Response.WriteAsync(output);
            }
    }
);
app.Run();

1. Rebuild the Hello World app image and push the image to the Azure Container Registry by running the following command.

az acr build --registry $ACR_NAME --image $APP_NAME .

1. Redeploy the updated Hello World app to Azure Container Apps, using the following command.

az containerapp create \
  --name $APP_NAME \
  --resource-group $RESOURCE_GROUP \
  --environment $ENVIRONMENT \
  --image $ACR_NAME.azurecr.io/$APP_NAME \
  --target-port 3500 \
  --ingress 'external' \
  --registry-server $ACR_NAME.azurecr.io \
  --query properties.configuration.ingress.fqdn

This command will output the deployed Hello World app's fully qualified domain name (FQDN). Copy and paste the FQDN into a browser to see the updated Hello World app running in Azure Container Apps.

Observe the Hello World web app

Now that we’ve instrumented the web app to send observability data to Elastic Observability, we can now use Elastic Cloud to monitor the web app’s operations.

1. In Elastic Cloud, select the Observability Services menu item.

2. Click the helloworld service.

3. Click the Transactions tab.

4. Scroll down and click the GET / transaction.Scroll down to the Trace Sample section to see the GET / , HelloSpan trace sample.

Observability made to scale

You’ve seen the entire process of deploying a web app to Azure Container Apps that is instrumented with Elastic Observability. This web app is now fully available on the web running on a platform that will auto-scale to serve visitors worldwide. And it’s instrumented for Elastic Observability APM using OpenTelemetry to ingest data into Elastic Cloud’s Kibana dashboards.

Now that you’ve seen how to deploy a Hello World web app with a basic observability setup, visit Elastic Observability to learn more about expanding to a full scale observability coverage solution for your apps. Or visit Getting started with Elastic on Microsoft Azure for more examples of how you can drive the data insights you need by combining Microsoft Azure’s cloud computing services with Elastic’s search-powered platform.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

This blog post will show you how to deploy a simple Hello World web app to App Runner and then walk you through the steps to instrument the Hello World web app to enable observation of the application’s operations with Elastic Cloud.

Elastic Observability setup

We’ll start with setting up an Elastic Cloud deployment, which is where observability will take place for the web app we’ll be deploying.

From the Elastic Cloud console, select Create deployment.

Enter a deployment name and click Create deployment. It takes a few minutes for your deployment to be created. While waiting, you are prompted to save the admin credentials for your deployment, which provides you with superuser access to your Elastic® deployment. Keep these credentials safe as they are shown only once.

Elastic Observability requires an APM Server URL and an APM Secret token for an app to send observability data to Elastic Cloud. Once the deployment is created, we’ll copy the Elastic Observability server URL and secret token and store them somewhere safely for adding to our web app code in a later step.

To copy the APM Server URL and the APM Secret Token, go to Elastic Cloud. Then go to theDeployments page, which lists all of the deployments you have created. Select the deployment you want to use, which will open the deployment details page. In the Kibana® row of links, click on Open to open Kibana for your deployment.

Select Integrations from the top-level menu. Then click the APM tile.

On the APM Agents page, copy the secretToken and the serverUrl values and save them for use in a later step.

Now that we’ve completed the Elastic Cloud setup, the next step is to set up our AWS project for deploying apps to App Runner.

AWS App Runner setup

To start using AWS App Runner, you need an AWS account. If you’re a brand new user, go to aws.amazon.com to sign up for a new account.

Set up AWS CloudShell

We’ll perform the process of creating a Python Hello World App image and pushing it to the AWS ECR using AWS CloudShell.

We’re going to use Docker to build the sample app image. Perform the following five steps to set up Docker within CloudShell.

1. Open AWS CloudShell.

1. Run the following two commands to install Docker in CloudShell:

sudo yum update -y
sudo amazon-linux-extras install docker

1. Start Docker by running the command:

sudo dockerd

1. With Docker running, open a new tab in CloudShell by clicking the Actions dropdown menu and selecting New tab.

1. Run the following command to authenticate Docker within CloudShell. Replace \<account_id> with your AWS Account ID in the Docker command below, and then run it in CloudShell.

aws ecr get-login-password --region us-east-2 | sudo docker login --username AWS --password-stdin <account_id>.dkr.ecr.us-east-2.amazonaws.com

Build the Hello World web app image and push it to AWS ECR

We’ll be using AWS ECR, Amazon’s fully managed container registry for storing and deploying application images. To build and push the Hello World app image to AWS ECR, we’ll perform the following six steps in AWS CloudShell:

1. Run the command below in CloudShell to create a repository in AWS ECR.

aws ecr create-repository \
    --repository-name elastic-helloworld/web \
    --image-scanning-configuration scanOnPush=true \
    --region us-east-2

“elastic-helloworld” will be the application's name and “ web” will be the service name.

1. In the newly created tab within CloudShell, clone a Python Hello World sample app repo from GitHub by entering the following command.

git clone https://github.com/elastic/observability-examples

1. Change directory to the location of the Hello World web app code by running the following command:

cd observability-examples/aws/app-runner/helloworld

1. Build the Hello World sample app from the application’s directory. Run the following Docker command in CloudShell.

sudo docker build -t elastic-helloworld/web .

1. Tag the application image. Replace \<account_id> with your AWS Account ID in the Docker command below, and then run it in CloudShell.

sudo docker tag elastic-helloworld/web:latest <account_id>.dkr.ecr.us-east-2.amazonaws.com/elastic-helloworld/web:latest

1. Push the application image to ECR. Replace \<account_id> with your AWS Account ID in the command below, and then run it in CloudShell.

sudo docker push <account_id>.dkr.ecr.us-east-2.amazonaws.com/elastic-helloworld/web:latest

Deploy a Hello World web app to AWS App Runner

We’ll perform the process of deploying a Python Hello World App to App Runner using the AWS App Runner console.

1. Open the App Runner console and click the Create an App Runner service button.

1. On the Source and deployment page, set the following deployment details:

• In the Source section, for Repository type, choose Container registry.
• For Provider, choose Amazon ECR.
• For Container image URI, choose Browse to select the Hello World application image that we previously pushed to AWS ECR.
• In the Select Amazon ECR container image dialog box, for Image repository, select the “ elastic-helloworld/web” repository.
• For Image tag, select “ latest” and then choose Continue.
• In the Deployment settings section, choose Automatic.
• For ECR access role, choose Create new service role.
• Click Next.

1. On the Configure service page, in the Service settings section, enter the service name “ helloworld-app.” Leave all the other settings as they are and click Next.

1. On the Review and create page, click Create & deploy.

After a few minutes, the Hello World app will be deployed to App Runner.

1. Click the Default domain URL to view the Hello World app running in App Runner.

Instrument the Hello World web app with Elastic Observability

With a web app successfully running in App Runner, we’re now ready to add the minimal code necessary to start monitoring the app. To enable observability for the Hello World app in Elastic Cloud, we’ll perform the following five steps in AWS CloudShell:

1. Edit the Dockerfile file to add the following Elastic Open Telemetry environment variables along with the commands to install and run the Elastic APM agent. Use the “nano” text editor by typing “nano Dockerfile”. Be sure to replace the \<ELASTIC_APM_SERVER_URL> text and the \<ELASTIC_APM_SECRET_TOKEN> text with the APM Server URL and the APM Secret Token values that you copied and saved in an earlier step. The updated Dockerfile should look something like this:

FROM python:3.9-slim as base

# get packages
COPY requirements.txt .
RUN pip install -r requirements.txt

WORKDIR /app

# install opentelemetry packages
RUN pip install opentelemetry-distro opentelemetry-exporter-otlp
RUN opentelemetry-bootstrap -a install

ENV OTEL_EXPORTER_OTLP_ENDPOINT='<ELASTIC_APM_SERVER_URL>'
ENV OTEL_EXPORTER_OTLP_HEADERS='Authorization=Bearer%20<ELASTIC_APM_SECRET_TOKEN>'
ENV OTEL_LOG_LEVEL=info
ENV OTEL_METRICS_EXPORTER=otlp
ENV OTEL_RESOURCE_ATTRIBUTES=service.version=1.0,deployment.environment=production
ENV OTEL_SERVICE_NAME=helloworld
ENV OTEL_TRACES_EXPORTER=otlp

COPY . .
ENV FLASK_APP=helloworld
ENV FLASK_RUN_HOST=0.0.0.0
ENV FLASK_RUN_PORT=8080
EXPOSE 8080
ENTRYPOINT [ "opentelemetry-instrument", "flask", "run" ]

Note: You can close the nano text editor and save the file by typing “Ctrl + x”. Press the “y” key and then the “Enter” key to save the changes.

1. Edit the helloworld.py file to add observability traces. In CloudShell, type “nano helloworld.py” to edit the file.

• After the import statements at the top of the file, add the code required to initialize the Elastic Open Telemetry APM agent:

from opentelemetry import trace
tracer = trace.get_tracer("hello-world")

• Replace the “Hello World!” output code . . .

return "<h1>Hello World!</h1>";

• … with the Hello Elastic Observability code block.

return '''
<div style="text-align: center;">
<h1 style="color: #005A9E; font-family:'Verdana'">
Hello Elastic Observability - AWS App Runner - Python
</h1>
<img src="https://elastic-helloworld.s3.us-east-2.amazonaws.com/elastic-logo.png">
</div>
'''

• Then add a “hi” trace before the Hello Elastic Observability code block along with an additional “@app.after_request” method placed afterward to implement a “bye” trace.

@app.route("/")
def helloworld():
    with tracer.start_as_current_span("hi") as span:
        logging.info("hello")
        return '''
        <div style="text-align: center;">
        <h1 style="color: #005A9E; font-family:'Verdana'">
        Hello Elastic Observability - AWS App Runner - Python
        </h1>
        <img src="https://elastic-helloworld.s3.us-east-2.amazonaws.com/elastic-logo.png">
        </div>
        '''

@app.after_request
def after_request(response):
    with tracer.start_as_current_span("bye"):
        logging.info("goodbye")
        return response

The completed helloworld.py file should look something like this:

import logging
from flask import Flask

from opentelemetry import trace
tracer = trace.get_tracer("hello-world")

app = Flask(__name__)

@app.route("/")
def helloworld():
    with tracer.start_as_current_span("hi") as span:
        logging.info("hello")
        return '''
        <div style="text-align: center;">
        <h1 style="color: #005A9E; font-family:'Verdana'">
        Hello Elastic Observability - AWS App Runner - Python
        </h1>
        <img src="https://elastic-helloworld.s3.us-east-2.amazonaws.com/elastic-logo.png">
        </div>
        '''

@app.after_request
def after_request(response):
    with tracer.start_as_current_span("bye"):
        logging.info("goodbye")
        return response

Note: You can close the nano text editor and save the file by typing “Ctrl + x”. Press the “y” key and then the “Enter” key to save the changes.

1. Rebuild the updated Hello World sample app using Docker from within the application’s directory. Run the following command in CloudShell.

sudo docker build -t elastic-helloworld/web .

1. Tag the application image using Docker. Replace \<account_id> with your AWS Account ID in the Docker command below and then run it in CloudShell.

sudo docker tag elastic-helloworld/web:latest <account_id>.dkr.ecr.us-east-2.amazonaws.com/elastic-helloworld/web:latest

1. Push the updated application image to ECR. Replace \<account_id> with your AWS Account ID in the Docker command below and then run it in CloudShell.

sudo docker push <account_id>.dkr.ecr.us-east-2.amazonaws.com/elastic-helloworld/web:latest

Pushing the image to ECR will automatically deploy the new version of the Hello World app.

Open the App Runner console. After a few minutes, the Hello World app will be deployed to App Runner. Click the Default domain URL to view the updated Hello World app running in App Runner.

Observe the Hello World web app

Now that we’ve instrumented the web app to send observability data to Elastic Observability, we can now use Elastic Cloud to monitor the web app’s operations.

1. In Elastic Cloud, select the Observability Services menu item.

2. Click the helloworld service.

3. Click the Transactions tab.

4. Scroll down and click the “/” transaction.

5. Scroll down to the Trace Sample section to see the “/,” “hi,” and “bye” trace samples.

Observability made to scale

You’ve seen the complete process of deploying a web app to AWS App Runner that is instrumented with Elastic Observability. The end result is a web app that will scale up and down with usage, combined with the observability tools to monitor the web app as it serves one user or millions of users.

Now that you’ve seen how to deploy a serverless web app instrumented with observability, visit Elastic Observability to learn more about how to implement a complete observability solution for your apps. Or visit Getting started with Elastic on AWS for more examples of how you can drive the data insights you need by combining AWS’s cloud computing services with Elastic’s search-powered platform.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

SREs and developers who want to maintain robust, efficient systems and achieve optimal code performance need effective tools to measure and improve code performance. Profilers are invaluable for these tasks, as they can help you boost your app's throughput, ensure consistent system reliability, and gain a deeper understanding of your code's behavior at runtime. However, traditional profilers can be cumbersome to use, as they often require code recompilation and are limited to specific languages. Additionally, they can also have a high overhead that negatively affects performance and makes them less suitable for quick, real-time debugging in production environments.

To address the limitations of traditional profilers, Elastic^® recently announced the general availability of Elastic Universal Profiling, a continuous profiling product that is refreshingly straightforward to use, eliminating the need for instrumentation, recompilations, or restarts. Moreover, Elastic Universal Profiling does not require on-host debug symbols and is language-agnostic, allowing you to profile any process running on your machines — from your application's code to third-party libraries and even kernel functions.

However, even the most advanced tools require a certain level of expertise to interpret the data effectively. The wealth of visual profiling data — flamegraphs, stacktraces, or functions — can initially seem overwhelming. This blog post aims to demystify continuous profiling and guide you through its unique visualizations. We will equip you with the knowledge to derive quick, actionable insights from Universal Profiling.

Let’s begin.

Stacktraces: The cornerstone for profiling

It all begins with a stacktrace — a snapshot capturing the cascade of function calls.

A stacktrace is a snapshot of the call stack of an application at a specific point in time. It captures the sequence of function calls that the program has made up to that point. In this way, a stacktrace serves as a historical record of the call stack, allowing you to trace back the steps that led to a particular state in your application.

Further, stacktraces are the foundational data structure that profilers rely on to determine what an application is executing at any given moment. This is particularly useful when, for instance, your infrastructure monitoring indicates that your application servers are consuming 95% of CPU resources. While utilities such as 'top -H' can show the top processes that are consuming CPU, they lack the granularity needed to identify the specific lines of code (in the top process) responsible for the high usage.

In the case of Elastic Universal Profiling, eBPF is used to perform sampling of every process that is keeping a CPU core busy. Unlike most instrumentation profilers that focus solely on your application code, Elastic Universal Profiling provides whole-system visibility — it profiles not just your code, but also code you don't own, including third-party libraries and even kernel operations.

The diagram below shows how the Universal Profiling agent works at a very high level. Step 5 indicates the ingestion of the stacktraces into the profiling collector, a new part of the Elastic Stack.

Just deploy the profiling host agent and receive profiling data (in Kibana^®) a few minutes later. Get started now .

1. Unwinder eBPF programs (bytecode) are sent to the kernel.

2. The kernel verifies that the BPF program is safe. If accepted, the program is attached to the probes and executed when the event occurs.

3. The eBPF programs pass the collected data to userspace via maps.

4. The agent reads the collected data from maps. The data transferred from the agent to the maps are process-specific and interpreter-specific meta-information that help the eBPF unwinder programs perform unwinding.

5. Stacktraces, metrics, and metadata are pushed to the Elastic Stack.

6. Visualize data as flamegraphs, stacktraces, and functions via Kibana.

While stacktraces are the key ingredient for most profiling tools, interpreting them can be tricky. Let's take a look at a simple example to make things a bit easier. The table below shows a group of stacktraces from a Java application and assigns each a percentage to indicate its share of CPU time consumption.

Table 1: Grouped Stacktraces with CPU Time Percentage

| Percentage | Function Calls | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | 60% | startApp -> authenticateUser -> processTransaction | | 20% | startApp -> loadAccountDetails -> fetchRecentTransactions | | 10% | startApp -> authenticateUser -> processTransaction -> verifyFunds | | 2% | startApp -> authenticateUser -> processTransaction ->libjvm.so | | 1% | startApp -> authenticateUser -> processTransaction ->libjvm.so ->vmlinux: asm_common_interrupt ->vmlinux: asm_sysvec_apic_timer_interrupt |

The percentages above represent the relative frequency of each specific stacktrace compared to the total number of stacktraces collected over the observation period, not actual CPU usage percentages. Also, the libjvm.so and kernel frames (vmlinux:*) in the example are commonly observed with whole-system profilers like Elastic Universal Profiling.

Also, we can see that 60% of the time is spent in the sequence startApp; authenticateUser; processTransaction. An additional 10% of the processing time is allocated to verifyFunds, a function invoked by processTransaction. Given these observations, it becomes evident that optimization initiatives would yield the most impact if centered on the processTransaction function, as it is one of the most expensive functions. However, real-world stacktraces can be far more intricate than this example. So how do we make sense of them quickly? The answer to this problem resulted in the creation of flamegraphs.

Flamegraphs: A visualization of stacktraces

While the above example may appear straightforward, it scarcely reflects the complexities encountered when aggregating multiple stacktraces across a fleet of machines on a continuous basis. The depth of the stack traces and the numerous branching paths can make it increasingly difficult to pinpoint where code is consuming resources. This is where flamegraphs, a concept popularized by Brendan Gregg, come into play.

A flamegraph is a visual interpretation of stacktraces, designed to quickly and accurately identify the functions that are consuming the most resources. Each function is represented by a rectangle, where the width of the rectangle represents the amount of time spent in the function, and the number of stacked rectangles represents the stack depth. The stack depth is the number of functions that were called to reach the current function.

Elastic Universal Profiling uses icicle graphs, which is an inverted variant of the standard flamegraph. In an icicle graph, the root function is at the top, and its child functions are shown below their parents –– making it easier to see the hierarchy of functions and how they are related to each other.

In most flamegraphs, the y-axis represents stack depth, but there is no standardization for the x-axis. Some profiling tools use the x-axis to indicate the passage of time; in these instances, the graph is more accurately termed a flame chart. Others sort the x-axis alphabetically. Universal Profiling sorts functions on the x-axis based on relative CPU percentage utilization, starting with the function that consumes the most CPU time on the left, as shown in the example icicle graph below.

Debugging and optimizing performance issues: Stacktraces, TopN functions, flamegraphs

SREs and SWEs can use Universal Profiling for troubleshooting, debugging, and performance optimization. It builds stacktraces that go from the kernel, through userspace native code, all the way into code running in higher level runtimes, enabling you to identify performance regressions , reduce wasteful computations , and debug complex issues faster.

To this end, Universal Profiling offers three main visualizations: Stacktraces, TopN Functions, and flamegraphs.

Stacktrace view

The stacktraces view shows grouped stacktrace graphs by threads, hosts, Kubernetes deployments, and containers. It can be used to detect unexpected CPU spikes across threads and drill down into a smaller time range to investigate further with a flamegraph. Refer to the documentation for details.

TopN functions view

Universal Profiling's topN functions view shows the most frequently sampled functions, broken down by CPU time, annualized CO₂, and annualized cost estimates. You can use this view to identify the most expensive functions across your entire fleet, and then apply filters to focus on specific components for a more detailed analysis. Clicking on a function name will redirect you to the flamegraph, enabling you to examine the call hierarchy.

Flamegraphs view

The flamegraph page is where you will most likely spend the most time, especially when debugging and optimizing. We recommend that you use the guide below to identify performance bottlenecks and optimization opportunities with flamegraphs. The three key elements-conditions to look for are width , hierarchy , and height.

Width matters: In icicle graphs, wider rectangles signify functions taking up more CPU time. Always read the graph from left to right and note the widest rectangles, as these are the prime hot spots.

Hierarchy matters: Navigate the graph's stack to understand function relationships. This vertical examination will help you identify whether one or multiple functions are responsible for performance bottlenecks. This could also uncover opportunities for code improvements, such as swapping an inefficient library or avoiding unnecessary I/O operations.

Height matters: Elevated or tall stacks in the graph usually point to deep call hierarchies. These can be an indicator of complex and less efficient code structures that may require attention.

Also, when navigating a flamegraph, you may want to look for specific function names to validate your assumptions on their presence: in the Universal Profiling flamegraphs view, there is a “Search” bar at the bottom left corner of the view. You can input a regex, and the match will be highlighted in the flamegraph; by clicking on the left and right arrows next to the Search bar, you can move across the occurrences on the flamegraph and spot callers and callee of the matched function.

In summary,

• Scan horizontally from left to right, focusing on width for CPU-intensive functions.
• Examine vertically to examine the stack and spot bottlenecks.
• Look for towering stacks to identify potential complexities in the code.

To recap, use topN functions to generate optimization hypotheses and validate them with stacktraces and/or flamegraphs. Use stacktraces to monitor CPU utilization trends and to delve into the finer details. Use flamegraphs to quickly debug and optimize your code, using width, hierarchy, and height as guides.

Identify. Optimize. Measure. Repeat!

Measure the impact of your change

For the very first time in history, developers can now measure the performance (gained or lost), cloud cost, and carbon footprint impact of every deployed change.

Once you have identified a performance issue and applied fixes or optimizations to your code, it is essential to measure the impact of your changes. The differential topN functions and differential flamegraph pages are invaluable for this, as they can help you identify regressions and measure your change impact not only in terms of performance but also in terms of carbon emissions and cost savings.

The Diff column indicates a change in the function’s rank.

You may need to use tags or other metadata, such as container and deployment name, in combination with time ranges to differentiate between the optimized and non-optimized changes.

Universal Profiling: The key to optimizing application resources

Computational efficiency is no longer just a nice-to-have, but a must-have from both a financial and environmental sustainability perspective. Elastic Universal Profiling provides unprecedented visibility into the runtime behavior of all your applications, so you can identify and optimize the most resource-intensive areas of your code. The result is not merely better-performing software but also reduced resource consumption, lower cloud costs, and a reduction in carbon footprint. Optimizing your code with Universal Profiling is not only the right thing to do for your business, it’s the right thing to do for our world.

Get started with Elastic Universal Profiling today.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

OpenTelemetry is more than just becoming the open ingestion standard for observability. As one of the major Cloud Native Computing Foundation (CNCF) projects, with as many commits as Kubernetes, it is gaining support from major ISVs and cloud providers delivering support for the framework. Many global companies from finance, insurance, tech, and other industries are starting to standardize on OpenTelemetry. With OpenTelemetry, DevOps teams have a consistent approach to collecting and ingesting telemetry data providing a de-facto standard for observability.

Elastic^® is strategically standardizing on OpenTelemetry for the main data collection architecture for observability and security. Additionally, Elastic is making a commitment to help OpenTelemetry become the best de facto data collection infrastructure for the observability ecosystem. Elastic is deepening its relationship with OpenTelemetry beyond the recent contribution of Elastic Common Schema (ECS) to OpenTelemetry (OTel).

Today, Elastic supports OpenTelemetry natively, since Elastic 7.14, by being able to directly ingest OpenTelemetry protocol (OTLP) based traces, metrics, and logs.

In this blog, we’ll review the current OpenTelemetry support provided by Elastic, which includes the following:

• Easy ingest of distributed tracing and metrics for applications configured with OpenTelemetry agents for Python, NodeJS, Java, Go, and .NET
• OpenTelemetry logs instrumentation and ingest using various configurations
• Open semantic conventions for logs and more through ECS, which is not part of OpenTelemetry
• Machine learning based AIOps capabilities, such as latency correlations, failure correlations, anomaly detection, log spike analysis, predictive pattern analysis, Elastic AI Assistant support, and more, all apply to native OTLP telemetry.
• Migrate applications to OpenTelemetry at your own speed. Elastic’s APM capabilities all work seamlessly even with a mix of services using OpenTelemetry and/or Elastic APM agents. You can even combine OpenTelemetry instrumentation with Elastic Agent.
• Integrated views and analysis with Kubernetes clusters, which most OpenTelemetry applications are running on. Elastic can highlight specific pods and containers related to each service when analyzing issues for applications based on OpenTelemetry.

Ingesting OpenTelemetry into Elastic

If you’re interested in seeing how simple it is to ingest OpenTelemetry traces and metrics into Elastic, follow the steps outlined in this blog.

Let’s outline what Elastic provides for ingesting OpenTelemetry data. Here are all your options:

Using the OpenTelemetry Collector

When using the OpenTelemetry Collector, which is the most common configuration option, you simply have to add two key variables.

The instructions utilize a specific opentelemetry-collector configuration for Elastic. Essentially, the Elastic values.yaml file specified in the elastic/opentelemetry-demo configure the opentelemetry-collector to point to the Elastic APM Server using two main values:

OTEL_EXPORTER_OTLP_ENDPOINT is Elastic’s APM Server
OTEL_EXPORTER_OTLP_HEADERS Elastic Authorization

These two values can be found in the OpenTelemetry setup instructions under the APM integration instructions (Integrations->APM) in your Elastic Cloud.

Native OpenTelemetry agents embedded in code

If you are thinking of using OpenTelemetry libraries in your code, you can simply point the service to Elastic’s APM server, because it supports native OLTP protocol. No special Elastic conversion is needed.

To demonstrate this effectively and provide some education on how to use OpenTelemetry, we have two applications you can use to learn from:

• Elastic’s version of OpenTelemetry demo: As with all the other observability vendors, we have our own forked version of the OpenTelemetry demo.
• Elastiflix: This demo application is an example to help you learn how to instrument on various languages and telemetry signals.

Check out our blogs on using the Elastiflix application and instrumenting with OpenTelemetry:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation

We have created YouTube videos on these topics as well:

• How to Manually Instrument Java with OpenTelemetry (Part 1)
• How to Manually Instrument Java with OpenTelemetry (Part 2)
• Custom Java Instrumentation with OpenTelemetry
• Elastic APM - Automatic .NET Instrumentation with OpenTelemetry
• How to Manually Instrument .NET Applications with OpenTelemetry

Given Elastic and OpenTelemetry’s vast user base, these provide a rich source of education for anyone trying to learn the intricacies of instrumenting with OpenTelemetry.

Elastic Agents supporting OpenTelemetry

If you’ve already implemented OpenTelemetry, you can still use them with OpenTelemetry. Elastic APM agents today are able to ship OpenTelemetry spans as part of a trace. This means that if you have any component in your application that emits an OpenTelemetry span, it’ll be part of the trace the Elastic APM agent captures.

OpenTelemetry logs in Elastic

If you look at OpenTelemetry documentation, you will see that a lot of language libraries are still in experimental or not implemented yet state. Java is in stable state, per the documentation. Depending on your service’s language, and your appetite for adventure, there exist several options for exporting logs from your services and applications and marrying them together in your observability backend.

In a previous blog, we discussed 3 different configurations to properly get logging data into Elastic for Java. The blog explores the current state of the art of OpenTelemetry logging and provides guidance on the available approaches with the following tenants in mind:

• Correlation of service logs with OTel-generated tracing where applicable
• Proper capture of exceptions
• Common context across tracing, metrics, and logging
• Support for slf4j key-value pairs (“structured logging”)
• Automatic attachment of metadata carried between services via OTel baggage
• Use of an Elastic Observability backend
• Consistent data fidelity in Elastic regardless of the approach taken

Three models, which are covered in the blog, currently exist for getting your application or service logs to Elastic with correlation to OTel tracing and baggage:

• Output logs from your service (alongside traces and metrics) using an embedded OpenTelemetry Instrumentation library to Elastic via the OTLP protocol
• Write logs from your service to a file scrapped by the OpenTelemetry Collector, which then forwards to Elastic via the OTLP protocol
• Write logs from your service to a file scrapped by Elastic Agent (or Filebeat), which then forwards to Elastic via an Elastic-defined protocol

Note that (1), in contrast to (2) and (3), does not involve writing service logs to a file prior to ingestion into Elastic.

OpenTelemetry is Elastic’s preferred schema

Elastic recently contributed the Elastic Common Schema (ECS) to the OpenTelemetry (OTel) project, enabling a unified data specification for security and observability data within the OTel Semantic Conventions framework.

ECS, an open source specification, was developed with support from the Elastic user community to define a common set of fields to be used when storing event data in Elasticsearch^®. ECS helps reduce management and storage costs stemming from data duplication, improving operational efficiency.

Similarly, OTel’s Semantic Conventions (SemConv) also specify common names for various kinds of operations and data. The benefit of using OTel SemConv is in following a common naming scheme that can be standardized across a codebase, libraries, and platforms for OTel users.

The merging of ECS and OTel SemConv will help advance OTel’s adoption and the continued evolution and convergence of observability and security domains.

Elastic Observability APM and machine learning capabilities

All of Elastic Observability’s APM capabilities are available with OTel data (read more on this in our blog, Independence with OpenTelemetry):

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services
• Transactions (traces)
• ML correlations (specifically for latency)
• Service logs

In addition to Elastic’s APM and unified view of the telemetry data, you will now be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR. Here are some of the ML based AIOps capabilities we have:

• Anomaly detection: Elastic Observability, when turned on (see documentation), automatically detects anomalies by continuously modeling the normal behavior of your OpenTelemetry data — learning trends, periodicity, and more.
• Log categorization: Elastic also identifies patterns in your OpenTelemetry log events quickly, so that you can take action quicker.
• High-latency or erroneous transactions: Elastic Observability’s APM capability helps you discover which attributes are contributing to increased transaction latency and identifies which attributes are most influential in distinguishing between transaction failures and successes.
• Log spike detector helps identify reasons for increases in OpenTelemetry log rates. It makes it easy to find and investigate causes of unusual spikes by using the analysis workflow view.
• Log pattern analysis helps you find patterns in unstructured log messages and makes it easier to examine your data.

Elastic allows you to migrate to OTel on your schedule

Although OpenTelemetry supports many programming languages, the status of its major functional components — metrics, traces, and logs — are still at various stages. Thus migrating applications written in Java, Python, and JavaScript are good choices to start with as their metrics, traces, and logs (for Java) are stable.

For the other languages that are not yet supported, you can easily instrument those using Elastic Agents, therefore running your full stack observability platform in mixed mode (Elastic agents with OpenTelemetry agents).

Here is a simple example:

The above shows a simple variation of our standard Elastic Agent application with one service flipped to OTel — the newsletter-otel service. But we can easily and as needed convert each of these services to OTel as development resources allow.

Hence you can migrate what you need to OpenTelemetry with Elastic as specific languages reach a stable state, and you can then continue your migration to OpenTelemetry agents.

Integrated Kubernetes and OpenTelemetry views in Elastic

Elastic manages your Kubernetes cluster using the Elastic Agent, and you can use it on your Kubernetes cluster where your OpenTelemetry application is running. Hence you can not only use OpenTelemetry for your application, but Elastic can also monitor the corresponding Kubernetes cluster.

There are two configurations for Kubernetes:

1. Simply deploying the Elastic Agent daemon set on the kubernetes cluster. We outline this out in the article entitled Managing your Kubernetes cluster with Elastic Observability. This would also push just the Kubernetes metrics and logs to Elastic.

2. Deploying the Elastic Agent with not only the Kubernetes Daemon set, but also Elastic’s APM integration, the Defend (Security) integration, and Network Packet capture integration to provide more comprehensive Kubernetes cluster observability. We outline this configuration in the following article Modern observability and security on Kubernetes with Elastic and OpenTelemetry.

Both OpenTelemetry visualization examples use the OpenTelemetry demo, and in Elastic, we tie the Kubernetes information with the application to provide you an ability to see Kubernetes information from your traces in APM. This provides a more integrated approach when troubleshooting.

Summary

In essence, Elastic's commitment goes beyond mere support for OpenTelemetry. We are dedicated to ensuring our customers not only adopt OpenTelemetry but thrive with it. Through our solutions, expertise, and resources, we aim to elevate the observability journey for every business, turning data into actionable insights that drive growth and innovation.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Observability across the entire stack of modern distributed applications requires data collection, processing, and correlation often in the form of dashboards. Ingesting all system data requires installing agents across stacks, frameworks, and providers — a process that can be challenging and time-consuming for teams who have to deal with version changes, compatibility issues, and proprietary code that doesn't scale as systems change.

Thanks to OpenTelemetry (OTel), DevOps and SRE teams now have a standard way to collect and send data that doesn't rely on proprietary code and have a large support community reducing vendor lock-in.

In this blog post, we will show you how to manually instrument Go applications using OpenTelemetry. This approach is slightly more complex than using auto-instrumentation

In a previous blog, we also reviewed how to use the OpenTelemetry demo and connect it to Elastic^®, as well as some of Elastic’s capabilities with OpenTelemetry. In this blog, we will use an alternative demo application, which helps highlight manual instrumentation in a simple way.

Finally, we will discuss how Elastic supports mixed-mode applications, which run with Elastic and OpenTelemetry agents. The beauty of this is that there is no need for the otel-collector! This setup enables you to slowly and easily migrate an application to OTel with Elastic according to a timeline that best fits your business.

Application, prerequisites, and config

The application that we use for this blog is called Elastiflix, a movie streaming application. It consists of several micro-services written in .NET, NodeJS, Go, and Python.

Before we instrument our sample application, we will first need to understand how Elastic can receive the telemetry data.

All of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Prerequisites

• An Elastic Cloud account — sign up now
• A clone of the Elastiflix demo application, or your own Go application
• Basic understanding of Docker — potentially install Docker Desktop
• Basic understanding of Go

View the example source code

The full source code including the Dockerfile used in this blog can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

Before we begin, let’s look at the non-instrumented code first.

This is our simple go application that can receive a GET request. Note that the code shown here is a slightly abbreviated version.

package main

import (
    "log"
    "net/http"
    "os"
    "time"

    "github.com/go-redis/redis/v8"

    "github.com/sirupsen/logrus"

    "github.com/gin-gonic/gin"
    "strconv"
    "math/rand"
)

var logger = &logrus.Logger{
    Out:   os.Stderr,
    Hooks: make(logrus.LevelHooks),
    Level: logrus.InfoLevel,
    Formatter: &logrus.JSONFormatter{
        FieldMap: logrus.FieldMap{
            logrus.FieldKeyTime:  "@timestamp",
            logrus.FieldKeyLevel: "log.level",
            logrus.FieldKeyMsg:   "message",
            logrus.FieldKeyFunc:  "function.name", // non-ECS
        },
        TimestampFormat: time.RFC3339Nano,
    },
}

func main() {
    delayTime,  := strconv.Atoi(os.Getenv("TOGGLE_SERVICE_DELAY"))

    redisHost := os.Getenv("REDIS_HOST")
    if redisHost == "" {
        redisHost = "localhost"
    }

    redisPort := os.Getenv("REDIS_PORT")
    if redisPort == "" {
        redisPort = "6379"
    }

    applicationPort := os.Getenv("APPLICATION_PORT")
    if applicationPort == "" {
        applicationPort = "5000"
    }

    // Initialize Redis client
    rdb := redis.NewClient(&redis.Options{
        Addr:     redisHost + ":" + redisPort,
        Password: "",
        DB:       0,
    })

    // Initialize router
    r := gin.New()
    r.Use(logrusMiddleware)

    r.GET("/favorites", func(c *gin.Context) {
        // artificial sleep for delayTime
        time.Sleep(time.Duration(delayTime) * time.Millisecond)

        userID := c.Query("user_id")

        contextLogger(c).Infof("Getting favorites for user %q", userID)

        favorites, err := rdb.SMembers(c.Request.Context(), userID).Result()
        if err != nil {
            contextLogger(c).Error("Failed to get favorites for user %q", userID)
            c.String(http.StatusInternalServerError, "Failed to get favorites")
            return
        }

        contextLogger(c).Infof("User %q has favorites %q", userID, favorites)

        c.JSON(http.StatusOK, gin.H{
            "favorites": favorites,
        })
    })

    // Start server
    logger.Infof("App startup")
    log.Fatal(http.ListenAndServe(":"+applicationPort, r))
    logger.Infof("App stopped")
}

Step-by-step guide

Step 0. Log in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Install and initialize OpenTelemetry

As a first step, we’ll need to add some additional packages to our application.

import (
      "github.com/go-redis/redis/extra/redisotel/v8"
      "go.opentelemetry.io/otel"
      "go.opentelemetry.io/otel/attribute"
      "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

    "go.opentelemetry.io/otel/propagation"

    "google.golang.org/grpc/credentials"
    "crypto/tls"

      sdktrace "go.opentelemetry.io/otel/sdk/trace"

    "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"

    "go.opentelemetry.io/otel/trace"
    "go.opentelemetry.io/otel/codes"
)

This code imports necessary OpenTelemetry packages, including those for tracing, exporting, and instrumenting specific libraries like Redis.

Next we read the "OTEL_EXPORTER_OTLP_ENDPOINT" variable and initialize the exporter.

var (
    collectorURL = os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
)
var tracer trace.Tracer


func initTracer() func(context.Context) error {
    tracer = otel.Tracer("go-favorite-otel-manual")

    // remove https:// from the collector URL if it exists
    collectorURL = strings.Replace(collectorURL, "https://", "", 1)
    secretToken := os.Getenv("ELASTIC_APM_SECRET_TOKEN")
    if secretToken == "" {
        log.Fatal("ELASTIC_APM_SECRET_TOKEN is required")
    }

    secureOption := otlptracegrpc.WithInsecure()
    exporter, err := otlptrace.New(
        context.Background(),
        otlptracegrpc.NewClient(
            secureOption,
            otlptracegrpc.WithEndpoint(collectorURL),
            otlptracegrpc.WithHeaders(map[string]string{
                "Authorization": "Bearer " + secretToken,
            }),
            otlptracegrpc.WithTLSCredentials(credentials.NewTLS(&tls.Config{})),
        ),
    )

    if err != nil {
        log.Fatal(err)
    }

    otel.SetTracerProvider(
        sdktrace.NewTracerProvider(
            sdktrace.WithSampler(sdktrace.AlwaysSample()),
            sdktrace.WithBatcher(exporter),
        ),
    )
    otel.SetTextMapPropagator(
        propagation.NewCompositeTextMapPropagator(
            propagation.Baggage{},
            propagation.TraceContext{},
        ),
    )
    return exporter.Shutdown
}

For instrumenting connections to Redis, we will add a tracing hook to it, and in order to instrument Gin, we will add the OTel middleware. This will automatically capture all interactions with our application, since Gin will be fully instrumented. In addition, all outgoing connections to Redis will also be instrumented.

// Initialize Redis client
    rdb := redis.NewClient(&redis.Options{
        Addr:     redisHost + ":" + redisPort,
        Password: "",
        DB:       0,
    })
    rdb.AddHook(redisotel.NewTracingHook())
    // Initialize router
    r := gin.New()
    r.Use(logrusMiddleware)
    r.Use(otelgin.Middleware("go-favorite-otel-manual"))

Adding custom spans
Now that we have everything added and initialized, we can add custom spans.

If we want to have additional instrumentation for a part of our app, we simply start a custom span and then defer ending the span.

// start otel span
ctx := c.Request.Context()
ctx, span := tracer.Start(ctx, "add_favorite_movies")
defer span.End()

For comparison, this is the instrumented code of our sample application. You can find the full source code in GitHub.

package main

import (
    "log"
    "net/http"
    "os"
    "time"
    "context"

    "github.com/go-redis/redis/v8"
    "github.com/go-redis/redis/extra/redisotel/v8"


    "github.com/sirupsen/logrus"

    "github.com/gin-gonic/gin"

  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/attribute"
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

    "go.opentelemetry.io/otel/propagation"

    "google.golang.org/grpc/credentials"
    "crypto/tls"

  sdktrace "go.opentelemetry.io/otel/sdk/trace"

    "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"

    "go.opentelemetry.io/otel/trace"

    "strings"
    "strconv"
    "math/rand"
    "go.opentelemetry.io/otel/codes"

)

var tracer trace.Tracer

func initTracer() func(context.Context) error {
    tracer = otel.Tracer("go-favorite-otel-manual")

    collectorURL = strings.Replace(collectorURL, "https://", "", 1)

    secureOption := otlptracegrpc.WithInsecure()

    // split otlpHeaders by comma and convert to map
    headers := make(map[string]string)
    for _, header := range strings.Split(otlpHeaders, ",") {
        headerParts := strings.Split(header, "=")

        if len(headerParts) == 2 {
            headers[headerParts[0]] = headerParts[1]
        }
    }

    exporter, err := otlptrace.New(
        context.Background(),
        otlptracegrpc.NewClient(
            secureOption,
            otlptracegrpc.WithEndpoint(collectorURL),
            otlptracegrpc.WithHeaders(headers),
            otlptracegrpc.WithTLSCredentials(credentials.NewTLS(&tls.Config{})),
        ),
    )

    if err != nil {
        log.Fatal(err)
    }

    otel.SetTracerProvider(
        sdktrace.NewTracerProvider(
            sdktrace.WithSampler(sdktrace.AlwaysSample()),
            sdktrace.WithBatcher(exporter),
            //sdktrace.WithResource(resources),
        ),
    )
    otel.SetTextMapPropagator(
        propagation.NewCompositeTextMapPropagator(
            propagation.Baggage{},
            propagation.TraceContext{},
        ),
    )
    return exporter.Shutdown
}

var (
  collectorURL = os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
    otlpHeaders = os.Getenv("OTEL_EXPORTER_OTLP_HEADERS")
)


var logger = &logrus.Logger{
    Out:   os.Stderr,
    Hooks: make(logrus.LevelHooks),
    Level: logrus.InfoLevel,
    Formatter: &logrus.JSONFormatter{
        FieldMap: logrus.FieldMap{
            logrus.FieldKeyTime:  "@timestamp",
            logrus.FieldKeyLevel: "log.level",
            logrus.FieldKeyMsg:   "message",
            logrus.FieldKeyFunc:  "function.name", // non-ECS
        },
        TimestampFormat: time.RFC3339Nano,
    },
}

func main() {
    cleanup := initTracer()
  defer cleanup(context.Background())

    redisHost := os.Getenv("REDIS_HOST")
    if redisHost == "" {
        redisHost = "localhost"
    }

    redisPort := os.Getenv("REDIS_PORT")
    if redisPort == "" {
        redisPort = "6379"
    }

    applicationPort := os.Getenv("APPLICATION_PORT")
    if applicationPort == "" {
        applicationPort = "5000"
    }

    // Initialize Redis client
    rdb := redis.NewClient(&redis.Options{
        Addr:     redisHost + ":" + redisPort,
        Password: "",
        DB:       0,
    })
    rdb.AddHook(redisotel.NewTracingHook())


    // Initialize router
    r := gin.New()
    r.Use(logrusMiddleware)
    r.Use(otelgin.Middleware("go-favorite-otel-manual"))


    // Define routes
    r.GET("/", func(c *gin.Context) {
        contextLogger(c).Infof("Main request successful")
        c.String(http.StatusOK, "Hello World!")
    })

    r.GET("/favorites", func(c *gin.Context) {
        // artificial sleep for delayTime
        time.Sleep(time.Duration(delayTime) * time.Millisecond)

        userID := c.Query("user_id")

        contextLogger(c).Infof("Getting favorites for user %q", userID)

        favorites, err := rdb.SMembers(c.Request.Context(), userID).Result()
        if err != nil {
            contextLogger(c).Error("Failed to get favorites for user %q", userID)
            c.String(http.StatusInternalServerError, "Failed to get favorites")
            return
        }

        contextLogger(c).Infof("User %q has favorites %q", userID, favorites)

        c.JSON(http.StatusOK, gin.H{
            "favorites": favorites,
        })
    })

    // Start server
    logger.Infof("App startup")
    log.Fatal(http.ListenAndServe(":"+applicationPort, r))
    logger.Infof("App stopped")
}

Step 2. Running the Docker image with environment variables

As specified in the OTEL documentation, we will use environment variables and pass in the configuration values that are found in your APM Agent’s configuration section.

Because Elastic accepts OTLP natively, we just need to provide the Endpoint and authentication where the OTEL Exporter needs to send the data, as well as some other environment variables.

Where to get these variables in Elastic Cloud and Kibana ^®
You can copy the endpoints and token from Kibana under the path /app/home#/tutorial/apm.

You will need to copy the OTEL_EXPORTER_OTLP_ENDPOINT as well as the OTEL_EXPORTER_OTLP_HEADERS.

Build the image

docker build -t  go-otel-manual-image .

Run the image

docker run \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="<REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT>" \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <REPLACE WITH TOKEN>" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production,service.name=go-favorite-otel-manual" \
       -p 5000:5000 \
       go-otel-manual-image

You can now issue a few requests in order to generate trace data. Note that these requests are expected to return an error, as this service relies on a connection to Redis that you don’t currently have running. As mentioned before, you can find a more complete example using Docker compose here.

curl localhost:500/favorites
# or alternatively issue a request every second

while true; do curl "localhost:5000/favorites"; sleep 1; done;

How do the traces show up in Elastic?

Now that the service is instrumented, you should see the following output in Elastic APM when looking at the transactions section of your Node.js service:

Conclusion

In this blog, we discussed the following:

• How to manually instrument Go with OpenTelemetry
• How to properly initialize OpenTelemetry and add a custom span
• How to easily set the OTLP ENDPOINT and OTLP HEADERS with Elastic without the need for a collector

Hopefully, this provides an easy-to-understand walk-through of instrumenting Go with OpenTelemetry and how easy it is to send traces into Elastic.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

DevOps engineers continuously optimize software delivery, while SRE teams act as the stewards of application reliability, scalability, and top-tier performance. The challenge? These teams require a cutting-edge observability solution, one that encompasses full-stack insights, empowering them to rapidly manage, monitor, and rectify potential disruptions before they culminate into operational challenges.

Observability in our modern distributed software ecosystem goes beyond mere monitoring — it demands limitless data collection, precision in processing, and the correlation of this data into actionable insights. However, the road to achieving this holistic view is paved with obstacles, from navigating version incompatibilities to wrestling with restrictive proprietary code.

Enter OpenTelemetry (OTel), with the following benefits for those who adopt it:

• Escape vendor constraints with OTel, freeing yourself from vendor lock-in and ensuring top-notch observability.
• See the harmony of unified logs, metrics, and traces come together to provide a complete system view.
• Improve your application oversight through richer and enhanced instrumentations.
• Embrace the benefits of backward compatibility to protect your prior instrumentation investments.
• Embark on the OpenTelemetry journey with an easy learning curve, simplifying onboarding and scalability.
• Rely on a proven, future-ready standard to boost your confidence in every investment.
• Explore manual instrumentation, enabling customized data collection to fit your unique needs.
• Ensure monitoring consistency across layers with a standardized observability data framework.
• Decouple development from operations, driving peak efficiency for both.

In this post, we will dive into the methodology to instrument a .NET application manually using Docker.

What's covered?

• Instrumenting the .NET application manually
• Creating a Docker image for a .NET application with the OpenTelemetry instrumentation baked in
• Installing and running the OpenTelemetry .NET Profiler for automatic instrumentation

Prerequisites

• An understanding of Docker and .NET
• Elastic Cloud
• Docker installed on your machine (we recommend docker desktop)

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Step-by-step guide

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Getting started

In our demonstration, we will manually instrument a .NET Core application - Login. This application simulates a simple user login service. In this example, we are only looking at Tracing since the OpenTelemetry logging instrumentation is currently at mixed maturity, as mentioned here.

The application has the following files:

1. Program.cs

2. Startup.cs

3. Telemetry.cs

4. LoginController.cs

Step 2. Instrumenting the application

When it comes to OpenTelemetry, the .NET ecosystem presents some unique aspects. While OpenTelemetry offers its API, .NET leverages its native System.Diagnostics API to implement OpenTelemetry's Tracing API. The pre-existing constructs such as ActivitySource and Activity are aptly repurposed to comply with OpenTelemetry.

That said, understanding the OpenTelemetry API and its terminology remains crucial for .NET developers. It's pivotal in gaining full command over instrumenting your applications, and as we've seen, it also extends to understanding elements of the System.Diagnostics API.

For those who might lean toward using the original OpenTelemetry APIs over the System.Diagnostics ones, there is also a way. OpenTelemetry provides an API shim for tracing that you can use. It enables developers to switch to OpenTelemetry APIs, and you can find more details about it in the OpenTelemetry API Shim documentation.

By integrating such practices into your .NET application, you can take full advantage of the powerful features OpenTelemetry provides, irrespective of whether you're using OpenTelemetry's API or the System.Diagnostics API.

In this blog, we are sticking to the default method and using the Activity convention which the System.Diagnostics API dictates.

To manually instrument a .NET application, you need to make changes in each of these files. Let's take a look at these changes one by one.

Program.cs

This is the entry point for our application. Here, we create an instance of IHostBuilder with default configurations. Notice how we set up a console logger with Serilog.

public static void Main(string[] args)
{
    Log.Logger = new LoggerConfiguration().WriteTo.Console().CreateLogger();
    CreateHostBuilder(args).Build().Run();
}

Startup.cs

In the Startup.cs file, we use the ConfigureServices method to add the OpenTelemetry Tracing.

public void ConfigureServices(IServiceCollection services)
{
    services.AddOpenTelemetry().WithTracing(builder => builder.AddOtlpExporter()
        .AddSource("Login")
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
        .ConfigureResource(resource =>
            resource.AddService(
                serviceName: "Login"))
    );
    services.AddControllers();
}

The WithTracing method enables tracing in OpenTelemetry. We add the OTLP (OpenTelemetry Protocol) exporter, which is a general-purpose telemetry data delivery protocol. We also add the AspNetCoreInstrumentation, which will automatically collect traces from our application. This is a critically important step that is not mentioned in the OpenTelemetry docs. Without adding this method, the instrumentation was not working for me for the Login application.

Telemetry.cs

This file contains the definition of our ActivitySource. The ActivitySource represents the source of the telemetry activities. It is named after the service name for your application, and this name can come from a configuration file, constants file, etc. We can use this ActivitySource to start activities.

using System.Diagnostics;

public static class Telemetry
{
    //...

    // Name it after the service name for your app.
    // It can come from a config file, constants file, etc.
    public static readonly ActivitySource LoginActivitySource = new("Login");

    //...
}

In our case, we've created an ActivitySource named Login. In our LoginController.cs, we use this LoginActivitySource to start a new activity when we begin our operations.

using (Activity activity = Telemetry.LoginActivitySource.StartActivity("SomeWork"))
{
    // Perform operations here
}

This piece of code starts a new activity named SomeWork , performs some operations (in this case, generating a random user and logging them in), and then ends the activity. These activities are traced and can be analyzed later to understand the performance of the operations.

This ActivitySource is fundamental to OpenTelemetry's manual instrumentation. It represents the source of the activities and provides a way to start and stop activities.

LoginController.cs

In the LoginController.cs file, we are tracing the operations performed by the GET and POST methods. We start a new activity, SomeWork , before we begin our operations and dispose of it once we're done.

using (Activity activity = Telemetry.LoginActivitySource.StartActivity("SomeWork"))
{
    var user = GenerateRandomUserResponse();
    Log.Information("User logged in: {UserName}", user);
    return user;
}

This will track the time taken by these operations and send this data to any configured telemetry backend via the OTLP exporter.

Step 3. Base image setup

Now that we have our application source code created and instrumented, it’s time to create a Dockerfile to build and run our .NET Login service.

Start with the .NET runtime image for the base layer of our Dockerfile:

FROM ${ARCH}mcr.microsoft.com/dotnet/aspnet:7.0. AS base
WORKDIR /app
EXPOSE 8000

Here, we're setting up the application's runtime environment.

Step 4. Building the .NET application

This feature of Docker is just the best. Here, we compile our .NET application. We'll use the SDK image. In the bad old days, we used to build on a different platform and then put the compiled code into the Docker container. This way, we are much more confident our build will replicate from a developers desktop and into production by using Docker all the way through.

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-preview AS build
ARG TARGETPLATFORM

WORKDIR /src
COPY ["login.csproj", "./"]
RUN dotnet restore "./login.csproj"
COPY . .
WORKDIR "/src/."
RUN dotnet build "login.csproj" -c Release -o /app/build

This section ensures that our .NET code is properly restored and compiled.

Step 5. Publishing the application

Once built, we'll publish the app:

FROM build AS publish
RUN dotnet publish "login.csproj" -c Release -o /app/publish

Step 6. Preparing the final image

Now, let's set up the final runtime image:

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .

Step 7. Entry point setup

Lastly, set the Docker image's entry point to both source the OpenTelemetry instrumentation, which sets up the Environment variables required to bootstrap the .NET Profiler, and then we start our .NET application:

ENTRYPOINT ["/bin/bash", "-c", "dotnet login.dll"]

Step 8. Running the Docker image with environment variables

To build and run the Docker image, you'd typically follow these steps:

Build the Docker image

First, you'd want to build the Docker image from your Dockerfile. Let's assume the Dockerfile is in the current directory, and you'd like to name/tag your image dotnet-login-otel-image.

docker build -t dotnet-login-otel-image .

Run the Docker image

After building the image, you'd run it with the specified environment variables. For this, the docker run command is used with the -e flag for each environment variable.

docker run \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ${ELASTIC_APM_SECRET_TOKEN}" \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="${ELASTIC_APM_SERVER_URL}" \
       -e OTEL_METRICS_EXPORTER="otlp" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production" \
       -e OTEL_SERVICE_NAME="dotnet-login-otel-manual" \
       -e OTEL_TRACES_EXPORTER="otlp" \
       dotnet-login-otel-image

Make sure that ${ELASTIC_APM_SECRET_TOKEN} and ${ELASTIC_APM_SERVER_URL} are set in your shell environment, replace them with their actual values from the cloud as shown below.

Getting Elastic Cloud variables
You can copy the endpoints and token from Kibana under the path /app/home#/tutorial/apm.

You can also use an environment file with docker run --env-file to make the command less verbose if you have multiple environment variables.

Once you have this up and running, you can ping the endpoint for your instrumented service (in our case, this is /login), and you should see the app appear in Elastic APM, as shown below:

It will begin by tracking throughput and latency critical metrics for SREs to pay attention to.

Digging in, we can see an overview of all our Transactions.

And look at specific transactions, including the “SomeWork” activity/span we created in the code above:

There is clearly an outlier here, where one transaction took over 20ms. This is likely to be due to the CLR warming up.

Wrapping up

With the code here instrumented and the Dockerfile bootstrapping the application, you've transformed your simple .NET application into one that's instrumented with OpenTelemetry. This will aid greatly in understanding application performance, tracing errors, and gaining insights into how users interact with your software.

Remember, observability is a crucial aspect of modern application development, especially in distributed systems. With tools like OpenTelemetry, understanding complex systems becomes a tad bit easier.

In this blog, we discussed the following:

• How to manually instrument .NET with OpenTelemetry.
• Using standard commands in a Docker file, our instrumented application was built and started.
• Using OpenTelemetry and its support for multiple languages, DevOps and SRE teams can instrument their applications with ease, gaining immediate insights into the health of the entire application stack and reducing mean time to resolution (MTTR).

Since Elastic can support a mix of methods for ingesting data whether it be using auto-instrumentation of open-source OpenTelemetry or manual instrumentation with its native APM agents, you can plan your migration to OTel by focusing on a few applications first and then using OpenTelemety across your applications later on in a manner that best fits your business needs.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

DevOps engineers continuously optimize software delivery, while SRE teams act as the stewards of application reliability, scalability, and top-tier performance. The challenge? These teams require a cutting-edge observability solution, one that encompasses full-stack insights, empowering them to rapidly manage, monitor, and rectify potential disruptions before they culminate into operational challenges.

Observability in our modern distributed software ecosystem goes beyond mere monitoring — it demands limitless data collection, precision in processing, and the correlation of this data into actionable insights. However, the road to achieving this holistic view is paved with obstacles, from navigating version incompatibilities to wrestling with restrictive proprietary code.

Enter OpenTelemetry (OTel), with the following benefits for those who adopt it:

• Escape vendor constraints with OTel, freeing yourself from vendor lock-in and ensuring top-notch observability.
• See the harmony of unified logs, metrics, and traces come together to provide a complete system view.
• Improve your application oversight through richer and enhanced instrumentations.
• Embrace the benefits of backward compatibility to protect your prior instrumentation investments.
• Embark on the OpenTelemetry journey with an easy learning curve, simplifying onboarding and scalability.
• Rely on a proven, future-ready standard to boost your confidence in every investment.
• Explore manual instrumentation, enabling customized data collection to fit your unique needs.
• Ensure monitoring consistency across layers with a standardized observability data framework.
• Decouple development from operations, driving peak efficiency for both.

Given this context, OpenTelemetry emerges as an unmatched observability solution for cloud-native software, seamlessly enabling tracing, monitoring, and debugging. One of its strengths is the ability to auto-instrument applications, allowing developers the luxury of collecting invaluable telemetry without delving into code modifications.

In this post, we will dive into the methodology to instrument a .NET application using Docker, blending the best of both worlds: powerful observability without the code hassles.

What's covered?

• How APM works with .NET using CLR Profiler functionality
• Creating a Docker image for a .NET application with the OpenTelemetry instrumentation baked in
• Installing and running the OpenTelemetry .NET Profiler for automatic instrumentation

How APM works with .NET using CLR Profiler functionality

Before we delve into the details, let's clear up some confusion around .NET Profilers and CPU Profilers like Elastic^®’s Universal Profiling tool — we don’t want to get these two things mixed up, as they have very different purposes.

When discussing profiling tools, especially in the context of .NET, it's not uncommon to encounter confusion between a ".NET profiler" and a "CPU profiler." Though both are used to diagnose and optimize applications, they serve different primary purposes and operate at different levels. Let's clarify the distinction:

.NET Profiler

1. Scope: Specifically targets .NET applications. It is designed to work with the .NET runtime (i.e., the Common Language Runtime (CLR)).

2. Functionality:

3. Use cases:

CPU Profiler

1. Scope: More general than a .NET profiler. It can profile any application, irrespective of the language or runtime, as long as it runs on the CPU being profiled.

2. Functionality:

3. Use cases:

While both .NET profilers and CPU profilers aid in optimizing and diagnosing application performance, their approach and depth differ. A .NET profiler offers deep insights specifically into the .NET ecosystem, allowing for fine-grained analysis and instrumentation. In contrast, a CPU profiler provides a broader view, focusing on CPU usage patterns across any application, regardless of its development platform.

It's worth noting that for comprehensive profiling of a .NET application, you might use both: the .NET profiler to understand code-level behaviors specific to .NET and the CPU profiler to get an overview of CPU resource utilization.

Now that we've cleared that up, let's focus on the .NET Profiler, which we are discussing in this blog for automatic instrumentation of .NET applications. First, let's familiarize ourselves with some foundational concepts and terminologies relevant to a .NET Profiler:

• CLR (Common Language Runtime): CLR is a core component of the .NET framework, acting as the execution engine for .NET apps. It provides key services like memory management, exception handling, and type safety.
• Profiler API:.NET provides a set of APIs for profiling applications. These APIs let tools and developers monitor or manipulate .NET applications during runtime.
• IL (Intermediate Language): After compiling, .NET source code turns into IL, a low-level, platform-agnostic representation. This IL code is then compiled just-in-time (JIT) into machine code by the CLR during application execution.
• JIT compilation: JIT stands for just-in-time. In .NET, the CLR compiles IL to native code just before its execution.

Now, let's explore how automatic instrumentation works using CLR Profiler.

Automatic instrumentation in .NET, much like Java's bytecode instrumentation, revolves around modifying the behavior of your application's methods during runtime, without changing the actual source code.

Here’s a step-by-step breakdown:

1. Attach the profiler: When launching your .NET application, you'll have to specify to load the profiler. The CLR checks for the presence of a profiler by reading environment variables. If it finds one, the CLR initializes the profiler before any user code is executed.

2. Use Profiler API to monitor events: The Profiler API allows a profiler to monitor various events. For instance, method JIT compilation events can be tracked. When a method is about to be JIT compiled, the profiler gets notified.

3. Manipulate IL code: Upon getting notified of a JIT compilation, the profiler can manipulate the IL code of the method. Using the Profiler API, the profiler can insert, delete, or replace IL instructions. This is analogous to how Java agents modify bytecode. For example, if you want to measure a method's execution time, you'd modify the IL to insert calls to start and stop a timer at the beginning and end of the method, respectively.

4. Execution of transformed code: Once the IL has been modified, the JIT compiler will translate it into machine code. The application will then execute this machine code, which includes the additions made by the profiler.

5. Gather and report data: The added instrumentation can collect various data, such as method execution times or call counts. This data can then be relayed to an application performance management (APM) tool, which can provide insights, visualizations, and alerts based on the data.

In essence, automatic instrumentation with CLR Profiler is about modifying the behavior of your .NET methods at runtime. This is invaluable for monitoring, diagnosing, and fine-tuning the performance of .NET applications without intruding on the application's actual source code.

Prerequisites

• A basic understanding of Docker and .NET
• Elastic Cloud
• Docker installed on your machine (we recommend docker desktop)

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Step-by-step guide

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Base image setup

Start with the .NET runtime image for the base layer of our Dockerfile:

FROM ${ARCH}mcr.microsoft.com/dotnet/aspnet:7.0. AS base
WORKDIR /app
EXPOSE 8000

Here, we're setting up the application's runtime environment.

Step 2. Building the .NET application

This feature of Docker is just the best. Here, we compile our .NET application using the SDK image. In the bad old days, we used to build on a different platform and then put the compiled code into the Docker container. This way, we are much more confident our build will replicate from a developer’s desktop and into production by using Docker all the way through.

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-preview AS build
ARG TARGETPLATFORM

WORKDIR /src
COPY ["login.csproj", "./"]
RUN dotnet restore "./login.csproj"
COPY . .
WORKDIR "/src/."
RUN dotnet build "login.csproj" -c Release -o /app/build

This section ensures that our .NET code is properly restored and compiled.

Step 3. Publishing the application

Once built, we'll publish the app:

FROM build AS publish
RUN dotnet publish "login.csproj" -c Release -o /app/publish

Step 4. Preparing the final image

Now, let's set up the final runtime image:

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish

Step 5. Installing OpenTelemetry

We'll install dependencies and download the OpenTelemetry auto-instrumentation script:

RUN apt-get update && apt-get install -y zip curl
RUN mkdir /otel
RUN curl -L -o /otel/otel-dotnet-install.sh https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/download/v0.7.0/otel-dotnet-auto-install.sh
RUN chmod +x /otel/otel-dotnet-install.sh

Step 6. Configure OpenTelemetry

Designate where OpenTelemetry should reside and execute the installation script. Note that the ENV OTEL_DOTNET_AUTO_HOME is required as the script looks for it:

ENV OTEL_DOTNET_AUTO_HOME=/otel
RUN /bin/bash /otel/otel-dotnet-install.sh

Step 7. Additional configuration

Make sure the auto-instrumentation and platform detection scripts are executable and run the platform detection script.

COPY platform-detection.sh /otel/
RUN chmod +x /otel/instrument.sh
RUN chmod +x /otel/platform-detection.sh && /otel/platform-detection.sh

This platform detection script will check if the Docker build is for ARM64 and implement a workaround to get the OpenTelemetry instrumentation to work on MacOS. If you happen to be running locally on MacOS M1 or M2 processors, you will be grateful for this script.

Step 8. Entry point setup

Lastly, set the Docker image's entry point to both source the OpenTelemetry instrumentation, which sets up the environment variables required to bootstrap the .NET Profiler, and then we start our .NET application:

ENTRYPOINT ["/bin/bash", "-c", "source /otel/instrument.sh && dotnet login.dll"]

Step 9. Running the Docker image with environment variables

To build and run the Docker image, you'd typically follow these steps:

Build the Docker image

First, you'd want to build the Docker image from your Dockerfile. Let's assume the Dockerfile is in the current directory, and you'd like to name/tag your image dotnet-login-otel-image.

docker build -t dotnet-login-otel-image .

Run the Docker image

After building the image, you'd run it with the specified environment variables. For this, the docker run command is used with the -e flag for each environment variable.

docker run \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ${ELASTIC_APM_SECRET_TOKEN}" \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="${ELASTIC_APM_SERVER_URL}" \
       -e OTEL_METRICS_EXPORTER="otlp" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production" \
       -e OTEL_SERVICE_NAME="dotnet-login-otel-auto" \
       -e OTEL_TRACES_EXPORTER="otlp" \
       dotnet-login-otel-image

Make sure that ${ELASTIC_APM_SECRET_TOKEN} and ${ELASTIC_APM_SERVER_URL} are set in your shell environment, and replace them with their actual values from the cloud as shown below.
Getting Elastic Cloud variables

You can copy the endpoints and token from Kibana^® under the path /app/home#/tutorial/apm.

You can also use an environment file with docker run --env-file to make the command less verbose if you have multiple environment variables.

Once you have this up and running, you can ping the endpoint for your instrumented service (in our case, this is /login), and you should see the app appear in Elastic APM, as shown below:

It will begin by tracking throughput and latency critical metrics for SREs to pay attention to.

Digging in, we can see an overview of all our Transactions.

And look at specific transactions:

There is clearly an outlier here, where one transaction took over 200ms. This is likely to be due to the .NET CLR warming up. Click on Logs , and we see that logs are also brought over. The OTel Agent will automatically bring in logs and correlate them with traces for you:

Wrapping up

With this Dockerfile, you've transformed your simple .NET application into one that's automatically instrumented with OpenTelemetry. This will aid greatly in understanding application performance, tracing errors, and gaining insights into how users interact with your software.

Remember, observability is a crucial aspect of modern application development, especially in distributed systems. With tools like OpenTelemetry, understanding complex systems becomes a tad bit easier.

In this blog, we discussed the following:

• How to auto-instrument .NET with OpenTelemetry.
• Using standard commands in a Docker file, auto-instrumentation was done efficiently and without adding code in multiple places enabling manageability.
• Using OpenTelemetry and its support for multiple languages, DevOps and SRE teams can auto-instrument their applications with ease gaining immediate insights into the health of the entire application stack and reduce mean time to resolution (MTTR).

Since Elastic can support a mix of methods for ingesting data, whether it be using auto-instrumentation of open-source OpenTelemetry or manual instrumentation with its native APM agents, you can plan your migration to OTel by focusing on a few applications first and then using OpenTelemety across your applications later on in a manner that best fits your business needs.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Observability across the entire stack of modern distributed applications requires data collection, processing, and correlation often in the form of dashboards. Ingesting all system data requires installing agents across stacks, frameworks, and providers — a process that can be challenging and time-consuming for teams who have to deal with version changes, compatibility issues, and proprietary code that doesn't scale as systems change.

Thanks to OpenTelemetry (OTel), DevOps and SRE teams now have a standard way to collect and send data that doesn't rely on proprietary code and have a large support community reducing vendor lock-in.

In a previous blog, we also reviewed how to use the OpenTelemetry demo and connect it to Elastic^®, as well as some of Elastic’s capabilities with OpenTelemetry and Kubernetes.

In this blog, we will show how to use manual instrumentation for OpenTelemetry with the Python service of our application called Elastiflix. This approach is slightly more complex than using automatic instrumentation.

The beauty of this is that there is no need for the otel-collector! This setup enables you to slowly and easily migrate an application to OTel with Elastic according to a timeline that best fits your business.

Application, prerequisites, and config

The application that we use for this blog is called Elastiflix, a movie streaming application. It consists of several micro-services written in .NET, NodeJS, Go, and Python.

Before we instrument our sample application, we will first need to understand how Elastic can receive the telemetry data.

All of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Prerequisites

• An Elastic Cloud account — sign up now
• A clone of the Elastiflix demo application, or your own Python application
• Basic understanding of Docker — potentially install Docker Desktop
• Basic understanding of Python

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Before we begin, let’s look at the non-instrumented code first.

This is our simple Python Flask application that can receive a GET request. (This is a portion of the full main.py file.)

from flask import Flask, request
import sys

import logging
import redis
import os
import ecs_logging
import datetime
import random
import time

redis_host = os.environ.get('REDIS_HOST') or 'localhost'
redis_port = os.environ.get('REDIS_PORT') or 6379

application_port = os.environ.get('APPLICATION_PORT') or 5000

app = Flask(__name__)

# Get the Logger
logger = logging.getLogger("app")
logger.setLevel(logging.DEBUG)

# Add an ECS formatter to the Handler
handler = logging.StreamHandler()
handler.setFormatter(ecs_logging.StdlibFormatter())
logger.addHandler(handler)
logging.getLogger('werkzeug').setLevel(logging.ERROR)
logging.getLogger('werkzeug').addHandler(handler)

r = redis.Redis(host=redis_host, port=redis_port, decode_responses=True)

@app.route('/favorites', methods=['GET'])
def get_favorite_movies():
    user_id = str(request.args.get('user_id'))

    logger.info('Getting favorites for user ' + user_id, extra={
        "event.dataset": "favorite.log",
        "user.id": request.args.get('user_id')
    })

    favorites = r.smembers(user_id)

    # convert to list
    favorites = list(favorites)
    logger.info('User ' + user_id + ' has favorites: ' + str(favorites), extra={
        "event.dataset": "favorite.log",
        "user.id": user_id
    })
    return { "favorites": favorites}

logger.info('App startup')
app.run(host='0.0.0.0', port=application_port)
logger.info('App Stopped')

Step-by-step guide

Step 0. Log in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Install and initialize OpenTelemetry

As a first step, we’ll need to add some additional libraries to our application.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.redis import RedisInstrumentor
from opentelemetry.sdk.resources import Resource

This code imports necessary OpenTelemetry libraries, including those for tracing, exporting, and instrumenting specific libraries like Flask, Requests, and Redis.

Next we read the variables:

OTEL_EXPORTER_OTLP_HEADERS
OTEL_EXPORTER_OTLP_ENDPOINT

And then initialize the exporter.

otel_exporter_otlp_headers = os.environ.get('OTEL_EXPORTER_OTLP_HEADERS')

otel_exporter_otlp_endpoint = os.environ.get('OTEL_EXPORTER_OTLP_ENDPOINT')

exporter = OTLPSpanExporter(endpoint=otel_exporter_otlp_endpoint, headers=otel_exporter_otlp_headers)

In order to pass additional parameters to OpenTelemetry, we will read the OTEL_RESOURCE_ATTRIBUTES variable and convert it into an object.

resource_attributes = os.environ.get('OTEL_RESOURCE_ATTRIBUTES') or 'service.version=1.0,deployment.environment=production'
key_value_pairs = resource_attributes.split(',')
result_dict = {}

for pair in key_value_pairs:
    key, value = pair.split('=')
    result_dict[key] = value

Next, we will then use these parameters to populate the resources configuration.

resourceAttributes = {
     "service.name": otel_service_name,
     "service.version": result_dict['service.version'],
     "deployment.environment": result_dict['deployment.environment']
}

resource = Resource.create(resourceAttributes)

We then set up the trace provider using the previously created resource. The trace provider will allow us to create spans later after getting a tracer instance from it.

Additionally, we specify the use of BatchSPanProcessor. The Span processor is an interface that allows hooks for span start and end method invocations.

In OpenTelemetry, different Span processors are offered. The BatchSPanProcessor batches span and sends them in bulk. Multiple Span processors can be configured to be active at the same time using the MultiSpanProcessor. See OpenTelemetry documentation.

Additionally, we added the resource module. This allows us to specify attributes such as service.name, version, and more. See OpenTelemetry semantic conventions documentation for more details.

provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)

# Sets the global default tracer provider
trace.set_tracer_provider(provider)

# Creates a tracer from the global tracer provider
tracer = trace.get_tracer(otel_service_name)

Finally, because we are using Flask and Redis, we also add the following, which allows us to automatically instrument both Flask and Redis.

Technically you could consider this “cheating.” We are using some parts of the Python auto-instrumentation. However, it’s generally a good approach to resort to using some of the auto-instrumentation modules. This saves you a lot of time, and in addition, it ensures that functionality like distributed tracing will work automatically for any requests you receive or send.

FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()
RedisInstrumentor().instrument()

Step 2. Adding Custom Spans

Now that we have everything added and initialized, we can add custom spans.

If we want to have additional instrumentation for a part of our app, we simply wrap the /favoritesGET function code using Python with:

with tracer.start_as_current_span("add_favorite_movies", set_status_on_exception=True) as span:
        ...

The wrapped code is as follows:

@app.route('/favorites', methods=['GET'])
def get_favorite_movies():
    # add artificial delay if enabled
    if delay_time > 0:
        time.sleep(max(0, random.gauss(delay_time/1000, delay_time/1000/10)))

    with tracer.start_as_current_span("get_favorite_movies") as span:
        user_id = str(request.args.get('user_id'))

        logger.info('Getting favorites for user ' + user_id, extra={
            "event.dataset": "favorite.log",
            "user.id": request.args.get('user_id')
        })

        favorites = r.smembers(user_id)

        # convert to list
        favorites = list(favorites)
        logger.info('User ' + user_id + ' has favorites: ' + str(favorites), extra={
            "event.dataset": "favorite.log",
            "user.id": user_id
        })

Additional code

In addition to modules and span instrumentation, the sample application also checks some environment variables at startup. When sending data to Elastic without an OTel collector, the OTEL_EXPORTER_OTLP_HEADERS variable is required as it contains the authentication. The same is true for OTEL_EXPORTER_OTLP_ENDPOINT, the host where we’ll send the telemetry data.

otel_exporter_otlp_headers = os.environ.get('OTEL_EXPORTER_OTLP_HEADERS')
# fail if secret token not set
if otel_exporter_otlp_headers is None:
    raise Exception('OTEL_EXPORTER_OTLP_HEADERS environment variable not set')


otel_exporter_otlp_endpoint = os.environ.get('OTEL_EXPORTER_OTLP_ENDPOINT')
# fail if server url not set
if otel_exporter_otlp_endpoint is None:
    raise Exception('OTEL_EXPORTER_OTLP_ENDPOINT environment variable not set')
else:
    exporter = OTLPSpanExporter(endpoint=otel_exporter_otlp_endpoint, headers=otel_exporter_otlp_headers)

Final code
For comparison, this is the instrumented code of our sample application. You can find the full source code in GitHub.

from flask import Flask, request
import sys

import logging
import redis
import os
import ecs_logging
import datetime
import random
import time

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

#Using grpc exporter since per the instructions in OTel docs this is needed for any endpoint receiving OTLP.

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.redis import RedisInstrumentor
#from opentelemetry.instrumentation.wsgi import OpenTelemetryMiddleware
from opentelemetry.sdk.resources import Resource

redis_host = os.environ.get('REDIS_HOST') or 'localhost'
redis_port = os.environ.get('REDIS_PORT') or 6379
otel_traces_exporter = os.environ.get('OTEL_TRACES_EXPORTER') or 'otlp'
otel_metrics_exporter = os.environ.get('OTEL_TRACES_EXPORTER') or 'otlp'
environment = os.environ.get('ENVIRONMENT') or 'dev'
otel_service_version = os.environ.get('OTEL_SERVICE_VERSION') or '1.0.0'
resource_attributes = os.environ.get('OTEL_RESOURCE_ATTRIBUTES') or 'service.version=1.0,deployment.environment=production'

otel_exporter_otlp_headers = os.environ.get('OTEL_EXPORTER_OTLP_HEADERS')
# fail if secret token not set
if otel_exporter_otlp_headers is None:
    raise Exception('OTEL_EXPORTER_OTLP_HEADERS environment variable not set')
#else:
#    otel_exporter_otlp_fheaders= f"Authorization=Bearer%20{secret_token}"

otel_exporter_otlp_endpoint = os.environ.get('OTEL_EXPORTER_OTLP_ENDPOINT')
# fail if server url not set
if otel_exporter_otlp_endpoint is None:
    raise Exception('OTEL_EXPORTER_OTLP_ENDPOINT environment variable not set')
else:
    exporter = OTLPSpanExporter(endpoint=otel_exporter_otlp_endpoint, headers=otel_exporter_otlp_headers)


key_value_pairs = resource_attributes.split(',')
result_dict = {}

for pair in key_value_pairs:
    key, value = pair.split('=')
    result_dict[key] = value

resourceAttributes = {
     "service.name": result_dict['service.name'],
     "service.version": result_dict['service.version'],
     "deployment.environment": result_dict['deployment.environment']
#     # Add more attributes as needed
}

resource = Resource.create(resourceAttributes)


provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)

# Sets the global default tracer provider
trace.set_tracer_provider(provider)

# Creates a tracer from the global tracer provider
tracer = trace.get_tracer("favorite")


application_port = os.environ.get('APPLICATION_PORT') or 5000

app = Flask(__name__)


FlaskInstrumentor().instrument_app(app)
#OpenTelemetryMiddleware().instrument()
RequestsInstrumentor().instrument()
RedisInstrumentor().instrument()

#app.wsgi_app = OpenTelemetryMiddleware(app.wsgi_app)

# Get the Logger
logger = logging.getLogger("app")
logger.setLevel(logging.DEBUG)

# Add an ECS formatter to the Handler
handler = logging.StreamHandler()
handler.setFormatter(ecs_logging.StdlibFormatter())
logger.addHandler(handler)
logging.getLogger('werkzeug').setLevel(logging.ERROR)
logging.getLogger('werkzeug').addHandler(handler)

r = redis.Redis(host=redis_host, port=redis_port, decode_responses=True)

@app.route('/favorites', methods=['GET'])
def get_favorite_movies():
    with tracer.start_as_current_span("get_favorite_movies") as span:
        user_id = str(request.args.get('user_id'))

        logger.info('Getting favorites for user ' + user_id, extra={
            "event.dataset": "favorite.log",
            "user.id": request.args.get('user_id')
        })

        favorites = r.smembers(user_id)

        # convert to list
        favorites = list(favorites)
        logger.info('User ' + user_id + ' has favorites: ' + str(favorites), extra={
            "event.dataset": "favorite.log",
            "user.id": user_id
        })
        return { "favorites": favorites}

logger.info('App startup')
app.run(host='0.0.0.0', port=application_port)
logger.info('App Stopped')

Step 3. Running the Docker image with environment variables

As specified in the OTEL documentation, we will use environment variables and pass in the configuration values to enable it to connect with Elastic Observability’s APM server.

Because Elastic accepts OTLP natively, we just need to provide the Endpoint and authentication where the OTEL Exporter needs to send the data, as well as some other environment variables.

Getting Elastic Cloud variables
You can copy the endpoints and token from Kibana^® under the path /app/home#/tutorial/apm.

You will need to copy the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS

Build the image

docker build -t  python-otel-manual-image .

Run the image

docker run \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="<REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT>" \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <REPLACE WITH TOKEN>" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production,service.name=python-favorite-otel-manual" \
       -p 3001:3001 \
       python-otel-manual-image

You can now issue a few requests in order to generate trace data. Note that these requests are expected to return an error, as this service relies on a connection to Redis that you don’t currently have running. As mentioned before, you can find a more complete example using docker-compose here.

curl localhost:500/favorites
# or alternatively issue a request every second

while true; do curl "localhost:5000/favorites"; sleep 1; done;

Step 4. Explore traces, metrics, and logs in Elastic APM

Now that the service is instrumented, you should see the following output in Elastic APM when looking at the transactions section of your Python service:

Notice how this is slightly different from the auto-instrumented version, as we now also have our custom span in this view.

Is it worth it?

This is the million-dollar question. Depending on what level of detail you need, it's potentially necessary to manually instrument. Manual instrumentation lets you add custom spans, custom labels, and metrics where you want or need them. It allows you to get a level of detail that otherwise would not be possible and is oftentimes important for tracking business-specific KPIs.

Your operations, and whether you need to troubleshoot or analyze the performance of specific parts of the code, will dictate when and what to instrument. But it’s helpful to know that you have the option to manually instrument.

If you noticed we didn’t yet instrument metrics, that is another blog. We discussed logs in a previous blog.

Conclusion

In this blog, we discussed the following:

• How to manually instrument Python with OpenTelemetry
• How to properly initialize OpenTelemetry and add a custom span
• How to easily set the OTLP ENDPOINT and OTLP HEADERS with Elastic without the need for a collector

Hopefully, this provides an easy-to-understand walk-through of instrumenting Python with OpenTelemetry and how easy it is to send traces into Elastic.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Observability across the entire stack of modern distributed applications requires data collection, processing, and correlation often in the form of dashboards. Ingesting all system data requires installing agents across stacks, frameworks, and providers — a process that can be challenging and time-consuming for teams who have to deal with version changes, compatibility issues, and proprietary code that doesn't scale as systems change.

Thanks to OpenTelemetry (OTel), DevOps and SRE teams now have a standard way to collect and send data that doesn't rely on proprietary code and have a large support community reducing vendor lock-in.

In a previous blog, we also reviewed how to use the OpenTelemetry demo and connect it to Elastic^®, as well as some of Elastic’s capabilities with OpenTelemetry and Kubernetes.

In this blog, we will show how to use manual instrumentation for OpenTelemetry with the Node.js service of our application called Elastiflix. This approach is slightly more complex than using auto-instrumentation.

The beauty of this is that there is no need for the otel-collector! This setup enables you to slowly and easily migrate an application to OTel with Elastic according to a timeline that best fits your business.

Application, prerequisites, and config

The application that we use for this blog is called Elastiflix, a movie streaming application. It consists of several micro-services written in .NET, NodeJS, Go, and Python.

Before we instrument our sample application, we will first need to understand how Elastic can receive the telemetry data.

All of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Prerequisites

• An Elastic Cloud account — sign up now
• A clone of the Elastiflix demo application, or your own Node.js application
• Basic understanding of Docker — potentially install Docker Desktop
• Basic understanding of Node.js

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

Before we begin, let’s look at the non-instrumented code first.

This is our simple index.js file that can receive a POST request. See the full code here.

const pino = require("pino");
const ecsFormat = require("@elastic/ecs-pino-format"); //
const log = pino({ ...ecsFormat({ convertReqRes: true }) });
const expressPino = require("express-pino-logger")({ logger: log });

var API_ENDPOINT_FAVORITES =
  process.env.API_ENDPOINT_FAVORITES || "127.0.0.1:5000";
API_ENDPOINT_FAVORITES = API_ENDPOINT_FAVORITES.split(",");

const express = require("express");
const cors = require("cors")({ origin: true });
const cookieParser = require("cookie-parser");
const { json } = require("body-parser");

const PORT = process.env.PORT || 3001;

const app = express().use(cookieParser(), cors, json(), expressPino);

const axios = require("axios");

app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use((err, req, res, next) => {
  log.error(err.stack);
  res.status(500).json({ error: err.message, code: err.code });
});

var favorites = {};

app.post("/api/favorites", (req, res) => {
  var randomIndex = Math.floor(Math.random() * API_ENDPOINT_FAVORITES.length);
  if (process.env.THROW_NOT_A_FUNCTION_ERROR == "true" && Math.random() < 0.5) {
    // randomly choose one of the endpoints
    axios
      .post(
        "http://" +
          API_ENDPOINT_FAVORITES[randomIndex] +
          "/favorites?user_id=1",
        req.body
      )
      .then(function (response) {
        favorites = response.data;
        // quiz solution: "42"
        res.jsonn({ favorites: favorites });
      })
      .catch(function (error) {
        res.json({ error: error, favorites: [] });
      });
  } else {
    axios
      .post(
        "http://" +
          API_ENDPOINT_FAVORITES[randomIndex] +
          "/favorites?user_id=1",
        req.body
      )
      .then(function (response) {
        favorites = response.data;
        res.json({ favorites: favorites });
      })
      .catch(function (error) {
        res.json({ error: error, favorites: [] });
      });
  }
});

app.listen(PORT, () => {
  console.log(`Server listening on ${PORT}`);
});

Step-by-step guide

Step 0. Log in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Install and initialize OpenTelemetry

As a first step, we’ll need to add some additional modules to our application.

const opentelemetry = require("@opentelemetry/api");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { BatchSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const { Resource } = require("@opentelemetry/resources");
const {
  SemanticResourceAttributes,
} = require("@opentelemetry/semantic-conventions");

const { registerInstrumentations } = require("@opentelemetry/instrumentation");
const { HttpInstrumentation } = require("@opentelemetry/instrumentation-http");
const {
  ExpressInstrumentation,
} = require("@opentelemetry/instrumentation-express");

We start by creating a collectorOptions object with parameters such as the url and headers for connecting to the Elastic APM Server or OpenTelemetry collector.

const collectorOptions = {
  url: OTEL_EXPORTER_OTLP_ENDPOINT,
  headers: OTEL_EXPORTER_OTLP_HEADERS,
};

In order to pass additional parameters to OpenTelemetry, we will read the OTEL_RESOURCE_ATTRIBUTES variable and convert it into an object.

const envAttributes = process.env.OTEL_RESOURCE_ATTRIBUTES || "";

// Parse the environment variable string into an object
const attributes = envAttributes.split(",").reduce((acc, curr) => {
  const [key, value] = curr.split("=");
  if (key && value) {
    acc[key.trim()] = value.trim();
  }
  return acc;
}, {});

Next we will then use these parameters to populate the resources configuration.

const resource = new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]:
    attributes["service.name"] || "node-server-otel-manual",
  [SemanticResourceAttributes.SERVICE_VERSION]:
    attributes["service.version"] || "1.0.0",
  [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]:
    attributes["deployment.environment"] || "production",
});

We then set up the trace provider using the previously created resource, followed by the exporter which takes the collectorOptions from before. The trace provider will allow us to create spans later.

Additionally, we specify the use of BatchSPanProcessor. The Span processor is an interface that allows hooks for span start and end method invocations.

In OpenTelemetry, different Span processors are offered. The BatchSPanProcessor batches span and sends them in bulk. Multiple Span processors can be configured to be active at the same time using the MultiSpanProcessor. See OpenTelemetry documentation.

Additionally, we added the resource module. This allows us to specify attributes such as service.name, version, and more. See OpenTelemetry semantic conventions documentation for more details.

const tracerProvider = new NodeTracerProvider({
  resource: resource,
});

const exporter = new OTLPTraceExporter(collectorOptions);
tracerProvider.addSpanProcessor(new BatchSpanProcessor(exporter));
tracerProvider.register();

Next, we are going to register some instrumentations. This will automatically instrument Express and HTTP for us. While it’s possible to do this step fully manually as well, it would be complex and a waste of time. This way we can ensure that any incoming and outgoing request is captured properly and that functionality such as distributed tracing works without any additional work.

registerInstrumentations({
  instrumentations: [new HttpInstrumentation(), new ExpressInstrumentation()],
  tracerProvider: tracerProvider,
});

As a last step, we will now get an instance of the tracer that we can use to create custom spans.

const tracer = opentelemetry.trace.getTracer();

Step 2. Adding custom spans

Now that we have the modules added and initialized, we can add custom spans.

Our sample application has a POST request which calls a downstream service. If we want to have additional instrumentation for this part of our app, we simply wrap the function code with:

tracer.startActiveSpan('favorites',   tracer.startActiveSpan('favorites', (span) => {...

The wrapped code is as follows:

app.post("/api/favorites", (req, res, next) => {
  tracer.startActiveSpan("favorites", (span) => {
    axios
      .post(
        "http://" + API_ENDPOINT_FAVORITES + "/favorites?user_id=1",
        req.body
      )
      .then(function (response) {
        favorites = response.data;
        span.end();
        res.jsonn({ favorites: favorites });
      })
      .catch(next);
  });
});

Automatic error handling
For automatic error handling, we are adding a function that we use in Express which captures the exception for any error that happens during runtime.

app.use((err, req, res, next) => {
  log.error(err.stack);
  span = opentelemetry.trace.getActiveSpan();
  span.recordException(error);
  span.end();
  res.status(500).json({ error: err.message, code: err.code });
});

Additional code
n addition to modules and span instrumentation, the sample application also checks some environment variables at startup. When sending data to Elastic without an OTel collector, the OTEL_EXPORTER_OTLP_HEADERS variable is required as it contains the authentication. The same is true for OTEL_EXPORTER_OTLP_ENDPOINT, the host where we’ll send the telemetry data.

const OTEL_EXPORTER_OTLP_HEADERS = process.env.OTEL_EXPORTER_OTLP_HEADERS;
// error if secret token is not set
if (!OTEL_EXPORTER_OTLP_HEADERS) {
  throw new Error("OTEL_EXPORTER_OTLP_HEADERS environment variable is not set");
}

const OTEL_EXPORTER_OTLP_ENDPOINT = process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
// error if server url is not set
if (!OTEL_EXPORTER_OTLP_ENDPOINT) {
  throw new Error(
    "OTEL_EXPORTER_OTLP_ENDPOINT environment variable is not set"
  );
}

Final code
For comparison, this is the instrumented code of our sample application. You can find the full source code in GitHub.

const pino = require("pino");
const ecsFormat = require("@elastic/ecs-pino-format"); //
const log = pino({ ...ecsFormat({ convertReqRes: true }) });
const expressPino = require("express-pino-logger")({ logger: log });

// Add OpenTelemetry packages
const opentelemetry = require("@opentelemetry/api");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { BatchSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const {
  OTLPTraceExporter,
} = require("@opentelemetry/exporter-trace-otlp-grpc");
const { Resource } = require("@opentelemetry/resources");
const {
  SemanticResourceAttributes,
} = require("@opentelemetry/semantic-conventions");

const { registerInstrumentations } = require("@opentelemetry/instrumentation");

// Import OpenTelemetry instrumentations
const { HttpInstrumentation } = require("@opentelemetry/instrumentation-http");
const {
  ExpressInstrumentation,
} = require("@opentelemetry/instrumentation-express");

var API_ENDPOINT_FAVORITES =
  process.env.API_ENDPOINT_FAVORITES || "127.0.0.1:5000";
API_ENDPOINT_FAVORITES = API_ENDPOINT_FAVORITES.split(",");

const OTEL_EXPORTER_OTLP_HEADERS = process.env.OTEL_EXPORTER_OTLP_HEADERS;
// error if secret token is not set
if (!OTEL_EXPORTER_OTLP_HEADERS) {
  throw new Error("OTEL_EXPORTER_OTLP_HEADERS environment variable is not set");
}

const OTEL_EXPORTER_OTLP_ENDPOINT = process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
// error if server url is not set
if (!OTEL_EXPORTER_OTLP_ENDPOINT) {
  throw new Error(
    "OTEL_EXPORTER_OTLP_ENDPOINT environment variable is not set"
  );
}

const collectorOptions = {
  // url is optional and can be omitted - default is http://localhost:4317
  // Unix domain sockets are also supported: 'unix:///path/to/socket.sock'
  url: OTEL_EXPORTER_OTLP_ENDPOINT,
  headers: OTEL_EXPORTER_OTLP_HEADERS,
};

const envAttributes = process.env.OTEL_RESOURCE_ATTRIBUTES || "";

// Parse the environment variable string into an object
const attributes = envAttributes.split(",").reduce((acc, curr) => {
  const [key, value] = curr.split("=");
  if (key && value) {
    acc[key.trim()] = value.trim();
  }
  return acc;
}, {});

// Create and configure the resource object
const resource = new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]:
    attributes["service.name"] || "node-server-otel-manual",
  [SemanticResourceAttributes.SERVICE_VERSION]:
    attributes["service.version"] || "1.0.0",
  [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]:
    attributes["deployment.environment"] || "production",
});

// Create and configure the tracer provider
const tracerProvider = new NodeTracerProvider({
  resource: resource,
});
const exporter = new OTLPTraceExporter(collectorOptions);
tracerProvider.addSpanProcessor(new BatchSpanProcessor(exporter));
tracerProvider.register();

//Register instrumentations
registerInstrumentations({
  instrumentations: [new HttpInstrumentation(), new ExpressInstrumentation()],
  tracerProvider: tracerProvider,
});

const express = require("express");
const cors = require("cors")({ origin: true });
const cookieParser = require("cookie-parser");
const { json } = require("body-parser");

const PORT = process.env.PORT || 3001;

const app = express().use(cookieParser(), cors, json(), expressPino);

const axios = require("axios");

app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use((err, req, res, next) => {
  log.error(err.stack);
  span = opentelemetry.trace.getActiveSpan();
  span.recordException(error);
  span.end();
  res.status(500).json({ error: err.message, code: err.code });
});

const tracer = opentelemetry.trace.getTracer();

var favorites = {};

app.post("/api/favorites", (req, res, next) => {
  tracer.startActiveSpan("favorites", (span) => {
    var randomIndex = Math.floor(Math.random() * API_ENDPOINT_FAVORITES.length);

    if (
      process.env.THROW_NOT_A_FUNCTION_ERROR == "true" &&
      Math.random() < 0.5
    ) {
      // randomly choose one of the endpoints
      axios
        .post(
          "http://" +
            API_ENDPOINT_FAVORITES[randomIndex] +
            "/favorites?user_id=1",
          req.body
        )
        .then(function (response) {
          favorites = response.data;
          // quiz solution: "42"
          span.end();
          res.jsonn({ favorites: favorites });
        })
        .catch(next);
    } else {
      axios
        .post(
          "http://" +
            API_ENDPOINT_FAVORITES[randomIndex] +
            "/favorites?user_id=1",
          req.body
        )
        .then(function (response) {
          favorites = response.data;
          span.end();
          res.json({ favorites: favorites });
        })
        .catch(next);
    }
  });
});

app.listen(PORT, () => {
  log.info(`Server listening on ${PORT}`);
});

Step 3. Running the Docker image with environment variables

We will use environment variables and pass in the configuration values to enable it to connect with Elastic Observability’s APM server.

Because Elastic accepts OTLP natively, we just need to provide the Endpoint and authentication where the OTEL Exporter needs to send the data, as well as some other environment variables.

Getting Elastic Cloud variables
You can copy the endpoints and token from Kibana^® under the path /app/home#/tutorial/apm.

You will need to copy the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS

Build the image

docker build -t  node-otel-manual-image .

Run the image

docker run \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="<REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT>" \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <REPLACE WITH TOKEN>" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production,service.name=node-server-otel-manual" \
       -p 3001:3001 \
       node-otel-manual-image

You can now issue a few requests in order to generate trace data. Note that these requests are expected to return an error, as this service relies on some downstream services that you may not have running on your machine.

curl localhost:3001/api/login
curl localhost:3001/api/favorites

# or alternatively issue a request every second

while true; do curl "localhost:3001/api/favorites"; sleep 1; done;

Step 4. Explore in Elastic APM

Now that the service is instrumented, you should see the following output in Elastic APM when looking at the transactions section of your Node.js service:

Notice how this mirrors the auto-instrumented version.

Is it worth it?

This is the million-dollar question. Depending on what level of detail you need, it's potentially necessary to manually instrument. Manual instrumentation lets you add custom spans, custom labels, and metrics where you want or need them. It allows you to get a level of detail that otherwise would not be possible and is oftentimes important for tracking business-specific KPIs.

Your operations, and whether you need to troubleshoot or analyze the performance of specific parts of the code, will dictate when and what to instrument. But it’s helpful to know that you have the option to manually instrument.

If you noticed we didn’t yet instrument metrics, that is another blog. We discussed logs in a previous blog.

Conclusion

In this blog, we discussed the following:

• How to manually instrument Node.js with OpenTelemetry
• The different modules needed when using Express
• How to properly initialize and instrument span
• How to easily set the OTLP ENDPOINT and OTLP HEADERS from Elastic without the need for a collector

Hopefully, this provides an easy-to-understand walk-through of instrumenting Node.js with OpenTelemetry and how easy it is to send traces into Elastic.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

DevOps engineers continuously optimize software delivery, while SRE teams act as the stewards of application reliability, scalability, and top-tier performance. The challenge? These teams require a cutting-edge observability solution, one that encompasses full-stack insights, empowering them to rapidly manage, monitor, and rectify potential disruptions before they culminate into operational challenges.

Observability in our modern distributed software ecosystem goes beyond mere monitoring—it demands limitless data collection, precision in processing, and the correlation of this data into actionable insights. However, the road to achieving this holistic view is paved with obstacles: from navigating version incompatibilities to wrestling with restrictive proprietary code.

Enter OpenTelemetry (OTel), with the following benefits for those who adopt it:

• Escape vendor constraints with OTel, freeing yourself from vendor lock-in and ensuring top-notch observability.
• See the harmony of unified logs, metrics, and traces come together to provide a complete system view.
• Improve your application oversight through richer and enhanced instrumentations.
• Embrace the benefits of backward compatibility to protect your prior instrumentation investments.
• Embark on the OpenTelemetry journey with an easy learning curve, simplifying onboarding and scalability.
• Rely on a proven, future-ready standard to boost your confidence in every investment.

In this blog, we will explore how you can use manual instrumentation in your Java application using Docker, without the need to refactor any part of your application code. We will use an application called Elastiflix. This approach is slightly more complex than using automatic instrumentation.

The beauty of this is that there is no need for the otel-collector! This setup enables you to slowly and easily migrate an application to OTel with Elastic according to a timeline that best fits your business.

Application, prerequisites, and config

The application that we use for this blog is called Elastiflix, a movie streaming application. It consists of several micro-services written in .NET, NodeJS, Go, and Python.

Before we instrument our sample application, we will first need to understand how Elastic can receive the telemetry data.

All of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Prerequisites

• An Elastic Cloud account — sign up now
• A clone of the Elastiflix demo application, or your own Java application
• Basic understanding of Docker — potentially install Docker Desktop
• Basic understanding of Java

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

In particular, we will be working through the following file:

Elastiflix/java-favorite/src/main/java/com/movieapi/ApiServlet.java

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Before we begin, let’s look at the non-instrumented code first.

Step-by-step guide

Step 0. Log in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Set up OpenTelemetry

The first step is to set up the OpenTelemetry SDK in your Java application. You can start by adding the OpenTelemetry Java SDK and its dependencies to your project's build file, such as Maven or Gradle. In our example application, we are using Maven. Add the dependencies below to your pom.xml:

<dependency>
      <groupId>io.opentelemetry.instrumentation</groupId>
      <artifactId>opentelemetry-logback-mdc-1.0</artifactId>
      <version>1.25.1-alpha</version>
    </dependency>

    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-api</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-sdk</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-exporter-otlp</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-semconv</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-exporter-otlp-logs</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry.instrumentation</groupId>
      <artifactId>opentelemetry-logback-appender-1.0</artifactId>
      <version>1.25.1-alpha</version>
    </dependency>

And add the following bill of materials from OpenTelemetry too:

<dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-bom</artifactId>
        <version>1.25.0</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
      <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-bom-alpha</artifactId>
        <version>1.25.0-alpha</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

Step 2. Add the application configuration

We recommend that you add the following configuration to the application’s main method, to start before any application code. Doing it like this gives you a bit more control and flexibility and ensures that OpenTelemetry will be available at any stage of the application lifecycle. In the examples, we put this code before the Spring Boot Application startup. Elastic supports OTLP over HTTP and OTLP over GRPC. In this example, we are using GRPC.

String SERVICE_NAME = System.getenv("OTEL_SERVICE_NAME");

// set service name on all OTel signals
Resource resource = Resource.getDefault().merge(Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME,SERVICE_NAME,ResourceAttributes.SERVICE_VERSION,"1.0",ResourceAttributes.DEPLOYMENT_ENVIRONMENT,"production")));

// init OTel logger provider with export to OTLP
SdkLoggerProvider sdkLoggerProvider = SdkLoggerProvider.builder().setResource(resource).addLogRecordProcessor(BatchLogRecordProcessor.builder(OtlpGrpcLogRecordExporter.builder().setEndpoint(System.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")).addHeader("Authorization", "Bearer " + System.getenv("ELASTIC_APM_SECRET_TOKEN")).build()).build()).build();

// init OTel trace provider with export to OTLP
SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder().setResource(resource).setSampler(Sampler.alwaysOn()).addSpanProcessor(BatchSpanProcessor.builder(OtlpGrpcSpanExporter.builder().setEndpoint(System.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")).addHeader("Authorization", "Bearer " + System.getenv("ELASTIC_APM_SECRET_TOKEN")).build()).build()).build();

// init OTel meter provider with export to OTLP
SdkMeterProvider sdkMeterProvider = SdkMeterProvider.builder().setResource(resource).registerMetricReader(PeriodicMetricReader.builder(OtlpGrpcMetricExporter.builder().setEndpoint(System.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")).addHeader("Authorization", "Bearer " + System.getenv("ELASTIC_APM_SECRET_TOKEN")).build()).build()).build();

// create sdk object and set it as global
OpenTelemetrySdk sdk = OpenTelemetrySdk.builder().setTracerProvider(sdkTracerProvider).setLoggerProvider(sdkLoggerProvider).setMeterProvider(sdkMeterProvider).setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance())).build();

GlobalOpenTelemetry.set(sdk);
// connect logger
GlobalLoggerProvider.set(sdk.getSdkLoggerProvider());
// Add hook to close SDK, which flushes logs
Runtime.getRuntime().addShutdownHook(new Thread(sdk::close));

Step 3. Create the Tracer and start the OpenTelemetry Span inside the TracingFilter

In the Spring Boot, example you will notice that we have a TracingFilter class which extends the OncePerRequestFilter class. This Filter is a component placed at the front of the request processing chain. Its primary roles are to intercept incoming requests and outgoing responses, performing tasks such as logging, authentication, transformation of request/response entities, and more. So what we do here is intercept the request as it comes into the Favorite service, so that we can pull out the headers which may contain tracing information from upstream systems.

We start by using the OpenTelemetry Tracer, which is a core component of OpenTelemetry that allows you to create spans, start and stop them, and add attributes and events. In your Java code, import the necessary OpenTelemetry classes and create an instance of the Tracer within your application.

We use this to create a new downstream span, which will continue as a child from the span created in the upstream system using the information we got from the upstream request. In our Elastiflix example, this will be the nodejs application.

@Override
protected void doFilterInternal(jakarta.servlet.http.HttpServletRequest request, jakarta.servlet.http.HttpServletResponse response, jakarta.servlet.FilterChain filterChain) throws jakarta.servlet.ServletException, IOException {
        Tracer tracer = GlobalOpenTelemetry.getTracer(SERVICE_NAME);

        Context extractedContext = GlobalOpenTelemetry.getPropagators()
                .getTextMapPropagator()
                .extract(Context.current(), request, getter);

        Span span = tracer.spanBuilder(request.getRequestURI())
                .setSpanKind(SpanKind.SERVER)
                .setParent(extractedContext)
                .startSpan();

        try (Scope scope = span.makeCurrent()) {
            filterChain.doFilter(request, response);
        } catch (Exception e) {
            span.setStatus(StatusCode.ERROR);
            throw e;
        } finally {
            span.end();
        }
    }

Step 4. Instrument other interesting code with spans

To instrument with spans and track specific regions of your code, you can use the Tracer's SpanBuilder to create spans. To accurately measure the duration of a specific operation, make sure to start and stop the spans at the appropriate locations in your code. Use the startSpan and endSpan methods provided by the Tracer to mark the beginning and end of the span. For example, you can create a span around a specific method or operation in your code, as shown here in the handleCanary method:

private void handleCanary() throws Exception {
        Span span = GlobalOpenTelemetry.getTracer(SERVICE_NAME).spanBuilder("handleCanary").startSpan();
        Scope scope = span.makeCurrent();

///.....


 span.setStatus(StatusCode.OK);

        span.end();

        scope.close();
    }

Step 5. Add attributes and events to spans

You can enhance the spans with additional attributes and events to provide more context and details about the operation being tracked. Attributes can be key-value pairs that describe the span, while events can be used to mark significant points in the span's lifecycle. This is also shown in the handleCanary method:

private void handleCanary() throws Exception {

            Span.current().setAttribute("canary", "test-new-feature");
            Span.current().setAttribute("quiz_solution", "correlations");

            span.addEvent("a span event", Attributes
                    .of(AttributeKey.longKey("someKey"), Long.valueOf(93)));
    }

Step 6. Instrument backends

Let's consider an example where we are instrumenting a Redis database call. We're using the Java OpenTelemetry SDK, and our goal is to create a trace that captures each "Post User Favorites" operation to the database.

Below is the Java method that performs the operation and collects telemetry data:

public void postUserFavorites(String user_id, String movieID) {
  ...
}

Let's go through it line by line:

Initializing a span
The first important line of our method is where we initialize a span. A span represents a single operation within a trace, which could be a database call, a remote procedure call (RPC), or any segment of code that you want to measure.

Span span = GlobalOpenTelemetry.getTracer(SERVICE_NAME).spanBuilder("Redis.Post").setSpanKind(SpanKind.CLIENT).startSpan();

Setting span attributes
Next, we add attributes to our span. Attributes are key-value pairs that provide additional information about the span. In order to get the backend call to appear correctly in the service map, it is critical that the attributes are set correctly for the backend call type. In this example, we set the db.system attribute to redis.

span.setAttribute("db.system", "redis");
span.setAttribute("db.connection_string", redisHost);
span.setAttribute(
  "db.statement",
  "POST user_id " + user_id + " AND movie_id " + movieID
);

This will ensure calls to the backend redis backend are tracked as shown below:

Capturing the result of the operation
We then execute the operation we're interested in, within a try-catch block. If an exception occurs during the execution of the operation, we record it in the span.

try (Scope scope = span.makeCurrent()) {
    ...
} catch (Exception e) {
    span.setStatus(StatusCode.ERROR, "Error while getting data from Redis");
    span.recordException(e);
}

Closing resources
Finally, we close the Redis connection and end the span.

finally {
    jedis.close();
    span.end();
}

Step 7. Configure logging

Logging is an essential part of application monitoring and troubleshooting. OpenTelemetry allows you to integrate with existing logging frameworks, such as Logback or Log4j, to capture logs along with the telemetry data. Configure the logging framework of your choice to capture logs related to the instrumented spans. In our example application, check out the logback configuration, which shows how to export logs directly to Elastic.

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">

    <appender name="otel-otlp"
        class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
        <captureExperimentalAttributes>false</captureExperimentalAttributes>
        <captureCodeAttributes>true</captureCodeAttributes>
        <captureKeyValuePairAttributes>true</captureKeyValuePairAttributes>
    </appender>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="DEBUG">
     <appender-ref ref="otel-otlp" />
        <appender-ref ref="STDOUT" />

    </root>
</configuration>

Step 8. Running the Docker image with environment variables

As specified in the OTEL Java documentation, we will use environment variables and pass in the configuration values to enable it to connect with Elastic Observability’s APM server.

Because Elastic accepts OTLP natively, we just need to provide the Endpoint and authentication where the OTEL Exporter needs to send the data, as well as some other environment variables.

Getting Elastic Cloud variables
You can copy the endpoints and token from Kibana under the path /app/home#/tutorial/apm.

You will need to copy the following environment variable:

OTEL_EXPORTER_OTLP_ENDPOINT

As well as the token from:

OTEL_EXPORTER_OTLP_HEADERS

Build the Docker image

docker build -t java-otel-manual-image .

Run the Docker image

docker run \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT" \
       -e ELASTIC_APM_SECRET_TOKEN="REPLACE WITH TOKEN" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production" \
       -e OTEL_SERVICE_NAME="java-favorite-otel-manual" \
       -p 5000:5000 \
       java-otel-manual-image

You can now issue a few requests in order to generate trace data. Note that these requests are expected to return an error, as this service relies on a connection to Redis that you don’t currently have running. As mentioned before, you can find a more complete example using docker-compose here.

curl localhost:5000/favorites

# or alternatively issue a request every second

while true; do curl "localhost:5000/favorites"; sleep 1; done;

Step 9. Explore traces and logs in Elastic APM

Once you have this up and running, you can ping the endpoint for your instrumented service (in our case, this is /favorites), and you should see the app appear in Elastic APM, as shown below:

It will begin by tracking throughput and latency critical metrics for SREs to pay attention to.

Digging in, we can see an overview of all our Transactions.

And look at specific transactions:

Click on Logs , and we see that logs are also brought over. The OTel Agent will automatically bring in logs and correlate them with traces for you:

This gives you complete visibility across logs, metrics, and traces!

Wrapping up

Manually instrumenting your Java applications with OpenTelemetry gives you greater control over what to track and monitor. By following the steps outlined in this blog post, you can effectively monitor the performance of your Java applications, identify issues, and gain insights into the overall health of your application.

Remember, OpenTelemetry is a powerful tool, and proper instrumentation requires careful consideration of what metrics, traces, and logs are essential for your specific use case. Experiment with different configurations, leverage the OpenTelemetry SDK for Java documentation, and continuously iterate to achieve the observability goals of your application.

In this blog, we discussed the following:

• How to manually instrument Java with OpenTelemetry
• How to properly initialize and instrument span
• How to easily set the OTLP ENDPOINT and OTLP HEADERS from Elastic without the need for a collector

Hopefully, this provided an easy-to-understand walk-through of instrumenting Java with OpenTelemetry and how easy it is to send traces into Elastic.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Observability across the entire stack of modern distributed applications requires data collection, processing, and correlation often in the form of dashboards. Ingesting all system data requires installing agents across stacks, frameworks, and providers — a process that can be challenging and time-consuming for teams who have to deal with version changes, compatibility issues, and proprietary code that doesn't scale as systems change.

Thanks to OpenTelemetry (OTel), DevOps and SRE teams now have a standard way to collect and send data that doesn't rely on proprietary code and has a large support community reducing vendor lock-in.

In a previous blog, we also reviewed how to use the OpenTelemetry demo and connect it to Elastic^®, as well as some of Elastic’s capabilities with OpenTelemetry visualizations and Kubernetes.

In this blog, we will show how to use automatic instrumentation for OpenTelemetry with the Python service of our application called Elastiflix, which helps highlight auto-instrumentation in a simple way.

The beauty of this is that there is no need for the otel-collector! This setup enables you to slowly and easily migrate an application to OTel with Elastic according to a timeline that best fits your business.

Application, prerequisites, and config

The application that we use for this blog is called Elastiflix, a movie-streaming application. It consists of several micro-services written in .NET, NodeJS, Go, and Python.

Before we instrument our sample application, we will first need to understand how Elastic can receive the telemetry data.

All of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Prerequisites

• An Elastic Cloud account — sign up now
• A clone of the Elastiflix demo application, or your own Python application
• Basic understanding of Docker — potentially install Docker Desktop
• Basic understanding of Python

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Step-by-step guide

Step 0. Log in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Configure auto-instrumentation for the Python Service

We are going to use automatic instrumentation with Python service from the Elastiflix demo application.

We will be using the following service from Elastiflix:

Elastiflix/python-favorite-otel-auto

Per the OpenTelemetry Automatic Instrumentation for Python documentation, you will simply install the appropriate Python packages using pip install.

>pip install opentelemetry-distro \
    opentelemetry-exporter-otlp

>opentelemetry-bootstrap -a install

If you are running the Python service on the command line, then you can use the following command:

opentelemetry-instrument python main.py

For our application, we do this as part of the Dockerfile.

Dockerfile

FROM python:3.9-slim as base

# get packages
COPY requirements.txt .
RUN pip install -r requirements.txt
WORKDIR /favoriteservice

#install opentelemetry packages
RUN pip install opentelemetry-distro \
    opentelemetry-exporter-otlp

RUN opentelemetry-bootstrap -a install

# Add the application
COPY . .

EXPOSE 5000
ENTRYPOINT [ "opentelemetry-instrument", "python", "main.py"]

Step 2. Running the Docker image with environment variables

As specified in the OTEL Python documentation, we will use environment variables and pass in the configuration values to enable it to connect with Elastic Observability’s APM server.

Because Elastic accepts OTLP natively, we just need to provide the Endpoint and authentication where the OTEL Exporter needs to send the data, as well as some other environment variables.

Getting Elastic Cloud variables
You can copy the endpoints and token from Kibana^® under the path /app/home#/tutorial/apm.

You will need to copy the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS

Build the image

docker build -t  python-otel-auto-image .

Run the image

docker run \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="<REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT>" \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20<REPLACE WITH TOKEN>" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production" \
       -e OTEL_SERVICE_NAME="python-favorite-otel-auto" \
       -p 5001:5001 \
       python-otel-auto-image

Important: Note that the “OTEL_EXPORTER_OTLP_HEADERS” variable has the whitespace after Bearer escaped as “%20” — this is a requirement for Python.

You can now issue a few requests in order to generate trace data. Note that these requests are expected to return an error, as this service relies on a connection to Redis that you don’t currently have running. As mentioned before, you can find a more complete example using docker-compose here.

curl localhost:5000/favorites

# or alternatively issue a request every second

while true; do curl "localhost:5000/favorites"; sleep 1; done;

Step 3: Explore traces, metrics, and logs in Elastic APM

Exploring the Services section in Elastic APM, you’ll see the Python service displayed.

Clicking on the python-favorite-otel-auto service , you can see that it is ingesting telemetry data using OpenTelemetry.

In this blog, we discussed the following:

• How to auto-instrument Python with OpenTelemetry
• Using standard commands in a Dockerfile, auto-instrumentation was done efficiently and without adding code in multiple places

Since Elastic can support a mix of methods for ingesting data, whether it be using auto-instrumentation of open-source OpenTelemetry or manual instrumentation with its native APM agents, you can plan your migration to OTel by focusing on a few applications first and then using OpenTelemety across your applications later on in a manner that best fits your business needs.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

DevOps engineers continuously optimize software delivery, while SRE teams act as the stewards of application reliability, scalability, and top-tier performance. The challenge? These teams require a cutting-edge observability solution, one that encompasses full-stack insights, empowering them to rapidly manage, monitor, and rectify potential disruptions before they culminate into operational challenges.

Observability in our modern distributed software ecosystem goes beyond mere monitoring — it demands limitless data collection, precision in processing, and the correlation of this data into actionable insights. However, the road to achieving this holistic view is paved with obstacles, from navigating version incompatibilities to wrestling with restrictive proprietary code.

Enter OpenTelemetry (OTel), with the following benefits for those who adopt it:

• Escape vendor constraints with OTel, freeing yourself from vendor lock-in and ensuring top-notch observability.
• See the harmony of unified logs, metrics, and traces come together to provide a complete system view.
• Improve your application oversight through richer and enhanced instrumentations.
• Embrace the benefits of backward compatibility to protect your prior instrumentation investments.
• Embark on the OpenTelemetry journey with an easy learning curve, simplifying onboarding and scalability.
• Rely on a proven, future-ready standard to boost your confidence in every investment.

In this blog, we will explore how you can use automatic instrumentation in your Java application using Docker, without the need to refactor any part of your application code. We will use an application called Elastiflix, which helps highlight auto-instrumentation in a simple way.

Application, prerequisites, and config

The application that we use for this blog is called Elastiflix, a movie-streaming application. It consists of several micro-services written in .NET, NodeJS, Go, and Python.

Before we instrument our sample application, we will first need to understand how Elastic can receive the telemetry data.

All of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Prerequisites

• An Elastic Cloud account — sign up now.
• A clone of the Elastiflix demo application, or your own Java application
• Basic understanding of Docker — potentially install Docker Desktop
• Basic understanding of Java

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Step-by-step guide

Step 0. Log in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Configure auto-instrumentation for the Java service

We are going to use automatic instrumentation with Java service from the Elastiflix demo application.

We will be using the following service from Elastiflix:

Elastiflix/java-favorite-otel-auto

Per the OpenTelemetry Automatic Instrumentation for Java documentation and documentation, you will simply install the appropriate Java packages.

Create a local OTel directory to download the OpenTelemetry Java agent. Download opentelemetry-javaagent.jar.

>mkdir /otel

>curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar –output /otel/opentelemetry-javaagent.jar

If you are going to run the service on the command line, then you can use the following command:

java -javaagent:/otel/opentelemetry-javaagent.jar \
-jar /usr/src/app/target/favorite-0.0.1-SNAPSHOT.jar --server.port=5000

For our application, we will do this as part of the Dockerfile.

Dockerfile

Start with a base image containing Java runtime
FROM maven:3.8.2-openjdk-17-slim as build

# Make port 8080 available to the world outside this container
EXPOSE 5000

# Change to the app directory
WORKDIR /usr/src/app

# Copy the local code to the container
COPY . .

# Build the application
RUN mvn clean install

USER root
RUN apt-get update && apt-get install -y zip curl
RUN mkdir /otel
RUN curl -L -o /otel/opentelemetry-javaagent.jar https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v1.28.0/opentelemetry-javaagent.jar

COPY start.sh /start.sh
RUN chmod +x /start.sh

ENTRYPOINT ["/start.sh"]

Step 2. Running the Docker Image with environment variables

As specified in the OTEL Java documentation, we will use environment variables and pass in the configuration values to enable it to connect with Elastic Observability’s APM server.

Because Elastic accepts OTLP natively, we just need to provide the Endpoint and authentication where the OTEL Exporter needs to send the data, as well as some other environment variables.

Getting Elastic Cloud variables
You can copy the endpoints and token from Kibana under the path /app/home#/tutorial/apm.

You will need to copy the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS

Build the Docker image

docker build -t java-otel-auto-image .

Run the Docker image

docker run \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT" \
       -e ELASTIC_APM_SECRET_TOKEN="REPLACE WITH THE BIT AFTER Authorization=Bearer " \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production" \
       -e OTEL_SERVICE_NAME="java-favorite-otel-auto" \
       -p 5000:5000 \
       java-otel-auto-image

You can now issue a few requests in order to generate trace data. Note that these requests are expected to return an error, as this service relies on a connection to Redis that you don’t currently have running. As mentioned before, you can find a more complete example using docker-compose here.

curl localhost:5000/favorites

# or alternatively issue a request every second

while true; do curl "localhost:5000/favorites"; sleep 1; done;

Step 3: Explore traces and logs in Elastic APM

Once you have this up and running, you can ping the endpoint for your instrumented service (in our case, this is /favorites), and you should see the app appear in Elastic APM, as shown below:

It will begin by tracking throughput and latency critical metrics for SREs to pay attention to.

Digging in, we can see an overview of all our Transactions.

And look at specific transactions:

Click on Logs, and we see that logs are also brought over. The OTel Agent will automatically bring in logs and correlate them with traces for you:

This gives you complete visibility across logs, metrics, and traces!

Basic concepts: How APM works with Java

Before we continue, let's first understand a few basic concepts and terms.

• Java Agent: This is a tool that can be used to instrument (or modify) the bytecode of class files in the Java Virtual Machine (JVM). Java agents are used for many purposes like performance monitoring, logging, security, and more.
• Bytecode: This is the intermediary code generated by the Java compiler from your Java source code. This code is interpreted or compiled on the fly by the JVM to produce machine code that can be executed.
• Byte Buddy: Byte Buddy is a code generation and manipulation library for Java. It is used to create, modify, or adapt Java classes at runtime. In the context of a Java Agent, Byte Buddy provides a powerful and flexible way to modify bytecode. Both the Elastic APM Agent and the OpenTelemetry Agent use Byte Buddy under the covers.

Now, let's talk about how automatic instrumentation works with Byte Buddy:

Automatic instrumentation is the process by which an agent modifies the bytecode of your application's classes, often to insert monitoring code. The agent doesn't modify the source code directly, but rather the bytecode that is loaded into the JVM. This is done while the JVM is loading the classes, so the modifications are in effect during runtime.

Here's a simplified explanation of the process:

1. Start the JVM with the agent: When starting your Java application, you specify the Java agent with the -javaagent command line option. This instructs the JVM to load your agent before the main method of your application is invoked. At this point, the agent has the opportunity to set up class transformers.

2. Register a class file transformer with Byte Buddy: Your agent will register a class file transformer with Byte Buddy. A transformer is a piece of code that is invoked every time a class is loaded into the JVM. This transformer receives the bytecode of the class, and it can modify this bytecode before the class is actually used.

3. Transform the bytecode: When your transformer is invoked, it will use Byte Buddy's API to modify the bytecode. Byte Buddy allows you to specify your transformations in a high-level, expressive way rather than manually writing complex bytecode. For example, you could specify a certain class and method within that class that you want to instrument and provide an "interceptor" that will add new behavior to that method.

4. Use the transformed classes: Once the agent has set up its transformers, the JVM continues to load classes as usual. Each time a class is loaded, your transformers are invoked, allowing them to modify the bytecode. Your application then uses these transformed classes as if they were the original ones, but they now have the extra behavior that you've injected through your interceptor.

In essence, automatic instrumentation with Byte Buddy is about modifying the behavior of your Java classes at runtime, without needing to alter the source code directly. This is especially useful for cross-cutting concerns like logging, monitoring, or security, as it allows you to centralize this code in your Java Agent, rather than scattering it throughout your application.

Summary

With this Dockerfile, you've transformed your simple Java application into one that's automatically instrumented with OpenTelemetry. This will aid greatly in understanding application performance, tracing errors, and gaining insights into how users interact with your software.

Remember, observability is a crucial aspect of modern application development, especially in distributed systems. With tools like OpenTelemetry, understanding complex systems becomes a tad bit easier.

In this blog, we discussed the following:

• How to auto-instrument Java with OpenTelemetry.
• Using standard commands in a Docker file, auto-instrumentation was done efficiently and without adding code in multiple places enabling manageability.
• Using OpenTelemetry and its support for multiple languages, DevOps and SRE teams can auto-instrument their applications with ease gaining immediate insights into the health of the entire application stack and reduce mean time to resolution (MTTR).

Since Elastic can support a mix of methods for ingesting data, whether it be using auto-instrumentation of open-source OpenTelemetry or manual instrumentation with its native APM agents, you can plan your migration to OTel by focusing on a few applications first and then using OpenTelemety across your applications later on in a manner that best fits your business needs.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.