Observability across the entire stack of modern distributed applications requires data collection, processing, and correlation often in the form of dashboards. Ingesting all system data requires installing agents across stacks, frameworks, and providers — a process that can be challenging and time-consuming for teams who have to deal with version changes, compatibility issues, and proprietary code that doesn't scale as systems change.

Thanks to OpenTelemetry (OTel), DevOps and SRE teams now have a standard way to collect and send data that doesn't rely on proprietary code and has a large support community reducing vendor lock-in.

In a previous blog, we also reviewed how to use the OpenTelemetry demo and connect it to Elastic^®, as well as some of Elastic’s capabilities with OpenTelemetry visualizations and Kubernetes.

In this blog, we will show how to use automatic instrumentation for OpenTelemetry with the Python service of our application called Elastiflix, which helps highlight auto-instrumentation in a simple way.

The beauty of this is that there is no need for the otel-collector! This setup enables you to slowly and easily migrate an application to OTel with Elastic according to a timeline that best fits your business.

Application, prerequisites, and config

The application that we use for this blog is called Elastiflix, a movie-streaming application. It consists of several micro-services written in .NET, NodeJS, Go, and Python.

Before we instrument our sample application, we will first need to understand how Elastic can receive the telemetry data.

All of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Prerequisites

• An Elastic Cloud account — sign up now
• A clone of the Elastiflix demo application, or your own Python application
• Basic understanding of Docker — potentially install Docker Desktop
• Basic understanding of Python

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Step-by-step guide

Step 0. Log in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Configure auto-instrumentation for the Python Service

We are going to use automatic instrumentation with Python service from the Elastiflix demo application.

We will be using the following service from Elastiflix:

Elastiflix/python-favorite-otel-auto

Per the OpenTelemetry Automatic Instrumentation for Python documentation, you will simply install the appropriate Python packages using pip install.

>pip install opentelemetry-distro \
	opentelemetry-exporter-otlp

>opentelemetry-bootstrap -a install

If you are running the Python service on the command line, then you can use the following command:

opentelemetry-instrument python main.py

For our application, we do this as part of the Dockerfile.

Dockerfile

FROM python:3.9-slim as base

# get packages
COPY requirements.txt .
RUN pip install -r requirements.txt
WORKDIR /favoriteservice

#install opentelemetry packages
RUN pip install opentelemetry-distro \
	opentelemetry-exporter-otlp

RUN opentelemetry-bootstrap -a install

# Add the application
COPY . .

EXPOSE 5000
ENTRYPOINT [ "opentelemetry-instrument", "python", "main.py"]

Step 2. Running the Docker image with environment variables

As specified in the OTEL Python documentation, we will use environment variables and pass in the configuration values to enable it to connect with Elastic Observability’s APM server.

Because Elastic accepts OTLP natively, we just need to provide the Endpoint and authentication where the OTEL Exporter needs to send the data, as well as some other environment variables.

Getting Elastic Cloud variables
You can copy the endpoints and token from Kibana^® under the path /app/home#/tutorial/apm.

You will need to copy the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS

Build the image

docker build -t  python-otel-auto-image .

Run the image

docker run \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="<REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT>" \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20<REPLACE WITH TOKEN>" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production" \
       -e OTEL_SERVICE_NAME="python-favorite-otel-auto" \
       -p 5001:5001 \
       python-otel-auto-image

Important: Note that the “OTEL_EXPORTER_OTLP_HEADERS” variable has the whitespace after Bearer escaped as “%20” — this is a requirement for Python.

You can now issue a few requests in order to generate trace data. Note that these requests are expected to return an error, as this service relies on a connection to Redis that you don’t currently have running. As mentioned before, you can find a more complete example using docker-compose here.

curl localhost:5000/favorites

# or alternatively issue a request every second

while true; do curl "localhost:5000/favorites"; sleep 1; done;

Step 3: Explore traces, metrics, and logs in Elastic APM

Exploring the Services section in Elastic APM, you’ll see the Python service displayed.

Clicking on the python-favorite-otel-auto service , you can see that it is ingesting telemetry data using OpenTelemetry.

In this blog, we discussed the following:

• How to auto-instrument Python with OpenTelemetry
• Using standard commands in a Dockerfile, auto-instrumentation was done efficiently and without adding code in multiple places

Since Elastic can support a mix of methods for ingesting data, whether it be using auto-instrumentation of open-source OpenTelemetry or manual instrumentation with its native APM agents, you can plan your migration to OTel by focusing on a few applications first and then using OpenTelemety across your applications later on in a manner that best fits your business needs.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this article, we will explore the central configuration capabilities for EDOT SDKs with the EDOT Gateway Collector. You will learn how to configure the EDOT SDKs and the EDOT Gateway Collector to enable central configuration. Finally, we'll cover the configuration settings supported through central configuration.

Central configuration based on the OpenTelemetry Open Agent Management Protocol

The OpenTelemetry project provides OpAMP for the remote management of large fleets of data collection Agents among other capabilities. The central management of EDOT SDKs leverages OpAMP for dispatching the configurations. It's a client-server network protocol where the OpAMP server is part of the EDOT Collector and the OpAMP client is part of the EDOT SDK. The EDOT SDK polls at regular intervals the OpAMP server for configuration updates. The OpAMP server in the EDOT collector is part of the Elastic APM central configuration extension. This extension reads the configuration for the EDOT SDKs from Elasticsearch. The apmconfigextension, which is the technical name of the Elastic APM central configuration extension, is included and needs to be configured in the EDOT Collector configuration to be activated. The OpAMP specification includes WebSocket and plain HTTP connection for transport. The EDOT SDKs pull the configuration utilizing the plain HTTP connection for transport from the EDOT Collector.

Prerequisites

Central configuration of EDOT SDKs requires a standalone EDOT Collector running in Gateway mode. Other collectors like the OpenTelemetry contrib collector or a custom distribution of the collector you build yourself require the Elastic APM central configuration extension.

EDOT versions supporting central configuration

The following table gives an overview of the versions of the EDOT SDKs and EDOT Collector that provide central configuration support. Applications and services have to be instrumented with the EDOT SDKs with a version listed below to pull and apply central configuration changes.

Central configuration is not blocking application startup for EDOT Java, Node.js, PHP and Python. The application starts with default configuration or the configuration provided by environment variables. When the central configuration settings are successfully pulled, they will take precedence over local configuration settings. EDOT .NET currently does not support central configuration although it’s planned to add support.

Furthermore, central configuration for EDOT SDKs is not supported on Elastic Cloud Serverless or the Elastic Cloud Managed OTLP Endpoint, yet. The following table gives an overview of the versions of the Elastic stack that support central configuration of EDOT SDKs.

Retrieve Elasticsearch endpoint and API key

The Elastic APM central configuration extension needs the Elasticsearch endpoint in the configuration to be able to read the central configuration settings for the EDOT SDKs. The Elasticsearch endpoint is the same that the Elasticsearch exporter uses to export telemetry data to Elasticsearch. The central configuration documentation describes the steps to obtain the endpoint and the API key in more detail.

Enable central configuration for EDOT

To enable central configuration, the EDOT Collector needs the apmconfigextension to be configured as part of the extensions section. This requires the Elasticsearch endpoint obtained above and an Elasticsearch API key. The environment variable ELASTIC_OTEL_OPAMP_ENDPOINT needs to be set to enable central configuration in the EDOT SDK.

In the following, the configuration is explained for the EDOT Gateway Collector and the EDOT SDKs.

Configure the EDOT Collector to enable central configuration

Central configuration support in the EDOT Collector is enabled by adding the configuration of the Elastic APM central configuration extension configuration to the configuration file. For the authentication of the apmconfigextension with the Elasticsearch endpoint, bearertokenauth authenticator is configured. This configures a client type authenticator for outgoing requests to the Elasticsearch endpoint. The apmconfigextension acts as client and Elasticsearch endpoint as server. The apmconfig section configures the OpAMP server endpoint. EDOT SDKs will connect to the endpoint to fetch the configuration. The service section activates the apmconfig and bearertokenauth extension. The following code snippet shows the configuration excerpt of the EDOT Collector including the bearertoken authenticator configuration and the apmconfig configuration for central configuration.

extensions:
  bearertokenauth:
    scheme: "APIKey"
    token: "<ENCODED_ELASTICSEARCH_APIKEY>"
  source:
     elasticsearch:
       endpoint: "<YOUR_ELASTICSEARCH_ENDPOINT>"
       auth:
         authenticator: bearertokenauth
  apmconfig:
    opamp:
      protocols:
        http:
          # Default is localhost:4320
          # To specify a custom endpoint, uncomment the following line
          # and set the endpoint to the custom endpoint
          # endpoint: "<CUSTOM_OPAMP_ENDPOINT>"
  service:
    extensions: [bearertokenauth, apmconfig]

Download the EDOT collector and include the configuration from the snippet above in the otel.yml configuration file to enable central configuration. The otel.yml configuration file examples are available in the EDOT Collector documentation. Consider the example for direct ingestion into Elasticsearch.

Configure the EDOT SDKs to enable central configuration

To enable central configuration in the EDOT SDKs Java, Node.js, PHP and Python, set the ELASTIC_OTEL_OPAMP_ENDPOINT environment variable to the OpAMP server endpoint of the EDOT Collector and set the required resource attributes.

Enable central configuration of EDOT SDKs

The following code snippet shows how to set the ELASTIC_OTEL_OPAMP_ENDPOINT environment variable with the export command in a shell.

export ELASTIC_OTEL_OPAMP_ENDPOINT="http://<your-opamp-end-point>:4320/v1/opamp"

<your-opamp-end-point> must be set to the address or host name of the EDOT Gateway Collector that provides the OpAMP server endpoint for central configuration.

When you are using EDOT SDKs for mobile, the documentation shows how to activate central configuration support in EDOT Android and EDOT iOS.

Configure resource attributes

Central configuration requires the OpenTelemetry resource attributes service.name and deployment.environment.name to be set. While service.name is mandatory, deployment.environment.name is optional but recommended. If deployment.environment.name is unset, no configuration can be created that applies to a whole environment.

Set the OTEL_RESOURCE_ATTRIBUTES environment variable including the service.name and deployment.environment.name like in the following code snippet. The key-value pairs are concatenated with a comma as separator and provided as value to the OTEL_RESOURCE_ATTRIBUTES environment variable.

export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,service.name=my-app"

Supported configuration settings

The following tables give an overview of the supported central configuration settings at the time of writing.

Non-mobile EDOT SDKs

The table shows the supported central configuration settings of EDOT Java, Node.js, PHP, and Python SDK.

Mobile EDOT SDKs

The table below shows the supported central configuration settings of EDOT Android, and iOS SDK.

Use Elastic Observability to change configuration settings of EDOT SDKs

An application must produce and send telemetry data otherwise the EDOT SDK will not appear in the Agent configuration UI in Elastic Observability. Because the Agent configuration in Elastic Observability has no knowledge about the existence of the EDOT SDK until it's receiving telemetry data. The OpenTelemetry resource attribute service.name is used as the key to assign a configuration to an EDOT SDK. Currently, EDOT SDKs will not show up in the Agent Explorer.

Go to Kibana -> Observability -> Applications -> Service Inventory -> Settings -> Agent Configuration to create a new configuration for your EDOT SDK.

The EDOT Java SDK configuration above deactivates all instrumentations by setting deactivate_all_instrumentations to true. This is useful in situations when switching off the instrumentations is necessary. The sampling_rate setting becomes handy when the sampling rate of the EDOT SDK should be changed. A sampling rate can be selected according to the needs as shown below.

Disable central configuration

Disabling central configuration is easy. Remove the environment variable ELASTIC_OTEL_OPAMP_ENDPOINT and restart the application to disable central configuration support in the EDOT SDKs. Remove the apmconfig extension from the service section of the EDOT Collector configuration and restart the collector to disable the Elastic APM central configuration extension in the EDOT Collector.

Elastic's contribution to OpenTelemetry

Elastic is committed to the OpenTelemetry project. Elastic contributed the Java OpAMP client implementation to the OpenTelemetry project (Github PR), and is working on the contribution for Python (Github PR) and Node.js. The OpAMP client for PHP will be part of a larger contribution (Github issue) that Elastic is doing.

Conclusion

In this article, you learned how to configure the EDOT SDKs centrally in Elastic Observability with the Gateway Collector and OpAMP at scale. You learned how to configure the apmconfigextension in the collector and how to set the ELASTIC_OTEL_OPAMP_ENDPOINT environment variable to enable central configuration in the EDOT SDKs, which versions of the SDKs and collector support central configuration, and what configuration settings are currently supported. Now you can leverage central configuration in large deployments to manage the configuration of the EDOT SDKs.

Prerequisites

• An Elastic Cloud account (Serverless or Cloud Hosted v9.0+)
• Python 3.9+
• Kibana's built-in sample eCommerce orders dataset (Add data > Sample data > Sample eCommerce orders)

Install the required packages:

pip install flask elasticsearch python-dotenv \
  opentelemetry-api opentelemetry-sdk \
  opentelemetry-exporter-otlp

What is the Elastic managed OTLP endpoint?

The Elastic Managed OTLP Endpoint is a hosted ingestion layer that accepts standard OTLP over HTTP or gRPC directly from your application, with no collector required.

OpenTelemetry uses OTLP (OpenTelemetry Protocol) to transport telemetry data.

In a traditional setup, your application sends data to an OpenTelemetry Collector, which then forwards it to a backend like Elasticsearch.

The Collector handles batching, retries, and routing, but it also means one more component to deploy, configure, and maintain.

Elastic's managed OTLP endpoint, recently GA, removes that step.

Your application sends OTLP data directly to an endpoint that Elastic operates for you.

The endpoint accepts standard OTLP over HTTP or gRPC, backed by a managed ingestion layer that handles scaling, buffering, and durability.

You use the same opentelemetry-exporter-otlp packages you would use with any backend.

Building a Python Flask API to instrument with OpenTelemetry

We'll build a small REST API in Python with Flask that lists and looks up eCommerce orders stored in Elasticsearch.

As a data source, we'll use Kibana's built-in sample data.

You can find the complete application code in the companion repository.

The walkthrough below builds the code step by step, but you can clone the repo and follow along if you prefer.

Before adding telemetry, let's look at the base application: a Flask API that queries Kibana's sample eCommerce index.

It has two endpoints: list recent orders and look up a single order by ID.

Create a file called app.py:

import os

from dotenv import load_dotenv
from elasticsearch import Elasticsearch
from flask import Flask, jsonify

load_dotenv()

# Elasticsearch client
es = Elasticsearch(
    hosts=[os.environ["ES_URL"]],
    api_key=os.environ["ES_API_KEY"],
)
INDEX = "kibana_sample_data_ecommerce"

app = Flask(__name__)

@app.route("/orders")
def list_orders():
    response = es.search(
        index=INDEX,
        size=10,
        sort=[{"order_date": "desc"}],
        aggs={"total_revenue": {"sum": {"field": "taxful_total_price"}}},
    )

    hits = response["hits"]["hits"]
    total_revenue = response["aggregations"]["total_revenue"]["value"]
    orders = [
        {
            "order_id": h["_source"]["order_id"],
            "customer": h["_source"]["customer_full_name"],
            "total": h["_source"]["taxful_total_price"],
            "date": h["_source"]["order_date"],
        }
        for h in hits
    ]

    return jsonify({"orders": orders, "total_revenue": total_revenue})

@app.route("/orders/<order_id>")
def get_order(order_id):
    response = es.search(
        index=INDEX,
        size=1,
        query={"term": {"order_id": order_id}},
    )

    hits = response["hits"]["hits"]
    if not hits:
        return jsonify({"error": "Order not found"}), 404

    return jsonify(hits[0]["_source"])

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5001)

Create a .env file with your Elasticsearch credentials.

To get the OTLP endpoint and API key, follow the Quickstart: Send OTLP data to Elastic guide.

OTEL_EXPORTER_OTLP_ENDPOINT=
OTEL_EXPORTER_OTLP_HEADERS=Authorization=ApiKey <api_key>
ES_URL=
ES_API_KEY=

This is a working API with no instrumentation. The next sections add traces, metrics, and logs one signal at a time.

Sending traces from Python

Let's start with traces using the Python OpenTelemetry SDK.

We'll wrap the Elasticsearch calls in spans so we can see how long each query takes.

Set up the tracer

Add the following to the top of app.py:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME

# Create a resource identifying your service
resource = Resource.create({
    SERVICE_NAME: "my-python-app"
})

# Set up the tracer provider with OTLP export
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(tracer_provider)

tracer = trace.get_tracer(__name__)

The OTLPSpanExporter() reads the endpoint and headers from the environment variables we set earlier, so no arguments are needed in the code.

Add spans to your endpoints

Now wrap each route handler in a parent span and nest a child span around the Elasticsearch call. This gives you a two-level trace: one span for the HTTP request and one for the database query inside it.

@app.route("/orders")
def list_orders():
    with tracer.start_as_current_span("list-orders") as span:

        with tracer.start_as_current_span("es.search") as es_span:
            es_span.set_attribute("db.system", "elasticsearch")
            es_span.set_attribute("db.elasticsearch.index", INDEX)

            response = es.search(
                index=INDEX,
                size=10,
                sort=[{"order_date": "desc"}],
                aggs={"total_revenue": {"sum": {"field": "taxful_total_price"}}},
            )

        hits = response["hits"]["hits"]
        total_revenue = response["aggregations"]["total_revenue"]["value"]
        orders = [
            {
                "order_id": h["_source"]["order_id"],
                "customer": h["_source"]["customer_full_name"],
                "total": h["_source"]["taxful_total_price"],
                "date": h["_source"]["order_date"],
            }
            for h in hits
        ]

        span.set_attribute("orders.returned", len(orders))
        return jsonify({"orders": orders, "total_revenue": total_revenue})

Each request creates a parent span (list-orders) with a child span (es.search) wrapping the Elasticsearch call.

The db.system and db.elasticsearch.index attributes are what Kibana uses to recognize the call as an Elasticsearch query in the trace waterfall.

Sending metrics from Python

Metrics reveal patterns across all requests: how many calls hit each endpoint, how response times trend over time. Let's add a counter and a histogram to capture that.

Set up the meter

Add the following to app.py:

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter

# Set up the metric reader and provider
metric_reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(),
    export_interval_millis=10000  # Export every 10 seconds
)
meter_provider = MeterProvider(
    resource=resource,
    metric_readers=[metric_reader]
)
metrics.set_meter_provider(meter_provider)

meter = metrics.get_meter(__name__)

Create instruments

# Counter for total requests
request_counter = meter.create_counter(
    name="app.requests.total",
    description="Total number of requests",
    unit="1"
)

# Histogram for request duration
request_duration = meter.create_histogram(
    name="app.request.duration",
    description="Request duration in seconds",
    unit="s"
)

Record metrics in your endpoints

Update the route handler to record metrics:

import time

@app.route("/orders")
def list_orders():
    start_time = time.time()

    with tracer.start_as_current_span("list-orders") as span:

        with tracer.start_as_current_span("es.search") as es_span:
            es_span.set_attribute("db.system", "elasticsearch")
            es_span.set_attribute("db.elasticsearch.index", INDEX)

            response = es.search(
                index=INDEX,
                size=10,
                sort=[{"order_date": "desc"}],
                aggs={"total_revenue": {"sum": {"field": "taxful_total_price"}}},
            )

        hits = response["hits"]["hits"]
        total_revenue = response["aggregations"]["total_revenue"]["value"]
        orders = [
            {
                "order_id": h["_source"]["order_id"],
                "customer": h["_source"]["customer_full_name"],
                "total": h["_source"]["taxful_total_price"],
                "date": h["_source"]["order_date"],
            }
            for h in hits
        ]

        duration = time.time() - start_time
        span.set_attribute("orders.returned", len(orders))
        request_counter.add(1, {"endpoint": "/orders", "status": "200"})
        request_duration.record(duration, {"endpoint": "/orders"})

        return jsonify({"orders": orders, "total_revenue": total_revenue})

Sending logs from Python

Traces and metrics tell you what happened and how often. Logs capture the details: error messages, parameter values, and debug context. OpenTelemetry bridges Python's standard logging module so your existing log statements are exported as OTLP data alongside traces and metrics.

When you emit a log inside a span context, the SDK automatically attaches the trace and span IDs, so you can jump from a log line to the exact trace that produced it.

Set up the log provider

import logging
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter

# Set up the logger provider
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(
    BatchLogRecordProcessor(OTLPLogExporter())
)
set_logger_provider(logger_provider)

# Bridge Python's logging to OTel
handler = LoggingHandler(
    level=logging.INFO,
    logger_provider=logger_provider
)
logging.getLogger().addHandler(handler)
logging.getLogger().addHandler(logging.StreamHandler())  # Also print to terminal
logging.getLogger().setLevel(logging.INFO)

logger = logging.getLogger(__name__)

The StreamHandler() ensures logs also appear in your terminal while developing.

Without it, logs only go to Elastic and the terminal stays silent, which can be confusing during testing.

Emit logs from your endpoints

@app.route("/orders")
def list_orders():
    start_time = time.time()

    with tracer.start_as_current_span("list-orders") as span:
        logger.info("Listing recent orders")

        with tracer.start_as_current_span("es.search") as es_span:
            es_span.set_attribute("db.system", "elasticsearch")
            es_span.set_attribute("db.elasticsearch.index", INDEX)

            response = es.search(
                index=INDEX,
                size=10,
                sort=[{"order_date": "desc"}],
                aggs={"total_revenue": {"sum": {"field": "taxful_total_price"}}},
            )

        hits = response["hits"]["hits"]
        total_revenue = response["aggregations"]["total_revenue"]["value"]
        orders = [
            {
                "order_id": h["_source"]["order_id"],
                "customer": h["_source"]["customer_full_name"],
                "total": h["_source"]["taxful_total_price"],
                "date": h["_source"]["order_date"],
            }
            for h in hits
        ]

        duration = time.time() - start_time
        span.set_attribute("orders.returned", len(orders))
        logger.info(
            "Orders listed",
            extra={"orders.returned": len(orders), "duration_s": round(duration, 4)}
        )

        request_counter.add(1, {"endpoint": "/orders", "status": "200"})
        request_duration.record(duration, {"endpoint": "/orders"})

        return jsonify({"orders": orders, "total_revenue": total_revenue})

Because the logger.info() calls happen inside a span context, OpenTelemetry automatically attaches the trace ID and span ID to each log record.

In Kibana, this means you can jump from a log line directly to the related trace.

Running the instrumented Flask app and generating traffic

With traces, metrics, and logs wired up, the application is fully instrumented. Let's run it, send a few requests, and verify the data arrives in Kibana.

python app.py

Note: Flask's built-in development server is fine for this tutorial. For production workloads, use a WSGI server like gunicorn.

Generate some traffic:

curl http://localhost:5001/orders
curl http://localhost:5001/orders/584677
curl http://localhost:5001/orders/does-not-exist

After a few seconds, open Kibana and go to Observability > APM > Services.

You should see my-python-app listed.

Viewing traces, metrics, and logs in Kibana

Now that telemetry data is flowing, let's walk through where each signal lands in Kibana and how they connect to each other.

Traces

Go to APM > Services > my-python-app > Transactions. Click on a list-orders transaction to open the trace waterfall.

You should see the parent span list-orders with a child span es.search.

The db.system and db.elasticsearch.index attributes we set earlier appear in the span details, and Kibana recognizes the child as an Elasticsearch query.

Failed transactions

For error visibility, open a get-order transaction for the /orders/does-not-exist request.

The transaction details show event.outcome: failure along with the order.id: does-not-exist attribute we attached.

This confirms that error status and custom attributes propagate correctly through the managed endpoint.

Correlated logs

Go to Discover and filter by service.name: my-python-app.

Expand a log document to see the body.text, trace_id, and span_id fields.

Because we emitted logs inside a span context, every log record carries the trace ID.

You can copy the trace_id value and go to APM to jump directly to the trace that produced that log.

Metrics

In Discover, switch to the metrics-* data view.

You should see the app.requests.total counter and app.request.duration histogram arriving at regular intervals.

You can also create a Lens visualization with app.requests.total on the Y axis and @timestamp on the X axis to see request volume over time.

Service map

Kibana builds the service map automatically from the span attributes we set earlier.

Because we tagged our Elasticsearch spans with db.system: elasticsearch, Kibana draws the dependency between my-python-app and elasticsearch, giving you a visual overview of your service and its backends.

Conclusion: full OpenTelemetry observability in Python without a collector

In this walkthrough, we instrumented a Flask API with traces, metrics, and logs using the standard OpenTelemetry SDK and two environment variables, then verified all three signals in Kibana with log-to-trace correlation built in.

The managed OTLP endpoint takes care of scaling, buffering, and durability, so you can focus on your application instead of operating ingestion infrastructure.

From here, consider auto-instrumenting with EDOT Python to remove the manual spans, or adding custom metrics that match your domain.

Next steps

• Explore the Elastic Managed OTLP Endpoint documentation for advanced configuration.
• Try opentelemetry-instrument or EDOT Python to auto-instrument Flask and the Elasticsearch client without manual spans.
• Check the OpenTelemetry Python SDK docs for additional instrumentation options.
• For fine-grained control over histogram temporality, see the managed OTLP endpoint documentation.

OpenLIT is an open-source Observability and Evaluation tool that helps take your LLM apps from playground to debug to production. With OpenLit you get an ability to choose from a range of Integrations (across LLMs, VectorDBs, frameworks, and GPUs) to start tracking LLM performance, usage, and costs without hassle. In this blog we will look at tracking OpenAI and LangChain. to send telemetry to an OpenTelemetry compatible endpoint like Elastic.

Elastic supports OpenTelemetry natively, it can take telemetry directly from the application (via the OpenTelemetry SDKs) or through a native OTel collector. No special agents are needed. Additionally Elastic's EDOT provides a supported set of OTel SDKs and an OTel Collector. In this blog we will connect our application directly to Elastic without a collector for simplicity.

Why Observability Matters for LLM Applications

Monitoring LLM applications is crucial for several reasons.

1. It’s vital to keep track of how often LLMs are being used for usage and cost tracking.

2. Latency is important to track since the response time from the model can vary based on the inputs passed to the LLM.

3. Rate limiting is a common challenge, particularly for external LLMs, as applications depend more on these external API calls. When rate limits are hit, it can hinder these applications from performing their essential functions using these LLMs.

By keeping a close eye on these aspects, you can not only save costs but also avoid hitting request limits, ensuring your LLM applications perform optimally.

What are the signals that you should be looking at?

Using Large Language Models (LLMs) in applications differs from traditional machine learning (ML) models. Primarily, LLMs are often accessed through external API calls instead of being run locally or in-house. It is crucial to capture the sequence of events (using traces), especially in a RAG-based application where there can be events before and after LLM usage. Also, analyzing the aggregated data (through metrics) provides a quick overview like request, tokens and cost is important for optimizing performance and managing costs. Here are the key signals to monitor:

Traces

Request Metadata: This is important in the context of LLMs, given the variety of parameters (like temperature and top_p) that can drastically affect both the response quality and the cost. Specific aspects to monitor are:

1. Temperature: Indicates the level of creativity or randomness desired from the model’s outputs. Varying this parameter can significantly impact the nature of the generated content.

2. top_p: Decides how selective the model is by choosing from a certain percentage of most likely words. A high “top_p” value means the model considers a wider range of words, making the text more varied.

3. Model Name or Version: Essential for tracking over time, as updates to the LLM might affect performance or response characteristics.

4. Prompt Details: The exact inputs sent to the LLM, which, unlike in-house ML models where inputs might be more controlled and homogeneous, can vary wildly and affect output complexity and cost implications.

Response Metadata: Given the API-based interaction with LLMs, tracking the specifics of the response is key for cost management and quality assessment:

1. Tokens: Directly impacts cost and is a measure of response length and complexity.

2. Cost: Critical for budgeting, as API-based costs can scale with the number of requests and the complexity of each request.

3. Completion Details: Similar to the prompt details but from the response perspective, providing insights into the model’s output characteristics and potential areas of inefficiency or unexpected cost.

Metrics

Request Volume: The total number of requests made to the LLM service. This helps in understanding the demand patterns and identifying any anomaly in usage, such as sudden spikes or drops.

Request Duration: The time it takes for a request to be processed and a response to be received from the LLM. This includes network latency and the time the LLM takes to generate a response, providing insights into the performance and reliability of the LLM service.

Costs and Tokens Counters: Keeping track of the total cost accrued and tokens consumed over time is essential for budgeting and cost optimization strategies. Monitoring these metrics can alert you to unexpected increases that may indicate inefficient use of the LLM or the need for optimization.

Implementing Automatic Instrumentation with OpenLIT

OpenLIT automates telemetry data capture, simplifying the process for developers. Here’s a step-by-step guide to setting it up:

1. Install the OpenLIT SDK:

First, you must install the following package: 

pip install openlit

Note: OpenLIT currently supports Python, a popular language for Generative AI. The team is also working on expanding support to JavaScript soon.

2. Get your Elastic APM Credentials

1. Sign in to your Elastic cloud account.

2. Open the side navigation and click on APM under Observability.

3. Make sure the APM Server is running

1. In the APM Agents section, Select OpenTelemetry and directly jump to Step 5 (Configure OpenTelemetry in your application):

2. Copy and save the configuration value for OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS

3. Set Environment Variables:

OpenTelemetry Environment variables for Elastic can be set as follows in linux (or in the code). Elastic OTel Documentation

export OTEL_EXPORTER_OTLP_ENDPOINT="YOUR_ELASTIC_APM_OTLP_URL"
export OTEL_EXPORTER_OTLP_HEADERS="YOUR_ELASTIC_APM_AUTH"

Note: Make sure to replace the space after Bearer with %20:

OTEL_EXPORTER_OTLP_HEADERS=“Authorization=Bearer%20[APIKEY]”

4. Initialize the SDK:

You will need to add the following to the LLM Application code.

import openlit
openlit.init()

Optionally, you can customize the application name and environment by setting the application_name and environment attributes when initializing OpenLIT in your application. These variables configure the OTel attributes service.name and deployment.environment, respectively. For more details on other configuration settings, check out the OpenLIT GitHub Repository.

openlit.init(application_name="YourAppName",environment="Production")

The most popular libraries in GenAI are OpenAI (for accessing LLMs) and Langchain (for orchestrating steps). An example instrumentation of a Langchain and OpenAI based LLM Application will look like:

import getpass
import os
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage
import openlit 

# Auto-instruments LLM and VectorDB calls, sending OTel traces and metrics to the configured endpoint
openlit.init()

os.environ["OPENAI_API_KEY"] = getpass.getpass()
model = ChatOpenAI(model="gpt-4")
messages = [
    SystemMessage(content="Translate the following from English into Italian"),
    HumanMessage(content="hi!"),
]
model.invoke(messages)

Visualizing Data with Kibana

Once your LLM application is instrumented, visualizing the collected data is the next step. Follow the below steps to import a pre-built Kibana dashboard to get yourself started:

1. Copy the dashboard NDJSON provided here and save it in a file with an extension .ndjson.

2. Log into your Elastic Instance.

3. Go to Stack Management > Saved Objects.

4. Click Import and upload your file containing the dashboard NDJSON.

5. Click Import and you should have the dashboard available.

The dashboard provides an in-depth overview of system metrics through eight key areas: Total Successful Requests, Request Duration Distribution, Request Rates, Usage Cost and Tokens, Top GenAI Models, GenAI Requests by Platform and Environment, Token Consumption vs. Cost. These metrics collectively help identify peak usage times, latency issues, rate limits, and resource allocation, facilitating performance tuning and cost management. This comprehensive breakdown aids in understanding LLM performance, ensuring consistent operation across environments, budget needs, and troubleshooting issues, ultimately optimizing overall system efficiency.

Also, you can see OpenTelemetry Traces from OpenLIT in Elastic APM, letting you look into each LLM request in detail. This setup ensures better system efficiency by helping with model performance checks, smooth running across environments, budget planning, and troubleshooting.

Conclusion

Observability is crucial for the efficient operation of LLM applications. OpenTelemetry's open standards and extensive support, combined with Elastic's APM, AIOps, and analytics and OpenLIT's powerful and easy auto-instrumentation for 20+ GenAI tools from LLMs to VectorDBs, enable complete visibility into LLM performance. 

Hopefully, this provides an easy-to-understand walk-through of instrumenting Langchain with OpenTelemetry and OpenLit and how easy it is to send traces into Elastic.

Additional resources for OpenTelemetry with Elastic:

• Independence with OpenTelemetry on Elastic

• Modern observability and security on Kubernetes with Elastic and OpenTelemetry

• 3 models for logging with OpenTelemetry and Elastic

• Adding free and open Elastic APM as part of your Elastic Observability deployment

• Monitor OpenAI API and GPT models with OpenTelemetry and Elastic

• Futureproof your observability platform with OpenTelemetry and Elastic

• Instrumentation resources:

  • Python: Auto-instrumentation, Manual instrumentation

  • Java: Auto-instrumentation, Manual instrumentation 

  • Node.js: Auto-instrumentation, Manual instrumentation

  • .NET: Auto-instrumentation, Manual instrumentation

There are several options to trace for Langchain. Langtrace is one such option. Langtrace is an open-source observability software that lets you capture, debug and analyze traces and metrics from all your applications. Langtrace automatically captures traces from LLM APIs/inferences, Vector Databases, and LLM-based Frameworks. Langtrace stands out due to its seamless integration with popular LLM frameworks and its ability to provide deep insights into complex AI workflows without requiring extensive manual instrumentation.

Langtrace has an SDK, a lightweight library that can be installed and imported into your project to collect traces. The traces are OpenTelemetry-based and can be exported to Elastic without using a Langtrace API key.

OpenTelemetry (OTel) is now broadly accepted as the industry standard for tracing. As one of the major Cloud Native Computing Foundation (CNCF) projects, with as many commits as Kubernetes, it is gaining support from major ISVs and cloud providers delivering support for the framework. 

Hence, many LangChain-based applications will have multiple components beyond just LLM interactions. Using OpenTelemetry with LangChain is essential. 

This blog will cover how you can use Langtrace SDK to trace a simple LangChain Chat app connecting to Azure OpenAI, perform a search in DuckDuckGoSearch and export the output to Elastic.

Pre-requisites:

• An Elastic Cloud account — sign up now, and become familiar with Elastic’s OpenTelemetry configuration

• Have a LangChain app to instrument

• Be familiar with using OpenTelemetry’s Python SDK 

• An account on your favorite LLM (AzureOpen AI), with API keys

• The application we used in this blog, called langchainChat can be found in Github langhcainChat. It is built using Azure OpenAI and DuckDuckGo, but you can easily modify it for your LLM and search of choice.

App Overview and output in Elastic:

To showcase the combined power of Langtrace and Elastic, we created a simple LangChain app that performs the following steps:

1. Takes customer input on the command line. (Queries)

2. Sends these to the Azure OpenAI LLM via a LangChain.

3. Utilizes chain tools to perform a search using DuckDuckGo.

4. The LLM processes the search results and returns the relevant information to the user.

Here is a sample interaction:

Here is what the service view looks like after we ran a few queries.

As you can see, Elastic Observability’s APM recognizes the LangChain app and also shows the average latency, throughput, and transactions. Our average latency is 30s since it takes that log for humans to type the query (twice).

You can also select other tabs to see, dependencies, errors, metrics, and more. One interesting part of Elastic APM is the ability to use universal profiling (eBPF) output also analyzed for this service. Here is what our service’s dependency is (Azure OpenAI) with its average latency, throughput, and failed transactions:

We see Azure OpenAI is on average 4s to give us the results.

If we drill into transactions and look at the trace for our queries on Taylor Swift and Pittsburgh Steelers, we can see both queries and their corresponding spans.

In this trace:

1. The user makes a query

2. Azure OpenAI is called, but it uses a tool (DuckDuckGo) to obtain some results

3. Azure OpenAI reviews and returns a summary to the end user

4. Repeats for another query

We noticed that the other long span (other than Azure OpenAI) is Duckduckgo (~1000ms). We can individually look at the span and review the data:

Configuration:

How do we make all this show up in Elastic? Let's go over the steps:

OpenTelemetry Configuration

To leverage the full capabilities of OpenTelemetry with Langtrace and Elastic, we need to configure the SDK to generate traces and properly set up Elastic’s endpoint and authorization. Detailed instructions can be found in the OpenTelemetry Auto-Instrumentation setup documentation.

OpenTelemetry Environment variables:

For Elastic, you can set the following OpenTelemetry environment variables either in your Linux/Mac environment or directly in the code:

OTEL_EXPORTER_OTLP_ENDPOINT=12345.apm.us-west-2.aws.cloud.es.io:443
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20ZZZZZZZ"
OTEL_RESOURCE_ATTRIBUTES="service.name=langchainChat,service.version=1.0,deployment.environment=production"

In this setup:

• OTEL_EXPORTER_OTLP_ENDPOINT is configured to send traces to Elastic.

• OTEL_EXPORTER_OTLP_HEADERS provides the necessary authorization for the Elastic APM server.

• OTEL_RESOURCE_ATTRIBUTES define key attributes like the service name, version, and deployment environment.

These values can be easily obtained from Elastic’s APM configuration screen under the OpenTelemetry section.

Note: No agent is required; the OTLP trace messages are sent directly to Elastic’s APM server, simplifying the setup process.

Langtrace Library:

OpenTelemetry's auto-instrumentation can be extended to trace additional frameworks using instrumentation packages. For this blog post, you will need to install the Langtrace Python SDK:

pip install langtrace-python-sdk 

After installation, you can add the following code to your project:

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

from langtrace_python_sdk import langtrace, with_langtrace_root_span

Instrumentation:

Once the necessary libraries are installed and the environment variables are configured, you can use auto-instrumentation to trace your application. For example, run the following command to instrument your LangChain application with Elastic:

opentelemetry-instrument python langtrace-elastic-demo.py

The Langtrace OpenTelemetry library correctly captures the flow with minimal manual instrumentation, apart from integrating the OpenTelemetry library. Additionally, the LLM spans captured by Langtrace also include useful metadata such as token counts, model hyper-parameter settings etc. Note that the generated spans follow the OTEL GenAI semantics described here.

In summary, the instrumentation process involves:

1. Capturing customer input from the command line (Queries).

2. Sending these queries to the Azure OpenAI LLM via a LangChain.

3. Utilizing chain tools, such as DuckDuckGo, to perform searches.

4. The LLM processes the results and returns the relevant information to the user.

Conclusion

By combining the power of Langtrace with Elastic, developers can achieve unparalleled visibility into their LangChain applications, ensuring optimized performance and quicker debugging. This powerful combination simplifies the complex task of monitoring AI-driven systems, enabling you to focus on what truly matters—delivering value to your users. Throughout this blog,we've covered the following essential steps and concepts:

• How to manually instrument Langchain with OpenTelemetry

• How to properly initialize OpenTelemetry and add a custom span

• How to easily set the OTLP ENDPOINT and OTLP HEADERS with Elastic without the need for a collector

• How to view and analyze traces in Elastic Observability APM

These steps provide a clear and actionable guide for developers looking to integrate robust tracing capabilities into their LangChain applications.

We hope this guide makes understanding and implementing OpenTelemetry tracing for LangChain simple, ensuring seamless integration with Elastic.

Additional resources for OpenTelemetry with Elastic:

• Independence with OpenTelemetry on Elastic

• Modern observability and security on Kubernetes with Elastic and OpenTelemetry

• 3 models for logging with OpenTelemetry and Elastic

• Adding free and open Elastic APM as part of your Elastic Observability deployment

• Monitor OpenAI API and GPT models with OpenTelemetry and Elastic

• Futureproof your observability platform with OpenTelemetry and Elastic

• Instrumentation resources:

  • Python: Auto-instrumentation, Manual instrumentation

  • Java: Auto-instrumentation, Manual instrumentation 

  • Node.js: Auto-instrumentation, Manual instrumentation

  • .NET: Auto-instrumentation, Manual instrumentation

• Elastic APM - Langtrace AI Docs 

There are several options to trace for LangChain. One is Langsmith, ideal for detailed tracing and a complete breakdown of requests to large language models (LLMs). However, it is specific to Langchain. OpenTelemetry (OTel) is now broadly accepted as the industry standard for tracing. As one of the major Cloud Native Computing Foundation (CNCF) projects, with as many commits as Kubernetes, it is gaining support from major ISVs and cloud providers delivering support for the framework. 

Hence, many LangChain-based applications will have multiple components beyond just LLM interactions. Using OpenTelemetry with LangChain is essential. OpenLLMetry is an available option for tracing Langchain apps in addition to Langsmith.

This blog will show how you can get LangChain tracing into Elastic using the OpenLLMetry library opentelemetry-instrumentation-langchain.

Pre-requisites:<a id="pre-requisites"></a>

• An Elastic Cloud account — sign up now, and become familiar with Elastic’s OpenTelemetry configuration

• Have a LangChain app to instrument

• Be familiar with using OpenTelemetry’s Python SDK 

• An account on your favorite LLM, with API keys

Overview

In highlighting tracing I created a simple LangChain app that does the following:

1. Takes customer input on the command line. (Queries)

2. Sends these to the Azure OpenAI LLM via a LangChain.

3. Chain tools are set to use the search with Tavily 

4. The LLM uses the output which returns the relevant information to the user.

As you can see Elastic Observability’s APM recognizes the LangChain App, and also shows the full trace (done with manual instrumentation):

As the above image shows:

1. The user makes a query
2. Azure OpenAI is called, but it uses a tool (Tavily) to obtain some results
3. Azure OpenAI reviews and returns a summary to the end user

The code was manually instrumented, but auto-instrument can also be used.

OpenTelemetry Configuration<a id="opentelemetry-configuration"></a>

In using OpenTelemetry, we need to configure the SDK to generate traces and configure Elastic’s endpoint and authorization. Instructions can be found in OpenTelemetry Auto-Instrumentation setup documentation.

OpenTelemetry Environment variables:<a id="opentelemetry-environment-variables"></a>

OpenTelemetry Environment variables for Elastic can be set as follows in linux (or in the code).

OTEL_EXPORTER_OTLP_ENDPOINT=12345.apm.us-west-2.aws.cloud.es.io:443
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20ZZZZZZZ"
OTEL_RESOURCE_ATTRIBUTES="service.name=langchainChat,service.version=1.0,deployment.environment=production"

As you can see OTEL_EXPORTER_OTLP_ENDPOINT is set to Elastic, and the corresponding authorization header is also provided. These can be easily obtained from Elastic’s APM configuration screen under OpenTelemetry

Note: No agent is needed, we simply send the OTLP trace messages directly to Elastic’s APM server. 

OpenLLMetry Library:<a id="openllmetry-library"></a>

OpenTelemetry's auto-instrumentation can be extended to trace other frameworks via instrumentation packages.

First, you must install the following package: 

pip install opentelemetry-instrumentation-langchain

This library was developed by OpenLLMetry. 

Then you will need to add the following to the code.

from opentelemetry.instrumentation.langchain import LangchainInstrumentor
LangchainInstrumentor().instrument()

Instrumentation<a id="instrumentation"></a>

Once the libraries are added, and the environment variables are set, you can use auto-instrumentation With auto-instrumentation, the following:

opentelemetry-instrument python tavilyAzureApp.py

The OpenLLMetry library does pull out the flow correctly with minimal manual instrumentation except for adding the OpenLLMetry library.

1. Takes customer input on the command line. (Queries)

2. Sends these to the Azure OpenAI LLM via a Lang chain.

3. Chain tools are set to use the search with Tavily 

4. The LLM uses the output which returns the relevant information to the user.

Manual-instrumentation<a id="manual-instrumentation"></a>

If you want to get more details out of the application, you will need to manually instrument. To get more traces follow my Python instrumentation guide. This guide will walk you through setting up the necessary OpenTelemetry bits, Additionally, you can also look at the documentation in OTel for instrumenting in Python.

Note that the env variables OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_ENDPOINT are set as noted in the section above. You can also set up the OTEL_RESOURCE_ATTRIBUTES.

Once you follow the steps in either guide and initiate the tracer, you will have to essentially just add the span where you want to get more details. In the example below, only one line of code is added for span initialization.

Look at the placement of with tracer.start_as_current_span("getting user query") as span: below

# Creates a tracer from the global tracer provider
tracer = trace.get_tracer("newsQuery")

async def chat_interface():
    print("Welcome to the AI Chat Interface!")
    print("Type 'quit' to exit the chat.")
    
    with tracer.start_as_current_span("getting user query") as span:
        while True:
            user_input = input("\nYou: ").strip()
            
            if user_input.lower() == 'quit':
                print("Thank you for chatting. Goodbye!")
                break
        
            print("AI: Thinking...")
            try:
                result = await chain.ainvoke({"query": user_input})
                print(f"AI: {result.content}")
            except Exception as e:
                print(f"An error occurred: {str(e)}")


if __name__ == "__main__":
    asyncio.run(chat_interface())

As you can see, with manual instrumentation, we get the following trace:

Which calls out when we enter our query function. async def chat_interface()

Conclusion<a id="conclusion"></a>

In this blog, we discussed the following:

• How to manually instrument LangChain with OpenTelemetry

• How to properly initialize OpenTelemetry and add a custom span

• How to easily set the OTLP ENDPOINT and OTLP HEADERS with Elastic without the need for a collector

• See traces in Elastic Observability APM

Hopefully, this provides an easy-to-understand walk-through of instrumenting LangChain with OpenTelemetry and how easy it is to send traces into Elastic.

Additional resources for OpenTelemetry with Elastic:

• Independence with OpenTelemetry on Elastic

• Modern observability and security on Kubernetes with Elastic and OpenTelemetry

• 3 models for logging with OpenTelemetry and Elastic

• Adding free and open Elastic APM as part of your Elastic Observability deployment

• Monitor OpenAI API and GPT models with OpenTelemetry and Elastic

• Futureproof your observability platform with OpenTelemetry and Elastic

• Instrumentation resources:

  • Python: Auto-instrumentation, Manual instrumentation

  • Java: Auto-instrumentation, Manual instrumentation 

  • Node.js: Auto-instrumentation, Manual instrumentation

  • .NET: Auto-instrumentation, Manual instrumentation

Also log into cloud.elastic.co to try out Elastic with a free trial.

Last year, we announced Elastic Distribution of OpenTelemetry (a.k.a. EDOT) language SDKs, which collect logs, traces and metrics from applications. When this was announced, we didn’t yet support Large Language Model (LLM) providers such as OpenAI. This limited insight developers had into Generative AI (GenAI) applications.

In a prior post, we reviewed LLM observability focus, such as token usage, chat latency and knowing which tools (like DuckDuckGo) your application uses. With the right logs, traces and metrics, developers can answer questions like "Which version of a model generated this response?" or "What was the exact chat prompt created by my RAG application?"

In the last six months, Elastic invested a lot of energy alongside others in the OpenTelemetry community towards shared specifications on these areas, including code to collect LLM related logs, metrics and traces. Our goal was to extend the zero code (agent) approach EDOT brings to GenAI use cases.

Today, we announce our first GenAI instrumentation capability in the EDOT language SDKs: OpenAI. Below, you’ll see how to observe GenAI applications using our Python, Node.js and Java EDOT SDKs.

Example application

Many of us may be familiar with ChatGPT, which is frontend for OpenAI’s GPT model family. Using this, you can ask a question and the assistant might reply correctly depending on what you ask and text the LLM was trained on.

Here’s an example of an esoteric question answered by ChatGPT:

Our example application will simply ask this predefined question and print the result. We’ll write it in three languages: Python, JavaScript and Java.

We’ll execute each with a "zero code" (agent) approach, so that logs, metrics and traces are captured and visible in an Elastic Stack configured with Kibana and APM server. If you don’t have a stack running, use instructions from Elasticsearch Labs to set one up.

Regardless of programming language, three variables are needed: the OpenAI API key, the location of your Elastic APM server, and the service name of the application. You’ll write these to a file named .env.

OPENAI_API_KEY=sk-YOUR_API_KEY
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8200
OTEL_SERVICE_NAME=openai-example

By default instrumentations does not capture the content sent to the OpenAI API in the GenAI events sent to logs, if you want to capture it add the following:

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

Each time the application is run, it sends logs, traces and metrics to the APM server, which you can find by querying Kibana like this for the application "openai-example"

http://localhost:5601/app/apm/services/openai-example/transactions

When you choose a trace, you’ll see the LLM request made by the OpenAI SDK, and HTTP traffic caused by it:

Select the logs tab to see the exact request and response to OpenAI. This data is critical for Q/A and evaluation use cases.

You can also go to the Metrics Explorer and make a graph of "gen_ai.client.token.usage" or "gen_ai.client.operation.duration" over all the times you ran the application:

http://localhost:5601/app/metrics/explorer

Continue to see exactly how this application looks and is run, in Python, Java and Node.js. Those already using our EDOT language SDKs will be familiar with how this works.

Python

Assuming you have python installed, the first thing would be to setup a virtual environment and install the required packages: the OpenAI client, a helper tool to read the .env file and our EDOT Python package:

python3 -m venv .venv
source .venv/bin/activate
pip install openai "python-dotenv[cli]" elastic-opentelemetry

Next, run edot-bootstrap which analyzes the code to install any relevant instrumentation available:

edot-bootstrap —-action=install

Now, create your .env file, as described earlier in this article, and the below source code in chat.py

import os

import openai

CHAT_MODEL = os.environ.get("CHAT_MODEL", "gpt-4o-mini")


def main():
  client = openai.Client()

  messages = [
    {
      "role": "user",
        "content": "Answer in up to 3 words: Which ocean contains Bouvet Island?",
    }
  ]

  chat_completion = client.chat.completions.create(model=CHAT_MODEL, messages=messages)
  print(chat_completion.choices[0].message.content)

if __name__ == "__main__":
  main()

Now you can run everything with:

dotenv run -- opentelemetry-instrument python chat.py

Finally, look for a trace for the service named "openai-example" in Kibana. You should see a transaction named "chat gpt-4o-mini".

Rather than copy/pasting above, you can find a working copy of this example (along with the instructions) in the Python EDOT repository here.

Finally, if you would like to try a more comprehensive example, take a look at chatbot-rag-app which uses OpenAI with Elasticsearch’s Elser retrieval model.

Java

There are multiple popular ways to initialize a Java project. Since we are using OpenAI, the first step is to configure the dependency com.openai:openai-java and write the below source as Chat.java.

package openai.example;

import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.*;


final class Chat {

  public static void main(String[] args) {
    String chatModel = System.getenv().getOrDefault("CHAT_MODEL", "gpt-4o-mini");

    OpenAIClient client = OpenAIOkHttpClient.fromEnv();

    String message = "Answer in up to 3 words: Which ocean contains Bouvet Island?";
    ChatCompletionCreateParams params = ChatCompletionCreateParams.builder()
        .addMessage(ChatCompletionUserMessageParam.builder()
          .content(message)
          .build())
        .model(chatModel)
        .build();

    ChatCompletion chatCompletion = client.chat().completions().create(params);
    System.out.println(chatCompletion.choices().get(0).message().content().get());
  }
}

Build the project such that all dependencies are in a single jar. For example, if using Gradle, you would use the com.gradleup.shadow plugin.

Next, create your .env file, as described earlier, and download shdotenv which we’ll use to load it.

curl -O -L https://github.com/ko1nksm/shdotenv/releases/download/v0.14.0/shdotenv
chmod +x ./shdotenv

At this point, you have a jar and configuration you can use to run the OpenAI example. The next step is to download the EDOT Java javaagent binary. This is the part that records and exports logs, metrics and traces.

curl -o elastic-otel-javaagent.jar -L 'https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=co.elastic.otel&a=elastic-otel-javaagent&v=LATEST'

Assuming you assembled a file named openai-example-all.jar, run it with EDOT like this:

./shdotenv java -javaagent:elastic-otel-javaagent.jar -jar openai-example-all.jar

Finally, look for a trace for the service named "openai-example" in Kibana. You should see a transaction named "chat gpt-4o-mini".

Rather than copy/pasting above, you can find a working copy of this example in the EDOT Java source repository here.

Node.js

Assuming you already have npm installed and configured, run the following commands to initialize a project for the example. This includes the openai package and @elastic/opentelemetry-node (EDOT Node.js)

npm init -y
npm install openai @elastic/opentelemetry-node

Next, create your .env file, as described earlier in this article and the below source code in index.js

const {OpenAI} = require('openai');

let chatModel = process.env.CHAT_MODEL ?? 'gpt-4o-mini';

async function main() {
 const client = new OpenAI();
 const completion = await client.chat.completions.create({
  model: chatModel,
  messages: [
   {
    role: 'user',
    content: 'Answer in up to 3 words: Which ocean contains Bouvet Island?',
   },
  ],
 });
 console.log(completion.choices[0].message.content);
}

main();

With this in place, run the above source with EDOT like this:

node --env-file .env --require @elastic/opentelemetry-node index.js

Finally, look for a trace for the service named "openai-example" in Kibana. You should see a transaction named "chat gpt-4o-mini".

Rather than copy/pasting above, you can find a working copy of this example in the EDOT Node.js source repository here.

Finally, if you would like to try a more comprehensive example, take a look at openai-embeddings which uses OpenAI with Elasticsearch as a vector database!

Closing Notes

Above you’ve seen how to observe the official OpenAI SDK in three different languages, using Elastic Distribution of OpenTelemetry (EDOT).

It is important to note that some of the OpenAI SDKs and also OpenTelemetry specifications around generative AI are experimental. If you find this helps you, or find glitches, please join our slack and let us know about it.

Several LLM platforms accept requests from the OpenAI client SDK, by setting OPENAI_BASE_URL and choosing relevant models. During development, we tested against OpenAI Platform and Azure OpenAI Service. We also ran integration tests against Ollama, contributing improvements its OpenAI support released in v0.5.12. Whatever your choice of OpenAI compatible platform, we hope this new tooling helps you understand your LLM usage.

Finally, while the first Generative AI SDK instrumented with EDOT is OpenAI, you’ll see more soon. We are already working on Bedrock, and collaborating with others in the OpenTelemetry community for other platforms. Keep watching this blog for exciting updates.

Background

Elastic is standardizing on OpenTelemetry (OTel) for observability and security data collection. As part of that effort, we are providing distributions of the OpenTelemetry Language SDKs. We have recently released alpha distributions for Java, .NET and Node.js. Our Android and iOS SDKs have been OpenTelemetry-based from the start. The Elastic Distribution for OpenTelemetry Python is the latest addition.

Design choices

We have chosen to provide a lean distribution that does not install all the instrumentations by default but that instead provides tools to do so. We leverage the opentelemetry-bootstrap tool provided by OpenTelemetry Python project to scan the packages installed in your environment and recognizes libraries we are able to instrument. This tool can just report the instrumentations available and optionally is able to install them as well. This allows you to avoid installing packages you are not going to need or instrument libraries you are not interested in tracing.

Getting started

To get started with Elastic Distribution for OpenTelemetry Python you need to install the package elastic-opentelemetry in your project environment. We'll use pip in our examples but you are free to use any python package and environment manager of your choice.

pip install elastic-opentelemetry

Once you have installed our distro you'll have also the opentelemetry-bootstrap command available. Running it:

opentelemetry-bootstrap

will list all available packages for your instrumentation, e.g. you can expect something like the following:

opentelemetry-instrumentation-asyncio==0.46b0
opentelemetry-instrumentation-dbapi==0.46b0
opentelemetry-instrumentation-logging==0.46b0
opentelemetry-instrumentation-sqlite3==0.46b0
opentelemetry-instrumentation-threading==0.46b0
opentelemetry-instrumentation-urllib==0.46b0
opentelemetry-instrumentation-wsgi==0.46b0
opentelemetry-instrumentation-grpc==0.46b0
opentelemetry-instrumentation-requests==0.46b0
opentelemetry-instrumentation-system-metrics==0.46b0
opentelemetry-instrumentation-urllib3==0.46b0

It also provides a command option to install the packages automatically

opentelemetry-bootstrap --action=install

It is advised to run this command every time you release a new version of your application so that you can install or just revise any instrumentation packages for your code.

Some environment variables are needed to provide the needed configuration for instrumenting your services. These mostly concern the destination of your traces but also for easily identifying your service. A service name is required to have your service distinguishable from the others. Then you need to provide the authorization headers for authentication with Elastic Observability cloud and the Elastic cloud endpoint where the data is sent.

The API Key you get from your Elastic cloud serverless project must be URL-encoded, you can do that with the following Python snippet:

from urllib.parse import quote
quote("ApiKey <your api key>)

Once you have all your configuration values you can export via environment variables as below:

export OTEL_RESOURCE_ATTRIBUTES=service.name=<service-name>
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=<url encoded apikey header value>"
export OTEL_EXPORTER_OTLP_ENDPOINT=<your elastic cloud url>

We are done with the configuration and the last piece of the puzzle is wrapping your service invocation with opentelemetry-instrument, the wrapper that provides zero-code instrumentation. Zero-code (or Automatic) instrumentation means that the distribution will set up the OpenTelemetry SDK and enable all the previously installed instrumentations for you. Unfortunately Zero-code instrumentation does not cover all libraries and some — web frameworks in particular — will require minimal manual configuration.

For a web service running with gunicorn it may look like:

opentelemetry-instrument gunicorn main:app

The result is an observable application using the industry-standard OpenTelemetry — offering high-quality instrumentation of many popular Python libraries, a portable API to avoid vendor lock-in and an active community.

Using Elastic Observability, some out-of-the-box benefits you can expect are: rich trace viewing, Service maps, integrated metrics and log analysis, and more.

What's next?

Elastic is committed to helping OpenTelemetry succeed and to helping our customers use OpenTelemetry effectively in their systems. Last year, we donated ECS and continue to work on integrating it with OpenTelemetry Semantic Conventions. More recently, we are working on donating our eBPF-based profiler to OpenTelemetry. We contribute to many of the language SDKs and other OpenTelemetry projects.

In the Python ecosystem we are active reviewers and contributors of both the opentelemetry-python and opentelemetry-python-contrib repositories.

The Elastic Distribution for OpenTelemetry Python is currently an alpha. Please try it out and let us know if it might work for you. Watch for the latest releases here. You can engage with us on the project issue tracker.

We are eager to know your use cases to help you succeed in your Observability journey.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Resources

• https://www.elastic.co/blog/elastic-opentelemetry-sdk-distributions
• https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-java-agent
• https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-dotnet-applications
• https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-node-js
• https://www.elastic.co/observability-labs/blog/manual-instrumentation-python-apps-opentelemetry
• https://www.elastic.co/observability-labs/blog/auto-instrumentation-python-applications-opentelemetry
• https://www.elastic.co/observability-labs/blog/opentelemetry-observability

Observability across the entire stack of modern distributed applications requires data collection, processing, and correlation often in the form of dashboards. Ingesting all system data requires installing agents across stacks, frameworks, and providers — a process that can be challenging and time-consuming for teams who have to deal with version changes, compatibility issues, and proprietary code that doesn't scale as systems change.

Thanks to OpenTelemetry (OTel), DevOps and SRE teams now have a standard way to collect and send data that doesn't rely on proprietary code and have a large support community reducing vendor lock-in.

In a previous blog, we also reviewed how to use the OpenTelemetry demo and connect it to Elastic^®, as well as some of Elastic’s capabilities with OpenTelemetry and Kubernetes.

In this blog, we will show how to use manual instrumentation for OpenTelemetry with the Python service of our application called Elastiflix. This approach is slightly more complex than using automatic instrumentation.

The beauty of this is that there is no need for the otel-collector! This setup enables you to slowly and easily migrate an application to OTel with Elastic according to a timeline that best fits your business.

Application, prerequisites, and config

The application that we use for this blog is called Elastiflix, a movie streaming application. It consists of several micro-services written in .NET, NodeJS, Go, and Python.

Before we instrument our sample application, we will first need to understand how Elastic can receive the telemetry data.

All of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Prerequisites

• An Elastic Cloud account — sign up now
• A clone of the Elastiflix demo application, or your own Python application
• Basic understanding of Docker — potentially install Docker Desktop
• Basic understanding of Python

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Before we begin, let’s look at the non-instrumented code first.

This is our simple Python Flask application that can receive a GET request. (This is a portion of the full main.py file.)

from flask import Flask, request
import sys

import logging
import redis
import os
import ecs_logging
import datetime
import random
import time

redis_host = os.environ.get('REDIS_HOST') or 'localhost'
redis_port = os.environ.get('REDIS_PORT') or 6379

application_port = os.environ.get('APPLICATION_PORT') or 5000

app = Flask(__name__)

# Get the Logger
logger = logging.getLogger("app")
logger.setLevel(logging.DEBUG)

# Add an ECS formatter to the Handler
handler = logging.StreamHandler()
handler.setFormatter(ecs_logging.StdlibFormatter())
logger.addHandler(handler)
logging.getLogger('werkzeug').setLevel(logging.ERROR)
logging.getLogger('werkzeug').addHandler(handler)

r = redis.Redis(host=redis_host, port=redis_port, decode_responses=True)

@app.route('/favorites', methods=['GET'])
def get_favorite_movies():
    user_id = str(request.args.get('user_id'))

    logger.info('Getting favorites for user ' + user_id, extra={
        "event.dataset": "favorite.log",
        "user.id": request.args.get('user_id')
    })

    favorites = r.smembers(user_id)

    # convert to list
    favorites = list(favorites)
    logger.info('User ' + user_id + ' has favorites: ' + str(favorites), extra={
        "event.dataset": "favorite.log",
        "user.id": user_id
    })
    return { "favorites": favorites}

logger.info('App startup')
app.run(host='0.0.0.0', port=application_port)
logger.info('App Stopped')

Step-by-step guide

Step 0. Log in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Install and initialize OpenTelemetry

As a first step, we’ll need to add some additional libraries to our application.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.redis import RedisInstrumentor
from opentelemetry.sdk.resources import Resource

This code imports necessary OpenTelemetry libraries, including those for tracing, exporting, and instrumenting specific libraries like Flask, Requests, and Redis.

Next we read the variables:

OTEL_EXPORTER_OTLP_HEADERS
OTEL_EXPORTER_OTLP_ENDPOINT

And then initialize the exporter.

otel_exporter_otlp_headers = os.environ.get('OTEL_EXPORTER_OTLP_HEADERS')

otel_exporter_otlp_endpoint = os.environ.get('OTEL_EXPORTER_OTLP_ENDPOINT')

exporter = OTLPSpanExporter(endpoint=otel_exporter_otlp_endpoint, headers=otel_exporter_otlp_headers)

In order to pass additional parameters to OpenTelemetry, we will read the OTEL_RESOURCE_ATTRIBUTES variable and convert it into an object.

resource_attributes = os.environ.get('OTEL_RESOURCE_ATTRIBUTES') or 'service.version=1.0,deployment.environment=production'
key_value_pairs = resource_attributes.split(',')
result_dict = {}

for pair in key_value_pairs:
    key, value = pair.split('=')
    result_dict[key] = value

Next, we will then use these parameters to populate the resources configuration.

resourceAttributes = {
     "service.name": otel_service_name,
     "service.version": result_dict['service.version'],
     "deployment.environment": result_dict['deployment.environment']
}

resource = Resource.create(resourceAttributes)

We then set up the trace provider using the previously created resource. The trace provider will allow us to create spans later after getting a tracer instance from it.

Additionally, we specify the use of BatchSPanProcessor. The Span processor is an interface that allows hooks for span start and end method invocations.

In OpenTelemetry, different Span processors are offered. The BatchSPanProcessor batches span and sends them in bulk. Multiple Span processors can be configured to be active at the same time using the MultiSpanProcessor. See OpenTelemetry documentation.

Additionally, we added the resource module. This allows us to specify attributes such as service.name, version, and more. See OpenTelemetry semantic conventions documentation for more details.

provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)

# Sets the global default tracer provider
trace.set_tracer_provider(provider)

# Creates a tracer from the global tracer provider
tracer = trace.get_tracer(otel_service_name)

Finally, because we are using Flask and Redis, we also add the following, which allows us to automatically instrument both Flask and Redis.

Technically you could consider this “cheating.” We are using some parts of the Python auto-instrumentation. However, it’s generally a good approach to resort to using some of the auto-instrumentation modules. This saves you a lot of time, and in addition, it ensures that functionality like distributed tracing will work automatically for any requests you receive or send.

FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()
RedisInstrumentor().instrument()

Step 2. Adding Custom Spans

Now that we have everything added and initialized, we can add custom spans.

If we want to have additional instrumentation for a part of our app, we simply wrap the /favoritesGET function code using Python with:

with tracer.start_as_current_span("add_favorite_movies", set_status_on_exception=True) as span:
        ...

The wrapped code is as follows:

@app.route('/favorites', methods=['GET'])
def get_favorite_movies():
    # add artificial delay if enabled
    if delay_time > 0:
        time.sleep(max(0, random.gauss(delay_time/1000, delay_time/1000/10)))

    with tracer.start_as_current_span("get_favorite_movies") as span:
        user_id = str(request.args.get('user_id'))

        logger.info('Getting favorites for user ' + user_id, extra={
            "event.dataset": "favorite.log",
            "user.id": request.args.get('user_id')
        })

        favorites = r.smembers(user_id)

        # convert to list
        favorites = list(favorites)
        logger.info('User ' + user_id + ' has favorites: ' + str(favorites), extra={
            "event.dataset": "favorite.log",
            "user.id": user_id
        })

Additional code

In addition to modules and span instrumentation, the sample application also checks some environment variables at startup. When sending data to Elastic without an OTel collector, the OTEL_EXPORTER_OTLP_HEADERS variable is required as it contains the authentication. The same is true for OTEL_EXPORTER_OTLP_ENDPOINT, the host where we’ll send the telemetry data.

otel_exporter_otlp_headers = os.environ.get('OTEL_EXPORTER_OTLP_HEADERS')
# fail if secret token not set
if otel_exporter_otlp_headers is None:
    raise Exception('OTEL_EXPORTER_OTLP_HEADERS environment variable not set')


otel_exporter_otlp_endpoint = os.environ.get('OTEL_EXPORTER_OTLP_ENDPOINT')
# fail if server url not set
if otel_exporter_otlp_endpoint is None:
    raise Exception('OTEL_EXPORTER_OTLP_ENDPOINT environment variable not set')
else:
    exporter = OTLPSpanExporter(endpoint=otel_exporter_otlp_endpoint, headers=otel_exporter_otlp_headers)

Final code
For comparison, this is the instrumented code of our sample application. You can find the full source code in GitHub.

from flask import Flask, request
import sys

import logging
import redis
import os
import ecs_logging
import datetime
import random
import time

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

#Using grpc exporter since per the instructions in OTel docs this is needed for any endpoint receiving OTLP.

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.redis import RedisInstrumentor
#from opentelemetry.instrumentation.wsgi import OpenTelemetryMiddleware
from opentelemetry.sdk.resources import Resource

redis_host = os.environ.get('REDIS_HOST') or 'localhost'
redis_port = os.environ.get('REDIS_PORT') or 6379
otel_traces_exporter = os.environ.get('OTEL_TRACES_EXPORTER') or 'otlp'
otel_metrics_exporter = os.environ.get('OTEL_TRACES_EXPORTER') or 'otlp'
environment = os.environ.get('ENVIRONMENT') or 'dev'
otel_service_version = os.environ.get('OTEL_SERVICE_VERSION') or '1.0.0'
resource_attributes = os.environ.get('OTEL_RESOURCE_ATTRIBUTES') or 'service.version=1.0,deployment.environment=production'

otel_exporter_otlp_headers = os.environ.get('OTEL_EXPORTER_OTLP_HEADERS')
# fail if secret token not set
if otel_exporter_otlp_headers is None:
    raise Exception('OTEL_EXPORTER_OTLP_HEADERS environment variable not set')
#else:
#    otel_exporter_otlp_fheaders= f"Authorization=Bearer%20{secret_token}"

otel_exporter_otlp_endpoint = os.environ.get('OTEL_EXPORTER_OTLP_ENDPOINT')
# fail if server url not set
if otel_exporter_otlp_endpoint is None:
    raise Exception('OTEL_EXPORTER_OTLP_ENDPOINT environment variable not set')
else:
    exporter = OTLPSpanExporter(endpoint=otel_exporter_otlp_endpoint, headers=otel_exporter_otlp_headers)


key_value_pairs = resource_attributes.split(',')
result_dict = {}

for pair in key_value_pairs:
    key, value = pair.split('=')
    result_dict[key] = value

resourceAttributes = {
     "service.name": result_dict['service.name'],
     "service.version": result_dict['service.version'],
     "deployment.environment": result_dict['deployment.environment']
#     # Add more attributes as needed
}

resource = Resource.create(resourceAttributes)


provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)

# Sets the global default tracer provider
trace.set_tracer_provider(provider)

# Creates a tracer from the global tracer provider
tracer = trace.get_tracer("favorite")


application_port = os.environ.get('APPLICATION_PORT') or 5000

app = Flask(__name__)


FlaskInstrumentor().instrument_app(app)
#OpenTelemetryMiddleware().instrument()
RequestsInstrumentor().instrument()
RedisInstrumentor().instrument()

#app.wsgi_app = OpenTelemetryMiddleware(app.wsgi_app)

# Get the Logger
logger = logging.getLogger("app")
logger.setLevel(logging.DEBUG)

# Add an ECS formatter to the Handler
handler = logging.StreamHandler()
handler.setFormatter(ecs_logging.StdlibFormatter())
logger.addHandler(handler)
logging.getLogger('werkzeug').setLevel(logging.ERROR)
logging.getLogger('werkzeug').addHandler(handler)

r = redis.Redis(host=redis_host, port=redis_port, decode_responses=True)

@app.route('/favorites', methods=['GET'])
def get_favorite_movies():
    with tracer.start_as_current_span("get_favorite_movies") as span:
        user_id = str(request.args.get('user_id'))

        logger.info('Getting favorites for user ' + user_id, extra={
            "event.dataset": "favorite.log",
            "user.id": request.args.get('user_id')
        })

        favorites = r.smembers(user_id)

        # convert to list
        favorites = list(favorites)
        logger.info('User ' + user_id + ' has favorites: ' + str(favorites), extra={
            "event.dataset": "favorite.log",
            "user.id": user_id
        })
        return { "favorites": favorites}

logger.info('App startup')
app.run(host='0.0.0.0', port=application_port)
logger.info('App Stopped')

Step 3. Running the Docker image with environment variables

As specified in the OTEL documentation, we will use environment variables and pass in the configuration values to enable it to connect with Elastic Observability’s APM server.

Because Elastic accepts OTLP natively, we just need to provide the Endpoint and authentication where the OTEL Exporter needs to send the data, as well as some other environment variables.

Getting Elastic Cloud variables
You can copy the endpoints and token from Kibana^® under the path /app/home#/tutorial/apm.

You will need to copy the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS

Build the image

docker build -t  python-otel-manual-image .

Run the image

docker run \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="<REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT>" \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <REPLACE WITH TOKEN>" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production,service.name=python-favorite-otel-manual" \
       -p 3001:3001 \
       python-otel-manual-image

You can now issue a few requests in order to generate trace data. Note that these requests are expected to return an error, as this service relies on a connection to Redis that you don’t currently have running. As mentioned before, you can find a more complete example using docker-compose here.

curl localhost:500/favorites
# or alternatively issue a request every second

while true; do curl "localhost:5000/favorites"; sleep 1; done;

Step 4. Explore traces, metrics, and logs in Elastic APM

Now that the service is instrumented, you should see the following output in Elastic APM when looking at the transactions section of your Python service:

Notice how this is slightly different from the auto-instrumented version, as we now also have our custom span in this view.

Is it worth it?

This is the million-dollar question. Depending on what level of detail you need, it's potentially necessary to manually instrument. Manual instrumentation lets you add custom spans, custom labels, and metrics where you want or need them. It allows you to get a level of detail that otherwise would not be possible and is oftentimes important for tracking business-specific KPIs.

Your operations, and whether you need to troubleshoot or analyze the performance of specific parts of the code, will dictate when and what to instrument. But it’s helpful to know that you have the option to manually instrument.

If you noticed we didn’t yet instrument metrics, that is another blog. We discussed logs in a previous blog.

Conclusion

In this blog, we discussed the following:

• How to manually instrument Python with OpenTelemetry
• How to properly initialize OpenTelemetry and add a custom span
• How to easily set the OTLP ENDPOINT and OTLP HEADERS with Elastic without the need for a collector

Hopefully, this provides an easy-to-understand walk-through of instrumenting Python with OpenTelemetry and how easy it is to send traces into Elastic.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

One of the main components of Elastic's Synthetic Monitoring is the ability to create user journeys, which can be done with or without code. There is a Synthetics agent,, a CLI tool that guides you through the process of creating both heartbeat monitors and user journeys and deploying your code to Elastic Observability. If you are using code to create user journeys, you are using Playwright under the hood with some additional configuration to make it easier to work with Elastic Observability.

To automatically create user journeys using TypeScript, you can create Playwright tests based on a prompt using Warp, an AI-assisted terminal, Gemini 2.5 Pro, and MCP. This application was built using Python and FastMCP, which wraps the synthetic agent to deploy browser tests to Elastic automatically. This blog post will guide you through how the application works, how to use it, and its development process. You can find the complete code on GitHub.

Solution overview

Currently, this solution is set up to run inside Warp as an MCP server; however, you can also use another client, such as Claude Desktop or Cursor. From there, you create a Python script using FastMCP, which allows you to create functions that are callable by an LLM. Within Warp, you can make a configuration file in JSON that enables you to point to your Python script and pass in all the environment variables you are working with. From there, you'll want to toggle agent mode and ask a question about creating synthetic testing or call the MCP function directly. There are many options for which LLM you can select, be sure to check out Warp's documentation to learn more about the options available.

After that, you should ask a question about creating synthetic testing or call the MCP function you are looking for. The following three functions can be used:

• diagnose_warp_mcp_config Used for debugging environment variable issues that may arise. This function likely won't be needed unless there is an issue with your configuration.

• create_and_deploy_browser_test Will automatically create Playwright tests if given the test name, the URL you want to test, and a schedule. This approach uses a template-based method, rather than a machine learning-based method, and all the tests it outputs will appear similar.

• llm_create_and_deploy_test_from_prompt Similar to create_and_deploy_browser_test, but the main difference is that it uses an LLM to create tests based on a prompt you give it. The tests should reflect the prompt you provided. To run this function you'll provide a test name, URL, prompt, and schedule.

Why create this solution as an MCP server?

The reason this was developed as an MCP server, as opposed to just a standalone script or a standard CLI, is that it can be structured and interacted with in a more conversational manner. It enables an LLM to generate dynamic Playwright testing while maintaining consistent arguments, environment variables, and responses to ensure accuracy and reliability. Thus, it becomes a reliable workflow that other agents or developers can compose with additional tools. In other words, the MCP layer turns your LLM-based test authoring into a standardized, reusable capability instead of a one-off script. To learn more about the direction of MCP, be sure to check out our article on the topic.

Implementation considerations

When creating a solution like this one, one thing to be mindful of is your use of tokens. An early version of this solution took approximately twenty minutes to create synthetic tests and ultimately led to severe rate-limiting.

Another issue faced during the building process was striking a balance between creating a template that facilitates the creation of a Playwright script and having an LLM create Playwright scripts based on prompts that didn't feel cookie-cutter. While using a more LLM approach an issue faced was that the scripts often didn't work or were based on parameters that didn't exist and a more templated approach was more reliable but felt repetitive. The final version of this solution attempted to balance this by using elements of the template while adjusting the LLM parameter of temperature, which controls the randomness or creativity of a large language model's output.

While testing this solution, a failing test also emerged that required navigating past a pop-up. In more complex cases, this may serve as a building block that requires additional domain knowledge to create a complete passing Playwright test.

How to get started

Prerequisites

• The version of Python that is used is Python 3.12.1 but you can use any version of Python higher than 3.10.
• This application uses Elastic Observability version 9.1.2, but you can use any version of Elastics Observability that is higher than 8.10. You can also use Elastic Cloud Serverless as well.
• You will also need an OpenAI API key to use the LLM capabilities of this application. You will want to configure an environment variable for your OpenAI API Key, which you can find on the API keys page in OpenAI's developer portal.

Step 1: Install the packages and clone the repository

In order for this MCP server to run locally you will need to install the the following packages:

pip install fastmcp openai
npm install -g playwright @elastic/synthetics

You will use FastMCP 2.0 to create the MCP server, and OpenAI to generate tests based on prompts that you provide. Additionally, you will want to clone the repository to obtain a local copy of the server.

Step 2: Set up a configuration file in Warp

Inside of Warp, you will want to go to the side panel, where it says MCP servers and where it says “add”.

After that, you will be prompted to add a JSON configuration file that should resemble the following. Be sure to add your own Kibana URL, update the correct path, and include your own keys and tokens.

{
 "elastic-synthetics": {
   "command": "python",
   "args": ["elastic_synthetics_server.py"],
   "env": {
     "PYTHONPATH": ".",
     "ELASTIC_KIBANA_URL": "https://your-kibana-url.elastic-cloud.com",
     "ELASTIC_API_KEY": "your-api-key-here",
     "ELASTIC_PROJECT_ID": "mcp-synthetics-demo",
     "ELASTIC_SPACE": "default",
     "ELASTIC_AUTO_PUSH": "true",
     "ELASTIC_USE_JAVASCRIPT": "false",
     "ELASTIC_INSTALL_DEPENDENCIES": "true",
     "OPENAI_API_KEY": "sk-your-openai-key",
     "LLM_MODEL": "gpt-4o"
   },
   "working_directory": "/path/to/your/file",
   "start_on_launch": true 
   }
}

Step 3: Ask a question or call the tools directly

Now that you've set up locally, you will want to toggle agent mode and select the LLM you wish to use. The reason why Gemini-Pro-2.5 was chosen for this blog post is that it provides a straightforward answer, while other LLMs selected returned a very lengthy response.

To start using the MCP tools, from your MCP server, you can ask a question that contains the test name, URL, prompt, and schedule.

You can also call the directly by typing llm_create_and_deploy_test_from_prompt() and the program will prompt you for the relevant details:

Inside Kibana, you should see your monitor listed if you click under Applications and select Monitors listed under Synthetics. You can also find a link to your monitor in the response of your MCP tool.

What's Going On Inside

This code sample consists of three primary functions, which are MCP tools that you can call from your MCP client, including diagnose_warp_mcp_config, create_and_deploy_browser_test and llm_create_and_deploy_test_from_prompt.

Debugging environment issues

There were various issues that came up while creating this application around environment variable loading, so there was a need to create an MCP that could be called depending on errors that may be present.

The tool diagnose_warp_mcp_config kicks off with a decorator @mcp.tool() which allows it to be called and listed in the list of available tools. This tool is designed to help debug issues with Elastic-specific environment variables for troubleshooting purposes. First, it loads in the environment variables and looks for the Elastic specific variables, after it does some security masking so it doesn't show any variables and hides sensitive information like API keys in the output, showing only the first eight characters followed by "...". This tool determines if the minimum required credentials (Kibana URL and API Key) are present to proceed with deployment and provides a report letting you know to address any issues that may exist.

@mcp.tool()
def diagnose_warp_mcp_config() -> Dict[str, Any]:
   """Diagnose Warp MCP environment configuration for Elastic Synthetics"""
   try:
       env_vars = load_env_from_warp_mcp()
      
       # Check for required variables
       kibana_url = env_vars.get('ELASTIC_KIBANA_URL') or env_vars.get('KIBANA_URL')
       api_key = env_vars.get('ELASTIC_API_KEY') or env_vars.get('API_KEY')
       project_id = env_vars.get('ELASTIC_PROJECT_ID') or env_vars.get('PROJECT_ID')
       space = env_vars.get('ELASTIC_SPACE') or env_vars.get('SPACE', 'default')
      
       # Mask sensitive values for display
       masked_vars = {}
       for key, value in env_vars.items():
           if 'API_KEY' in key or 'TOKEN' in key:
               masked_vars[key] = f"{value[:8]}..." if value and len(value) > 8 else "***"
           else:
               masked_vars[key] = value
      
       deployment_ready = bool(kibana_url and api_key)
      
       return safe_json_response({
           "status": "success",
           "environment_variables": masked_vars,
           "required_check": {
               "kibana_url": bool(kibana_url),
               "api_key": bool(api_key),
               "project_id": bool(project_id),
               "space": bool(space)
           },
           "deployment_ready": deployment_ready,
           "recommendations": [
               "Environment variables detected" if env_vars else "No environment variables found",
               "Kibana URL configured" if kibana_url else "Missing ELASTIC_KIBANA_URL or KIBANA_URL",
               "API Key configured" if api_key else "Missing ELASTIC_API_KEY or API_KEY",
               "Ready for deployment" if deployment_ready else "Missing required credentials"
           ]
       })
      
   except Exception as e:
       return safe_json_response({
           "status": "error",
           "error": str(e),
           "error_type": type(e).__name__
       })

Creating synthetic tests based on a template

While developing this solution to generate tests based on a prompt, the process wasn't always smooth. Early versions encountered issues with accuracy, hallucinations, and the creation of loops. To make progress, a version that relied on creating a test template to verify the mechanics of the solution, such as whether the test could pass and be deployed to Elastic correctly, was a logical next step.

This solution automates the entire process of creating a synthetic browser test that will regularly check if a website is working correctly, then deploys it to Elastic Observability Synthetics. Similar to diagnose_warp_mcp_config, the MCP tool create_and_deploy_browser_test starts with the decorator @mcp.tool() and checks to make sure that the proper environment variables are loaded.

From there, it creates a TypeScript test file that is based on templates and generates dynamic test steps based on the target website's characteristics, including navigating to the website, verifying the page title exists, checking page load performance, taking a screenshot, verifying page content is visible, and finally saves the test file in a synthetic_tests directory.

Finally, it wraps Elastic's CLI tool @elastic/synthetics to push the test to Kibana, allowing you to set which geographic locations to run tests from, how often to run the test, and the project and workspace settings.

You check out the full code for this MCP tool here.

Creating synthetic tests based on a prompt

While creating browser tests based on a templated approach is a good starting point, it felt generic and cookie-cutter. But it made a helpful structure to build an LLM-based function on top of.

The MCP tool llm_create_and_deploy_test_from_prompt begins by ensuring that basic parameters, including locations, schedule, and directories, are listed. Additionally, it aims to learn more about the target website to inform the AI and initialize the OpenAI client and model, which is GPT-4o.

After setting up the LLM, it converts natural language requests into actual Playwright test code, then cleans and validates the AI-generated code to prevent issues like injection attacks or malformed syntax. It draws inspiration from the templated approach, wrapping AI-generated steps within a proven, reliable test framework template. Finally, it deploys the test to Elastic in a similar manner to the previous tool.

You can find the code for this tool here.

Conclusion and next steps

Synthetic monitoring in Elastic Observability makes it easy to test complete user journeys and keep your site reliable, with simple setup and a Playwright integration. A tool like this can provide a starting point for tests that you can iterate on after.

A solution like this is just the start of an MCP implementation that automatically generates Playwright tests for you and can be expanded in the future to include heartbeat monitors, utilize the Playwright MCP server, or consider experimenting with Claude for Chrome to create synthetic testing.

Check out more articles on Observability Labs on Synthetic Monitoring

• Distributed components: A single request might hit an API gateway, retrieve data from a feature store, evaluate a predictive model in a Python inference service, query a vector database, and call an external LLM.
• Non-determinism: AI agents make autonomous decisions and tool calls. If an agent fails, you need a full trace to understand its reasoning loop and what external tools it tried to invoke.
• Context dependence: You don't just care that an error happened; you need to know what model version was running, what hyperparameters were used, what the input data looked like, what was the commit that made that change. Many of these attributes are custom to your app, and you need an Observability environment that has the flexibility of creating new parameters on the fly and use them to find and fix issues.

On top of that, with the increased use of AI agents to generate code and make autonomous decisions, Observability becomes key to understanding what is working and what is not. It creates a critical feedback loop to quickly fix problems. More than ever, ML and AI applications need to adopt the best practices of mature software engineering systems to succeed.

This guide shows how to use OpenTelemetry and Elastic to correlate traces, logs, and metrics to track runs, compare model behavior, and trace requests across Python and Go services with one shared context.

Problem context: why AI systems are harder to debug

Traditional services already have distributed failure modes, but ML and AI systems add more moving parts:

• notebook experiments and ad hoc jobs
• batch training and evaluation pipelines
• online inference services
• external API calls, including LLM providers
• changing model versions and hyperparameters

When one prediction path gets slower or starts failing, plain isolated logs do not answer enough questions. You need to correlate:

• what ran (run ID, model version, parameters)
• where time was spent (pipeline stage latencies)
• what was the result (model stats, predictions, API calls, compare with other runs)
• what changed (code, data, dependencies)

In a future blog post, we'll show you how to set up automatic RCA and remediations with Elastic Workflows and our AI integrations. But as a first step, ML and AI pipelines need a robust Observability framework, which is very easy to set up with OpenTelemetry and Elastic.

Solution overview

OpenTelemetry gives you a standard way to emit traces, metrics, and logs. Elastic provides full OpenTelemetry ingestion, giving you a single place to store and query that telemetry. Kibana's UI is fully integrated with OpenTelemetry, allowing you to explore your services, service dependencies, service latencies, spans, and metrics out-of-the-box.

You can start with two deployment options:

• Cloud: send OpenTelemetry data directly to Elastic Cloud Managed OTLP Endpoint (mOTLP docs), without the overhead of managing collectors
• Local: run Elastic and the EDOT Collector with start-local, the EDOT Collector will be automatically listening for OTLP data in localhost:4317

Both options let you keep your application code unchanged for the initial implementation.

Step 1: zero-code baseline for Python services

Start by just installing the Elastic Distribution of OpenTelemetry Python (EDOT Python) package and using the opentelemetry-instrument wrapper to run your script. By simply running your script with this wrapper—without modifying your application code—your Python services begin emitting standard telemetry right away. This includes any logs exported via logging, alongside metrics and traces for auto-instrumented libraries. This data can be routed directly to Elastic's managed OTLP endpoint or a local EDOT collector.

pip install elastic-opentelemetry
edot-bootstrap --action=install

Export the OpenTelemetry environment variables, then run opentelemetry-instrument on your script to enable auto-instrumentation.

export OTEL_EXPORTER_OTLP_ENDPOINT="https://<motlp-endpoint>" # No need when using start-local with EDOT
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey <key>" # No need when using start-local with EDOT
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=prod,service.version=1.0.0" # Set the environment and version for your app
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
export ELASTIC_OTEL_SYSTEM_METRICS_ENABLED=true
export OTEL_METRIC_EXPORT_INTERVAL=5000 # Choose the interval for your application metrics

opentelemetry-instrument --service_name=<pipeline-name> python3 <your_python_script>.py # Set your chosen name for your service

With this baseline, you can quickly get:

• Centralized logs with trace context. Any logs exported via logging will be searchable in Elastic and Kibana, with the ability to perform full-text search on your logs
• Set alerting on log errors
• Process and system metrics. System and process metrics from the execution will be automatically exported to Elastic. You can visualize them, and analyse memory usage (leaks, OOM errors), CPU utilization (Bottlenecks / Spikes), thread counts, disk I/O bottlenecks or network I/O saturation.
• Set alerting on metrics
• Spans for auto instrumented libraries
• Service latency baselines and error trends
• Set manual or Anomaly detection alerting on error rates, latencies or throughput
• Correlate logs, metrics, and traces in a single shared context to quickly find the root cause of issues, using OpenTelemetry for instrumentation and Elastic for analysis.

Once ingested, Kibana immediately populates out-of-the-box dashboards. You can explore full-text searchable logs, monitor system and process metrics, investigate auto-instrumented trace waterfalls, map out your ML dependencies with service maps, and easily set up alerts for latency spikes, memory or CPU usage or log errors.

For LLM-specific observability, OpenTelemetry provides official Semantic Conventions for Generative AI to standardize how you track token usage, model names, and prompts. These semantic conventions are still in development and not stable yet. Some instrumentations for the most used libraries in this space are being developed as part of the OpenTelemetry Python Contrib repository. Alternatively you can implement these conventions manually in your custom spans. LLM related OpenTelemetry logs, metrics and traces sent to Elastic will be in context and automatically correlated with the rest of your application or stack of applications.

Step 2: add ML-specific context with custom spans and log fields

Auto-instrumentation is a starting point. For ML and AI Ops, add explicit spans around business stages and attach run metadata. Elastic's schema flexibility and dynamic mappings make it a perfect fit for custom attributes or metrics that are exclusive to your pipelines or specific experiments. There is no need to know what the data will look like before writing it. You have the flexibility of creating new parameters on the fly, Elastic maps them automatically, and you can track them instantly.

Add custom fields and metric-like values as structured log fields so you can chart and alert on them later:

logger.info("training metrics", extra={
    "ml.run_id": run_id,
    "ml.training_accuracy": train_accuracy,
    "ml.validation_accuracy": val_accuracy,
    "ml.drift_detected": drift_detected,
})

Because Elastic handles dynamic mapping, any custom metrics or attributes you log, like model ids, training accuracy or drift detection, are instantly indexed and available to search in Discover or visualize via Dashboards.

This makes dashboards and rules practical:

• alert when ml.validation_accuracy < 0.8
• alert when ml.drift_detected == true
• compare stage latency by ml.model_version

You can use these custom attributes to build targeted visualizations, and trigger alerts when ML-specific metrics like validation accuracy drop below a critical threshold.

Adding custom spans allows you to break down the specific stages of your ML pipelines, such as data loading and model training, wrapping them in their own measurable execution blocks, and analyze average latency or error rates for specific pipeline stages.

from opentelemetry import trace

tracer = trace.get_tracer("ml.pipeline")

with tracer.start_as_current_span("load_data") as span:
    span.set_attribute("ml.run_id", run_id)
    span.set_attribute("ml.dataset", dataset_source)
    load_data()

with tracer.start_as_current_span("train_model") as span:
    span.set_attribute("ml.model_version", model_version)
    span.set_attribute("ml.learning_rate", learning_rate)
    train_model()

Custom spans will be reflected in the APM UI alongside your traces. So you can explore their latency, impact in total execution, stack traces, error rates.

Step 3: trace across Python and Go in production

Real inference paths often cross service boundaries. For example:

In a production environment, a user request might pass through a Go-based API before hitting your Python ML inference service. OpenTelemetry ensures tracing context is preserved seamlessly across these boundaries.

In our example, we have a simple Go HTTP service that acts as the entry point and demonstrates OpenTelemetry instrumentation in Go. This REST API service stores and retrieves ML predictions by querying Elasticsearch based on data IDs from the source dataset. All of its endpoints are natively instrumented with OTel spans.

The full request lifecycle looks like this:

1. The Go API receives the client request.
2. It searches Elasticsearch for an existing prediction or calls the Python model service to run inference.
3. The Python service loads features, runs the model, and returns predictions.

When both services use OpenTelemetry, trace context is propagated automatically through headers. In Elastic, you can inspect one end-to-end trace and locate latency or errors by service and span.

The resulting distributed trace in Elastic pieces the entire journey together. You can see the exact breakdown of time spent in the Go API versus the Python model, and correlate logs from both services in a single unified view.

Validation checklist

After instrumentation, validate with a short runbook:

1. Confirm logs, metrics, and traces arrive for each service.
2. Verify your custom attributes (e.g. run_id, model_version, llm_ground_truth_score) are present in traces and logs.
3. Compare p95 latency per stage (load_data, train_model, predict).
4. Trigger a controlled failure and confirm error traces include stack context.
5. Test one rule for errors, one rule for latency spikes, and one rule for model-quality fields. Set up a connector and attach it to the rule to reach you in Slack, email, or trigger an auto-remediation workflow.

Conclusion and next steps

OpenTelemetry gives ML and AI teams a unified telemetry layer, while Elastic makes that data instantly queryable and actionable across your entire lifecycle—from notebook experiments to production inference. By starting with zero-code instrumentation and incrementally adding ML-specific attributes and cross-language tracing, your team can easily adopt the Observability best practices of mature software engineering systems and succeed in the new era of complex AI operations.

Try this setup in Elastic Cloud, and use mOTLP for a managed ingest path. If you want a local sandbox first, start with Elastic start-local + EDOT Collector.

Introduction

Data pipelines, particularly ETL processes, form the backbone of modern data architectures. These pipelines are responsible for extracting raw data from various sources, transforming it into meaningful information, and loading it into data warehouses or data lakes for analysis and reporting.

In our organization, we have Python-based ETL scripts that play a pivotal role in exporting and processing data from Elasticsearch (ES) clusters and loading it into Google BigQuery (BQ). This processed data then feeds into DBT (Data Build Tool) models, which further refine the data and make it available for analytics and reporting. To see the full architecture and learn how we monitor our DBT pipelines with Elastic see Monitor your DBT pipelines with Elastic Observability. In this article we focus on the ETL scripts. Given the critical nature of these scripts, it is imperative to set up mechanisms to control and ensure the quality of the data they generate.

The strategies discussed here can be extended to any script or application that handles data processing or machine learning models, regardless of the programming language used as long as there exists a corresponding agent that supports OTEL instrumentation.

Motivation

Observability in data pipelines involves monitoring the entire lifecycle of data processing to ensure that everything works as expected. It includes:

1. Data Quality Control:

• Detecting anomalies in the data, such as unexpected drops in record counts.
• Verifying that data transformations are applied correctly and consistently.
• Ensuring the integrity and accuracy of the data loaded into the data warehouse.

2. Performance Monitoring:

• Tracking the execution time of ETL scripts to identify bottlenecks and optimize performance.
• Monitoring resource usage, such as memory and CPU consumption, to ensure efficient use of infrastructure.

3. Real-time Alerting:

• Setting up alerts for immediate notification of issues such as failed ETL jobs, data quality issues, or performance degradation.
• Identify the root case of such incidents
• Proactively addressing incidents to minimize downtime and impact on business operations

Issues such as failed ETL jobs, can even point to larger infrastructure or data source data quality issues.

Steps for Instrumentation

Here are the steps to automatically instrument your Python script for exporting OTEL traces, metrics, and logs.

Step 1: Import Required Libraries

We first need to install the following libraries.

pip install elastic-opentelemetry google-cloud-bigquery[opentelemetry]

You can also them to your project's requirements.txt file and install them with pip install -r requirements.txt.

Explanation of Dependencies

1. elastic-opentelemetry: This package is the Elastic Distribution for OpenTelemetry Python. Under the hood it will install the following packages:

   • opentelemetry-distro: This package is a convenience distribution of OpenTelemetry, which includes the OpenTelemetry SDK, APIs, and various instrumentation packages. It simplifies the setup and configuration of OpenTelemetry in your application.

   • opentelemetry-exporter-otlp: This package provides an exporter that sends telemetry data to the OpenTelemetry Collector or any other endpoint that supports the OpenTelemetry Protocol (OTLP). This includes traces, metrics, and logs.

   • opentelemetry-instrumentation-system-metrics: This package provides instrumentation for collecting system metrics, such as CPU usage, memory usage, and other system-level metrics.

2. google-cloud-bigquery[opentelemetry]: This package integrates Google Cloud BigQuery with OpenTelemetry, allowing you to trace and monitor BigQuery operations.

Step 2: Export OTEL Variables

Set the necessary OpenTelemetry (OTEL) variables by getting the configuration from APM OTEL from Elastic.

Go to APM -> Services -> Add data (top left corner).

In this section you will find the steps how to configure various APM agents. Navigate to OpenTelemetry to find the variables that you need to export.

Find OTLP Endpoint:

• Look for the section related to OpenTelemetry or OTLP configuration.
• The OTEL_EXPORTER_OTLP_ENDPOINT is typically provided as part of the setup instructions for integrating OpenTelemetry with Elastic APM. It might look something like https://<your-apm-server>/otlp.

Obtain OTLP Headers:

• In the same section, you should find instructions or a field for OTLP headers. These headers are often used for authentication purposes.
• Copy the necessary headers provided by the interface. They might look like Authorization: Bearer <your-token>.

Note: Notice you need to replace the whitespace between Bearer and your token with %20 in the OTEL_EXPORTER_OTLP_HEADERS variable when using Python.

Alternatively you can use a different approach for authentication using API keys (see instructions). If you are using our serverless offering you will need to use this approach instead.

Set up the variables:

• Replace the placeholders in your script with the actual values obtained from the Elastic APM interface and execute it in your shell via the source command source env.sh.

Below is a script to set these variables:

#!/bin/bash
echo "--- :otel: Setting OTEL variables"
export OTEL_EXPORTER_OTLP_ENDPOINT='https://your-apm-server/otlp:443'
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=Bearer%20your-token'
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
export OTEL_PYTHON_LOG_CORRELATION=true
export ELASTIC_OTEL_SYSTEM_METRICS_ENABLED=true
export OTEL_METRIC_EXPORT_INTERVAL=5000
export OTEL_LOGS_EXPORTER="otlp,console"

With these variables set, we are ready for auto-instrumentation without needing to add anything to the code.

Explanation of Variables

• OTEL_EXPORTER_OTLP_ENDPOINT: This variable specifies the endpoint to which OTLP data (traces, metrics, logs) will be sent. Replace placeholder with your actual OTLP endpoint.

• OTEL_EXPORTER_OTLP_HEADERS: This variable specifies any headers required for authentication or other purposes when sending OTLP data. Replace placeholder with your actual OTLP headers.

• OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED: This variable enables auto-instrumentation for logging in Python, allowing logs to be automatically enriched with trace context.

• OTEL_PYTHON_LOG_CORRELATION: This variable enables log correlation, which includes trace context in log entries to correlate logs with traces.

• OTEL_METRIC_EXPORT_INTERVAL: This variable specifies the metric export interval in milliseconds, in this case 5s.

• OTEL_LOGS_EXPORTER: This variable specifies the exporter to use for logs. Setting it to "otlp" means that logs will be exported using the OTLP protocol. Adding "console" specifies that logs should be exported to both the OTLP endpoint and the console. In our case for better visibility on the infa side, we choose to export to console as well.

• ELASTIC_OTEL_SYSTEM_METRICS_ENABLED: It is needed to use this variable when using the Elastic distribution as by default it is set to false.

Note: OTEL_METRICS_EXPORTER and OTEL_TRACES_EXPORTER: This variables specify the exporter to use for metrics/traces, and are set to "otlp" by default, which means that metrics and traces will be exported using the OTLP protocol.

Running Python ETLs

We run Python ETLs with the following command:

OTEL_RESOURCE_ATTRIBUTES="service.name=x-ETL,service.version=1.0,deployment.environment=production" && opentelemetry-instrument python3 X_ETL.py 

Explanation of the Command

• OTEL_RESOURCE_ATTRIBUTES: This variable specifies additional resource attributes, such as service name, service version and deployment environment, that will be included in all telemetry data, you can customize these values per your needs. You can use a different service name for each script.

• opentelemetry-instrument: This command auto-instruments the specified Python script for OpenTelemetry. It sets up the necessary hooks to collect traces, metrics, and logs.

• python3 X_ETL.py: This runs the specified Python script (X_ETL.py).

Tracing

We export the traces via the default OTLP protocol.

Tracing is a key aspect of monitoring and understanding the performance of applications. Spans form the building blocks of tracing. They encapsulate detailed information about the execution of specific code paths. They record the start and end times of activities and can have hierarchical relationships with other spans, forming a parent/child structure.

Spans include essential attributes such as transaction IDs, parent IDs, start times, durations, names, types, subtypes, and actions. Additionally, spans may contain stack traces, which provide a detailed view of function calls, including attributes like function name, file path, and line number, which is especially useful for debugging. These attributes help us analyze the script's execution flow, identify performance issues, and enhance optimization efforts.

With the default instrumentation, the whole Python script would be a single span. In our case we have decided to manually add specific spans per the different phases of the Python process, to be able to measure their latency, throughput, error rate, etc individually. This is how we define spans manually:

from opentelemetry import trace

if __name__ == "__main__":

    tracer = trace.get_tracer("main")
    with tracer.start_as_current_span("initialization") as span:
            # Init code
            … 
    with tracer.start_as_current_span("search") as span:
            # Step 1 - Search code
            …
   with tracer.start_as_current_span("transform") as span:
           # Step 2 - Transform code
           …
   with tracer.start_as_current_span("load") as span:
           # Step 3 - Load code
           …

You can explore traces in the APM interface as shown below.

Metrics

We export metrics via the default OTLP protocol as well, such as CPU usage and memory. No extra code needs to be added in the script itself.

Note: Remember to set ELASTIC_OTEL_SYSTEM_METRICS_ENABLED to true.

Logging

We export logs via the default OTLP protocol as well.

For logging, we modify the logging calls to add extra fields using a dictionary structure (bq_fields) as shown below:

        job.result()  # Waits for table load to complete
        job_details = client.get_job(job.job_id)  # Get job details

        # Extract job information
        bq_fields = {
            # "slot_time_ms": job_details.slot_ms,
            "job_id": job_details.job_id,
            "job_type": job_details.job_type,
            "state": job_details.state,
            "path": job_details.path,
            "job_created": job_details.created.isoformat(),
            "job_ended": job_details.ended.isoformat(),
            "execution_time_ms": (
                job_details.ended - job_details.created
            ).total_seconds()
            * 1000,
            "bytes_processed": job_details.output_bytes,
            "rows_affected": job_details.output_rows,
            "destination_table": job_details.destination.table_id,
            "event": "BigQuery Load Job", # Custom event type
            "status": "success", # Status of the step (success/error)
            "category": category # ETL category tag 
        }

        logging.info("BigQuery load operation successful", extra=bq_fields)

This code shows how to extract BQ job stats, execution time, bytes processed, rows affected and destination table among them. You can add other metadata like we do such as custom event type, status, and category.

Any calls to logging (of all levels above the set threshold, in this case INFO logging.getLogger().setLevel(logging.INFO)) will create a log that will be exported to Elastic. This means that in Python scripts that already use logging there is no need to make any changes to export logs to Elastic.

For each of the log messages, you can go into the details view (click on the … when you hover over the log line and go into View details) to examine the metadata attached to the log message. You can also explore the logs in Discover.

Explanation of Logging Modification

• logging.info: This logs an informational message. The message "BigQuery load operation successful" is logged.

• extra=bq_fields: This adds additional context to the log entry using the bq_fields dictionary. This context can include details making the log entries more informative and easier to analyze. This data will be later used to set up alerts and data anomaly detection jobs.

Monitoring in Elastic's APM

As shown, we can examine traces, metrics, and logs in the APM interface. To make the most out of this data, we make use on top of nearly the whole suit of features in Elastic Observability alongside Elastic Analytic's ML capabilities.

Rules and Alerts

We can set up rules and alerts to detect anomalies, errors, and performance issues in our scripts.

The error count threshold rule is used to create a trigger when the number of errors in a service exceeds a defined threshold.

To create the rule go to Alerts and Insights -> Rules -> Create Rule -> Error count threshold, set the error count threshold, the service or environment you want to monitor (you can also set an error grouping key across services), how often to run the check, and choose a connector.

Next, we create a rule of type custom threshold on a given ETL logs data view (create one for your index) filtering on "labels.status: error" to get all the logs with status error from any of the steps of the ETL which have failed. The rule condition is set to document count > 0. In our case, in the last section of the rule config, we also set up Slack alerts every time the rule is activated. You can pick from a long list of connectors Elastic supports.

Then we can set up alerts for failures. We add status to the logs metadata as shown in the code sample below for each of the steps in the ETLs. It then becomes available in ES via labels.status.

logging.info(
            "Elasticsearch search operation successful",
            extra={
                "event": "Elasticsearch Search",
                "status": "success",
                "category": category,
                "index": index,
            },
        )

More Rules

We could also add rules to detect anomalies in the execution time of the different spans we define. This is done by selecting transaction/span -> Alerts and rules -> Custom threshold rule -> Latency. In the example below, we want to generate an alert whenever the search step takes more than 25s.

Alternatively, for finer-grained control, you can go with Alerts and rules -> Anomaly rule, set up an anomaly job, and pick a threshold severity level.

Anomaly detection job

In this example we set an anomaly detection job on the number of documents before transform.

We set up an Anomaly Detection jobs on the number of document before the transform using the [Single metric job] (https://www.elastic.co/guide/en/machine-learning/current/ml-anomaly-detection-job-types.html#multi-metric-jobs) to detect any anomalies with the incoming data source.

In the last step, you can create alerting similarly to what we did before to receive alerts whenever there is an anomaly detected, by setting up a severity level threshold. Using the anomaly score which is assigned to every anomaly, every anomaly is characterized by a severity level.

Similarly to the previous example, we set up a Slack connector to receive alerts whenever an anomaly is detected.

You can go to your custom dashboard by going to Add Panel -> ML -> Anomaly Swim Lane -> Pick your job.

Similarly, we add jobs for the number of documents after the transform, and a Multi-Metric one on the execution_time_ms, bytes_processed and rows_affected similarly to how it was done in Monitor your DBT pipelines with Elastic Observability.

Custom Dashboard

Now that your logs, metrics, and traces are in Elastic, you can use the full potential of our Kibana dashboards to extract the most from them. We can create a custom dashboard like the following one: a pie chart based on labels.event (category field for every type of step in the ETLs), a chart for every type of step broken down by status, a timeline of steps broken down by status, BQ stats for the ETL, and anomaly detection swim lane panels for the various anomaly jobs.

Conclusion

Elastic’s APM, in combination with other Observability and ML features, provides a unified view of our data pipelines, allowing us to bring a lot of value with minimal code changes:

• Logging of new logs (no need to add custom logging) alongside their execution context
• Monitor the runtime behavior of our models
• Track data quality issues
• Identify and troubleshoot real-time incidents
• Optimize performance bottlenecks and resource usage
• Identify dependencies on other services and their latency
• Optimize data transformation processes
• Set up alerts on latency, data quality issues, error rates of transactions or CPU usage)

With these capabilities, we can ensure the resilience and reliability of our data pipelines, leading to more robust and accurate BI system and reporting.

In conclusion, setting up OpenTelemetry (OTEL) in Python for data pipeline observability has significantly improved our ability to monitor, detect, and resolve issues proactively. This has led to more reliable data transformations, better resource management, and enhanced overall performance of our data transformation, BI and Machine Learning systems.

This app is also now insturmented with EDOT, and you can visualize the Chatbot's traces to OpenAI, as well as relevant logs, and metrics from the application. By running the app as instructed in the github repo with Docker you can see these traces on a local stack. But how about running it against serverless, Elastic cloud or even with Kubernetes?

In this blog we will walk through how to set up Elastic's RAG Based Chatbot application with Elastic cloud and Kubernetes.

Prerequisites:

In order to follow along, these few pre-requisites are needed

• An Elastic Cloud account — sign up now, and become familiar with Elastic's OpenTelemetry configuration. With Serverless no version required. With regular cloud minimally 8.17

• Git clone the RAG based Chatbot application and go through the tutorial on how to bring it up and become more familiar and how to bring up the application using Docker.

• An account on OpenAI with API keys

• Kubernetes cluster to run the RAG based Chatbot app

• The instructions in this blog are also found in observability-examples in github.

Application OpenTelemetry output in Elastic

Chatbot-rag-app

The first item that you will need to get up and running is the ChatBotApp, and once up you should see the following:

As you select some of the questions you will set a response based on the index that was created in Elasticsearch when the app initializes. Additionally there will be queries that are made to LLMs.

Traces, logs, and metrics from EDOT in Elastic

Once you have the application running on your K8s cluster or with Docker, and Elastic Cloud up and running you should see the following:

Logs:

In Discover you will see logs from the Chatbotapp, and be able to analyze the application logs, any specific log patterns, which saves you time in analysis.

Traces:

In Elastic Observability APM, you can also see tha chatbot details, which include transactions, dependencies, logs, errors, etc.

When you look at traces, you will be able to see the chatbot interactions in the trace.

1. You will see the end to end http call

2. Individual calls to elasticsearch

3. Specific calls such as invoke actions, and calls to the LLM

You can also get individual details of the traces, and look at related logs, and metrics related to that trace,

Metrics:

In addition to logs, and traces, any instrumented metrics will also get ingested into Elastic.

Setting it all up with Docker

In order to properly set up the Chatbot-app on Docker with telemetry sent over to Elastic, a few things must be set up:

1. Git clone the chatbot-rag-app

2. Modify the env file as noted in the github README with the following exception:

Use your Elastic cloud's OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADER instead.

You can find these in the Elastic Cloud under integrations->APM

Envs for sending the OTel instrumentation you will need the following:

OTEL_EXPORTER_OTLP_ENDPOINT="https://123456789.apm.us-west-2.aws.cloud.es.io:443"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20xxxxx"

Notice the %20 in the headers. This will be needed to account for the space in credentials.

3. Set the following to false - OTEL_SDK_DISABLED=false

4. Set the envs for LLMs

In this example we're using OpenAI, hence only three variables are needed.

LLM_TYPE=openai
OPENAI_API_KEY=XXXX
CHAT_MODEL=gpt-4o-mini

5. Run the docker container as noted

docker compose up --build --force-recreate

6. Play with the app at localhost:4000

7. Then log into Elastic cloud and see the output as shown previously.

Run chatbot-rag-app on Kubernetes

In order to set this up, you can follow the following repo on Observability-examples which has the Kubernetes yaml files being used. These will also point to Elastic Cloud.

1. Set up the Kubernetes Cluster (we're using EKS)

2. Get the appropriate ENV variables:

• Find the OTEL_EXPORTER_OTLP_ENDPOINT/HEADER variables as noted in the pervious for Docker.

• Get your OpenAI Key

• Elasticsearch URL, and username and password.

3. Follow the instructions in the following github repo in observability examples to run two Kubernetes yaml files.

Essentially you need only replace the secret variables in k8s-deployment.yaml, and run

kubectl create -f k8s-deployment.yaml
kubectl create -f init-index-job.yaml

The app needs to be running first, then we use the app to initialize Elasticsearch with indices for the app.

Init-index-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: init-elasticsearch-index-test
spec:
  template:
    spec:
      containers:
      - name: init-index
        image: ghcr.io/elastic/elasticsearch-labs/chatbot-rag-app:latest
        workingDir: /app/api
        command: ["python3", "-m", "flask", "--app", "app", "create-index"]
        env:
        - name: FLASK_APP
          value: "app"
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: ES_INDEX
          value: "workplace-app-docs"
        - name: ES_INDEX_CHAT_HISTORY
          value: "workplace-app-docs-chat-history"
        - name: ELASTICSEARCH_URL
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_URL
        - name: ELASTICSEARCH_USER
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_USER
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_PASSWORD
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
      restartPolicy: Never
  backoffLimit: 4

k8s-deployment.yaml

apiVersion: v1
kind: Secret
metadata:
  name: chatbot-regular-secrets
type: Opaque
stringData:
  ELASTICSEARCH_URL: "https://yourelasticcloud.es.us-west-2.aws.found.io"
  ELASTICSEARCH_USER: "elastic"
  ELASTICSEARCH_PASSWORD: "elastic"
  OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer%20xxxx"
  OTEL_EXPORTER_OTLP_ENDPOINT: "https://12345.apm.us-west-2.aws.cloud.es.io:443"
  OPENAI_API_KEY: "YYYYYYYY"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chatbot-regular
spec:
  replicas: 2
  selector:
    matchLabels:
      app: chatbot-regular
  template:
    metadata:
      labels:
        app: chatbot-regular
    spec:
      containers:
      - name: chatbot-regular
        image: ghcr.io/elastic/elasticsearch-labs/chatbot-rag-app:latest
        ports:
        - containerPort: 4000
        env:
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=chatbot-regular,service.version=0.0.1,deployment.environment=dev"
        - name: OTEL_SDK_DISABLED
          value: "false"
        - name: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT
          value: "true"
        - name: OTEL_EXPERIMENTAL_RESOURCE_DETECTORS
          value: "process_runtime,os,otel,telemetry_distro"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "http/protobuf"
        - name: OTEL_METRIC_EXPORT_INTERVAL
          value: "3000"
        - name: OTEL_BSP_SCHEDULE_DELAY
          value: "3000"
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

---
apiVersion: v1
kind: Service
metadata:
  name: chatbot-regular-service
spec:
  selector:
    app: chatbot-regular
  ports:
  - port: 80
    targetPort: 4000
  type: LoadBalancer

Open App with LoadBalancer URL

Run the kubectl get services command and get the URL for the chatbot app

% kubectl get services
NAME                                 TYPE           CLUSTER-IP    EXTERNAL-IP                                                               PORT(S)                                                                     AGE
chatbot-regular-service            LoadBalancer   10.100.130.44    xxxxxxxxx-1515488226.us-west-2.elb.amazonaws.com   80:30748/TCP                                                                6d23h

4. Play with app and review telemetry in Elastic

5. Once you go to the URL, you should see all the screens we described earlier in the beginning of this blog.

Conclusion

With Elastic's Chatbot-rag-app you have an example of how to build out a OpenAI driven RAG based chat application. However, you still need to understand how well it performs, whether its working properly, etc. Using OTel and Elastic’s EDOT gives you the ability to achieve this. Additionally, you will generally run this application on Kubernetes. Hopefully this blog provides the outline of how to achieve this. Here are the other Tracing blogs:

App Observability with LLM (Tracing)-

• Observing LangChain with Langtrace and OpenTelemetry

• Observing LangChain with OpenLit Tracing

• Instrumenting LangChain with OpenTelemetry

LLM Observability -

• Elevate LLM Observability with GCP Vertex AI Integration

• LLM Observability on AWS Bedrock

• LLM Observability for Azure OpenAI

• LLM Observability for Azure OpenAI v2

Elastic provides the ability to ingest OpenTelemetry data with native OTel SDKs, the off the shelf OTel collector, or even Elastic’s Distributions of OpenTelemetry (EDOT). EDOT enables you to bring in logs, metrics and traces for your GenAI application and for K8s. However you will also generally need libraries to help trace specific components in your application. In tracing GenAI applications you can pick from a large set of libraries.

• OpenTelemetry OpenAI Instrumentation-v2 - allows tracing LLM requests and logging of messages made by the OpenAI Python API library. (note v2 is built by OpenTelemetry, the non v2 version is from a specific vendor and not OpenTelemetry)

• OpenTelemetry VertexAI Instrumentation - allows tracing LLM requests and logging of messages made by the VertexAI Python API library

• Langtrace - commercially available library which supports all LLMs in one library, and all traces are also OTel native.

• Elastic’s EDOT - which recently added tracing. See blog.

As you can see OpenTelemetry is the defacto mechanism that is converging to collect and ingest. OpenTelemetry is growing its support for this but it is also early days.

In this blog, we will walk through how to, with minimal code, observe a RAG based chatbot application with tracing using Langtrace. We previously covered Langtrace in a blog to highlight tracing Langchain.

In this blog we used langtrace OpenAI, Amazon Bedrock, Cohere, and others in one library.

Pre-requisites:

In order to follow along, these few pre-requisites are needed

• An Elastic Cloud account — sign up now, and become familiar with Elastic’s OpenTelemetry configuration. With Serverless no version required. With regular cloud minimally 8.17

• Git clone the RAG based Chatbot application and go through the tutorial on how to bring it up and become more familiar.

• An account on your favorite LLM (OpenAI, AzureOpen AI, etc), with API keys

• Be familiar with EDOT to understand how we bring in logs, metrics, and traces from the application through the OTel Collector

• Kubernetes cluster - I’ll be using Amazon EKS

• Look at Langtrace documentation also.

Application OpenTelemetry output in Elastic

Chatbot-rag-app

The first item that you will need to get up and running is the ChatBotApp, and once up you should see the following:

As you select some of the questions you will set a response based on the index that was created in Elasticsearch when the app initializes. Additionally there will be queries that are made to LLMs.

Traces, logs, and metrics from EDOT in Elastic

Once you have OTel Collector with EDOT configuration on your K8s cluster, and Elastic Cloud up and running you should see the following:

Logs:

In Discover you will see logs from the Chatbotapp, and be able to analyze the application logs, any specific log patterns (saves you time in analysis), and view logs from K8s.

Traces:

In Elastic Observability APM, you can also see tha chatbot details, which include transactions, dependencies, logs, errors, etc.

When you look at traces, you will be able to see the chatbot interactions in the trace.

1. You will see the end to end http call

2. Individual calls to elasticsearch

3. Specific calls such as invoke actions, and calls to the LLM

You can also get individual details of the traces, and look at related logs, and metrics related to that trace,

Metrics:

In addition to logs, and traces, any instrumented metrics will also get ingested into Elastic.

Setting it all up

In order to properly set up the Chatbot-app on K8s with telemetry sent over to Elastic, a few things must be set up:

1. Git clone the chatbot-rag-app, and modify one of the python files.

2. Next create a docker container that can be used in Kubernetes. The Docker build here in the Chatbot-app is good to use.

3. Collect all needed env variables. In this example we are using OpenAI, but the files can be modified for any of the LLMs. Hence you will have to get a few environmental variables loaded into the cluster. In the github repo there is a env.example for docker. You can pick and chose what is needed or not needed and adjust appropriately in the K8s file below.

4. Set up your K8s Cluster, and then install the OpenTelemetry collector with the appropriate yaml file and credentials. This will help collect K8s cluster logs and metrics also.

5. Utilize the two yaml files listed below to ensure you can run it on Kubernetes.

• Init-index-job.yaml - Initiates the index in elasticsearch with the local corporate information

• k8s-deployment-chatbot-rag-app.yaml - initializes the application frontend and backend.

6. Open the app on the load balancer URL against the chatbot-app service in K8s

7. Go to Elasticsearch and look at Discover for logs, go to APM and look for your chatbot-app and review the traces, and finally.

Modify the code for tracing with Langtrace

Once you curl the app and untar, go to the chatbot-rag-app directory:

curl https://codeload.github.com/elastic/elasticsearch-labs/tar.gz/main | 
tar -xz --strip=2 elasticsearch-labs-main/example-apps/chatbot-rag-app
cd elasticsearch-labs-main/example-apps/chatbot-rag-app

Next open the app.py file in the api directory and add the following

from opentelemetry.instrumentation.flask import FlaskInstrumentor

from langtrace_python_sdk import langtrace

langtrace.init(batch=False)

FlaskInstrumentor().instrument_app(app)

into the code:

import os
import sys
from uuid import uuid4

from chat import ask_question
from flask import Flask, Response, jsonify, request
from flask_cors import CORS

from opentelemetry.instrumentation.flask import FlaskInstrumentor

from langtrace_python_sdk import langtrace

langtrace.init(batch=False)

app = Flask(__name__, static_folder="../frontend/build", static_url_path="/")
CORS(app)

FlaskInstrumentor().instrument_app(app)

@app.route("/")

See the items in BOLD which will add in the langtrace library, and the opentelemetry flask instrumentation. This combination will provide and end to end trace for the https call all the way down to the calls to Elasticsearch, and to OpenAI (or other LLMs).

Create the docker container

Use the Dockerfile that is in the chatbot-rag-app directory as is and add the following line:

RUN pip3 install --no-cache-dir langtrace-python-sdk

into the Dockerfile:

COPY requirements.txt ./requirements.txt
RUN pip3 install -r ./requirements.txt
RUN pip3 install --no-cache-dir langtrace-python-sdk
COPY api ./api
COPY data ./data

EXPOSE 4000

This enables the langtrace-python-sdk to be installed into the docker container so the langtrace libraries can be used properly.

Collecting the proper env variables:

First collect the env variables from Elastic:

Envs for index initialization in Elastic:

ELASTICSEARCH_URL=https://aws.us-west-2.aws.found.io
ELASTICSEARCH_USER=elastic
ELASTICSEARCH_PASSWORD=elastic

# The name of the Elasticsearch indexes
ES_INDEX=workplace-app-docs
ES_INDEX_CHAT_HISTORY=workplace-app-docs-chat-history

The ELASTICSEARCH_URL can be found in cloud.elastic.co when you bring up your instance. The user and password, you will need to setup in Elastic.

Envs for sending the OTel instrumentation you will need the following:

OTEL_EXPORTER_OTLP_ENDPOINT="https://123456789.apm.us-west-2.aws.cloud.es.io:443"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer xxxxx"

These credentials are found in Elastic under APM integration and under OpenTelemetry

Envs for LLMs

In this example we’re using OpenAI, hence only three variables are needed.

LLM_TYPE=openai
OPENAI_API_KEY=XXXX
CHAT_MODEL=gpt-4o-mini

All these variables will be needed in the Kubernetes yamls in the next step

Setup K8s cluster and load up OTel Collector with EDOT

This step is outlined in the following Blog. It’s a simple three step process.

This step will bring in all the K8s cluster logs and metrics and setup the OTel collector.

Setup secrets, initialize indices, and start the app

Now that the cluster is up, and you have your environmental variables, you will need to

1. Install and run the k8s-deployments.yaml with the variables

2. Initialize the index

Essentially run the following:

kubectl create -f k8s-deployment.yaml
kubectl create -f init-index-job.yaml

Here are the two yamls you should use. Also found here

k8s-deployment.yaml

apiVersion: v1
kind: Secret
metadata:
  name: genai-chatbot-langtrace-secrets
type: Opaque
stringData:
  OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer%20xxxx"
  OTEL_EXPORTER_OTLP_ENDPOINT: "https://1234567.apm.us-west-2.aws.cloud.es.io:443"
 ELASTICSEARCH_URL: "YOUR_ELASTIC_SEARCH_URL"
  ELASTICSEARCH_USER: "elastic"
  ELASTICSEARCH_PASSWORD: "elastic"
  OPENAI_API_KEY: "XXXXXXX"  

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: genai-chatbot-langtrace
spec:
  replicas: 2
  selector:
    matchLabels:
      app: genai-chatbot-langtrace
  template:
    metadata:
      labels:
        app: genai-chatbot-langtrace
    spec:
      containers:
      - name: genai-chatbot-langtrace
        image:65765.amazonaws.com/genai-chatbot-langtrace2:latest
        ports:
        - containerPort: 4000
        env:
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: OTEL_SDK_DISABLED
          value: "false"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=genai-chatbot-langtrace,service.version=0.0.1,deployment.environment=dev"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "http/protobuf"
        envFrom:
        - secretRef:
            name: genai-chatbot-langtrace-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

---
apiVersion: v1
kind: Service
metadata:
  name: genai-chatbot-langtrace-service
spec:
  selector:
    app: genai-chatbot-langtrace
  ports:
  - port: 80
    targetPort: 4000
  type: LoadBalancer

Init-index-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: init-elasticsearch-index-test
spec:
  template:
    spec:
      containers:
      - name: init-index
#update your image location for chatbot rag app
        image: your-image-location:latest
        workingDir: /app/api
        command: ["python3", "-m", "flask", "--app", "app", "create-index"]
        env:
        - name: FLASK_APP
          value: "app"
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: ES_INDEX
          value: "workplace-app-docs"
        - name: ES_INDEX_CHAT_HISTORY
          value: "workplace-app-docs-chat-history"
        - name: ELASTICSEARCH_URL
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_URL
        - name: ELASTICSEARCH_USER
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_USER
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_PASSWORD
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
      restartPolicy: Never
  backoffLimit: 4

Open App with LoadBalancer URL

Run the kubectl get services command and get the URL for the chatbot app

% kubectl get services
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                                                                     AGE
chatbot-langtrace-service            LoadBalancer   10.100.130.44    xxxxxxxxx-1515488226.us-west-2.elb.amazonaws.com   80:30748/TCP                                                                6d23h

Play with app and review telemetry in Elastic

Once you go to the URL, you should see all the screens we described earlier in the beginning of this blog.

Conclusion

With Elastic's Chatbot-rag-app you have an example of how to build out a OpenAI driven RAG based chat application. However, you still need to understand how well it performs, whether its working properly, etc. Using OTel, Elastic’s EDOT and Langtrace gives you the ability to achieve this. Additionally, you will generally run this application on Kubernetes. Hopefully this blog provides the outline of how to achieve this.

Here are the other Tracing blogs:

App Observability with LLM (Tracing)-

• Observing LangChain with Langtrace and OpenTelemetry

• Observing LangChain with OpenLit Tracing

• Instrumenting LangChain with OpenTelemetry

LLM Observability -

• Elevate LLM Observability with GCP Vertex AI Integration

• LLM Observability on AWS Bedrock

• LLM Observability for Azure OpenAI

• LLM Observability for Azure OpenAI v2

This blog post will showcase a different use-case: within Elastic APM, we have our own APM Agents. Some of these have download numbers in the tens of millions, and some of them predate OpenTelemetry. Of course we realize OpenTelemetry is very important and it’s here to stay, so we wanted to make these agents OpenTelemetry compatible and illustrate them using OpenTelemetry visualizations in this blog.

Most of our Elastic APM Agents today are able to ship OpenTelemetry spans as part of a trace. This means that if you have any component in your application that emits an OpenTelemetry span, it’ll be part of the trace the Elastic APM Agent captures. This can be a library you use that is already instrumented by the OpenTelemetry API, or it can be any other OpenTelemetry span that an application developer added into the application’s code for manual instrumentation.

This feature of the Elastic APM Agents not only reports those spans but also properly maintains parent-child relationships between all spans, making OpenTelemetry a first-class citizen for these agents. If, for example, an Elastic APM Agent starts a span for a specific action by auto-instrumentation and then within that span the OpenTelemetry API starts another span, then the OpenTelemetry span will be the child of the outer span created by the agent. This is reflected in the parent.id field of the spans. It’s the same the other way around as well: if a span is created by the OpenTelemetry API and within that span an Elastic APM agent captures another span, then the span created by the Elastic APM Agent will be the child of the other span created by the OpenTelemetry API.

This feature is present in the following agents:

• Java
• .NET
• Python
• Node.js
• Go

Capturing OpenTelemetry spans in the Elastic .NET APM Agent

As a first example, let’s take an ASP.NET Core application. We’ll put the .NET Elastic APM Agent into this application, and we’ll turn on the feature, which automatically bridges OpenTelemetry spans, so the Elastic APM Agent will make those spans part of the trace it reports.

The following code snippet shows a controller:

namespace SampleAspNetCoreApp.Controllers
{
	public class HomeController : Controller
	{
		private readonly SampleDataContext _sampleDataContext;
		private ActivitySource _activitySource = new ActivitySource("HomeController");
		public HomeController(SampleDataContext sampleDataContext) => _sampleDataContext = sampleDataContext;
		public async Task<IActionResult> Index()
		{
			await ReadGitHubStars();
			return View();
		}
		public async Task ReadGitHubStars()
		{
			using var activity = _activitySource.StartActivity();
			var httpClient = new HttpClient();
			httpClient.DefaultRequestHeaders.Add("User-Agent", "APM-Sample-App");
			var responseMsg = await httpClient.GetAsync("https://api.github.com/repos/elastic/apm-agent-dotnet");
			var responseStr = await responseMsg.Content.ReadAsStringAsync();
			// …use responseStr
		}
	}
}

The Index method calls the ReadGitHubStars method and after that we simply return the corresponding view from the method.

The incoming HTTP call and the outgoing HTTP call by the HttpClient are automatically captured by the Elastic APM Agent — this is part of the auto instrumentation we had for a very long time.

The ReadGitHubStars is the one where we use the OpenTelemetry API. OpenTelemetry in .NET uses the ActivitySource and Activity APIs. The _activitySource.StartActivity() call simply creates an OpenTelemetry span that automatically takes the name of the method by using the CallerMemberNameAttribute C# language feature, and this span will end when the method runs to completion.

Additionally, within this span we call the GitHub API with the HttpClient type. For this type, the .NET Elastic APM Agent again offers auto instrumentation, so the HTTP call will be also captured as a span by the agent automatically.

And here is how the water-flow chart for this transaction looks in Kibana:

As you can see, the agent was able to capture the OpenTelemetry span as part of the trace.

Bridging OpenTelemetry spans in Python by using the Python Elastic APM Agent

Let’s see how this works in the case of Python. The idea is the same, so all the concepts introduced previously apply to this example as well.

We take a very simple Django example:

from django.http import HttpResponse
from elasticapm.contrib.opentelemetry import Tracer
import requests


def index(request):
   tracer = Tracer(__name__)
   with tracer.start_as_current_span("ReadGitHubStars"):
       url = "https://api.github.com/repos/elastic/apm-agent-python"
       response = requests.get(url)
       return HttpResponse(response)

The first step to turn on capturing OpenTelemetry spans in Python is to import the Tracer implementation from elasticapm.contrib.opentelemetry.

And then on this Tracer you can start a new span — in this case, we manually name the span ReadGitHubStars.

Similarly to the previous example, the call to http://127.0.0.1:8000/otelsample/ is captured by the Elastic APM Python Agent, and then the next span is created by the OpenTelemetry API, which, as you can see, is captured by the agent automatically, and then finally the HTTP call to the GitHub API is captured again by the auto instrumentation of the agent.

Here is how it looks in the water-flow chart:

As already mentioned, the agent maintains the parent-child relationship for all the OTel spans. Let’s take a look at the parent.id of the GET api.github.com call:

As you can see, the id of this span is c98401c94d40b87a.

If we look at the span.id of the ReadGitHubStars OpenTelemetry span, then we can see that the id of this span is exactly c98401c94d40b87a — so the APM Agent internally maintains parent-child relationships across OpenTelemetry and non-OpenTelemetry spans, which makes OpenTelemetry spans first-class citizens in Elastic APM Agents.

Other languages

At this point, I'll stop to just replicate the exact same sample code in further languages — I think you already got the point here: in each language listed above, our Elastic APM Agents are able to bridge OpenTelemetry traces and show them in Kibana as native spans. We also blogged about using the same API in Java, and you can see examples for the rest of the languages in the corresponding agent documentation (linked above).

When to use this feature and when to use pure OpenTelemetry SDKs

This is really up to you. If you want to only have pure OpenTelemetry usage in your applications and you really want to avoid any vendor-related software, then feel free to use OpenTelemetry SDKs directly — that is a use case we clearly support. If you go that route, this feature is not so relevant to you.

However, our Elastic APM Agents already have a very big user base and they offer features that are not present in OpenTelemetry. Some of these features are span compression, central configuration, inferred spans, distributed tail based sampling with multiple APM Servers, and many more.

If you are one of the many existing Elastic APM Agent users, or you plan to use an Elastic APM Agent because of the features mentioned above, then bridging OpenTelemetry spans enables you to still use the OpenTelemetry API and not rely on any vendor related API usage. That way your developer teams can instrument your application with OpenTelemetry, and you can also use any third-party library already instrumented by OpenTelemetry, and Elastic APM Agents will happily report those spans as part of the traces they report. With this, you can combine the vendor independent nature of OpenTelemetry and still use the feature rich Elastic APM Agents.

The OpenTelemetry bridge feature is also a good tool to use if you wish to change your telemetry library from an Elastic APM Agent to OpenTelemetry (and vice-versa), as it allows you to use both libraries together and switch them using atomic changes.

Next steps

In this blog post, we discussed how you can bridge OpenTelemetry spans with Elastic APM Agents. Of course OpenTelemetry is more than just traces. We know that, and we plan to cover further areas: currently we are working on bridging OpenTelemetry metrics in our Elastic APM Agents in a very similar fashion. You can watch the progress here.

Learn more about adding Elastic APM as part of your Elastic Observability deployment.