OpenTelemetry integrationedit

OpenTelemetry is a set of APIs, SDKs, tooling, and integrations that enable the capture and management of telemetry data from your services for greater observability. For more information about the OpenTelemetry project, see the spec.

Elastic OpenTelemetry integrations allow you to reuse your existing OpenTelemetry instrumentation to quickly analyze distributed traces and metrics to help you monitor business KPIs and technical components with the Elastic Stack.

APM Server native support of OpenTelemetry protocoledit

Elastic APM Server natively supports the OpenTelemetry protocol. This means trace data and metrics collected from your applications and infrastructure can be sent directly to Elastic APM Server using the OpenTelemetry protocol.

OpenTelemetry Elastic architecture diagram
Instrument applicationsedit

To export traces and metrics to APM Server, ensure that you have instrumented your services and applications with the OpenTelemetry API, SDK, or both. For example, if you are a Java developer, you need to instrument your Java app using the OpenTelemetry agent for Java.

By defining the following environment variables, you can configure the OTLP endpoint so that the OpenTelemetry agent communicates with APM Server.

export OTEL_RESOURCE_ATTRIBUTES=service.name=checkoutService,service.version=1.1,deployment.environment=production
export OTEL_EXPORTER_OTLP_ENDPOINT=https://apm_server_url:8200
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer an_apm_secret_token"
export OTEL_METRICS_EXPORTER="otlp" \
export OTEL_LOGS_EXPORTER="otlp" \
java -javaagent:/path/to/opentelemetry-javaagent-all.jar \
     -classpath lib/*:classes/ \
     com.mycompany.checkout.CheckoutServiceServer

OTEL_RESOURCE_ATTRIBUTES

Fields that describe the service and the environment that the service runs in. See Resource attributes for more information.

OTEL_EXPORTER_OTLP_ENDPOINT

APM Server URL. The host and port that APM Server listens for events on.

OTEL_EXPORTER_OTLP_HEADERS

Authorization header that includes the Elastic APM Secret token or API key: "Authorization=Bearer an_apm_secret_token" or "Authorization=ApiKey an_api_key".

For information on how to format an API key, see our API key docs.

Please note the required space between Bearer and an_apm_secret_token, and APIKey and an_api_key.

OTEL_EXPORTER_OTLP_CERTIFICATE

The trusted certificate used to verify the TLS credentials of the client. (optional)

You are now ready to collect traces and metrics before verifying metrics and visualizing metrics in Kibana.

Connect OpenTelemetry Collector instancesedit

Connect your OpenTelemetry collector instances to Elastic Observability using the OTLP exporter.

receivers: 
  # ...
  otlp:

processors: 
  # ...
  memory_limiter:
    check_interval: 1s
    limit_mib: 2000
  batch:

exporters:
  logging:
    loglevel: warn 
  otlp/elastic: 
    # Elastic APM server https endpoint without the "https://" prefix
    endpoint: "${ELASTIC_APM_SERVER_ENDPOINT}"  
    headers:
      # Elastic APM Server secret token
      Authorization: "Bearer ${ELASTIC_APM_SECRET_TOKEN}"  

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, otlp/elastic]
    metrics:
      receivers: [otlp]
      exporters: [logging, otlp/elastic]
    logs: 
      receivers: [otlp]
      exporters: [logging, otlp/elastic]

The receivers, such as the OTLP receiver, that forward data emitted by APM agents or the host metrics receiver.

We recommend using the Batch processor and also suggest using the memory limiter processor. For more information, see Recommended processors.

The logging exporter is helpful for troubleshooting and supports various logging levels: debug, info, warn, and error.

Elastic Observability endpoint configuration. APM Server supports a ProtoBuf payload via both the OTLP protocol over gRPC transport (OTLP/gRPC) and the OTLP protocol over HTTP transport (OTLP/HTTP). To learn more about these exporters, see the OpenTelemetry Collector documentation: OTLP/HTTP Exporter or OTLP/gRPC exporter.

Hostname and port of the APM Server endpoint. For example, elastic-apm-server:8200.

Credential for Elastic APM secret token authorization (Authorization: "Bearer a_secret_token") or API key authorization (Authorization: "ApiKey an_api_key").

Environment-specific configuration parameters can be conveniently passed in as environment variables documented here (e.g. ELASTIC_APM_SERVER_ENDPOINT and ELASTIC_APM_SECRET_TOKEN).

To send OpenTelemetry logs to Elastic Stack version 8.0+, declare a logs pipeline.

You’re now ready to export traces and metrics from your services and applications.

Collect metricsedit

When collecting metrics, please note that the DoubleValueRecorder and LongValueRecorder metrics are not yet supported.

Here’s an example of how to capture business metrics from a Java application.

// initialize metric
Meter meter = GlobalMetricsProvider.getMeter("my-frontend");
DoubleCounter orderValueCounter = meter.doubleCounterBuilder("order_value").build();

public void createOrder(HttpServletRequest request) {

   // create order in the database
   ...
   // increment business metrics for monitoring
   orderValueCounter.add(orderPrice);
}

See the Open Telemetry Metrics API for more information.

Verify OpenTelemetry metrics dataedit

Use Discover to validate that metrics are successfully reported to Kibana.

  1. Launch Kibana:

    1. Log in to your Elastic Cloud account.
    2. Navigate to the Kibana endpoint in your deployment.
  2. Open the main menu, then click Discover.
  3. Select apm-* as your index pattern.
  4. Filter the data to only show documents with metrics: processor.name :"metric"
  5. Narrow your search with a known OpenTelemetry field. For example, if you have an order_value field, add order_value: * to your search to return only OpenTelemetry metrics documents.
Visualize in Kibanaedit

TSVB within Kibana is the recommended visualization for OpenTelemetry metrics. TSVB is a time series data visualizer that allows you to use the Elasticsearch aggregation framework’s full power. With TSVB, you can combine an infinite number of aggregations to display complex data.

In this example eCommerce OpenTelemetry dashboard, there are four visualizations: sales, order count, product cache, and system load. The dashboard provides us with business KPI metrics, along with performance-related metrics.

OpenTelemetry visualizations

Let’s look at how this dashboard was created, specifically the Sales USD and System load visualizations.

  1. Open the main menu, then click Dashboard.
  2. Click Create dashboard.
  3. Click Save, enter the name of your dashboard, and then click Save again.
  4. Let’s add a Sales USD visualization. Click Edit.
  5. Click Create new and then select TSVB.
  6. For the label name, enter Sales USD, and then select the following:

    • Aggregation: Counter Rate.
    • Field: order_sum.
    • Scale: auto.
    • Group by: Everything
  7. Click Save, enter Sales USD as the visualization name, and then click Save and return.
  8. Now let’s create a visualization of load averages on the system. Click Create new.
  9. Select TSVB.
  10. Select the following:

    • Aggregation: Average.
    • Field: system.cpu.load_average.1m.
    • Group by: Terms.
    • By: host.ip.
    • Top: 10.
    • Order by: Doc Count (default).
    • Direction: Descending.
  11. Click Save, enter System load per host IP as the visualization name, and then click Save and return.

    Both visualizations are now displayed on your custom dashboard.

By default, Discover shows data for the last 15 minutes. If you have a time-based index and no data displays, you might need to increase the time range.

AWS Lambda Supportedit

AWS Lambda functions can be instrumented with OpenTelemetry and monitored with Elastic Observability.

To get started, follow the official AWS Distro for OpenTelemetry Lambda getting started documentation and configure the OpenTelemetry Collector to output traces and metrics to your Elastic cluster.

Instrumenting AWS Lambda Java functionsedit

For a better startup time, we recommend using SDK-based instrumentation, i.e. manual instrumentation of the code, rather than auto instrumentation.

To instrument AWS Lambda Java functions, follow the official AWS Distro for OpenTelemetry Lambda Support For Java.

Noteworthy configuration elements:

  • AWS Lambda Java functions should extend com.amazonaws.services.lambda.runtime.RequestHandler,

    public class ExampleRequestHandler implements RequestHandler<APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent> {
        public APIGatewayProxyResponseEvent handleRequest(APIGatewayProxyRequestEvent event, Context context) {
            // add your code ...
        }
    }
  • When using SDK-based instrumentation, frameworks you want to gain visibility of should be manually instrumented

  • The configuration of the OpenTelemetry Collector, with the definition of the Elastic Observability endpoint, can be added to the root directory of the Lambda binaries (e.g. defined in src/main/resources/opentelemetry-collector.yaml)

    # Copy opentelemetry-collector.yaml in the root directory of the lambda function
    # Set an environment variable 'OPENTELEMETRY_COLLECTOR_CONFIG_FILE' to '/var/task/opentelemetry-collector.yaml'
    receivers:
      otlp:
        protocols:
          http:
          grpc:
    
    exporters:
      logging:
        loglevel: debug
      otlp/elastic:
        # Elastic APM server https endpoint without the "https://" prefix
        endpoint: "${ELASTIC_OTLP_ENDPOINT}" 
        headers:
          # Elastic APM Server secret token
          Authorization: "Bearer ${ELASTIC_OTLP_TOKEN}" 
    
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [logging, otlp/elastic]
        metrics:
          receivers: [otlp]
          exporters: [logging, otlp/elastic]
        logs:
          receivers: [otlp]
          exporters: [logging, otlp/elastic]

    Environment-specific configuration parameters can be conveniently passed in as environment variables: ELASTIC_OTLP_ENDPOINT and ELASTIC_OTLP_TOKEN

  • Configure the AWS Lambda Java function with:

    • Function layer: The latest AWS Lambda layer for OpenTelemetry (e.g. arn:aws:lambda:eu-west-1:901920570463:layer:aws-otel-java-wrapper-ver-1-2-0:1)
    • TracingConfig / Mode set to PassTrough
    • FunctionConfiguration / Timeout set to more than 10 seconds to support the longer cold start inherent to the Lambda Java Runtime
    • Export the environment variables:

      • AWS_LAMBDA_EXEC_WRAPPER="/opt/otel-proxy-handler" for wrapping handlers proxied through the API Gateway (see here)
      • OTEL_PROPAGATORS="tracecontext, baggage" to override the default setting that also enables X-Ray headers causing interferences between OpenTelemetry and X-Ray
      • OPENTELEMETRY_COLLECTOR_CONFIG_FILE="/var/task/opentelemetry-collector.yaml" to specify the path to your OpenTelemetry Collector configuration
Instrumenting AWS Lambda Java functions with Terraformedit

We recommend using an infrastructure as code solution like Terraform or Ansible to manage the configuration of your AWS Lambda functions.

Here is an example of AWS Lambda Java function managed with Terraform and the AWS Provider / Lambda Functions:

Instrumenting AWS Lambda Node.js functionsedit

For a better startup time, we recommend using SDK-based instrumentation for manual instrumentation of the code rather than auto instrumentation.

To instrument AWS Lambda Node.js functions, see AWS Distro for OpenTelemetry Lambda Support For JavaScript.

The configuration of the OpenTelemetry Collector, with the definition of the Elastic Observability endpoint, can be added to the root directory of the Lambda binaries: src/main/resources/opentelemetry-collector.yaml.

# Copy opentelemetry-collector.yaml in the root directory of the lambda function
# Set an environment variable 'OPENTELEMETRY_COLLECTOR_CONFIG_FILE' to '/var/task/opentelemetry-collector.yaml'
receivers:
  otlp:
    protocols:
      http:
      grpc:

exporters:
  logging:
    loglevel: debug
  otlp/elastic:
    # Elastic APM server https endpoint without the "https://" prefix
    endpoint: "${ELASTIC_OTLP_ENDPOINT}" 
    headers:
      # Elastic APM Server secret token
      Authorization: "Bearer ${ELASTIC_OTLP_TOKEN}" 

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, otlp/elastic]
    metrics:
      receivers: [otlp]
      exporters: [logging, otlp/elastic]
    logs:
      receivers: [otlp]
      exporters: [logging, otlp/elastic]

Environment-specific configuration parameters can be conveniently passed in as environment variables: ELASTIC_OTLP_ENDPOINT and ELASTIC_OTLP_TOKEN

Configure the AWS Lambda Node.js function:

  • Function layer: The latest AWS Lambda layer for OpenTelemetry. For example, arn:aws:lambda:eu-west-1:901920570463:layer:aws-otel-nodejs-ver-0-23-0:1)
  • TracingConfig / Mode set to PassTrough
  • FunctionConfiguration / Timeout set to more than 10 seconds to support the cold start of the Lambda JavaScript Runtime
  • Export the environment variables:

    • AWS_LAMBDA_EXEC_WRAPPER="/opt/otel-handler" for wrapping handlers proxied through the API Gateway. See enable auto instrumentation for your lambda-function.
    • OTEL_PROPAGATORS="tracecontext" to override the default setting that also enables X-Ray headers causing interferences between OpenTelemetry and X-Ray
    • OPENTELEMETRY_COLLECTOR_CONFIG_FILE="/var/task/opentelemetry-collector.yaml" to specify the path to your OpenTelemetry Collector configuration.
    • OTEL_TRACES_SAMPLER="AlwaysOn" define the required sampler strategy if it is not sent from the caller. Note that Always_on can potentially create a very large amount of data, so in production set the correct sampling configuration, as per the specification.
Instrumenting AWS Lambda Node.js functions with Terraformedit

To manage the configuration of your AWS Lambda functions, we recommend using an infrastructure as code solution like Terraform or Ansible.

Here is an example of AWS Lambda Node.js function managed with Terraform and the AWS Provider / Lambda Functions:

Resource attributesedit

A resource attribute is a key/value pair containing information about the entity producing telemetry. Resource attributes are mapped to Elastic Common Schema (ECS) fields like service.*, cloud.*, process.*, etc. These fields describe the service and the environment that the service runs in.

The examples below set the Elastic (ECS) service.environment field for the resource, i.e. service, that is producing trace events. Note that Elastic maps the OpenTelemetry deployment.environment field to the ECS service.environment field on ingestion.

OpenTelemetry agent

Use the OTEL_RESOURCE_ATTRIBUTES environment variable to pass resource attributes at process invocation.

export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production

OpenTelemetry collector

Use the resource processor to set or apply changes to resource attributes.

...
processors:
  resource:
    attributes:
    - key: deployment.environment
      action: insert
      value: production
...

Need to add event attributes instead? Use attributes—​not to be confused with resource attributes—​to add data to span, log, or metric events. Attributes can be added as a part of the OpenTelemetry instrumentation process or with the attributes processor.

Proxy requests to APM Serveredit

APM Server supports both the (OTLP/gRPC) and (OTLP/HTTP) protocol on the same port as Elastic APM agent requests. For ease of setup, we recommend using OTLP/HTTP when proxying or load balancing requests to the APM Server.

If you use the OTLP/gRPC protocol, requests to the APM Server must use either HTTP/2 over TLS or HTTP/2 Cleartext (H2C). No matter which protocol is used, OTLP/gRPC requests will have the header: "Content-Type: application/grpc".

When using a layer 7 (L7) proxy like AWS ALB, requests must be proxied in a way that ensures requests to the APM Server follow the rules outlined above. For example, with ALB you can create rules to select an alternative backend protocol based on the headers of requests coming into ALB. In this example, you’d select the gRPC protocol when the "Content-Type: application/grpc" header exists on a request.

For more information on how to configure an AWS ALB to support gRPC, see this AWS blog post: Application Load Balancer Support for End-to-End HTTP/2 and gRPC.

For more information on how APM Server services gRPC requests, see Muxing gRPC and HTTP/1.1.

Limitationsedit

OpenTelemetry tracesedit
  • Traces of applications using messaging semantics might be wrongly displayed as transactions in the APM UI, while they should be considered spans. #7001
  • Inability to see Stack traces in spans
  • Inability in APM views to view the "Time Spent by Span Type" #5747
  • Metrics derived from traces (throughput, latency, and errors) are not accurate when traces are sampled before being ingested by Elastic Observability (for example, by an OpenTelemetry Collector or OpenTelemetry APM agent or SDK) #472
OpenTelemetry metricsedit
  • Inability to see host metrics in Elastic Metrics Infrastructure view when using the OpenTelemetry Collector host metrics receiver #5310
OpenTelemetry logsedit
  • OpenTelemetry logs are supported with beta support from 8.0 #6768.
OpenTelemetry Line Protocol (OTLP)edit

APM Server supports both the (OTLP/gRPC) and (OTLP/HTTP) protocol with ProtoBuf payload. APM Server does not yet support JSON Encoding for OTLP/HTTP.

OpenTelemetry Collector exporter for Elasticedit

The OpenTelemetry Collector exporter for Elastic was deprecated in 7.13 and replaced by the native support of the OpenTelemetry Line Protocol in Elastic Observability (OTLP). To learn more, see migration.