Find answers quickly, correlate OpenTelemetry traces with existing ECS logs in Elastic Observability

OpenTelemetry (OTel) is rapidly becoming the de-facto standard for vendor-neutral instrumentation. However, many organizations still rely on Elastic Common Schema (ECS)-based logging pipelines and dashboards they've built and refined over years. Elastic Observability provides a way to combine the benefits of modern observability instrumentation with OTel-specifically using the Elastic Distribution of OpenTelemetry (EDOT), while maintaining compatibility with your existing ECS-based systems.

We will walk through this using a Java application to show:

• Modify an application using an existing ECS logging library (Log4j2/ECS-Java) to inject the
  trace.id
  and
  span.id
  contexts generated by the EDOT SDK.
• Configure the EDOT Collector to ingest these ECS-formatted logs from a file and forward them back to Elasticsearch, ensuring that tracing and logging data are perfectly correlated and immediately usable with Kibana's built-in dashboards and tools.

This approach allows for the full adoption of OTel's unified observability (traces, metrics, and logs) without having to abandon or modify your established ECS-based log pipelines and dashboards.

In the final part of the blog we will discuss how we can make collect data in an OpenTelemetry-first approach.

The ECS Foundation and the rise of EDOT

Imagine an ecosystem of applications already configured to send logs in the ECS format. This is a great starting point for structured logging. Historically, in the Elastic ecosystem, the Elastic Common Schema (ECS) was adopted as the standard for log formatting. Elastic simplifies this standardization at the source by providing ECS logging plugins that easily integrate with common logging libraries across various programming languages. These plugins automatically generate structured JSON logs adhering to ECS. For this demonstration, we'll use a custom Java application that generates random logs relying on the logging library Log4j2 configured with using the ecs-java-plugin library.

The Elastic documentation outlines an example configuration that serves as a reliable foundation for your application's setup. This process involves incorporating the necessary ECS Java logging plugin libraries and modifying the Log4j2 configuration file to utilize the ECS layout configuration. This setup assumes prior configuration of Log4j2 dependencies to include the required ECS plugin libraries. An extract of the Log4j2 configuration template follows:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="DEBUG">
    <Appenders>
        <Console name="LogToConsole" target="SYSTEM_OUT">
            <EcsLayout serviceName="logger-app" serviceVersion="v1.0.0"/>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="LogToConsole"/>
        </Root>
    </Loggers>
</Configuration>

The full code of the application used for this blog can be found here.

Introducing the Elastic Distribution of OpenTelemetry (EDOT)

The Elastic Distribution of OpenTelemetry (EDOT) is a collection of OpenTelemetry components (OTel Collector and language SDKs) customized by Elastic. Released with Elastic Observability v8.15, the EDOT Collector enhances Elastic's existing OTel capabilities by being able to collect and forward application logs, infrastructure logs, metrics, and traces using standard OTel Collector receivers. EDOT users benefit from automatic enrichment of container logs with Kubernetes metadata, leveraging a powerful log parser contributed by Elastic. EDOT's Primary Benefits:

Deliver Enhanced Features Earlier: Provides features not yet available in "vanilla" OTel components, which Elastic continuously contributes upstream.

Enhanced OTel Support: Offers enterprise-grade support and maintenance for fixes outside of standard OTel release cycles.

The question then becomes: How can users transition their ingestion architecture to an OTel-native approach while maintaining the ability to collect logs in ECS format?

This involves replacing classic collection and instrumentation components (like Elastic Agent and the Elastic APM Java Agent). Let us show you how this can be done step by step replacing it with the full suite of components provided by EDOT. A comprehensive view of the EDOT architecture components in Kubernets is shown below.

In a Kubernetes environment, EDOT components are typically installed via an OTel Operator and HELM chart. The main components are:

• EDOT Collector Cluster: deployment used to collect cluster-wide metrics.
• EDOT Collector Daemon: daemonset used to collect node metrics, logs, and application telemetry data.
• EDOT Collector Gateway: performs pre-processing, aggregation, and ingestion of data into Elastic.

Elastic provides a curated configuration file for all the EDOT components available as part of the the OpenTelemetry Operator using the

Trace and Log Correlation with the EDOT SDK

Our Java application needs to be instrumented with the EDOT Java SDK to collect traces and propagate trace context to its logs. While the EDOT SDK can collect logs directly, a generally more resilient approach is to stick to file collection. This is important because if the OTel Collector is down, logs written to a file are buffered locally on the disk, preventing the data loss that can occur if the SDK's in-memory queue reaches its limit and starts discarding new logs. For an in-depth discussion on this topic we refer to the OpenTelemetry Documentation.

Instrumenting the Application

The EDOT Java SDK is a customized version of the OpenTelemetry Java Agent. In Kubernetes, zero-code Java autoinstrumentation is supported by adding an annotation in the pod template configuration in the deployment manifest:

apiVersion: apps/v1
kind: Deployment
...
spec:
  ..
  template:
    metadata:
      # Auto-Instrumentation
      annotations:
        instrumentation.opentelemetry.io/inject-java: "opentelemetry-operator-system/elastic-instrumentation"

The Orchestration: EDOT SDK and ECS Logging

Correlating logs with traces relies on a two-part orchestration:

• The Injector (EDOT Java SDK): Whenever a span is active (indicating a tracked operation), the EDOT agent extracts the current Trace and Span IDs and injects them into the Java logging library's MDC (Mapped Diagnostic Context). By default, the SDK uses

  trace_id
  and
  span_id
  . To align with the ECS standard, we must configure the instrumentation object to inject the compliant field names
  trace.id
  and
  span.id
  instead. This is achieved by applying the following environment variables:
  instrumentation:
    java:
      image: ...
      env:
        # disable direct export (we rely on filelog collection)
        - name: OTEL_LOGS_EXPORTER
          value: none
        # Override default keys to match ECS standard
        - name: OTEL_INSTRUMENTATION_COMMON_LOGGING_SPAN_ID
          value: span.id
        - name: OTEL_INSTRUMENTATION_COMMON_LOGGING_TRACE_ID
          value: trace.id
  

  • The Formatter (ECS Logging Plugin): The ECS Java logging plugin (e.g., EcsLayout) formats the log event as structured JSON. Because we reconfigured the injector above, the plugin can now seamlessly map the data from the thread context directly to the final JSON log:

  MDC	JSON Log
  trace.id	trace.id
  span.id	span.id

Collecting and Processing Logs with the EDOT Collector

With the application now emitting logs in a pretty ECS JSON formatx containing the correct

EDOT Collector Configuration: Dynamic Workload Discovery and filelog receiver

Applications running on containers become moving targets for monitoring systems. To handle this, we rely on Dynamic workload discovery on Kubernetes. This allows the EDOT Collector to track pod lifecycles and dynamically attach log collection configurations based on specific annotations.

In our example, we have a Deployment with a Pod consisting of one container. We use Kubernetes annotations to:

1. Enable auto-instrumentation (Java).

2. Enable log collection for this pod.

3. Instruct the collector to parse the output as JSON immediately (json-parser configuration).

4. Add custom attributes (e.g. identify the Application souce code)

Deployment Manifest Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: logger-app-deployment
  labels:
    app: logger-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logger-app
  template:
    metadata:
      annotations:
        # 1. Turn on Auto-Instrumentation
        instrumentation.opentelemetry.io/inject-java: "opentelemetry-operator-system/elastic-instrumentation"
        # 2. Enable Log Collection for this pod
        io.opentelemetry.discovery.logs/enabled: "true"
        # 3. Provide the parsing "hint" (Treat logs as JSON)
        io.opentelemetry.discovery.logs.ecs-log-producer/config: |
            max_log_size: "2MiB"
            operators:
            - type: container
              id: container-parser
            - type: json_parser
              id: json-parser
         # 4. Identify this application as Java (To allow for user interface rendering in Kibana)
        resource.opentelemetry.io/telemetry.sdk.language: "java"
      labels:
        app: logger-app
    spec:
      containers:
      - name: logger-app-container

This setup provides a bare-minimum configuration for ingesting ECS library logs. Crucially, it decouples log collection from application logic. Developers simply need to provide a hint via annotations that their logs are in JSON format (structurally guaranteed by the ECS libraries). We then define the standardized enrichment and processing rules centrally at the processor level in the (Daemon) EDOT Collector.

This centralization ensures consistency across the platform: if we need to update our standard formatting or enrichment strategies later, we apply the change once in the collector, and it automatically propagates to all services without developers needing to touch their manifests.

(Daemon) EDOT Collector Configuration

To enable this, we configure a Receiver Creator in the Daemon Collector. This component uses the

daemon:
  ...
  config:
    ...
    extensions:
      extensions:
        k8s_observer:
          auth_type: serviceAccount
          node: ${env:K8S_NODE_NAME}
          observe_nodes: true
          observe_pods: true
          observe_services: true
          ...
    receivers:
        receiver_creator/logs:
          watch_observers: [k8s_observer]
          discovery:
            enabled: true
    ...
...

Finally, we reference the

daemon:
  ...
  config:
    ...
    service:
      extensions:
      - k8s_observer
      pipelines:
        # Pipeline for node-level logs
        logs/node:
          receivers:
            # - filelog             # We disable direct filelog receiver
            - receiver_creator/logs # Using the configured receiver_creator instead of filelog
          processors:
            - batch
            - k8sattributes
            - resourcedetection/system
          exporters:
            - otlp/gateway # Forward to the Gateway Collector for ingestion

Transforming your log

To finalize the pipeline, we use the transform processor, which allows us to modify and restructure telemetry signals using the OpenTelemetry Transformation Language (OTTL).

While our logs are now valid structured JSON (ECS), the OpenTelemetry Collector initially reads them as generic log attributes. To enable proper correlation and backend storage, we must map these attributes to the strict OpenTelemetry Log Data Model.

We use the processor to promote specific ECS fields into the top-level OpenTelemetry fields and renaming attributes according to OpenTelemetry Semantic Conventions:

• Promote the
  message
  attribute to the top-level
  Body
  field.
• Promote the
  log.level
  attribute to the OTel
  SeverityText
  field.
• Move the
  @timestamp
  attribute to the OTel
  Time
  field.
• Rename
  trace.id
  and
  span.id
  to their OTel-compliant counterparts.
• Move "resource" attributes like
  service.name
  to their resource attributes counterparts

This ensures that all log data moving forward is standardized, enabling seamless correlation with traces and metrics, and simplifying eventual ingestion into any OTel-compatible backend.

 processors:
    transform/ecs_handler:
      log_statements:
        - context: log
          conditions:
            # Only apply if the log was actually generated by our ECS library
            - log.attributes["ecs.version"] != nil
          statements:
            # 1. Promote message to Body
            - set(log.body, log.attributes["message"])
            - delete_key(log.attributes, "message")

            # 2. Parse and promote Timestamp
            - set(log.time, Time(log.attributes["@timestamp"], "%Y-%m-%dT%H:%M:%SZ"))
            - delete_key(log.attributes, "@timestamp")

            # 3. Map Trace/Span IDs for correlation
            - set(log.trace_id.string, log.attributes["trace.id"])
            - delete_key(log.attributes, "trace.id")
            - set(log.span_id.string, log.attributes["span.id"])
            - delete_key(log.attributes, "span.id")

            # 4. Map log level to severity text
            - set(log.severity_text, log.attributes["log.level"])
            - delete_key(log.attributes, "log.level")

            # 5. Map resource attributes
            - set(resource.attributes["service.name"], log.attributes["service.name"]) where resource.attributes["service.name"] == null
            - delete_key(log.attributes, "service.name")
            - set(resource.attributes["service.version"], log.attributes["service.version"]) where resource.attributes["service.version"] == null
            - delete_key(log.attributes, "service.version")

            # Add here additional transformations as needed...

We need to reference the newly created processor in the Daemon Collector logs pipeline:

service:
  pipelines:
    logs/node:
      receivers:
        - receiver_creator/logs
      processors:
        - batch
        - k8sattributes
        - resourcedetection/system
        - transform/ecs_handler          # Newly created transform processor
      exporters:
        - otlp/gateway

Exporting to Elasticsearch with ECS Compatibility

The final step is configuring the EDOT Gateway Collector to export data. To maintain compatibility with existing ECS-based dashboards while supporting OTel-native signals, we rely on two key components: the Elasticsearch Exporter's mapping mode and a Routing Connector.

Mapping Mode

The Elasticsearch Exporter supports a

• mode: otel
  (Default): Stores documents using Elastic's preferred OTel-native schema. It preserves the original attribute names and structure of the OTLP event.
• mode: ecs
  : Tries to automatically map OpenTelemetry Semantic Conventions back to the Elastic Common Schema (ECS). This is the setting required to keep your legacy dashboards working.

Refer to Mapping Modes and ECS & OpenTelemetry for more details.

Routing Connector

Since we may have a mix of log types (e.g., k8sevents, other container logs not using ECS), we use a Routing Connector. This component inspects the logs in-flight. If it detects the ecs.version attribute (which we preserved in earlier steps), it routes the log to a dedicated ECS pipeline. All other logs fall back to the default pipeline.

Putting all together

Here is the setting provided in the Gateway Collector's Elasticsearch exporter configuration:

gateway:
  ...
  config:
    ...
    connectors:
      routing/logs:
        match_once: true
        default_pipelines: [logs]
        table:
          - context: log
            condition: attributes["ecs.version"] != null
            pipelines: [logs/ecs]

    exporters:
      elasticsearch/ecs:
        endpoint: <<Your Elasticsearch Endpoint URL>>
        # **Crucial setting for ECS compatibility**
        mapping:
          mode: ecs
        headers:
          Authorization: <<Your API Key>>

Finally, ensure the exporter you created is included in the logs pipeline of the Gateway Collector:

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [routing/logs]
    logs/otel:
      receivers: [routing/logs]
      processors: [batch]
      exporters: [elasticsearch]            # use default elasticsearch exporter
    logs/ecs:
      receivers: [routing/logs]
      processors: [batch]
      exporters: [elasticsearch/ecs]        # use elasticsearch ecs exporters

Conclusion

By implementing this architecture, we have successfully transitioned to an OTel-native observability stack without discarding our investment in ECS-based tooling.

We used the EDOT SDK to correlate logs and traces at the source, and the EDOT Collector to handle the "heavy lifting"—centralized discovery, enrichment, and schema translation. This gives us the best of both worlds: the modern flexibility of OpenTelemetry and the robust structure of ECS.

Looking ahead to the Future: The Path to Pure OTel

The greatest benefit of this OTel-native architecture is the flexibility of the pipeline. Because we used the Transform Processor earlier to map our logs to the OTel (Log) Data Model, our data is already compliant internally.

When you are ready to fully adopt OpenTelemetry Semantic Conventions (SemConv) and move away from ECS, you don't need to rewrite or redeploy your applications. You simply update the Collector to stop routing to the ECS pipeline and rely on the default OTel-native export.

This is achieved by using the default elasticsearch exporter configuration (where mapping: mode is set to otel):

gateway:
  ...
  config:
    service:
      pipelines:
        metrics:
          ...
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [elasticsearch]
        traces:
          ...

Optimizing the Signal

Once you have switched to the native OTel mode, you can further optimize your pipeline. Since you no longer need to maintain backward compatibility with ECS dashboards, you can modify your transform processor to remove the redundant ECS attributes (like ecs.version or specific labels) before they reach your storage.

This leaves you with a lean, clean, and fully standardized log stream that correlates seamlessly with your traces and metrics.

Summary

In this article, we demonstrated how to transition to an OTel-native instrumentation and collection architecture using the Elastic Distribution of OpenTelemetry (EDOT).

We adopted a hybrid approach: we established a vendor-neutral OTel observability stack (for traces, metrics, and logs) while guaranteeing full backward compatibility with existing ECS-based components like dashboards and pipelines.

In the end we demonstrated how once the full collecting architecture deployed, it is as easy as clicking on a button to deploy a fully compatible OTEL native collector to send logs in SemConv to your Elasticsearch backend. This strategy enables teams to transition smoothly and regularly to a full OTel-ready environment without disruption.