Traffic grows. Teams want metrics derived from traces. The backend goes down for maintenance and you lose an hour of telemetry. A compliance requirement means PII must be stripped before data leaves the cluster. Suddenly, that single collector isn't enough — and the question becomes: what should the architecture actually look like?

The OpenTelemetry Collector is designed to be composed. It can run in multiple deployment modes, be chained into pipelines, and scaled independently at each stage. But the documentation describes individual components, not how to think about assembling them. That thinking is what this article is about.

What follows is a conceptual framework for reasoning about collector architectures — not a set of rigid templates. The building blocks described here are reference points. In practice, they combine, overlap, and adapt to your constraints. A tail sampling tier might also need Kafka-backed resilience. A gateway might absorb the role of a sampling tier at low volumes. The goal is to understand the concepts well enough to compose the right architecture for your situation, not to pick a pre-built one off a shelf.

Three conceptual layers

It helps to think about collector architectures in three layers: edge, processing, and resilience. These aren't physical tiers that must exist as separate deployments — they're categories of concern. A single collector can address multiple layers. A complex deployment might have several components within one layer. The layers are a thinking tool, not a deployment diagram.

Edge: how telemetry enters the pipeline

The edge layer is about the first hop — how telemetry gets from your applications and infrastructure into the pipeline. At this stage, the collector gathers data in two fundamentally different ways. Pull-based receivers like filelog and hostmetrics actively reach out to collect data — tailing log files on disk or scraping system-level metrics from the host. Push-based receivers like otlp listen for data sent to them — applications instrumented with OpenTelemetry SDKs export traces, metrics, and logs directly to the collector's OTLP endpoint. A single edge collector typically runs both: pull receivers for infrastructure telemetry the application doesn't know about, and push receivers for application telemetry the SDK produces. There are several common deployment patterns, and the right one depends on your environment and what you need to collect.

DaemonSet Agent — One OpenTelemetry Collector per Kubernetes node, deployed as a DaemonSet. Applications export to the agent running on the same node (typically via status.hostIP:4317 using the Kubernetes Downward API). The agent also tails container log files from disk via the filelog receiver and scrapes host-level metrics via the hostmetrics receiver. This is the most common Kubernetes pattern because it handles both application and infrastructure telemetry with a single deployment, and applications only need to know about localhost.

Sidecar Agent — One OpenTelemetry Collector per pod, deployed as a sidecar container. Each service gets its own collector with a custom configuration. This is required on managed container platforms like AWS Fargate or Azure Container Apps where DaemonSets aren't available, and it's useful when services have different processing requirements. When running alongside a DaemonSet, the sidecar handles application telemetry while the DaemonSet independently collects node-level telemetry — applications don't send to both.

Host Agent — A standalone OpenTelemetry Collector running as a systemd service on bare-metal or VM hosts. It serves the same role as the DaemonSet agent but outside Kubernetes: collecting host metrics, tailing log files, and receiving OTLP from local applications.

Direct SDK Export — Applications export directly to the next stage (gateway or backend) with no local collector. This is the simplest option but only works when you don't need infrastructure collection. For log collection, the recommended pattern is still to write to stdout and use a collector with the filelog receiver — even if the SDK is exporting traces and metrics directly.

These patterns aren't mutually exclusive. A Kubernetes cluster might run DaemonSet agents for infrastructure collection alongside sidecars for services that need custom processing. A VM environment might use host agents for some services and direct SDK export for others. The edge layer is about matching the collection pattern to the workload, not picking one pattern for everything.

Processing: central policy, sampling, and transformation

Not every architecture needs a processing layer. If your edge collectors can export directly to your backend and you don't need centralized policy, you can skip it to favour simplicity. But several scenarios push you toward central processing — and the way you address them can range from a single gateway to a multi-stage pipeline.

Centralized policy (Gateway) — A pool of OpenTelemetry Collectors that sits between edge collectors and the backend. This is where you enforce consistent filtering, transformation, and PII redaction across all services. It's also where you manage backend credentials — edge collectors export to the gateway over OTLP, and only the gateway holds the API keys. Credential isolation is often the primary reason teams add a gateway.

Replica count scales with data volume. At low volumes (under 1K events/sec), 2 replicas co-located with workloads is sufficient. At medium volumes, 3–5 replicas on a dedicated node pool. At high volumes, 5–20+ replicas, potentially in a separate cluster. This is a general rule of thumb, and you should adapt it to your specific needs as loads might vary significantly between payload types.

Tail-based sampling — Sampling decisions that consider the complete trace (e.g., "keep all traces with errors, sample 10% of successful traces") require that all spans of a trace reach the same collector instance. This is achieved with the loadbalancingexporter using routing_key: traceID, which consistently routes spans from the same trace to the same downstream collector.

There's a critical subtlety here: if you're deriving span metrics (RED metrics) from traces using the spanmetrics connector, the derivation must happen before sampling. Otherwise, your metrics only reflect the sampled subset, not the true traffic. The correct pattern is a two-step pipeline within the sampling stage:

1. Receive traces, derive spanmetrics from 100% of traffic, forward via a forward connector.
2. Apply tail_sampling to the forwarded traces, export only kept traces.
3. A separate metrics pipeline exports the derived RED metrics.

This ensures accurate metrics regardless of your sampling rate.

The key point about processing is that these capabilities — gateway policy, tail sampling, span metrics derivation — are not separate products or fixed modules. They're configurations of the same OpenTelemetry Collector. At low volumes, a single gateway deployment might handle policy enforcement, sampling, and metrics derivation all at once. At high volumes, you might split them into dedicated stages for independent scaling. The architecture adapts to your scale, not the other way around.

Resilience: what happens when the backend is down

The resilience layer determines how much data you're willing to lose during backend outages or collector restarts. This isn't a separate tier you bolt on — it's a property you apply to any stage of the pipeline.

In-Memory Queues — The default. The collector's sending_queue retries failed exports with exponential backoff. If the collector process crashes or restarts, queued data is lost. This is acceptable for development and for workloads where some data loss during incidents is tolerable.

Persistent Queues (WAL) — The file_storage extension writes queued data to disk before export. If the collector crashes, it resumes from where it left off after restart. In Kubernetes, this requires a PersistentVolumeClaim. This is the right choice for most production workloads — it survives collector restarts and brief backend outages without the operational complexity of an external message bus.

Kafka Buffer — An external Kafka cluster sits between collectors and the backend. Producer collectors write to Kafka topics; consumer collectors read from Kafka and export to the backend. This provides the strongest durability guarantee — Kafka can buffer hours of telemetry during extended outages and enables replay. But it adds significant operational complexity.

The important thing to understand is that resilience is orthogonal to the other layers. You can add persistent queues to an edge agent, a gateway, or a sampling tier. You can put Kafka in front of a gateway, in front of a sampling tier, or in front of the backend. A tail sampling deployment that needs to survive extended outages might use Kafka-backed ingestion — combining what might look like two separate "modules" into a single stage. The building blocks compose freely based on what you need to protect against.

Where to start with your architecture

The Agent + Gateway two-tier pattern is the de facto production standard, used by the vast majority of organizations running OpenTelemetry at scale. DaemonSet agents on every node handle local collection — pulling infrastructure telemetry via filelog and hostmetrics, receiving application telemetry via OTLP — while a centralized gateway pool enforces policy, manages credentials, and exports to the backend. Persistent queues (WAL) on the gateway protect against backend outages without external dependencies.

Every other configuration either simplifies this pattern or extends it. Smaller environments might drop the gateway and export directly from agents. Larger ones might add a tail sampling tier with traceID-based load balancing, a Kafka buffer for extended resilience, or span metrics derivation before sampling. The building blocks described in the previous sections — edge, processing, resilience — are the modules you add or remove from this foundation.

The key is to start with the two-tier pattern and evolve incrementally:

• Need credential isolation or centralized PII redaction? You already have the gateway.
• Need tail-based sampling? Add a load-balancing exporter and a sampling tier between agents and gateway.
• Need hours of buffer during extended outages? Insert Kafka between agents and the processing tier.
• Running on Fargate or Azure Container Apps? Swap DaemonSet agents for sidecars — the rest of the pipeline stays the same.

Start here. Add modules as your needs grow. The architecture adapts to your scale, not the other way around.

Decision points that shape where you need to take your architecture

When designing a collector architecture, these are the questions that determine which patterns you need:

| Question | Impact | |----------|--------| | Do I need infrastructure telemetry (host metrics, disk logs)? | Determines whether you need a local collector or can use direct SDK export | | Am I on a managed container platform (Fargate, ACA)? | Forces sidecar pattern instead of DaemonSet | | Do I need centralized filtering, PII redaction, or credential isolation? | Adds a gateway stage | | Do I need tail-based sampling? | Adds a sampling stage with load-balancing exporter and traceID routing | | Do I want span-derived metrics (RED metrics)? | Requires spanmetrics before sampling in a two-step pipeline | | How much data loss is acceptable during outages? | Determines in-memory queues vs. persistent queues vs. Kafka — applied to whichever stage needs protection | | What is my expected data volume? | Determines whether capabilities can be co-located in a single deployment or need dedicated stages |

The answers to these questions don't map to a single "correct" architecture. They constrain the design space, and within those constraints, you make trade-offs between simplicity and capability.

Exploring these patterns interactively

If you'd rather explore how these building blocks compose than assemble them by hand, OpenTelemetry Blueprints is an open-source tool that generates reference architectures from your requirements.

Toggle your environment, signals, volume, resilience, and processing needs — and get a composed diagram with animated data flow, interactive tooltips, and reference collector configurations you can open directly in OTelBin for validation.

The generated configurations export via OTLP, so they work with any OTLP-compatible backend — including Elastic Observability, which natively accepts and stores OTLP traces, metrics, and logs.

The architectures Blueprints generates are reference compositions — starting points for understanding how the building blocks fit together, not turnkey deployments. Every architecture should be adapted to your organisation's scale, security, networking, and compliance requirements. The patterns might combine or overlap differently in your environment than in anyone else's, and that's the point.

Get started

The architectures described here export over OTLP, so they work with any compatible backend. If you don't have one yet, the fastest way to see your telemetry flowing end-to-end is with Elastic Observability — it natively ingests OTLP traces, metrics, and logs with no additional configuration.

1. Start a free trial on Elastic Cloud Serverless — no credit card required.
2. Point your collector's OTLP exporter at the managed OTLP endpoint.
3. Explore your traces, metrics, and logs in Kibana within minutes.

Check out these resources to go further:

• Elastic's managed OTLP endpoint documentation
• EDOT Collector — Elastic's distribution of the OpenTelemetry Collector
• OpenTelemetry Blueprints — generate reference architectures interactively

Today, we introduce EDOT Cloud Forwarder, built on OTel Collector, it is the simplest, fastest, and possibly most boring way to connect your Cloud environment to Elastic Observability, and it is now Generally Available on AWS. With EDOT Cloud Forwarder you can get started in seconds, get observability across your entire cloud estate, and easily handle telemetry at any volume.

Deploying Cloud Forwarder from a District Line train

So, I got to work deploying EDOT Cloud Forwarder in my AWS account. I was doing it on the commute, using nothing more than a decent 4G signal and hoping that my 27% battery would be enough.

I hit deploy on the terraform template and waited. I started seeing events flowing into Elastic Observability.

As the train pulled into Putney Bridge, the flow of logs peaked, and one million events per second scrolled across my screen.

Once deployed, I got the best three zeros I could expect:

• Zero Intervention: I watched as traffic ramped up to a full 1M EPS. Lambda functions automatically scaled out from a few instances to the 60-65 concurrent executions needed. There were zero manual adjustments required. The scaling was instant and hands-free.
• Zero Data Loss: It achieved a consistent processing rate, with every single event indexed in Elasticsearch.
• Zero Idle Cost: When there are no events, Cloud Forwarder scales to zero - it has no fixed infrastructure cost. You only pay for the moment data is being processed, not for permanently over-provisioned servers sitting idle.

Right before the train came to a standstill, I looked at the total cost for running Cloud Forwarder for the two minutes between Parsons Green and Putney Bridge - we forwarded about 120GB of telemetry and the total cost was below £0.10. Well, unless you count the £2.50 train ticket!

Making Observability easy at any scale

Getting started observing your infrastructure is hard and once it's observable, deriving actionable value from it requires sifting and winnowing through massive volumes of telemetry data, sometimes millions of events per second.

The new EDOT Cloud Forwarder for AWS (also available in Preview for GCP and Azure) is easy to deploy, with just a Terraform template. To make sure that it was easy to get started with, we designed it to be as close to a single-click deployment as possible:

Just click the link below to launch the CloudFormation stack in your AWS account:

The best part? The fastest way to get started with Elastic Observability scales to any size workload! With EDOT Cloud Forwarder you have one solution which automatically scales down to zero and up to millions of events per second.

So, what is EDOT Cloud Forwarder?

EDOT Cloud Forwarder is a serverless OpenTelemetry Collector that, on AWS, runs as a Lambda function. In AWS, it is triggered by events and processes logs and metrics from services such as VPC Flow Logs, ELB Access Logs, CloudTrail, CloudWatch Logs and CloudWatch Metrics.

It has the following core capabilities:

• Collects observability and security data from Cloud Service Providers
• Parses data into native OpenTelemetry format
• Forwards data over OTLP
• Scales up and down based on traffic

ECF for AWS is a pure serverless solution, no VMs, containers, or Kubernetes control planes to manage.

Off the train, a more controlled scenario

For a more controlled testing scenario, we used synthetic VPC Flow Log data generated to show how easy EDOT Cloud Forwarder can sustain a million events per second reliably and without data loss.

For the configuration, we left all EDOT Cloud Forwarder configuration/settings at their defaults. In AWS, the Lambda max concurrency defaults to 5. If left to the default of 5, up to around 50k EPS could be expected. For our scenario, we bumped this to 100 to ensure we'd have plenty of headroom for our test.

We ran the scenario in 10 minute stages, with each stage resulting in a larger data volume. We held ingest flat during the stage to provide short-term steady state windows for us to grab metrics.

We experienced no errors across all stages of the scenarios, no retries outside expected behavior, no data loss across 5.4 billion ingested events.

Stats for Observability Nerds

Because we know you love them:

Incremental Load Stages

We tested EDOT Cloud Forwarder using incremental load stages, gradually increasing traffic from approximately 300,000 events per second to over 1 million events per second.

The graph below shows the Elasticsearch ingestion rate throughout the entire test duration. You can see the clear progression as we ramped up through six distinct stages, with each plateau representing a 10 minute stabilization period.

The system handled each traffic increase smoothly, culminating in sustained 1 million documents per second ingestion with no bottlenecks or data loss.

Lambda

As seen in CloudWatch metrics (no manual adjustments required). No errors and no throttles were seen during the full duration of the test.

Concurrent executions

60 to 65 instances running at the same time.

Average execution time per Lambda

Each execution is taking about 5 seconds.

Memory Usage

Memory use stabilized at around 450 MB, below the default limit of 512 MB.

Elasticsearch Indexing

Elasticsearch indexed one million documents per second, with events visible in Discover within seconds and no indexing delays or bottlenecks.

Efficient by design: Lambda at 1M EPS from S3

Running ECF for AWS at 1M EPS costs about $3.87 per hour. Around 66 percent ($2.57 per hour) is data transfer (same region), 34 percent ($1.32 per hour) is Lambda compute, and less than 1 percent is S3 requests.

This is fully serverless with no idle cost. You only pay while events are forwarded. With S3, data arrives pre-batched in large objects, which keeps Lambda invocations low and compute costs tightly controlled. At sustained throughput, Lambda costs are comparable to EKS—but without cluster management or idle capacity.

Other options: OTel Collector on EKS at 1M events per second

An OTel Collector on EKS sized for 1M EPS has a baseline cost of about $3.69 per hour. Roughly $0.33 per hour of this is compute related, EC2 nodes, EKS control plane, and EBS. The rest comes from data transfer and SQS, which scale with traffic and do not change with utilization.

Idle compute impact on EKS real costs

Considering EKS is typically provisioned for peak load, the real cost of the compute portion is affected by idle capacity. At 100 percent utilization, total cost is $3.69 per hour. At 50 percent utilization, a common baseline to absorb burstiness, total cost rises to about $4.02 per hour. At 30 percent utilization, it increases further to about $4.46 per hour.

Pay for Work vs Pay for Capacity

ECF for AWS delivers 1M EPS at a cost comparable to EKS at peak utilization, with no idle compute or capacity planning required. EKS can reach the same peak throughput, but total cost increases further as average utilization drops because compute capacity must be provisioned in advance.

Conclusion

The boring truth: to the EDOT Cloud Forwarder, a million events per second is no different from any other workload.

With no infrastructure to deploy, no idle cost and no manual scaling, I guess the best thing to do is to stop overthinking and start forwarding! It's like stepping onto the fastest train on the line: you just get on and you're instantly en route to your destination, effortlessly handling any distance or in this case, any volume.

So, we're shipping it. ECF for AWS is now Generally Available.

Get Started

1. Deploy EDOT Cloud Forwarder via CloudFormation (below) or using the AWS Serverless Application Repository

1. Create an Observability project using an Elastic Cloud free trial or deploy locally with start-local if you don't already have an Elastic project or deployment.

Visit EDOT Cloud Forwarder for AWS Documentation for more details.

Check out these other resources on OpenTelemetry at Elastic

Discover how Elastic is evolving data ingestion with OpenTelemetry

Learn how OpAMP enables centralized configuration of OpenTelemetry SDKs

• Native OTel Integrity: Elastic is now 100% OTel-native, retaining OTel data natively without requiring data translation This eliminates the need for SREs to handle tedious schema conversions and develop customized views. All Elastic Observability capabilities—such as entity discovery, entity-centric insights, APM, infrastructure monitoring, and AI-driven issue analysis— now seamlessly work with native OTel data.

• Powerful end to end OTel based Kubernetes observability with Elastic Distributions of OpenTelemetry (EDOT): Elastic now supports EDOT deployment and management on Kubernetes via the OTel Operator, enabling streamlined EDOT collector deployment, application auto-instrumentation, and lifecycle management. With out-of-the-box OTel-based Kubernetes integration and dashboards, SREs gain instant, real-time visibility into cluster and application metrics, logs, and traces—with no manual configuration needed.

For organizations, it signals our commitment to open standards, streamlined data collection, and delivering insights from native OpenTelemetry data. Bring the power of Elastic Observability to your Kubernetes and OpenTelemetry deployments for maximum visibility and performance. 

Fully native OTel architecture with in-depth data analysis

Elastic’s OpenTelemetry-first architecture is 100% OTel-native, fully retaining the OTel data model, including OTel Semantic Conventions and Resource attributes, so your observability data remains in OpenTelemetry standards. OTel data in Elastic is also backward compatible with the Elastic Common Schema (ECS).

SREs now gain a holistic view of resources, as Elastic accurately identifies entities through OTel resource attributes. For example, in a Kubernetes environment, Elastic identifies containers, hosts, and services and connects these entities to logs, metrics, and traces.

Once OTel data is in Elastic’s scalable vector datastore, Elastic’s capabilities such as the AI Assistant, zero-config machine learning-based anomaly detection, pattern analysis, and latency correlation empower SREs to quickly analyze and pinpoint potential issues in production environments.

Kubernetes insights with Elastic Distributions of OpenTelemetry (EDOT)

EDOT reduces manual effort through automated onboarding and pre-configured dashboards. With EDOT and OpenTelemetry, Elastic makes Kubernetes monitoring straightforward and accessible for organizations of any size.

EDOT paired with Elasticsearch,  enables storage for all signal types—logs, metrics, traces, and soon profiling—while maintaining essential resource attributes and semantic conventions.

Elastic’s OpenTelemetry-native solution enables customers to quickly extract insights from their data rather than manage complex infrastructure to ingest data. Elastic automates the deployment and configuration of observability components to deliver a user experience focused on ease and scalability, making it well-suited for large-scale environments and diverse industry needs.

Let’s take a look at how Elastic’s EDOT enables visibility into Kubernetes environments.

1. Simple 3-step OTel ingest with lifecycle management and auto-instrumentation 

Elastic leverages the upstream OpenTelemetry Operator to automate its EDOT lifecycle management—including deployment, scaling, and updates—allowing customers to focus on visibility into their Kubernetes infrastructure and applications instead of their observability infrastructure for data collection.

The Operator integrates with the EDOT Collector and language SDKs to provide a consistent, vendor-agnostic experience. For instance, when customers deploy a new application, they don’t need to manually configure instrumentation for various languages; the OpenTelemetry Operator manages this through auto-instrumentation, as supported by the upstream OpenTelemetry project.

This integration simplifies observability by ensuring consistent application instrumentation across the Kubernetes environment. Elastic’s collaboration with the upstream OpenTelemetry project strengthens this automation, enabling users to benefit from the latest updates and improvements in the OpenTelemetry ecosystem. By relying on open source tools like the OpenTelemetry Operator, Elastic ensures that its solutions stay aligned with the latest advancements in the OpenTelemetry project, reinforcing its commitment to open standards and community-driven development.

The diagram above shows how the operator can deploy multiple OTel collectors, helping SREs deploy individual EDOT Collectors for specific applications and infrastructure. This configuration improves availability for OTel ingest and the telemetry is sent directly to Elasticsearch servers via OTLP.

Check out our recent blog on how to set this up.

2. Out-of-the-box OTel-based Kubernetes integration with dashboards

Elastic delivers an OTel-based Kubernetes configuration for the OTel collector by packaging all necessary receivers, processors, and configurations for Kubernetes observability. This enables users to automatically collect, process, and analyze Kubernetes metrics, logs, and traces without the need to configure each component individually.

The OpenTelemetry Kubernetes Collector components provide essential building blocks, including receivers like the Kubernetes Receiver for cluster metrics, Kubeletstats Receiver for detailed node and container metrics, along with processors for data transformation and enrichment. By packaging these components, Elastic offers a turnkey solution that simplifies Kubernetes observability and eliminates the need for users to set up and configure individual collectors or processors.

This pre-packaged approach, which includes OTel-native Kibana assets such as dashboards, allows users to focus on analyzing their observability data rather than managing configuration details. Elastic’s Unified OpenTelemetry Experience ensures that users can harness OpenTelemetry’s full potential without needing deep expertise. Whether you’re monitoring resource usage, container health, or API server metrics, users gain comprehensive observability through EDOT.

For more details on OpenTelemetry Kubernetes Collector components, visit OpenTelemetry Collector Components.

3. Streamlined ingest architecture with OTel data and Elasticsearch

Elastic’s ingest architecture minimizes infrastructure overhead by enabling users to forward trace data directly into Elasticsearch with the EDOT Collector, removing the need for the Elastic APM server. This approach:

• Reduces the costs and complexity associated with maintaining additional infrastructure, allowing users to deploy, scale, and manage their observability solutions with fewer resources.

• Allows all OTel data, metrics, logs, and traces to be ingested and stored in Elastic’s singular vector database store enabling further analysis with Elastic’s AI-driven capabilities.

SREs can now reduce operational burdens while also gaining high performance analytics and observability insights provided by Elastic.

Elastic’s ongoing commitment to open source and OpenTelemetry

With Elasticsearch fully open source once again under the AGPL license,  this change reinforces our deep commitment to open standards and the open source community. This aligns with Elastic’s OpenTelemetry-first approach to observability, where Elastic Distributions of OpenTelemetry (EDOT) streamline OTel ingestion and schema auto-detection, providing real-time insights for Kubernetes and application telemetry.

As users increasingly adopt OTel as their schema and data collection architecture for observability, Elastic’s Distribution of OpenTelemetry (EDOT), currently in tech preview, enhances standard OpenTelemetry capabilities and improves troubleshooting while also serving as a commercially supported OTel distribution. EDOT, together with Elastic’s recent contributions of the Elastic Profiling Agent and Elastic Common Schema (ECS) to OpenTelemetry, reinforces Elastic’s commitment to establishing OpenTelemetry as the industry standard.

Customers can now embrace open standards and enjoy the advantages of an open, extensible platform that integrates seamlessly with their environment. End result?  Reduced costs, greater visibility, and vendor independence.

Getting hands-on with Elastic Observability and EDOT

Ready to try out the OTel Operator with EDOT collector and SDKs to see how Elastic utilizes ingested OTel data in APM, Discover, Analysis, and out-of-the-box dashboards? 

• Get an account on Elastic Cloud

• Learn about Elastic Distributions of OpenTelemetry Overview

• Utilize the OpenTelemetry Demo with EDOT

• Understand how you can monitor Kubernetes with EDOT

• Utilize the EDOT Operator and the EDOT OTel collector

If you have your own application and want to configure EDOT the application with auto-instrumentation, read the following blogs on Go, Java, PHP, Python

• Auto-Instrumenting Go Applications with OpenTelemetry

• Elastic Distribution OpenTelemetry Java Agent

• Elastic OpenTelemetry Distribution for PHP

• Elastic OpenTelemetry Distribution for Python

The Elastic Distributions of OpenTelemetry (EDOT) are composed of OpenTelemetry (OTel) project components, OTel Collector, and language SDKs,  which provide users with the necessary capabilities and out-of-the-box configurations, enabling quick and effortless infra and application monitoring.

While OTel components are feature-rich, enhancements through the community can take time. Additionally, support is left up to the community or individual users and organizations. Hence EDOT will bring the following to end users:

• Deliver enhanced features earlier than OTel: By providing features unavailable in the “vanilla” OpenTelemetry components, we can quickly meet customers’ requirements while still providing an OpenTelemetry native and vendor-agnostic instrumentation for their applications. Elastic will continuously upstream these enhanced features.

• Enhanced OTel support - By maintaining Elastic distributions, we can better support customers with enhancements and fixes outside of the OTel release cycles. In addition, Elastic support can troubleshoot issues on the EDOT.

EDOT currently includes the following tech preview components, which will  grow over time:

• Elastic Distribution of OpenTelemetry (EDOT) Collector

• Elastic Distribution of OpenTelemetry (EDOT) Java.

• Elastic Distribution of OpenTelemetry (EDOT) Python

• Elastic Distribution of OpenTelemetry (EDOT) NodeJS

• Elastic Distribution of OpenTelemetry (EDOT) .NET

• Elastic Distribution of OpenTelemetry (EDOT)  iOS and Android

Details and documentation for all EDOT are available in our public OpenTelemetry GitHub repository. 

Elastic Distribution of OpenTelemetry (EDOT) Collector

The EDOT Collector, recently released with the 8.15 release of Elastic Observability enhances Elastic’s existing OTel capabilities. The EDOT Collector can, in addition to service monitoring, forward application logs, infrastructure logs, and metrics using standard OpenTelemetry Collector receivers like file logs and host metrics receivers.

Additionally, users of the Elastic Distribution of the OpenTelemetry Collector benefit from container logs automatically enriched with Kubernetes metadata by leveraging the powerful container log parser that Elastic recently contributed. This OpenTelemetry-based enrichment enhances the context and value of the collected logs, providing deeper insights and more effective troubleshooting capabilities.

This new collector distribution ensures that exported data is fully compatible with the Elastic Platform, enhancing the overall observability experience. Elastic also ensures that Elastic-curated UIs can seamlessly handle both the Elastic Common Schema (ECS) and OpenTelemetry formats.

Elastic Distributions for Language SDKs

Elastic's APM agents have capabilities yet to be available in the OTel SDKs. EDOT brings these capabilities into the OTel language SDKs while maintaining seamless integration with Elastic Observability. Elastic will release OTel versions of all its APM agents, and continue to add additional language SDKs mirroring OTel.

Continued support for Native OTel components

EDOT does not preclude users from using native components. Users are still able to use:

• OpenTelemetry Vanilla Language SDKs: use standard OpenTelemetry code instrumentation for many popular programming languages sending OTLP traces to Elastic via APM server.

• Upstream Distribution of OpenTelemetry Collector (Contrib or Custom): Send traces using the OpenTelemetry Collector with OTLP receiver and OTLP exporter to Elastic via APM server.

Elastic is committed to contributing EDOT features or components upstream into the OpenTelemetry community, fostering a collaborative environment, and enhancing the overall OpenTelemetry ecosystem.

Extending our commitment to vendor-agnostic data collection

Elastic remains committed to supporting OpenTelemetry by being OTel first and building a vendor-agnostic framework. As OpenTelemetry constantly grows its support of SDKs and components,  Elastic will continue to refine and mirror EDOT to OpenTelemetry and push enhancements upstream. 

Over the past year, Elastic has been active in OTel through its donation of Elastic Common Schema (ECS), contributions to the native OpenTelemetry Collector and language SDKs, and a recent donation of its Universal Profiling agent to OpenTelemetry. 

EDOT  builds on our decision to fully adopt and recommend OpenTelemetry as the preferred solution for observing applications. With EDOT, Elastic customers can future-proof their investments and adopt OpenTelemetry, giving them vendor-neutral instrumentation with Elastic enterprise-grade support.

Our vision is that Elastic will work with the OpenTelemetry community to donate features through the standardization processes and contribute the code to implement those in the native OpenTelemetry components. In time, as OTel capabilities advance, and many of the Elastic-exclusive features transition into OpenTelemetry, we look forward to no longer having Elastic Distributions for OpenTelemetry.. In the meantime, we can deliver those capabilities via our OpenTelemetry distributions.