<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
    <channel>
        <title>Elastic Observability Labs - Articles by Miguel Luna</title>
        <link>https://www.elastic.co/observability-labs</link>
        <description>Trusted security news &amp; research from the team at Elastic.</description>
        <lastBuildDate>Tue, 21 Jul 2026 16:45:20 GMT</lastBuildDate>
        <docs>https://validator.w3.org/feed/docs/rss2.html</docs>
        <generator>https://github.com/jpmonette/feed</generator>
        <image>
            <title>Elastic Observability Labs - Articles by Miguel Luna</title>
            <url>https://www.elastic.co/observability-labs/assets/observability-labs-thumbnail.png</url>
            <link>https://www.elastic.co/observability-labs</link>
        </image>
        <copyright>© 2026. Elasticsearch B.V. All Rights Reserved</copyright>
        <item>
            <title><![CDATA[Introducing Elastic Distribution of OpenTelemetry Collector]]></title>
            <link>https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-collector</link>
            <guid isPermaLink="false">elastic-distribution-opentelemetry-collector</guid>
            <pubDate>Fri, 09 Aug 2024 00:00:00 GMT</pubDate>
            <description><![CDATA[We are thrilled to announce the technical preview of the Elastic Distribution of OpenTelemetry Collector. This new offering underscores Elastic dedication to this important framework and highlights our ongoing contributions to make OpenTelemetry the best vendor agnostic data collection framework.]]></description>
            <content:encoded><![CDATA[<p>OpenTelemetry is an open-source framework that ensures vendor-agnostic data collection, providing a standardized approach for the collection, processing, and ingestion of observability data. Elastic is fully committed to this principle, aiming to make observability truly vendor-agnostic and eliminating the need for users to reinstrument their observability when switching platforms.</p>
<p>Over the past year, Elastic has made several notable contributions to the OpenTelemetry ecosystem. We <a href="https://opentelemetry.io/blog/2023/ecs-otel-semconv-convergence/">donated our Elastic Common Schema (ECS)</a> to OpenTelemetry, successfully <a href="https://opentelemetry.io/blog/2024/elastic-contributes-continuous-profiling-agent/">integrated the eBPF-based profiling agent</a>, and have consistently been one of the top contributing companies across the OpenTelemetry project. Additionally, Elastic has significantly improved upstream logging capabilities within OpenTelemetry with enhancements to key areas such as <a href="https://opentelemetry.io/blog/2024/otel-collector-container-log-parser/">container logging</a>, further enhancing the framework’s robustness.</p>
<p>These efforts demonstrate our strategic focus on enhancing and expanding the capabilities of OpenTelemetry for the broader observability community and reinforce the vendor-agnostic benefits of using OpenTelemetry.</p>
<p>Today, we are thrilled to announce the technical preview of the Elastic Distribution of OpenTelemetry Collector. This new offering underscores Elastic’s dedication to this important framework and highlights our ongoing contributions to make OpenTelemetry the best vendor agnostic data collection framework.</p>
<h2>Elastic Agent as an OpenTelemetry Collector&lt;a id=&quot;elastic-agent-as-an-opentelemetry-collector&quot;&gt;&lt;/a&gt;</h2>
<p>Technically, the Elastic Distribution of OpenTelemetry Collector represents an evolution of the Elastic Agent. In its latest version, the Elastic Agent can operate in an OpenTelemetry mode. This mode invokes a module within the Elastic Agent which is essentially a distribution of the OpenTelemetry collector. It is crafted using a selection of upstream components from the contrib distribution.</p>
<p>The Elastic OpenTelemetry Collector also includes configuration for this set of <a href="https://github.com/elastic/elastic-agent/tree/main/internal/pkg/otel#components">upstream OpenTelemetry Collector components</a>, providing out-of-the-box functionality with Elastic Observability. This integration allows users to seamlessly utilize Elastic’s advanced observability features with minimal setup.</p>
<p>The technical preview version of the Elastic OpenTelemetry Collector has been tailored with out-of-the-box configurations for the below use cases, we will keep working to add more as we progress: :</p>
<ul>
<li>
<p><strong><em>Collect and ship logs</em></strong>: Use the Elastic OpenTelemetry Collector to gather log data from various sources and ship it directly to Elastic where it can be analyzed in Kibana Discover, and Elastic Observability’s Explorer (also in Tech Preview in 8.15).</p>
</li>
<li>
<p><strong><em>Assess host health</em></strong>: Leverage the OpenTelemetry host metrics and Kubernetes receivers to monitor to evaluate the performance of hosts and pods. This data can then be visualized and analyzed in Elastic’s Infrastructure Observability UIs, providing deep insights into host performance and health. Details of how this is configured in the OTel collector is outlined in this <a href="https://www.elastic.co/observability-labs/blog/infrastructure-monitoring-with-opentelemetry-in-elastic-observability">blog</a>.</p>
</li>
<li>
<p><strong>Kubernetes container logs</strong>: Additionally, users of the Elastic OpenTelemetry Collector benefit from out-of-the-box Kubernetes container and application logs enriched with Kubernetes metadata by leveraging the powerful <a href="https://opentelemetry.io/blog/2024/otel-collector-container-log-parser/">container log parser</a> Elastic recently contributed to OTel. This OpenTelemetry-based enrichment enhances the context and value of the collected logs, providing deeper insights and more effective troubleshooting capabilities.</p>
</li>
</ul>
<p>While the Elastic OpenTelemetry Collector comes pre-built and preconfigured for the sake of easier onboarding and getting started experience, Elastic is committed to the vision of vendor-neutral collection of data. Thus, we strive to contribute any Elastic specific features back to the upstream OpenTelemetry components, to advance and help grow the OpenTelemetry landscape and capabilities.</p>
<p>Stay tuned for upcoming announcements sharing our plans to combine the best of Elastic Agent and OpenTelemetry Collector.</p>
<h2>Get started the Elastic Distribution of OpenTelemetry Collector&lt;a id=&quot;get-started-the-elastic-distribution-for-opentelemetry-collector&quot;&gt;&lt;/a&gt;</h2>
<p>To get started with a guided onboarding flow for the Elastic Distribution of the OpenTelemetry Collector for Kubernetes, Linux, and Mac environments, visit the<a href="https://github.com/elastic/opentelemetry/blob/main/docs/guided-onboarding.md"> guided onboarding documentation</a>.</p>
<p>For more advanced manual configuration, follow the<a href="https://github.com/elastic/opentelemetry/blob/main/docs/manual-configuration.md"> manual configuration instructions</a>.</p>
<p>Once the Elastic Distribution of the OpenTelemetry Collector is set up and running, you’ll be able to analyze your systems within various features of the Elastic Observability solution.</p>
<p>Analyze the performance and health of your infrastructure, through corresponding metrics and logs collected through OpenTelemetry Collector receivers, such as the host metrics receiver and different Kubernetes receivers.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/elastic-distribution-opentelemetry-collector/hosts.png" alt="OTel Monitoring Hosts" /></p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/elastic-distribution-opentelemetry-collector/otel-daemonset-green-logs.png" alt="OTel Logs" /></p>
<p>With Elastic OpenTelemetry Collector, container and application logs are enriched with Kubernetes metadata out-of-the-box making filtering, grouping and logs analysis easier and more efficient.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/elastic-distribution-opentelemetry-collector/explorer.png" alt="OTel Discover" /></p>
<p>The Elastic Distribution of the OpenTelemetry Collector allows for tracing just like any other collector distribution made of upstream components. Explore and analyze the performance and runtime behavior of your applications and services through RED metric, service maps and distributed traces collected from OpenTelemetry SDKs.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/elastic-distribution-opentelemetry-collector/apm.png" alt="OTel APM" /></p>
<p>The above capabilities and features packed with the Elastic OpenTelemetry Collector can be achieved in a similar way with a custom build of the upstream OpenTelemetry Collector packing the right set of upstream components. To do just that follow our <a href="https://www.elastic.co/docs/reference/opentelemetry/edot-collector/customization">guidance here</a>.</p>
<h2>Outlook&lt;a id=&quot;outlook&quot;&gt;&lt;/a&gt;</h2>
<p>The launch of the technical preview of the Elastic Distribution of OpenTelemetry Collector is another step on Elastic’s journey towards OpenTelemetry based observability. On that journey we are committed to a vendor-agnostic approach to data collection and therefore prioritize upstream contribution to OpenTelemetry over Elastic-specific data collection features.</p>
<p>Stay tuned to see more of Elastic’s contributions to OpenTelemetry and observe Elastic’s journey towards fully OpenTelemetry-based observability.</p>
<p>Additional resources for OpenTelemetry with Elastic:</p>
<ul>
<li>
<p>Elastic Distributions recently introduced:</p>
<ul>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-java-agent">Elastic Distribution of OpenTelemetry's Java SDK</a>.</p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-python">Elastic Distribution of OpenTelemetry's Python SDK</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-node-js">Elastic Distribution of OpenTelemetry's NodeJS SDK</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-dotnet-applications">Elastic Distribution of OpenTelemetry's .NET SDK</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/apm-ios-android-native-apps">Elastic Distribution of OpenTelemetry of iOS and Android</a></p>
</li>
</ul>
</li>
<li>
<p>Other Elastic OpenTelemetry resources:</p>
<ul>
<li>
<p><a href="https://www.elastic.co/blog/opentelemetry-observability">Independence with OpenTelemetry on Elastic</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/blog/implementing-kubernetes-observability-security-opentelemetry">Modern observability and security on Kubernetes with Elastic and OpenTelemetry</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/blog/3-models-logging-opentelemetry-elastic">3 models for logging with OpenTelemetry and Elastic</a></p>
</li>
</ul>
</li>
<li>
<p>Instrumentation resources:</p>
<ul>
<li>
<p>Python: <a href="https://www.elastic.co/blog/auto-instrumentation-of-python-applications-opentelemetry">Auto-instrumentation</a>, <a href="https://www.elastic.co/blog/manual-instrumentation-of-python-applications-opentelemetry">Manual instrumentation</a></p>
</li>
<li>
<p>Java: <a href="https://www.elastic.co/blog/auto-instrumentation-of-java-applications-opentelemetry">Auto-instrumentation</a>, <a href="https://www.elastic.co/blog/manual-instrumentation-of-java-applications-opentelemetry">Manual instrumentation </a></p>
</li>
<li>
<p>Node.js: <a href="https://www.elastic.co/blog/auto-instrument-nodejs-applications-opentelemetry">Auto-instrumentation</a>, <a href="https://www.elastic.co/blog/manual-instrumentation-of-nodejs-applications-opentelemetry">Manual instrumentation</a></p>
</li>
<li>
<p>.NET: <a href="https://www.elastic.co/blog/auto-instrumentation-of-net-applications-opentelemetry">Auto-instrumentation</a>, <a href="https://www.elastic.co/blog/manual-instrumentation-of-net-applications-opentelemetry">Manual instrumentation</a></p>
</li>
</ul>
</li>
</ul>
]]></content:encoded>
            <category>observability-labs</category>
            <enclosure url="https://www.elastic.co/observability-labs/assets/images/elastic-distribution-opentelemetry-collector/otel-collector-announcement.jpeg" length="0" type="image/jpeg"/>
        </item>
        <item>
            <title><![CDATA[Elastic Distributions of OpenTelemetry (EDOT) Now GA: Open-Source, Production-Ready OTel]]></title>
            <link>https://www.elastic.co/observability-labs/blog/elastic-distributions-opentelemetry-ga</link>
            <guid isPermaLink="false">elastic-distributions-opentelemetry-ga</guid>
            <pubDate>Wed, 02 Apr 2025 00:00:00 GMT</pubDate>
            <description><![CDATA[Elastic is proud to introduce General Availability of Elastic Distributions of OpenTelemetry (EDOT), which contains Elastic’s versions of the OpenTelemetry Collector and several language SDKs like Python, Java, .NET, and NodeJS. These help provide enhanced features and enterprise-grade support for EDOT.]]></description>
            <content:encoded><![CDATA[<p>We’re excited to announce <strong>general availability of</strong> <strong>Elastic Distributions of OpenTelemetry (EDOT)!</strong> EDOT is a fully open distribution of the OpenTelemetry collector and language SDKs, providing SREs and developers with a stable, production-tested OTel ecosystem backed by enterprise-grade support.</p>
<p>While OTel components are feature-rich, enhancements through the community can take time. Additionally, support is left up to the community or individual users and organizations.  EDOT delivers the following benefits to end users:</p>
<ul>
<li>
<p>Production-ready, backed by expert OTel support</p>
</li>
<li>
<p>No vendor lock-In - no proprietary add-ons</p>
</li>
<li>
<p>Preserving OpenTelemetry standards - no schema conversion</p>
</li>
</ul>
<h2>EDOT Collector and SDKs are GA</h2>
<p>Elastic Distributions of OpenTelemetry (EDOT) is a curated collection of OpenTelemetry components, EDOT Collector and language SDKs. It is designed to support OTel telemetry from applications and shared infrastructure such as hosts or Kubernetes.</p>
<p>Highlighted below are all the EDOT components that are now GA.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/elastic-distributions-opentelemetry-ga/edot-components.png" alt="EDOT Components" /></p>
<ul>
<li>
<p><strong>Elastic Distribution of OpenTelemetry (EDOT) Collector</strong> - The EDOT Collector is the OTel collector with Elastic’s set of processors, receivers and exporters to send OTel to Elastic</p>
</li>
<li>
<p><strong>Elastic Distribution of OpenTelemetry (EDOT) SDKs</strong> <strong>&amp; zero-code instrumentation</strong> - Users have an option to instrument with the SDKs or choose to use zero-code instrumentation. Here are all the SDKs currently available in EDOT:</p>
<ul>
<li>
<p>Elastic Distribution of OpenTelemetry (EDOT) Java.</p>
</li>
<li>
<p>Elastic Distribution of OpenTelemetry (EDOT) Python</p>
</li>
<li>
<p>Elastic Distribution of OpenTelemetry (EDOT) NodeJS</p>
</li>
<li>
<p>Elastic Distribution of OpenTelemetry (EDOT) .NET</p>
</li>
<li>
<p>Elastic Distribution of OpenTelemetry (EDOT) PHP</p>
</li>
<li>
<p>Elastic Distribution of OpenTelemetry (EDOT) iOS</p>
</li>
<li>
<p>Elastic Distribution of OpenTelemetry (EDOT) Android</p>
</li>
</ul>
</li>
</ul>
<p>Details and documentation for EDOT are available in our public <a href="https://www.elastic.co/docs/reference/opentelemetry/">EDOT documentation</a>, and our <a href="https://github.com/elastic/opentelemetry">EDOT github repository</a>.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/elastic-distributions-opentelemetry-ga/EDOT-Overview.png" alt="EDOT overview" /></p>
<p>To learn more about the ease of use, particularly with Kubernetes check out our previous blog <a href="https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-otel-operator">Ingest Kubernetes and application telemetry in 3 steps with EDOT</a>.</p>
<h2>What an SRE gains with EDOT</h2>
<p><strong>Production-ready, Backed by Expert OTel Support</strong></p>
<p>Enterprises adopting OpenTelemetry often struggle with unreliable support, slow bug fixes, and untested updates, leading to operational risks and downtime, and increased troubleshooting efforts. Without enterprise-grade guarantees, teams are left to resolve issues on their own, increasing maintenance overhead and slowing adoption.</p>
<p>EDOT delivers enterprise-grade support backed by OpenTelemetry experts, ensuring stability, proactive fixes beyond OpenTelemetry’s release cycles and production-tested reliability. With rapid issue resolution and expert guidance, EDOT enables organizations to confidently adopt and scale OpenTelemetry without operational disruptions or added maintenance burden.</p>
<p><strong>No Vendor Lock-In—No Proprietary Add-Ons</strong></p>
<p>Observability vendors have traditionally built proprietary agents and ingestion pipelines allowing them to control data flows and lock in users.</p>
<p>Elastic Distributions of OpenTelemetry (EDOT) offers a fully open, vendor-neutral approach to observability. As a curated portfolio of OpenTelemetry components, EDOT enhances infrastructure and application monitoring with Elastic Observability—without proprietary modifications.</p>
<p>All enhancements and fixes are contributed back to the OpenTelemetry community, ensuring EDOT remains a stable, standards-compliant distribution that stays aligned with upstream OpenTelemetry. This guarantees interoperability, seamless upgrades, and freedom from vendor lock-in.</p>
<p><strong>Preserving OpenTelemetry Standards for Richer Context</strong></p>
<p>When vendors modify OpenTelemetry data and schemas by introducing proprietary translations that disrupt interoperability, they create vendor lock-in and increase complexity. These modifications force operations teams to manage custom integrations, convert schemas, and sometimes result in each signal requiring its own query language and tooling, adding unnecessary overhead and limiting flexibility.</p>
<p>Elastic has re-architected its platform with an OTel-first approach that preserves the OpenTelemetry data model. OTel data can now be used in its original specification to power Elastic dashboards, analytics, alerts, and other functionality without the need for schema conversions – it just works.</p>
<p>With Elasticsearch as a single backend for all OpenTelemetry signals, users can store and query observability data in a unified, OTel-native format. Combined with ES|QL, a powerful and flexible query language, SREs get effortless correlation of logs, metrics, and traces using OpenTelemetry resource attributes. The result is a faster, more intuitive way to analyze system health and performance—all in one place.</p>
<h2>Get Started Today</h2>
<p>EDOT is available to all Elastic customers. Whether you’re adopting OpenTelemetry for the first time or looking for a reliable distribution with enterprise-grade support, EDOT ensures a smooth, OpenTelemetry-first experience.</p>
<p>Check out our <a href="https://www.elastic.co/docs/reference/opentelemetry/">EDOT documentation</a>, and our <a href="https://github.com/elastic/opentelemetry">EDOT github repository</a> and get started today!</p>
<p>Additionally check out some of the blogs detailing our components</p>
<ul>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-collector">Elastic Distribution of OpenTelemetry (EDOT) Collector</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-java-agent">Elastic Distribution of OpenTelemetry (EDOT) Java</a>.</p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-python">Elastic Distribution of OpenTelemetry (EDOT) Python</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-node-js">Elastic Distribution of OpenTelemetry (EDOT) NodeJS</a></p>
</li>
</ul>
]]></content:encoded>
            <category>observability-labs</category>
            <enclosure url="https://www.elastic.co/observability-labs/assets/images/elastic-distributions-opentelemetry-ga/edot-image.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Introducing Elastic Distributions of OpenTelemetry]]></title>
            <link>https://www.elastic.co/observability-labs/blog/elastic-distributions-opentelemetry</link>
            <guid isPermaLink="false">elastic-distributions-opentelemetry</guid>
            <pubDate>Thu, 15 Aug 2024 00:00:00 GMT</pubDate>
            <description><![CDATA[Elastic is proud to introduce Elastic Distributions of OpenTelemetry (EDOT), which contains Elastic’s versions of the OpenTelemetry Collector and several language SDKs like Python, Java, .NET, and NodeJS. These help provide enhanced features and enterprise-grade support for EDOT.]]></description>
            <content:encoded><![CDATA[<p>We are announcing the availability of Elastic Distributions of OpenTelemetry (EDOT). These Elastic distributions, currently in tech preview,  have been developed to enhance the capabilities of standard OpenTelemetry distributions and improve existing OpenTelemetry support from Elastic. </p>
<p>The Elastic Distributions of OpenTelemetry (EDOT) are composed of OpenTelemetry (OTel) project components, OTel Collector, and language SDKs,  which provide users with the necessary capabilities and out-of-the-box configurations, enabling quick and effortless infra and application monitoring.</p>
<p>While OTel components are feature-rich, enhancements through the community can take time. Additionally, support is left up to the community or individual users and organizations. Hence EDOT will bring the following to end users:</p>
<ul>
<li>
<p><strong>Deliver enhanced features earlier than OTel</strong>: By providing features unavailable in the “vanilla” OpenTelemetry components, we can quickly meet customers’ requirements while still providing an OpenTelemetry native and vendor-agnostic instrumentation for their applications. Elastic will continuously upstream these enhanced features.</p>
</li>
<li>
<p><strong>Enhanced OTel support</strong> - By maintaining Elastic distributions, we can better support customers with enhancements and fixes outside of the OTel release cycles. In addition, Elastic support can troubleshoot issues on the EDOT.</p>
</li>
</ul>
<p>EDOT currently includes the following tech preview components, which will  grow over time:</p>
<ul>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-collector">Elastic Distribution of OpenTelemetry (EDOT) Collector</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-java-agent">Elastic Distribution of OpenTelemetry (EDOT) Java</a>.</p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-python">Elastic Distribution of OpenTelemetry (EDOT) Python</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-node-js">Elastic Distribution of OpenTelemetry (EDOT) NodeJS</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-dotnet-applications">Elastic Distribution of OpenTelemetry (EDOT) .NET</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/apm-ios-android-native-apps">Elastic Distribution of OpenTelemetry (EDOT)  iOS and Android</a></p>
</li>
</ul>
<p>Details and documentation for all EDOT are available in our public <a href="https://github.com/elastic/opentelemetry">OpenTelemetry GitHub repository</a>. </p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/elastic-distributions-opentelemetry/edot-components.png" alt="EDOT Components" /></p>
<h2>Elastic Distribution of OpenTelemetry (EDOT) Collector&lt;a id=&quot;elastic-distribution-of-opentelemetry-edot-collector&quot;&gt;&lt;/a&gt;</h2>
<p>The EDOT Collector, recently released with the 8.15 release of Elastic Observability enhances Elastic’s existing OTel capabilities. The EDOT Collector can, in addition to service monitoring, forward application logs, infrastructure logs, and metrics using standard OpenTelemetry Collector receivers like file logs and host metrics receivers.</p>
<p>Additionally, users of the Elastic Distribution of the OpenTelemetry Collector benefit from container logs automatically enriched with Kubernetes metadata by leveraging the powerful <a href="https://opentelemetry.io/blog/2024/otel-collector-container-log-parser/">container log parser</a> that Elastic recently contributed. This OpenTelemetry-based enrichment enhances the context and value of the collected logs, providing deeper insights and more effective troubleshooting capabilities.</p>
<p>This new collector distribution ensures that exported data is fully compatible with the Elastic Platform, enhancing the overall observability experience. Elastic also ensures that Elastic-curated UIs can seamlessly handle both the Elastic Common Schema (ECS) and OpenTelemetry formats.</p>
<h2>Elastic Distributions for Language SDKs&lt;a id=&quot;elastic-distributions-for-language-sdks&quot;&gt;&lt;/a&gt;</h2>
<p><a href="https://www.elastic.co/guide/en/apm/agent/index.html">Elastic's APM agents</a> have capabilities yet to be available in the OTel SDKs. EDOT brings these capabilities into the OTel language SDKs while maintaining seamless integration with Elastic Observability. Elastic will release OTel versions of all its APM agents, and continue to add additional language SDKs mirroring OTel.</p>
<h2>Continued support for Native OTel components&lt;a id=&quot;continued-support-for-native-otel-components&quot;&gt;&lt;/a&gt;</h2>
<p>EDOT does not preclude users from using native components. Users are still able to use:</p>
<ul>
<li>
<p><strong>OpenTelemetry Vanilla Language SDKs:</strong> use standard OpenTelemetry code instrumentation for many popular programming languages sending OTLP traces to Elastic via APM server.</p>
</li>
<li>
<p><strong>Upstream Distribution of OpenTelemetry Collector (Contrib or Custom):</strong> Send traces using the OpenTelemetry Collector with OTLP receiver and OTLP exporter to Elastic via APM server.</p>
</li>
</ul>
<p>Elastic is committed to contributing EDOT features or components upstream into the OpenTelemetry community, fostering a collaborative environment, and enhancing the overall OpenTelemetry ecosystem.</p>
<h2>Extending our commitment to vendor-agnostic data collection&lt;a id=&quot;extending-our-commitment-to-vendor-agnostic-data-collection&quot;&gt;&lt;/a&gt;</h2>
<p>Elastic remains committed to supporting OpenTelemetry by being OTel first and building a vendor-agnostic framework. As OpenTelemetry constantly grows its support of SDKs and components,  Elastic will continue to refine and mirror EDOT to OpenTelemetry and push enhancements upstream. </p>
<p>Over the past year, Elastic has been active in OTel through its <a href="https://opentelemetry.io/blog/2023/ecs-otel-semconv-convergence/">donation of Elastic Common Schema (ECS)</a>, contributions to the native <a href="https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-collector">OpenTelemetry Collector</a> and language SDKs, and a recent <a href="https://www.elastic.co/observability-labs/blog/elastic-profiling-agent-acceptance-opentelemetry">donation of its Universal Profiling agent</a> to OpenTelemetry. </p>
<p>EDOT  builds on our decision to fully adopt and recommend OpenTelemetry as the preferred solution for observing applications. With EDOT, Elastic customers can future-proof their investments and adopt OpenTelemetry, giving them vendor-neutral instrumentation with Elastic enterprise-grade support.</p>
<p>Our vision is that Elastic will work with the OpenTelemetry community to donate features through the standardization processes and contribute the code to implement those in the native OpenTelemetry components. In time, as OTel capabilities advance, and many of the Elastic-exclusive features transition into OpenTelemetry, we look forward to no longer having Elastic Distributions for OpenTelemetry.. In the meantime, we can deliver those capabilities via our OpenTelemetry distributions.</p>
]]></content:encoded>
            <category>observability-labs</category>
            <enclosure url="https://www.elastic.co/observability-labs/assets/images/elastic-distributions-opentelemetry/edot-image.png" length="0" type="image/png"/>
        </item>
        <item>
            <title><![CDATA[Elastic’s Managed OTLP Endpoint: Simpler, Scalable OpenTelemetry for SREs]]></title>
            <link>https://www.elastic.co/observability-labs/blog/elastic-managed-otlp-endpoint-for-opentelemetry</link>
            <guid isPermaLink="false">elastic-managed-otlp-endpoint-for-opentelemetry</guid>
            <pubDate>Thu, 14 Aug 2025 00:00:00 GMT</pubDate>
            <description><![CDATA[Streamline OpenTelemetry data ingestion with Elastic Observability's new managed OTLP endpoint available on Elastic Cloud Serverless. Get native OTel storage and Elastic-grade scaling for logs, metrics, and traces, simplifying observability for SREs.]]></description>
            <content:encoded><![CDATA[<p>We’re excited to announce the <strong>managed OTLP endpoint for Elastic Observability Serverless.</strong> This feature marks a major milestone in Elastic’s shift to OpenTelemetry as the backbone of our data ingestion strategy and makes it dramatically easier to get high-fidelity OpenTelemetry data into Elastic Cloud.</p>
<h2>What is Elastic’s Managed OTLP Endpoint?</h2>
<p>The managed OTLP endpoint delivers on that promise offering a fully hosted OpenTelemetry ingestion path that’s scalable, reliable, and designed from the ground up for OpenTelemetry.</p>
<p>Data from OpenTelemetry SDKs, OpenTelemetry Collectors, or any OTLP service can send data to the OTLP endpoint. The OTLP endpoint is available on Elastic Cloud Serverless, and is fully managed by Elastic. This helps minimize the burden on customers of managing the OpenTelemetry ingestion layer. Whenever your production environment scales, the OTLP end point will also auto scale without any management from an SRE.</p>
<p>OpenTelemetry data is stored without any schema translation, preserving both semantic conventions and resource attributes. Additionally, it supports ingesting OTLP logs, metrics, and traces in a unified manner, ensuring consistent treatment across all telemetry data. This marks a significant improvement over the existing functionality, which primarily focuses on traces and APM use cases.</p>
<p>Hence, SREs gain: </p>
<ul>
<li>
<p><strong>Native OTLP ingestion</strong> with Elastic-managed reliability and scale</p>
</li>
<li>
<p><strong>OTel-native data storage</strong>, enabling richer analytics and future-proof observability</p>
</li>
<li>
<p><strong>Elastic-grade scaling</strong>, ready for production and multi-tenant workloads</p>
</li>
<li>
<p><strong>Frictionless onboarding</strong>, with a drop-in endpoint for logs, metrics and traces..</p>
</li>
</ul>
<h2>Native OTLP ingestion</h2>
<p>Whether you are using native OTel SDKs, OpenTelemetry Collector, EDOT, or other OpenTelemetry instrumentation, the OTLP endpoint will ingest any native OTLP data.</p>
<p>The managed OTLP endpoint will automatically scale with Observability data that is notoriously bursty. A sudden spike in requests, a scaling event in Kubernetes, or a deployment gone sideways can lead to massive surges in telemetry, often when you need visibility the most. That’s exactly what the managed OTLP endpoint in Elastic Observability Serverless is built to handle.</p>
<p>This isn’t just a thin wrapper on a collector. It’s a <strong>multi-tenant, auto-scaling service</strong> architected to absorb high volumes of OpenTelemetry data without you having to manage infrastructure, pre-provision capacity, or worry about dropped data.</p>
<p>Whether you’re routing data directly from OpenTelemetry SDKs or via an intermediate Collector, Elastic handles the scale behind the scenes. The endpoint is designed to scale with your telemetry traffic and recover gracefully from bursts, giving you one less thing to monitor. Just point your instrumentation at the endpoint and let Elastic take care of the rest.</p>
<h2>Natively stored OpenTelemetry </h2>
<p>With this feature, developers can now <strong>send OpenTelemetry signals directly to an Elastic Cloud</strong> <strong>Serverless project</strong> using the OTLP output of a collector or SDK regardless of the distribution contrib, EDOT and any other distribution will work). </p>
<p>The endpoint also supports data forwarded from any OpenTelemetry Collectors, SDKs or OTLP compliant forwarder. This gives teams full control to send directly from an SDK or route, enrich, or batch telemetry when needed. Elasticsearch stores OpenTelemetry data using the OpenTelemetry data model, including resource attributes, to identify emitting entities and enable ES|QL queries that correlate logs, metrics, and traces.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/elastic-managed-otlp-endpoint-for-opentelemetry/resource-attributes.jpg" alt="OTel resource attributes" /></p>
<h2>Faster time-to-insight</h2>
<p>Whether you’re building in serverless, Kubernetes, or classic VMs, this endpoint lets you focus on instrumentation and insights—not ingestion plumbing. It dramatically shortens the time from telemetry to value, while embracing the OpenTelemetry data model by preserving the original attributes and built-in correlation</p>
<h2>Easy connectivity to Managed OTLP Endpoint</h2>
<p>Connecting to the Managed OTLP endpoint is as simple as setting your SDK or the OTel collector OTLP export setting to the Elastic Managed OTLP Endpoint URL, and authentication key. Getting your endpoint is extremely straight-forward, go to project management, then edit alias and you will find your project’s OTLP endpoint. </p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/elastic-managed-otlp-endpoint-for-opentelemetry/otlp-endpoint-config.jpg" alt="OTel OTLP endpoint" /></p>
<h2>Get Started Today</h2>
<p>The managed OTLP endpoint can be used today <strong>on Elastic Observability Serverless</strong>. Support for <strong>Elastic Cloud Hosted</strong> deployments is coming soon.</p>
<p>For more detail and examples, follow <a href="https://www.elastic.co/docs/reference/opentelemetry/motlp">this guide</a>.</p>
<p>Whether you’re running microservices in Kubernetes, workloads in serverless, or apps on classic VMs, the OTLP endpoint helps you <strong>streamline your observability pipeline</strong>, <strong>standardize on OpenTelemetry</strong>, and <strong>accelerate your mean time to resolution (MTTR)</strong>.</p>
<p>Also check out our OTel resources about instrumenting and ingesting OTel into Elastic</p>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-distributions-opentelemetry-ga">Elastic Distributions of OpenTelemetry</a></p>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-otel-operator">Monitoring Kubernetes with Elastic and OpenTelemetry</a></p>
<p><a href="https://www.elastic.co/observability-labs/blog/k8s-discovery-with-EDOT-collector">Dynamic workload discovery with EDOT Collector</a></p>
<p><a href="https://www.elastic.co/observability-labs/blog/assembling-an-opentelemetry-nginx-ingress-controller-integration">Assembling an OpenTelemetry NGINX Ingress Controller Integration</a></p>
]]></content:encoded>
            <category>observability-labs</category>
            <enclosure url="https://www.elastic.co/observability-labs/assets/images/elastic-managed-otlp-endpoint-for-opentelemetry/otlp-endpoint.jpg" length="0" type="image/jpg"/>
        </item>
        <item>
            <title><![CDATA[Now GA: Managed OTLP Endpoint on Elastic Cloud Hosted]]></title>
            <link>https://www.elastic.co/observability-labs/blog/elastic-managed-otlp-endpoint-ga-elastic-cloud-hosted</link>
            <guid isPermaLink="false">elastic-managed-otlp-endpoint-ga-elastic-cloud-hosted</guid>
            <pubDate>Thu, 19 Mar 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[The Elastic Managed OTLP Endpoint is now generally available on Elastic Cloud Hosted, bringing managed Kafka-backed resilience and native OTLP ingestion to any OpenTelemetry shipper.]]></description>
            <content:encoded><![CDATA[<p>The Elastic Managed OTLP Endpoint (mOTLP) is now generally available on Elastic Cloud Hosted. Any OTLP-compliant source, whether it's an upstream OpenTelemetry SDK, any Collector distribution, EDOT, or a custom forwarder, can send traces, metrics, and logs to Elastic Cloud without deploying or managing ingestion infrastructure.</p>
<p>mOTLP was <a href="https://www.elastic.co/observability-labs/blog/elastic-managed-otlp-endpoint-for-opentelemetry">already GA on Elastic Cloud Serverless</a>. With this release, Elastic Cloud Hosted deployments get the same managed ingestion path: an OpenTelemetry Collector-based architecture with Kafka-backed resilience that absorbs traffic spikes and protects against data loss during the moments that matter most.</p>
<h2>You only need to set two environment variables</h2>
<p>From the outside, sending OpenTelemetry data to Elastic using the Managed OTLP Endpoint is as simple as setting up these environment variables:</p>
<pre><code class="language-bash">export OTEL_EXPORTER_OTLP_ENDPOINT=&quot;https://&lt;your-motlp-endpoint&gt;&quot;
export OTEL_EXPORTER_OTLP_HEADERS=&quot;Authorization=ApiKey &lt;your-api-key&gt;&quot;
</code></pre>
<p>Two environment variables. That's the entire integration surface. No Collector gateway to deploy, no ingestion pipelines to manage, no credentials to distribute across edge agents.</p>
<p>What makes such a simple design possible is a full OpenTelemetry Collector-based ingestion architecture designed to handle the worst moments in production, from the traffic spike during a deployment rollout to the burst of error traces during an incident. These are the moments when telemetry matters most, and they're also the moments when ingestion is most likely to be overwhelmed.</p>
<p>The managed endpoint receives OTLP data through an OpenTelemetry Collector layer that buffers to a managed Kafka cluster before indexing into Elasticsearch. Kafka absorbs bursts, decouples ingestion from indexing, and provides durability guarantees that in-memory queues can't. If Elasticsearch is temporarily under pressure, data sits in Kafka rather than getting dropped. When pressure subsides, the buffer drains and everything catches up. This is the same <a href="https://www.elastic.co/observability-labs/blog/opentelemetry-collector-reference-architectures">resilience pattern</a> you'd build yourself with a Kafka exporter and Kafka receiver in a self-managed collector pipeline, except Elastic operates it for you.</p>
<p>This makes the entire path OpenTelemetry end-to-end:</p>
<ol>
<li>Your applications produce telemetry with OTel SDKs.</li>
<li>Your collectors (or SDKs directly) export over OTLP.</li>
<li>The managed endpoint receives that OTLP through an OTel Collector-based layer, buffers it through Kafka, and stores it natively in Elasticsearch using the OpenTelemetry data model.</li>
</ol>
<p>No proprietary protocols, no schema translation, no format conversion at any stage of the pipeline. Just pure OTel.</p>
<h2>Bring your OTLP data in, no matter the source</h2>
<p>The endpoint accepts standard OTLP over HTTP and gRPC. Any tool that speaks OTLP can send data to it.</p>
<p>This means you can send data from:</p>
<ul>
<li><strong>OpenTelemetry SDKs</strong> (upstream, EDOT, or any distribution) exporting directly from your application.</li>
<li><strong>OpenTelemetry Collectors</strong> (Contrib, EDOT, or custom builds) running as agents, gateways, or sidecars.</li>
<li><strong>EDOT Cloud Forwarder</strong> forwarding cloud provider logs and metrics.</li>
<li><strong>Any OTLP-compliant forwarder</strong> you've built or adopted.</li>
</ul>
<p>This is worth pausing on, because it's not how most vendors work.</p>
<p>Many observability backends require vendor-specific components in your pipeline. Even when those components live in the OpenTelemetry Collector Contrib repository, they often reshape your data on the way out. A vendor-specific exporter might flatten resource attributes, drop the hierarchy between resources and scopes, or translate semantic conventions into a proprietary schema. By the time your telemetry reaches the backend, it's no longer standard OpenTelemetry data. It just started as OpenTelemetry data.</p>
<p>Elastic doesn't require any of that. The managed endpoint ingests standard OTLP, which means you use the standard <code>otlphttp</code> or <code>otlp</code> exporter that ships with every OpenTelemetry Collector. No Elastic-specific exporter, no vendor plugin, no translation layer. Your data arrives in Elasticsearch with the same resource hierarchy, the same semantic conventions, and the same attribute structure your instrumentation produced.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/elastic-managed-otlp-endpoint-ga-elastic-cloud-hosted/motlp-reference-architecture.png" alt="Reference architecture: OTel SDKs and Collectors on the edge exporting over OTLP to the Managed OTLP Endpoint on Elastic Cloud" /></p>
<p>The practical result: teams adopt OpenTelemetry at different speeds and with different tools. Some start with the upstream SDK and a Contrib Collector. Others use EDOT for Elastic-specific optimizations. Many run a mix. The managed endpoint doesn't impose a choice, and it doesn't quietly reshape your data behind a vendor exporter.</p>
<p>For organizations already running OpenTelemetry in production, this means switching to Elastic or adding Elastic as a destination requires changing an exporter URL, not rearchitecting the pipeline or adding vendor-specific components.</p>
<h2>What you no longer have to manage</h2>
<p>Before the managed OTLP endpoint, sending OpenTelemetry data to Elastic Cloud Hosted required deploying your own OTLP-compatible ingestion layer, typically an <a href="https://www.elastic.co/docs/reference/edot-collector/modes#edot-collector-as-gateway">EDOT Collector running as a gateway</a>. That gateway needed to be sized, scaled, monitored, and kept available. If it went down, telemetry stopped flowing.</p>
<p>With mOTLP, that entire layer is Elastic's responsibility. Here's what moves off your plate:</p>
<ul>
<li>The endpoint scales with your traffic. On Elastic Cloud Hosted, rate limits scale dynamically based on Elasticsearch backpressure. No pre-provisioning required.</li>
<li>The Collector-to-Kafka buffer handles burst absorption and backpressure. You don't need to operate your own Kafka cluster.</li>
<li>Your shippers authenticate directly with the endpoint using an API key. No intermediate gateway holding and distributing backend credentials.</li>
<li>The endpoint is managed and multi-tenant. You don't need to run redundant Collector replicas or configure health checks for your ingestion layer.</li>
</ul>
<p>This doesn't mean you should remove all collectors from your architecture. Edge collectors (DaemonSet agents, sidecars, host agents) still serve a purpose: collecting infrastructure telemetry via pull-based receivers like <code>filelog</code> and <code>hostmetrics</code>, applying local transformations, and batching data before export. What changes is that the destination is now a managed endpoint rather than a self-operated gateway.</p>
<h2>Dynamic rate scaling: the system adapts to your cluster</h2>
<p>On Elastic Cloud Hosted, the managed endpoint doesn't have a fixed throughput ceiling. It uses dynamic rate scaling that adjusts based on your Elasticsearch cluster's capacity and current load.</p>
<p>This is a fundamentally different model from static rate limits. Instead of provisioning for peak and paying for idle capacity, the system continuously calibrates ingestion to what your cluster can actually handle. Sudden load spikes may still trigger temporary <code>429</code> responses while the system scales, but these resolve automatically.</p>
<p>If you are seeing consistent 429 errors, the signal is clear, your Elasticsearch cluster needs more capacity. Scaling the cluster reduces backpressure, which in turn increases the ingestion rate limit. The autoscaling capabilities in Elastic Cloud Hosted can help automate this, and AutoOps can assist by monitoring the deployment and providing recommendations to scale or adjust resources when capacity constraints are detected.</p>
<h2>Native OTLP storage: OTel from first mile to last</h2>
<p>The end-to-end OTel story doesn't stop at ingestion. Data that arrives through the managed endpoint is stored using the OpenTelemetry data model. Resource attributes, semantic conventions, and signal structure are preserved as-is. There's no translation to ECS or any other schema at any point in the pipeline.</p>
<p>This means the attribute names your SDK produces are the same you query in ES|QL and Discover: <code>service.name</code>, <code>http.request.method</code>, <code>k8s.pod.name</code> are stored exactly as the OpenTelemetry specification defines them. You're not debugging a mapping layer or wondering which schema translation dropped an attribute. What your instrumentation emits is what Elasticsearch stores and what you search.</p>
<p>If no specific dataset or namespace is configured, telemetry lands in default data streams: <code>traces-generic.otel-default</code>, <code>metrics-generic.otel-default</code>, and <code>logs-generic.otel-default</code>. You can route logs to dedicated datasets by setting the <code>data_stream.dataset</code> attribute, either in your collector configuration or via <code>OTEL_RESOURCE_ATTRIBUTES</code>:</p>
<pre><code class="language-yaml">processors:
  transform:
    log_statements:
      - set(log.attributes[&quot;data_stream.dataset&quot;], &quot;app.orders&quot;) where resource.attributes[&quot;service.name&quot;] == &quot;orders-service&quot;
</code></pre>
<h2>The failure store: a safety net for mapping conflicts</h2>
<p>Even with careful schema design, mapping conflicts happen. A field that's a string in one service might be an integer in another. In traditional setups, these conflicts cause indexing failures and data loss.</p>
<p>The managed endpoint has the <a href="https://www.elastic.co/docs/manage-data/data-store/data-streams/failure-store">failure store</a> enabled by default for all OTLP data streams. Documents that fail indexing due to mapping conflicts or ingest pipeline exceptions are stored in a separate index rather than dropped. You can inspect failed documents from the <a href="https://www.elastic.co/docs/solutions/observability/data-set-quality-monitoring">Data Set Quality</a> page and fix the underlying issue without losing the data.</p>
<p>This is particularly valuable in OpenTelemetry environments where multiple teams instrument independently and attribute types can drift across services.</p>
<h2>Getting started</h2>
<p>The managed OTLP endpoint is available today on Elastic Cloud Hosted deployments (version 9.0+) in <a href="https://www.elastic.co/docs/reference/opentelemetry/motlp">supported regions</a>.</p>
<p>To find your endpoint:</p>
<ol>
<li>Log in to the <a href="https://cloud.elastic.co">Elastic Cloud Console</a>.</li>
<li>Select your deployment and go to <strong>Manage</strong>.</li>
<li>In the <strong>Application endpoints</strong> section, select <strong>Managed OTLP</strong> and copy the public endpoint.</li>
</ol>
<p>Then point any OTLP exporter at it:</p>
<pre><code class="language-bash">export OTEL_EXPORTER_OTLP_ENDPOINT=&quot;https://&lt;your-motlp-endpoint&gt;&quot;
export OTEL_EXPORTER_OTLP_HEADERS=&quot;Authorization=ApiKey &lt;your-api-key&gt;&quot;
</code></pre>
<p>That's it. Traces, metrics, and logs will start flowing into your deployment within seconds.</p>
<p>For a detailed walkthrough, follow the <a href="https://www.elastic.co/docs/solutions/observability/get-started/quickstart-elastic-cloud-otel-endpoint">Send data to the Elastic Cloud Managed OTLP Endpoint</a> quickstart.</p>
<h2>Learn more</h2>
<ul>
<li><a href="https://www.elastic.co/docs/reference/opentelemetry/motlp">Managed OTLP Endpoint reference documentation</a></li>
<li><a href="https://www.elastic.co/observability-labs/blog/opentelemetry-collector-reference-architectures">Composing OpenTelemetry Reference Architectures</a></li>
<li><a href="https://www.elastic.co/docs/reference/opentelemetry/edot-collector">EDOT Collector documentation</a></li>
<li><a href="https://www.elastic.co/docs/reference/opentelemetry">OpenTelemetry at Elastic</a></li>
</ul>
]]></content:encoded>
            <category>observability-labs</category>
            <enclosure url="https://www.elastic.co/observability-labs/assets/images/elastic-managed-otlp-endpoint-ga-elastic-cloud-hosted/managed-otlp-endpoint-elastic-cloud-hosted.jpg" length="0" type="image/jpg"/>
        </item>
        <item>
            <title><![CDATA[Unlock possibilities with native OpenTelemetry: prioritize reliability, not proprietary limitations]]></title>
            <link>https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-native-kubernetes-observability</link>
            <guid isPermaLink="false">elastic-opentelemetry-native-kubernetes-observability</guid>
            <pubDate>Tue, 12 Nov 2024 00:00:00 GMT</pubDate>
            <description><![CDATA[Elastic now supports Elastic Distributions of OpenTelemetry (EDOT) deployment and management on Kubernetes, using OTel Operator. SREs can now access out-of the-box configurations and dashboards designed to streamline collector deployment, application auto-instrumentation and lifecycle management with Elastic Observability.]]></description>
            <content:encoded><![CDATA[<p>OpenTelemetry (OTel) is emerging as the standard for data ingestion since it delivers a vendor-agnostic way to ingest data across all telemetry signals. Elastic Observability is leading the OTel evolution with the following announcements:</p>
<ul>
<li>
<p><strong>Native OTel Integrity:</strong> Elastic is now 100% OTel-native, retaining OTel data natively without requiring data translation This eliminates the need for SREs to handle tedious schema conversions and develop customized views. All Elastic Observability capabilities—such as entity discovery, entity-centric insights, APM, infrastructure monitoring, and AI-driven issue analysis— now seamlessly work with  native OTel data.</p>
</li>
<li>
<p><strong>Powerful end to end OTel based Kubernetes observability with</strong> <a href="https://www.elastic.co/observability-labs/blog/elastic-distributions-opentelemetry"><strong>Elastic Distributions of OpenTelemetry (EDOT)</strong></a><strong>:</strong> Elastic now supports EDOT deployment and management on Kubernetes via the OTel Operator, enabling streamlined EDOT collector deployment, application auto-instrumentation, and lifecycle management. With out-of-the-box OTel-based Kubernetes integration and dashboards, SREs gain instant, real-time visibility into cluster and application metrics, logs, and traces—with no manual configuration needed.</p>
</li>
</ul>
<p>For organizations, it signals our commitment to open standards, streamlined data collection, and delivering insights from native OpenTelemetry data. Bring the power of Elastic Observability to your Kubernetes and OpenTelemetry deployments for maximum visibility and performance. </p>
<h1>Fully native OTel architecture with in-depth data analysis</h1>
<p>Elastic’s OpenTelemetry-first architecture is 100% OTel-native, fully retaining the OTel data model, including OTel Semantic Conventions and Resource attributes, so your observability data remains in OpenTelemetry standards. OTel data in Elastic is also backward compatible with the Elastic Common Schema (ECS).</p>
<p>SREs now gain a holistic view of resources, as Elastic accurately identifies entities through OTel resource attributes. For example, in a Kubernetes environment, Elastic identifies containers, hosts, and services and connects these entities to logs, metrics, and traces.</p>
<p>Once OTel data is in Elastic’s scalable vector datastore, Elastic’s capabilities such as the AI Assistant, zero-config machine learning-based anomaly detection, pattern analysis, and latency correlation empower SREs to quickly analyze and pinpoint potential issues in production environments.</p>
<h1>Kubernetes insights with Elastic Distributions of OpenTelemetry (EDOT)</h1>
<p>EDOT reduces manual effort through automated onboarding and pre-configured dashboards. With EDOT and OpenTelemetry, Elastic makes Kubernetes monitoring straightforward and accessible for organizations of any size.</p>
<p>EDOT paired with Elasticsearch,  enables storage for all signal types—logs, metrics, traces, and soon profiling—while maintaining essential resource attributes and semantic conventions.</p>
<p>Elastic’s OpenTelemetry-native solution enables customers to quickly extract insights from their data rather than manage complex infrastructure to ingest data. Elastic automates the deployment and configuration of observability components to deliver a user experience focused on ease and scalability, making it well-suited for large-scale environments and diverse industry needs.</p>
<p>Let’s take a look at how Elastic’s EDOT enables visibility into Kubernetes environments.</p>
<h2>1. Simple 3-step OTel ingest with lifecycle management and auto-instrumentation </h2>
<p>Elastic leverages the upstream OpenTelemetry Operator to automate its EDOT lifecycle management—including deployment, scaling, and updates—allowing customers to focus on visibility into their Kubernetes infrastructure and applications instead of their observability infrastructure for data collection.</p>
<p>The Operator integrates with the EDOT Collector and language SDKs to provide a consistent, vendor-agnostic experience. For instance, when customers deploy a new application, they don’t need to manually configure instrumentation for various languages; the OpenTelemetry Operator manages this through auto-instrumentation, as supported by the upstream OpenTelemetry project.</p>
<p>This integration simplifies observability by ensuring consistent application instrumentation across the Kubernetes environment. Elastic’s collaboration with the upstream OpenTelemetry project strengthens this automation, enabling users to benefit from the latest updates and improvements in the OpenTelemetry ecosystem. By relying on open source tools like the OpenTelemetry Operator, Elastic ensures that its solutions stay aligned with the latest advancements in the OpenTelemetry project, reinforcing its commitment to open standards and community-driven development.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/elastic-opentelemetry-native-kubernetes-observability/unified-otel-based-k8s-experience.png" alt="Unified OTel-based Kubernetes Experience" /></p>
<p>The diagram above shows how the operator can deploy multiple OTel collectors, helping SREs deploy individual EDOT Collectors for specific applications and infrastructure. This configuration improves availability for OTel ingest and the telemetry is sent directly to Elasticsearch servers via OTLP.</p>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-otel-operator">Check out our recent blog on how to set this up</a>.</p>
<h2>2. Out-of-the-box OTel-based Kubernetes integration with dashboards</h2>
<p>Elastic delivers an OTel-based Kubernetes configuration for the OTel collector by packaging all necessary receivers, processors, and configurations for Kubernetes observability. This enables users to automatically collect, process, and analyze Kubernetes metrics, logs, and traces without the need to configure each component individually.</p>
<p>The OpenTelemetry Kubernetes Collector components provide essential building blocks, including receivers like the Kubernetes Receiver for cluster metrics, Kubeletstats Receiver for detailed node and container metrics, along with processors for data transformation and enrichment. By packaging these components, Elastic offers a turnkey solution that simplifies Kubernetes observability and eliminates the need for users to set up and configure individual collectors or processors.</p>
<p>This pre-packaged approach, which includes <a href="https://github.com/elastic/integrations/tree/main/packages/kubernetes_otel">OTel-native Kibana assets</a> such as dashboards, allows users to focus on analyzing their observability data rather than managing configuration details. Elastic’s Unified OpenTelemetry Experience ensures that users can harness OpenTelemetry’s full potential without needing deep expertise. Whether you’re monitoring resource usage, container health, or API server metrics, users gain comprehensive observability through EDOT.</p>
<p>For more details on OpenTelemetry Kubernetes Collector components, visit<a href="https://opentelemetry.io/docs/kubernetes/collector/components/"> OpenTelemetry Collector Components</a>.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/elastic-opentelemetry-native-kubernetes-observability/otel-based-k8s-dashboard.png" alt="OTel-based Kubernetes Dashboard" /></p>
<h2>3. Streamlined ingest architecture with OTel data and Elasticsearch</h2>
<p>Elastic’s ingest architecture minimizes infrastructure overhead by enabling users to forward trace data directly into Elasticsearch with the EDOT Collector, removing the need for the Elastic APM server. This approach:</p>
<ul>
<li>
<p>Reduces the costs and complexity associated with maintaining additional infrastructure, allowing users to deploy, scale, and manage their observability solutions with fewer resources.</p>
</li>
<li>
<p>Allows all OTel data, metrics, logs, and traces to be ingested and stored in Elastic’s singular vector database store enabling further analysis with Elastic’s AI-driven capabilities.</p>
</li>
</ul>
<p>SREs can now reduce operational burdens while also gaining high performance analytics and observability insights provided by Elastic.</p>
<h1>Elastic’s ongoing commitment to open source and OpenTelemetry</h1>
<p>With <a href="https://www.elastic.co/blog/elasticsearch-is-open-source-again">Elasticsearch fully open source once again</a> under the AGPL license,  this change reinforces our deep commitment to open standards and the open source community. This aligns with Elastic’s OpenTelemetry-first approach to observability, where Elastic Distributions of OpenTelemetry (EDOT) streamline OTel ingestion and schema auto-detection, providing real-time insights for Kubernetes and application telemetry.</p>
<p>As users increasingly adopt OTel as their schema and data collection architecture for observability, Elastic’s Distribution of OpenTelemetry (EDOT), currently in tech preview, enhances standard OpenTelemetry capabilities and improves troubleshooting while also serving as a commercially supported OTel distribution. EDOT, together with Elastic’s recent contributions of the Elastic Profiling Agent and Elastic Common Schema (ECS) to OpenTelemetry, reinforces Elastic’s commitment to establishing OpenTelemetry as the industry standard.</p>
<p>Customers can now embrace open standards and enjoy the advantages of an open, extensible platform that integrates seamlessly with their environment. End result?  Reduced costs, greater visibility, and vendor independence.</p>
<h1>Getting hands-on with Elastic Observability and EDOT</h1>
<p>Ready to try out the OTel Operator with EDOT collector and SDKs to see how Elastic utilizes ingested OTel data in APM, Discover, Analysis, and out-of-the-box dashboards? </p>
<ul>
<li>
<p><a href="https://cloud.elastic.co/">Get an account on Elastic Cloud</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-distributions-opentelemetry">Learn about Elastic Distributions of OpenTelemetry Overview</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/opentelemetry-demo-with-the-elastic-distributions-of-opentelemetry">Utilize the OpenTelemetry Demo with EDOT</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/infrastructure-monitoring-with-opentelemetry-in-elastic-observability">Understand how you can monitor Kubernetes with EDOT</a></p>
</li>
<li>
<p><a href="https://github.com/elastic/opentelemetry">Utilize the EDOT Operator </a>and the <a href="https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-collector">EDOT OTel collector</a></p>
</li>
</ul>
<p>If you have your own application and want to configure EDOT the application with auto-instrumentation, read the following blogs on Go, Java, PHP, Python</p>
<ul>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/auto-instrumentation-go-applications-opentelemetry">Auto-Instrumenting Go Applications with OpenTelemetry</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-java-agent">Elastic Distribution OpenTelemetry Java Agent</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-php">Elastic OpenTelemetry Distribution for PHP</a></p>
</li>
<li>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-distribution-python">Elastic OpenTelemetry Distribution for Python</a></p>
</li>
</ul>
]]></content:encoded>
            <category>observability-labs</category>
            <enclosure url="https://www.elastic.co/observability-labs/assets/images/elastic-opentelemetry-native-kubernetes-observability/Kubecon-main-blog.jpg" length="0" type="image/jpg"/>
        </item>
        <item>
            <title><![CDATA[A train ride away from a million events per second with EDOT Cloud Forwarder]]></title>
            <link>https://www.elastic.co/observability-labs/blog/one-million-events-per-second-with-edot-cloud-forwarder</link>
            <guid isPermaLink="false">one-million-events-per-second-with-edot-cloud-forwarder</guid>
            <pubDate>Tue, 20 Jan 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[EDOT Cloud Forwarder for AWS from Elastic Observability is now Generally Available. Deploying EDOT Cloud Forwarder and reliably handling one million events per second with zero intervention, zero data loss, and zero idle cost.]]></description>
            <content:encoded><![CDATA[<p>Infrastructure observability is critical for maintaining uptime, optimizing cloud environments, and securing the cloud perimeter. Cloud environments generate observability data at massive scale. VPC Flow Logs, ELB Access Logs, CloudTrail and CloudWatch logs can easily reach hundreds of thousands of events per second. Dealing with scale like this is a complex problem all by itself.</p>
<p>Today, we introduce <strong>EDOT Cloud Forwarder</strong>, built on OTel Collector, it is the simplest, fastest, and possibly most boring way to connect your Cloud environment to Elastic Observability, and it is <strong>now Generally Available on AWS</strong>. With EDOT Cloud Forwarder you can get started in seconds, get observability across your entire cloud estate, and easily handle telemetry at any volume.</p>
<h2>Deploying Cloud Forwarder from a District Line train</h2>
<p>So, I got to work deploying EDOT Cloud Forwarder in my AWS account. I was doing it on the commute, using nothing more than a decent 4G signal and hoping that my 27% battery would be enough.</p>
<p>I hit deploy on the terraform template and waited. I started seeing events flowing into Elastic Observability.</p>
<p>As the train pulled into Putney Bridge, the flow of logs peaked, and one million events per second scrolled across my screen.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/one-million-events-per-second-with-edot-cloud-forwarder/putney_bridge_1MEPS.png" alt="Putney_Bridge" /></p>
<p>Once deployed, I got the best three zeros I could expect:</p>
<ul>
<li><strong>Zero Intervention:</strong> I watched as traffic ramped up to a full 1M EPS. Lambda functions automatically scaled out from a few instances to the 60-65 concurrent executions needed. There were <strong>zero manual adjustments required</strong>. The scaling was instant and hands-free.</li>
<li><strong>Zero Data Loss:</strong> It achieved a consistent processing rate, with every single event indexed in Elasticsearch.</li>
<li><strong>Zero Idle Cost:</strong> When there are no events, Cloud Forwarder scales to zero - it has no fixed infrastructure cost. You only pay for the moment data is being processed, not for permanently over-provisioned servers sitting idle.</li>
</ul>
<p>Right before the train came to a standstill, I looked at the total cost for running Cloud Forwarder for the two minutes between Parsons Green and Putney Bridge - we forwarded about 120GB of telemetry and the total cost was below £0.10. Well, unless you count the £2.50 train ticket!</p>
<h2>Making Observability easy at any scale</h2>
<p>Getting started observing your infrastructure is hard and once it's observable, deriving actionable value from it requires sifting and winnowing through massive volumes of telemetry data, sometimes millions of events per second.</p>
<p>The new EDOT Cloud Forwarder for AWS (also available in Preview for GCP and Azure) is easy to deploy, with just a Terraform template. To make sure that it was easy to get started with, we designed it to be as close to a <a href="https://www.elastic.co/docs/reference/opentelemetry/edot-cloud-forwarder/aws#quick-deployment-direct-link">single-click deployment</a> as possible:</p>
<p>Just click the link below to launch the CloudFormation stack in your AWS account:</p>
<p><a href="https://console.aws.amazon.com/cloudformation/home?%23/stacks/new?templateURL=https%3A%2F%2Fedot-cloud-forwarder.s3.amazonaws.com%2Fv1%2Flatest%2Fcloudformation%2Fs3_logs-cloudformation.yaml"><img src="https://www.elastic.co/observability-labs/assets/images/one-million-events-per-second-with-edot-cloud-forwarder/cloudformation-launch-stack.png" alt="Launch_stack" /></a></p>
<p>The best part? The fastest way to get started with Elastic Observability scales to any size workload! With EDOT Cloud Forwarder you have one solution which automatically scales down to zero and up to millions of events per second.</p>
<h2>So, what is EDOT Cloud Forwarder?</h2>
<p>EDOT Cloud Forwarder is a serverless OpenTelemetry Collector that, on AWS, runs as a Lambda function. In AWS, it is triggered by events and processes logs and metrics from services such as VPC Flow Logs, ELB Access Logs, CloudTrail, CloudWatch Logs and CloudWatch Metrics.</p>
<p>It has the following core capabilities:</p>
<ul>
<li>Collects observability and security data from Cloud Service Providers</li>
<li>Parses data into native OpenTelemetry format</li>
<li>Forwards data over OTLP</li>
<li>Scales up and down based on traffic</li>
</ul>
<p>ECF for AWS is a pure serverless solution, no VMs, containers, or Kubernetes control planes to manage.</p>
<h2>Off the train, a more controlled scenario</h2>
<p>For a more controlled testing scenario, we used synthetic VPC Flow Log data generated to show how easy EDOT Cloud Forwarder can sustain a million events per second reliably and without data loss.</p>
<p>For the configuration, we left all EDOT Cloud Forwarder configuration/settings at their <a href="https://www.elastic.co/docs/reference/opentelemetry/edot-cloud-forwarder/aws#optional-settings">defaults</a>. In AWS, the Lambda max concurrency defaults to 5. If left to the default of 5, up to around 50k EPS could be expected. For our scenario, we bumped this to 100 to ensure we'd have plenty of headroom for our test.</p>
<p>We ran the scenario in 10 minute stages, with each stage resulting in a larger data volume. We held ingest flat during the stage to provide short-term steady state windows for us to grab metrics.</p>
<p>We experienced no errors across all stages of the scenarios, no retries outside expected behavior, no data loss across 5.4 billion ingested events.</p>
<h2>Stats for Observability Nerds</h2>
<p>Because we know you love them:</p>
<h3>Incremental Load Stages</h3>
<p>We tested EDOT Cloud Forwarder using incremental load stages, gradually increasing traffic from approximately 300,000 events per second to over 1 million events per second.</p>
<p>The graph below shows the Elasticsearch ingestion rate throughout the entire test duration. You can see the clear progression as we ramped up through six distinct stages, with each plateau representing a 10 minute stabilization period.</p>
<p>The system handled each traffic increase smoothly, culminating in sustained 1 million documents per second ingestion with no bottlenecks or data loss.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/one-million-events-per-second-with-edot-cloud-forwarder/es-ingestion-rate.png" alt="ES Ingestion Rate" /></p>
<h3>Lambda</h3>
<p>As seen in CloudWatch metrics (no manual adjustments required). No errors and no throttles were seen during the full duration of the test.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/one-million-events-per-second-with-edot-cloud-forwarder/lambda-errors-throttles.png" alt="Lambda errors and throttles" /></p>
<h4>Concurrent executions</h4>
<p>60 to 65 instances running at the same time.
<img src="https://www.elastic.co/observability-labs/assets/images/one-million-events-per-second-with-edot-cloud-forwarder/lambda-instances.png" alt="Lambda concurrent executions" /></p>
<h4>Average execution time per Lambda</h4>
<p>Each execution is taking about 5 seconds.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/one-million-events-per-second-with-edot-cloud-forwarder/lambda-execution-time.png" alt="Lambda average execution time" /></p>
<h4>Memory Usage</h4>
<p>Memory use stabilized at around 450 MB, below the default limit of 512 MB.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/one-million-events-per-second-with-edot-cloud-forwarder/lambda-memory-used.png" alt="Lambda memory usage" /></p>
<h3>Elasticsearch Indexing</h3>
<p>Elasticsearch indexed one million documents per second, with events visible in Discover within seconds and no indexing delays or bottlenecks.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/one-million-events-per-second-with-edot-cloud-forwarder/es-1m-eps.png" alt="1M EPS" /></p>
<h2>Efficient by design: Lambda at 1M EPS from S3</h2>
<p>Running ECF for AWS at 1M EPS costs about <strong>$3.87 per hour</strong>. Around 66 percent ($2.57 per hour) is data transfer (same region), 34 percent ($1.32 per hour) is Lambda compute, and less than 1 percent is S3 requests.</p>
<p>This is fully serverless with no idle cost. You only pay while events are forwarded. With S3, data arrives pre-batched in large objects, which keeps Lambda invocations low and compute costs tightly controlled. At sustained throughput, Lambda costs are comparable to EKS—but without cluster management or idle capacity.</p>
<h3>Other options: OTel Collector on EKS at 1M events per second</h3>
<p>An OTel Collector on EKS sized for 1M EPS has a baseline cost of about <strong>$3.69 per hour</strong>. Roughly $0.33 per hour of this is compute related, EC2 nodes, EKS control plane, and EBS. The rest comes from data transfer and SQS, which scale with traffic and do not change with utilization.</p>
<h4>Idle compute impact on EKS real costs</h4>
<p>Considering EKS is typically provisioned for peak load, the real cost of the compute portion is affected by idle capacity. At <strong>100 percent utilization</strong>, total cost is <strong>$3.69 per hour</strong>. At <strong>50 percent utilization</strong>, a common baseline to absorb burstiness, total cost rises to about <strong>$4.02 per hour</strong>. At <strong>30 percent utilization</strong>, it increases further to about <strong>$4.46 per hour</strong>.</p>
<h4>Pay for Work vs Pay for Capacity</h4>
<p>ECF for AWS delivers 1M EPS at a cost comparable to EKS at peak utilization, with no idle compute or capacity planning required. EKS can reach the same peak throughput, but total cost increases further as average utilization drops because compute capacity must be provisioned in advance.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/one-million-events-per-second-with-edot-cloud-forwarder/otel-collector-lambda.png" alt="Collector-lambda" /></p>
<h2>Conclusion</h2>
<p>The boring truth: to the EDOT Cloud Forwarder, a million events per second is no different from any other workload.</p>
<p>With no infrastructure to deploy, no idle cost and no manual scaling, I guess the best thing to do is to stop overthinking and start forwarding! It's like stepping onto the fastest train on the line: you just get on and you're instantly en route to your destination, effortlessly handling any distance or in this case, any volume.</p>
<p>So, we're shipping it. ECF for AWS is now Generally Available.</p>
<h2>Get Started</h2>
<ol>
<li>Deploy EDOT Cloud Forwarder via CloudFormation (below) or using the AWS Serverless Application Repository</li>
</ol>
<p><a href="https://console.aws.amazon.com/cloudformation/home?%23/stacks/new?templateURL=https%3A%2F%2Fedot-cloud-forwarder.s3.amazonaws.com%2Fv1%2Flatest%2Fcloudformation%2Fs3_logs-cloudformation.yaml"><img src="https://www.elastic.co/observability-labs/assets/images/one-million-events-per-second-with-edot-cloud-forwarder/cloudformation-launch-stack.png" alt="Launch_stack" /></a></p>
<ol start="2">
<li>Create an Observability project using an <a href="https://cloud.elastic.co/login?redirectTo=%2Fhome">Elastic Cloud</a> free trial or deploy locally with start-local if you don't already have an Elastic project or deployment.</li>
</ol>
<p>Visit EDOT Cloud Forwarder for AWS <a href="https://www.elastic.co/docs/reference/opentelemetry/edot-cloud-forwarder/aws">Documentation</a> for more details.</p>
<p>Check out these other resources on OpenTelemetry at Elastic</p>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-agent-pivot-opentelemetry">Discover how Elastic is evolving data ingestion with OpenTelemetry</a></p>
<p><a href="https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-sdk-central-configuration-opamp">Learn how OpAMP enables centralized configuration of OpenTelemetry SDKs</a></p>
]]></content:encoded>
            <category>observability-labs</category>
            <enclosure url="https://www.elastic.co/observability-labs/assets/images/one-million-events-per-second-with-edot-cloud-forwarder/ecf-for-aws.jpg" length="0" type="image/jpg"/>
        </item>
        <item>
            <title><![CDATA[Composing OpenTelemetry Reference Architectures]]></title>
            <link>https://www.elastic.co/observability-labs/blog/opentelemetry-collector-reference-architectures</link>
            <guid isPermaLink="false">opentelemetry-collector-reference-architectures</guid>
            <pubDate>Tue, 31 Mar 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[A conceptual framework for reasoning about OpenTelemetry Collection architectures — edge, processing, and resilience layers that compose into the right pipeline for your environment.]]></description>
            <content:encoded><![CDATA[<p>Most OpenTelemetry tutorials end at the same place: an application instrumented with the SDK, exporting traces to a single collector, forwarding to a backend. It works. Then production happens.</p>
<p>Traffic grows. Teams want metrics derived from traces. The backend goes down for maintenance and you lose an hour of telemetry. A compliance requirement means PII must be stripped before data leaves the cluster. Suddenly, that single collector isn't enough — and the question becomes: what should the architecture actually look like?</p>
<p>The OpenTelemetry Collector is designed to be composed. It can run in multiple deployment modes, be chained into pipelines, and scaled independently at each stage. But the documentation describes individual components, not how to think about assembling them. That thinking is what this article is about.</p>
<p>What follows is a conceptual framework for reasoning about collector architectures — not a set of rigid templates. The building blocks described here are reference points. In practice, they combine, overlap, and adapt to your constraints. A tail sampling tier might also need Kafka-backed resilience. A gateway might absorb the role of a sampling tier at low volumes. The goal is to understand the concepts well enough to compose the right architecture for your situation, not to pick a pre-built one off a shelf.</p>
<h2>Three conceptual layers</h2>
<p>It helps to think about collector architectures in three layers: <strong>edge</strong>, <strong>processing</strong>, and <strong>resilience</strong>. These aren't physical tiers that must exist as separate deployments — they're categories of concern. A single collector can address multiple layers. A complex deployment might have several components within one layer. The layers are a thinking tool, not a deployment diagram.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/opentelemetry-collector-reference-architectures/three-layers.png" alt="The three conceptual layers: Edge, Processing, and Resilience" /></p>
<h3>Edge: how telemetry enters the pipeline</h3>
<p>The edge layer is about the first hop — how telemetry gets from your applications and infrastructure into the pipeline. At this stage, the collector gathers data in two fundamentally different ways. <strong>Pull-based receivers</strong> like <code>filelog</code> and <code>hostmetrics</code> actively reach out to collect data — tailing log files on disk or scraping system-level metrics from the host. <strong>Push-based receivers</strong> like <code>otlp</code> listen for data sent to them — applications instrumented with OpenTelemetry SDKs export traces, metrics, and logs directly to the collector's OTLP endpoint. A single edge collector typically runs both: pull receivers for infrastructure telemetry the application doesn't know about, and push receivers for application telemetry the SDK produces. There are several common deployment patterns, and the right one depends on your environment and what you need to collect.</p>
<p><strong>DaemonSet Agent</strong> — One OpenTelemetry Collector per Kubernetes node, deployed as a DaemonSet. Applications export to the agent running on the same node (typically via status.hostIP:4317 using the Kubernetes Downward API). The agent also tails container log files from disk via the filelog receiver and scrapes host-level metrics via the hostmetrics receiver. This is the most common Kubernetes pattern because it handles both application and infrastructure telemetry with a single deployment, and applications only need to know about localhost.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/opentelemetry-collector-reference-architectures/daemonset-agent.png" alt="DaemonSet Agent pattern: Application with OTel SDK exporting over OTLP to a per-node DaemonSet collector" /></p>
<p><strong>Sidecar Agent</strong> — One OpenTelemetry Collector per pod, deployed as a sidecar container. Each service gets its own collector with a custom configuration. This is required on managed container platforms like AWS Fargate or Azure Container Apps where DaemonSets aren't available, and it's useful when services have different processing requirements. When running alongside a DaemonSet, the sidecar handles application telemetry while the DaemonSet independently collects node-level telemetry — applications don't send to both.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/opentelemetry-collector-reference-architectures/sidecar-agent.png" alt="Sidecar Agent pattern: Application with OTel SDK exporting over OTLP to a per-pod sidecar collector" /></p>
<p><strong>Host Agent</strong> — A standalone OpenTelemetry Collector running as a systemd service on bare-metal or VM hosts. It serves the same role as the DaemonSet agent but outside Kubernetes: collecting host metrics, tailing log files, and receiving OTLP from local applications.</p>
<p><strong>Direct SDK Export</strong> — Applications export directly to the next stage (gateway or backend) with no local collector. This is the simplest option but only works when you don't need infrastructure collection. For log collection, the recommended pattern is still to write to stdout and use a collector with the <code>filelog</code> receiver — even if the SDK is exporting traces and metrics directly.</p>
<p>These patterns aren't mutually exclusive. A Kubernetes cluster might run DaemonSet agents for infrastructure collection alongside sidecars for services that need custom processing. A VM environment might use host agents for some services and direct SDK export for others. The edge layer is about matching the collection pattern to the workload, not picking one pattern for everything.</p>
<h3>Processing: central policy, sampling, and transformation</h3>
<p>Not every architecture needs a processing layer. If your edge collectors can export directly to your backend and you don't need centralized policy, you can skip it to favour simplicity. But several scenarios push you toward central processing — and the way you address them can range from a single gateway to a multi-stage pipeline.</p>
<p><strong>Centralized policy (Gateway)</strong> — A pool of OpenTelemetry Collectors that sits between edge collectors and the backend. This is where you enforce consistent filtering, transformation, and PII redaction across all services. It's also where you manage backend credentials — edge collectors export to the gateway over OTLP, and only the gateway holds the API keys. Credential isolation is often the primary reason teams add a gateway.</p>
<p>Replica count scales with data volume. At low volumes (under 1K events/sec), 2 replicas co-located with workloads is sufficient. At medium volumes, 3–5 replicas on a dedicated node pool. At high volumes, 5–20+ replicas, potentially in a separate cluster. This is a general rule of thumb, and you should adapt it to your specific needs as loads might vary significantly between payload types.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/opentelemetry-collector-reference-architectures/gateway-pool.png" alt="Gateway pattern: Load Balancer distributing traffic to a Gateway Pool of OTel Collectors" /></p>
<p><strong>Tail-based sampling</strong> — Sampling decisions that consider the complete trace (e.g., &quot;keep all traces with errors, sample 10% of successful traces&quot;) require that all spans of a trace reach the same collector instance. This is achieved with the <code>loadbalancingexporter</code> using <code>routing_key: traceID</code>, which consistently routes spans from the same trace to the same downstream collector.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/opentelemetry-collector-reference-architectures/tail-sampling.png" alt="Tail sampling pattern: LB Exporter routing to Sampling Collectors with tail_sampling" /></p>
<p>There's a critical subtlety here: if you're deriving span metrics (RED metrics) from traces using the <code>spanmetrics</code> connector, the derivation must happen <strong>before</strong> sampling. Otherwise, your metrics only reflect the sampled subset, not the true traffic. The correct pattern is a two-step pipeline within the sampling stage:</p>
<ol>
<li>Receive traces, derive spanmetrics from 100% of traffic, forward via a <code>forward</code> connector.</li>
<li>Apply <code>tail_sampling</code> to the forwarded traces, export only kept traces.</li>
<li>A separate metrics pipeline exports the derived RED metrics.</li>
</ol>
<p>This ensures accurate metrics regardless of your sampling rate.</p>
<p><strong>The key point about processing</strong> is that these capabilities — gateway policy, tail sampling, span metrics derivation — are not separate products or fixed modules. They're configurations of the same OpenTelemetry Collector. At low volumes, a single gateway deployment might handle policy enforcement, sampling, and metrics derivation all at once. At high volumes, you might split them into dedicated stages for independent scaling. The architecture adapts to your scale, not the other way around.</p>
<h3>Resilience: what happens when the backend is down</h3>
<p>The resilience layer determines how much data you're willing to lose during backend outages or collector restarts. This isn't a separate tier you bolt on — it's a property you apply to any stage of the pipeline.</p>
<p><strong>In-Memory Queues</strong> — The default. The collector's <code>sending_queue</code> retries failed exports with exponential backoff. If the collector process crashes or restarts, queued data is lost. This is acceptable for development and for workloads where some data loss during incidents is tolerable.</p>
<p><strong>Persistent Queues (WAL)</strong> — The <code>file_storage</code> extension writes queued data to disk before export. If the collector crashes, it resumes from where it left off after restart. In Kubernetes, this requires a PersistentVolumeClaim. This is the right choice for most production workloads — it survives collector restarts and brief backend outages without the operational complexity of an external message bus.</p>
<p><strong>Kafka Buffer</strong> — An external Kafka cluster sits between collectors and the backend. Producer collectors write to Kafka topics; consumer collectors read from Kafka and export to the backend. This provides the strongest durability guarantee — Kafka can buffer hours of telemetry during extended outages and enables replay. But it adds significant operational complexity.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/opentelemetry-collector-reference-architectures/kafka-buffer.png" alt="Kafka buffer pattern: Collector Pool producing to Kafka, consumed by another Collector Pool" /></p>
<p>The important thing to understand is that resilience is orthogonal to the other layers. You can add persistent queues to an edge agent, a gateway, or a sampling tier. You can put Kafka in front of a gateway, in front of a sampling tier, or in front of the backend. A tail sampling deployment that needs to survive extended outages might use Kafka-backed ingestion — combining what might look like two separate &quot;modules&quot; into a single stage. The building blocks compose freely based on what you need to protect against.</p>
<h2>Where to start with your architecture</h2>
<p>The Agent + Gateway two-tier pattern is the de facto production standard, used by the vast majority of organizations running OpenTelemetry at scale. DaemonSet agents on every node handle local collection — pulling infrastructure telemetry via <code>filelog</code> and <code>hostmetrics</code>, receiving application telemetry via OTLP — while a centralized gateway pool enforces policy, manages credentials, and exports to the backend. Persistent queues (WAL) on the gateway protect against backend outages without external dependencies.</p>
<p><img src="https://www.elastic.co/observability-labs/assets/images/opentelemetry-collector-reference-architectures/where-to-start.png" alt="A Kubernetes architecture with DaemonSet agents, a processing tier with tail sampling and gateway pool, exporting over OTLP to an observability backend" /></p>
<p>Every other configuration either simplifies this pattern or extends it. Smaller environments might drop the gateway and export directly from agents. Larger ones might add a tail sampling tier with traceID-based load balancing, a Kafka buffer for extended resilience, or span metrics derivation before sampling. The building blocks described in the previous sections — edge, processing, resilience — are the modules you add or remove from this foundation.</p>
<p>The key is to start with the two-tier pattern and evolve incrementally:</p>
<ul>
<li>Need credential isolation or centralized PII redaction? You already have the gateway.</li>
<li>Need tail-based sampling? Add a load-balancing exporter and a sampling tier between agents and gateway.</li>
<li>Need hours of buffer during extended outages? Insert Kafka between agents and the processing tier.</li>
<li>Running on Fargate or Azure Container Apps? Swap DaemonSet agents for sidecars — the rest of the pipeline stays the same.</li>
</ul>
<p>Start here. Add modules as your needs grow. The architecture adapts to your scale, not the other way around.</p>
<h2>Decision points that shape where you need to take your architecture</h2>
<p>When designing a collector architecture, these are the questions that determine which patterns you need:</p>
<table>
<thead>
<tr>
<th>Question</th>
<th>Impact</th>
</tr>
</thead>
<tbody>
<tr>
<td>Do I need infrastructure telemetry (host metrics, disk logs)?</td>
<td>Determines whether you need a local collector or can use direct SDK export</td>
</tr>
<tr>
<td>Am I on a managed container platform (Fargate, ACA)?</td>
<td>Forces sidecar pattern instead of DaemonSet</td>
</tr>
<tr>
<td>Do I need centralized filtering, PII redaction, or credential isolation?</td>
<td>Adds a gateway stage</td>
</tr>
<tr>
<td>Do I need tail-based sampling?</td>
<td>Adds a sampling stage with load-balancing exporter and traceID routing</td>
</tr>
<tr>
<td>Do I want span-derived metrics (RED metrics)?</td>
<td>Requires spanmetrics before sampling in a two-step pipeline</td>
</tr>
<tr>
<td>How much data loss is acceptable during outages?</td>
<td>Determines in-memory queues vs. persistent queues vs. Kafka — applied to whichever stage needs protection</td>
</tr>
<tr>
<td>What is my expected data volume?</td>
<td>Determines whether capabilities can be co-located in a single deployment or need dedicated stages</td>
</tr>
</tbody>
</table>
<p>The answers to these questions don't map to a single &quot;correct&quot; architecture. They constrain the design space, and within those constraints, you make trade-offs between simplicity and capability.</p>
<h2>Exploring these patterns interactively</h2>
<p>If you'd rather explore how these building blocks compose than assemble them by hand, <a href="https://mlunadia.github.io/otel-blueprints/">OpenTelemetry Blueprints</a> is an open-source tool that generates reference architectures from your requirements.</p>
<p>Toggle your environment, signals, volume, resilience, and processing needs — and get a composed diagram with animated data flow, interactive tooltips, and reference collector configurations you can open directly in <a href="https://www.otelbin.io">OTelBin</a> for validation.</p>
<p><a href="https://mlunadia.github.io/otel-blueprints/"><img src="https://www.elastic.co/observability-labs/assets/images/opentelemetry-collector-reference-architectures/architecture.png" alt="Screenshot of a composed architecture diagram showing a Kubernetes cluster with DaemonSet agent, processing tier, and observability backend" /></a></p>
<p>The generated configurations export via OTLP, so they work with any OTLP-compatible backend — including <a href="https://www.elastic.co/docs/reference/opentelemetry/motlp">Elastic Observability</a>, which natively accepts and stores OTLP traces, metrics, and logs.</p>
<p>The architectures Blueprints generates are reference compositions — starting points for understanding how the building blocks fit together, not turnkey deployments. Every architecture should be adapted to your organisation's scale, security, networking, and compliance requirements. The patterns might combine or overlap differently in your environment than in anyone else's, and that's the point.</p>
<h2>Get started</h2>
<p>The architectures described here export over OTLP, so they work with any compatible backend. If you don't have one yet, the fastest way to see your telemetry flowing end-to-end is with Elastic Observability — it natively ingests OTLP traces, metrics, and logs with no additional configuration.</p>
<ol>
<li><a href="https://cloud.elastic.co/serverless-registration?onboarding_token=observability">Start a free trial</a> on Elastic Cloud Serverless — no credit card required.</li>
<li>Point your collector's OTLP exporter at the managed OTLP endpoint.</li>
<li>Explore your traces, metrics, and logs in Kibana within minutes.</li>
</ol>
<p>Check out these resources to go further:</p>
<ul>
<li><a href="https://www.elastic.co/docs/reference/opentelemetry/motlp">Elastic's managed OTLP endpoint documentation</a></li>
<li><a href="https://www.elastic.co/docs/reference/opentelemetry/edot-collector">EDOT Collector — Elastic's distribution of the OpenTelemetry Collector</a></li>
<li><a href="https://mlunadia.github.io/otel-blueprints/">OpenTelemetry Blueprints — generate reference architectures interactively</a></li>
</ul>
]]></content:encoded>
            <category>observability-labs</category>
            <enclosure url="https://www.elastic.co/observability-labs/assets/images/opentelemetry-collector-reference-architectures/opentelemetry-collector-reference-architectures.jpg" length="0" type="image/jpg"/>
        </item>
    </channel>
</rss>