OpenTelemetry has won the hearts of the industry. Adoption is accelerating: the CNCF's 2024 Observability survey found OTel to be the fastest-growing project in the foundation's history, with the OTel Collector registering hundreds of millions of downloads. The proposition is compelling: write instrumentation once, ship it anywhere, avoid lock-in.

But here is what every platform team discovers once they cross into production: the collector sprawl problem. Hundreds of collector instances deployed across regions, Kubernetes namespaces, and bare-metal hosts. Configuration drift creeping in. An upgrade that has to be co-ordinated across a fleet of independent processes. A security patch that someone has to manually roll out to each one. And zero visibility into which collectors are running, healthy, or stuck.

This is the gap between "deploying OpenTelemetry" and "operating OpenTelemetry at scale." With Elastic 9.3, Elastic Agent closes that gap entirely. The Elastic Agent is now built on Elastic's Distribution of the OpenTelemetry Collector (EDOT) and, when managed by Fleet, gives platform teams a single control plane for configuring, updating, and monitoring every OTel collector in their estate — all while remaining compatible with the Beats-based integrations they already rely on.

The Collector Sprawl Problem and Why It Matters

OpenTelemetry's success has created a quiet operational debt for many organisations. Individual teams adopt the collector for their services: logs here, metrics there, a custom pipeline for the new microservice. Without a centralised management layer, each of these collectors becomes an independent snowflake: its own config file, its own upgrade cycle, its own failure domain.

The consequences are predictable. Configuration drift means collectors running different versions of the same pipeline, producing subtly incompatible data. Compliance teams ask "show me all the places data is collected and where it goes", and the honest answer is a spreadsheet that's already out of date.

This isn't a niche problem. A Gartner analysis of enterprise observability programmes consistently identifies operational overhead as the top barrier to expanding OTel adoption beyond initial pilots. The technology works. The tooling to manage it at scale is what's been missing.

How Elastic Agent Became an OTel Collector

To understand the significance of this, it helps to understand what Elastic Agent used to be, and what it is now.

Elastic Agent acts as a supervisor process: Before version 9.3, it managed a collection of separate Beats sub-processes (Filebeat, Metricbeat, Winlogbeat and so on), each running its own input/output lifecycle, each consuming its own memory footprint. The agent coordinated them, but the fundamental model was a collection of discrete daemons running under a parent.

With 9.3, that model has been replaced. Elastic Agent is now itself an instance of the EDOT Collector: Elastic's hardened, production-supported distribution of the upstream OTel Collector. The architectural shift has three important consequences.

First, the process model simplifies dramatically. Instead of a supervisor managing multiple sub-process lifecycles, there is a single EDOT Collector process. This means a smaller memory footprint, fewer things that can fail independently, and fewer processes to observe for health and performance.

Second, Beats functionality is preserved, not discarded. Rather than forcing a breaking migration, Elastic has introduced Beats Receivers: beat inputs and processors re-packaged as native OTel receiver components. A Filestream input is enabled by a filebeatreceiver. The same Filebeat configuration YAML you write today is automatically translated into the corresponding EDOT receiver configuration at runtime. Existing integrations, dashboards, and ingest pipelines continue to work without modification.

Third, the agent is now a first-class participant in the OTel ecosystem. It speaks OTLP natively, it runs standard OTel receivers, and it can be configured to sit alongside any other OTel-compatible tool in a modern observability pipeline.

Central Management with Fleet: Configuration, Lifecycle, and Visibility

The architectural shift above would be valuable on its own. But it becomes transformative when combined with Elastic Fleet, the centralised management plane for Elastic Agents.

Fleet gives platform and SRE teams a single console from which to manage every Elastic Agent (and by extension, every EDOT Collector instance) in their estate. The capabilities break into three categories: configuration management, lifecycle management, and fleet-wide observability.

Configuration management at scale

With Fleet, you define an Agent Policy — a declarative description of what a collector should do. What data should it collect? Via which receivers? Where should it export? The policy is authored once in Fleet's UI (or via its API), and pushed automatically to every agent enrolled in that policy. Change the policy, and every affected collector receives the update. No SSH. No Ansible playbook to maintain. No configuration drift.

Fleet pushes policies to enrolled agents across any environment. Agents send heartbeat and health data back, giving a live inventory of every collector in the estate.

Lifecycle management: upgrades, enrolment, and remediation

Perhaps the most operationally significant benefit of Fleet management is lifecycle control. With Fleet, upgrading a collector is a policy action: select the target version, select the scope (all agents, a specific policy group, a canary subset), and click. Fleet orchestrates the rolling upgrade, tracking status per agent and surfacing failures immediately.

This changes the security calculus fundamentally. When a vulnerability is disclosed in the OTel Collector binary, patching is a Fleet operation measured in minutes, not a change-management ceremony measured in days across SSH sessions to individual hosts.

Fleet also handles enrolment and de-enrolment. New hosts added to your infrastructure can be auto-enrolled into the appropriate policy based on tags or deployment tooling. Agents on decommissioned hosts can be removed from Fleet's inventory, ensuring your observability map reflects your actual infrastructure.

Fleet-wide observability of your collectors

Every Fleet-managed Elastic Agent ships monitoring telemetry about itself: CPU and memory consumption, event throughput, error rates, pipeline latency. This data flows into Elastic and is surfaced in the Fleet UI, giving you a live dashboard of every collector in your estate, not just the ones you happen to be watching.

For the first time, "how healthy is my observability pipeline?" becomes a question with a real-time, fleet-wide answer. You can identify agents that have stopped sending data, agents consuming unexpectedly high resources, and agents that have fallen behind on queue processing — before those problems surface as gaps in your monitoring data.

In the near future this capability will be offered to non-Fleet managed agents (aka standalone) and/or 3rd party OTel collectors provided by other vendors. These collectors can be configured via some other means but be monitored in Fleet - from both resource consumption and/or component pipeline health.

The Hybrid Agent: Beats Data and OTel Data, Simultaneously

One of the most practically significant capabilities introduced in 9.3 is what Elastic calls the Hybrid Agent: an Elastic Agent that can run both Beats-based receivers and native OTel receivers in the same pipeline, at the same time. This does not change anything for existing installations.

This matters enormously for real-world adoption. Most organisations arriving at OTel in 2025 and 2026 are not starting from a blank slate. They have years of investment in Beats-based integrations: Filebeat-powered log collection, Metricbeat-powered host metrics, bespoke ingest pipelines in Elasticsearch that normalise and enrich that data into ECS (Elastic Common Schema) format. The business value locked in those integrations (the dashboards, the alerts, the correlation logic) is not something they can afford to throw away in order to "go OTel."

The Hybrid Agent solves this by making the two worlds coexist. For example, in a single agent policy you can simultaneously configure:

• A filebeatreceiver collecting application logs in ECS format, routed through your existing ingest pipeline to its existing data stream
• A native OTel filelog receiver collecting OTel-native telemetry from your new services instrumented with the OTel SDK, stored in OTel-native data streams without touching ingest pipelines
• An OTel hostmetrics receiver collecting system metrics in semantic convention format alongside your existing Metricbeat-derived system metrics

The two lanes are independent. Beats-receiver data travels through ingest pipelines and lands in ECS-formatted data streams, exactly as it always has. Native OTel data follows OTel semantic conventions and is stored directly in OTel-native data streams, bypassing ingest pipelines. Your existing dashboards and alerts continue to work. Your new OTel-native workloads get the full OTel experience. The same agent, the same Fleet policy, the same management console.

This co-existence is the practical answer to the question every platform team eventually faces: "We want to adopt OTel properly but we can't break what we already have." The Hybrid Agent lets you migrate incrementally, service by service, on your timeline.

The Integration Catalogue: Turning Configuration into a One-Click Operation

Configuration management at scale is only as good as the configurations themselves. Elastic's integration catalogue — over 500 packages covering everything from NGINX and PostgreSQL to AWS CloudTrail and Kubernetes — extends naturally to the Hybrid Agent model.

From 9.3 onwards, the catalogue includes OTel integration packages alongside the existing Beats-based ones. Each OTel package contains two components:

• An Input package: the configuration for the corresponding OTel receiver (receivers, processors, pipeline wiring), ready to be applied to a Hybrid Agent policy
• A Content package: the assets associated with the application: pre-built dashboards, alerts, index templates, and saved queries, all calibrated for OTel semantic convention data

When an operator adds an OTel integration to an Agent Policy in Fleet, the receiver configuration is pushed to all enrolled agents. When those agents start ingesting data and it arrives in Elasticsearch, the content package assets are automatically installed based on metadata in the data received. The dashboard is ready before you've had time to wonder where it is.

The same policy can hold both OTel integrations and legacy Beats integrations. A real-world agent policy might simultaneously collect system metrics via the OTel hostmetrics receiver, application logs via filebeat receiver, and APM data via OTLP — all from one policy, all managed from Fleet, all visible in a unified Kibana experience.

A technical walk through of how this is done for NGINX data collection can be found here for reference. Currently management of Elastic Agents is done via existing Fleet protocols, however in the near future this will move over to OPAMP so that Fleet will be able to provide management to 3rd party OTel collectors as well.

For organisations on platforms not yet in Elastic's OS support matrix, 3rd-party OTel Collectors (such as Red Hat's OpenShift-native collector) can send data to Elastic using the OTLP exporter and be observed alongside all other collectors in their fleet.

What This Means in Practice: A Migration Story

Consider a mid-sized platform team operating 200 Linux hosts across three regions, currently running Elastic Agent 8.x with a mix of Filebeat and Metricbeat integrations. Their new services are being instrumented with the OTel SDK and they want to standardise on OTel going forward without disrupting the monitoring coverage they already have.

With a Fleet-managed upgrade to 9.3, their existing agents become Hybrid Agents automatically. Their Filebeat and Metricbeat configurations are internally translated to Beats receiver configurations and continue to run unmodified. Their existing dashboards still populate. Their ingest pipelines still fire. Nothing breaks.

They then add OTel integration packages to their Fleet policies for each new service. The OTel-instrumented microservices start sending OTLP data, received by native OTel receivers in the same agents. OTel-native dashboards appear automatically in Kibana. They now have both data universes in one place, managed from one console, visible in one interface.

Over the following quarters, as Beats-based integrations for their remaining services are superseded by OTel equivalents in the catalogue, they migrate them one by one, updating the Agent Policy in Fleet and watching the transition happen across all 200 hosts simultaneously, without touching a single one directly.

Looking Forward

Elastic has made a clear architectural bet: OpenTelemetry is the future of observability data collection, and the right response to that future is not to build a parallel OTel tool alongside the existing stack — it is to evolve the existing stack into OTel. The Hybrid Agent and EDOT Collector are the result of that bet.

Fleet central management is the operational layer that makes that bet practical at scale. OpenTelemetry gives you standardised, vendor-neutral instrumentation. Fleet gives you the operational control plane to manage those collectors like the production infrastructure they are, not like artisanal YAML files scattered across your estate.

The collector sprawl problem is solvable. The answer is a managed, policy-driven, centrally observable fleet of EDOT Collectors, and in Elastic 9.3, that answer is production-ready today.

In this blog, we examine a simple way to integrate Elastic Agent with Confluent Cloud's Kafka offering to reduce the operational burden of ingesting business-critical data.

Benefits of Elastic Agent and Confluent Cloud

When combined, Elastic Agent and Confluent Cloud's updated Elasticsearch Sink connector provide a myriad of advantages for organizations of all sizes. This combined solution offers flexibility in handling any type of data ingest workload in an efficient and resilient manner.

Fully Managed

When combined, Elastic Cloud Serverless and Confluent Cloud provide users with a fully managed service. This makes it effortless to deploy and ingest nearly unlimited data volumes without having to worry about nodes, clusters, or scaling.

Full Elastic Integrations Support

Sending data through Kafka is fully supported with any of the 300+ Elastic Integrations. In this blog post, we outline how to set up the connection between the two platforms. This ensures you can benefit from our investments in built-in alerts, SLOs, AI Assistants, and more.

Decoupled Architecture

Kafka acts as a resilient buffer between data sources (such as Elastic Agent and Logstash) and Elasticsearch, decoupling data producers from consumers. This can significantly reduce total cost of ownership by enabling you to size your Elasticsearch cluster based on typical data ingest volume, not maximum ingest volume. It also ensures system resilience during spikes in data volume.

Ultimate control over your data

With our new Output per Integration capability, customers can now send different data to different destinations using the same agent. Customers can easily send security logs directly to Confluent Cloud/Kafka, which can provide delivery guarantees, while sending less critical application logs and system metrics directly to Elasticsearch.

Deploying the reference architecture

In the following sections, we will walk you through one of the ways Confluent Kafka can be integrated with Elastic Agent and Elasticsearch using Confluent Cloud's Elasticsearch Sink Connector. As with any streaming and data collection technology, there are many ways a pipeline can be configured depending on the particular use case. This blog post will focus on a simple architecture that can be used as a starting point for more complex deployments.

Some of the highlights of this architecture are:

• Dynamic Kafka topic selection at Elastic Agents
• Elasticsearch Sink Connectors for fully managed transfer from Confluent Kafka to Elasticsearch
• Processing data leveraging Elastic's 300+ Integrations

Prerequisites

Before getting started ensure you have a Kafka cluster deployed in Confluent Cloud, an Elasticsearch cluster or project deployed in Elastic Cloud, and an installed and enrolled Elastic Agent.

Configure Confluent Cloud Kafka Cluster for Elastic Agent

Navigate to the Kafka cluster in Confluent Cloud, and select Cluster Settings. Locate and note the Bootstrap Server address, we will need this value later when we create the Kafka Output in Fleet.

Navigate to Topics in the left-hand navigation menu and create two topics:

1. A topic named logs
2. A topic named metrics

Next, navigate to API Keys in the left-hand navigation menu:

1. Click + Add API Key
2. Select the Service Account API key type
3. Provide a meaningful name for this API Key
4. Grant the key write permission to the metrics and logs topics
5. Create the key

Note the provided Key and the Secret, we will need it later when we configure the Kafka Output in Fleet.

Configure Elasticsearch and Elastic Agent

In this section, we will configure the Elastic Agent to send data to Confluent Cloud's Kafka cluster and we will configure Elasticsearch so it can receive data from the Confluent Cloud Elasticsearch Sink Connector.

Configure Elastic Agent to send data to Confluent Cloud

Elastic Fleet simplifies sending data to Kafka and Confluent Cloud. With Elastic Agent, a Kafka "output" can be easily attached to all data coming from an agent or it can be applied only to data coming from a specific data source.

Find Fleet in the left-hand navigation, click the Settings tab. On the Settings tab, find the Outputs section and click Add Output.

Perform the following steps to configure the new Kafka output:

1. Provide a Name for the output
2. Set the Type to Kafka
3. Populate the Hosts field with the Bootstrap Server address we noted earlier .
4. Under Authentication, populate the Username with the API Key and the Password with the Secret we noted earlier
5. Under Topics, select Dynamic Topic and set Topic from field to data_stream.type
6. Click Save and apply settings

Next, we will navigate to the Agent Policies tab in Fleet and click to edit the Agent Policy that we want to attach the Kafka output to. With the Agent Policy open, click the Settings tab and change Output for integrations and Output for agent monitoring to the Kafka output we just created.

Selecting an Output per Elastic Integration: To set the Kafka output to be used for specific data sources, see the integration-level outputs documentation.

A note about Topic Selection: The data_stream.type field is a reserved field which Elastic Agent automatically sets to logs if the data we're sending is a log and metrics if the data we're sending is a metric. Enabling Dynamic Topic selection using data_stream.type, will cause Elastic Agent to automatically route metrics to a metrics topic and logs to a logs topic. For information on topic selection, see the Kafka Output's Topics settings documentation.

Configuring a publishing endpoint in Elasticsearch

Next, we will set up two publishing endpoints (data streams) for the Confluent Cloud Sink Connector to use when publishing documents to Elasticsearch:

1. We will create a data stream logs-kafka.reroute-default for handling logs
2. We will create a data stream metrics-kafka.reroute-default for handling metrics

If we were to leave the data in those data streams as-is, the data would be available but we would find the data is unparsed and lacking vital enrichment. So we will also create two index templates and two ingest pipelines to make sure the data is processed by our Elastic Integrations.

Creating the Elasticsearch Index Templates and Ingest Pipelines

The following steps use Dev Tools in Kibana, but all of these steps can be completed via the REST API or using the relevant user interfaces in Stack Management.

First, we will create the Index Template and Ingest Pipeline for handling logs:

PUT _index_template/logs-kafka.reroute
{
  "template": {
    "settings": {
      "index.default_pipeline": "logs-kafka.reroute"
    }
  },
  "index_patterns": [
    "logs-kafka.reroute-default"
  ],
  "data_stream": {}
}

PUT _ingest/pipeline/logs-kafka.reroute
{
  "processors": [
    {
      "reroute": {
        "dataset": [
          "{{data_stream.dataset}}"
        ],
        "namespace": [
          "{{data_stream.namespace}}"
        ]
      }
    }
  ]
}

Next, we will create the Index Template and Ingest Pipeline for handling metrics:

PUT _index_template/metrics-kafka.reroute
{
  "template": {
    "settings": {
      "index.default_pipeline": "metrics-kafka.reroute"
    }
  },
  "index_patterns": [
    "metrics-kafka.reroute-default"
  ],
  "data_stream": {}
}

PUT _ingest/pipeline/metrics-kafka.reroute
{
  "processors": [
    {
      "reroute": {
        "dataset": [
          "{{data_stream.dataset}}"
        ],
        "namespace": [
          "{{data_stream.namespace}}"
        ]
      }
    }
  ]
}

A note about rerouting: For a practical example of how this works, a document related to a Linux Network Metric would be first land in metrics-kafka.reroute-default and this Ingest Pipeline would inspect the document and find data_stream.dataset set to system.network and data_stream.namespace set to default. It would use these values to reroute the document from metrics-kafka.reroute-default to metrics-system.network-default where it would be processed by the system integration.

Configure the Confluent Cloud Elasticsearch Sink Connector

Now it's time to configure the Confluent Cloud Elasticsearch Sink Connector. We will perform the following steps twice and create two separate connectors, one connector for logs and one connector for metrics. Where the required settings differ, we will highlight the correct values.

Navigate to your Kafka cluster in Confluent Cloud and select Connectors from the left-hand navigation menu. On the Connectors page, select Elasticsearch Service Sink from a catalog of connectors available.

Confluent Cloud presents a simplified workflow for the user to configure a connector. Here we will walk through each step of the process:

Step 1: Topic Selection

First, we will select the topic that the connector will consume data from based on which connector we are deploying:

• When deploying the Elasticsearch Sink Connector for logs, select the logs topic.
• When deploying the Elasticsearch Sink Connector for metrics, select the metrics topic.

Step 2: Kafka Credentials

Choose KAFKA_API_KEY as the cluster authentication mode. Provide the API Key and Secret noted earlier when we gather required Confluent Cloud Cluster information.

Step 3: Authentication

Provide the Elasticsearch Endpoint address of our Elasticsearch cluster as the Connection URI. The Connection user and Connection password are the authentication information for the account in Elasticsearch that will be used by the Elasticsearch Sink Connector to write data to Elasticsearch.

Step 4: Configuration

In this step we will keep the Input Kafka record value format set to JSON. Next, expand Advanced Configuration.

1. We will set Data Stream Dataset to kafka.reroute
2. We will set Data Stream Typebased on the connector we are deploying:
   • When deploying the Elasticsearch Sink Connector for logs, we will set Data Stream Type to logs
   • When deploying the Elasticsearch Sink Connector for metrics, we will set Data Stream Type to metrics
3. The correct values for other settings will depend on the specific environment.

Step 5: Sizing

In this step, notice that Confluent Cloud provides a recommended minimum number of tasks for our deployment. Following the recommendation here is a good starting place for most deployments.

Step 6: Review and Launch

Review the Connector configuration and Connector pricing sections and if everything looks good, it's time to click continue and launch the connector! The connector may report as provisioning but will soon start consuming data from the Kafka topic and writing it to the Elasticsearch cluster.

You can now navigate to Discover in Kibana and find your logs flowing into Elasticsearch! Also check out the real time metrics that Confluent Cloud provides for your new Elasticsearch Sink Connector deployments.

If you have only deployed the first logs sink connector, you can now repeat the steps above to deploy the second metrics sink connector.

Enjoy your fully managed data ingest architecture

If you followed the steps above, congratulations. You have successfully:

1. Configured Elastic Agent to send logs and metrics to dedicated topics in Kafka
2. Created publishing endpoints (data streams) in Elasticsearch dedicated to handling data from the Elasticsearch Sink Connector
3. Configured managed Elasticsearch Sink connectors to consume data from multiple topics and publish that data to Elasticsearch

Next you should enable additional integrations, deploy more Elastic Agents, explore your data in Kibana, and enjoy the benefits of a fully managed data ingest architecture with Elastic Serverless and Confluent Cloud!

Elastic has fully embraced OpenTelemetry as the backbone of its data ingestion strategy, aligning with the open-source community and contributing to make it the best data collection platform for a broad user base. This move benefits users by providing enhanced flexibility, efficiency, and control over telemetry data.

Why OpenTelemetry?

OpenTelemetry provides a powerful set of capabilities that make it a compelling choice for open-source-focused users. Elastic is re-architecting its data ingest tools around OpenTelemetry to offer users vendor-agnostic flexibility, performance optimization through OTel's efficient data model for correlating telemetry, and enhanced flexibility and control over data pipelines. This move brings the benefits of open-source telemetry to Elastic users.

Elastic engineers are active contributors to the Otel project in several areas of the project. Demonstrating its commitment to open source, Elastic continues to make significant contributions to OpenTelemetry.

OpenTelemetry as the Core of Elastic's Data Ingestion

Elastic is transforming its data ingestion strategy by basing all ingestion mechanisms on the OpenTelemetry components. Elastic currently supports the following OTel based ingest architecture, which support OTel SDKs and Collectors from OTel or Elastic's Distribution of OpenTelemetry (EDOT).

This marks a fundamental shift, ensuring a more standardized and scalable telemetry pipeline. All the existing Elastic ingest components will become OTel based.

Let's discuss how each component of Elastic's data ingestion platform will be based on an OpenTelemetry collector whilst still providing the same functionality to the user.

Beats

Elastic's traditional data shippers will be re-architected as OpenTelemetry Collectors, aligning with OTel's extensibility model. Current Beat architecture is essentially made up of a few stages in its pipeline, as shown in the diagram below. It consists of an Input, Processors for enrichments, Queuing of events and Output for batching and writing the data to a specific output.

Beatreceiver Concept

To ensure a smooth transition without major disruptions, a "beatsreceiver" concept is being implemented. These beatreceivers (like filebeatreceiver or metricbeatreceiver) act as dedicated Beat inputs integrated into the OpenTelemetry Collector as native receivers. They support all existing inputs and processors, guaranteeing that the final architecture accepts the user's current configuration and delivers the same functionality as today's Beats, all without introducing any breaking changes.

An OTel based Beats architecture will see the Input phase embedded as an OTel receiver (eg.  filebeatreceiver to represent the functionality of filebeat). This receiver would only be available as part of Elastic's distribution of OTel in support of our current user base and not a functionality that would be available upstream.

All the remaining components of the pipeline will be based on OTel. The new Beat will accept the same filebeat configuration (as an example) and will transform it to an OTel based configuration in order to avoid any deployment disruption. It should be noted that in this architecture the Beats will continue to only support ECS formatted data. In order to keep the Beat functionality inline with what exists today, the Elasticsearch exporter (as an example) will output ECS formatted data only.

The following diagram illustrates the beatreceiver concept by showing how a basic filebeat configuration is automatically translated into an OpenTelemetry-based configuration. This new configuration retains the original inputs and processors but leverages the native OpenTelemetry pipeline and exporter to achieve the same overall filebeat functionality. Existing filebeat configurations will be automatically converted, eliminating the need for manual adjustments or introducing breaking changes.

Elastic Agent

Elastic Agent is a unified agent for data collection, security, and observability. It can also be deployed in an OpenTelemetry only mode, enabling native OTel workflows. Elastic Agent is a supervisor that manages many other Beats as sub-processes in order to provide a more comprehensive data collection tool. It is capable of translating Agent Policy received from Fleet into configuration acceptable by the various sub-processes.

Expanding on the Beat receiver concept described above, the Elastic Agent, which currently can be deployed as an OTel collector (see blog), will be also modified to a much simpler OTel based architecture based on these receivers. As shown below, this architecture will streamline the components within the Elastic Agent and remove duplicated functionality such as queuing and output. Whilst supporting the current functionality, these changes will reduce the agent footprint and also present a reduction in number of connections opened to pipeline elements egress of the agent (such as Elasticsearch clusters, Logstash or Kafka brokers).

By moving to an OTel based architecture Elastic Agent is now able to operate as a truly hybrid Elastic Agent which provides not only the Beat functionality but also allows our users to create OTel native pipelines and take advantage of plethora of functionality available as part of the open source project.

Elastic's commitment to OpenTelemetry will deepen through increased contributions, resulting in OpenTelemetry receivers gradually superseding Beats receiver features. This evolution will eventually reduce the need for a distinct Beats receiver within the Elastic Agent architecture. The envisioned architecture will empower the Elastic Agent to transmit data in OTLP format as well, granting users the flexibility to select any OTLP-compatible backend, thereby upholding the principle of vendor neutrality.

Fleet & Integrations: Managing OpenTelemetry at Scale

Elastic's centralized management system will support OpenTelemetry-based configurations, making large-scale deployments easier to manage. Managing thousands of telemetry agents at scale presents a significant challenge. Elastic's Fleet & Integrations simplify this process by providing robust lifecycle management for these new OpenTelemetry-based Elastic agents.

Key Capabilities Offered:

• Scalability: Manage up to 100K+ agents across distributed environments.

• Automated Upgrades: Staged rollouts and automatic upgrades ensure minimal downtime.

• Monitoring & Diagnostics: Real-time status updates, failure detection, and diagnostic downloads improve system reliability.

• Policy-Based Configuration Management: Enables centralized control over agent configurations, improving consistency across deployments.

• Pre-Built Integrations: Elastic offers a catalog of 470+ pre-built integrations, allowing users to ingest data seamlessly from various sources. These will also include OTel based packages making configuration much more efficient across a large deployment.

The goal is for Fleet to also provide monitoring capabilities for native OTel collectors as well in a vendor agnostic fashion.

Conclusion

Elastic's adoption of OpenTelemetry marks a significant milestone in the evolution of open-source observability. By standardizing on OpenTelemetry, Elastic is ensuring that its data ingestion strategy remains open, scalable, and future-proof.

For open-source users, this shift means:

• Greater interoperability across observability tools.

• Enhanced flexibility in choosing telemetry backends.

• A stronger commitment to community-driven observability standards.

• Existing Beats and Elastic Agent users can seamlessly adopt OpenTelemetry without rearchitecting their pipelines.

• OpenTelemetry users can integrate with Elastic's observability stack without additional complexity.

Stay tuned for more updates as Elastic continues to expand its OpenTelemetry-based data collection capabilities! In the mean time here are some other references:

• Elastic Distributions of OpenTelemetry (EDOT) Now GA

• Dynamic workload discovery on Kubernetes now supported with EDOT Collector

• Introducing the OTTL Playground for OpenTelemetry