With today’s acceptance, we are officially contributing our continuous profiler technology to OpenTelemetry. We will also dedicate a team of profiling domain experts to co-maintain and advance the profiling capabilities within OTel.

We want to thank the OpenTelemetry community for the great and constructive cooperation on the donation proposal. We look forward to jointly establishing continuous profiling as an integral part of OpenTelemetry.

What is continuous profiling?

Profiling is a technique used to understand the behavior of a software application by collecting information about its execution. This includes tracking the duration of function calls, memory usage, CPU usage, and other system resources.

However, traditional profiling solutions have significant drawbacks limiting adoption in production environments:

• Significant cost and performance overhead due to code instrumentation
• Disruptive service restarts
• Inability to get visibility into third-party libraries

Unlike traditional profiling, which is often done only in a specific development phase or under controlled test conditions, continuous profiling runs in the background with minimal overhead. This provides real-time, actionable insights without replicating issues in separate environments. SREs, DevOps, and developers can see how code affects performance and cost, making code and infrastructure improvements easier.

Contribution of production-grade features

Elastic Universal Profiling is a whole-system, always-on, continuous profiling solution that eliminates the need for code instrumentation, recompilation, on-host debug symbols or service restarts. Leveraging eBPF, Elastic Universal Profiling profiles every line of code running on a machine, including application code, kernel, and third-party libraries. The solution measures code efficiency in three dimensions, CPU utilization, CO2, and cloud cost, to help organizations manage efficient services by minimizing computational waste.

The Elastic profiling agent facilitates identifying non-optimal code paths, uncovering "unknown unknowns", and provides comprehensive visibility into the runtime behavior of all applications. Elastic’s continuous profiling agent supports various runtimes and languages, such as C/C++, Rust, Zig, Go, Java, Python, Ruby, PHP, Node.js, V8, Perl, and .NET.

Additionally, organizations can meet sustainability objectives by minimizing computational wastage, ensuring seamless alignment with their strategic ESG goals.

Benefits to OpenTelemetry

This contribution not only boosts the standardization of continuous profiling for observability but also accelerates the practical adoption of profiling as the fourth key signal in OTel. Customers get a vendor-agnostic way of collecting profiling data and enabling correlation with existing signals, like tracing, metrics, and logs, opening new potential for observability insights and a more efficient troubleshooting experience. 

OTel-based continuous profiling unlocks the following possibilities for users:

• Improved customer experience: delivering consistent service quality and performance through continuous profiling ensures customers have an application that performs optimally, remains responsive, and is reliable.

• Maximize gross margins: Businesses can optimize their cloud spend and improve profitability by reducing the computational resources needed to run applications. Whole system continuous profiling identifies the most expensive functions (down to the lines of code) across diverse environments that may span multiple cloud providers. In the cloud context, every CPU cycle saved translates to money saved. 

• Minimize environmental impact: energy consumption associated with computing is a growing concern (source: MIT Energy Initiative ). More efficient code translates to lower energy consumption, reducing carbon (CO2) footprint. 

• Accelerate engineering workflows: continuous profiling provides detailed insights to help troubleshoot complex issues faster, guide development, and improve overall code quality.

• Improved vendor neutrality and increased efficiency: an OTel eBPF-based profiling agent removes the need to use proprietary APM agents and offers a more efficient way to collect profiling telemetry.

With these benefits, customers can now manage the overall application’s efficiency on the cloud while ensuring their engineering teams optimize it.

What comes next?

While the acceptance of Elastic’s donation of the profiling agent marks a significant milestone in the evolution of OTel’s eBPF-based continuous profiling capabilities, it represents the beginning of a broader journey. Moving forward, we will continue collaborating closely with the OTel Profiling and Collector SIGs to ensure seamless integration of the profiling agent within the broader OTel ecosystem. During this phase, users can test early preview versions of the OTel profiling integration by following the directions in the otel-profiling-agent repository.

Elastic remains deeply committed to OTel’s vision of enabling cross-signal correlation. We plan to further contribute to the community by sharing our innovative research and implementations, specifically those facilitating the correlation between profiling data and distributed traces, across several OTel language SDKs and the profiling agent.

We are excited about our growing relationship with OTel and the opportunity to donate our profiling agent in a way that benefits both the Elastic community and the broader OTel community. Learn more about Elastic’s OpenTelemetry support and learn how to contribute to the ongoing profiling work in the community.

Additional Resources

Additional details on Elastic’s Universal Profiling can be found in the FAQ.

For insights into observability, visit Observability labs where OTel specific articles are also available.

Elastic Universal Profiling™ Agent goes open source under Apache 2

At Elastic, open source is more than just a philosophy — it's our DNA. We believe the benefits of whole-system continuous profiling extend far beyond performance optimization. It's a win for businesses and the planet alike. For instance, since launching Elastic Universal Profiling in general availability (GA), we've observed a wide variety of use cases from customers.

These range from customers relying fully on Universal Profiling's differential flame graphs and topN functions for insights during release management to utilizing AI assistants for quickly optimizing expensive functions. This includes using profiling data to identify the optimal energy-efficient cloud region to run certain workloads. Additionally, customers are using insights that Universal Profiling provides to build evidence to challenge cloud provider bills. As it turns out, cloud providers' in-VM agents can consume a significant portion of the CPU time, which customers are billed for.

In a move that will empower the community to take advantage of continuous profiling's benefits, we're thrilled to announce that the Elastic Universal Profiling agent , a pioneering eBPF-based continuous profiling agent, is now open source under the Apache 2 license!

This move democratizes hyper-scaler efficiency for everyone , opening exciting new possibilities for the future of continuous profiling, as well as its role in observability and OpenTelemetry.

Implementation of the OpenTelemetry (OTel) Profiling protocol

Our commitment to open source goes beyond just the agent itself. We recently announced our intent to donate the agent to OpenTelemetry and have further solidified this goal by implementing the experimental OTel Profiling data model. This allows the open-sourced eBPF-based continuous profiling agent to communicate seamlessly with OpenTelemetry backends.

But that's not all! We've also launched an innovative feature that correlates profiling data with OpenTelemetry distributed traces. This powerful capability offers a deeper level of insight into application performance, enabling the identification of bottlenecks with greater precision. Upon donating the Profiling agent to OTel, Elastic will also contribute critical components that enable distributed trace correlation within the Elastic distribution of the OTel Java agent to the upstream OTel Java SDK. This underscores Elastic Observability's commitment to both open source and the support of open standards like OpenTelemetry while pushing the boundaries of what is possible in observability.

What does this mean for Elastic Universal Profiling customers?

We'd like to express our immense gratitude to all our customers who have been part of this journey, from the early stages of private beta to GA. Your feedback has been invaluable in shaping Universal Profiling into the powerful product it is today.

By open-sourcing the Universal Profiling agent and contributing it to OpenTelemetry, we're fostering a win-win situation for both you and the broader community. This move opens doors for innovation and collaboration, ultimately leading to a more robust and versatile whole-system continuous profiling solution for everyone.

Furthermore, we're actively working on exciting novel ways to integrate Universal Profiling seamlessly within Elastic Observability. Expect further announcements soon, outlining how you can unlock even greater value from your profiling data within a unified observability experience in a way that has never been done before.

The open-sourced agent is using the recently released (experimental) OTel Profiling signal. As a precaution, we recommend not using it in production environments.

Please continue using the official Elastic distribution of the Universal Profiling agent until the agent is formally accepted by OTel and the protocol reaches a stable phase. There's no need to take any action at this time, and we will ensure to have a smooth transition plan in place for you.

What does this mean for the OpenTelemetry community?

OpenTelemetry is adopting continuous profiling as a key signal. By open-sourcing the eBPF-based profiling agent and working towards donating it to OTel, Elastic is making it possible to accelerate the standardization of continuous profiling within OpenTelemetry. This move has a massive impact on the observability community, empowering everyone to continuously profile their systems with a standardized protocol.

This is particularly timely as Moore's Law slows down and cloud computing takes hold, making computational efficiency critical for businesses.

Here's how whole-system continuous profiling benefits you:

• Maximize gross margins: By reducing the computational resources needed to run applications, businesses can optimize their cloud spend and improve profitability. Whole-system continuous profiling is one way of identifying the most expensive applications (down to the lines of code) across diverse environments that may span multiple cloud providers. This principle aligns with the familiar adage, "a penny saved is a penny earned." In the cloud context, every CPU cycle saved translates to money saved.

• Minimize environmental impact: Energy consumption associated with computing is a growing concern (source: MIT Energy Initiative). More efficient code translates to lower energy consumption, contributing to a reduction in carbon footprint.

• Accelerate engineering workflows: Continuous profiling provides detailed insights to help debug complex issues faster, guide development, and improve overall code quality.

This is where Elastic Universal Profiling comes in — designed to help organizations run efficient services by minimizing computational wastage. To this end, it measures code efficiency in three dimensions: CPU utilization , CO** 2 , and cloud cost**.

Elastic's journey with continuous profiling began by joining forces with optimyze.cloud –– this became the foundation for Elastic Universal Profiling. We are excited to see this product evolve into its next growth phase in the open-source world.

Ready to give it a spin?

As Elastic Universal Profiling transitions into this new open source era, the potential for transformative impact on performance optimization, cost efficiency, and environmental sustainability is immense. Elastic's approach — balancing innovation with responsibility — paves the way for a future where technology not only powers our world but does so in a way that is sustainable and accessible to all.

Get started with the open source Elastic Universal Profiling agent today! Download it directly from GitHub and follow the instructions in the repository.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

As a core contributor, Elastic is proud to have accelerated this effort by previously donating its Universal Profiling™ eBPF-based continuous profiling agent to OpenTelemetry. This production-grade agent enables whole-system visibility across all applications, covering a multitude of programming languages and runtimes including third-party libraries and kernel operations with minimal overhead. It allows SREs and developers to quickly identify performance bottlenecks, maximize resource utilization, and optimize cloud spend.

Additionally, over the last two years, Elastic has been heavily contributing to the OpenTelemetry Collector, Semantic Conventions and Profiling Special Interest Groups (SIGs) to lay the technical foundation for the promotion of Profiles to Alpha.

This Alpha milestone not only boosts the standardization of continuous profiling but also accelerates the practical adoption of profiling as the fourth key signal in observability. Customers now have a vendor-agnostic way of collecting profiling data and enabling correlation with existing signals, like logs, metrics and traces, unveiling new potential for observability insights and a more efficient troubleshooting experience.

What is continuous profiling?

Profiling is a technique used to understand the behavior of a software application by collecting information about its execution. This includes tracking the duration of function calls, memory usage, CPU usage, and other system resources.

However, traditional profiling solutions have significant drawbacks limiting adoption in production environments:

• Significant cost and performance overhead due to code instrumentation
• Disruptive service restarts
• Inability to get visibility into third-party libraries

Unlike traditional profiling, which is often done only in a specific development phase or under controlled test conditions, continuous profiling runs in the background with minimal overhead, eliminating the need for service restarts or manual intervention. This provides real-time, actionable insights without replicating issues in separate environments. SREs, DevOps, and developers can see how code affects performance and cost, making code and infrastructure improvements easier.

Elastic's contribution: Powering the Alpha

The Elastic-donated profiler now forms the reference eBPF-based profiler implementation within OpenTelemetry: opentelemetry-ebpf-profiler. With the Alpha release, the eBPF profiler operates as an OpenTelemetry Collector receiver and contains numerous improvements such as automatic Go symbolization and support for new language runtimes. Operating as an OpenTelemetry Collector receiver enables the profiler to seamlessly leverage existing OpenTelemetry processing and filtering pipelines.

For example, the k8sattributesprocessor can use the container.id resource attribute to automatically enrich every profile with its corresponding Kubernetes context. This means you don't just see a raw stack trace; you see exactly which namespace, pod, and deployment produced it.

receivers:
  # Profiling receiver
  profiling: {}

processors:
  k8sattributes:
    passthrough: false 
    pod_association:
      - sources:
          - from: resource_attribute
            name: container.id
    extract:
      metadata:
        - "k8s.namespace.name"
        - "k8s.deployment.name"
        - "k8s.replicaset.name"
        - "k8s.statefulset.name"
        - "k8s.daemonset.name"
        - "k8s.node.name"
        - "k8s.pod.name"
        - "k8s.pod.ip"
        - "k8s.pod.uid"

Besides improvements to the eBPF profiler, Elastic has made significant contributions to:

• Correlating profiles with the information produced by OpenTelemetry eBPF instrumentation (OBI), a powerful auto-instrumentation tool that can enable distributed tracing.
• Process Context Sharing OTEP which is designed to bridge the gap between application SDKs and the profiler. This mechanism will allow OpenTelemetry SDKs to "publish" their resource attributes (like service.name) into a small, standardized memory region. Because this data is stored in the process's own memory map, the eBPF Profiler can instantly discover and associate it with its corresponding Profile.
• Semantic conventions and integration of OpenTelemetry Profiles with Google's pprof format (transparent conversion)
• OpenTelemetry Collector processing pipelines, allowing it to better integrate with the profiling receiver

Elastic's Next-Generation Profiling Development

Elastic remains deeply committed to OpenTelemetry's vision and is pushing the boundaries of what is possible with profiling data. We are dedicating a team of profiling domain experts to co-maintain and advance profiling capabilities within OpenTelemetry, while simultaneously working on groundbreaking features built on this new open standard.

Exciting areas of internal profiling-specific development include:

• OpenTelemetry Profiles derived Metrics: We are developing innovative ways to automatically generate actionable performance metrics directly from the raw OTel Profiles data, providing a new dimension for infrastructure modeling and alerting.
• Rapid Integration with the Elastic Stack: We are making swift progress on first-class support for OTLP Profiles within the Elastic Stack, ensuring seamless ingestion (the ebpf-profiler receiver is already integrated with the Elastic Distributions of OpenTelemetry (EDOT) collector), storage, and visualization of this new signal alongside your existing logs, metrics and traces.
• AI-Powered Workflows: We are leveraging the deep insights provided by continuous profiling data to power new AI-driven workflows, enabling automatic root-cause analysis, anomaly detection, and intelligent optimization suggestions for both code and infrastructure.

While the Alpha release marks a significant milestone, it is just the beginning. We encourage the community to start testing early preview versions of the OTel Profiles integration and contribute to the ongoing profiling work. To get started with an actual, local deployment, you can use the OpenTelemetry eBPF profiler in combination with a self-hosted Elastic Observability Stack or devfiler, a standalone desktop application that acts as an OpenTelemetry Profiles compliant backend aimed at experimentation and development.

The processing pipeline is straightforward: The profiler samples every CPU core on the system at a fixed rate (19Hz by default), unwinds execution stacks, symbolizes the resulting stacktraces and ships the profiles to a backend like Elasticsearch.

And then... the user has to figure out what to do with them.

That last step is where continuous profiling has historically faced adoption challenges, as the path from "profiling is on" to "profiling is useful" is steeper than it should be.

Four barriers to adoption

• Storage cost: Full stacktraces, even after deduplication and clever storage schemas, are expensive to store at fleet scale. That cost makes continuous profiling an opt-in feature in practice: a lot of potential users never enable it, and the ones who do, tend to enable it only on a subset of hosts.

• Query friction: A normalized stacktrace schema is optimized for ingestion and storage but complicates ad-hoc questions. "How much CPU time does my service spend in TLS?" is a simple question that may require intricate ES|QL or custom code in order to be answered.

• AI-hostile data: Normalized stacktrace data (typically involving multiple levels of indirection) resists straightforward algorithmic analysis. LLMs in particular struggle with it and necessitate further data transformations into representations more amenable to LLM processing.

• UX barrier: Flamegraphs are extremely useful when you know how to read them but intimidating when you don't.

These four barriers compound: storage cost limits coverage, the UX barrier limits who benefits from coverage, query friction limits what questions users can ask and the AI-hostile data representation limits what the system can do when users don't know what questions to ask.

How profiling-derived metrics work: classify at the edge

The core idea is simple: instead of sending full stacktraces all the way to a backend and asking the user to make sense of them there, we classify and count at the edge, inside an OpenTelemetry Collector pipeline, and emit ordinary OpenTelemetry time-series counters. The profiling logic itself doesn't change; it's still the OpenTelemetry eBPF profiler running inside the OpenTelemetry Collector. All the new work happens in a stateless connector inside the Collector: the connector inspects each stacktrace produced by the profiler, classifies its frames into one or more categories and increments counters.

We've released profilingmetricsconnector as part of Elastic's opentelemetry-collector-components repository. It sits between the OpenTelemetry eBPF profiler receiver and any metrics exporter, and turns symbolized stacktraces into named, aggregated counters with attributes.

Because the profilingmetricsconnector lives inside the standard OpenTelemetry Collector pipeline, every metric it produces flows through the same processors as the rest of your telemetry. In the following example, the resourcedetectionprocessor enriches each counter with host-derived attributes.

connectors:
  profilingmetrics:
    flush_interval: 30s

receivers:
  profiling: {}

exporters:
  elasticsearch:
    endpoints:
      - # ENDPOINT
    api_key: # API_KEY
    mapping:
      mode: otel

processors:
  resourcedetection:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: false
        host.arch:
          enabled: true
        os.description:
          enabled: true
        os.type:
          enabled: true

service:
  pipelines:
    profiles:
      receivers: [ profiling ]
      exporters: [ profilingmetrics ]
    metrics:
      receivers: [ profilingmetrics ]
      processors: [resourcedetection]
      exporters: [ elasticsearch ]

Profiling-derived CPU metrics: what gets emitted

The connector ships with a set of pre-baked counters built from useful classification rules. Each metric is a count of stacktrace samples whose leaf frame matched a particular category, with the frequency value standing in for CPU consumption.

The kernel categorization is worth a closer look as a modern Linux kernel has more than 400 system calls. However, most of what shows up in CPU stacktraces falls into a handful of subsystems: filesystem read/write, network read/write, memory management, scheduling, synchronization. Some syscalls (e.g. read, write) are ambiguous on their own and only become specific when one examines more frames down the stack: ext4_file_read_iter points to filesystem, tcp_v4_rcv to network. The connector handles this disambiguation as part of frame iteration.

Native frames typically lack symbolic information beyond shared library names, but those names are still informative: libssl and libcrypto mean cryptographic work as part of OpenSSL or one of its variants; libz means compression; libclrjit means the .NET JIT is busy. We don't need to enumerate libraries statically as the connector dynamically generates shlib_name attribute values using the trimmed library name (e.g. libssl not libssl.so.3) for clean cardinality.

Currently, for each stacktrace, the connector computes a Self CPU count (the leaf frame matched the category) corresponding to exclusive CPU usage. A complication exists for fine-grained kernel categories like network/tcp/write where the actual leaf frame is usually a device-driver call that we can't meaningfully match. We deal with that by trying to match frames further up the stack (e.g. tcp_sendmsg is enough to correctly classify the sample).

Users can also add their own categories by specifying a frame pattern (e.g. a function or package) and the connector will generate counters for them.

Benefits of profiling-derived metrics for observability

This shift looks small from the outside — "we're emitting counters" — but it changes almost everything about how profiling fits into an observability stack.

• Orders of magnitude less storage: A counter aggregated over a 5-second (or 30-second or one-minute) window is dramatically cheaper than the full stacktraces it distills. The pre-aggregation interval is configurable with the trade-off being time resolution rather than categorization fidelity. For most "where is my CPU being spent?" questions, 30 seconds is plenty.

• On by default: Because the storage cost is now in line with regular metrics, profiling-derived metrics can be on for everyone, on every host, from the moment the profiler is deployed. Users get a CPU breakdown by runtime, syscall, kernel category and shared library on day one.

• Standard dashboards: These are ordinary OpenTelemetry time-series counters and can be visualized ad-hoc using stacked bar graphs, pie charts, top-N panels or any other visualization Kibana supports out of the box. The same Lens and TSDB-backed views for application metrics work here.

• AI and query-friendly: Standard time-series data is trivially consumable by ES|QL, ML jobs, anomaly detectors and by LLMs. "Show me the top services by network/udp/write time, filtered to the payments namespace, over the last six hours" is one query that is not only simple for the system to answer but also simple for an LLM to generate.

• Cross-signal correlation: Because the metrics flow through the standard OpenTelemetry Collector pipeline, they pick up the same resource attributes (e.g. service.name, k8s.pod.name, host.name, deployment.environment) that logs, other metrics and traces already carry.

• Instant value, with a path to more detail: A user who just wants to know "what's burning my CPU?" gets a meaningful answer without ever opening a flamegraph. A user who wants to dig deeper still has the full eBPF profiler underneath, ready to hand back complete stacktraces when they're warranted.

User-programmable profiling and adaptive sampling

The longer-term direction is for the profiler to stop being something users consume and start being something they program. User-defined metrics are the first step in this direction, complemented by on-demand (full) profiling and adaptive sampling.

Profiling-derived metrics or other signals can act as a trigger for on-demand profiling where the system enables full profiling on a specific host or service to capture complete stacktraces. In that way, the full profiling processing and storage cost is paid only when it matters.

We can apply the same idea to the sampling rate. 19Hz is a sensible baseline for steady state but when the metrics signal an interesting event or an anomaly, the system can automatically ramp to 100Hz or higher to capture high-fidelity data for the time window during which it's relevant. It can then ramp down to baseline.

How profiling-derived metrics enable self-driving observability

Most observability stacks today use an open-loop model: the profiler emits data with a fixed configuration. Then a human looks at flamegraphs and dashboards, potentially correlates with logs, other metrics and traces, forms a hypothesis and triggers a deeper investigation. Every link in this chain requires a human decision. Nothing feeds back into the profiler at speed and the system cannot act on its own observations.

Profiling-derived metrics close that loop.

1. A "significant host events" metric, an anomaly on network/udp/write or a spike in native.count/libz: something crosses a threshold.

2. The profiler adjusts in response: sampling rate increases, full profiling turns on for the affected hosts.

3. The richer data is correlated against logs, traces, and other metrics by an LLM, by a human or both. The same resource attributes that make cross-signal correlation easy for the user make it easy for the system.

4. A root cause is identified. A remediation is suggested or applied. The metric returns to baseline and the loop continues.

This is what we mean when we talk about self-driving observability. The profiler is no longer just an instrument that someone wields. It is the sensory organ of an autonomous feedback loop: a system that observes itself, decides what to look at more closely and adjusts its own configuration in response to what it sees.

What's next: inclusive CPU, off-CPU metrics, and runtime-specific profiling

Any piece of data visible in a stacktrace can be a metric source and several extensions are already on the roadmap.

• Inclusive-CPU metrics: Today's pre-baked counters attribute CPU at the leaf frame (exclusive-CPU). Inclusive-CPU metrics will attribute the entire call chain which is useful when you care about the total cost of a function call — the function plus everything it transitively calls — not just the work done directly in its own body.

• Runtime-specific metrics: GC time per runtime, JSON/Protobuf serialization, RPC frameworks, FFI boundaries. The kinds of questions every team eventually asks about their language runtime, answered by default.

• Off-CPU metrics: On-CPU profiling tells you where you're spending CPU but Off-CPU profiling tells you where you're not (e.g. blocked on I/O, locks). The same classification logic applies, with the only change being the source signal.

Profiling-derived metrics are an active area of work within Elastic and the profilingmetricsconnector is the place to start if you want to play with this today. A ready-made Kibana integration ships dashboards for all the metrics described above.

If you're already using Elastic's continuous profiling, expect these metrics to show up as first-class citizens in the Elastic stack. If you're not, this is a very low-friction way in as no flamegraph expertise is required and storage cost is minimal.

The flamegraph isn't going anywhere, but for the first time, it isn't the only way profiling yields results.