The processing pipeline is straightforward: The profiler samples every CPU core on the system at a fixed rate (19Hz by default), unwinds execution stacks, symbolizes the resulting stacktraces and ships the profiles to a backend like Elasticsearch.

And then… the user has to figure out what to do with them.

That last step is where continuous profiling has historically faced adoption challenges, as the path from "profiling is on" to "profiling is useful" is steeper than it should be.

Four barriers to adoption

• Storage cost: Full stacktraces, even after deduplication and clever storage schemas, are expensive to store at fleet scale. That cost makes continuous profiling an opt-in feature in practice: a lot of potential users never enable it, and the ones who do, tend to enable it only on a subset of hosts.

• Query friction: A normalized stacktrace schema is optimized for ingestion and storage but complicates ad-hoc questions. "How much CPU time does my service spend in TLS?" is a simple question that may require intricate ES|QL or custom code in order to be answered.

• AI-hostile data: Normalized stacktrace data (typically involving multiple levels of indirection) resists straightforward algorithmic analysis. LLMs in particular struggle with it and necessitate further data transformations into representations more amenable to LLM processing.

• UX barrier: Flamegraphs are extremely useful when you know how to read them but intimidating when you don't.

These four barriers compound: storage cost limits coverage, the UX barrier limits who benefits from coverage, query friction limits what questions users can ask and the AI-hostile data representation limits what the system can do when users don't know what questions to ask.

How profiling-derived metrics work: classify at the edge

The core idea is simple: instead of sending full stacktraces all the way to a backend and asking the user to make sense of them there, we classify and count at the edge, inside an OpenTelemetry Collector pipeline, and emit ordinary OpenTelemetry time-series counters. The profiling logic itself doesn't change; it's still the OpenTelemetry eBPF profiler running inside the OpenTelemetry Collector. All the new work happens in a stateless connector inside the Collector: the connector inspects each stacktrace produced by the profiler, classifies its frames into one or more categories and increments counters.

We've released profilingmetricsconnector as part of Elastic's opentelemetry-collector-components repository. It sits between the OpenTelemetry eBPF profiler receiver and any metrics exporter, and turns symbolized stacktraces into named, aggregated counters with attributes.

Because the profilingmetricsconnector lives inside the standard OpenTelemetry Collector pipeline, every metric it produces flows through the same processors as the rest of your telemetry. In the following example, the resourcedetectionprocessor enriches each counter with host-derived attributes.

connectors:
  profilingmetrics:
    flush_interval: 30s

receivers:
  profiling: {}

exporters:
  elasticsearch:
    endpoints:
      - # ENDPOINT
    api_key: # API_KEY
    mapping:
      mode: otel

processors:
  resourcedetection:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: false
        host.arch:
          enabled: true
        os.description:
          enabled: true
        os.type:
          enabled: true

service:
  pipelines:
    profiles:
      receivers: [ profiling ]
      exporters: [ profilingmetrics ]
    metrics:
      receivers: [ profilingmetrics ]
      processors: [resourcedetection]
      exporters: [ elasticsearch ]

Profiling-derived CPU metrics: what gets emitted

The connector ships with a set of pre-baked counters built from useful classification rules. Each metric is a count of stacktrace samples whose leaf frame matched a particular category, with the frequency value standing in for CPU consumption.

| Metric | Classifies | Attached metadata | |---|---|---| | kernel.count | Kernel leaf frames | syscall, category (disk/rw, ipc/rw, network/{tcp,udp,other}/rw, memory, synchronization, …) | | native.count | Native C/C++/Rust leaf frames | shared library name (libcrypto, libclrjit, libsystemd, …) | | hotspot.count, go.count, python.count, … | Runtime-specific leaf frames | runtime-specific attributes |

The kernel categorization is worth a closer look as a modern Linux kernel has more than 400 system calls. However, most of what shows up in CPU stacktraces falls into a handful of subsystems: filesystem read/write, network read/write, memory management, scheduling, synchronization. Some syscalls (e.g. read, write) are ambiguous on their own and only become specific when one examines more frames down the stack: ext4_file_read_iter points to filesystem, tcp_v4_rcv to network. The connector handles this disambiguation as part of frame iteration.

Native frames typically lack symbolic information beyond shared library names, but those names are still informative: libssl and libcrypto mean cryptographic work as part of OpenSSL or one of its variants; libz means compression; libclrjit means the .NET JIT is busy. We don't need to enumerate libraries statically as the connector dynamically generates shlib_name attribute values using the trimmed library name (e.g. libssl not libssl.so.3) for clean cardinality.

Currently, for each stacktrace, the connector computes a Self CPU count (the leaf frame matched the category) corresponding to exclusive CPU usage. A complication exists for fine-grained kernel categories like network/tcp/write where the actual leaf frame is usually a device-driver call that we can't meaningfully match. We deal with that by trying to match frames further up the stack (e.g. tcp_sendmsg is enough to correctly classify the sample).

Users can also add their own categories by specifying a frame pattern (e.g. a function or package) and the connector will generate counters for them.

Benefits of profiling-derived metrics for observability

This shift looks small from the outside — "we're emitting counters" — but it changes almost everything about how profiling fits into an observability stack.

• Orders of magnitude less storage: A counter aggregated over a 5-second (or 30-second or one-minute) window is dramatically cheaper than the full stacktraces it distills. The pre-aggregation interval is configurable with the trade-off being time resolution rather than categorization fidelity. For most "where is my CPU being spent?" questions, 30 seconds is plenty.

• On by default: Because the storage cost is now in line with regular metrics, profiling-derived metrics can be on for everyone, on every host, from the moment the profiler is deployed. Users get a CPU breakdown by runtime, syscall, kernel category and shared library on day one.

• Standard dashboards: These are ordinary OpenTelemetry time-series counters and can be visualized ad-hoc using stacked bar graphs, pie charts, top-N panels or any other visualization Kibana supports out of the box. The same Lens and TSDB-backed views for application metrics work here.

• AI and query-friendly: Standard time-series data is trivially consumable by ES|QL, ML jobs, anomaly detectors and by LLMs. "Show me the top services by network/udp/write time, filtered to the payments namespace, over the last six hours" is one query that is not only simple for the system to answer but also simple for an LLM to generate.

• Cross-signal correlation: Because the metrics flow through the standard OpenTelemetry Collector pipeline, they pick up the same resource attributes (e.g. service.name, k8s.pod.name, host.name, deployment.environment) that logs, other metrics and traces already carry.

• Instant value, with a path to more detail: A user who just wants to know "what's burning my CPU?" gets a meaningful answer without ever opening a flamegraph. A user who wants to dig deeper still has the full eBPF profiler underneath, ready to hand back complete stacktraces when they're warranted.

User-programmable profiling and adaptive sampling

The longer-term direction is for the profiler to stop being something users consume and start being something they program. User-defined metrics are the first step in this direction, complemented by on-demand (full) profiling and adaptive sampling.

Profiling-derived metrics or other signals can act as a trigger for on-demand profiling where the system enables full profiling on a specific host or service to capture complete stacktraces. In that way, the full profiling processing and storage cost is paid only when it matters.

We can apply the same idea to the sampling rate. 19Hz is a sensible baseline for steady state but when the metrics signal an interesting event or an anomaly, the system can automatically ramp to 100Hz or higher to capture high-fidelity data for the time window during which it's relevant. It can then ramp down to baseline.

How profiling-derived metrics enable self-driving observability

Most observability stacks today use an open-loop model: the profiler emits data with a fixed configuration. Then a human looks at flamegraphs and dashboards, potentially correlates with logs, other metrics and traces, forms a hypothesis and triggers a deeper investigation. Every link in this chain requires a human decision. Nothing feeds back into the profiler at speed and the system cannot act on its own observations.

Profiling-derived metrics close that loop.

1. A "significant host events" metric, an anomaly on network/udp/write or a spike in native.count/libz: something crosses a threshold.

2. The profiler adjusts in response: sampling rate increases, full profiling turns on for the affected hosts.

3. The richer data is correlated against logs, traces, and other metrics by an LLM, by a human or both. The same resource attributes that make cross-signal correlation easy for the user make it easy for the system.

4. A root cause is identified. A remediation is suggested or applied. The metric returns to baseline and the loop continues.

This is what we mean when we talk about self-driving observability. The profiler is no longer just an instrument that someone wields. It is the sensory organ of an autonomous feedback loop: a system that observes itself, decides what to look at more closely and adjusts its own configuration in response to what it sees.

What's next: inclusive CPU, off-CPU metrics, and runtime-specific profiling

Any piece of data visible in a stacktrace can be a metric source and several extensions are already on the roadmap.

• Inclusive-CPU metrics: Today's pre-baked counters attribute CPU at the leaf frame (exclusive-CPU). Inclusive-CPU metrics will attribute the entire call chain which is useful when you care about the total cost of a function call — the function plus everything it transitively calls — not just the work done directly in its own body.

• Runtime-specific metrics: GC time per runtime, JSON/Protobuf serialization, RPC frameworks, FFI boundaries. The kinds of questions every team eventually asks about their language runtime, answered by default.

• Off-CPU metrics: On-CPU profiling tells you where you're spending CPU but Off-CPU profiling tells you where you're not (e.g. blocked on I/O, locks). The same classification logic applies, with the only change being the source signal.

Profiling-derived metrics are an active area of work within Elastic and the profilingmetricsconnector is the place to start if you want to play with this today. A ready-made Kibana integration ships dashboards for all the metrics described above.

If you're already using Elastic's continuous profiling, expect these metrics to show up as first-class citizens in the Elastic stack. If you're not, this is a very low-friction way in as no flamegraph expertise is required and storage cost is minimal.

The flamegraph isn't going anywhere, but for the first time, it isn't the only way profiling yields results.

As a core contributor, Elastic is proud to have accelerated this effort by previously donating its Universal Profiling™ eBPF-based continuous profiling agent to OpenTelemetry. This production-grade agent enables whole-system visibility across all applications, covering a multitude of programming languages and runtimes including third-party libraries and kernel operations with minimal overhead. It allows SREs and developers to quickly identify performance bottlenecks, maximize resource utilization, and optimize cloud spend.

Additionally, over the last two years, Elastic has been heavily contributing to the OpenTelemetry Collector, Semantic Conventions and Profiling Special Interest Groups (SIGs) to lay the technical foundation for the promotion of Profiles to Alpha.

This Alpha milestone not only boosts the standardization of continuous profiling but also accelerates the practical adoption of profiling as the fourth key signal in observability. Customers now have a vendor-agnostic way of collecting profiling data and enabling correlation with existing signals, like logs, metrics and traces, unveiling new potential for observability insights and a more efficient troubleshooting experience.

What is continuous profiling?

Profiling is a technique used to understand the behavior of a software application by collecting information about its execution. This includes tracking the duration of function calls, memory usage, CPU usage, and other system resources.

However, traditional profiling solutions have significant drawbacks limiting adoption in production environments:

• Significant cost and performance overhead due to code instrumentation
• Disruptive service restarts
• Inability to get visibility into third-party libraries

Unlike traditional profiling, which is often done only in a specific development phase or under controlled test conditions, continuous profiling runs in the background with minimal overhead, eliminating the need for service restarts or manual intervention. This provides real-time, actionable insights without replicating issues in separate environments. SREs, DevOps, and developers can see how code affects performance and cost, making code and infrastructure improvements easier.

Elastic's contribution: Powering the Alpha

The Elastic-donated profiler now forms the reference eBPF-based profiler implementation within OpenTelemetry: opentelemetry-ebpf-profiler. With the Alpha release, the eBPF profiler operates as an OpenTelemetry Collector receiver and contains numerous improvements such as automatic Go symbolization and support for new language runtimes. Operating as an OpenTelemetry Collector receiver enables the profiler to seamlessly leverage existing OpenTelemetry processing and filtering pipelines.

For example, the k8sattributesprocessor can use the container.id resource attribute to automatically enrich every profile with its corresponding Kubernetes context. This means you don't just see a raw stack trace; you see exactly which namespace, pod, and deployment produced it.

receivers:
  # Profiling receiver
  profiling: {}

processors:
  k8sattributes:
    passthrough: false 
    pod_association:
      - sources:
          - from: resource_attribute
            name: container.id
    extract:
      metadata:
        - "k8s.namespace.name"
        - "k8s.deployment.name"
        - "k8s.replicaset.name"
        - "k8s.statefulset.name"
        - "k8s.daemonset.name"
        - "k8s.node.name"
        - "k8s.pod.name"
        - "k8s.pod.ip"
        - "k8s.pod.uid"

Besides improvements to the eBPF profiler, Elastic has made significant contributions to:

• Correlating profiles with the information produced by OpenTelemetry eBPF instrumentation (OBI), a powerful auto-instrumentation tool that can enable distributed tracing.
• Process Context Sharing OTEP which is designed to bridge the gap between application SDKs and the profiler. This mechanism will allow OpenTelemetry SDKs to "publish" their resource attributes (like service.name) into a small, standardized memory region. Because this data is stored in the process's own memory map, the eBPF Profiler can instantly discover and associate it with its corresponding Profile.
• Semantic conventions and integration of OpenTelemetry Profiles with Google's pprof format (transparent conversion)
• OpenTelemetry Collector processing pipelines, allowing it to better integrate with the profiling receiver

Elastic's Next-Generation Profiling Development

Elastic remains deeply committed to OpenTelemetry's vision and is pushing the boundaries of what is possible with profiling data. We are dedicating a team of profiling domain experts to co-maintain and advance profiling capabilities within OpenTelemetry, while simultaneously working on groundbreaking features built on this new open standard.

Exciting areas of internal profiling-specific development include:

• OpenTelemetry Profiles derived Metrics: We are developing innovative ways to automatically generate actionable performance metrics directly from the raw OTel Profiles data, providing a new dimension for infrastructure modeling and alerting.
• Rapid Integration with the Elastic Stack: We are making swift progress on first-class support for OTLP Profiles within the Elastic Stack, ensuring seamless ingestion (the ebpf-profiler receiver is already integrated with the Elastic Distributions of OpenTelemetry (EDOT) collector), storage, and visualization of this new signal alongside your existing logs, metrics and traces.
• AI-Powered Workflows: We are leveraging the deep insights provided by continuous profiling data to power new AI-driven workflows, enabling automatic root-cause analysis, anomaly detection, and intelligent optimization suggestions for both code and infrastructure.

While the Alpha release marks a significant milestone, it is just the beginning. We encourage the community to start testing early preview versions of the OTel Profiles integration and contribute to the ongoing profiling work. To get started with an actual, local deployment, you can use the OpenTelemetry eBPF profiler in combination with a self-hosted Elastic Observability Stack or devfiler, a standalone desktop application that acts as an OpenTelemetry Profiles compliant backend aimed at experimentation and development.

Elastic integrations are designed to simplify observability by providing tools to ingest application data, process it through Ingest pipelines, and deliver prebuilt dashboards for visualization. With OpenTelemetry support, data collection and processing will transition to the OpenTelemetry Collector, while dashboards will need to adopt the OpenTelemetry data structure.

From a Log to an Integration

Although the concept of an OpenTelemetry Integration has not yet been officially defined, we envision it as a structured collection of artifacts that enables users to start monitoring an application from scratch. Each artifact has a specific role; for example, an OpenTelemetry Collector configuration file, which must be integrated into the main Collector setup. This bundled configuration instructs the Collector on how to gather and process data from the relevant application.

In the OpenTelemetry Collector, data collection is handled by the receivers component. Some receivers are tailored for specific applications, such as Kafka or MySQL, while others are designed to support general data collection methods. The specialized receivers combine data gathering and transformation within a single component. For the more generic receivers, however, additional components are needed to refine and transform the incoming data into a more application-specific format. Let’s take a look at how we can build an integration for monitoring a Nginx Ingress Controller.

The Ingress Nginx is an Ingress controller for Kubernetes, using NGINX as a reverse proxy and load balancer. Widely adopted, it plays a crucial role in directing external traffic into Kubernetes services, making its usage, performance and health essential to observe. How can we start observing the external requests done to our Ingress controller? Fortunately, the NGINX Ingress Controller generates a structured log entry for each processed request. This structured format ensures that each log entry follows a consistent structure, making it straightforward to parse and generate consistent output.

log_format upstreaminfo '$remote_addr - $remote_user [$time_local]
    "$request" ' '$status $body_bytes_sent "$http_referer" "$http_user_agent" '
    '$request_length $request_time [$proxy_upstream_name]
    [$proxy_alternative_upstream_name] $upstream_addr ' '$upstream_response_length
    $upstream_response_time $upstream_status $req_id';

All the field's definition can be found here.

The OpenTelemetry Contrib Collector does not include a receiver capable of reading and parsing all fields in an NGINX Ingress log. There are two primary reasons for this:

• Application Diversity: The landscape of applications is vast, with each generating logs in unique formats. Developing and maintaining a dedicated receiver for every application would be resource-intensive and difficult to scale.
• Data Source Flexibility: Receivers are typically designed to collect data from a specific source, like an HTTP endpoint. However, in some cases, we may want to parse logs from an alternate source, such as an NGINX Ingress log file stored in an AWS S3 bucket.

These challenges can be addressed by combining receivers and processors. Receivers handle the collection of raw data, while processors can extract specific values when a known data structure is detected. Do we need a dedicated processor to parse NGINX logs? Not necessarily. The transform processor can handle this by modifying telemetry data according to a specified configuration. This configuration is written in the OpenTelemetry Transformation Language (OTTL), a language for transforming open telemetry data based on the OpenTelemetry Collector Processing Exploration.

The concept of processors in OpenTelemetry is quite similar to the Ingest pipeline strategy currently used in Elastic integrations. The main challenge, therefore, lies in migrating Ingest pipeline configurations to OpenTelemetry Collector configurations. For a deeper dive into the challenges of such migrations, check out this article.

For reference, you can view the current Elastic NGINX Ingress Controller Ingest pipeline configuration in the following link: Elastic NGINX Ingress Controller Ingest Pipeline.

Let’s start with the data collection. By default, the NGINX Ingress Controller logs to stdout, and Kubernetes captures and stores these logs in a file. Assuming that the OpenTelemetry Collector running the following configuration has access to the Kubernetes Pod logs, we can use the filelog receiver to read the controller logs:

receivers:
  filelog/nginx:
    include_file_path: true
    include: [/var/log/pods/*nginx-ingress-nginx-controller*/controller/*.log]
    operators:
      - id: container-parser
        type: container

This configuration is designed to exclusively read the controller's pod logs, focusing on their default file path within a Kubernetes node. Furthermore, since the Ingress controller does not inherently have access to its associated Kubernetes metadata, the container-parser operator has been implemented to bridge this gap. This operator appends Kubernetes-specific attributes, such as k8s.pod.name and k8s.namespace.name, based solely on information available from the filename. For a detailed overview of the container-parser operator, see the following OpenTelemetry blog post.

Avoiding duplicated logs

The configuration outlined in this blog is designed for Kubernetes environments, where the collector runs as a Kubernetes Pod. In such setups, handling Pod restarts properly is crucial. By default, the filelog receiver reads the entire content of log files on startup. This behavior can lead to duplicate log entries being reprocessed and sent through the pipeline if the collector Pod is restarted.

To make the configuration resilient to restarts, you can use a storage extension to track file offsets. These offsets allow the filelog receiver to resume reading from the last processed position in the log file after a restart. Below is an example of how to add a file storage extension and update the filelog receiver configuration to store the offsets in a file:

extensions:
  file_storage:
  directory: /var/lib/otelcol

receivers:
  filelog/nginx:
    storage: file_storage
    ...

Important: The /var/lib/otelcol directory must be mounted as part of a Kubernetes persistent volume to ensure the stored offsets persist across Pod restarts.

Data transformation with OpenTelemetry processors

Now it’s time to parse the structured log fields and transform them into queryable OpenTelemetry fields. Initially, we considered using regular expressions with the extract_patterns function available in the OpenTelemetry Transformation Language (OTTL). However, Elastic recently contributed a new OTTL function, ExtractGrokPatterns, based on Grok—a regular expression dialect that supports reusable, aliased expressions. The function’s underlying library Elastic Go-Grok ships with numerous predefined grok patterns that simplify working with pattern matching, like %NUMBER that will match any number type; "123", "456.789", "-0.123".

Each Ingress Controller log entry begins with the client's source IP address (which may be a single IP or a list of IPs) and the username provided via Basic authentication, represented as “$remote_addr - $remote_user”. The Grok IP alias can be used to parse either an IPv4 or IPv6 address from the remote_addr field, while the %GREEDYDATA alias can capture the remote_user value.

For example, the following OTTL configuration will transform an unstructured body message to a structured one with two fields:

• Parses a single IP address and assign it to the source.address key.
• Delimited by a “-”, captures the optional value of the authenticated username in the user.name key.

transform/parse_nginx_ingress_access/log:
  log_statements:
    - context: log
      statements:
        - set(body, ExtractGrokPatterns(body, "%{IP:source.address} - (-|%{GREEDYDATA:user.name})", true))

The screenshot below illustrates the transformation process, showing the original input data alongside the resulting structured format (diff):

In real-world scenarios, NGINX Ingress Controller logs may begin with a list of IP addresses or, at times, a domain name. These variations can be handled with an extended Grok pattern. Similarly, we can use Grok to parse an HTTP UserAgent and URL strings, but additional OTTL functions, such as URL or UserAgent, are required to extract meaningful data from these fields.

The complete configuration is available in the documentation for Elastic’s OpenTelemetry NGINX Ingress Controller integration: Integration Documentation.

Usage

The Elastic OpenTelemetry NGINX Ingress Controller is currently on Technical preview. To access it, you must enable the "Display beta integrations" toggle in the Integrations menu within Kibana.

By installing the Elastic OpenTelemetry NGINX Ingress Controller integration, a couple of dashboards will become available in your Kibana profile. One of these dashboards provides insights into access events for the controller, displaying information such as HTTP response status codes over time, request volume per URL, distribution of incoming requests by browser, top requested pages, and more. The screenshot below shows the NGINX Ingress Controller Access Logs dashboard, displaying data from a controller routing requests to an OpenTelemetry Demo deployment:

The second dashboard focuses on errors within the Nginx Ingress controller, highlighting the volume of error events generated over time:

To start gathering and processing controller logs, we recommend incorporating the OpenTelemetry Collector pipeline outlined in the integration’s documentation into your collector configuration: Integration Documentation. Keep in mind that this configuration requires access to the Kubernetes node's Pods logs, typically stored in /var/log/pods/*. To ensure proper access, we recommend deploying the OpenTelemetry Collector as a daemonset in Kubernetes, as this deployment type allows the collector to access the necessary log directory on each node.

The OpenTelemetry Collector configuration service pipeline should include a similar configuration:

service:
  extensions: [file_storage]
  pipelines:
    logs/nginx_ingress_controller:
      receivers:
        - filelog
      processors:
        - transform/parse_nginx_ingress_access/log
        - transform/parse_nginx_ingress_error/log
        - resourcedetection/system
      exporters:
        - elasticsearch

Adding GeoIP Metadata

As an optional enhancement, the OpenTelemetry Collector GeoIP processor can be configured and added to the pipeline to enrich each NGINX Ingress Controller log with geographical attributes, such as the request’s originating country, region, and city, enabling geo maps in Kibana to visualize traffic distribution and geographic patterns.

While the OpenTelemetry GeoIP processor is similar to Elastic's GeoIP processor, it requires users to provide their own local GeoLite2 database. The following configuration extends the Integration’s configuration to include the GeoIP processor with a MaxMind's database.

processors:
  geoip:
    context: record
    providers:
      maxmind:
        database_path: /tmp/GeoLite2-City.mmdb

service:
  extensions: [file_storage]
  pipelines:
    logs/nginx_ingress_controller:
      receivers:
        - filelog
      processors:
        - transform/parse_nginx_ingress_access/log
        - transform/parse_nginx_ingress_error/log
        - resourcedetection/system
        - geoip
      exporters:
        - elasticsearch

Sample Kibana Map with the OpenTelemetry Nginx Ingress Controller integration:

Next steps

OpenTelemetry Log Event

A closer look at the OTTL integration’s statements reveals that the raw log message is replaced by the parsed fields. In other words, the configuration transforms the body log field* from a string into a structured map of key-value pairs, as seen in “set(body, ExtractGrokPatterns(body,…)”. This approach is based on treating each NGINX Ingress Controller log entry as an OpenTelemetry Event—a specialized type of LogRecord. Events are OpenTelemetry’s standardized semantic formatting for LogRecords, containing an “event.name” attribute which defines the structure of the body field. An NGINX Ingress Controller log record aligns well with the OpenTelemetry Event data model. It follows a structured format and clearly distinguishes between two event types: access logs and error logs. There is an ongoing PR to incorporate the NGINX Ingress controller log into the OpenTelemetry semantic convention: https://github.com/open-telemetry/semantic-conventions/pull/982

Operating system breakdown

Each controller log contains the source UserAgent, from which the integration extracts the browser that originated the request. This information is valuable for understanding user access patterns, as it provides insights into the types of browsers commonly interacting with your services. Additionally, an ongoing pull request into OTTL aims to extend this functionality by extracting operating system (OS) details as well, providing even deeper insights into the environments interacting with the NGINX Ingress Controller.

Configuration encapsulation

Setting up the configuration for the NGINX Ingress Controller integration can be somewhat tedious, as it involves adding several complex processor configurations to the existing collector pipelines. This process can quickly become cumbersome, especially for non-expert users or in cases where the collector configuration is already quite complex. In an ideal scenario, users would simply reference a pre-defined integration configuration, and the collector would automatically "unwrap" all the necessary components into the corresponding pipelines. This would significantly simplify the setup process, making it more accessible and reducing the risk of misconfigurations. To address this, there is a RFC (Request for Comments) proposing support for shareable, modular configurations within the OpenTelemetry Collector. This feature would allow users to easily collect signals from specific services or applications by referencing modular configurations, streamlining the setup and enhancing usability for complex scenarios.

*The OpenTelemetry community is currently discussing whether structured body-extracted information should be stored in the attributes or body field. For details, see this ongoing issue.

This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com

What is the OpenTelemetry Demo?

The OpenTelemetry Demo is a microservices-based application created by OpenTelemetry's community to showcase its capabilities in a realistic, and distributed system environment. This demo application, known as the OpenTelemetry Astronomy Shop, simulates an e-commerce website composed of over 10 interconnected microservices (written in multiple languages: Go, Java, .NET, Node.js, etc.), communicating via HTTP and gRPC. Each service is fully instrumented with OpenTelemetry, generating comprehensive traces, metrics, and logs. The demo serves as an invaluable resource for understanding how to implement and use OpenTelemetry in real-world applications.

One of the microservices, called loadgenerator, automatically starts generating requests to the various endpoints of the demo, simulating a real-world environment where multiple clients are interacting with the system. This helps replicate the behavior of a busy, live application with concurrent user activity.

Elastic's fork

Elastic recognized an opportunity to enhance the OpenTelemetry Demo by forking it and integrating advanced Elastic features for deeper observability and simpler monitoring. While forking is the recommended OpenTelemetry approach, we aim to leverage the robust foundation and latest updates from the upstream version as much as possible. To achieve this, Elastic’s fork of the OpenTelemetry Demo performs daily pulls from upstream, seamlessly integrating them with Elastic-specific changes. To avoid conflicts, we continuously contribute upstream, ensuring Elastic's modifications are always additive or configurable through environment variables. One such contribution is the .env.override file, designed exclusively for vendor forks to override the microservices images and configuration files used in the demo.

Deeper Insights with Elastic Distributions

In our current update of Elastic's OpenTelemetry Demo fork, we have replaced some of the microservices OTel SDKs used for instrumentation with Elastic's specialized distributions. These changes ensure deeper integration with Elastic's observability tools, offering richer insights and more robust monitoring capabilities. These are some of the fork's changes:

Java services: The Ad, Fraud Detection, and Kafka services now utilize the Elastic distribution of the OpenTelemetry Java Agent. One of the included features in the distribution are stack traces, which provides precise information of where in the code path a span was originated. Learn more about the Elastic Java Agent here.

The Cart service has been upgraded to use the Elastic distribution of the OpenTelemetry .NET Agent. This replacement gives visibility on how the Elastic Distribution of OpenTelemetry .NET (EDOT .NET) can be used to get started using OpenTelemetry in your .NET applications with zero code changes. Discover more about the Elastic .NET Agent in this blog post.

In the Payment service, we've configured the Elastic distribution of the OpenTelemetry Node.js Agent. The distribution ships with the host-metrics extension, and Kibana provides a curated service metrics UI. Read more about the Elastic Node.js Agent here.

The Recommendation service now leverages the EDOT Python, replacing the standard OpenTelemetry Python agent. The Python distribution is another example of a Zero-code (or Automatic) instrumentation, meaning that the distribution will set up the OpenTelemetry SDK and enable all the recommended instrumentations for you. Find out more about the Elastic Python Agent in this blog post.

It's important to highlight that Elastic Distributions of OpenTelemetry don't bundle proprietary software, they have been build on top of the vanilla OTel SDKs but they offer some advantages, such as single package for installation, easy auto-instrumentation with reasonable default configuration, automatic logs telemetry sending, and many more. Along these lines, the ultimate goal is to contribute as many features from EDOT's back to the upstream OpenTelemetry agents; they are designed in such a way that the additional features, realized as extensions, work directly with the OTel SDKs.

Collecting Data with the Elastic Collector Distribution

The OpenTelemetry Demo applications generate and send their signals to an OpenTelemetry Collector OTLP endpoint. In the Demo's fork, the EDOT collector is set up to forward all OTLP signals from the microservices to an APM server OTLP endpoint. Additionally, it sends all other metrics and logs collected by the collector to an Elasticsearch endpoint.

If the fork is deployed in a Kubernetes environment, the collector will automatically start collecting the system's metrics. The collector will be configured to use the hostmetrics receivers to monitor all the K8s node's metrics, the kuebeletstats receiver to retrieve Kubelet's metrics and the filelog receiver, that will collect all cluster's.

Both the signals generated by the microservices and those collected by the EDOT collector are enriched with Kubernetes metadata, allowing users to correlate them seamlessly. This makes it easy to track and observe which Kubernetes nodes and pods each service is running on, providing deep insights into both application performance and infrastructure health.

Learn more about the Elastic's OpenTelemetry Collector distribution: https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-collector

Error detection with Elastic

The OpenTelemetry Demo incorporates flagd, a feature flag evaluation engine used to simulate error scenarios. For example, the paymentServiceFailure flag will force an error for every request to the payment service charge endpoint. Since the service is instrumented with OpenTelemetry, the error will be captured in the generated traces. We can then use Kibana's powerful visualization and search tools to trace the error back to its root cause.

Another available flag is named adServiceHighCpu, which causes a high CPU load in the ad service. This increased CPU usage can be monitored either through the service's metrics or the related metrics of its Kubernetes pod:

The full list of simulated scenarios can be found at this link.

Start your own exploration

Ready to explore the OpenTelemetry Demo with Elastic and its enhanced observability capabilities? Follow the link to Kibana and begin your own exploration of how Elastic and OpenTelemetry can transform your approach to observability.

Live demo: https://ela.st/demo-otel

But that's not all—if you want to take it a step further, you can deploy the OpenTelemetry Demo directly with your own Elasticsearch stack. Follow the steps provided here to set it up and start gaining valuable insights from your own environment.