OpenTelemetry is emerging as the standard for modern observability. As a highly active project within the Cloud Native Computing Foundation (CNCF)—second only to Kubernetes—it has become the monitoring solution of choice for cloud-native applications. OpenTelemetry provides a unified method for collecting traces, metrics, and logs across Kubernetes, microservices, and infrastructure.

However, for many enterprises—especially in banking, insurance, healthcare, and government—the reality is more complex than just “cloud native.” Although most organizations have deployed mobile apps and adopted microservices architectures, much of their critical core processing still relies on IBM mainframe applications. These systems process credit card swipes, financial transactions, patient records, and premium calculations.

This creates a dilemma: while the modern distributed systems of the hybrid environment are well-observed, the critical backend remains a black box.

The “Broken Trace”

A common challenge we see with customers involves a request that originates from a modern mobile application. The request hits microservices running on Kubernetes, initiates a service call to the mainframe, and suddenly, visibility stops.

When latency spikes or a transaction fails, Site Reliability Engineers (SREs) are left guessing. Is it the network? The API gateway? Or underlying mainframe applications like CICS? Without a unified, end-to-end view of the services involved—from the frontend Node.js microservices to the backend CICS service—mean time to resolution (MTTR) becomes “mean time to innocence,” with teams simply proving it wasn't their microservice rather than fixing root causes.

We need a unified view where a trace flows seamlessly from a cloud-native frontend (like React) all the way into mainframe transactions.

IBM Z Observability Connect

With the recent release of Z Observability Connect, IBM has introduced OpenTelemetry-native instrumentation into mainframe applications. This creates a bridge between modern cloud-native services and mainframe transactions.

This means the mainframe is no longer a special case; it acts just like any other microservice in a mesh. It functions as an OpenTelemetry data producer, emitting traces, metrics, and logs to OpenTelemetry-compliant backends like Elastic.

The Architecture

The architecture is straightforward:

• The Collector: IBM Z Observability Connect runs on z/OS. It collects logs, metrics, or traces and converts them into the OTLP (OpenTelemetry Protocol) format.
• The Processor: The Elastic Cloud Managed OTLP Endpoint acts as a gateway collector, providing fully hosted, scalable, and reliable native OTLP ingestion.
• The Consumer: Elastic APM enables OpenTelemetry-native application performance monitoring, making it easy to pinpoint and fix performance problems quickly.

Putting it all together in Kubernetes

We deploy an OpenTelemetry Collector within our Kubernetes cluster. This collector acts as a specialized gateway. It is configured to receive OTLP traffic directly from IBM Z Observability Connect on the mainframe and forward it securely to our observability backend, Elastic APM, by using the otlp/elastic exporter.

Here is the configuration for the OpenTelemetry Collector. Note the exporters section, which handles the authentication and batched transmission to Elastic:

exporters:
  # Exporter to print the first 5 logs/metrics and then every 1000th
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 1000

  # Exporter to send logs and metrics to Elasticsearch Managed OTLP Input
  otlp/elastic:
    endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
    headers:
      Authorization: ApiKey ${env:ELASTIC_API_KEY}
    sending_queue:
      enabled: true
      sizer: bytes
      queue_size: 50000000 # 50MB uncompressed
      block_on_overflow: true
    batch:
      flush_timeout: 1s
      min_size: 1_000_000 # 1MB uncompressed
      max_size: 4_000_000 # 4MB uncompressed

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/elastic, debug]

Note: We strongly recommend using environment variables for your endpoints and API keys to keep your manifest secure.

Why the OTel specification matters

Elastic’s managed OTLP endpoint and observability solution is built with native OTel support and adheres to the OTel specification and semantic conventions. Once we wired everything up and the data started to flow, we noticed that some of the traces in Elastic APM were not being represented correctly.

Most observability solutions derive the so-called RED metrics (rate, error, and duration) for the most important spans in a trace—i.e., incoming and outgoing spans of each individual service. This allows for an efficient indication of a service’s performance without the need to comb through all of the tracing data to show something as simple as the latency of a service’s endpoint or the error rate on outgoing requests.

For an efficient calculation of such derived metrics for incoming spans on a service, the OTel community introduced the SPAN_FLAGS_CONTEXT_HAS_IS_REMOTE_MASK and SPAN_FLAGS_CONTEXT_IS_REMOTE_MASK flags on the span entities within the OTLP protocol. These flags provide an unambiguous indication of whether an individual span is an entry span and, thus, allow observability backends to efficiently calculate metrics for entry-level spans.

If these flags are set incorrectly for an entry span, the span cannot be recognized as an entry span, and metrics are not derived properly—leading to a broken experience. This is what we initially experienced with the ingested OTel data from the IBM mainframe instrumentation.

In a proprietary world, this might have been a dead end or a months-long troubleshooting exercise. However, since OpenTelemetry is an open standard, we were able to debug the issue rapidly and share our findings with IBM engineers, who quickly developed a fix.

Streamline observability

We now have end-to-end visibility that spans from modern mobile or web applications deep into the IBM mainframe. This unlocks significant value:

• Unified Service Maps: You can visually see the dependency between the cloud-native cart service and the backend inventory system on z/OS.
• Single Pane of Glass: SREs no longer need to switch between modern observability tools and separate mainframe monitoring tools to view service health.
• Operational Efficiency: By eliminating the “blind spot” in the trace, you reduce the time spent on coordinating between cloud and mainframe teams, making issue resolution faster.

Conclusion

If you are running hybrid workloads, it is time to stop treating your mainframe as a black box. With IBM Z Observability Connect, the Elastic Managed OTLP Endpoint, and Elastic APM, your entire stack can finally speak a single language: OpenTelemetry.

The gain is in where the intermediate set lives. When you ask what the saturated hosts logged, the list of saturated hosts is computed and consumed inside Elasticsearch. Six host names or 500 trace IDs never land in a clipboard or in an AI agent's context window, and the set is recomputed from current data every time the query runs.

That changes the unit of investigation. You go from "one slow request stalled on a lock" to "most of the slowest requests did," and only the second answer tells you which team to page. It matters more the further apart your signals are: in many observability stacks, logs, metrics, and traces live in three separate systems, each with its own query language, its own time picker, and its own idea of what a host is, so the same question has to be asked two or three times and the answers joined by hand.

In this post we walk through four investigations, each of which is self-contained. For every one of them, we set out the scenario that started it, the query that answers it, the table it returns, and a note on the difficulties you run into when you try to get the same answer any other way.

The data was collected with OpenTelemetry and lands in the logs-*.otel-*, traces-*.otel-*, and metrics-*.otel-* data streams, where fields keep their semantic convention names rather than being rewritten into another schema. The correlation pattern works just as well on Elastic Agent integrations, though the queries need translating rather than just renaming: ECS carries log severity as the text field log.level instead of a numeric severity_number, the System integration reports CPU as separate system.cpu.*.pct fields instead of one metric with a state dimension, and APM records durations in microseconds. All of it lands in the same cluster either way, which is the part the subquery depends on.

In Discover, the time picker already applies the range, so the examples below omit an explicit @timestamp filter. Outside Discover, add a filter by time yourself, either with literal timestamps in the query or with ?_tstart and ?_tend in the query and values in the params array of your _query request. Every result below comes from a one hour window over a synthetic fleet of 300 hosts.

Metrics to logs: what are the saturated hosts complaining about?

An infrastructure alert tells you a handful of hosts in a fleet of a few hundred sat above 90% CPU over the last hour. That tells you which hosts are hot and nothing about why. The question worth answering is whether those hosts share a failure mode, or whether they are busy for unrelated reasons and the alert is a coincidence.

FROM logs-*.otel-*
| WHERE severity_number >= 17
  AND resource.attributes.host.name IN (
      TS metrics-hostmetrics.otel-*
      | WHERE attributes.state == "idle"
      | STATS idle = AVG(AVG_OVER_TIME(metrics.system.cpu.utilization))
          BY resource.attributes.host.name
      | WHERE idle < 0.1
      | KEEP resource.attributes.host.name
    )
| STATS errors = COUNT(*), hosts = COUNT_DISTINCT(resource.attributes.host.name)
    BY pattern = CATEGORIZE(body.text), service = resource.attributes.service.name
| SORT errors DESC

The two halves of the query map onto the two halves of the question. The subquery works out which hosts were saturated, averaging CPU utilization per host and keeping the ones that averaged under 10% idle, which is another way of saying above 90% busy for the window. The outer query then works out what those hosts were complaining about, pulling their error logs and using CATEGORIZE to collapse thousands of individual lines into a handful of error classes.

Two choices in there are worth pausing on.

The subquery uses TS rather than FROM because a host does not report one CPU number. It reports a separate time series per CPU state, and per logical core as well if your collector is configured to break them out, so the reduction has to happen in two stages. AVG_OVER_TIME collapses each series to a single value first, and the outer AVG then combines those into one figure per host.

Naming that inner function matters more than it looks. Write AVG(metrics.system.cpu.utilization) on its own and TS supplies LAST_OVER_TIME for you, averaging each series' final sample rather than its average over the window. In this dataset that one substitution moves a host from 7% idle to 10% idle, which is the difference between appearing in the results and not. Querying metrics with the TS command goes into the two aggregation phases in more depth.

The log filter tests severity_number (17 is the ERROR floor on the OpenTelemetry scale) rather than the severity text, because the numeric scale is fixed by the spec while the text is whatever the emitting library decided to write. That is not a hypothetical distinction here: the error logs in this cluster carry four different labels, including SEVERE from a Java service. Matching on the text alone returns 2,754 of checkout's errors and misses the billing service entirely, while the numeric filter returns all 5,910 of them and keeps billing too.

When you run the query in Discover, the result is a short table of error classes, scoped to the saturated hosts:

The subquery returned six saturated hosts, and on all six the same service is timing out against an upstream dependency and draining its connection pool. The hosts column is what lets you set the other two rows aside without opening anything: the certificate errors reach only two of the six, and the gateway declines amount to five lines in an hour. Neither tracks the cohort the way the checkout patterns do.

Having all three signals in one store already removed the exports from this investigation. The subquery removes the step after that, and closing that last gap matters more than it sounds. By the time you have read six host names off a chart and typed them into a log search, the set has moved: a host that crossed the threshold a minute ago is missing from your list, and one that has since recovered is still in it. Here the host list is derived from current data on every run, so re-running the query during an incident gives you the current cohort.

The stale list is only half of it. A correlation done by hand exists only in the head of the person who did it, so nobody else can check it, save it, or run it again tomorrow.

Logs to metrics: do the erroring hosts look different from the healthy ones?

Filtering metrics by a log-derived host set answers the opposite question. The payments service is throwing errors on some hosts and not others, and you want to know whether resource pressure explains the split before you start reading deploy history.

That is a comparison, so the query needs both cohorts. FORK runs two branches over the same input, and IN and NOT IN against the same log-derived host set divide the fleet between them.

TS metrics-hostmetrics.otel-*
| WHERE attributes.state == "idle"
| STATS idle = AVG(AVG_OVER_TIME(metrics.system.cpu.utilization))
    BY host = resource.attributes.host.name
| EVAL busy = 1 - idle
| FORK
    ( WHERE host IN (
        FROM logs-*.otel-*
        | WHERE severity_number >= 17
          AND resource.attributes.service.name == "payments"
          AND resource.attributes.host.name IS NOT NULL
        | STATS errors = COUNT(*) BY resource.attributes.host.name
        | KEEP resource.attributes.host.name )
      | EVAL cohort = "logging errors" )
    ( WHERE host NOT IN (
        FROM logs-*.otel-*
        | WHERE severity_number >= 17
          AND resource.attributes.service.name == "payments"
          AND resource.attributes.host.name IS NOT NULL
        | STATS errors = COUNT(*) BY resource.attributes.host.name
        | KEEP resource.attributes.host.name )
      | EVAL cohort = "no errors" )
| STATS hosts = COUNT(*), mean_busy = AVG(busy), busiest_host = MAX(busy)
    BY cohort

The query reads top to bottom as three stages. The metrics query runs first and reduces the whole fleet to one busy figure per host. FORK then splits that fleet in two using the same log query in both branches, separating the hosts that appear in it from the hosts that do not. The final STATS summarizes each group, so both cohorts come back as two rows of one table, measured the same way over the same window.

This query is longer than the others, and three parts of it are less obvious than they look.

The natural way to label the two cohorts would be EVAL cohort = CASE(host IN (...), "erroring", "healthy"), and ES|QL rejects it. In 9.5 an IN subquery has to be a top-level predicate in a WHERE condition rather than an argument to a scalar function, which is why the split happens at the command level with FORK.

The STATS ... BY inside each subquery looks redundant, since KEEP alone would return the same host names. It is not: without it, the subquery returns one row per matching log document instead of one row per host, and those rows are all held in memory for the outer query to filter against. Aggregating first turns millions of rows into a few hundred host names.

The IS NOT NULL filter guards the sharpest edge here, and this dataset is a live example rather than a hypothetical. NOT IN follows SQL null semantics, so a single null in the subquery result makes the predicate match nothing at all. Five of the payments error logs in this cluster came through a sidecar that dropped the host name. Remove that one line from both branches and the query still succeeds, but it returns a single row: the nulls quietly delete the entire 286-host "no errors" cohort, and what is left looks like a perfectly plausible answer to a different question.

Run the query in Discover and you get two rows, one per cohort:

CPU does not explain the split, and the numbers say so twice. The 14 erroring hosts run slightly cooler on average than the 286 quiet ones, and the busiest machine among them averaged 50% over the hour while the quiet cohort contains a host that averaged 95%. Whatever is failing on those 14, they had headroom the entire time, and deploy history is a better place to spend the next ten minutes.

A negative result like this is worth as much as a positive one, and it is usually the one people skip. Getting it the long way means running the metrics query twice against two hand-built host lists and lining the numbers up afterwards, which is enough friction that the check often just does not happen. Both cohorts here come from the same log query in the same execution, over identical time windows, so there is nothing to reconcile and no reason not to check.

Traces to logs: what did every service log during the slow requests?

A service level objective (SLO) burn alert fires on checkout latency. Tracing gives you the slow requests and their spans, and the next question is what the services involved were writing to their logs while those specific requests were in flight.

The trace ID is the join key, and there are far too many of them to move by hand.

FROM logs-*.otel-*
| WHERE trace_id IN (
      FROM traces-*.otel-*
      | WHERE kind == "Server"
        AND resource.attributes.service.name == "checkout"
        AND name == "POST /api/orders"
        AND duration > 2000000000
      | SORT duration DESC
      | LIMIT 500
      | KEEP trace_id
    )
| STATS lines = COUNT(*), traces = COUNT_DISTINCT(trace_id)
    BY pattern = CATEGORIZE(body.text),
       service = resource.attributes.service.name,
       severity_text
| SORT traces DESC

The subquery answers "which requests were slow." It looks at the inbound request span for the checkout endpoint rather than the client and internal spans beneath it, then keeps the 500 slowest requests over two seconds. Durations are recorded in nanoseconds, which is why the threshold has so many zeros.

The outer query answers "what got logged while they were running," gathering every log line that shares one of those trace IDs and grouping them into patterns.

Counting distinct traces per pattern is what makes the output readable. A log pattern that appears 30,000 times across four traces is one chatty request, while a pattern that shows up in 470 of the 500 slowest traces is a property of being slow.

From the results above, the top row is there by construction and can be set aside: checkout writes one order submitted line per order, so it appears in all 500 traces and says nothing about why these particular 500 were slow.

From the results above, the third row is the answer, and it points at a service nobody was looking at. A lock wait timeout in inventory, two hops downstream from where the alert fired, shows up in 470 of the 500 slowest requests, and the 947 lines behind those 470 traces mean a good share of them retried more than once. The payment gateway declines are real failures, and at 19 traces out of 500 they are not what is burning the SLO.

Done by hand, this means opening slow traces one at a time and reading the correlated logs for each, which is tedious at ten traces and nobody's idea of a plan at 500. When the spans and the logs are held in different systems, every trace you check is a copied ID and a context switch, and the sample size you can afford drops to about three. Three traces is enough to form a theory and not enough to test one. Treating the slow requests as a population is what turns "this trace had a lock wait" into "470 of the 500 slowest requests had a lock wait," and that difference decides whether you page the inventory team.

All three signals: from pod memory pressure to the log lines behind the failures

IN subqueries nest, so the pattern extends to as many signal types as the question needs.

A node pool starts reporting memory pressure after a rollout. You want the log lines from the requests that actually failed on the pods under pressure, which means going from metrics to traces to logs without stopping in between.

FROM logs-*.otel-*
| WHERE trace_id IN (
      FROM traces-*.otel-*
      | WHERE kind == "Server"
        AND status.code == "Error"
        AND resource.attributes.k8s.pod.uid IN (
            TS metrics-kubeletstats.otel-*
            | STATS peak = MAX(MAX_OVER_TIME(metrics.k8s.pod.memory_limit_utilization))
                BY resource.attributes.k8s.pod.uid
            | WHERE peak > 0.95
            | KEEP resource.attributes.k8s.pod.uid
          )
      | STATS failures = COUNT(*) BY trace_id
      | SORT failures DESC
      | LIMIT 1000
      | KEEP trace_id
    )
| STATS lines = COUNT(*), traces = COUNT_DISTINCT(trace_id)
    BY pattern = CATEGORIZE(body.text), service = resource.attributes.service.name
| SORT traces DESC

Reading the query inside out, you can see each layer answering one part of the question. The innermost subquery identifies the pods whose memory peaked above 95% of their limit, using MAX_OVER_TIME to take each pod's peak rather than its average. The middle one narrows to the requests that actually failed on those pods and reduces them to at most 1,000 trace IDs. The outer query then collects the logs for those traces from every service that took part, including services running on pods that were entirely healthy, and that last part turns out to be where the answer is.

The pods are matched on their UID rather than their name, because names repeat across namespaces and restarts. The query assumes Kubernetes metadata reaches your spans, which is what the collector's k8sattributes processor is for; if it does not, the host or container ID works the same way.

From the results above, the first row sits at exactly 1,000 because that is the subquery's LIMIT, so it describes the size of the sample rather than the size of the incident. Every trace in that sample carries the cart deadline line, which is the symptom you already knew about when you started.

Look at the second row instead. product-catalog is rejecting oversized payloads across 946 of the same 1,000 traces, and it never appeared in the pod subquery at all: its pods peaked at 75% of their memory limit, well under the 95% threshold. The rollout started sending larger payloads, which would account for both cart's memory climb and the failures. Checkout's retry budget gives out in about half of them, which is how the failure became visible to users.

Filtering logs directly by the pressured pods would have shown you the cart line and hidden the product-catalog one, which is to say it would have confirmed the symptom and buried the cause. Doing it without subqueries means three queries and two hand-built lists, and the second list is a thousand trace IDs. That is usually the point at which people stop after the first hop and go with the cart theory. The reason the second hop is cheap here is that all three signals sit in the same store behind the same query language, so widening from pods to traces to every service in the trace is a clause, not a project.

Why do ES|QL subqueries matter for AI agents?

Keeping the intermediate set inside the cluster is convenient for a person and close to essential for an agent querying on your behalf.

Split across two tool calls, the intermediate result has to travel. A list of 500 trace IDs comes back in a tool response and occupies the model's context, and the agent then has to rewrite every one of them into the next query. That costs tokens on every hop, and it is where truncation and transcription errors come from. With a subquery, the intermediate set stays inside Elasticsearch and the agent only ever sees the final table.

The problem compounds when the signals are spread across systems. An agent then needs credentials, a client, and a working knowledge of the query language for each one, plus the judgment to join results that use different names for the same host. One store and one query language reduce that to a single skill the agent has to be good at.

One ES|QL string is also a complete description of the correlation, which makes the investigation reproducible: an agent can put the query in its summary, and a human can paste it into Discover and get the same logic evaluated against current data. A two-call sequence with a hardcoded host list in the middle gives you neither. The one thing to watch is that an agent calling the _query API has to filter @timestamp itself, since nothing is binding a time picker.

Four things to know before writing ES|QL IN subqueries

IN subqueries are in technical preview in Elastic Stack 9.5 and on Elastic Cloud Serverless, while TS and FORK have been generally available since 9.4. CATEGORIZE has been generally available since 9.1 and requires a Platinum license; every query above works without it if you group by an existing field instead.

Four things are worth knowing before you write your own:

• In 9.5 the subquery returns exactly one column, which is what the trailing KEEP does in each example.
• Aggregate the subquery down to distinct values with STATS ... BY before returning them. Its result is materialized for the outer query to filter against, so handing back a few hundred host names instead of a few million rows is both faster and safer.
• Filter nulls out of any NOT IN subquery, because SQL null semantics mean one null makes the predicate match nothing.
• Subqueries are non-correlated. They run independently and cannot reference columns from the outer query, so this is a set filter rather than a row-by-row join. Reach for LOOKUP JOIN when you need per-row enrichment, which we covered in ES|QL joins for richer observability.

Try ES|QL signal correlation on your own data

The pattern under all four examples is the same. You start with a set you can describe in one signal and a question you can only answer in another, and the subquery carries that set across the boundary for you.

The syntax is the smaller part of what makes that work. It works because logs, metrics, and traces sit in one store behind one query engine, under field names they kept on the way in, so crossing from one signal to another is a clause in a query rather than an integration to build and maintain. Where that is not true, the same four investigations turn into a sequence of exports, translations, and manual joins, and that costs more than slower answers. It quietly shrinks the number of questions anyone is willing to ask, and the negative results are the first to go.

Each pattern here replaces two or three queries with one, and no host list or trace ID list has to move between them. Fewer steps mean fewer places to be wrong, and a correlation you can save as a single string and hand to someone else.

To try it:

1. Open an Observability project on Elastic Cloud Serverless, or upgrade to Elastic Stack 9.5.
2. Send data with the Elastic Distributions of OpenTelemetry, or point an existing collector at Elasticsearch.
3. In Discover, switch to ES|QL and start from the metrics to logs query above, swapping in your own data streams and thresholds.
4. Read the IN subquery reference for the full set of commands you can use inside a subquery.

What is Tail Sampling?

Distributed tracing is useful for debugging, but at production scale it comes with processing overhead and storage costs, at which point sampling becomes a natural way to maintain the value of tracing while keeping costs under control. Tail-based sampling, or tail sampling, is a technique that makes a sampling decision conditionally at a later stage, so that high-value traces like errors or slow transactions are more likely to be sampled. The opposite is head sampling, which makes the decision at the start of a trace, before any such information is available.

How does the tail sampling processor work?

The tail sampling processor buffers 100% of incoming traces (or spans, used interchangeably), then forwards the sampled subset after applying the sampling policies. Buffering is a major source of memory usage, and it scales proportionally to the volume of spans, a well known pain point in the community.

Memory usage is bounded by configuration parameters like decision_wait and num_traces. Setting decision_wait to 1 minute means a sampling decision is made for a trace after 1 minute, during which all spans for that trace are expected to have arrived. If a trace is slower than 1 minute, the decision is made with some spans missing.

As a side note, scaling out the tail sampling setup involves using the loadbalancingexporter to satisfy the requirement that all spans for a trace must be routed to the same collector. This introduces some operational complexity and potentially data loss during collector restarts. But this post focuses on the memory usage of a single tail sampling processor instance, regardless of horizontal scaling.

Why does tail sampling cause memory pressure?

These parameters introduce a tradeoff between data loss and memory usage, and they require assumptions about the shape of traces: how slow they can be, how many spans they contain, how large each span is. These assumptions can become stale as instrumentation evolves.

How much data loss is acceptable to limit memory usage, and can the tradeoff be improved? The following two contributions aim to give operators more flexibility.

How span-ingest reduces tail sampling memory by releasing spans early

sampling_strategy is a new configuration option added to the tail sampling processor in v0.149.0.

sampling_strategy defaults to trace-complete, which matches the original behavior: sampling policies are only evaluated when decision_wait has elapsed, at which point the trace is considered complete. (There is a similar config, decision_wait_after_root_received, for optimization, but it is excluded from this discussion for simplicity.) This means all spans are buffered in memory for roughly decision_wait before being released, regardless of whether a decision could have been made earlier. For example, health check spans that should always be dropped are still held in memory until policy evaluation time.

Alternatively, sampling_strategy can be set to span-ingest, where spans are evaluated individually at ingest time. This allows terminal decisions, specifically drop or sampled, to be made earlier, freeing memory by dropping or exporting all spans buffered so far for that trace before decision_wait elapses. In the health check example, a policy can be configured to drop the entire trace as soon as the root span belongs to a health check. It is worth noting that an unsampled decision, unlike an explicit drop, is not terminal, as it can be overruled by a sampled or drop decision from another span in the same trace, so unsampled traces cannot be released early.

Switching from trace-complete to span-ingest will require policy adjustments, as policies can no longer assume all spans are available at evaluation time. Moreover, not all policy types are supported with the span-ingest strategy.

Disk-backed tail sampling storage with Pebble

Even with span-ingest, all spans are still buffered in memory. As decision_wait is increased to accommodate slow traces and num_traces is increased to limit data loss, the collector will eventually hit its memory limit and get OOM killed, resulting in further data loss.

What if traces were buffered on disk instead, where there is an order of magnitude more capacity? The main drawback is performance: disk throughput and latency, even with SSDs, are at least an order of magnitude slower than memory, so disk writes need to be efficient. For this reason, Pebble, an LSM database, was chosen as the storage backend for its fast write performance. Read performance is less of a concern, as reads only happen for the sampled subset of traces when sampling_strategy is set to span-ingest.

The implementation introduces a TailStorage interface for trace storage operations, and a new tail_storage option in v0.150.0 (behind feature gate processor.tailsamplingprocessor.tailstorageextension) to configure the storage backend. The default in-memory behavior is unchanged, but it is now possible to swap in a different storage backend, like the new pebbletailstorageextension contributed to Collector Contrib.

Tail sampling memory benchmarks: trace-complete vs span-ingest with Pebble

Benchmark setup: OpenTelemetry Demo with fan-out collectors

The following benchmarks were produced by running OpenTelemetry Demo with increased load against a pipe collector, which receives all spans and fans them out to two identical collectors under observation (CUO-A and CUO-B), differing only in their tail sampling configuration. Measurements include pipe collector throughput, spans received, spans sent (sampled), CPU usage, and memory usage.

Benchmark setup diagram

                      demo ns
     +----------------------------------------+
     |  opentelemetry-demo                     |
     |    loadgenerator (locust)               |
     |    services: frontend, cart, ...        |
     |    demo-collector                       |
     +----------------------------------------+
                          |  OTLP/gRPC
                          v
                     chamber ns
     +----------------------------------------+
     |             pipe-collector             |
     |         receive once, fan out          |
     |      exporters: [otlp/a, otlp/b]       |
     +----------------------------------------+
              | OTLP                  | OTLP
              v                       v
     +----------------+      +----------------+
     |     CUO-A      |      |     CUO-B      |
     | tail_sampling  |      | tail_sampling  |
     |   (config A)   |      |   (config B)   |
     +----------------+      +----------------+

Tail sampling processor configurations

CUO-A

config:
  processors:
    tail_sampling:
      sampling_strategy: trace-complete
      decision_wait: 5m
      num_traces: 5000000
      block_on_overflow: true
      decision_cache:
        sampled_cache_size: 10000
        non_sampled_cache_size: 200000
      policies:
        - name: root_1pct
          type: and
          and:
            and_sub_policy:
              - name: root_span_only
                type: ottl_condition
                ottl_condition:
                  error_mode: ignore
                  span:
                    - "IsRootSpan()"
              - name: root_probabilistic
                type: probabilistic
                probabilistic:
                  sampling_percentage: 1.0

CUO-B

CUO-B uses the same tail sampling processor configuration as CUO-A, except it sets sampling_strategy: span-ingest and tail_storage: pebble_tail_storage/main, along with its corresponding pebbletailstorageextension configuration.

extensions:
  pebble_tail_storage/main:
    directory: /var/lib/otelcol/pebble

Memory, CPU and throughput results

The following tables compare trace-complete (CUO-A) against span-ingest with Pebble disk storage (CUO-B) across memory, CPU and throughput. The process RSS, Go heap allocation, and per-process CPU measurements come from OpenTelemetry Collector internal process and runtime metrics, while container working set and container CPU come from Kubernetes cgroup metrics scraped by kubelet/cAdvisor.

Memory (peak over the window)

CPU (total over the window)

Throughput (total over the window)

Tail sampling

• cuo-a = trace-complete, cuo-b = span-ingest-pebble
• Window: 15m 39s (t+0:00 start, t+10:06 drain start, t+15:39 drain end)

The results show a significant memory reduction when using pebbletailstorageextension with span-ingest, at the cost of increased CPU usage from event serialization and database overhead.

What's next for OpenTelemetry tail sampling

Both sampling_strategy and pebbletailstorageextension are still in their early stages at the time of writing. Feedback and contributions are welcome in the OpenTelemetry Collector Contrib repo. Stay tuned for more improvements.

While the benefits of using serverless functions are manifold, it is also necessary to make them observable in the wider end-to-end microservices architecture context.

Elastic Observability (APM) for Azure Functions: The architecture

Elastic Observability 8.7 introduced distributed tracing for Microsoft Azure Functions — available for the Elastic APM Agents for .NET, Node.js, and Python. Auto-instrumentation of HTTP requests is supported out-of-the-box, enabling the detection of performance bottlenecks and sources of errors.

The key components of the solution for observing Azure Functions are:

1. The Elastic APM Agent for the relevant language
2. Elastic Observability

The APM server validates and processes incoming events from individual APM Agents and transforms them into Elasticsearch documents. The APM Agent provides auto-instrumentation capabilities for the application being observed. The Node.js APM Agent can trace function invocations in an Azure Functions app.

Setting up Elastic APM for Azure Functions

To demonstrate the setup and usage of Elastic APM, we will use a sample Node.js application.

Application overview

The Node.js application has two HTTP-triggered functions named "Hello" and "Goodbye." Once deployed, they can be called as follows, and tracing data will be sent to the configured Elastic Observability deployment.

curl -i https://<APP_NAME>.azurewebsites.net/api/hello
curl -i https://<APP_NAME>.azurewebsites.net/api/goodbye

Setup

Step 0. Prerequisites

To run the sample application, you will need:

• An installation of Node.js (v14 or later)

• Access to an Azure subscription with an appropriate role to create resources

• The Azure CLI (az) logged into an Azure subscription

  1. Use az login to login
  2. See the output of az account show

• The Azure Functions Core Tools (func) (func --version should show a 4.x version)

• An Elastic Observability deployment to which monitoring data will be sent

  1. The simplest way to get started with Elastic APM Microsoft Azure is through Elastic Cloud. Get started with Elastic Cloud on Azure Marketplace or sign up for a trial on Elastic Cloud.

• The APM server URL (serverUrl) and secret token (secretToken) from your Elastic stack deployment for configuration below

  1. How to get the serverUrl and secretToken documentation

Step 1. Clone the sample application repo and install dependencies

git clone https://github.com/elastic/azure-functions-apm-nodejs-sample-app.git
cd azure-functions-apm-nodejs-sample-app
npm install

Step 2. Deploy the Azure Function App
Caution icon! Deploying a function app to Azure can incur costs. The following setup uses the free tier of Azure Functions. Step 5 covers the clean-up of resources.

Step 2.1
To avoid name collisions with others that have independently run this demo, we need a short unique identifier for some resource names that need to be globally unique. We'll call it the DEMO_ID. You can run the following to generate one and save it to DEMO_ID and the "demo-id" file.

if [[ ! -f demo-id ]]; then node -e 'console.log(crypto.randomBytes(3).toString("hex"))' >demo-id; fi
export DEMO_ID=$(cat demo-id)
echo $DEMO_ID

Step 2.2
Before you can deploy to Azure, you will need to create some Azure resources: a Resource Group, Storage Account, and the Function App. For this demo, you can use the following commands. (See this Azure docs section for more details.)

REGION=westus2   # Or use another region listed in 'az account list-locations'.
az group create --name "AzureFnElasticApmNodeSample-rg" --location "$REGION"
az storage account create --name "eapmdemostor${DEMO_ID}" --location "$REGION" \
    --resource-group "AzureFnElasticApmNodeSample-rg" --sku Standard_LRS
az functionapp create --name "azure-functions-apm-nodejs-sample-app-${DEMO_ID}" \
    --resource-group "AzureFnElasticApmNodeSample-rg" \
    --consumption-plan-location "$REGION" --runtime node --runtime-version 18 \
    --functions-version 4 --storage-account "eapmdemostor${DEMO_ID}"

Step 2.3
Next, configure your Function App with the APM server URL and secret token for your Elastic deployment. This can be done in the Azure Portal or with the az CLI.

In the Azure portal, browse to your Function App, then its Application Settings (Azure user guide). You'll need to add two settings:

First set your APM URL and token.

export ELASTIC_APM_SERVER_URL="<your serverUrl>"
export ELASTIC_APM_SECRET_TOKEN="<your secretToken>"

Or you can use the az functionapp config appsettings set ... CLI command as follows:

az functionapp config appsettings set \
  -g "AzureFnElasticApmNodeSample-rg" -n "azure-functions-apm-nodejs-sample-app-${DEMO_ID}" \
  --settings "ELASTIC_APM_SERVER_URL=${ELASTIC_APM_SERVER_URL}"
az functionapp config appsettings set \
  -g "AzureFnElasticApmNodeSample-rg" -n "azure-functions-apm-nodejs-sample-app-${DEMO_ID}" \
  --settings "ELASTIC_APM_SECRET_TOKEN=${ELASTIC_APM_SECRET_TOKEN}"

The ELASTIC_APM_SERVER_URL and ELASTIC_APM_SECRET_TOKEN are set in Azure function’s settings for the app and used by the Elastic APM Agent. This is initiated by the initapm.js file, which starts the Elastic APM agent with:

require("elastic-apm-node").start();

When you log in to Azure and look at the function’s configuration, you will see them set:

Step 2.4
Now you can publish your app. (Re-run this command every time you make a code change.)

func azure functionapp publish "azure-functions-apm-nodejs-sample-app-${DEMO_ID}"

You should log in to Azure to see the function running.

Step 3. Try it out

% curl https://azure-functions-apm-nodejs-sample-app-${DEMO_ID}.azurewebsites.net/api/Hello
{"message":"Hello."}
% curl https://azure-functions-apm-nodejs-sample-app-${DEMO_ID}.azurewebsites.net/api/Goodbye
{"message":"Goodbye."}

In a few moments, the APM app in your Elastic deployment will show tracing data for your Azure Function app.

Step 4. Apply some load to your app
To get some more interesting data, you can run the following to generate some load on your deployed function app:

npm run loadgen

This uses the autocannon node package to generate some light load (2 concurrent users, each calling at 5 requests/s for 60s) on the "Goodbye" function.

Step 5. Clean up resources
If you deployed to Azure, you should make sure to delete any resources so you don't incur any costs.

az group delete --name "AzureFnElasticApmNodeSample-rg"

Analyzing Azure Function APM data in Elastic

Once you have successfully set up the sample application and started generating load, you should see APM data appearing in the Elastic Observability APM Services capability.

Service map

With the default setup, you will see two services in the APM Service map.

The main function: azure-functions-apm-nodejs-sample-app

And the end point where your function is accessible: azure-functions-apm-nodejs-sample-app-ec7d4c.azurewebsites.net

You will see that there is a connection between the two as your application is taking requests and answering through the endpoint.

From the APM Service map you can further investigate the function, analyze traces, look at logs, and more.

Service details

When we dive into the details, we can see several items.

• Latency for the recent load we ran against the application
• Transactions (Goodbye and Hello)
• Average throughput
• And more

Transaction details

We can see transaction details.

An individual trace shows us that the "Goodbye" function calls the "Hello" function in the same function app before returning:

Machine learning based latency correlation

As we’ve mentioned in other blogs, we can also correlate issues such as higher than normal latency. Since we see a spike at 1s, we run the embedded latency correlation, which uses machine learning to help analyze the potential impacting component by analyzing logs, metrics, and traces.

The correlation indicated there is a potential cause (25%) due to the host sending the load (my machine).

Cold start detection

Also, we can see the impact a cold start can have on the latency of a request:

Summary

Elastic Observability provides real-time monitoring of Azure Functions in your production environment for a broad range of use cases. Curated dashboards assist DevOps teams in performing root cause analysis for performance bottlenecks and errors. SRE teams can quickly view upstream and downstream dependencies, as well as perform analyses in the context of distributed microservices architecture.

Learn more

To learn how to add the Elastic APM Agent to an existing Node.js Azure Function app, read Monitoring Node.js Azure Functions. Additional resources include:

• How to deploy and manage Elastic Observability on Microsoft Azure
• Elastic APM Quickstart

Want to run trace-based tests with Elastic APM? Today is your lucky day. We're happy to announce that Tracetest now integrates with Elastic Observability APM.

Check out this hands-on example of how Tracetest works with Elastic Observability APM and OpenTelemetry!

Tracetest is a CNCF project aiming to provide a solution for deep integration and system testing by leveraging the rich data in distributed system traces. In this blog, we intend to provide an introduction to Tracetest and its capabilities, including how it can be integrated with Elastic Application Performance Monitoring and OpenTelemetry to enhance the testing process.

Your good friend distributed tracing

Distributed tracing is a way to understand how a distributed system works by tracking the flow of requests through the system. It can be used for a variety of purposes, such as identifying and fixing performance issues, figuring out what went wrong when an error occurs, and making sure that the system is running smoothly. Here are a few examples of how distributed tracing can be used:

• Monitoring performance: Distributed tracing can help you keep an eye on how your distributed system is performing by showing you what's happening in real time. This can help you spot and fix problems like bottlenecks or slow response times that can make the system less reliable.
• Finding the source of problems: When something goes wrong, distributed tracing can help you figure out what happened by showing you the sequence of events that led up to the problem. This can help you pinpoint the specific service or component that's causing the issue and fix it.
• Debugging: Distributed tracing can help you find and fix bugs by giving you detailed information about what's happening in the system. This can help you understand why certain requests are behaving in unexpected ways and how to fix them.
• Security: Distributed tracing can help you keep an eye on security by showing you who is making requests to the system, where they are coming from, and what services are being accessed.
• Optimization: Distributed tracing can help you optimize the performance of the system by providing insight into how requests are flowing through it, which can help you identify areas that can be made more efficient and reduce the number of requests that need to be handled.

Distributed tracing — Now also for testing

Observability, previously only used in operations, is now being applied in other areas of development, such as testing. This shift has led to the emergence of "Observability-driven development" and "trace-based testing" as new methods for using distributed tracing to test distributed applications.

Instead of just checking that certain parts of the code are working, trace-driven testing follows the path that a request takes as it goes through the system. This way, you can make sure that the entire system is working properly and that the right output is produced for a given input. By using distributed tracing, developers can record what happens during the test and then use that information to check that everything is working as it should.

This method of testing can help to find problems that may be hard to detect with other types of testing and can better validate that the new code is working as expected. Additionally, distributed tracing provides information about what is happening during the test, such as how long it takes for a request to be processed and which services are being used, which can help developers understand how the code behaves in a real-world scenario.

Enters Tracetest

Tracetest is a CNCF project that can run tests by verifying new traces against previously created assertions against other traces captured from the real systems. Here's how you can use Tracetest:

• Capture the baseline good known trace. This will be the golden standard that you will use to write your tests and assertions. Trace-driven development is a better way to test how different parts of the system work together because it allows developers to test the entire process from start to finish, making sure that everything is working as it should and giving a more complete view of how the system is functioning instead of trying to create disjointed assertions validating the request execution.
• Now you can start validating your code changes against good known behavior captured previously.
• Tracetest can validate the resulting traces from the test and see if the system is working as it should. This can help you find problems that traditional testing methods might not catch.
• Create reports: Tracetest can also create reports that summarize the results of the test so that you can share the information with your team.
• Help you validate in production that the new requests follow the known path and run the predefined assertions against them.

The APM tool in Kibana, which is a familiar UI for many developers, can provide extra information when used with Tracetest. The APM tool can show you how the system is performing during the test and help you find issues using the familiar user interface. For example, the APM tool can show you how requests are moving through the system, how long requests take to be processed, and which parts of the system are being used. This information can help you identify and fix problems during testing.

Furthermore, the APM tool can be set to show you all the data in real-time, which allows you to monitor the system's behavior during the test or even in production and helps you make sense of what Tracetest is showing.

How Tracetest works with Elastic APM to test the application

The components work together to provide a complete solution for testing distributed systems. The telemetry captured by the OpenTelemetry agent is sent to the Elastic APM Server, which processes and formats the data for indexing in Elasticsearch. The data can then be queried and analyzed using Kibana APM UI, and Tracetest can be used to conduct deep integration and system tests by utilizing the rich data contained in the distributed system trace.

For more details on Elastic's support for OpenTelelemetry, check out Independence with OpenTelemetry on Elastic.

1. Tracetest initiates the test by sending a request to the application under test.
2. The application processes the request, and the built-in OpenTelemetry agent captures the telemetry data of the request. This data includes information such as request and response payloads, request and response headers, and any errors that occurred during the request processing. The agent then sends the captured telemetry data to the Elastic APM Server.
3. Elastic APM server consumes OpenTelemetry or Elastic APM spans and sends the data to be stored and indexed in Elasticsearch.
4. Tracetest polls Elasticsearch to retrieve the captured trace data. It makes use of Elasticsearch query to fetch the trace data. Tracetest compares the received trace data with the expected trace data and runs the assertions. This step is used to check whether the data received from the application matches the expected data and to check for any errors or issues that may have occurred during the request processing. Based on the results of the comparison, Tracetest will report any errors or issues found and will provide detailed information about the root cause of the problem. If the test passes, Tracetest will report that the test passed, and the test execution process will be completed.
5. The trace data is visible and can be analyzed in Kibana APM UI as well.

Running your first Tracetest environment with Elastic APM and Docker compose

In your existing observability setup, you have the OpenTelemetry Nodejs agent configured in your code and sending OpenTelemetry traces to the Elastic APM server that then stores them in Elasticsearch. Adding Tracetest to the infrastructure lets you write detailed trace-based tests based on the existing tracing infrastructure. Tracetest runs tests against endpoints and uses trace data to run assertions.

The example that we are going to run is from the Tracetest GitHub repository. It contains a docker-compose setup, which is a convenient way to run multiple services together in a defined environment. The example includes a sample application that has been instrumented with an OpenTelemetry agent. The example also includes the Tracetest server with its Postgres database, which is responsible for invoking the test, polling Elasticsearch to retrieve the captured trace data, comparing the received trace data with the expected trace data, and running the assertions. Finally, the example includes Elasticsearch, Kibana, and the Elastic APM server from the Elastic Stack.

To quickly access the example, you can run the following:

git clone https://github.com/kubeshop/tracetest.git
cd tracetest/examples/tracetest-elasticapm-with-otel
docker-compose up -d

Once you have Tracetest set up, open http://localhost:11633 in your browser to check out the Web UI.

Navigate to the Settings menu and ensure the connection to Elasticsearch is working by pressing Test Connection:

To create a test, click the Create dropdown and choose Create New Test. Select the HTTP Request and give it a name and description.

For this simple example, GET the Node.js app, which runs at http://app:8080.

With the test created, you can click the Trace tab to see the distributed trace. It’s simple, but you can start to see how it delivers immediate visibility into every transaction your HTTP request generates.

From here, you can continue by adding assertions.

To make an assertion based on the GET / span of our trace, select that span in the graph view and click Current span in the Test Spec modal. Or, copy this span selector directly, using the Tracetest Selector Language:

span[tracetest.span.type="http" name="GET /" http.target="/" http.method="GET"]

Below, add the attr:http.status_code attribute and the expected value, which is 200. You can add more complex assertions as well, like testing whether the span executes in less than 500ms. Add a new assertion for attr:http.status_code, choose <, and add 500ms as the expected value.

You can check against other properties, return statuses, timing, and much more, but we’ll keep it simple for now.

Then click Save Test Spec , followed by Publish , and you’ve created your first assertion.If you open the APM app in Kibana at https://localhost:5601 (find the username and password from the examples/tracetest-elasticapm- with -otel/.env file), you will be able to navigate to the transaction generated by the test representing the overall application call with three underlying spans:

Summary

Elastic APM and Tracetest are tools that can help make testing distributed applications easier by providing a more comprehensive view of the system's behavior and allowing developers to identify and diagnose performance issues more efficiently. Tracetest allows you to test the entire process from start to finish, making sure that everything is working as it should, by following the path that a request takes.

Elastic APM provides detailed information about the performance of a system, including how requests are flowing through the system, how long requests take to be processed, and which services are being called. Together, these tools can help developers to identify and fix issues more quickly, improve collaboration and communication among the team, and ultimately improve the overall quality of the system.

• Elastic APM documentation: https://www.elastic.co/guide/en/apm/guide/current/index.html
• Tracetest documentation: https://tracetest.io/docs/ 
• Tracetest Github page: https://github.com/kubeshop/tracetest 
• Elastic blog: https://www.elastic.co/blog/category/technical-topics 
• Elastic APM community forum: https://discuss.elastic.co/c/apm 
• Tracetest support: Discord channel