Three ES|QL tools and about two dozen lines of query text turn AI root cause analysis into something you can check. This article builds them in Elastic Agent Builder and wires them to an agent whose report names the tool behind every number, and a trace.id and document _id behind every root cause claim.

What you need to reproduce this root cause analysis

• Elasticsearch and Kibana 9.5, self-managed or on Elastic Cloud.
• A generative AI connector for Agent Builder. Ours is Anthropic Claude Sonnet 4.6.
• Python 3.12 with opentelemetry-distro[otlp] and the Flask and requests instrumentations.
• An Elasticsearch API key with write access for OTLP ingest and Kibana access for the Agent Builder APIs.

Use the companion notebook if you want to reproduce the use case in this article.

The demo environment: four Python services on OpenTelemetry

Four Python services run on one host. checkout-api handles the customer request and calls pricing-api, which calls fx-rates for a currency quote. A fourth service, search-api, serves product search and sits outside that call path.

Each service exports OTLP straight to the Elasticsearch native OTLP endpoint, with no collector in between. Logs land in logs-generic.otel-default and spans in traces-generic.otel-default. Two checkout-api processes run on different versions at once, which one of the tools below depends on.

Starting a service is one command, with the OTLP endpoint and the service identity in the environment:

export OTEL_EXPORTER_OTLP_ENDPOINT="${ES_URL}/_otlp"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey ${ES_API_KEY}"
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true

OTEL_SERVICE_NAME=checkout-api \
OTEL_RESOURCE_ATTRIBUTES="service.version=2026.07.26.2,deployment.environment=production" \
  opentelemetry-instrument python services/checkout_api.py

Why the loudest service is not the root cause

At 09:55:33Z, fx-rates starts refusing about a third of its requests because its cached FX snapshot is older than the max_age it enforces. pricing-api turns each refusal into a 500, and checkout-api returns a 500 to the customer.

Two unrelated things happen in the same window. Release 2026.07.26.2 of checkout-api rolls out at 09:55, the minute the errors start, and 120 milliseconds after fx-rates breaks, search-api starts timing out during a scheduled reindex of its catalog.

One failure propagates, and the two other signals share only the clock:

The on-call view is four unhealthy services and a fresh deploy. This query gives the failure rate per service over the window:

FROM traces-generic.otel-default
| WHERE kind == "Server"
| EVAL failed = CASE(attributes.http.status_code >= 500, 1, 0)
| STATS failures = SUM(failed), requests = COUNT(*)
    BY service = resource.attributes.service.name
| EVAL failure_rate_pct = ROUND(100.0 * failures / requests, 1)
| SORT failure_rate_pct DESC

Nothing in the result ranks the candidates. search-api has the highest failure rate at 36.4%, and it is the one service with no involvement in the failing checkouts.

Automated root cause analysis with the default Elastic AI Agent

Before writing any tools, we gave the incident to the default Elastic AI Agent to see how far the built-in observability skills get on their own:

checkout-api started returning HTTP 500s to customers today. Investigate the
window 2026-07-26T09:50:00.000Z to 2026-07-26T10:02:00.000Z and tell me the root
cause. Context you have from the deploy log: checkout-api release 2026.07.26.2
rolled out at 09:55 UTC, and the on-call channel also reported search-api
timeouts starting at 09:55 UTC.

It got the cascade right. It named fx-rates as the origin, quoted the stale-snapshot message, and separated the search-api timeouts as a different problem. Then it ranked its hypotheses:

Hypothesis 2 claims the release restarted fx-rates without refreshing its snapshot, or lowered its max_age. Neither appears in the telemetry: the release was on checkout-api, and fx-rates reports service.version 2026.07.19.1 across all 7,374 of its spans.

The agent had a timestamp: the deploy and the first error share a minute. Across three runs, it connected them with a different invented mechanism each time:

• a restart that dropped the snapshot
• a new call path that was not exercised before the release
• a previously tolerant code path that stopped tolerating stale data

The numbers have the same problem. The impact summary in that same answer reports the failure rate as "~38%", dividing the 2,043 failures by the 5,331 successes instead of the 7,374 requests, with no query next to the number for a reader to check.

The rest of this article closes that gap with tools.

The 16 built-in observability tools in Agent Builder

Agent Builder ships with tools that cover most of the exploration work, so check the built-in tools reference before writing anything. The catalog is under Agent Builder > Manage components > Tools:

Sixteen of the built-in tools are observability tools, organized around investigation steps instead of index operations:

| Tool | What it answers | | :---- | :---- | | observability.get_logs | What is the log volume and shape for this filter, with samples and message categories | | observability.get_traces | What documents belong to these traces, grouped by trace.id | | observability.get_service_topology | Which dependencies does this service have, with error rate per connection | | observability.get_apm_correlations | Which attributes are over-represented in the slow or failing transactions | | observability.run_log_rate_analysis | Which fields or patterns correlate with a change in log throughput | | observability.get_log_change_points | Which message categories spiked, dipped or shifted, and when |

One platform tool matters here too. platform.streams.investigation_progress_report is what an agent calls to publish its hypothesis list while it works, with a status per hypothesis, a conclusion, and an explicit list of what the data could not settle:

{
  "summary": "string",
  "hypotheses": [
    {
      "candidate": "string",
      "confidence": 0.0,
      "status": "investigating | dismissed | confirmed",
      "reason": "string"
    }
  ],
  "conclusion": "string",
  "gaps_found": ["string"]
}

Three ES|QL tools for AI root cause analysis

Agent Builder's built-in observability tools are shaped for showing the landscape. Ruling a candidate out needs one number that answers one question, and two questions in this incident have no built-in tool. A third returns the document IDs that make a claim checkable.

How to tell one incident from two coincident failures (rca_failure_shapes)

Two services failing in the same minute belong to the same incident only if they appear in the same requests. This tool groups error logs by trace, collapses each trace to the set of services that erred inside it, then counts each shape:

FROM logs-generic.otel-default
| WHERE @timestamp >= TO_DATETIME(?start) AND @timestamp <= TO_DATETIME(?end)
  AND severity_text == "ERROR" AND trace_id IS NOT NULL
| STATS services = VALUES(resource.attributes.service.name) BY trace_id
| EVAL failure_shape = MV_CONCAT(MV_SORT(services), " + ")
| STATS traces = COUNT(*) BY failure_shape
| SORT traces DESC
| LIMIT 20

MV_CONCAT(MV_SORT(...)) matters here. Grouping directly by a multivalue field makes ES|QL expand it, one row per service, which loses the combination. Collapsing the set to a single string first keeps the shape intact.

Two clusters come back, and they share no traces. The three checkout services appear together in 2,043 traces, search-api errs alone in 1,324, and that zero overlap settles the coincidence in one row.

How to rule out a deploy as the root cause (rca_version_split)

A release that caused the failures makes the version carrying it fail at a materially higher rate than the version it replaced. Server spans carry both service.version and the response status, so one query decides it:

FROM traces-generic.otel-default
| WHERE @timestamp >= TO_DATETIME(?start) AND @timestamp <= TO_DATETIME(?end)
  AND resource.attributes.service.name == ?service AND kind == "Server"
| EVAL failed = CASE(attributes.http.status_code >= 500, 1, 0)
| STATS failures = SUM(failed), requests = COUNT(*)
    BY version = resource.attributes.service.version
| EVAL failure_rate_pct = ROUND(100.0 * failures / requests, 1)
| SORT version

27.2% against 28.2%, on roughly 3,700 requests per version, is a difference within noise. The version that was already running fails as often as the one that shipped, which rules the release out.

The tool depends on both versions serving at once. With an instantaneous and total rollout, no query separates a broken new version from something else breaking at the same moment.

How to return document IDs an agent can cite (rca_evidence_sample)

METADATA _id on an ES|QL source command returns the Elasticsearch document ID, so a claim can point at a specific record:

FROM logs-generic.otel-default METADATA _id, _index
| WHERE @timestamp >= TO_DATETIME(?start) AND @timestamp <= TO_DATETIME(?end)
  AND severity_text == "ERROR" AND resource.attributes.service.name == ?service
| KEEP @timestamp, _id, _index, trace_id,
       attributes.error.kind, attributes.upstream.service, body.text
| SORT @timestamp DESC
| LIMIT 5

Each tool is registered with one POST kbn:/api/agent_builder/tools call carrying the query and its typed parameters. Registered, rca_failure_shapes looks like this; the agent fills start and end at call time:

Write the tool description as the decision it supports

The model reads the tool description to decide when to call the tool, so the description does more work than the name.

We describe rca_failure_shapes as the way to test whether two services that broke together are one failure or two. That phrasing got it called whenever a question mentioned a second failing service.

The description also outweighs the agent instructions. An agent with these three tools and one line of instruction, "You are an SRE assistant, help the user find the root cause of production incidents", still ruled out the deploy and the coincidence every time.

Wiring the tools into an incident root cause analysis agent

The agent gets the three custom tools, four built-in observability tools, and platform.streams.investigation_progress_report. Its instructions are five numbered steps:

1. Scope. Establish the affected service, the failure window, and the size of the
   symptom before you name any cause. State the window as an explicit ISO 8601
   range and reuse that same range in every tool call.

2. Enumerate. Write down at least three candidate causes before you test any of
   them. Include the candidate a human on call would reach for first, such as a
   recent deploy or another service that started failing at the same minute.

3. Refute. For each candidate, state the observation that would prove it wrong,
   then run the query that produces that observation. A candidate is dismissed
   when the refuting observation is present, not when a different candidate
   looks better.

4. Cite. Every number you report must name the tool that returned it. Every claim
   about a root cause must carry at least one trace_id and at least one document
   _id. A claim with no citation is not a finding, it is a guess.

5. Report. Put anything the available data cannot settle in gaps_found rather
   than resolving it with reasoning.

Two rules follow the steps. Each one blocks a mistake we saw in the default agent's answers:

• Do not rank services by error volume and call the loudest one the cause.
• Do not treat time correlation as causation. Two services that start failing in the same minute belong to the same incident only if they appear in the same traces.

The finished agent, with Elastic capabilities switched off so everything it produces comes from these eight tools:

Running the investigation: 11 tool calls, 72 seconds

We gave the new agent the same question, word for word. It opened with a progress report listing its candidates, then ran the refuting queries in parallel batches:

Eleven tool calls and 72 seconds later, the report opens with a citation table:

Three layers, three verbatim messages, three document IDs, and one trace ID shared by all of them. The origin message states the mechanism: StaleQuoteError: fx snapshot age 1215s exceeds max_age 900s (provider=ecb-eod).

The dismissals come next, each with the observation that ruled it out:

| Candidate | Refuting observation | Tool | | :---- | :---- | :---- | | checkout-api release 2026.07.26.2 | Both versions fail at the same rate, 27.2% against 28.2%, on roughly 3,700 requests each | rca_version_split | | search-api timeouts | Zero shared traces. 2,043 traces contain the three checkout services, 1,324 contain search-api alone | rca_failure_shapes | | Database failure | No database errors in the checkout-api logs | rca_evidence_sample |

The report closes with a gaps section:

The first gap covers the question the default agent answered with an invented mechanism. The snapshot was already 1,127 seconds old at the first error, so the feed stopped refreshing before the window opened, and no log in this system records when. The next place to look is the feed ingestion job.

Verifying the agent's citation by hand in ES|QL

The report gives a trace ID, so open it:

FROM logs-generic.otel-default
| WHERE trace_id == "a131d0cf7343195c0a7a14f6a98da6b7"
| KEEP @timestamp, resource.attributes.service.name,
       attributes.upstream.service, body.text
| SORT @timestamp ASC

Three records, one millisecond apart, in the order the report claimed. fx-rates errs first and names no upstream, because it is the origin. pricing-api names fx-rates, and checkout-api names pricing-api.

Add upstream.service to your error logs

upstream.service is a logging convention that OpenTelemetry does not provide: each service writes it on the error it emits when a dependency fails. With that one key, error records order themselves into a call chain, and the ordering survives the clock skew and sub-millisecond hops that break timestamp sorting.

Two other fields carry the rest of the weight: trace.id arrives free from auto-instrumentation as long as the log is emitted inside an active span, and a structured error.kind lets you count failure modes without matching on message text.

Extracting knowledge indicators from your streams

Knowledge indicators are facts Elastic extracts from your raw data with LLM models. It picks out things like the underlying infrastructure and the dependencies between services.

You find this option under Streams > {your_stream} > Significant events:

Click Generate and the indicators are created for you:

What three ES|QL tools changed about the agent's answer

Three ES|QL tools and about two dozen lines of query text turned an agent's answer into a report that names its candidates, shows the observation that ruled each one out, and carries a record ID behind every number.

Next steps

• Run the companion notebook to reproduce the incident, the tools, and the agent in your own cluster.
• Write the query that rules out your team's most common wrong answer, the recent deploy or the loudest service, and register it as a tool whose description states the decision it supports.
• Audit one service's error logs for trace.id, a structured error.kind, and a declared upstream.service. The last two are a one-line change in your logging.
• Register the failure modes your team already knows about as significant events, so the next investigation starts with a history instead of a blank window.
• For a different shape of Agent Builder investigation, read From five dashboards to one prompt, which scores APM service health with five ES|QL tools.

This is the gap that Elastic's Vercel Integration is designed to close. Not just logs, but the full picture. Real-user performance, server health, traces showing the complete request journey, governance events, traffic, and engagement. All of it is queryable in one place.

And this is the part that's easy to miss: there's nothing to deploy on your side. No OpenTelemetry Collector, no Elastic Agent, no agent policy. Vercel has a feature called Drains that streams your data as HTTP webhooks. Elastic operates a managed endpoint on Elastic Cloud that receives those payloads and writes them into Elasticsearch. You point a drain at a URL with an API key, and data starts flowing. That's the entirety of the infrastructure story.

What Vercel drains send to Elastic Cloud

The integration pulls in five signal types from Vercel, and they're more complementary than they might seem at first.

| Signal | What it carries | SDK required | Plan | Endpoint | |---|---|---|---|---| | Web Analytics | Page views and custom events, with traffic patterns and geography | @vercel/analytics | Pro or Enterprise | Vercel endpoint | | Speed Insights | Real user Web Vitals: LCP, INP, CLS, TTFB, tagged by route, device type, and country | @vercel/speed-insights | Pro or Enterprise | Vercel endpoint | | Logs | Build output, static asset requests, Lambda function output, edge function output | None | Pro or Enterprise | Vercel endpoint | | Audit Logs | Team governance events: environment variable changes, drain edits, project transfers | None | Enterprise | Vercel endpoint | | Traces | Infrastructure spans and outbound HTTP fetch calls, plus framework and custom spans | @vercel/otel (optional) | Pro or Enterprise | Managed OTLP endpoint, /v1/traces |

Vercel Web Analytics: traffic and custom events

Web Analytics captures page views and custom events from the @vercel/analytics SDK (traffic patterns, geography, what pages users are actually hitting). The SDK itself can be installed on any Vercel plan. Forwarding that data to Elastic requires a drain, which is available on Pro and Enterprise plans.

Vercel Speed Insights: real user Web Vitals

Speed Insights carries real-user Web Vitals, including LCP, INP, CLS, and TTFB. These are measured in the browser and tagged with route, device type, and country. Install the @vercel/speed-insights SDK in your app, then create a drain on a Pro or Enterprise plan to send Web Vitals to Elastic. When you see a Lambda error spike and want to know whether users actually noticed, this data can help answer that question.

Vercel logs: build, Lambda and edge output

Logs cover the full server-side runtime picture, including build output, static asset requests, Lambda function output, and edge function output. When something fails, this is where you start digging. You can configure sampling rates and narrow by environment directly in the Vercel drain settings. Logs require a Pro or Enterprise plan.

Vercel audit logs: team governance events

Audit Logs cover team-level governance events (who changed an environment variable, who modified a drain, and who transferred a project). It is surprisingly useful when you're trying to understand whether a configuration change preceded an incident. Audit Logs require an Enterprise plan.

Vercel traces: OpenTelemetry spans end to end

Traces are where the integration gets genuinely powerful for debugging. When you configure a Trace Drain, Vercel automatically instruments infrastructure spans and outbound HTTP fetch calls. You can add the @vercel/otel package for framework and custom spans. These traces surface in Kibana's Service Inventory: trace waterfalls, service maps, and the ability to jump directly from a trace to the related log lines. A Pro or Enterprise plan is required for traces. Traces use a separate endpoint from the other signals; they go to /v1/traces on the Managed OTLP endpoint.

Each signal has its own drain in Vercel and lands in its own data stream in Elasticsearch.

How to send Vercel logs, speed insights, web analytics and traces to Elastic Cloud

In Elastic Cloud, go to Add data → Connect directly to the endpoint. You'll see the Vercel endpoint listed alongside an option to create an API key. Grab both before heading to Vercel.

The Vercel endpoint URL and API key are both available here; everything you need before touching Vercel's settings.

Over in Vercel, open your team settings and go to Drains → Add Drain. Pick the signal type you want to collect, hit Next, paste in the Elastic endpoint URL, and add one custom header:

Authorization: ApiKey <your-key>

Vercel lets you drain Logs, Traces, Speed Insights, Web Analytics, Audit Logs and more, with each as a separate drain. Logs, Speed Insights, Web Analytics, and Audit Logs share the Vercel endpoint, while Traces use the Managed OTLP traces URL.

The destination URL and Authorization header are all it takes – Elastic handles everything from here.

Repeat for each signal type. For Traces, use the Managed OTLP endpoint with /v1/traces appended, and not the standard Vercel drain URL.

For the full setup walkthrough and configuration options, refer to the Vercel (OpenTelemetry) Integration docs.

Serverless monitoring in Kibana: dashboards, alerts and SLOs

Once data is flowing, all the assets (dashboards, alert and SLO templates) install automatically. But the real value isn't in isolation, it's having everything in the same place when something goes wrong.

Dashboards

The Logs dashboard is where most investigations start: request volume, error rates, HTTP status distribution, top routes, and regional breakdowns. If errors are concentrated in a single region or a specific route, you can spot it right away.

The Speed Insights dashboard shows Core Web Vitals at p75, over time and per page, using the same "good / needs improvement / poor" thresholds Vercel uses. When a deployment ships and LCP starts drifting, this is where you'll see it first.

The Web Analytics dashboard covers the client-side traffic story: page views, geography, devices, top pages, and custom events. It is useful on its own, but especially alongside the Logs dashboard when you're trying to understand whether a server-side problem is actually reaching users.

Ten alert rules for Vercel failure modes

Beyond dashboards, ten pre-built alert rules cover the failure modes that actually matter for Vercel workloads:

• 5xx spikes.
• Lambda and edge function hard crashes (the kind that never return an HTTP response and disappear from standard error-rate math).
• Sudden traffic drops on both server and client side.
• Regional error concentration.
• TTFB degradation as an early warning for LCP regressions.
• WAF deny spikes.
• Suspicious audit activity from a single actor.

Five SLO templates for error rate and Web Vitals

Five SLO templates round things out, all on a rolling 30-day window. Server-side error rate at 99% is the core target. The remaining four cover Core Web Vitals and TTFB at p75, so real-user performance becomes something you can formally commit to, not just a metric you check occasionally.

Trace waterfalls and service maps in Kibana

For traces, once data is flowing you get the full experience in Kibana's Service inventory: end-to-end trace waterfalls, service maps, and log lines correlated right alongside the spans.

Set up serverless observability for your Vercel project

If you're already on Elastic Cloud, you're closer than you think. The endpoint is already there; you just need to point Vercel at it. Start with Web Analytics or Speed Insights if you want a focused first step: install the SDK Vercel already recommends, then create a drain on your Pro or Enterprise team. You'll have real-user performance data in Kibana within minutes. From there, layer in logs, then traces when you're ready for the full debugging story.

The integration is available on Elastic Cloud Serverless and Elastic Cloud Hosted. Head to Add data → Connect directly to the endpoint in your Elastic Cloud project to grab your endpoint URL and get going.

Both features follow the OTel GenAI semantic conventions and work with any OTel-instrumented provider. If your framework already emits OTel GenAI span attributes, there is nothing to change.

How OTel GenAI spans are structured

A GenAI span stores everything as span attributes. A typical chat span includes:

• gen_ai.provider.name: the provider (openai, anthropic, aws.bedrock, etc.); gen_ai.system is supported as a fallback for older instrumentation.
• gen_ai.operation.name: the operation type (chat, embeddings, etc.).
• gen_ai.request.model: the model being called.
• gen_ai.usage.input_tokens: tokens consumed by the prompt.
• gen_ai.usage.output_tokens: tokens generated in the response.
• gen_ai.input.messages, gen_ai.output.messages: conversation messages.
• gen_ai.system_instructions: the system prompt.

Both features read from these attributes:

| Feature | What it shows | Where it appears | Attributes it reads | | --- | --- | --- | --- | | GenAI tab | Details (operation type, request model, provider, input and output token counts, response model, response ID) and Conversation (system prompt, user messages, model response) | Span flyout in the APM trace view, and the span flyout in Discover | Appears with any gen_ai.* attribute. Conversation needs gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages | | Token count badges | Input and output token counts for each GenAI span | Every GenAI span row in the trace waterfall | gen_ai.usage.input_tokens, gen_ai.usage.output_tokens |

How to read LLM prompts and responses in the GenAI tab

When any gen_ai.* attribute is present on a span, the span flyout shows a dedicated GenAI tab next to Metadata.

The Details section shows model metadata from the span attributes: operation type, request model, provider, input and output token counts, response model, and response ID. The Conversation section shows the full exchange, populated from gen_ai.system_instructions (system prompt), gen_ai.input.messages (user messages), and gen_ai.output.messages (model response), each with a copy button so you can pull the exact prompt or response out of the trace without scraping text from a formatted table.

All raw span attributes remain accessible on the Metadata tab.

The GenAI tab is also available in the span flyout in Discover, so you can inspect LLM prompts and responses directly alongside your log and trace data without switching to the APM view.

What instrumentation does LLM tracing require?

No Kibana-side configuration is needed. The GenAI tab appears automatically when any gen_ai.* attribute is present on a span. Full Conversation support requires the OTel GenAI v1.37.0 span-attribute model: gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions.

Frameworks that emit the older span-events model (gen_ai.user.message, gen_ai.assistant.message, gen_ai.choice) will show the Details metadata section but will not populate the Conversation section. For a current list of compatible instrumentations, see the OTel GenAI semantic conventions.

To verify, open the span in Discover, check that gen_ai.input.messages and gen_ai.output.messages are present, and confirm the Conversation section renders.

If your application already sends APM data to Elastic from a GenAI workload, open any GenAI span in the trace view and check for the GenAI tab.

LLM token usage in the trace waterfall

Token count badges now appear on each GenAI span row in the waterfall, so you can scan the full trace without drilling in. In agentic traces with ten or twenty LLM calls, this lets you identify which span is driving token consumption before opening any span.

Each row shows input and output token count badges sourced from gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. The row label is the span name, which instrumentation frameworks typically set to something like chat gpt-4o-mini.

Waterfall-level token counts are most useful in agentic traces where a chain of LLM calls uses different models or the same model with varying context sizes.

Which LLM providers does Elastic APM support for GenAI tracing?

Elastic APM's LLM tracing works with any OTel-instrumented provider: the GenAI tab and waterfall token counts use the same OTel attribute schema regardless of which provider your application uses. Provider is read from gen_ai.provider.name, falling back to gen_ai.system for older instrumentation.

The OTel GenAI semantic conventions that enable this provider detection are currently in a Development lifecycle. Check the release notes before upgrading instrumentation.

How to enable LLM tracing in Elastic APM

Availability: Both features are available as a Technical Preview on Elastic Serverless and will be available as a Technical Preview in Elastic Stack 9.6.

To try these features:

1. Instrument your GenAI application with an OTel SDK that follows the OTel GenAI semantic conventions (v1.37.0 or later for full Conversation support).
2. Send traces to Elastic Observability using OTLP, the Elastic APM agent, or an EDOT SDK.
3. Open the APM section in Kibana, navigate to a service that makes LLM calls, and open the trace waterfall for any transaction.

The GenAI tab appears on any span with at least one gen_ai.* attribute set; token count badges appear when gen_ai.usage.input_tokens or gen_ai.usage.output_tokens are present.

If you don't have a GenAI application to test with, the otel-genai-chat-app repository is a minimal OpenAI chat app pre-instrumented with EDOT. Set OPENAI_API_KEY and follow the EDOT commands in the README to send traces to Elastic and see both features in action.

What's next for LLM observability in Elastic APM

We're exploring cost estimation per span (estimated spend based on model pricing and token counts, surfaced in the waterfall) and tool call rendering (structured display of tool/function call inputs and outputs for agentic spans).

If you are building GenAI applications and want early access or to share feedback, reach out through the Elastic community forums or open an issue in the kibana repository.

Those traces show how the agent reached its recommendation and what it cost. They do not record who approved it. Below: how to read the traces, scope the three identities a run touches, and append the approval decision to a data stream the pipeline cannot rewrite.

Prerequisites

• Elastic Stack 9.5 or Serverless
• Privileges to manage Kibana advanced settings, needed to install the traces dashboard.

Where Elastic Observability records each part of an AI agent action

Four questions come up in every review of an agentic operations pipeline, and each one is answered by a different record.

| Question | Where the answer lives | Who creates it | | :---- | :---- | :---- | | How did the agent reach its recommendation? | traces-agent_builder.otel-* spans | Agent Builder, automatically | | Which tools did it call, and did they fail? | execute_tool spans in the same data stream | Agent Builder, automatically | | Whose privileges did the run execute with? | Workflow execution record and Elasticsearch security audit logs | Kibana, partly | | What did a human decide, and did the action run? | An index you write to yourself | You |

The first two are new in 9.5 and cost nothing but a toggle. The last one has no automatic source, so it is the one most pipelines are missing.

The scenario: a stale pricing cache in checkout

Three checkout-service workers serve production traffic. One of them, checkout-worker-1, was rolled to version 2026.07.26.1 and now returns HTTP 500 on every quote because its pricing cache stopped refreshing. The other two stay on 2026.07.25.3 and serve normally.

The SRE control plane pattern behind this setup connects telemetry, an Agent Builder agent that reasons over it, and Elastic Workflows that run known actions, with a human approval gate before the first step that changes production.

Telemetry arrives through the documented OpenTelemetry path, so the agent reads standard OTel fields. Elasticsearch 9.5 exposes a native OTLP endpoint, which lets an OTel SDK write directly to the cluster with no collector in between:

from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource

provider = LoggerProvider(
    resource=Resource.create({
        "service.name": "checkout-service",
        "deployment.environment": "production",
    })
)
provider.add_log_record_processor(
    BatchLogRecordProcessor(
        OTLPLogExporter(
            endpoint=f"{ES_URL}/_otlp/v1/logs",
            headers={"Authorization": f"ApiKey {API_KEY}"},
        )
    )
)

The endpoint speaks OTLP over protobuf and rejects application/json with HTTP 406, so send it through an SDK or collector rather than hand-built JSON. That writes 450 records to logs-generic.otel-default: 360 healthy events across the three workers and 90 PricingCacheStaleError events from the broken one. The agent is given none of that context and has to find it by querying.

Reading AI agent observability traces in Elastic

Everything Agent Builder records about a run lives in two data streams, and all of it is queryable with ES|QL.

How to turn on AI agent tracing in GenAI Settings

Open Stack Management, then GenAI Settings, and find the Agent Builder Traces section. Collect conversation traces is on by default in 9.5.

Two details on that panel matter more than the toggle. Traces are written to traces-agent_builder.otel-<space-id>, one data stream per Kibana space, with a companion logs-agent_builder.otel-<space-id> for agent-side events. These are ordinary data streams on the standard OTel index templates, not hidden system indices, so Discover, Lens, and ES|QL query them directly.

The callout states the access model plainly: anyone who can read the index can read every trace in it. Trace access is not scoped per user, so restrict the index pattern through a role before granting access to a space with sensitive conversations.

Reading the LLM trace waterfall for one agent run

A single question through the converse API asks the built-in Elastic AI Agent to investigate logs-generic.otel-default, find which pod and version are returning HTTP 500, and propose one bounded action. It answers correctly, naming checkout-worker-1 on 2026.07.26.1 with a 43% error rate against zero errors on the two workers still on 2026.07.25.3.

Select the trace icon under any agent response to open the waterfall.

One question through the converse API produced 24 spans over 43.7 seconds. The structure is an invoke_agent root, a generate_title side branch, then alternating chat and execute_tool spans as the agent queries, reads the result, and picks the next query.

Three span families carry everything you will aggregate on.

| Span name prefix | What it represents | Key attributes | | :---- | :---- | :---- | | invoke_agent | A conversation round (CHAIN) or an agent execution (AGENT) | elastic.inference.span.kind, gen_ai.agent.id | | chat | One model call | gen_ai.request.model, gen_ai.provider.name, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens | | execute_tool | One tool invocation | gen_ai.tool.name, gen_ai.tool.call.id, status.code |

The token breakdown for that run:

| Model | Calls | Input tokens | Output tokens | | :---- | ----: | ----: | ----: | | anthropic-claude-4.6-sonnet | 5 | 113,315 | 2,039 | | anthropic-claude-4.5-haiku | 5 | 47,211 | 651 |

Half the model calls went to the smaller model. That is fast model routing, which sends low-effort steps to a cheaper model, and the split is only visible in the trace.

The single agent investigation consumed roughly 160,000 input tokens in total. Each round replays the accumulated context, so cost scales with conversation length rather than with the length of the question.

How do you query agent tool calls with ES|QL?

Tool invocations are the part of agent behavior most worth watching, because that is where the agent touches your data. Every call is one execute_tool span, and the documented query aggregates them directly:

FROM traces-agent_builder.otel-*
| WHERE span.name LIKE "execute_tool *"
| STATS calls = COUNT(*),
        errors = COUNT(*) WHERE status.code == "Error",
        avg_ms = ROUND(AVG(duration) / 1000000.0, 1)
  BY tool = attributes.gen_ai.tool.name
| SORT calls DESC

Across that run the agent leaned on platform.core.execute_esql, with platform.core.generate_esql and load_skill behind it at two calls each. duration is in nanoseconds on the root of the document, which is why the query divides by a million for milliseconds.

Check the companion logs data stream as well. During a different run in the same session, the agent tried to call a tool that was not in its available set, and the attempt was recorded as an exception event in logs-agent_builder.otel-default, correlated to the trace by trace_id:

{
  "trace_id": "3f8b9722dbd371ac4b7ad75e4bed13b6",
  "event_name": "exception",
  "attributes": {
    "exception.type": "toolNotFoundError",
    "exception.message": "Tool \"platform.streams.query_documents\" called but was not available"
  }
}

A blocked tool attempt is audit-relevant, and it does not appear in the trace waterfall. Querying only the traces data stream will miss it.

For aggregate views there is a managed dashboard, installed per space from the same settings panel.

Over a fifteen-minute window covering these runs, it reported 903,914 input tokens, 11,734 output tokens, and 44 LLM requests, with 35 tool spans at a 100% success rate and 0.42 seconds average duration. The dashboard is managed and read-only, so duplicate it to change a panel, and Elastic can still ship improvements to the original.

What OpenTelemetry LLM traces do not capture by default

By default, a trace records structure and cost, not content. Six toggles under Advanced privacy settings control prompts, responses, tool call details, system prompts, real tool and agent names, and real conversation and workflow IDs, and all six are off.

By default, a trace records structure and cost, not content. Select any span in the waterfall and the detail panel says so directly: "No input/output data available for this span."

Identifiers are hashed rather than dropped. The run with the blocked tool call returned conversation be30fb53-d351-4fa1-b5e1-a569816f85d9 from the API, but its spans carry gen_ai.conversation.id: b1141340d0851a46. That lets you group every span belonging to one conversation and compare conversations against each other, without exposing an identifier that ties back to a user's session.

The consequence is that you cannot join traces to conversations on the conversation ID unless you enable real IDs, and you rarely need to. The converse API hands you the correlation key directly:

{
  "conversation_id": "be30fb53-d351-4fa1-b5e1-a569816f85d9",
  "trace_id": "3f8b9722dbd371ac4b7ad75e4bed13b6",
  "model_usage": {
    "llm_calls": 22,
    "input_tokens": 518251,
    "output_tokens": 6423,
    "model": "anthropic-claude-4.6-sonnet"
  }
}

Store that trace_id in your own decision record, and the join works without weakening the privacy defaults. Enable real IDs only when exact response-to-decision attribution is required, and restrict the trace index in the same change.

Auditing AI agent actions beyond the trace

The traces stop at what the agent did, so the records that show who authorised it have to come from somewhere else.

Whose privileges does an AI agent action run with?

Three identities are involved in an agentic pipeline, and each has its own boundary.

| Identity | Runs with | Determined by | How to scope | | :---- | :---- | :---- | :---- | | Agent Builder tools | The privileges of whoever is chatting | The current user, so two people can get different data from the same question | Roles, as described in Agent Builder permissions | | Workflow steps | One stored API key shared by every elasticsearch.* and kibana.* step | The trigger: manual runs use the person who started them, scheduled runs use whoever last saved the workflow | Workflow authorization | | Trace readers | Index-level access, all or nothing | A role grant on traces-agent_builder.otel-* | A role boundary on the trace index pattern |

One consequence of the stored key belongs in any review. Deactivating a user or changing their role does not refresh it, and the workflow keeps running with the privileges it captured until someone saves it again or toggles Enabled off and back on. Revoking an engineer's access does not, by itself, stop workflows that still run as them.

Scope the investigation role to reads only:

POST /_security/role/agent-builder-observability-investigator
{
  "cluster": ["monitor_inference"],
  "indices": [
    {
      "names": ["logs-*", "metrics-*", "traces-*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ]
}

Reading the agent's own traces is a separate grant, needing read and view_index_metadata on traces-agent_builder.otel-*. Keep the two roles apart, because the people who investigate incidents and the people who audit the agent are not always the same people.

Recording who approved an agent action in Elasticsearch

Run the workflow, and it stops at the approval gate, where the reviewer sees the agent's structured output rendered into the request rather than a bare confirmation prompt.

The execution record is detailed. It captures resumedAt, resumedBy, the full resumeInput payload, per-step token usage, and a deep link back to itself.

That execution history is an operational view, not an audit store. The underlying .workflows-events data stream is reserved for system operations and rejects user queries outright, so you cannot run ES|QL across a quarter of decisions, and execution history is subject to retention rather than to your compliance policy.

Write the decision to an index you control:

- name: review
  type: waitForInput
  with:
    message: |
      ## Approve the proposed checkout remediation?

      The evidence query matched {{ steps.collect_evidence.output.hits.total.value }} error events in the last hour.

      Agent classification: {{ steps.investigate.output.structured_output.incident_class }}
      Affected pod: {{ steps.investigate.output.structured_output.affected_pod }}
      Proposed action: {{ steps.investigate.output.structured_output.recommended_action }}
    schema:
      type: object
      properties:
        decision:
          type: string
          enum: ["approve", "decline"]
        reason:
          type: string
          enum: ["supported-by-evidence", "insufficient-evidence", "wrong-target", "unsafe-action"]
        notes:
          type: string
      required: ["decision", "reason"]

- name: record_decision
  type: elasticsearch.index
  with:
    index: "agent-action-audit"
    document:
      "@timestamp": "{{ now | date: '%Y-%m-%dT%H:%M:%S.%LZ' }}"
      "event.action": "agent_recommendation_reviewed"
      "incident.id": "{{ consts.incident_id }}"
      "agent.conversation_id": "{{ steps.investigate.output.conversation_id }}"
      "agent.incident_class": "{{ steps.investigate.output.structured_output.incident_class }}"
      "agent.affected_pod": "{{ steps.investigate.output.structured_output.affected_pod }}"
      "agent.recommended_action": "{{ steps.investigate.output.structured_output.recommended_action }}"
      "agent.evidence_count": "{{ steps.collect_evidence.output.hits.total.value }}"
      "review.decision": "{{ steps.review.output.response.decision }}"
      "review.reason": "{{ steps.review.output.response.reason }}"
      "review.notes": "{{ steps.review.output.response.notes }}"
      "review.responded_by": "{{ steps.review.output.respondedBy }}"
      "workflow.execution_id": "{{ execution.id }}"
      "workflow.executed_by": "{{ execution.executedBy }}"
      "workflow.execution_url": "{{ execution.url }}"

Two details in the workflow snippet above differ from the reference page.

The reviewer payload is nested one level deeper. The docs describe steps.<name>.output.<field>, but the running build returns the submitted values under response, alongside a respondedBy field:

{
  "response": { "decision": "approve", "reason": "supported-by-evidence" },
  "respondedBy": "1506416774"
}

execution.executedBy records who started the run, and respondedBy records who approved the action, which in a human-in-the-loop pipeline are usually different people.

The second detail is the timestamp. {{ now }} renders a JavaScript date string like Sun Jul 26 2026 07:37:11 GMT+0000 (Coordinated Universal Time), which Elasticsearch rejects with failed to parse date field, and execution.startedAt has the same problem. The Liquid date filter fixes it.

The workflow editor also flags steps.review.output.* as an invalid variable before the first run, because the reviewer payload shape is only known once someone responds. The warning clears after the step has real output, and the templates resolve correctly at runtime.

Making the audit data stream append-only

An audit trail the agent's own pipeline can rewrite is not an audit trail. Elasticsearch provides two independent controls, and they compose.

First, write to a data stream rather than an index, because data streams accept appends and nothing else:

PUT _index_template/agent-action-audit
{
  "index_patterns": ["agent-action-audit"],
  "data_stream": {},
  "priority": 500,
  "template": {
    "mappings": {
      "properties": {
        "@timestamp":            { "type": "date" },
        "event.action":          { "type": "keyword" },
        "incident.id":           { "type": "keyword" },
        "agent.conversation_id": { "type": "keyword" },
        "agent.evidence_count":  { "type": "long" },
        "agent.recommended_action": { "type": "keyword" },
        "review.decision":       { "type": "keyword" },
        "review.reason":         { "type": "keyword" },
        "review.responded_by":   { "type": "keyword" },
        "workflow.execution_id": { "type": "keyword" }
      }
    }
  }
}

Second, give the writer create_doc and nothing else, so it can add records but cannot reach for the by-query escape hatches:

PUT _security/role/agent-action-audit-writer
{
  "indices": [
    { "names": ["agent-action-audit"], "privileges": ["create_doc", "auto_configure"] }
  ]
}

Tested against the running cluster, that pair behaves the way an audit store should:

| Attempt as the audit writer | Result | | :---- | :---- | | Append a decision record | 201 Created | | Overwrite a record by ID | 400, only op_type: create is allowed in data streams | | _update_by_query to change a decision | 403, action unauthorized | | _delete_by_query to erase history | 403, action unauthorized | | _search to read the trail back | 403, action unauthorized |

The write-only behaviour in the last row is deliberate. The workflow that writes decisions has no reason to read them, so auditors get a separate read role and the writer stays write-only.

The two controls fail differently, which matters. The 400 comes from the data stream itself and applies to everyone, including a superuser. The 403 rows come from the role, and a superuser could still run them, which is why tamper-resistant retention means shipping records off the cluster the agent's operators administer.

For cluster-level activity, enable Elasticsearch and Kibana security audit logging and forward the logs to a monitoring deployment. On 9.5 xpack.security.audit.enabled became a dynamic cluster setting, so Elasticsearch no longer needs a restart to turn it on, though on orchestrated deployments the logs still have to be shipped somewhere readable.

Query the decision trail with ES|QL

Two runs of the workflow, one approved and one declined, produce two rows you can query alongside everything else in Elastic.

FROM agent-action-audit
| KEEP @timestamp, agent.incident_class, agent.affected_pod, agent.recommended_action,
       agent.evidence_count, review.decision, review.reason, review.responded_by,
       workflow.execution_id
| SORT @timestamp DESC

Both runs saw the same 90 error events and proposed restart-checkout-worker on checkout-worker-1. The first review approved it as supported-by-evidence, and the second declined it as wrong-target, on the argument that restarting the pod hides a pricing-feed problem rather than fixing it.

Because both decisions are structured fields, disagreement between reviews is queryable. You can count rejections per incident class and group them by reason: insufficient-evidence sends you back to the investigation path, and unsafe-action sends you to the workflow and its permission boundary.

AI agent observability limits to design around

Four behaviors are worth designing around, and each is cheaper to handle before the workflows are written.

1. Trace access is index-level, not per user. A space with sensitive conversations needs a role boundary on traces-agent_builder.otel-* rather than a UI setting.
2. The managed dashboard is not installed automatically in a new space. Add it to your space provisioning checklist.
3. The workflow execution carries its own APM traceId. It is not the same trace as the Agent Builder spans its ai.agent step produced, so correlate through the conversation ID or the trace_id returned by the agent rather than expecting one trace to span both.
4. The waitForInput output shape differs from the reference page. The submitted values arrive under response, alongside respondedBy, as covered above.

None of these blocks the pattern.

Where to start with AI agent observability

Turn trace collection on, install the dashboard in the space your agents run in, and open the waterfall for one real conversation. It shows the tool sequence, the model split, and the latency distribution that the answer text does not.

Then pick the single incident class where you already trust the runbook, and add one elasticsearch.index step after its approval gate. An append-only decision record costs one workflow step and answers the three questions a review needs: who approved this, on what evidence, and what happened next.

For the details, see Collect Agent Builder traces, the traces overview dashboard, Agent Builder permissions, workflow authorization, and the waitForInput reference.

Why does AI agent observability need a different model?

Standard application monitoring answers: "Did this request succeed? How long did the database query take?" These questions work because traditional software is deterministic: same input, same output, errors have error codes.

AI agents break that model. A single user request to a Foundry hosted agent might trigger ten LLM calls, five tool executions, two file searches, and an MCP server call, each with its own latency and potential failure mode. The agent can return HTTP 200 and still produce a wrong answer. It can silently loop on the same tool call until it hits a token limit. It can hand off a task to a specialist agent and lose context in transit, and none of that shows up in your error rate or P99 latency.

The questions you actually need to answer in production are different:

• Why did this run consume 40,000 tokens when the average is 3,000?
• Which tool call is responsible for the long tail?
• When the orchestrator handed off to the search agent, did the trace context survive? For those, you need hierarchical trace data that maps the agent's decision tree, not just its I/O boundary.

Foundry Agent Service has strong built-in observability. For Prompt agents, server-side tracing is zero-config: connect an Application Insights resource to your project and Foundry automatically captures inputs, outputs, tool calls, token usage, and latency with no code changes required. For Hosted agents, you add client-side instrumentation to your container code, which is what this post focuses on.

What Foundry's built-in tracing doesn't give you is the ability to run arbitrary queries over the raw OTel data or correlate agent traces with infrastructure telemetry from the rest of your stack. That's where routing those same traces to Elastic adds value. The Instrument → Debug → Evaluate → Optimize loop that Foundry supports in its portal is even more powerful when you can drive it from ES|QL queries against the full trace corpus.

A note on Application Insights: it receives OTel traces but converts them into its own schema on ingestion. Elastic stores the data as-is (resource attributes, semantic convention fields, everything preserved) so you can query exactly what was emitted, without a translation layer between you and the data.

LLM tracing with the OpenTelemetry GenAI conventions

Foundry uses OpenTelemetry's GenAI semantic conventions to structure its traces. Three core span types cover most agent workloads.

Stability note: All gen_ai.* span names and attributes currently carry a Development stability badge in the OpenTelemetry registry; they are pre-1.0 and have already changed once (for example, gen_ai.system was renamed to gen_ai.provider.name). Pin your SDK versions and expect attribute strings to shift before these conventions reach stable status.

invoke_agent wraps the entire agent execution. Every run gets one of these as the root span.

chat is a single LLM API call. It carries gen_ai.provider.name (the provider: openai, anthropic, aws.bedrock), gen_ai.request.model, and gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on every single call, which is what makes per-call token cost attribution possible.

execute_tool is a tool or function invocation triggered by the model, nested under the chat span that requested it.

For multi-agent systems, Microsoft (in collaboration with Cisco Outshift) extended these conventions with additional span types now integrated into Foundry, Agent Framework, LangChain, LangGraph, and the OpenAI Agents SDK:

• execute_task captures task planning and how work is decomposed and distributed across agents.
• agent_to_agent_interaction (child of invoke_agent) traces direct communication between agents.
• agent_planning logs an agent's internal planning steps
• agent.state.management covers context and memory operations

The resulting trace for a multi-step Foundry agent looks like this:

[invoke_agent: research-agent]             ← root: the whole task
  [agent_planning]                         ← agent decides its approach
  [chat: azure]                            ← first LLM call
  [execute_tool: file_search]              ← Foundry built-in tool call
  [chat: azure]                            ← LLM call to reason over results
  [agent_to_agent_interaction: summarizer] ← handoff to specialist agent
    [invoke_agent: summarizer]             ← nested agent execution
      [chat: azure]
  [chat: azure]                            ← final synthesis call

In Elastic, the agent trace renders as a waterfall. You see how long each step took, which one errored, and where the tokens went, including across the agent handoff boundary. That's the data model we're getting into Elastic.

How to instrument a Foundry Hosted agent to emit OTel traces

Foundry Hosted agents run your code in a container managed by Foundry. Because you own the container, you control the instrumentation. Which path you take depends on the framework you're using.

| Framework | Language | Key package | Instrumentation method | |---|---|---|---| | Agent Framework | Python | azure-ai-projects | AIProjectInstrumentor().instrument() | | LangGraph | Python | langchain-azure-ai | AzureAIOpenTelemetryTracer callback | | Custom container | TypeScript / Node.js | @opentelemetry/sdk-node | Manual startActiveSpan |

Trace Agent Framework agents with the Azure AI Projects SDK (Python)

Agent Framework is Microsoft's framework for building Hosted agents on Foundry. It calls the Foundry Responses API for model inference and tool orchestration. To route traces to Elastic, configure the OTel SDK at container startup and add the OTLP exporter pointing at your Elastic endpoint.

Install the packages:

pip install azure-ai-projects azure-identity opentelemetry-sdk azure-core-tracing-opentelemetry opentelemetry-exporter-otlp-proto-http

Configure at startup in your agent container:

import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.telemetry import AIProjectInstrumentor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# OTLPSpanExporter reads OTEL_EXPORTER_OTLP_ENDPOINT and
# OTEL_EXPORTER_OTLP_HEADERS automatically, no need to pass them explicitly
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

# Tracing is off by default — opt in explicitly (experimental preview as of 2026)
# Set AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING=true in your container environment
AIProjectInstrumentor().instrument()

client = AIProjectClient(
    endpoint=os.getenv("AZURE_AI_FOUNDRY_PROJECT_ENDPOINT"),
    credential=DefaultAzureCredential(),
)

Every model call, tool invocation, and agent handoff made through the Responses API now produces a structured OTel span with token usage, model identity, and tool metadata. Note that GenAI tracing in the Azure AI Projects SDK is an experimental preview — you must set AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING=true in your container environment and call AIProjectInstrumentor().instrument() before any agent runs, or no spans are produced.

Trace LangGraph agents with AzureAIOpenTelemetryTracer (Python)

LangGraph is a supported framework for Foundry Hosted agents. Microsoft's langchain-azure-ai package provides an OTel-compliant tracer for LangGraph that emits spans for graph steps, tool invocations, and model calls. Configure the OTLP exporter the same way, then attach the tracer as a callback on each invocation:

pip install langchain-azure-ai langgraph langchain langchain-openai opentelemetry-sdk opentelemetry-exporter-otlp-proto-http azure-identity

from langchain_azure_ai.callbacks.tracers import AzureAIOpenTelemetryTracer
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# OTLPSpanExporter reads OTEL_EXPORTER_OTLP_ENDPOINT and
# OTEL_EXPORTER_OTLP_HEADERS automatically, no need to pass them explicitly
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

# AzureAIOpenTelemetryTracer is Microsoft's supported tracer for LangChain and LangGraph.
# Pass it as a callback when invoking your graph — it uses the active TracerProvider above.
azure_tracer = AzureAIOpenTelemetryTracer(name="my-langgraph-agent")

# app is your compiled LangGraph workflow (e.g. workflow.compile())
config = {"callbacks": [azure_tracer]}
result = app.invoke({"messages": [...]}, config=config)

Trace custom Node.js agents with the OpenTelemetry SDK (TypeScript)

For custom agent architectures in Node.js, wrap your orchestration logic directly with the OTel SDK. The startActiveSpan API handles parent-child nesting automatically. Any span created inside the callback is a child of the current span, giving you the decision tree hierarchy without explicit parent references.

import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { trace, SpanKind, SpanStatusCode } from "@opentelemetry/api";

// OTLPTraceExporter reads OTEL_EXPORTER_OTLP_ENDPOINT and
// OTEL_EXPORTER_OTLP_HEADERS automatically, no need to pass them explicitly
const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter(),
  serviceName: "my-foundry-agent",
});
sdk.start();

const tracer = trace.getTracer("foundry-agent", "1.0.0");

// Wrap your agent's entry point: everything inside becomes a child span
async function runAgent(userMessage: string) {
  return tracer.startActiveSpan(
    "invoke_agent my-agent",
    {
      kind: SpanKind.INTERNAL,
      attributes: {
        "gen_ai.operation.name": "invoke_agent",
        "gen_ai.agent.name": "my-agent",
        "gen_ai.provider.name": "azure",
      },
    },
    async (span) => {
      try {
        const result = await agentLoop(userMessage);
        span.setStatus({ code: SpanStatusCode.OK });
        return result;
      } catch (err) {
        span.recordException(err as Error);
        span.setStatus({ code: SpanStatusCode.ERROR });
        throw err;
      } finally {
        span.end();
      }
    }
  );
}

// Record token usage on every LLM call
async function callAzureOpenAI(messages: Message[]) {
  return tracer.startActiveSpan(
    "chat azure",
    {
      kind: SpanKind.CLIENT,
      attributes: {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": "azure",
        "gen_ai.request.model": "gpt-4o",
      },
    },
    async (span) => {
      const response = await client.chat.completions.create({ model: "gpt-4o", messages });
      span.setAttributes({
        "gen_ai.usage.input_tokens": response.usage.prompt_tokens,
        "gen_ai.usage.output_tokens": response.usage.completion_tokens,
        "gen_ai.response.model": response.model,
      });
      span.end();
      return response;
    }
  );
}

If your Node.js agent calls the OpenAI SDK directly, OpenTelemetry JS also ships an instrumentation-openai auto-instrumentation package as an alternative to the manual spans above (it works with Azure OpenAI clients too). Check its semantic-convention version against what's shown here before relying on it; third-party instrumentations for LangChain and others also exist but vary in how current they are.

How to get your Elastic managed OTLP endpoint and API key

The managed OTLP endpoint (mOTLP) is generally available on both Elastic Cloud Serverless and Elastic Cloud Hosted. It is not available for self-managed, ECE, or ECK deployments. For those, use the EDOT Collector as a gateway instead.

Serverless: Log in to Elastic Cloud → find your project → Manage → Application endpoints, cluster and component IDs → Ingest. Copy the endpoint value. Alternatively, go to Add data → Applications → OpenTelemetry inside your project, which also generates a pre-configured API key.

Elastic Cloud Hosted: Log in → Hosted deployments → Manage → Application endpoints → Managed OTLP. Copy the public endpoint value.

The API key must have event:write privilege on the apm application. The Elastic Cloud quickstart wizard generates this for you. Set two environment variables in your Foundry Hosted agent container:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://<your-motlp-endpoint>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey <your-api-key>"

Note the header format: ApiKey \<key>, not Bearer. And the env var uses = as the separator between header name and value, not :.

Traces sent to the endpoint land in the traces-generic.otel-default data stream by default. In Kibana, find them under Observability → APM → Traces or query them directly with ES|QL against traces-generic.otel-*. No OTel Collector required. No schema translation.

Auto-instrument Foundry agents on AKS with the OpenTelemetry Operator

Foundry Hosted agents support bring-your-own VNet and can run container workloads on AKS. If you're in that configuration, you can skip the SDK-level OTLP configuration entirely. Add this annotation to your pod spec and the OpenTelemetry Operator injects the SDK automatically:

annotations:
  instrumentation.opentelemetry.io/inject-python: "true"

You still set OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS to point at Elastic. The SDK wiring is handled for you.

For agents running in Azure Container Apps, the platform's built-in managed OTel collector routes to Application Insights by default. To also get traces into Elastic, use the SDK-level configuration from the examples above alongside Foundry's built-in observability, or run a sidecar OTel Collector with an OTLP exporter configured for your Elastic endpoint.

How to keep a multi-agent handoff in one trace

If your architecture involves an orchestrator delegating to specialist agents, you want all of that to appear as one connected trace, not N disconnected fragments.

OTel handles this via the W3C TraceContext standard (traceparent and tracestate headers). For HTTP-based inter-agent calls, the SDK propagates these automatically. For queue-based handoffs (Service Bus, Event Hubs), you carry the context in the message itself:

from opentelemetry import propagate, context

# Sending agent: inject the active trace context into the message
carrier = {}
propagate.inject(carrier)

await queue.send_message({
    "payload": task_data,
    "trace_context": carrier,  # {"traceparent": "00-abc123...", "tracestate": "..."}
})

# Receiving agent: restore the trace context before doing any work
incoming_ctx = propagate.extract(message["trace_context"])
with context.use_context(incoming_ctx):
    await process_task(message["payload"])

With this in place, the entire work item appears as one trace in Elastic, from orchestrator through every worker agent, across process boundaries. The waterfall shows exactly where time was spent at each tier.

What Foundry agent traces look like in Elastic APM

Traces arrive as a hierarchical waterfall in the APM UI. For each agent invocation, you get the full span tree: every LLM call, tool execution, and sub-agent call nested under the root invoke_agent span. Every chat span carries gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, so you can see at a glance which model invocations are expensive and which are routine. Errors surface as specific failed spans with full stack traces, not a red line on a latency graph.

You can also query the trace data directly with ES|QL. To find every agent run in the past hour that consumed more than 20,000 input tokens:

FROM traces-generic.otel-default
| WHERE attributes.gen_ai.operation.name == "invoke_agent"
  AND @timestamp > NOW() - 1 hour
| STATS total_input_tokens = SUM(attributes.gen_ai.usage.input_tokens)
    BY trace.id, attributes.gen_ai.agent.name
| WHERE total_input_tokens > 20000
| SORT total_input_tokens DESC

Combining trace structure with token accounting in a single query is what OTel native storage makes possible.

How to sample agent traces without losing errors

For tail-based sampling, add a local OTel Collector as an intermediate hop before the Elastic endpoint. A policy that keeps all error traces, all slow traces, and samples down routine successes is a reasonable starting point:

# Run this Collector between your Foundry Hosted agent and the Elastic endpoint
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 5000 }
      - name: sample-routine
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  otlp/elastic:
    endpoint: "${OTEL_EXPORTER_OTLP_ENDPOINT}"
    headers:
      Authorization: "ApiKey ${ELASTIC_API_KEY}"
    sending_queue:
      enabled: true
      sizer: bytes
      queue_size: 50_000_000
      block_on_overflow: true

If you're sending directly from the SDK without a Collector, start at 100% sampling. Agent trace volume is usually smaller than expected. Evaluate storage costs after a week and tune from there. For most Foundry Hosted agent workloads, the direct SDK path is the right starting point.

How to use agent traces for evaluation and optimization

The Build session demo framed tracing as a four-step production loop: instrument, debug, evaluate, optimize. Debugging is the obvious first payoff: when an agent run fails or produces a wrong answer, the trace shows exactly which span introduced the problem. But the evaluate and optimize steps are where the loop compounds.

With traces flowing into Elastic, you can identify which agent runs produced incorrect or low-quality outputs, then pull those traces directly into an evaluation workflow in Foundry. The trace gives you the full context (the prompt, the tool calls, the model's reasoning path) that an eval needs to score quality and catch regressions. From there, optimization targets are concrete: this tool call is slow, this model is expensive for its output quality, this planning step runs unnecessarily on every request.

Foundry's agent optimizer can use trace data to improve agent instructions automatically. Elastic gives you the query layer to find the traces worth optimizing in the first place.

What you need to start tracing Foundry agents

You need four things: an Elastic Cloud Serverless project (the managed OTLP endpoint is included), your endpoint URL and API key from Project Management → Edit alias, instrumentation that matches your stack (AIProjectInstrumentor for Agent Framework, AzureAIOpenTelemetryTracer from langchain-azure-ai for LangGraph, or the OTel SDK directly for custom container code), and one agent run to verify the trace appears in Kibana's APM view.

The first trace that shows you exactly which tool call caused that 45-second timeout makes the setup worth it.

More resources: Microsoft Foundry Agent Service overview | Set up tracing in Foundry | Tracing integrations by framework | Elastic managed OTLP endpoint docs | OpenTelemetry GenAI semantic conventions | Build 2026 DEM341: Any agent, any cloud

Forking the agent would fix this, but then you own every upstream merge. You could also use manual instrumentation for complete control, but that requires code changes and ongoing upkeep. OpenTelemetry Java extensions give you a cleaner path: a separate JAR the agent loads at startup, giving you precise control over what gets captured and exported, without touching agent or application code.

For example, the following challenges are very common:

• Health check probes are flooding your trace view.
• Span names reflect generic framework patterns rather than your domain operations.
• Some span names or attributes have high cardinality creating noise in your traces.
• Spans are missing attributes relevant to your business logic.
• Baggage headers are propagating to downstream services when they shouldn't.
• Resource attributes that describe your deployment are not automatically captured because they rely on custom environment variables.

Some of those can be solved through configuration, or by using an intermediate OpenTelemetry Collector for processing. However, this also might add complexity to the telemetry pipeline, and you might prefer to solve this at the source, where the data is captured.

What are OpenTelemetry Java extensions

An extension is a JAR file the agent loads at startup. It hooks into the agent's extension points through Java's Service Provider Interface (SPI) mechanism, the same mechanism the agent uses internally.

The extension mechanism works identically with the upstream OpenTelemetry Java agent and with Elastic's OpenTelemetry distribution. You write the extension once and it works with either.

For reference, the upstream extension documentation provides an exhaustive overview of extension points and a few examples.

This post does not aim to provide a complete reference, but focuses on simple use cases you're likely to reach for in production: renaming spans, filtering noisy traces, or propagating context that the agent doesn't cover in your environment.

Extensions also let you modify and extend the agent instrumentation itself. That goes beyond what this post covers. Here are two starting points:

• Modify instrumentation using instrumenter customizers.
• Modify or add instrumentation using the instrumentation module.

Setting up an OpenTelemetry Java extension project

An extension is a standard Java Gradle project with two requirements: the output must be a shadow JAR (a fat JAR with all extension dependencies bundled), and OpenTelemetry dependencies must be declared compileOnly so you don't bundle the SDK itself.

The shadow JAR requirement exists because the agent loads the extension in its own classloader. If you declare a dependency as implementation, it gets bundled and may conflict with the version already in the agent. Using compileOnly keeps those JARs out of the extension JAR entirely.

Here is a minimal build.gradle.kts for a simple extension that does not customize instrumentation and thus relies only on the OpenTelemetry SDK/API.

plugins {
  id("java")
  id("com.gradleup.shadow")
}

repositories {
  mavenCentral()
}

java {
  toolchain {
    languageVersion.set(JavaLanguageVersion.of(8))
  }
}

dependencies {
  // Use BOM to manage OpenTelemetry dependency versions
  compileOnly(platform("io.opentelemetry:opentelemetry-bom:1.64.0"))
  // OpenTelemetry SDK autoconfiguration SPI (provided by agent)
  compileOnly("io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi")
  // OpenTelemetry SDK
  compileOnly("io.opentelemetry:opentelemetry-sdk")
  // Annotation processor for automatic SPI registration
  compileOnly("com.google.auto.service:auto-service:1.1.1")
  annotationProcessor("com.google.auto.service:auto-service:1.1.1")
}

tasks.assemble {
  dependsOn(tasks.shadowJar)
}

Check Maven Central for the latest version of the BOM before starting.

Extensions only depend at compile-time on the OpenTelemetry SDK and the autoconfiguration SPI. The agent provides the rest of the SDK and instrumentation implementation at runtime.

Loading OpenTelemetry Java extensions at runtime

To load an OpenTelemetry Java extension at runtime, you can use the otel.javaagent.extensions system property or OTEL_JAVAAGENT_EXTENSIONS environment variable. The value is a comma-separated list of paths to extension JARs:

java -Dotel.javaagent.extensions=/path/to/my-extension.jar -javaagent:/path/to/opentelemetry-javaagent.jar -jar myapp.jar

The upstream OpenTelemetry Java agent also lets you embed extensions directly into the agent JAR to simplify deployment.

Filtering and renaming spans with OpenTelemetry Java extensions

You can modify spans in two ways:

• Using a SpanProcessor that is called synchronously when the span starts or ends.
• Using a SpanExporter that is called asynchronously when the span is exported.

Rename spans with a SpanProcessor

SpanProcessor.onStart receives a ReadWriteSpan, which means you can call span.updateName() before the span is exported. This is the right hook for renaming based on attributes that are available at span start.

public class OperationRenamingSpanProcessor implements SpanProcessor {

  @Override
  public void onStart(Context parentContext, ReadWriteSpan span) {
    String operation = span.getAttribute(AttributeKey.stringKey("app.operation"));
    if (operation != null) {
      span.updateName(operation);
    }
  }

  @Override
  public boolean isStartRequired() { return true; }

  @Override
  public void onEnd(ReadableSpan span) {}

  @Override
  public boolean isEndRequired() { return false; }

  @Override
  public CompletableResultCode shutdown() { return CompletableResultCode.ofSuccess(); }

  @Override
  public CompletableResultCode forceFlush() { return CompletableResultCode.ofSuccess(); }
}

Register the SpanProcessor via AutoConfigurationCustomizerProvider, composing it with whatever processor you have already configured:

@AutoService(AutoConfigurationCustomizerProvider.class)
public class RenamingCustomizerProvider implements AutoConfigurationCustomizerProvider {

  @Override
  public void customize(AutoConfigurationCustomizer customizer) {
    customizer.addTracerProviderCustomizer(this::configureSdkTracerProvider);
  }

  private SdkTracerProviderBuilder configureSdkTracerProvider(
      SdkTracerProviderBuilder tracerProvider, ConfigProperties config) {
    return tracerProvider.addSpanProcessor(new OperationRenamingSpanProcessor());
  }

}

The modify-span EDOT Java extension example provides a complete implementation.

Filter spans with a SpanExporter

A SpanExporter wrapper lets you modify or drop spans before they leave the process. This works well for known noisy endpoints like health checks.

public class FilteringSpanExporter implements SpanExporter {

  private final SpanExporter delegate;

  public FilteringSpanExporter(SpanExporter delegate) {
    this.delegate = delegate;
  }

  @Override
  public CompletableResultCode export(Collection<SpanData> spans) {
    List<SpanData> filtered = new ArrayList<>();
    for (SpanData span : spans) {
      if (!"GET /health".equals(span.getName())) {
        filtered.add(span);
      }
    }
    return delegate.export(filtered);
  }

  @Override
  public CompletableResultCode flush() { return delegate.flush(); }

  @Override
  public CompletableResultCode shutdown() { return delegate.shutdown(); }
}

Register the FilteringSpanExporter via addSpanExporterCustomizer:

customizer.addSpanExporterCustomizer((existing, config) -> new FilteringSpanExporter(existing));

The modify-span EDOT Java extension example provides a complete implementation.

The approach has two limitations:

• This won't discard any child span that may have been created, for example, if the healthcheck calls the database.
• Spans filtered at the exporter have already passed through the full processor pipeline and occupied buffer space in the batch processor.

If you're dropping a large fraction of your traffic at this stage, a custom Sampler (shown below) is more efficient because it drops spans before any processing happens and also filters out child spans. Also, when using declarative configuration, the rule-based sampler lets you implement filtering on rules using only configuration.

Adding custom resource attributes with a ResourceProvider

Resource attributes describe what's running: the service name, its version, the host. A ResourceProvider lets you attach additional attributes that the agent doesn't know about, such as deployment metadata your platform injects through environment variables.

The example below uses environment variables, but it could also be a configuration file, a cloud metadata service, or any other source available to the agent at startup.

Because the SDK initialization is synchronous, when querying an external service like a metadata endpoint, this can make the agent (and thus the application) startup slower. If possible, prefer checking environment variables and local config first before calling an external service.

@AutoService(ResourceProvider.class)
public class DeploymentResourceProvider implements ResourceProvider {

  @Override
  public Resource createResource(ConfigProperties config) {
    AttributesBuilder attributes = Attributes.builder();

    String region = System.getenv("DEPLOY_REGION");
    if (region != null) {
      attributes.put(AttributeKey.stringKey("deployment.region"), region);
    }

    String buildVersion = System.getenv("BUILD_VERSION");
    if (buildVersion != null) {
      attributes.put(AttributeKey.stringKey("build.version"), buildVersion);
    }

    return Resource.create(attributes.build());
  }
}

Attributes from a ResourceProvider merge with the agent's own resource. When two providers supply the same key, the one with the higher order() value wins. The agent's built-in providers use order 0, so overriding order() to return a positive integer gives your provider priority.

The resource-attribute EDOT Java extension example provides a complete implementation.

Custom sampling in OpenTelemetry Java

When filtering at the exporter is too late or too expensive, implement a Sampler directly. The sampler runs before any span processing, so dropped spans never touch the batch buffer.

However, the sampling decision can only rely on attributes that are provided when the span starts. For example, the status code of an HTTP response can't be used as it is only available when the span ends.

The key detail: wrap the existing sampler rather than replacing it. That way, your logic composes with whatever you configured, and parent-based decisions from an upstream service are still respected.

public class HealthCheckSampler implements Sampler {

  private final Sampler delegate;

  public HealthCheckSampler(Sampler delegate) {
    this.delegate = delegate;
  }

  @Override
  public SamplingResult shouldSample(
      Context parentContext,
      String traceId,
      String name,
      SpanKind spanKind,
      Attributes attributes,
      List<LinkData> parentLinks) {
    if (spanKind == SpanKind.SERVER && name.contains("health")) {
      return SamplingResult.create(SamplingDecision.DROP);
    }
    return delegate.shouldSample(parentContext, traceId, name, spanKind, attributes, parentLinks);
  }

  @Override
  public String getDescription() {
    return "HealthCheckSampler{" + delegate.getDescription() + "}";
  }
}

Register the HealthCheckSampler via addSamplerCustomizer, which gives you both the existing sampler and the resolved config:

customizer.addSamplerCustomizer((existing, config) -> new HealthCheckSampler(existing));

Community extensions in opentelemetry-java-contrib

The opentelemetry-java-contrib repository contains several community-maintained extensions.

Some of them are already included in the OpenTelemetry Java agent (and inherited in the Elastic distribution), but are opt-in:

• azure-resources
• aws-resources
• gcp-resources
• cloudfoundry-resources
• baggage-processor

Most Elastic distribution features exist as extensions in the contrib repository, so you can use them with the upstream agent in a vendor-neutral way.

• inferred-spans
• span-stacktrace

Further reading and extension examples

The upstream extension examples cover additional extension points not shown here, including custom propagators, ID generators, and ignored-type configurers.

The Elastic baggage example shows the filtering propagator for baggage running end-to-end with a two-service application, it also demonstrates custom instrumentation to add baggage without modifying the application code.

This post covered the project setup and the patterns most likely to come up in production. Both links above go deeper: the upstream examples add extension points not covered here, and the baggage example shows a complete two-service implementation you can run locally.

The Kubernetes OpenTelemetry Assets package now ships four Kubernetes SLO templates for Deployments, StatefulSets, DaemonSets, and Jobs on those OTel metrics. If you already followed Part 1 and have the dashboards and alert rules, create an SLO from a template and you get SLI, remaining budget, and burn rate without new instrumentation.

The diagram above extends the stack from Part 1. The Kubernetes OpenTelemetry Assets package (kubernetes_otel) 2.3.0 includes:

• Dashboards designed for drill-down (Part 1)
• Alert rule templates that fire on known bad states (Part 1)
• ML anomaly detection jobs with workload baselines (Part 1)
• SLO templates for rolling 30-day budgets (this post)

All four use the same OTel metrics. Burn rate alerts on an SLO send you back into Overview, Workloads, and Deployment Details when the number alone is not enough.

Why Kubernetes observability needs SLO monitoring alongside alerts

Part 1 built the reactive stack for the engineer who gets paged at 3 AM. SLOs serve the planning conversation on a 30-day horizon: Are we meeting our reliability commitments? They give platform and engineering leaders a number for prioritisation: how much error budget remains and which workload is burning it fastest. The table later in this post maps each SLO template to its Part 1 alert counterpart.

The SLO templates in this post are part of the Kubernetes OpenTelemetry Assets package (kubernetes_otel). Install the Kubernetes OpenTelemetry Assets package and confirm your cluster is already sending Kubernetes metrics through OpenTelemetry (the same pipeline from Part 1). No additional instrumentation is required.

Four SLO templates for Kubernetes Deployments, StatefulSets, DaemonSets and Jobs

In Integrations → Kubernetes OpenTelemetry → Assets, enable any of the four templates below. Names match Kibana; each includes the [Kubernetes OTel] prefix in the UI.

| Template | Rolling objective | Package description | | ----------------------------------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Deployment Replica Availability 99.5% Rolling 30 Days | 99.5% / 30d | Tracks Deployment availability from OTel metrics: 99.5% of intervals should have each Deployment at its desired replica count. When k8s.deployment.available < k8s.deployment.desired, the workload has fewer healthy replicas than configured. | | StatefulSet Replica Availability 99.5% Rolling 30 Days | 99.5% / 30d | Same pattern for StatefulSets, where pod identity and ordering matter for databases, queues, and caches. | | DaemonSet Scheduling Availability 99.0% Rolling 30 Days | 99.0% / 30d | Tracks whether each DaemonSet runs on all eligible nodes. Covers node-level agents such as log collectors, monitoring, security, and CNI plugins. | | Job Completion Success Rate 99.0% Rolling 30 Days | 99.0% / 30d | Tracks batch Jobs (ETL, backups, pipelines, scheduled tasks) completing without failed pods over the rolling window. |

Each is a timeslice-metric SLO: Elastic marks every five-minute window good or bad, then rolls those results into a 30-day rolling objective per namespace and workload.

Reliability is scored at two levels. Each five-minute slice gets one verdict: Elastic aggregates OTel metrics in that window, evaluates the template equation, and compares the result to the metric threshold. For Deployments, that is sum(available) / sum(desired) >= 1. At a ~30-second OTel scrape cadence, that is roughly ten measurements per slice, and the slice passes or fails on the aggregated result. The SLO target (99.5% or 99.0%) is the share of slices that must pass across the rolling window. Over 30 days at five-minute slices, that is 8,640 possible slices per workload (30 × 24 × 12). After you create an SLO from a template, the SLO detail view shows how many slices passed and how much error budget remains.

At 99.5%, a workload can miss roughly 43 of those slices (~3.6 hours of bad slices) before breach. At 99.0%, about 86 slices (~7.2 hours).

How to set SLO targets by Kubernetes workload type

We picked defaults per workload type, not one number for the whole cluster.

Deployments and StatefulSets at 99.5%: We considered 99.9% (~43 minutes per month), which fits a single critical API or a formal SLA buffer. For a default integration template across many Deployments, 99.5% (~3.6 hours) leaves room for normal rollout churn: a 20-minute bad image tag is roughly half the monthly budget at 99.9%, but a small fraction at 99.5%. Tune per workload; payment paths often warrant 99.9% or higher.

DaemonSets and Jobs at 99.0%: We considered 99.5% for DaemonSets, but node additions, replacements, and rolling updates often leave ready_nodes below desired_scheduled_nodes for several minutes per event. At 99.5%, that normal platform churn would burn error budget on infrastructure agents (log collectors, monitoring, CNI) as if they were user-facing outages. 99.0% (~7.2 hours) absorbs that lifecycle noise. Jobs get 99.0% for a different reason: a failed ETL run usually hurts data freshness, not live request availability, and failures can sit unnoticed until downstream teams see stale reports.

Deployment replica availability (99.5%)

Metric:     sum(k8s.deployment.available) / sum(k8s.deployment.desired) >= 1
Target:     99.5% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.deployment.name

When available < desired, the application runs fewer healthy replicas than configured. Failed rollouts, crash loops, and node loss all show up here. 99.5% leaves roughly 3.6 hours of degradation per deployment per month before breach.

Grouping by namespace and deployment name creates one SLO per workload. A cluster-wide average would let a healthy frontend mask a burning checkout. Linked dashboards (Overview and Workloads) connect the SLO view to investigation context in one click; from Workloads you drill into Deployment Detail for the failing deployment.

StatefulSet replica availability (99.5%)

Metric:     sum(k8s.statefulset.ready_pods) / sum(k8s.statefulset.desired_pods) >= 1
Target:     99.5% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.statefulset.name

When ready_pods < desired_pods, the StatefulSet reports fewer Ready replicas than configured. Ordered rollouts, stuck pods, and node loss show up here too. Rollouts proceed in order, and each pod keeps its name and volume, so a missing replica can stay below desired longer than a stateless pod would.

Grouping by namespace and StatefulSet name avoids a healthy workload masking another that is burning the SLO budget. 

DaemonSet scheduling availability (99.0%)

Metric:     sum(k8s.daemonset.ready_nodes) / sum(k8s.daemonset.desired_scheduled_nodes) >= 1
Target:     99.0% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.daemonset.name

DaemonSets run node-level infrastructure: log collectors, monitoring agents, security agents, and network plugins. When ready_nodes < desired_scheduled_nodes, an eligible node lacks a Ready pod, which can leave that node without logs or metrics from that agent. Rolling updates and new nodes drive most gaps; pods that never become Ready show the same signal. Cordoned nodes often still run DaemonSet pods. 99.0% (~7.2 hours per month) reflects that churn. 

Group by namespace and DaemonSet name so a healthy fluentd does not mask a broken node-exporter on the same SLO budget.

Job completion success rate (99.0%)

Metric:     max(k8s.job.failed_pods) <= 0
Target:     99.0% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.job.name

Jobs cover batch workloads: ETL pipelines, backups, database migrations, and scheduled reports. When failed_pods > 0, at least one pod created by the Job reached the Failed phase. Application errors, timeouts, and missing dependencies drive many failures; when retries reach the configured backoffLimit, Kubernetes marks the Job as Failed. Missed runs often surface as stale or delayed data, not as a serving outage. 99.0% (~7.2 hours per month) reflects that occasional batch failure is less time-sensitive than a Deployment or StatefulSet breach. 

How do SLOs and alerts work together in Kubernetes observability?

The SLO templates and the alert rules from Part 1 serve different people asking different questions at different times.

| SLO Template | Alert rule (Part 1) | Failure consequence | Monthly budget (30d) | | --------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------- | ------------------------ | | Deployment Replica Availability | Deployment below the desired replicas | Reduced throughput, degraded UX | ~3.6 hours at 99.5% | | StatefulSet Replica Availability | No dedicated rule. Covered by CrashLoopBackOff / OOMKilled at pod level | Split-brain risk, degraded durability | ~3.6 hours at 99.5% | | DaemonSet Scheduling Availability | Pod stuck in Pending / node disk pressure | Blind spots: unmonitored nodes and gaps in node-level coverage | ~7.2 hours at 99.0% | | Job Completion Success Rate | CrashLoopBackOff / OOMKilled | Stale or incomplete data | ~7.2 hours at 99.0% |

Alert rules answer: Is something broken right now? They fire within minutes, page the on-call engineer, and expect immediate action.

SLO templates answer: Are we meeting our reliability commitments over time? They accumulate signal across weeks and turn prioritisation debates into a number tied to remaining budget.

From incident to error budget burn: a Kubernetes walkthrough

A deployment drops from 3/3 to 2/3 available replicas during a rolling update. The new pod fails its readiness probe. Here is what happened in our test cluster, from dashboard signal through alert, SLO impact, and root cause.

Rollout begins. The Deployment dashboard shows available: 2, desired: 3. The Part 1 Deployment unavailable replicas rule has a 5-minute grace period, so on-call is not paged yet during a short rollout gap. The Deployment Detail view for web-frontend shows available replicas dropping while desired stays at 3. The Deployment replicas over time chart marks where the rollout started to fail.

Alert fires, then root cause. After the grace period, the alert rule triggers: Deployment unavailable replicas. The on-call engineer opens the Workloads dashboard, finds web-frontend at available: 2, desired: 3, and drills into Deployment Detail. The replicas-over-time chart confirms when availability dropped. 

In Discover, filter Kubernetes events for that pod with k8s.object.name: "web-frontend-796fcd55b9-jmlkh". The event stream shows ImagePullBackOff and Back-off pulling image "nginx:nonexistent-tag-999". The rollout references an image tag that does not exist.

Rollback and recovery. The engineer runs kubectl rollout undo deployment/web-frontend. Replicas return to 3/3.

How two rollouts consumed 88% of a 30-day error budget

The rollback fixed availability. The SLO still counted the day's failures.

Two rollout failures left web-frontend with 39 failed timeslices where available < desired. That consumed 88.0% of the 30-day error budget. The SLI still read 99.56%, above the 99.5% target, but only 12% of the monthly allowance remained.

The burn rate alert fired next, even though replicas were healthy again. Over the past day the deployment consumed budget at 26× the rate a 99.5% SLO can sustain long term. Each failed 5-minute timeslice uses roughly 2.3% of the monthly budget (about 43 failures allowed per 30 days). Thirty-nine failures across two rollouts is worth a reliability review, not a one-line postmortem. The burn rate alert often matters more than the raw SLI mid-month because it fires while you still have budget left to spend deliberately.

Try it yourself: trigger an error budget burn on a test Deployment

If you already have Kubernetes OpenTelemetry Assets installed, the SLO templates live under Integrations → Kubernetes OpenTelemetry → Assets.

Create a Deployment replica availability SLO for the deployment you use below. Open the SLO and note the baseline: current SLI, remaining error budget, and existing timeslice history.

Create an isolated namespace and a small deployment so the exercise does not affect production workloads. Wait a few minutes for the OTel collector to scrape metrics before you create the SLO.

kubectl create namespace blog-demo
kubectl create deployment web-frontend --namespace blog-demo --image=nginx:latest --replicas=3

Trigger a bad rollout with a non-existent image tag:

kubectl get deployment web-frontend -n blog-demo
kubectl set image deployment/web-frontend nginx=nginx:nonexistent-tag-999 --namespace blog-demo

Within a few minutes a new pod enters ImagePullBackOff, available replicas drop below desired, and the SLO records failed timeslices. Roll back to recover:

kubectl rollout undo deployment/web-frontend -n blog-demo

Refresh the SLO view. You should see new failed timeslices in the 30-day history and a reduction in remaining error budget.

One failed timeslice consumes about 2.3% of the monthly error budget at 99.5%. Repeat that across deployments in a week and the burn rate alert becomes the prioritisation signal.

When you are done, delete the test namespace with:

kubectl delete namespace blog-demo

What's next: from SLO monitoring to agentic remediation

Alerts tell you replicas dropped. SLOs tell you how much monthly budget that cost. In the walkthrough above, the same ImagePullBackOff showed up in Deployment Detail, the unavailable-replicas alert, and failed timeslices on the replica-availability SLO, all from the OTel pipeline you installed in Part 1. The SLI still read 99.56% while 88% of the monthly error budget was gone.

Part 1 closed by previewing Agentic Investigations: investigation workflows that run when an alert fires, with skills, tools, and MCP views. This post adds the SLO layer on those same metrics so you can quantify reliability debt before automating runbooks. A follow-up post will cover that agentic workflow and propose remediations you review before applying.

Which remediations would you trust a workflow to suggest on a Kubernetes incident, and which would you keep manual? Join the Elastic Community discussion.

What changed in Elastic Agent 9.3: A native OTel Collector under the hood

Previously, Elastic Agent acted as a supervisor process, spinning up Beats-based subprocesses, such as Filebeat or Metricbeat. From 9.3 onward, that architecture has been replaced. Elastic Agent itself is now built on the EDOT Collector, turning it into a first-class OTel Collector under the hood while preserving its original functionality.

Key benefits of this architectural shift include:

• Reduced footprint: Fewer subprocesses mean significantly less memory overhead and a simpler deployment model. In future releases, this footprint will be even further reduced.
• Unified telemetry pipeline: Logs and metrics flow through a single, standards-based OTel pipeline, as do traces.
• Ecosystem interoperability: Elastic Agent can now receive data from any OTel-compatible source out of the box. It can also be configured to emit to OTel-compatible destinations.
• Aligned with the OTel ecosystem: As the OTel ecosystem matures with new receivers, processors, and exporters, Elastic Agent deployments gain access to those capabilities automatically.

When you deploy or update Elastic Agent from version 9.3 onward, you're deploying an OpenTelemetry Collector. EDOT is the technology foundation; Elastic Agent is the product.

How existing Beats configurations run inside the OTel Collector pipeline

Elastic has introduced Beats Receivers, which are Beat inputs and processors that execute natively inside the new OTel Collector pipeline. For your teams and customers, this means:

• Existing elastic-agent.yml configurations require no modification.
• Fleet-managed agents automatically translate policy configurations into OTel format internally.
• All integrations, dashboards, ingest pipelines, and alerting rules continue to function exactly as before.
• Data written via Beats Receivers lands in the same data streams as always.

Upgrading to 9.3 is transparent because it uses the same inputs and produces the same outputs.

Running Beats and OTel Collector pipelines in one Elastic Agent

The new Elastic Agent is a collector capable of simultaneously running traditional Beats-based collections alongside native OTel pipelines, all in a single deployment. One agent policy can collect Elastic Common Schema–schematized (ECS-schematized) data via Beats Receivers and ingest native OpenTelemetry Protocol (OTLP) data from OTel-instrumented applications and infrastructure. This same agent policy can apply OTel processing stages across all telemetry before export.

OTel integrations from Elastic's catalog can be added to any agent policy. When native OTel data is ingested, Elastic automatically installs the relevant dashboards and alerts, in addition to necessary content packs, without any manual setup.

What's the relationship between Elastic Agent and EDOT?

You may be familiar with EDOT, the Elastic Distribution of OpenTelemetry Collector, as a stand-alone product. With this architectural change, EDOT is the technology foundation that now powers Elastic Agent, not a separate product that users need to track or deploy independently.

Going forward, Elastic Agent is the supported, Fleet-manageable, fully featured product. A stand-alone deployment remains available for specific niche scenarios (environments where the full version of Elastic Agent cannot be installed), but it isn't the recommended path for the vast majority of users.

Elastic Agent deployment options: Fleet-managed vs. stand-alone

| | Fleet-managed Elastic Agent | Stand-alone Elastic Agent | | :------------------------- | :-----------------------------: | :-----------------------------------------------------------------------------------------------------: | | Fleet lifecycle management | Yes | Can enroll into Fleet in-field without reinstallation | | Beats Receivers | Yes | Yes | | Elastic Defend | Yes | No | | Cloud Security | Yes | No | | Profiler support | Yes | No | | OTel-native pipeline | Yes | Yes | | Best for | Most deployments | Environments where full Elastic Agent cannot be installed or management is handled by other tools |

Do I need to change anything when upgrading to Elastic Agent 9.3?

For users running Elastic Agent today, upgrading to 9.3 requires no changes to configurations or integrations, and no changes to workflows. For customers evaluating OTel adoption, Elastic Agent now provides a fully supported, production-ready OTel Collector with Fleet management and rich integrations, along with Elastic's full support matrix, and none of this requires a separate OTel deployment.

With Elastic Agent 9.3, Elastic's data collection is fully OpenTelemetry-native. Elastic Agent is now an OpenTelemetry Collector. Everything you have today still works, and you also get all the capabilities of OTel.

The gain is in where the intermediate set lives. When you ask what the saturated hosts logged, the list of saturated hosts is computed and consumed inside Elasticsearch. Six host names or 500 trace IDs never land in a clipboard or in an AI agent's context window, and the set is recomputed from current data every time the query runs.

That changes the unit of investigation. You go from "one slow request stalled on a lock" to "most of the slowest requests did," and only the second answer tells you which team to page. It matters more the further apart your signals are: in many observability stacks, logs, metrics, and traces live in three separate systems, each with its own query language, its own time picker, and its own idea of what a host is, so the same question has to be asked two or three times and the answers joined by hand.

In this post we walk through four investigations, each of which is self-contained. For every one of them, we set out the scenario that started it, the query that answers it, the table it returns, and a note on the difficulties you run into when you try to get the same answer any other way.

| Pattern | Starts from | Question it answers | |---|---|---| | Metrics to logs | CPU saturation | Which error patterns show up only on the saturated hosts? | | Logs to metrics | Error logs | Do the erroring hosts look any different from the healthy ones? | | Traces to logs | Slow spans | What did every service log during those specific requests? | | All three signals | Pod memory pressure | Which log lines sit behind the requests that failed under that pressure? |

The data was collected with OpenTelemetry and lands in the logs-*.otel-*, traces-*.otel-*, and metrics-*.otel-* data streams, where fields keep their semantic convention names rather than being rewritten into another schema. The correlation pattern works just as well on Elastic Agent integrations, though the queries need translating rather than just renaming: ECS carries log severity as the text field log.level instead of a numeric severity_number, the System integration reports CPU as separate system.cpu.*.pct fields instead of one metric with a state dimension, and APM records durations in microseconds. All of it lands in the same cluster either way, which is the part the subquery depends on.

In Discover, the time picker already applies the range, so the examples below omit an explicit @timestamp filter. Outside Discover, add a filter by time yourself, either with literal timestamps in the query or with ?_tstart and ?_tend in the query and values in the params array of your _query request. Every result below comes from a one hour window over a synthetic fleet of 300 hosts.

Metrics to logs: what are the saturated hosts complaining about?

An infrastructure alert tells you a handful of hosts in a fleet of a few hundred sat above 90% CPU over the last hour. That tells you which hosts are hot and nothing about why. The question worth answering is whether those hosts share a failure mode, or whether they are busy for unrelated reasons and the alert is a coincidence.

FROM logs-*.otel-*
| WHERE severity_number >= 17
  AND resource.attributes.host.name IN (
      TS metrics-hostmetrics.otel-*
      | WHERE attributes.state == "idle"
      | STATS idle = AVG(AVG_OVER_TIME(metrics.system.cpu.utilization))
          BY resource.attributes.host.name
      | WHERE idle < 0.1
      | KEEP resource.attributes.host.name
    )
| STATS errors = COUNT(*), hosts = COUNT_DISTINCT(resource.attributes.host.name)
    BY pattern = CATEGORIZE(body.text), service = resource.attributes.service.name
| SORT errors DESC

The two halves of the query map onto the two halves of the question. The subquery works out which hosts were saturated, averaging CPU utilization per host and keeping the ones that averaged under 10% idle, which is another way of saying above 90% busy for the window. The outer query then works out what those hosts were complaining about, pulling their error logs and using CATEGORIZE to collapse thousands of individual lines into a handful of error classes.

Two choices in there are worth pausing on.

The subquery uses TS rather than FROM because a host does not report one CPU number. It reports a separate time series per CPU state, and per logical core as well if your collector is configured to break them out, so the reduction has to happen in two stages. AVG_OVER_TIME collapses each series to a single value first, and the outer AVG then combines those into one figure per host.

Naming that inner function matters more than it looks. Write AVG(metrics.system.cpu.utilization) on its own and TS supplies LAST_OVER_TIME for you, averaging each series' final sample rather than its average over the window. In this dataset that one substitution moves a host from 7% idle to 10% idle, which is the difference between appearing in the results and not. Querying metrics with the TS command goes into the two aggregation phases in more depth.

The log filter tests severity_number (17 is the ERROR floor on the OpenTelemetry scale) rather than the severity text, because the numeric scale is fixed by the spec while the text is whatever the emitting library decided to write. That is not a hypothetical distinction here: the error logs in this cluster carry four different labels, including SEVERE from a Java service. Matching on the text alone returns 2,754 of checkout's errors and misses the billing service entirely, while the numeric filter returns all 5,910 of them and keeps billing too.

When you run the query in Discover, the result is a short table of error classes, scoped to the saturated hosts:

The subquery returned six saturated hosts, and on all six the same service is timing out against an upstream dependency and draining its connection pool. The hosts column is what lets you set the other two rows aside without opening anything: the certificate errors reach only two of the six, and the gateway declines amount to five lines in an hour. Neither tracks the cohort the way the checkout patterns do.

Having all three signals in one store already removed the exports from this investigation. The subquery removes the step after that, and closing that last gap matters more than it sounds. By the time you have read six host names off a chart and typed them into a log search, the set has moved: a host that crossed the threshold a minute ago is missing from your list, and one that has since recovered is still in it. Here the host list is derived from current data on every run, so re-running the query during an incident gives you the current cohort.

The stale list is only half of it. A correlation done by hand exists only in the head of the person who did it, so nobody else can check it, save it, or run it again tomorrow.

Logs to metrics: do the erroring hosts look different from the healthy ones?

Filtering metrics by a log-derived host set answers the opposite question. The payments service is throwing errors on some hosts and not others, and you want to know whether resource pressure explains the split before you start reading deploy history.

That is a comparison, so the query needs both cohorts. FORK runs two branches over the same input, and IN and NOT IN against the same log-derived host set divide the fleet between them.

TS metrics-hostmetrics.otel-*
| WHERE attributes.state == "idle"
| STATS idle = AVG(AVG_OVER_TIME(metrics.system.cpu.utilization))
    BY host = resource.attributes.host.name
| EVAL busy = 1 - idle
| FORK
    ( WHERE host IN (
        FROM logs-*.otel-*
        | WHERE severity_number >= 17
          AND resource.attributes.service.name == "payments"
          AND resource.attributes.host.name IS NOT NULL
        | STATS errors = COUNT(*) BY resource.attributes.host.name
        | KEEP resource.attributes.host.name )
      | EVAL cohort = "logging errors" )
    ( WHERE host NOT IN (
        FROM logs-*.otel-*
        | WHERE severity_number >= 17
          AND resource.attributes.service.name == "payments"
          AND resource.attributes.host.name IS NOT NULL
        | STATS errors = COUNT(*) BY resource.attributes.host.name
        | KEEP resource.attributes.host.name )
      | EVAL cohort = "no errors" )
| STATS hosts = COUNT(*), mean_busy = AVG(busy), busiest_host = MAX(busy)
    BY cohort

The query reads top to bottom as three stages. The metrics query runs first and reduces the whole fleet to one busy figure per host. FORK then splits that fleet in two using the same log query in both branches, separating the hosts that appear in it from the hosts that do not. The final STATS summarizes each group, so both cohorts come back as two rows of one table, measured the same way over the same window.

This query is longer than the others, and three parts of it are less obvious than they look.

The natural way to label the two cohorts would be EVAL cohort = CASE(host IN (...), "erroring", "healthy"), and ES|QL rejects it. In 9.5 an IN subquery has to be a top-level predicate in a WHERE condition rather than an argument to a scalar function, which is why the split happens at the command level with FORK.

The STATS ... BY inside each subquery looks redundant, since KEEP alone would return the same host names. It is not: without it, the subquery returns one row per matching log document instead of one row per host, and those rows are all held in memory for the outer query to filter against. Aggregating first turns millions of rows into a few hundred host names.

The IS NOT NULL filter guards the sharpest edge here, and this dataset is a live example rather than a hypothetical. NOT IN follows SQL null semantics, so a single null in the subquery result makes the predicate match nothing at all. Five of the payments error logs in this cluster came through a sidecar that dropped the host name. Remove that one line from both branches and the query still succeeds, but it returns a single row: the nulls quietly delete the entire 286-host "no errors" cohort, and what is left looks like a perfectly plausible answer to a different question.

Run the query in Discover and you get two rows, one per cohort:

CPU does not explain the split, and the numbers say so twice. The 14 erroring hosts run slightly cooler on average than the 286 quiet ones, and the busiest machine among them averaged 50% over the hour while the quiet cohort contains a host that averaged 95%. Whatever is failing on those 14, they had headroom the entire time, and deploy history is a better place to spend the next ten minutes.

A negative result like this is worth as much as a positive one, and it is usually the one people skip. Getting it the long way means running the metrics query twice against two hand-built host lists and lining the numbers up afterwards, which is enough friction that the check often just does not happen. Both cohorts here come from the same log query in the same execution, over identical time windows, so there is nothing to reconcile and no reason not to check.

Traces to logs: what did every service log during the slow requests?

A service level objective (SLO) burn alert fires on checkout latency. Tracing gives you the slow requests and their spans, and the next question is what the services involved were writing to their logs while those specific requests were in flight.

The trace ID is the join key, and there are far too many of them to move by hand.

FROM logs-*.otel-*
| WHERE trace_id IN (
      FROM traces-*.otel-*
      | WHERE kind == "Server"
        AND resource.attributes.service.name == "checkout"
        AND name == "POST /api/orders"
        AND duration > 2000000000
      | SORT duration DESC
      | LIMIT 500
      | KEEP trace_id
    )
| STATS lines = COUNT(*), traces = COUNT_DISTINCT(trace_id)
    BY pattern = CATEGORIZE(body.text),
       service = resource.attributes.service.name,
       severity_text
| SORT traces DESC

The subquery answers "which requests were slow." It looks at the inbound request span for the checkout endpoint rather than the client and internal spans beneath it, then keeps the 500 slowest requests over two seconds. Durations are recorded in nanoseconds, which is why the threshold has so many zeros.

The outer query answers "what got logged while they were running," gathering every log line that shares one of those trace IDs and grouping them into patterns.

Counting distinct traces per pattern is what makes the output readable. A log pattern that appears 30,000 times across four traces is one chatty request, while a pattern that shows up in 470 of the 500 slowest traces is a property of being slow.

From the results above, the top row is there by construction and can be set aside: checkout writes one order submitted line per order, so it appears in all 500 traces and says nothing about why these particular 500 were slow.

From the results above, the third row is the answer, and it points at a service nobody was looking at. A lock wait timeout in inventory, two hops downstream from where the alert fired, shows up in 470 of the 500 slowest requests, and the 947 lines behind those 470 traces mean a good share of them retried more than once. The payment gateway declines are real failures, and at 19 traces out of 500 they are not what is burning the SLO.

Done by hand, this means opening slow traces one at a time and reading the correlated logs for each, which is tedious at ten traces and nobody's idea of a plan at 500. When the spans and the logs are held in different systems, every trace you check is a copied ID and a context switch, and the sample size you can afford drops to about three. Three traces is enough to form a theory and not enough to test one. Treating the slow requests as a population is what turns "this trace had a lock wait" into "470 of the 500 slowest requests had a lock wait," and that difference decides whether you page the inventory team.

All three signals: from pod memory pressure to the log lines behind the failures

IN subqueries nest, so the pattern extends to as many signal types as the question needs.

A node pool starts reporting memory pressure after a rollout. You want the log lines from the requests that actually failed on the pods under pressure, which means going from metrics to traces to logs without stopping in between.

FROM logs-*.otel-*
| WHERE trace_id IN (
      FROM traces-*.otel-*
      | WHERE kind == "Server"
        AND status.code == "Error"
        AND resource.attributes.k8s.pod.uid IN (
            TS metrics-kubeletstats.otel-*
            | STATS peak = MAX(MAX_OVER_TIME(metrics.k8s.pod.memory_limit_utilization))
                BY resource.attributes.k8s.pod.uid
            | WHERE peak > 0.95
            | KEEP resource.attributes.k8s.pod.uid
          )
      | STATS failures = COUNT(*) BY trace_id
      | SORT failures DESC
      | LIMIT 1000
      | KEEP trace_id
    )
| STATS lines = COUNT(*), traces = COUNT_DISTINCT(trace_id)
    BY pattern = CATEGORIZE(body.text), service = resource.attributes.service.name
| SORT traces DESC

Reading the query inside out, you can see each layer answering one part of the question. The innermost subquery identifies the pods whose memory peaked above 95% of their limit, using MAX_OVER_TIME to take each pod's peak rather than its average. The middle one narrows to the requests that actually failed on those pods and reduces them to at most 1,000 trace IDs. The outer query then collects the logs for those traces from every service that took part, including services running on pods that were entirely healthy, and that last part turns out to be where the answer is.

The pods are matched on their UID rather than their name, because names repeat across namespaces and restarts. The query assumes Kubernetes metadata reaches your spans, which is what the collector's k8sattributes processor is for; if it does not, the host or container ID works the same way.

From the results above, the first row sits at exactly 1,000 because that is the subquery's LIMIT, so it describes the size of the sample rather than the size of the incident. Every trace in that sample carries the cart deadline line, which is the symptom you already knew about when you started.

Look at the second row instead. product-catalog is rejecting oversized payloads across 946 of the same 1,000 traces, and it never appeared in the pod subquery at all: its pods peaked at 75% of their memory limit, well under the 95% threshold. The rollout started sending larger payloads, which would account for both cart's memory climb and the failures. Checkout's retry budget gives out in about half of them, which is how the failure became visible to users.

Filtering logs directly by the pressured pods would have shown you the cart line and hidden the product-catalog one, which is to say it would have confirmed the symptom and buried the cause. Doing it without subqueries means three queries and two hand-built lists, and the second list is a thousand trace IDs. That is usually the point at which people stop after the first hop and go with the cart theory. The reason the second hop is cheap here is that all three signals sit in the same store behind the same query language, so widening from pods to traces to every service in the trace is a clause, not a project.

Why do ES|QL subqueries matter for AI agents?

Keeping the intermediate set inside the cluster is convenient for a person and close to essential for an agent querying on your behalf.

Split across two tool calls, the intermediate result has to travel. A list of 500 trace IDs comes back in a tool response and occupies the model's context, and the agent then has to rewrite every one of them into the next query. That costs tokens on every hop, and it is where truncation and transcription errors come from. With a subquery, the intermediate set stays inside Elasticsearch and the agent only ever sees the final table.

The problem compounds when the signals are spread across systems. An agent then needs credentials, a client, and a working knowledge of the query language for each one, plus the judgment to join results that use different names for the same host. One store and one query language reduce that to a single skill the agent has to be good at.

One ES|QL string is also a complete description of the correlation, which makes the investigation reproducible: an agent can put the query in its summary, and a human can paste it into Discover and get the same logic evaluated against current data. A two-call sequence with a hardcoded host list in the middle gives you neither. The one thing to watch is that an agent calling the _query API has to filter @timestamp itself, since nothing is binding a time picker.

Four things to know before writing ES|QL IN subqueries

IN subqueries are in technical preview in Elastic Stack 9.5 and on Elastic Cloud Serverless, while TS and FORK have been generally available since 9.4. CATEGORIZE has been generally available since 9.1 and requires a Platinum license; every query above works without it if you group by an existing field instead.

Four things are worth knowing before you write your own:

• In 9.5 the subquery returns exactly one column, which is what the trailing KEEP does in each example.
• Aggregate the subquery down to distinct values with STATS ... BY before returning them. Its result is materialized for the outer query to filter against, so handing back a few hundred host names instead of a few million rows is both faster and safer.
• Filter nulls out of any NOT IN subquery, because SQL null semantics mean one null makes the predicate match nothing.
• Subqueries are non-correlated. They run independently and cannot reference columns from the outer query, so this is a set filter rather than a row-by-row join. Reach for LOOKUP JOIN when you need per-row enrichment, which we covered in ES|QL joins for richer observability.

Try ES|QL signal correlation on your own data

The pattern under all four examples is the same. You start with a set you can describe in one signal and a question you can only answer in another, and the subquery carries that set across the boundary for you.

The syntax is the smaller part of what makes that work. It works because logs, metrics, and traces sit in one store behind one query engine, under field names they kept on the way in, so crossing from one signal to another is a clause in a query rather than an integration to build and maintain. Where that is not true, the same four investigations turn into a sequence of exports, translations, and manual joins, and that costs more than slower answers. It quietly shrinks the number of questions anyone is willing to ask, and the negative results are the first to go.

Each pattern here replaces two or three queries with one, and no host list or trace ID list has to move between them. Fewer steps mean fewer places to be wrong, and a correlation you can save as a single string and hand to someone else.

To try it:

1. Open an Observability project on Elastic Cloud Serverless, or upgrade to Elastic Stack 9.5.
2. Send data with the Elastic Distributions of OpenTelemetry, or point an existing collector at Elasticsearch.
3. In Discover, switch to ES|QL and start from the metrics to logs query above, swapping in your own data streams and thresholds.
4. Read the IN subquery reference for the full set of commands you can use inside a subquery.

The challenge deepens when something goes wrong. A crash on the device, a slow screen, or an error response might originate in the Android app itself, in a backend service, or somewhere in the network path between them. Debugging these problems without a connected, E2E view from the mobile client to the backend is time-consuming and frustrating. And without a standard instrumentation format, mobile teams often end up maintaining separate tooling that doesn't integrate with what the backend and infrastructure teams already use.

OpenTelemetry offers a way out: a unified, open-standard instrumentation model that works across platforms and languages, backed by a large community. The Elastic Distribution of OpenTelemetry Android, or EDOT Android, is an APM agent for native Android applications built on top of OpenTelemetry. It gives Android teams a practical path to observe mobile app behavior in Elastic, providing them with distributed tracing, crash reporting, session tracking, disk buffering, and automatic instrumentation, with as little code as possible while staying grounded in open standards.

To see what it all looks like, we will instrument a demo Android weather application end to end. You will run Elasticsearch, Kibana, and the Elastic Agent locally. The Elastic Agent provides the OTLP endpoint that receives telemetry from the Android app and backend. You will then generate distributed traces, custom spans, logs, and Android crashes from the app, and explore the results in Kibana using the Android OpenTelemetry dashboards.

This article focuses on a hands-on experiment to explore the E2E experience of observing Android apps with Elastic, using the EDOT Android agent. For more specific details on the EDOT Android agent, such as a list of supported features and a setup guide for your own Android project, take a look at EDOT Android docs.

Setting up EDOT Android with Elasticsearch and Kibana

We will use the EDOT Android demo application. The demo is intentionally small but covers the main workflows you need when evaluating mobile observability with Elastic.

The demo has two main components: an Android app, and a Spring Boot backend. Additionally, you'll need an Elastic Stack environment up and running; we'll explain more about how to get one later in this guide.

Prerequisites

• Java 17 or higher.
• Docker.
• Android Studio.
• An Android emulator.
• On Windows, use Windows Subsystem for Linux (WSL) to run the demo scripts.

Step 1: Clone the demo app's repository

We'll start by cloning the EDOT Android demo application:

git clone git@github.com:elastic/android-agent-demo.git

Step 2: Start the Elastic Stack

The demo uses start-local to run Elasticsearch, Kibana, and the Elastic Agent with a single command. In this setup, the Elastic Agent provides the OTLP endpoint that receives telemetry from the application and backend. Run this from the directory where you want the local Elastic files to be created:

curl -fsSL https://elastic.co/start-local | sh -s -- --edot

For more information on this step, take a look at the demo app's instructions.

Step 3: Start the local backend

The demo backend is a Spring Boot service instrumented with the EDOT Java agent. It handles the app's weather requests and calls the Open-Meteo public API for weather data.

./backend-manager start

For more information on managing the backend service, take a look at the demo app's instructions.

Step 4: Launch the Android application

Use Android Studio to open up the EDOT Android demo application repo and run the application in your emulator. More info on how to run Android apps from Android Studio here.

Generating distributed traces, errors and crashes from an Android app

The Android app has two screens: a city selector and a weather display screen that shows the current weather for the selected city on the previous screen. It includes two intentional failure paths: the first one is reached by selecting New York, which causes the backend to reject the request (the demo backend only supports European cities), and tapping the floating crash button intentionally crashes the app so you can review crash reporting in Kibana after relaunch. We'll take a look at those use cases in more detail below.

Tracing a successful request end to end

In the EDOT Android demo app, selecting "Paris" as the city triggers a successful backend request on the second screen, for which a span will be automatically generated using EDOT Android's OkHttp auto-instrumentation, which supports all OkHttp-generated HTTP requests and tools using it, such as Retrofit. Aside from the Android HTTP span, the successful city request continues e2e and creates a backend HTTP client span to Open-Meteo.

How backend errors appear in the Android distributed trace

The demo backend only supports European cities, so selecting "New York" causes it to fail, which in turn automatically creates an error associated with our Android app's HTTP span. This is done automatically. We'll see later how to find and inspect these issues from Kibana.

How EDOT Android captures and reports app crashes

The crash button creates a crash event that appears in Kibana after the app is reopened. This event contains session information that will help us narrow down its root cause from Kibana, as we'll see later.

Note: EDOT Android automatically attaches Android session context to spans and logs. That means that any span or log created before the crash can be reviewed together with the crash event and nearby spans from the same session, giving you a complete picture of what the user was doing. This even applies to manually created spans and logs.

Visualizing Android application monitoring data in Kibana

To see the whole story from our Android app in a single place, we'll install Kibana's Android OpenTelemetry Assets package by following the steps below.

1. In Kibana, search for "Android OpenTelemetry Assets" in the global search field.
2. Open it and click Install to add the Android dashboards to your Kibana instance.

Exploring Android application monitoring dashboards in Kibana

Once the content package is installed, open the [Android OTel] Application Overview dashboard:

1. In Kibana, search for Dashboards in the global search field or in Kibana's menu.
2. In Dashboards, search for Android OTel and open the "[Android OTel] Application Overview" dashboard.
3. Select your application from the Applications panel at the top of the dashboard.

The dashboard provides a set of metric panels for an overview of your app's health, performance, and RUM, as well as a set of panels that can be further explored either in Discover or the Exception dashboard, as explained below.

How to inspect the distributed tracing waterfall in Kibana

From the Application Overview dashboard, go to one of the span tables (either All spans or Failed spans) and click its Explore in Discover button.

In Discover, click the expand icon on the left side of any span row to open its details panel. The trace waterfall UI appears inside, showing the full span hierarchy and timing for that trace. You can expand the waterfall to full screen and drill down from there.

Analyzing failed spans and backend errors in Kibana

While you can find all kinds of spans in the dashboard's All spans panel, you can narrow them down to failed ones only by exploring the Failed spans panel instead.

For the New York path, find a failed span and expand it. The trace waterfall highlights the backend error, and the exception details show the intentional backend rule that only supports European cities.

Reviewing crash details and stacktraces in the exception dashboard

Crash reporting is provided by the crash automatic instrumentation plugin. When an unhandled exception crashes the app, EDOT Android stores the crash event on disk. The event is exported the next time the app starts. Disk buffering ensures the crash event is not lost even if the network was unavailable at the time of the crash.

In the Application Overview dashboard, scroll to the Crashes section. You will see crash groups listed by a computed stacktrace group ID. Select a group and click View crash details to open the Exception Details dashboard.

The Exception Details dashboard shows a set of metrics to better understand the impact of the selected crash, as well as its full stacktrace. Crash events are grouped based on their stacktrace, which helps ensure that the same crash is counted and aggregated in this dashboard to better understand a single crash's impact on your application.

For this demo, the stacktrace points to the intentional crash in MainActivity. The nearby session events should also include the custom Crash button click log created just before the crash, which helps explain how the crash was triggered. We'll take a look at how to inspect a session to get an idea of the user's journey within your application that led them to a crash.

Using sessions to understand user flow in EDOT Android

Mobile troubleshooting often starts with a single bad outcome (a crash, an error, a slow UX), but the useful question is what happened before that outcome. EDOT Android helps answer that by attaching session.id to every span and log emitted by the application, even for manually created ones.

A session is meant to cover a single user interaction with your application. A new one is created when there is no previous active session or when the previous session has expired. Sessions expire after 30 minutes of inactivity. If the app stays active, a session can last up to 4 hours.

This lets you query all the telemetry from a single session and review it in order. After finding a crash group, drill into one affected session from the Top affected sessions panel and review the event timeline. You can see the custom logs, app startup spans, HTTP request spans, and crash data together in one investigation path.

The Event timeline panel on the Application Overview dashboard is also useful here: select a session from the dashboard's top filters, and the timeline shows the full sequence of spans and logs in that session chronologically.

What EDOT Android adds as a mobile APM for production apps

The demo uses a local stack and simple code, but the same agent features apply to production apps.

Disk buffering stores telemetry locally before export. This reduces data loss when the device has poor connectivity or the app is temporarily offline.

Automatic instrumentation creates telemetry for supported targets without adding code around every call. Today that includes OkHttp, crash reporting, and an adapter for OpenTelemetry Android instrumentation.

Manual instrumentation lets you add spans, logs, and metrics for app-specific workflows. This is useful for screen loading times, checkout flows, login steps, feature usage, or any area where framework-level telemetry alone is not enough.

Central configuration can remotely adjust selected EDOT Android behavior through Kibana when the OpAMP endpoint is configured. At the time of writing, central configuration for EDOT Android is in preview and supports settings such as recording and session sample rate.

Distributed tracing connects Android app requests to backend service spans so you can trace the full path of any user action, from the tap on the screen to the database query on the server. EDOT Android ensures that your application's telemetry timestamps are in sync with the coordinated universal time. This ensures a proper trace waterfall hierarchy later on in Kibana, where different components are properly coordinated in time.

Clean up the demo

When you are finished, stop the backend in case you're planning to restart it later, or uninstall it otherwise:

./backend-manager stop
# ./backend-manager uninstall

Then stop or uninstall the local Elastic Stack:

cd elastic-start-local
./stop.sh
# ./uninstall.sh

Getting started with EDOT Android in your own app

EDOT Android gives native Android teams an OpenTelemetry-based path for mobile APM in Elastic. With a small Gradle setup and one early initialization call, you get automatic HTTP spans, crash reporting, session tracking, and direct access to the OpenTelemetry SDK for custom telemetry, and you can see it all tied together in Kibana's Android dashboards.

Observability is a crucial part of modern mobile development. Crashes, slow screens, and backend errors all impact real users, and the sooner you can identify root causes across the full request path, from the device to the database, the better. The demo app is a good first step because it exercises the complete workflow without requiring a production deployment. After that, the same setup model applies to your own app with production endpoints, API key authentication, and custom spans and logs tailored to your use cases.

Developer resources:

• EDOT Android demo application
• EDOT Android documentation
• EDOT Android getting started guide
• EDOT Android automatic instrumentation
• EDOT Android manual instrumentation
• EDOT Android configuration
• EDOT Android troubleshooting
• Android OpenTelemetry Assets dashboard docs

Don't have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out mobile observability with EDOT Android as described in this guide. We'd love to hear about your experience gaining visibility into your Android application stack with Elastic.

What you need

• An Observability serverless project, or an Elastic Cloud Hosted or self-managed stack at version 9.4 or later. PromQL is generally available in Elastic Cloud Serverless and Elastic Stack 9.5, and available as a technical preview in Elastic Stack 9.4.
• Kubernetes state and container memory metrics in Elasticsearch.

What is the alert telling us?

This is the alert that opened the investigation:

It comes from this waiting-reason query:

PROMQL
  max by (namespace, pod, container) (
    max_over_time(
      kube_pod_container_status_waiting_reason{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api",
        reason="CrashLoopBackOff"
      }[2m]
    )
  ) == 1

A result of 1 means Kubernetes is delaying another start because the container has failed repeatedly. That is the correct paging signal here because the checkout path has a single replica: when that replica restarts, requests fail.

The max_over_time(...[2m]) range keeps the alert tied to recent samples. Without it, the last observed value of 1 can outlive the pod, and the rule keeps matching after that pod is gone.

That PromQL query ran every minute over a two-minute window and created an alert after one matching run.

Why did the last container stop?

The alert shows what Kubernetes is doing now. It does not show how the previous container ended:

PROMQL
  max by (namespace, pod, container) (
    kube_pod_container_status_last_terminated_reason{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      reason="OOMKilled"
    }
  ) == 1

A result of 1 for the same namespace, pod, and container means the last recorded exit was out of memory. Kube-state-metrics keeps that last reason as a gauge, so the value can stay visible after recovery. It points the investigation at memory; it does not prove that every restart in the window was an OOM kill.

Is the failure repeating?

A single restart can still be transient. The restart counter shows whether the failure keeps happening:

PROMQL
  max by (namespace, pod, container) (
    increase(
      kube_pod_container_status_restarts_total{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api"
      }[10m]
    )
  )

increase() shows how much the restart counter rose over the selected range. Repeated increases during the incident window explain why Kubernetes entered backoff.

How close is memory to the limit?

We need to know how close the container is to its memory limit, and whether that gap collapses right before each restart. This deployment allows only 128MiB, so the next query divides working-set memory by that configured limit:

PROMQL
  max by (namespace, pod, container) (
    container_memory_working_set_bytes{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      image!=""
    }
  )
  /
  max by (namespace, pod, container) (
    container_spec_memory_limit_bytes{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      image!=""
    }
  )

The chart shows a repeating sawtooth: memory approaches 90% of the limit, drops when the process stops, and climbs again after each restart.

Working set is the better signal here than total usage. Total usage includes reclaimable file cache, so it can sit near the limit without a kill. Working set is closer to the memory that triggers OOMKilled for this workload.

Is the node under memory pressure?

OOMKilled can mean the container hit its own limit, or the node ran low on memory and Kubernetes started reclaiming. To separate those cases, first find which node runs the pod, then check whether that node (or any peer) reported MemoryPressure.

PROMQL
  max by (namespace, pod, node) (
    kube_pod_info{
      namespace="checkout",
      pod=~"checkout-api-.*"
    }
  ) == 1

The pod sits on ip-10-0-2-18.ec2.internal. That is the node whose MemoryPressure result matters most for this incident:

PROMQL
  max by (node) (
    max_over_time(
      kube_node_status_condition{
        condition="MemoryPressure",
        status="true"
      }[30m]
    )
  )

Every node returns 0, including ip-10-0-2-18.ec2.internal. So the host was not under node-wide memory pressure. The kill came from the container limit itself.

Does raising the limit clear the alert?

checkout-api was healthy, then began building an in-memory cache that grows to 200MiB in 10MiB steps. The container only allows 128MiB, so the process is killed with OOMKilled before that cache is fully allocated.

We will raise the memory limit to 512MiB so the 200MiB cache fits with room for the runtime, then check whether CrashLoopBackOff clears:

kubectl set resources deployment/checkout-api -n checkout --limits=memory=512Mi

The same waiting-reason query then stops matching. CrashLoopBackOff drops off:

PROMQL
  max by (namespace, pod, container) (
    max_over_time(
      kube_pod_container_status_waiting_reason{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api",
        reason="CrashLoopBackOff"
      }[2m]
    )
  ) == 1

And the alert that started this investigation? Gone.

That is how you detect and investigate a Kubernetes CrashLoopBackOff with PromQL: from the firing alert, through OOMKilled and the limit mismatch, to a recovered alert. Elasticsearch holds the metrics; Kibana runs the same PromQL queries you already know from Prometheus.

Try it

1. Open an Observability project on Elastic Cloud Serverless, or use Elastic Stack 9.4 or later.
2. In the ES|QL editor in Kibana, run the waiting-reason query against a workload you care about.
3. Follow the same path from that alert to termination reason, restarts, memory versus the limit, and recovery.

For more PromQL in Elastic, see Elasticsearch supports PromQL, PromQL queries in Kibana, and Investigate Kubernetes infrastructure with PromQL.

Prerequisites for Elastic integration alerting rule templates

Elastic Stack 9.4.0 or later.

Alerting rule templates have been available since 9.2.1, under the integration Assets tab. This article covers three things that need 9.4.0: the dedicated Alerting tab, idle data streams rules, and the NGINX OpenTelemetry Assets package, which is in technical preview.

Step 1: Send NGINX logs and metrics to Elasticsearch with OpenTelemetry

First, get NGINX metrics and logs into Elasticsearch.

Enable the NGINX stub_status module and make the access and error logs readable by the collector. Then, configure an EDOT or upstream OpenTelemetry Collector with the nginx and filelog receivers to export metrics and logs to Elasticsearch.

The integration setup has the full receiver and pipeline configuration.

If you want to reproduce this example, you can use the companion repository.

Both signals matter for alerting, and each group of templates reads a different data stream:

• The log-based templates (4xx and 5xx error rates, error log spike) query logs-nginx.access.otel-* and logs-nginx.error.otel-*, which come from the filelog receiver.
• The metric-based templates (active connections, dropped connections) query metrics-nginxreceiver.otel-*, which comes from the nginx receiver.

This is easy to get wrong: the Fleet Nginx (OpenTelemetry) input package collects stub_status metrics only. Its companion for logs is the classic Nginx integration, which writes ECS-based nginx.access and nginx.error datasets, not the *.otel-* data streams the log-based templates query. If you rely on that pairing alone, the log-based rules have nothing to evaluate and silently never fire. Run the filelog receiver too, not just the nginx receiver.

Step 2: Install the NGINX OpenTelemetry Assets integration

NGINX OpenTelemetry Assets is a content-only package. It ships the dashboards, alerting rule templates, and SLO templates, but it does not collect data itself. The data comes from the collector you set up in Step 1.

You don't need to install it by hand. Once the NGINX OTel data from Step 1 starts arriving, Elastic detects it and installs the Assets package for you, which takes a minute or two. Confirm it under Management > Integrations > Installed integrations, where NGINX OpenTelemetry Assets should appear.

Step 3: Create Elasticsearch alert rules from a rule template

Open the integration and select the Alerting tab.

This package ships six templates: high 4xx and 5xx error rates, high active connections, an error log spike, dropped connections, and a generic High error rate by service template that points at a placeholder logs-myservicereceiver.otel-* index for you to repoint and rename. The five NGINX rules run ES|QL every minute and group results by host.name, so an alert points at the host with the problem. The generic template groups by service.name instead, since it is meant to be repointed at whichever service you choose.

Select a template, for example [Nginx OTel] High 5xx error rate. Kibana opens a prefilled Create rule form built on an Elasticsearch query rule. It runs the template's ES|QL on a schedule and alerts when the query returns rows.

The query looks like this:

FROM logs-nginx.access.otel-*
// Flag each access log entry as a server error (5xx) or not
| EVAL is_5xx = CASE(http.response.status_code >= 500, 1, 0)
// Aggregate total requests and 5xx count per NGINX host
| STATS total = COUNT(*), errors_5xx = SUM(is_5xx) BY host.name
// Minimum sample size to avoid noisy low-traffic hosts
| WHERE total > 50
// Calculate 5xx error rate as a percentage
| EVAL error_rate_pct = ROUND(TO_DOUBLE(errors_5xx) / TO_DOUBLE(total) * 100.0, 2)
// Alert threshold: adjust to tune sensitivity
| WHERE error_rate_pct > 5.0
| SORT error_rate_pct DESC
| LIMIT 10

It counts requests and 5xx responses per host, keeps hosts with enough traffic to matter, and returns those above five percent.

Three things to get right while the form is open:

• Send data first. ES|QL validates column names against the indices that exist when the query runs. Open a template before any NGINX data has been ingested and the editor reports Unknown column "http.response.status_code" and the form shows errors. Once data is flowing (Step 1), the same query validates and the error clears, so collect data before you create the rule.
• Set the time field to @timestamp.
• Leave "Create an alert for each row" selected. Because the query groups by host.name, this makes every affected host raise its own alert.

Add a connector and an action so the alert reaches Slack, email, or PagerDuty, then save and enable the rule.

Step 4: Tune alerting rule template thresholds in ES|QL

The thresholds are starting points, so confirm them against your own traffic. The threshold lives in the ES|QL WHERE clause.

To make the rule stricter, change error_rate_pct > 5.0 to error_rate_pct > 2.0. To require more traffic before it fires, raise total > 50. Use Test query in the rule form to confirm the edited query parses and returns rows before you save.

Three more settings are worth a look:

• Time window: the look-back period the query runs over. A shorter window reacts faster but is noisier on bursty traffic.
• Rule schedule: how often the query runs, every minute by default.
• Alert delay: the number of consecutive runs the condition must hold before an alert is created, which filters out single-run blips.

How do you detect idle data streams in Elasticsearch?

Threshold rules only fire while data keeps arriving. When an agent goes offline or an output breaks, the data stops, and a threshold rule has nothing to evaluate.

Many Elastic integrations include a dynamically generated idle data streams template for exactly this case. It is named [{Integration name}] Idle data streams and appears in the same Alerting tab, though it is generated automatically rather than bundled with the integration. It alerts when no data is written to any of the integration's data stream patterns within a set period.

The NGINX OpenTelemetry packages do not include an idle data streams template, which is why no such template appears in the Alerting tab from Step 3. The end of this section covers what to do instead.

The default period is 24 hours, which is usually too long. A production service can go quiet for most of a day before you hear about it.

When you create the rule, drop the period to match how fast you need to know. Fifteen minutes to one hour works for a critical service. For a batch job, set a period comfortably longer than its run interval, so the quiet gaps between runs do not trigger it.

So why is this example left out? The template is generated from the data stream patterns an integration defines, and it is not generated for input-only packages. A content-only package like NGINX OpenTelemetry Assets defines no data streams of its own either. To catch silence in a collector-based setup like this, recreate the rule by hand with an Elasticsearch query rule. Use the query DSL or KQL variant rather than ES|QL, because an ES|QL rule fires on returned rows and so cannot alert on the absence of data. Point it at the OTel data streams (logs-nginx.access.otel-*, or metrics-nginxreceiver.otel-*) and set the condition to fire when the number of matching documents is below 1 over a window of, say, the last 15 minutes. That reproduces what an idle data streams template does, scoped to the data streams your collector writes.

Get started with Elastic integration alerting rule templates

Alerting rule templates turn alert setup into a few steps: send data, install the integration, create a rule from a template, and adjust the threshold. Treat the bundled thresholds as defaults to confirm, not numbers to trust blindly. And where an idle data streams template is available, reduce its 24-hour default so you find out quickly when a service goes silent.

Resources

• Companion repository, to generate the NGINX demo data used here
• Alerting rule templates
• NGINX OpenTelemetry Assets integration
• End-to-end tracing for NGINX with OpenTelemetry
• Elasticsearch query rule
• Elastic Agent built-in alerts, for monitoring the agents themselves

The closest option today is to hand-code the mapping in the transform processor:

# otel.yml
processors:
  transform:
    log_statements:
      - context: log
        statements:
          - set(attributes["user.name"], "Alice") where attributes["user.id"] == "user001"
          - set(attributes["user.name"], "Bob") where attributes["user.id"] == "user002"
          - set(attributes["user.name"], "Carol") where attributes["user.id"] == "user003"
          # ...and one more line for every user

That works for a handful of entries, but it does not scale beyond 10 to 20 items. Every new mapping means another statement, the lookup data lives in the same file as your pipeline config, and there is no way to point at reference data that already exists. It proves the need is real, but it only covers the simplest, smallest cases.

If you run the Collector and you have been reaching for the transform processor, a sidecar script, or a downstream ingest pipeline just to add reference data, this component is for you.

Why the OpenTelemetry Collector needed a lookup processor

The Collector was already good at a couple of enrichment patterns. The transform processor reshapes and derives data from what is already on a record. The k8sattributes and resourcedetection processors attach system and environment metadata, like Kubernetes pod details or cloud host information.

What it could not do was look up related data by a key you already have. Three patterns in particular had no home:

• File-based lookups from static reference data in JSON, YAML, or CSV
• HTTP and API-based enrichment from an external service
• DNS lookups, such as resolving an IP to a hostname

These are everyday tasks in other data collectors and transformation tools. Mapping an internal service ID to a friendly name, attaching business metadata by customer ID, or resolving an IP all fall into this category. Without a native component, teams built brittle workarounds or pushed the work downstream where it is harder to reuse.

That gap is what the new lookup processor closes. Elastic's Data Processing team proposed it, the community accepted it, and it was built in partnership with Grafana, thanks to Sam DeHaan (GH: dehaansa). The lookup processor takes a value from your telemetry, uses OTTL to build a lookup key, queries a source such as a YAML file or DNS, and writes the result back as new attributes.

How the OpenTelemetry lookup processor is designed

The design grew out of repeated requests from the community for richer enrichment. A few examples that fed into it:

• Enrich attributes based on key matching from YAML or CSV definition (#40526)
• Enrich telemetry with resource metadata from an inventory datasource (#40936)
• Generic resource detector (#29627)
• Alert Manager receiver and exporter (#18526)
• gRPC processor / connector (#20888)

Rather than build a one-off component for each request, the goal was a single processor flexible enough to cover them. Four ideas shaped it:

• Multiple lookups per processor, so one instance can enrich several attributes in a single pass.
• Caching, so external sources like DNS do not get queried for the same key over and over.
• OTTL for key extraction, so you get a full expression language for pulling the lookup key off a record, including converters.
• Extensible sources, so you are not limited to the built-in set. You can register a custom source for your own data.

Extensible sources matter most for the processor's long-term value. A source is a small, well-defined interface, so the processor is a foundation for enrichment rather than a fixed list of features. YAML and CSV cover static reference data today, DNS covers dynamic resolution, and the same interface leaves the door open for HTTP APIs, key-value stores, or anything specific to your environment.

How the lookup processor evaluates keys with OTTL

Whatever source you configure, the processor runs the same three steps for every record.

First, it evaluates an OTTL expression to produce a lookup key from the record. Second, it hands that key to the configured source and gets a value back. Third, it writes the value to the attributes you name, on the record or on its parent resource. When a key has no match, the processor writes a configurable default so downstream queries stay predictable.

The next two sections walk through the two sources available today.

File-based lookups with YAML in the OpenTelemetry Collector

The most common case is static reference data. You keep a mapping file next to the Collector and enrich records against it. Here the processor reads a YAML file and adds user.name to each log based on its user.id.

# otel.yml
processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/mappings.yaml
    lookups:
      - key: log.attributes["user.id"]
        attributes:
          - destination: user.name
            default: "Unknown User"

The mapping file is a plain set of key-value pairs:

# /etc/otel/mappings.yaml
user001: "Alice"
user002: "Bob"

The key field is an OTTL value expression, so log.attributes["user.id"] reads the user.id attribute off the log record. The destination is where the looked-up value lands, and default is what gets written when the key is missing from the file.

Given this incoming log record:

{
  "body": "User logged in",
  "attributes": {
    "user.id": "user001",
    "http.method": "POST"
  }
}

The processor looks up user001, finds Alice, and produces:

{
  "body": "User logged in",
  "attributes": {
    "user.id": "user001",
    "user.name": "Alice",
    "http.method": "POST"
  }
}

The original attributes stay intact and the enriched value is added alongside them. The CSV source works the same way for teams that keep reference data in spreadsheets or exports rather than YAML.

DNS lookup enrichment inside the OpenTelemetry Collector

Static files are one thing, but some enrichment requires live, changing data. Resolving an IP address to a hostname is the classic example, and it is the first dynamic source the processor supports. Instead of a file, you point it at a DNS server.

# otel.yml
processors:
  lookup:
    source:
      type: dns
    lookups:
      - key: log.attributes["client.ip"]
        attributes:
          - destination: client.hostname
            default: "Not found"

The shape of the config is identical to the YAML example. Only the source changed. The processor pulls client.ip off the record, asks a DNS resolver to reverse-resolve it, and writes the hostname back.

Given this incoming record:

{
  "body": "User logged in",
  "attributes": {
    "client.ip": "8.8.8.8",
    "http.method": "POST"
  }
}

The DNS source resolves 8.8.8.8 and produces:

{
  "body": "User logged in",
  "attributes": {
    "client.ip": "8.8.8.8",
    "client.hostname": "dns.google",
    "http.method": "POST"
  }
}

Because DNS queries hit an external system, this is where caching earns its place. The processor keeps results in an in-memory cache so a stream of records sharing the same IP does not turn into a stream of identical DNS queries. That keeps latency down and avoids hammering your resolver.

Writing a custom lookup source

The built-in sources cover common cases, but the real design goal was extensibility. A source is a small contract: you implement a lookup function that takes a string key and returns a value. The processor takes care of OTTL key evaluation, caching, defaults, and writing attributes, so a custom source only has to answer the question "what value goes with this key?"

That means if you already run an internal metadata API, a Redis cache, or a custom database, you can wire it in as a source and reuse everything else the processor provides. HTTP-based sources and key-value stores are natural fits, and they are on the roadmap precisely because the interface makes them straightforward to add.

How Elastic plans to use this lookup processor

Elastic builds its Collector distributions on OpenTelemetry Collector Contrib. The plan is to include lookup processor so the enrichment work teams do today with brittle workarounds can run inside the pipeline instead of downstream.

Two patterns stand out. The first is reference-data enrichment: turning an internal service ID into a friendly name, or attaching metadata such as a team or customer by ID, so telemetry arrives already labeled for search and correlation. The second is DNS resolution: turning a client.ip into a hostname before the data lands, which matters for network and security telemetry where you want the name rather than the raw address.

Doing this in the Collector keeps reference data close to where telemetry is processed and avoids duplicating the logic in separate ingest steps. As the source interface grows to cover HTTP APIs and key-value stores, the same processor can back richer enrichment without changing how pipelines are configured.

Lookup processor roadmap and how to contribute

The main implementation of the lookup processor has merged upstream into OpenTelemetry Collector Contrib. The core processor and YAML source landed first, and the DNS source followed as the first dynamic lookup.

There is a healthy backlog of work ahead, and contributions are welcome:

• An HTTP lookup source for enrichment from external APIs.
• More DNS capabilities, including A and AAAA queries and support for multiple DNS servers.
• Component telemetry, so you can observe cache hit and miss rates, lookup latency, and error rates.
• Performance improvements as real-world usage grows.

If you want to try it, add the processor to a Collector build that includes Contrib, point a YAML source at a mapping file, and enrich a real record. Then open an issue or pull request on the OpenTelemetry Collector Contrib repository. The component is community-owned, and the more sources and feedback it gets, the more useful it becomes.

To go deeper, read the lookup processor README and the enrichment tracking issue. For more on how Elastic builds on OpenTelemetry, browse the OpenTelemetry articles on Elastic Observability Labs.

Availability

This is available in serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

What you need for RED metrics breakdown in Elastic Observability

You need trace data from a service instrumented with any method Elastic APM supports.

• Application instrumentation: one of the following:
• Elastic APM agents for Java, .NET, Node.js, Python, PHP, Ruby, Go, and other supported languages.
• OpenTelemetry SDKs sending OTLP via Elastic Agent or an upstream OpenTelemetry Collector with the elasticapm connector under connectors (not processors).
• Useful attributes: breakdown works best when spans include the dimensions you want to compare (k8s.pod.name, k8s.deployment.name, service.version, and others). You can also declare custom attributes on spans: add labels to transactions and spans with Elastic APM agents, or set OpenTelemetry attributes on spans and resources with OpenTelemetry SDKs. Those custom fields work as breakdown dimensions too.
• Backend: Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

How to go from a RED metrics alert to the root-cause error log

Step 1: Review RED metrics on the alert detail page

When you receive a notification for a RED metric threshold breach, if you open the alert detail page, you can review the symptoms for the impacted service on one page.

In our example, failed transactions have clearly increased for the cart service, so we want to understand what is driving that RED metric change:

Step 2: Break down RED metrics by span attributes in Discover

To investigate why a RED metric changed, open Traces in Discover and use the new breakdown feature to split RED metrics by any attribute on your spans. In our example, we're going to check Kubernetes attributes and service version, but you could break down by any span attribute you send (e.g. cloud.region, cloud.availability_zone, or container.id).

Each breakdown shows which attribute values moved the metric, so you can see whether the problem is isolated to one pod, deployment, version, or whatever dimension you split on.

In our example, error rate clusters on a single Kubernetes deployment, which points the investigation at a release. We will break down by service.version to validate our hypothesis:

Step 3: Open the trace waterfall and read the linked error log

Once trace breakdown has identified a specific service version as the likely cause, we can filter by that service.version and look at sample failing spans to see if they explain why the version is causing failures.

Open the trace waterfall for one failing span and follow through to the linked error log.

In our example, the error log points to bad configuration that could be causing the issue. Either way, we have narrowed the investigation to a solid hypothesis we can act on in just a few clicks:

From RED metrics alert to error log in Elastic Observability

From a RED metric alert, you can review the symptomatic service, break down Traces in Discover by any attribute on your spans, and open a failing span's trace waterfall to reach the error log in just a few clicks.

From a checkout RED metric alert to a failing shipping dependency on the APM service map (OpenTelemetry Demo).

Where the APM service map is available

This is available in Elastic Observability serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

Prerequisites for APM service map dependency analysis

You need trace data from services instrumented with any method Elastic APM supports.

• Application instrumentation: one of the following:
• Elastic APM agents for Java, .NET, Node.js, Python, PHP, Ruby, Go, and other supported languages
• Elastic Distributions of OpenTelemetry (EDOT) language SDKs
• OpenTelemetry SDKs sending OTLP via the EDOT Collector, Elastic Agent, APM Server, or the Managed OTLP endpoint. If you run a custom upstream Collector pipeline, include both the elasticapm connector and the elasticapm processor. Those components ship with the EDOT Collector (or a custom EDOT-like build); they are not part of the standard OpenTelemetry Collector Contrib distribution. For wiring details, see the upstream collector setup.
• Service map data: distributed traces that link the services in your architecture. The map draws connections from span parent-child relationships across instrumented services.
• Backend: Elastic Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

Dependency analysis walkthrough: from APM alert to shipping dependency

Step 1: APM service map on the alert detail page

When you receive a notification for a RED metric threshold breach on the checkout service, open the alert detail page.

The upgraded service map is embedded on the page, so you can start analysing dependencies the moment you land on the alert. You see checkout in context with its upstream and downstream connections without navigating away.

In our example, failed transactions have increased on checkout. The map is already scoped to the alert time range:

Step 2: Spotting a failing dependency on the service map

When we scan through the downstream dependencies, we can see some indicators that there is a problem with the checkout and shipping services. When we open the shipping service fly-out to view more details, we can see the transactions for /get-quote have an elevated failure rate:

Step 3: Drilling into failing transactions to identify RED metric drivers

Click on the /get-quote transaction to drill into these transactions. We can see a release marker for version 2.3.0 that seems to correlate to increase in failures. To validate this is the main contributing factor, we want to analyse these transactions and check a few dimensions to isolate the behaviour to the release:

Step 4: Comparing RED metrics by dimension in Traces in Discover

Using Traces in Discover, we can use the breakdown feature to compare various attributes to confirm the RED metric change is due to the release. As we break down by service.version, k8s.deployment.name and k8s.pod.name, we can see the failure rate is highly elevated for just this Kubernetes deployment.

These field names match the OpenTelemetry semantic conventions used by the demo (EDOT / OTel instrumentation). If you instrument with classic Elastic APM agents, use the ECS-style kubernetes.* equivalents instead (see ECS ↔ OTel field alignment).

Given we have clear indicators that this problem is almost certainly due to a problem caused by this Kubernetes deployment, we would next investigate the changes this deployment made so we can identify a mitigation strategy to return the system to a healthy status, most likely a rollback of the Kubernetes deployment.

Dependency analysis with the service map on custom dashboards

The APM service map can also be added as a panel to custom Kibana dashboards. If you have a custom Dashboard you would like to surface for a debugging problem with your service, you can attach it to your alert rule so users can use this to complement the default alert detail view.

When you receive the notification and land on the alert detail page, you can click Related Dashboards and open your custom dashboard. From here, you can use the same Service Map panel to quickly analyse dependencies just like the alert detail page:

What's new in the APM service map

Below is a list of the upgrades to the service map:

Service map embedded on APM alert pages and dashboards

• All APM alert detail pages will have the service map embedded to facilitate faster dependency analysis.
• There is a new Service Map Dashboard panel that you can add to any Dashboard.

Full-screen service map for large architectures

Open the map in full screen from the APM UI or from a dashboard panel. Full screen gives you more room on large architectures and works the same whether you opened the map from an alert, a service, or a dashboard.

Service map controls: search, filters, and orientation

On the service map in the APM UI and on dashboard embeds, the map toolbar adds controls to move faster on busy environments:

• Search — find services by text string

• Orientation — switch layout direction when dense graphs are hard to read

• Filters — narrow the map by alert status, whether a service has dependencies, anomaly status, and SLO status

Service map legend: node shapes, connections, and anomaly scores

A built-in legend explains node shapes, connection styles, and health indicators on the map. You spend less time decoding colours and icons when you are triaging under pressure.

The legend covers node shapes (instrumented services, databases and messaging, grouped resources), connection styles (one-way and two-way requests), and anomaly score colour rings from low through critical:

Minimap for navigating large service maps

A minimap helps you orient yourself on large maps. It also highlights anomalous services at a glance, so you can spot outliers without panning across the full graph.

RED metrics on connections with drill-through to Traces in Discover

Select a connection between two services to view RED metrics for the requests between them: rate, errors, and duration for that specific dependency edge.

From there, open Traces in Discover in one click to analyse those requests with full query flexibility. That is how we confirmed shipping was failing the requests checkout sent to it in the walkthrough above.

Adding the service map to a dashboard from the APM UI

From the service map in the APM UI, add the current map view to a dashboard in one click. You do not need to rebuild filters or time range settings manually when you want the same map on a team dashboard.

Service fly-out: preview RED metrics, anomalies and SLOs from the map

Wherever you view the service map in Kibana, you can open a service fly-out to preview a service without leaving the map.

The fly-out shows:

• RED metrics for the service
• Anomaly status — whether machine learning has flagged unusual behaviour
• SLO status — whether the service is meeting its objectives
• Transaction breakdown — how rate, errors, and duration split across transaction types

Use it to sanity-check a node before you open the full service page or follow a connection into Traces in Discover.

Summary: From APM alert to root cause with the service map

From a RED metric alert on checkout, the embedded service map showed shipping as the failing downstream dependency. Map controls, connection RED metrics, dashboard embeds, and the service fly-out are available on every map view in Kibana, so you can start dependency analysis wherever you already work.

Further Reading

You can see how we implemented these changes in the following post from Jenny Pavlova, who led the technical implementation:

6x faster at 500 services: how we rebuilt the Kibana APM service map from canvas to React DOM

Related Posts

See other recent improvements we've made for observing instrumented services in the blogs below:

• Four clicks from alert to root cause: how Elastic Observability links APM services to Kubernetes infrastructure
• Your SLO is on fire; here's how to find the arsonist in Elastic Observability
• Three clicks from alert to error log: breaking down RED metrics by any span attribute in Elastic Observability

Availability

This is available in Elastic Observability serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

Prerequisites for linking APM services to Kubernetes infrastructure

You need application traces and Kubernetes infrastructure metrics in the same Elastic Observability project.

• Application instrumentation: EDOT-instrumented services sending traces via the EDOT Collector or an upstream OpenTelemetry Collector with both the elasticapm connector and the elasticapm processor. The EDOT Collector includes both by default; for a custom upstream pipeline, see the upstream collector setup.
• Kubernetes observation: the cluster observed via OpenTelemetry with host and Kubernetes metrics from the EDOT Collector. See the Kubernetes quickstarts and Kubernetes observability with EDOT for setup.
• Backend: Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

APM alert triage: from notification to problematic pod

Step 1: Confirm the service degradation on the APM alert detail page

The redesigned alert detail page in Elastic Observability shows the impacted service, environment, endpoint and RED metrics in one view. Open it from the alert notification.

You can clearly see which service is impacted, which environment it runs in, what endpoint is being affected and easily look for correlations in their RED metrics. In this case, we can immediately rule out a spike in traffic as the throughput is clearly stable.

Step 2: Rule out service dependencies with the embedded service map

The newly embedded service map preview on the alert detail page shows the health and RED metrics of every dependent service, so you can rule out upstream causes without navigating away. In this case, we have been able to quickly rule out problems with other services causing the symptom with the symptomatic service:

Step 3: Review Kubernetes infrastructure metrics per pod, container and host

After ruling out service dependencies, open the service's updated Infrastructure tab in Elastic Observability to check for infrastructure-level patterns.

Step 4: Compare per-instance metric trends to find the root cause

The Infrastructure tab shows the average metric values over the specified time period. To really understand whether there is a problem with the infrastructure, we need to compare the pod, container and host metrics over time. This allows us to easily spot differences between different entities that may correlate with when the service started showing symptoms. In our example, we can clearly see a difference between some of the metrics between the pods that correlates with when the service symptoms began. So we know there is something going on with the infrastructure that needs investigating:

Summary: from APM alert to root cause in four clicks

In just a few clicks from an alert in Elastic Observability, you can rule out healthy dependent services without leaving the alert detail page, then compare per-pod infrastructure metrics to see which instance correlates with when the symptoms started.

What is Tail Sampling?

Distributed tracing is useful for debugging, but at production scale it comes with processing overhead and storage costs, at which point sampling becomes a natural way to maintain the value of tracing while keeping costs under control. Tail-based sampling, or tail sampling, is a technique that makes a sampling decision conditionally at a later stage, so that high-value traces like errors or slow transactions are more likely to be sampled. The opposite is head sampling, which makes the decision at the start of a trace, before any such information is available.

How does the tail sampling processor work?

The tail sampling processor buffers 100% of incoming traces (or spans, used interchangeably), then forwards the sampled subset after applying the sampling policies. Buffering is a major source of memory usage, and it scales proportionally to the volume of spans, a well known pain point in the community.

Memory usage is bounded by configuration parameters like decision_wait and num_traces. Setting decision_wait to 1 minute means a sampling decision is made for a trace after 1 minute, during which all spans for that trace are expected to have arrived. If a trace is slower than 1 minute, the decision is made with some spans missing.

As a side note, scaling out the tail sampling setup involves using the loadbalancingexporter to satisfy the requirement that all spans for a trace must be routed to the same collector. This introduces some operational complexity and potentially data loss during collector restarts. But this post focuses on the memory usage of a single tail sampling processor instance, regardless of horizontal scaling.

Why does tail sampling cause memory pressure?

These parameters introduce a tradeoff between data loss and memory usage, and they require assumptions about the shape of traces: how slow they can be, how many spans they contain, how large each span is. These assumptions can become stale as instrumentation evolves.

How much data loss is acceptable to limit memory usage, and can the tradeoff be improved? The following two contributions aim to give operators more flexibility.

How span-ingest reduces tail sampling memory by releasing spans early

sampling_strategy is a new configuration option added to the tail sampling processor in v0.149.0.

sampling_strategy defaults to trace-complete, which matches the original behavior: sampling policies are only evaluated when decision_wait has elapsed, at which point the trace is considered complete. (There is a similar config, decision_wait_after_root_received, for optimization, but it is excluded from this discussion for simplicity.) This means all spans are buffered in memory for roughly decision_wait before being released, regardless of whether a decision could have been made earlier. For example, health check spans that should always be dropped are still held in memory until policy evaluation time.

Alternatively, sampling_strategy can be set to span-ingest, where spans are evaluated individually at ingest time. This allows terminal decisions, specifically drop or sampled, to be made earlier, freeing memory by dropping or exporting all spans buffered so far for that trace before decision_wait elapses. In the health check example, a policy can be configured to drop the entire trace as soon as the root span belongs to a health check. It is worth noting that an unsampled decision, unlike an explicit drop, is not terminal, as it can be overruled by a sampled or drop decision from another span in the same trace, so unsampled traces cannot be released early.

Switching from trace-complete to span-ingest will require policy adjustments, as policies can no longer assume all spans are available at evaluation time. Moreover, not all policy types are supported with the span-ingest strategy.

Disk-backed tail sampling storage with Pebble

Even with span-ingest, all spans are still buffered in memory. As decision_wait is increased to accommodate slow traces and num_traces is increased to limit data loss, the collector will eventually hit its memory limit and get OOM killed, resulting in further data loss.

What if traces were buffered on disk instead, where there is an order of magnitude more capacity? The main drawback is performance: disk throughput and latency, even with SSDs, are at least an order of magnitude slower than memory, so disk writes need to be efficient. For this reason, Pebble, an LSM database, was chosen as the storage backend for its fast write performance. Read performance is less of a concern, as reads only happen for the sampled subset of traces when sampling_strategy is set to span-ingest.

The implementation introduces a TailStorage interface for trace storage operations, and a new tail_storage option in v0.150.0 (behind feature gate processor.tailsamplingprocessor.tailstorageextension) to configure the storage backend. The default in-memory behavior is unchanged, but it is now possible to swap in a different storage backend, like the new pebbletailstorageextension contributed to Collector Contrib.

Tail sampling memory benchmarks: trace-complete vs span-ingest with Pebble

Benchmark setup: OpenTelemetry Demo with fan-out collectors

The following benchmarks were produced by running OpenTelemetry Demo with increased load against a pipe collector, which receives all spans and fans them out to two identical collectors under observation (CUO-A and CUO-B), differing only in their tail sampling configuration. Measurements include pipe collector throughput, spans received, spans sent (sampled), CPU usage, and memory usage.

Benchmark setup diagram

                      demo ns
     +----------------------------------------+
     |  opentelemetry-demo                     |
     |    loadgenerator (locust)               |
     |    services: frontend, cart, ...        |
     |    demo-collector                       |
     +----------------------------------------+
                          |  OTLP/gRPC
                          v
                     chamber ns
     +----------------------------------------+
     |             pipe-collector             |
     |         receive once, fan out          |
     |      exporters: [otlp/a, otlp/b]       |
     +----------------------------------------+
              | OTLP                  | OTLP
              v                       v
     +----------------+      +----------------+
     |     CUO-A      |      |     CUO-B      |
     | tail_sampling  |      | tail_sampling  |
     |   (config A)   |      |   (config B)   |
     +----------------+      +----------------+

Tail sampling processor configurations

CUO-A

config:
  processors:
    tail_sampling:
      sampling_strategy: trace-complete
      decision_wait: 5m
      num_traces: 5000000
      block_on_overflow: true
      decision_cache:
        sampled_cache_size: 10000
        non_sampled_cache_size: 200000
      policies:
        - name: root_1pct
          type: and
          and:
            and_sub_policy:
              - name: root_span_only
                type: ottl_condition
                ottl_condition:
                  error_mode: ignore
                  span:
                    - "IsRootSpan()"
              - name: root_probabilistic
                type: probabilistic
                probabilistic:
                  sampling_percentage: 1.0

CUO-B

CUO-B uses the same tail sampling processor configuration as CUO-A, except it sets sampling_strategy: span-ingest and tail_storage: pebble_tail_storage/main, along with its corresponding pebbletailstorageextension configuration.

extensions:
  pebble_tail_storage/main:
    directory: /var/lib/otelcol/pebble

Memory, CPU and throughput results

The following tables compare trace-complete (CUO-A) against span-ingest with Pebble disk storage (CUO-B) across memory, CPU and throughput. The process RSS, Go heap allocation, and per-process CPU measurements come from OpenTelemetry Collector internal process and runtime metrics, while container working set and container CPU come from Kubernetes cgroup metrics scraped by kubelet/cAdvisor.

Memory (peak over the window)

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | process RSS | 916.4 MiB | 442.7 MiB | -51.7% | | Go heap alloc | 699.3 MiB | 241.7 MiB | -65.4% | | container working set | 763.0 MiB | 282.9 MiB | -62.9% |

CPU (total over the window)

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | per-process CPU | 11.9 core-s | 22.7 core-s | +90.7% | | container CPU | 11.9 core-s | 22.7 core-s | +90.1% |

Throughput (total over the window)

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | spans received | 257,804 | 257,804 | 0.0% | | spans sent | 2,477 | 2,477 | 0.0% |

Tail sampling

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | traces in memory peak | 29,284 | 29,245 | -0.1% | | traces sampled by root_1pct policy | 496 | 496 | 0.0% |

• cuo-a = trace-complete, cuo-b = span-ingest-pebble
• Window: 15m 39s (t+0:00 start, t+10:06 drain start, t+15:39 drain end)

The results show a significant memory reduction when using pebbletailstorageextension with span-ingest, at the cost of increased CPU usage from event serialization and database overhead.

What's next for OpenTelemetry tail sampling

Both sampling_strategy and pebbletailstorageextension are still in their early stages at the time of writing. Feedback and contributions are welcome in the OpenTelemetry Collector Contrib repo. Stay tuned for more improvements.

What you need before running these Kubernetes ES|QL queries

To follow the queries in this article, you need:

• Elasticsearch 9.2 or later.
• EDOT Collectors running in your cluster and shipping data to Elasticsearch, with the kubeletstats, k8s_cluster, and filelog receivers enabled.

Why ES|QL for Kubernetes monitoring

Elasticsearch ES|QL is a piped query language that lets you start from a data source and then add one operation per line: filter, compute, aggregate, sort. That structure fits investigation work well because you refine a query step by step as you narrow down a problem.

When to use the ES|QL TS command for Kubernetes metrics

The TS command understands time series. This matters because most Kubernetes metrics are counters or gauges sampled over time. Using TS for metrics avoids the common mistake of summing cumulative counters across pods and getting a meaningless number.

EDOT receivers these Kubernetes queries depend on

The queries below assume EDOT Collectors are running in your cluster and shipping data to Elasticsearch. A typical setup uses:

• The kubeletstats receiver for pod, container, and node resource metrics.
• The k8s_cluster receiver for object state such as pod phase and container restarts.
• The filelog receiver with the k8sattributes processor for container logs.

Resource attributes follow the OpenTelemetry semantic conventions. Pod, namespace, and node identifiers appear as k8s.pod.name, k8s.namespace.name, and k8s.node.name. Metric names are defined by the receiver that emits them, not by the semconv spec itself: k8s.pod.phase comes from the k8s_cluster receiver, while utilization metrics like k8s.container.memory_limit_utilization come from kubeletstats. EDOT preserves all of these names natively in Elasticsearch. The exact fields you have depend on which receivers you enabled, so treat the queries as templates and adjust field names if a metric is missing.

Exploring the cluster

Start broad. Before investigating a specific symptom, it helps to see what the cluster looks like in your data.

This query counts the pods reporting metrics in each namespace over the past hour.

FROM metrics-*
| WHERE @timestamp >= NOW() - 1 hour
| STATS pod_count = COUNT_DISTINCT(k8s.pod.name) BY k8s.namespace.name
| SORT pod_count DESC

COUNT_DISTINCT collapses the many metric samples per pod into a single count per namespace. The result is a quick inventory: which namespaces are busy and whether anything you expected to be running is missing.

Finding pod restarts and crash loops

Restarts are usually the first signal that something is wrong. The k8s.container.restarts metric is a gauge that reports the current restart count for each container.

This query surfaces the containers that have restarted the most in the last 24 hours.

FROM metrics-*
| WHERE @timestamp >= NOW() - 24 hours AND k8s.container.restarts IS NOT NULL
| STATS restarts = MAX(k8s.container.restarts)
    BY k8s.namespace.name, k8s.pod.name, k8s.container.name
| WHERE restarts > 0
| SORT restarts DESC
| LIMIT 20

MAX takes the highest restart count seen in the window, which reflects the latest value of the gauge. A container with a high and climbing restart count is the textbook sign of a crash loop. Once you have the pod name, you can pivot straight to its logs with the queries further down.

Spotting pods that are not running

A restart count tells you a pod recovered. The pod phase tells you whether it is healthy right now. The k8s.pod.phase metric encodes the phase as a number:

| Value | Phase | | :---- | :---- | | 1 | Pending | | 2 | Running | | 3 | Succeeded | | 4 | Failed | | 5 | Unknown |

This query uses TS to read the latest phase per pod and keeps anything that is not Running.

TS metrics-*
| WHERE TRANGE(15m)
| STATS phase = MAX(LAST_OVER_TIME(k8s.pod.phase))
    BY k8s.namespace.name, k8s.pod.name
| WHERE phase != 2
| EVAL phase_name = CASE(
    phase == 1, "Pending",
    phase == 3, "Succeeded",
    phase == 4, "Failed",
    phase == 5, "Unknown",
    "Other")
| SORT phase_name

LAST_OVER_TIME picks the most recent sample for each pod's time series, so you compare the current state rather than an average. Pods stuck in Pending often point to scheduling problems, such as insufficient CPU or memory on the nodes. Pods in Failed or Unknown are worth an immediate look.

Catching memory pressure before the OOM kill

Out-of-memory kills are one of the most common Kubernetes failures, and they are easier to prevent than to debug after the fact. When you enable limit metadata on the kubeletstats receiver, EDOT reports k8s.container.memory_limit_utilization as a fraction between 0 and 1 of the container's memory limit.

This query finds containers that ran close to their limit in the last hour.

TS metrics-*
| WHERE TRANGE(1h)
| STATS peak_mem_pct = MAX(MAX_OVER_TIME(k8s.container.memory_limit_utilization))
    BY k8s.namespace.name, k8s.pod.name, k8s.container.name
| EVAL peak_mem_pct = ROUND(peak_mem_pct * 100, 1)
| WHERE peak_mem_pct > 85
| SORT peak_mem_pct DESC

MAX_OVER_TIME finds the peak within each container's series, and the outer MAX keeps that peak per container. A container that repeatedly touches 95% or higher is a strong candidate for the next OOM kill. Pair this with the restart query above: a container with both a rising restart count and high memory utilization was very likely OOM killed.

Tracking CPU usage and node pressure

CPU problems show up as throttling and slow response times rather than crashes. The k8s.pod.cpu.node.utilization metric reports each pod's CPU use as a fraction of total node capacity.

This query charts the busiest pods over the last hour in five-minute buckets.

TS metrics-*
  | WHERE TRANGE(1h)
  | STATS avg_cpu = AVG(AVG_OVER_TIME(k8s.pod.cpu.node.utilization))
      BY k8s.pod.name, TBUCKET(5m)
  | SORT avg_cpu DESC

AVG_OVER_TIME averages the samples inside each pod's series for the bucket, and the outer AVG combines series that share a pod name. TBUCKET(5m) produces one point every five minutes, which renders cleanly as a time series chart.

To check whether the nodes themselves are saturated, query node utilization directly.

TS metrics-*
  | WHERE TRANGE(1h)
  | STATS cpu = AVG(AVG_OVER_TIME(k8s.node.cpu.usage)),
          mem = AVG(LAST_OVER_TIME(k8s.node.memory.usage))
      BY k8s.node.name, TBUCKET(5m)
  | SORT cpu DESC

A node sitting near full CPU explains throttled pods across many namespaces at once, which is easy to misread as an application bug when you only look at a single pod.

Investigating container logs

Once metrics point you at a pod, logs explain what it was doing. EDOT stores the log message in body.text and the level in severity_text, alongside the same k8s.* fields as the metrics.

This query ranks namespaces and pods by error volume in the last hour.

FROM logs-*
| WHERE @timestamp >= NOW() - 1 hour
  AND severity_text IN ("ERROR", "FATAL")
| STATS errors = COUNT(*)
    BY k8s.namespace.name, k8s.pod.name
| SORT errors DESC
| LIMIT 20

Counting by pod tells you whether errors are concentrated in one workload or spread across the cluster. A single noisy pod and a cluster-wide spike call for very different responses.

To read what a specific pod is logging, filter by pod name and search the message text.

FROM logs-*
| WHERE @timestamp >= NOW() - 1 hour
  AND k8s.pod.name == "checkout-<your-hash>"
  AND body.text LIKE "*timeout*"
| KEEP @timestamp, severity_text, body.text
| SORT @timestamp DESC
| LIMIT 50

LIKE "*timeout*" does a simple wildcard match on the message. For full-text relevance instead of wildcards, swap it for MATCH(body.text, "timeout").

Chaining ES|QL queries to investigate a Kubernetes incident

Real Kubernetes investigations chain multiple ES|QL queries together instead of running one in isolation.

A useful loop looks like this:

1. Count errors by pod to find the noisy workload.
2. Check that pod's restart count and memory utilization to see if it is crashing or starved.
3. Read its recent logs to find the specific failure.

Because every query uses the same k8s.namespace.name and k8s.pod.name fields, you can carry a pod name straight from one query to the next. The same fields let you build a single dashboard where a metrics panel and a logs panel filter together as you click through namespaces.

Turning queries into alerts and dashboards

Any ES|QL query in this post that produces an aggregated value can back an alert, not just support ad hoc investigation.

For example, the restart query becomes an alert when you keep only containers above a threshold and trigger on a non-empty result.

FROM metrics-*
| WHERE @timestamp >= NOW() - 15 minutes AND k8s.container.restarts IS NOT NULL
| STATS restarts = MAX(k8s.container.restarts)
    BY k8s.namespace.name, k8s.pod.name, k8s.container.name
| WHERE restarts >= 5

Wire this into an Elasticsearch query rule and you get notified the moment a container crosses five restarts in fifteen minutes, instead of finding out when a user does. The same pattern applies to memory utilization, node saturation, and error counts.

Building a Kubernetes monitoring toolkit with ES|QL

ES|QL gives you one language for every Kubernetes signal, from object state to resource metrics to container logs. Start with the exploration query to understand your cluster's shape, then keep the restart, phase, memory, CPU, and log queries close for the next incident. Use TS when you need time series functions like MAX_OVER_TIME or TBUCKET to aggregate correctly within each pod or container series. For counting distinct values or taking a simple MAX on a gauge, FROM is enough.

From here, you can adapt the field names to your own receivers, save the most useful queries as dashboard panels, and promote the critical ones to alerts.

To go deeper, see the ES|QL reference, the TS command documentation, and the EDOT Kubernetes guide.

Why multi-team log routing needs structure

Multi-team Elasticsearch deployments typically converge on a single shared index, which works until teams need different retention periods, sharding settings or processor pipelines.

Before Streams, you had to set up your ingestion scripts to send data to different indices, or data streams, or use the reroute processor to define the data destination based on some field name.

With AI Partitioning, you let the data arrive first. Then the AI analyzes what showed up, reviews the suggestions it proposes, refines them as needed, and applies them. The result is a set of wired child streams that inherit retention, processors, and schema from the parent, while still allowing you to override any of those per child.

What you need before using Streams AI Partitioning

Before running the example, three things need to be in place:

1. Wired Streams enabled. On Elastic Cloud Serverless and Elastic Cloud Hosted 9.4+, wired streams are on by default. If you upgraded from an earlier version, open the Streams app and confirm the toggle is on under Settings.
2. The Elastic Managed LLM connector. Go to Stack Management > Connectors > Create connector > Elastic Managed LLM. Keep in mind the following considerations:
3. This connector ships preconfigured and does not require an external account or API key.
4. Any generative AI connector works with the feature.
5. Note that Elastic Managed LLMs incur a cost per million tokens for input and output.
6. The account you use needs the manage_inference cluster privilege (the built-in inference_admin role grants it).
7. The Managed OTLP endpoint URL and an API key. Open Cloud Console > Manage > Application endpoints > Ingest. Copy the Managed OTLP endpoint URL and generate an API key from that same panel.

Once these are ready, open Observability > Streams and confirm a logs.otel wired stream is listed. That stream is the parent we will partition.

If Streams does not appear in the sidebar, your Kibana space may be using a solution view other than Observability. You can change it in Stack Management > Spaces > edit your space > set Solution view to Observability.

Generating multi-team log data

Our example uses three apps produced by teams from the same fictional company:

• payments-api: structured JSON with transaction_id and amount_cents. Sensitive data, long retention needs.
• checkout-web: JSON with cart_id and customer_id. Mostly INFO and ERROR.
• notifications-worker: less structured, with recipient and channel. High volume.

We use a Python script with the OpenTelemetry Python SDK to emit logs for all three teams over OTLP directly to the Managed OTLP endpoint. The full code, including setup and execution, is in the companion notebook.

Each team is defined with a service name, a set of message templates, and a function that generates team-specific attributes:

TEAMS = {
    "payments": {
        "service": "payments-api",
        "messages": [
            ("INFO", "charge captured tx={tx} amount_cents={amt}"),
            ("ERROR", "charge declined tx={tx} reason=insufficient_funds"),
            ("INFO", "refund issued tx={tx} amount_cents={amt}"),
        ],
        "extra": lambda: {
            "transaction_id": f"tx_{random.randint(10000, 99999)}",
            "amount_cents": random.randint(100, 50000),
        },
    },
    "checkout": {
        "service": "checkout-web",
        "messages": [
            ("INFO", "cart updated cart={cart} customer={cust}"),
            ("INFO", "checkout started cart={cart} customer={cust}"),
            ("ERROR", "checkout failed cart={cart} stage=address_validation"),
        ],
        "extra": lambda: {
            "cart_id": f"c_{random.randint(1000, 9999)}",
            "customer_id": f"u_{random.randint(100, 999)}",
        },
    },
    "notifications": {
        "service": "notifications-worker",
        "messages": [
            ("INFO", "email queued recipient={rcp} channel=email"),
            ("INFO", "sms queued recipient={rcp} channel=sms"),
            ("ERROR", "webhook failed recipient={rcp} channel=webhook status=503"),
        ],
        "extra": lambda: {
            "recipient": f"+1555{random.randint(1000000, 9999999)}",
            "channel": random.choice(["email", "sms", "webhook"]),
        },
    },
}

Setting elasticsearch.index to logs.otel as a resource attribute routes the data into the wired streams root instead of the default OTLP data stream.

def setup_provider():
    resource = Resource.create({"elasticsearch.index": "logs.otel"})
    provider = LoggerProvider(resource=resource)
    provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
    set_logger_provider(provider)
    handler = LoggingHandler(level=logging.INFO, logger_provider=provider)
    root = logging.getLogger()
    root.setLevel(logging.INFO)
    root.addHandler(handler)
    return provider

Run the notebook to emit 115 records with an uneven split across teams.

Open Observability > Streams > logs.otel and switch to the Partitioning tab. You should see the ingested data in the preview panel, with attributes like team, service.name, and the team-specific fields visible in the columns.

How Streams AI Partitioning proposes child streams

Streams AI Partitioning analyzes up to 1,000 documents from the parent stream, identifies attribute clustering and cardinality distribution, then proposes child streams, keyed on whichever field best separates the data logically (the ML approach behind this analysis is detailed in automated log parsing in Streams).

For the data we just emitted, the AI proposed three child streams keyed on attributes.service.name:

Each suggestion shows a Streamlang condition and the percentage of sampled documents that would match. The AI picked service.name because it is a standard OpenTelemetry attribute and a natural identifier for any single workload.

This is a reasonable first proposal, but it is worth thinking about what happens as the deployment grows. Right now there are three services because there are three teams. Tomorrow, Payments might add a refunds-api and a fraud-detector. Each new service would mechanically create another child stream, and over time you would end up with dozens of partitions for what is really just three organizational boundaries.

Elastic's partitioning recommendations prefer logical groupings like team or technology type, and aim for tens of partitions rather than hundreds. A team-keyed partitioning is more stable because Payments stays one child stream no matter how many services that team operates.

Refining Streams AI Partitioning suggestions in natural language

After reviewing the AI's initial suggestions in Streams AI Partitioning, click Modify suggestions to open a free-text prompt.

After submitting, the AI regenerates the suggestions. Now the three cards are keyed on attributes.team instead of service.name:

Select all three and click Accept selected. A confirmation dialog shows the streams that will be created, each with its WHERE attributes.team equals <team> condition.

Click Create all streams. The Partitioning tab now shows the three child streams as part of the logs.otel parent:

Every new document arriving at the OTLP endpoint will be routed into the correct child according to these conditions. You can open any child stream to verify its data. For example, logs.otel.checkout shows only checkout logs:

How do you set per-team log retention in Elasticsearch Streams?

After Streams AI Partitioning creates child streams, each one can have its own lifecycle configuration independently of the parent. Because wired streams use a parent-child hierarchy, every child inherits retention, processors, and schema from the parent by default. You only need to override the partitions you need to change.

Open the child stream logs.otel.payments and go to the Retention tab. Click Edit retention method, select Custom period, and set it to 90 days.

Do the same for the other teams with the retention that fits their needs:

| Stream | Retention | Rationale | | :---- | :---- | :---- | | logs.otel.payments | 90 days | Sensitive financial data, compliance requirements | | logs.otel.checkout | 30 days | Useful for debugging, no long-term need | | logs.otel.notifications | 7 days | High volume, low value after delivery confirmation |

Conclusion: from shared index to per-team streams, without routing rules

A shared Elastic deployment with several teams shipping logs is the normal starting point. Organizing it used to mean writing routing rules upfront or maintaining separate index templates and ILM policies by hand.

With Streams AI Partitioning, the workflow is different: you let the data arrive, let the AI read what showed up, refine the suggestions in natural language when they need adjusting, and accept.

The result is a set of child streams that inherit everything from the parent while giving each team its own retention and processing, without any manual template management.

Next steps

• Try the companion notebook to generate your own multi-team data.
• Read How Streams in Elastic Observability Simplifies Retention Management for a deeper look at the retention model.
• Read Streams Processing: Stop Fighting with Grok to explore the parsing side of Streams when teams need different processors.
• Read Introducing Streams for Observability for the broader investigation story Streams is part of.

• Elasticsearch is a Prometheus-compatible metrics backend — Prometheus Remote Write and PromQL now works natively in Kibana, no translation layer required.

• Metrics land in Elasticsearch's columnar TSDS architecture storing data up to 2.5× more efficient than Prometheus and 2× more efficient than ClickHouse.

• ES|QL time series queries run up to 30× faster than Prometheus on gauge averages and counter rates, including high-cardinality workloads.

• Elastic costs approximately 50% less than Datadog, with no custom metric classification and no cardinality-based billing.

• Grafana can query Elasticsearch directly through the native Prometheus API, keeping your visualization layer while replacing the backend.

• Kubernetes and AWS monitoring ship with pre-built dashboards, alert templates, ML anomaly jobs, and agentic investigation content ready at ingest. Additionally skills and MCP apps are available.

• Unified backend for Metrics, logs, and traces enabling agentic investigations without stitching context across tools.

• Metrics exploration in Discover lets anyone start querying and analyzing metrics immediately, no query language expertise required.

• Custom dashboarding is fast and flexible — dashboards-as-code, AI-assisted dashboard creation, variable controls, and collapsible panels mean less time building and more time investigating.

• Migration tooling to help easily migrate dashboards and alerting rules / monitors from Datadog and Grafana.

Elasticsearch metrics now competes on every dimension that matters to SREs: you can afford to keep every metric at full resolution, query it up to 30x faster than Prometheus, pay 50% less than Datadog, migrate dashboards and alerting rules from Grafana or Datadog easily, and go from alert to root cause without stitching context across disconnected tools. The rest of this post walks through each of these in detail.

Elasticsearch metrics performance: 30× faster than Prometheus and Mimir

Datadog and Prometheus force the same tradeoff: drop high-cardinality data or watch costs spiral. SREs managing Kubernetes, AWS, or any high-cardinality infrastructure know the specific shape of this problem. The Kubernetes labels, ephemeral pod data, and fine-grained OTel dimensions that matter most during an incident are the first to go when budgets tighten.

Elastic rebuilt the time series data store and ES|QL compute engine into a fully columnar metrics engine. Adding a new Kubernetes label, a new AWS instance tag, or a new application dimension doesn't strain the system; it adds far less cost than systems that index every label. OTel, Prometheus, and application-defined metrics all land in the same columnar backend at full resolution, with logs, traces, and metrics in a single store. No data dropped, no retention shortened.

Elasticsearch stores metrics up to 2.5× more efficiently than Prometheus (results may differ due to factors like compaction), and 2× more efficiently than ClickHouse. Query performance via ES|QL runs up to 30× faster than Prometheus on gauge averages and counter rates, including high-cardinality workloads where competitors stall. The architecture post covers how TSDS is organized and why the columnar layout produces these results.

| | | | | | :------------------------: | :----------------: | :--------------: | :----------------: | | Dimension | vs. Prometheus | vs. Mimir | vs. ClickHouse | | Query performance (ES|QL) | Up to 30× faster | Up to 30× faster | Up to 8× faster | | Storage efficiency | Up to 2.5× better | On par | 2× better |

The key architectural difference is that Elasticsearch metrics does not maintain a per-series in-memory state that scales with cardinality, so adding thousands of new Kubernetes pod labels or OTel dimensions doesn't drive up memory pressure.

OTel, Prometheus-native, and application-defined metrics are all stored the same way at full resolution, queried fast, at half the cost of Datadog.

Elastic Observability metrics pricing without the Datadog custom metric penalties

Observability cost is the #1 reason teams switch platforms. For Datadog customers, the pain comes down to one pricing mechanic: custom metrics. Any user-defined value outside of Datadog's built-in integrations is classified as a custom metric and billed at a premium rate. That includes the high-cardinality data that Kubernetes, OpenTelemetry, and cloud-native workloads generate by default. The more granular your instrumentation, the faster the bill compounds. Teams running modern infrastructure hit this ceiling quickly, and the response is predictable: drop data, shorten retention, lose the context that matters most when an incident happens.

Elasticsearch metrics removes that classification. Every metric is priced the same, with no per-metric penalties, no cardinality-based billing, and no forced rollups. You keep every metric at full resolution without a surprise invoice at the end of the month. And because Elastic is 50% the cost of Datadog, the conversation with finance changes: not what data you had to drop to stay on budget, but what you found because you kept everything. It's also why the AI investigation works. Unlike Grafana's fragmented LGTM stack, the context is already unified when the alert fires, not assembled by hand across disconnected tools.

Native Prometheus and PromQL support in Elasticsearch

Most SRE teams aren't running a clean, single-format telemetry pipeline. Prometheus is deeply embedded in applications, services, platforms, and automations. Migrating metrics backends historically meant rewriting queries, rebuilding dashboards, and retraining engineers — enough friction that teams stay on platforms they've outgrown rather than go through it.

Elasticsearch metrics has removed most of that friction. Prometheus metrics arrive via Prometheus Remote Write and land in the same columnar store without semantic changes, preserving full metric fidelity end to end. Point them at Elasticsearch instead of Mimir and the data flows. No translation layer, no changes to existing scrape configs.

PromQL now works natively in Kibana, so engineers who live in PromQL don't have to change how they work. Existing PromQL queries, dashboards, and alert rules migrate into Kibana directly. 

PromQL queries work unchanged on Elasticsearch

If your team already writes PromQL, nothing needs to change. These queries run as-is against Elasticsearch as your backend — copy, paste, and go.

CPU usage rate (container-level) The per-second CPU rate across containers, grouped by pod. Useful for spotting which pods are burning CPU during an incident.

PROMQL sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

Memory working set (container-level) Current memory in active use per container — the number that matters for OOM risk, not total allocated memory.

PROMQL sum by (container) (avg_over_time(container_memory_working_set_bytes[5m]))

HTTP request rate (application-level) Per-second request throughput grouped by instance. A standard first signal when investigating latency or error spikes.

PROMQL sum by (instance) (rate(http_requests_total[5m]))

All three follow standard PromQL syntax. If you use Elasticsearch as your backend, they run without modification. For the full syntax reference and what's covered, see the PromQL support documentation.

The native Prometheus API makes Elasticsearch a fully Prometheus-compatible backend. Any Prometheus-compatible frontend (Grafana included) can query Elasticsearch directly, so teams that want to keep Grafana as their visualization layer while consolidating onto Elasticsearch can do exactly that without modifying existing dashboards or alert rules.

When SREs need to go deeper than PromQL allows, ES|QL works across metrics, logs, and traces in a single interface. The TS command handles the time series specifics: counter rates, gauge averages, window functions, and multilevel aggregations across high-cardinality dimensions. The same query that pulls a CPU counter rate can join against logs from the same host and surface the deployment event that preceded the spike. No tool switching, no new query language. The query language, the dashboards, the alert rules, the visualization layer — all of it carries over. The only thing that changes is that Elasticsearch is the single backend powering everything.

Elastic Observability: out-of-the-box dashboards, alerts, and infrastructure content

Most Observability vendors require you to build everything from scratch. Elastic Observability has reduced this need across three areas:

Metrics exploration in Discover. The new Elasticsearch metrics exploration experience lets SREs explore metrics in the same interface used for logs — no tab switching, no duplicate queries. Connect an OTel pipeline or Prometheus scrape config, open Streams, and every metric in the data stream renders as a time series chart immediately. No dashboard to build, no query to write. This is where teams can validate data, spot patterns, and start building alerts and SLOs from a live view of what's flowing and cross correlate with logs, traces and other indexed data in Elasticsearch.

Dashboards. Kibana dashboards have gained collapsible panels with lazy loading, so panels that aren't immediately visible don't generate queries until they're needed and ES|QL control variables that let SREs manipulate visualizations through dropdowns without writing new queries. Dashboards-as-code is also shipping, enabling version-controlled dashboard definitions that can be templated, shared, and deployed programmatically across environments.

Out-of-the-box infrastructure content.  Elastic is shipping with two new infrastructure OOTB experiences:

• The new Kubernetes integration ships with hierarchical dashboards, alert rule templates, ML anomaly detection jobs, and the context and prompts needed for AI-assisted root cause analysis — all pre-configured and ready the moment data starts flowing. 

• AWS infrastructure monitoring follows the same pattern: OOTB content for core AWS services activates at ingest, so teams aren't starting from scratch every time a new service or account comes online. The same approach extends to databases and other core infrastructure — the platform arrives opinionated, not blank.

Agentic investigations across your infrastructure with Elastic Observability

Elasticsearch correlates metrics, logs, and traces in a single backend, so the investigation context is assembled before an engineer is paged.

The hard part is 2am. An RDS instance hitting connection limits, starving services upstream. An Auto Scaling group failing health checks for a reason buried in application logs. A pod restart cascading across a namespace.

In a Grafana LGTM stack, you're opening three tabs before you have enough context to form a hypothesis.

In Datadog, the context is unified but the AI is a black box: no BYO-LLM, no data residency options.

In Elastic, metrics, logs, and traces share a single backend and a common schema, so the investigation context is already assembled when the alert fires — no manual correlation across tools, no context lost in translation between query languages. ML anomaly detection runs automatically against infrastructure metrics (Kubernetes, AWS, databases), so the investigation starts from a scored anomaly with context about what's typical, what changed, and how severe the deviation is, not just a raw threshold breach.

When an alert fires, Elastic's investigation workflow correlates signals, assembles root cause context, and surfaces recommended next steps before anyone is paged. The agentic Kubernetes observability post walks through a complete example end to end. The EKS troubleshooting walkthrough shows how Agent Builder and MCP work together for a full root cause loop across EC2, EKS, and related AWS services.

In addition to investigating issues in Elastic Observability, you can use Claude, Cursor, VS Code, or your favorite tool to analyze issues using MCP Apps and agent skills from Elastic. The Observability MCP App extends the analysis to wherever your team already works. If your team investigates in Claude, Cursor, or VS Code, the same investigation capabilities (infrastructure health rollup, service dependency graph, anomaly detail, blast radius analysis) render as interactive views directly in the conversation. Neither Grafana nor Datadog offer this.

• Observability MCP App — Connects Claude, Cursor, VS Code, or any MCP-compatible tool directly to your Elasticsearch data, so infrastructure health, service dependencies, and anomaly context surface as interactive views inside the conversation without leaving your tool of choice. See how it works with Kubernetes.

• Agent Skills — Pre-built skills for Kubernetes, AWS, and other core infrastructure let any agent — in Elastic or your own — run structured investigations against your observability data without custom prompt engineering. Drop them into Claude, Cursor, or your own agent pipeline and they work out of the box. Explore the observability skills or browse the skills library on GitHub.

Migrating from Datadog or Grafana to Elastic Observability

The most common reason SRE teams don't switch observability platforms is migration. Moving years of alert rules, hundreds of dashboards, and runbook-embedded PromQL queries is a daunting operational task, and the cost of maintaining parallel stacks while doing it compounds every day.

The Observability Migration Platform handles the translation automatically. Point the CLI or Claude/Cursor (with Elastic’s agent skills) at your Datadog org or Grafana instance and it converts supported dashboards, alert rules, and PromQL queries into Kibana-native outputs. The tool allows you to see what was fully migrated, what needed tweaks and what is needed from you to migrate everything. You move what you've already built.

On the ingest side, Prometheus Remote Write means the pipeline requires no changes. Scrape configs point to Elasticsearch instead of another Prometheus-compatible backend and the data lands in the same columnar store. Workflows, queries, and alert configurations carry over without change. For teams that want to keep Grafana as a visualization layer during or after migration, the native Prometheus API and PromQL support in Kibana mean the transition can be phased rather than cut over all at once.

Elasticsearch as a backend for Grafana

For teams not ready to leave Grafana, replacing the backend is a migration path in its own right, and there are two ways to do it depending on your workflow.

If your team runs Prometheus today, the lowest-friction path is Grafana's Prometheus data source. Elasticsearch now exposes a native Prometheus-compatible API, so you can point Grafana's existing Prometheus plugin directly at Elasticsearch. No sidecars, no adapters, no pipeline changes required. Existing PromQL dashboards, alert rules, and variable dropdowns work without modification, including Grafana's Metrics Drilldown explorer. Add Elasticsearch as a remote_write target in your Prometheus config and swap the data source URL. That's the full migration for most teams. See the end-to-end setup guide.

For teams that want to go further and query logs, metrics, and traces together from a single Grafana query editor, the official Grafana Elasticsearch plugin now ships with ES|QL support. This unlocks cross-signal correlation directly in Grafana, with Elasticsearch handling all three data types in a unified columnar backend. See how to set it up.

Either way, keep Grafana, replace Mimir and Loki, and gain the full benefit of Elasticsearch's columnar storage and query performance underneath. Years of operational work, preserved. The migration that teams have been putting off becomes a backend swap.

What's GA and what's in tech preview

| Capability | Status | | ----------------------------------------- | ------------ | | Columnar metrics engine (TSDS) | GA | | ES|QL time series support | GA | | PromQL support in Kibana | GA | | Prometheus Remote Write ingest | GA | | Kubernetes infrastructure OOTB experience | GA | | AWS infrastructure OOTB experience | Tech Preview | | Observability MCP App | Tech Preview | | Agent skills | Tech Preview | | Observability Migration Platform | Tech Preview |

The individual posts linked throughout cover GA versus preview specifics and known limitations.

All of this (the columnar metrics engine, native PromQL, agentic investigations, and migration tooling) runs across Elastic's three deployment modes: serverless, Elastic Cloud, and self-managed. Datadog has no on-prem option; Grafana Cloud limits its highest-value features to hosted deployments. With Elastic, you choose where your data lives.

Elastic Observability: lower cost without dropping data

Modern cloud infrastructure broke the observability model built around separate tools for separate signals. The cost is real: duplicate tooling bills, manual correlation during incidents, and data dropped just to stay on budget.

A single backend that stores every signal efficiently means you keep what you need without the bill that usually comes with it. That's a different kind of conversation to have with finance: not "we had to drop data to stay on budget," but "here's what we found." The AI gets the full picture because there's only one picture, and the platform arrives with enough pre-built content to be useful on day one, not after weeks of dashboard toil.

That's possible because of how Elasticsearch is built differently from the platforms you're likely replacing:

• Columnar metrics storage stores stores metrics data highly efficiently in TSDS index mode. 

• Native Prometheus compatibility means existing scrape configs, PromQL queries, and dashboards work without rewriting.

• Unified metrics, logs, and traces in a single backend means investigation context is assembled at query time, not manually across tabs.

• Search and analytics in the same engine — an inverted index for logs, a columnar index for metrics, queried together with ES|QL.

• Agentic investigations that correlate signals, surface anomalies, and suggest remediation before anyone is paged.

• Serverless, Elastic Cloud, or self-managed — you choose where your data lives, which Datadog cannot offer.

The cost conversation with finance becomes about what you found, not what you spent.

Get started

• Start a free trial

• Elastic Observability documentation

• Elastic Observability Labs

Frequently asked questions

Is Elasticsearch now a production-ready metrics platform?

Yes. As of June 2026, Elasticsearch ships a rebuilt columnar storage engine purpose-built for time series data, native Prometheus Remote Write ingest, PromQL support in Kibana, ES|QL time series querying, and out-of-the-box infrastructure dashboards for Kubernetes and AWS. The columnar metrics engine, ES|QL time series support, PromQL, and Prometheus ingest are all generally available in Elastic Serverless and soon GA in Elastic Cloud Hosted.

How does Elasticsearch compare to Datadog for metrics cost?

In comparable metrics workloads, Elastic Observability Serverless costs significantly less than Datadog — in illustrative examples based on published list pricing, more than 50% less, and often closer to two-thirds less. The gap is structural: Datadog bills primarily per host, then adds charges for custom metrics and containers as instrumentation grows. The cost difference is largest for exactly the workloads where Datadog bills most: high-cardinality, densely instrumented environments like Kubernetes and OTel.

How does Elasticsearch metrics performance compare to Prometheus and Grafana Mimir?

ES|QL queries on Elasticsearch run up to 30× faster than Prometheus and Mimir on gauge averages and counter rates, including high-cardinality workloads. Elasticsearch stores OTel metrics at 3.75 bytes per data point; up to 2.5× more efficiently than Prometheus and 2× more efficiently than ClickHouse.

Can teams migrate from Datadog or Grafana to Elasticsearch without rebuilding everything?

Yes. Elastic's Observability Migration Platform converts Datadog and Grafana dashboards, alert rules, and migrates PromQL queries into Kibana as-is. Teams can also keep Grafana as a visualization layer while replacing the backend with Elasticsearch, using the native Prometheus API and PromQL support in Kibana.

What makes Elasticsearch different from Grafana for metrics observability?

Elasticsearch stores metrics, logs, and traces in a single unified backend with one query language (ES|QL), while Grafana's LGTM stack splits metrics (Mimir/Prometheus) and logs (Loki) across separate backends requiring separate query languages. Elasticsearch also ships agentic investigation capabilities, which includes AI Agent, Workflows, MCP App, and Agent skills, a more comprehensive set of capabilities than Grafana. 

Does Elasticsearch support Prometheus and PromQL natively?

Yes, in two distinct ways. First, Elasticsearch accepts Prometheus metrics via Prometheus Remote Write and exposes a native Prometheus-compatible API, so it can serve as a backend for any Prometheus-compatible frontend, including Grafana. Second, Kibana supports PromQL natively, meaning existing queries, dashboards, and alert rules run directly in Kibana without a translation layer or modification.

What infrastructure monitoring content ships out of the box with Elastic Observability?

Elastic ships pre-built dashboards, alert templates, and ML anomaly detection jobs across hundreds of infrastructure integrations covering hosts, containers, cloud services, databases, network devices, and more. For Kubernetes and AWS specifically, the platform also includes agentic investigation content such as agent skills and an Observability MCP App that lets teams run investigations directly from Claude, Cursor, or VS Code. All of this is available at ingest with no configuration required.

The processing pipeline is straightforward: The profiler samples every CPU core on the system at a fixed rate (19Hz by default), unwinds execution stacks, symbolizes the resulting stacktraces and ships the profiles to a backend like Elasticsearch.

And then… the user has to figure out what to do with them.

That last step is where continuous profiling has historically faced adoption challenges, as the path from "profiling is on" to "profiling is useful" is steeper than it should be.

Four barriers to adoption

• Storage cost: Full stacktraces, even after deduplication and clever storage schemas, are expensive to store at fleet scale. That cost makes continuous profiling an opt-in feature in practice: a lot of potential users never enable it, and the ones who do, tend to enable it only on a subset of hosts.

• Query friction: A normalized stacktrace schema is optimized for ingestion and storage but complicates ad-hoc questions. "How much CPU time does my service spend in TLS?" is a simple question that may require intricate ES|QL or custom code in order to be answered.

• AI-hostile data: Normalized stacktrace data (typically involving multiple levels of indirection) resists straightforward algorithmic analysis. LLMs in particular struggle with it and necessitate further data transformations into representations more amenable to LLM processing.

• UX barrier: Flamegraphs are extremely useful when you know how to read them but intimidating when you don't.

These four barriers compound: storage cost limits coverage, the UX barrier limits who benefits from coverage, query friction limits what questions users can ask and the AI-hostile data representation limits what the system can do when users don't know what questions to ask.

How profiling-derived metrics work: classify at the edge

The core idea is simple: instead of sending full stacktraces all the way to a backend and asking the user to make sense of them there, we classify and count at the edge, inside an OpenTelemetry Collector pipeline, and emit ordinary OpenTelemetry time-series counters. The profiling logic itself doesn't change; it's still the OpenTelemetry eBPF profiler running inside the OpenTelemetry Collector. All the new work happens in a stateless connector inside the Collector: the connector inspects each stacktrace produced by the profiler, classifies its frames into one or more categories and increments counters.

We've released profilingmetricsconnector as part of Elastic's opentelemetry-collector-components repository. It sits between the OpenTelemetry eBPF profiler receiver and any metrics exporter, and turns symbolized stacktraces into named, aggregated counters with attributes.

Because the profilingmetricsconnector lives inside the standard OpenTelemetry Collector pipeline, every metric it produces flows through the same processors as the rest of your telemetry. In the following example, the resourcedetectionprocessor enriches each counter with host-derived attributes.

connectors:
  profilingmetrics:
    flush_interval: 30s

receivers:
  profiling: {}

exporters:
  elasticsearch:
    endpoints:
      - # ENDPOINT
    api_key: # API_KEY
    mapping:
      mode: otel

processors:
  resourcedetection:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: false
        host.arch:
          enabled: true
        os.description:
          enabled: true
        os.type:
          enabled: true

service:
  pipelines:
    profiles:
      receivers: [ profiling ]
      exporters: [ profilingmetrics ]
    metrics:
      receivers: [ profilingmetrics ]
      processors: [resourcedetection]
      exporters: [ elasticsearch ]

Profiling-derived CPU metrics: what gets emitted

The connector ships with a set of pre-baked counters built from useful classification rules. Each metric is a count of stacktrace samples whose leaf frame matched a particular category, with the frequency value standing in for CPU consumption.

| Metric | Classifies | Attached metadata | |---|---|---| | kernel.count | Kernel leaf frames | syscall, category (disk/rw, ipc/rw, network/{tcp,udp,other}/rw, memory, synchronization, …) | | native.count | Native C/C++/Rust leaf frames | shared library name (libcrypto, libclrjit, libsystemd, …) | | hotspot.count, go.count, python.count, … | Runtime-specific leaf frames | runtime-specific attributes |

The kernel categorization is worth a closer look as a modern Linux kernel has more than 400 system calls. However, most of what shows up in CPU stacktraces falls into a handful of subsystems: filesystem read/write, network read/write, memory management, scheduling, synchronization. Some syscalls (e.g. read, write) are ambiguous on their own and only become specific when one examines more frames down the stack: ext4_file_read_iter points to filesystem, tcp_v4_rcv to network. The connector handles this disambiguation as part of frame iteration.

Native frames typically lack symbolic information beyond shared library names, but those names are still informative: libssl and libcrypto mean cryptographic work as part of OpenSSL or one of its variants; libz means compression; libclrjit means the .NET JIT is busy. We don't need to enumerate libraries statically as the connector dynamically generates shlib_name attribute values using the trimmed library name (e.g. libssl not libssl.so.3) for clean cardinality.

Currently, for each stacktrace, the connector computes a Self CPU count (the leaf frame matched the category) corresponding to exclusive CPU usage. A complication exists for fine-grained kernel categories like network/tcp/write where the actual leaf frame is usually a device-driver call that we can't meaningfully match. We deal with that by trying to match frames further up the stack (e.g. tcp_sendmsg is enough to correctly classify the sample).

Users can also add their own categories by specifying a frame pattern (e.g. a function or package) and the connector will generate counters for them.

Benefits of profiling-derived metrics for observability

This shift looks small from the outside — "we're emitting counters" — but it changes almost everything about how profiling fits into an observability stack.

• Orders of magnitude less storage: A counter aggregated over a 5-second (or 30-second or one-minute) window is dramatically cheaper than the full stacktraces it distills. The pre-aggregation interval is configurable with the trade-off being time resolution rather than categorization fidelity. For most "where is my CPU being spent?" questions, 30 seconds is plenty.

• On by default: Because the storage cost is now in line with regular metrics, profiling-derived metrics can be on for everyone, on every host, from the moment the profiler is deployed. Users get a CPU breakdown by runtime, syscall, kernel category and shared library on day one.

• Standard dashboards: These are ordinary OpenTelemetry time-series counters and can be visualized ad-hoc using stacked bar graphs, pie charts, top-N panels or any other visualization Kibana supports out of the box. The same Lens and TSDB-backed views for application metrics work here.

• AI and query-friendly: Standard time-series data is trivially consumable by ES|QL, ML jobs, anomaly detectors and by LLMs. "Show me the top services by network/udp/write time, filtered to the payments namespace, over the last six hours" is one query that is not only simple for the system to answer but also simple for an LLM to generate.

• Cross-signal correlation: Because the metrics flow through the standard OpenTelemetry Collector pipeline, they pick up the same resource attributes (e.g. service.name, k8s.pod.name, host.name, deployment.environment) that logs, other metrics and traces already carry.

• Instant value, with a path to more detail: A user who just wants to know "what's burning my CPU?" gets a meaningful answer without ever opening a flamegraph. A user who wants to dig deeper still has the full eBPF profiler underneath, ready to hand back complete stacktraces when they're warranted.

User-programmable profiling and adaptive sampling

The longer-term direction is for the profiler to stop being something users consume and start being something they program. User-defined metrics are the first step in this direction, complemented by on-demand (full) profiling and adaptive sampling.

Profiling-derived metrics or other signals can act as a trigger for on-demand profiling where the system enables full profiling on a specific host or service to capture complete stacktraces. In that way, the full profiling processing and storage cost is paid only when it matters.

We can apply the same idea to the sampling rate. 19Hz is a sensible baseline for steady state but when the metrics signal an interesting event or an anomaly, the system can automatically ramp to 100Hz or higher to capture high-fidelity data for the time window during which it's relevant. It can then ramp down to baseline.

How profiling-derived metrics enable self-driving observability

Most observability stacks today use an open-loop model: the profiler emits data with a fixed configuration. Then a human looks at flamegraphs and dashboards, potentially correlates with logs, other metrics and traces, forms a hypothesis and triggers a deeper investigation. Every link in this chain requires a human decision. Nothing feeds back into the profiler at speed and the system cannot act on its own observations.

Profiling-derived metrics close that loop.

1. A "significant host events" metric, an anomaly on network/udp/write or a spike in native.count/libz: something crosses a threshold.

2. The profiler adjusts in response: sampling rate increases, full profiling turns on for the affected hosts.

3. The richer data is correlated against logs, traces, and other metrics by an LLM, by a human or both. The same resource attributes that make cross-signal correlation easy for the user make it easy for the system.

4. A root cause is identified. A remediation is suggested or applied. The metric returns to baseline and the loop continues.

This is what we mean when we talk about self-driving observability. The profiler is no longer just an instrument that someone wields. It is the sensory organ of an autonomous feedback loop: a system that observes itself, decides what to look at more closely and adjusts its own configuration in response to what it sees.

What's next: inclusive CPU, off-CPU metrics, and runtime-specific profiling

Any piece of data visible in a stacktrace can be a metric source and several extensions are already on the roadmap.

• Inclusive-CPU metrics: Today's pre-baked counters attribute CPU at the leaf frame (exclusive-CPU). Inclusive-CPU metrics will attribute the entire call chain which is useful when you care about the total cost of a function call — the function plus everything it transitively calls — not just the work done directly in its own body.

• Runtime-specific metrics: GC time per runtime, JSON/Protobuf serialization, RPC frameworks, FFI boundaries. The kinds of questions every team eventually asks about their language runtime, answered by default.

• Off-CPU metrics: On-CPU profiling tells you where you're spending CPU but Off-CPU profiling tells you where you're not (e.g. blocked on I/O, locks). The same classification logic applies, with the only change being the source signal.

Profiling-derived metrics are an active area of work within Elastic and the profilingmetricsconnector is the place to start if you want to play with this today. A ready-made Kibana integration ships dashboards for all the metrics described above.

If you're already using Elastic's continuous profiling, expect these metrics to show up as first-class citizens in the Elastic stack. If you're not, this is a very low-friction way in as no flamegraph expertise is required and storage cost is minimal.

The flamegraph isn't going anywhere, but for the first time, it isn't the only way profiling yields results.

This post explains how the query, discovery, and metadata endpoints build on the earlier ingest and query work to form that API surface. Companion posts go deeper on individual pieces:

• Native PromQL support in ES|QL covers how PromQL queries are translated into ES|QL execution plans.
• Ship Prometheus Metrics to Elasticsearch with Remote Write covers ingestion setup.
• How Prometheus Remote Write Ingestion Works in Elasticsearch covers the remote write internals.

This is still a work in progress. The sections below call out what is supported today and which parts are still evolving.

The API surface

Today, the Prometheus-compatible API surface falls into three groups.

Query endpoints

The query endpoints let Prometheus-compatible clients evaluate PromQL expressions:

• GET /_prometheus/api/v1/query_range evaluates a PromQL expression over a time window (matrix results).
• GET /_prometheus/api/v1/query evaluates at a single point in time (vector results). Currently implemented as a short range query that returns the last sample.

Only GET is supported for query endpoints today. Some clients default to POST, so you may need to configure them to use GET. The Prometheus POST convention uses application/x-www-form-urlencoded bodies, which Elasticsearch's HTTP layer rejects as a CSRF safeguard before the request ever reaches the handler.

For the full PromQL coverage status, see the companion post on PromQL in ES|QL.

Metadata endpoints

The metadata endpoints serve the discovery information that clients need for autocomplete, variable dropdowns, and metric browsing.

The series, labels, and label values endpoints all accept match[] selectors and a time range (start/end). The match[] parameter takes a Prometheus series selector like http_requests_total{job="api"} and restricts the response to time series that match. This keeps responses fast and relevant on clusters with large numbers of metrics. For example:

GET /_prometheus/api/v1/series?match[]=http_requests_total{job="api"}
GET /_prometheus/api/v1/labels?match[]=http_requests_total
GET /_prometheus/api/v1/label/instance/values?match[]=http_requests_total{job="api"}

The first returns all series for http_requests_total where job="api", with their full label sets. The second returns only the label names that exist on http_requests_total series. The third returns only the instance values that appear on matching series.

GET /_prometheus/api/v1/metadata is different: it returns type and unit for each metric, optionally filtered by name via a metric parameter.

GET /_prometheus/api/v1/metadata?metric=http_requests_total

It does not accept match[] selectors or a time range. In Prometheus, metadata is collected from active scrape targets (the HELP, TYPE, and UNIT lines they expose), so the response does not involve a data scan. Elasticsearch does not have a dedicated metadata store like that, so the current implementation discovers metric metadata by visiting time series data from the last 24 hours. This keeps the query fast without requiring a full index scan. That 24-hour lookback is fixed today: the Prometheus metadata API does not expose start or end parameters that Elasticsearch could use to make it user-adjustable.

How the metadata endpoints work under the hood, including the TS_INFO and METRICS_INFO commands that power them, is covered below.

Index pre-filtering

All query and metadata endpoints accept an optional {index} path segment after /_prometheus/:

GET /_prometheus/metrics-prod-*/api/v1/query_range?query=up&start=...&end=...

This restricts which Elasticsearch indices the query runs against before any expression evaluation begins. On clusters with many data streams across teams or environments, this avoids scanning unrelated indices and can significantly reduce query latency. You can configure separate data sources per index pattern to give teams scoped access to their own metrics.

A note about Remote Write

For ingestion, Elasticsearch also exposes the standard Prometheus Remote Write endpoint:

• POST /_prometheus/api/v1/write ingests time series via the Prometheus Remote Write v1 protocol. v2 is not yet supported.

Remote Write writes into Elasticsearch's existing time series data streams (TSDS), not a separate Prometheus-specific storage layer. Prometheus labels become TSDS dimensions, and metric names become fields in the index mapping. The remote write architecture post covers the full mapping in detail, including how metric types are inferred and how labels are stored with a labels. prefix.

How it works

Under the hood, all endpoints work the same way: parse the incoming HTTP parameters, build an ES|QL query plan, execute it against time series data streams, and convert the columnar result back into the JSON format Prometheus clients expect.

TS_INFO and METRICS_INFO

The metadata endpoints need to answer questions like "what labels exist?" or "what metric types are defined?" across potentially millions of time series, without scanning every data point.

Internally, the Prometheus metadata endpoints answer those questions by building ES|QL plans around two new processing commands: METRICS_INFO and TS_INFO. You do not need to use these commands directly to use the Prometheus API, but they are the core execution primitives behind the metadata responses. Both work by visiting only one document per time series to extract its metadata, rather than scanning all samples. This means their cost scales with the number of distinct time series, not the number of data points.

METRICS_INFO returns one row per distinct metric with its name, type, unit, and associated dimension fields. TS_INFO is more granular: one row per (metric, time series) combination, including the actual dimension values as a JSON object.

TS metrics-*
| METRICS_INFO
| SORT metric_name

A dedicated blog post on TS_INFO and METRICS_INFO is coming soon, covering the two-phase execution model, how they scale, and how to use them directly in ES|QL queries beyond the Prometheus API.

How the metadata endpoints use them

Each metadata endpoint constructs an ES|QL plan with one of these commands at its core.

/api/v1/labels and /api/v1/series use TS_INFO, since they need per-time-series detail (which labels exist, which dimension values identify each series). /api/v1/metadata and /api/v1/label/__name__/values use METRICS_INFO, since they only need per-metric information (metric names, types, units).

/api/v1/label/{name}/values for regular labels (anything other than __name__) does not use either command. Regular labels like job or instance are actual dimension fields in the index, so the endpoint can query them directly with a group-by aggregation. When match[] selectors are provided, they are translated into a WHERE clause that filters the time series before the aggregation runs.

The __name__ label needs a different strategy because it is not always present as a dimension field. Prometheus Remote Write does store labels.__name__, but metrics ingested through other paths (OpenTelemetry, the bulk API) do not have it. The metric name is encoded in the field name itself (e.g., metrics.http_requests_total). You could look at the index mappings to enumerate field names, but mappings alone do not tell you which metric has which dimensions, and they cannot be filtered by label values from a match[] selector. METRICS_INFO can do both: it enumerates metric names across indices while respecting upstream WHERE filters.

In all cases, the API layer handles the translation back to Prometheus conventions: stripping the labels. and metrics. storage prefixes and synthesizing __name__ for non-Prometheus metrics that lack it.

In conclusion

The result: any Prometheus-compatible client can query and explore Elasticsearch metrics through endpoints it already understands. Remote Write metrics, OpenTelemetry metrics, and metrics indexed through other paths all show up through the same API, backed by the same TSDS indices.

All the Prometheus APIs mentioned here are available as tech preview in Elasticsearch Serverless today. For self-managed clusters and Elastic Cloud Hosted deployments, they will arrive as tech preview in Elasticsearch 9.4, with the exception of the GET /_prometheus/api/v1/metadata API which will arrive in 9.5. To try it locally, use start-local.

Frequently asked questions

Do the Prometheus-compatible query and metadata APIs only work with data that is ingested via Remote Write? No. The Prometheus-compatible API runs on top of Elasticsearch any time series data stream (TSDS). The same indices that hold metrics from Prometheus Remote Write, OpenTelemetry, or the Bulk API are queried directly through PromQL, with no extra storage layer to operate.

Why do my Prometheus clients fail when calling the query endpoints with POST? Today only GET is supported on the Prometheus query endpoints. Some clients default to POST with application/x-www-form-urlencoded, which Elasticsearch's HTTP layer rejects as a CSRF safeguard before the request reaches the handler. Configure the client to use GET.

Can I scope a Prometheus query to a subset of indices in Elasticsearch? Yes. Every query and metadata endpoint accepts an optional {index} segment, like /_prometheus/metrics-prod-*/api/v1/query_range. This restricts the query to the matching indices before evaluation, which avoids scanning unrelated data.

How does the native Prometheus API compare to running a dedicated Prometheus server? Elasticsearch consolidates Prometheus Remote Write, OpenTelemetry, and other ingestion paths into one TSDS-backed store, queryable through PromQL or ES|QL. You keep Prometheus client compatibility while reusing existing Elasticsearch capabilities like long-term storage, ILM, and correlation with logs and traces stored in the same cluster.

Histogram metrics

When dealing with metrics (in OpenTelemetry or Prometheus, for instance), counters and gauges are the most common metric types. Gauges allow you to monitor values that rise or fall (e.g., CPU utilization). Counters allow you to, well, count things, such as the total number of HTTP requests your service is handling. Counters normally just increase in value, with a few exceptions when they reset, like when a server reboots.

In the case of counters, you can additionally collect a counter measuring the total sum of your HTTP response times, which allows you to derive the average response time by dividing that sum by the total number of requests. However, average response times provide limited insights into the collected data and the system behavior. The best insights are gained by analyzing the collected metric distribution, e.g., through median and percentile calculations. This is where counters fall short.

In the past, workarounds have been applied: For example, classic Prometheus-style histograms attempt to capture the distribution using a set of counters. By defining fixed buckets (e.g., one for response times in the range [0s, 1s), one for [1s, 4s), and so on) and associating a counter with each, we can at least estimate percentiles broadly. However, the key problem here is that we have to know the distribution of our data up front to properly define these buckets.

To that end, the OpenTelemetry community has come up with a better solution: exponential histograms. Exponential histograms assign collected values to buckets, just like classic Prometheus-style histograms. The key differentiator is that these buckets vary dynamically based on the collected values. The name "exponential" comes from the fact that the bucket sizes increase exponentially: we use small buckets for small values and wider buckets for larger values. You can find an excellent introduction in the OpenTelemetry exponential histograms introduction.

Note that in addition to classic histograms, Prometheus also added native histograms, which directly map to OTel exponential histograms. Native histograms have their own PromQL syntax. We are actively working on adding support for that syntax to the Elasticsearch PromQL implementation, so that you can directly query exponential histograms using PromQL.

Demo setup

Let's start by collecting some histogram metrics to show how they can be stored and analyzed in Elasticsearch using ES|QL.

We'll focus on a Java JVM metric: garbage collection durations. OpenTelemetry defines the jvm.gc.duration, which is a histogram-typed metric. The OpenTelemetry Java agent natively supports collecting this metric.

We'll spin up a JVM running a Renaissance benchmark to put it under stress. We'll start that JVM with the vanilla OpenTelemetry Java agent attached and have it send the metrics directly to Elasticsearch.

You can find the ready-to-run Docker-compose file here. You'll just need to insert your Elasticsearch OTLP/HTTP endpoint and API key in the docker-compose.yml:

OTEL_EXPORTER_OTLP_ENDPOINT: https://<elasticsearch url>/_otlp
OTEL_EXPORTER_OTLP_HEADERS: "Authorization=ApiKey <base64 API key>"

Note that you don't have to use this demo setup. We even encourage you to try it with your own application. Here are the other important OpenTelemetry agent settings the demo already includes, which you should include too if you're bringing your own app:

OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE: delta
OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION: BASE2_EXPONENTIAL_BUCKET_HISTOGRAM
OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_ENABLED: "true"

Let's step through them:

• Temporality preference: OpenTelemetry supports both cumulative and delta-based histograms. Cumulative means that the histogram is only cleared after an application restart, while delta clears it after each export. At the time of writing, Elasticsearch only supports delta temporality for histograms. We are actively working on supporting cumulative histograms as well.
• Default Histogram Aggregation: By default, OpenTelemetry exports histograms in the Prometheus-style fixed bucket format. Since we want to reap the benefits of exponential histograms, we tell the agent to use them instead.
• Runtime Telemetry enabled: This tells the agent to actually collect the detailed JVM metrics, which include jvm.gc.duration.

Now we are ready to go! We'll let the application run in the background and switch over to Kibana to analyze the GC metric.

Querying with ES|QL

Now let's open up Kibana and navigate to "Discover". There we'll switch to ES|QL mode, and start querying the collected data:

TS metrics-* | STATS COUNT(jvm.gc.duration)

As a response, we now see the metric panel shown below. If you don't see any data, make sure to double-check the Kibana time range filter.

This number represents the total number of garbage collection operations that happened in our test application during the selected time frame.

Similarly, we can query the total time spent on those garbage collection operations:

TS metrics-* | STATS SUM(jvm.gc.duration)

So we have roughly 270k garbage collections, which in total took 713 seconds. Given these two numbers, we can now compute the average if we are still fluent in primary school-level math. Even if not, you can just let ES|QL do that for you:

TS metrics-* | STATS AVG(jvm.gc.duration)

Now we know that the average garbage collection operation took about 3 milliseconds. However, Java experts might know that there are different kinds of garbage collections happening, which can have significantly different pause times. Fortunately the OpenTelemetry metric comes with attributes, which allow us to slice the data accordingly:

TS metrics-* | STATS AVG(jvm.gc.duration) BY jvm.gc.action

As expected, major garbage collections take a lot more time per collection than minor ones, at least on average. So far, we have done nothing you couldn't also achieve by just using counters. Let's now use histograms to understand the actual distribution of the GC latency. We'll look at the data over time (by grouping using TBUCKET) and focus on the major garbage collections:

TS metrics-*
| WHERE jvm.gc.action == "end of major GC"
| STATS MAX(jvm.gc.duration),
        PERCENTILE(jvm.gc.duration, 99),
        MEDIAN(jvm.gc.duration),
        MIN(jvm.gc.duration)
 BY TBUCKET(100)

The graph now shows us the minimum, maximum, median and 99th percentile for major garbage collections. Note that we aren't bound to only querying the median and the 99th percentile. We can query any percentile we'd like to see, as these are estimated at query time from the raw exponential histograms.

A note on backwards compatibility

So far, we have seen how you can use the new shiny toy in Elasticsearch and ES|QL: exponential histograms. However, since this has just reached general availability (GA) in the 9.4 release, what about your historical data?

Before exponential histograms were added, Elasticsearch was already capable of storing OpenTelemetry histograms in the histogram field type. To do so, we converted them to a different data structure supported by the histogram field type: T-Digest. T-Digest provides good accuracy for extreme percentiles (e.g., 99th percentile) at the cost of accuracy for percentiles in the middle of the distribution, such as the median. In contrast, exponential histograms provide a guaranteed upper bound on the relative error for every percentile. As conversions always introduce errors, we are happy to now have native support for exponential histograms, allowing you to collect and analyze your metrics end-to-end without unnecessary conversions.

But still, what should you do if you have historical data and still want to query it? Thanks to ES|QL union types, the answer is actually easy: You just have to add a ::exponential_histogram suffix to the histogram metrics in your queries:

TS metrics-* | STATS AVG(jvm.gc.duration::exponential_histogram)

When this query encounters histogram fields, it will attempt to convert them to exponential histograms. When operating on exponential_histogram fields, the ::exponential_histogram cast has no effect. Note that this also works with mixed data sets: if your backing indices use both types, the query will just do the right thing.

So if you are building queries or dashboards that you expect to run on pre-9.4 ingested data, we recommend that you simply add: ::exponential_histogram casts.

Wrapping up

Native support for OpenTelemetry exponential histograms in Elasticsearch gives you better metric fidelity and more flexible analysis in ES|QL. In this blog post, we have shown you how to easily ingest and analyze your histogram metrics with ES|QL using various aggregations and the impact exponential histograms have.

Exponential histograms are generally available in Elasticsearch basic starting with the 9.4.0 release. They will be available in Elastic Cloud Serverless a few weeks after the 9.4.0 release, once mOTLP (the managed observability OTLP intake) switches to use the Elasticsearch OTLP endpoint. We'll update this blog post and add a note on the Elastic Cloud Serverless release notes when that happens.

Frequently asked questions

How do I query OpenTelemetry histogram metrics in Elasticsearch? Elasticsearch 9.4 natively stores OpenTelemetry exponential histograms and supports querying them in ES|QL. You can use standard aggregation functions like AVG, SUM, COUNT, PERCENTILE, MEDIAN, MIN, and MAX directly on histogram fields. Just send your OTel metrics to the Elasticsearch OTLP/HTTP endpoint and query them in Kibana Discover using ES|QL.

What's the difference between exponential histograms and classic Prometheus-style histograms? Classic Prometheus-style histograms require you to predefine fixed buckets, which means you need to know your data distribution upfront. Exponential histograms dynamically adapt their bucket boundaries based on collected values, giving you accurate percentile estimates without any upfront configuration. This makes them far more flexible for real-world workloads where distributions vary.

Why are exponential histograms better than T-Digest for metric storage? Before 9.4, Elasticsearch converted OTel histograms to T-Digest for storage. T-Digest provides good accuracy for extreme percentiles (like p99) but loses accuracy for mid-range percentiles like the median. Exponential histograms provide a guaranteed upper bound on relative error for every percentile, and native storage eliminates the lossy conversion step entirely. Also T-Digests don't support cumulative temporality.

Can I query historical histogram data after upgrading to Elasticsearch 9.4? Yes. If you have older data stored in the classic histogram field type, you can query it alongside new exponential_histogram data by adding a ::exponential_histogram cast to your ES|QL queries. ES|QL union types handle the conversion transparently, even across mixed indices.

How do I send OpenTelemetry exponential histograms to Elasticsearch? Configure your OpenTelemetry SDK or agent to use delta temporality (OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE: delta) and exponential bucket histograms (OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION: BASE2_EXPONENTIAL_BUCKET_HISTOGRAM). Point the OTLP exporter at your Elasticsearch OTLP/HTTP endpoint and the histograms will be stored natively — no intermediate collector or conversion needed.

Does Elasticsearch support downsampling for histogram metrics? Yes. Starting with Elasticsearch 9.4, both the exponential_histogram and classic histogram field types are supported in time series data stream downsampling. This lets you retain long-term histogram data at reduced storage cost while still being able to query percentiles.

How does Elasticsearch's histogram support compare to other observability platforms? Most observability platforms either require fixed-bucket histograms (losing accuracy) or convert distributions to sketches on ingest (losing raw fidelity). Elasticsearch 9.4 stores OTel exponential histograms natively and lets you compute any percentile at query time using ES|QL, without predefining buckets or losing data to intermediate conversions.

Elasticsearch has supported storing metrics in time-series data streams (TSDS) since version 8.7. This offering mainly focused on storage gains as explained in an earlier blog post. Still, performance was not on par with specialized systems for storing and querying metrics, in terms of storage efficiency, indexing throughput and query latency.

In the past year, we revisited the storage layer, optimized ingestion for OTel metrics and extended the ES|QL compute engine with vectorized processing for time series data. These efforts led to substantial performance wins across the board, compared to earlier versions of TSDS:

1. Up to 6.6x improvement in storage efficiency, reaching 3.75 bytes per data point in OTel metrics
2. Up to 50% improvement in indexing throughput for OTel data
3. Up to 160x improvement in query latency, including blazing fast counter rate evaluation and window support in time series aggregations

Elasticsearch has thus become a leading columnar metrics engine, matching or exceeding the competition (like Prometheus, Mimir, and ClickHouse) in indexing throughput and exceeding it by up to 2.5x in storage efficiency and 30x in query performance. All while maintaining the ability to store logs and other data and fully use the rich querying capabilities of ES|QL (e.g. inline stats, lookup join) — which other PromQL-based systems lack. Elasticsearch can thus serve as a unified storage and query engine for all user data, with no compromises for metrics and observability applications.

How TSDS is organized

TSDS has the following properties that help improve the performance of time-series codecs and produce correct results when aggregating data points per time series:

• The metric name and the dimension names and values are used to calculate the _tsid, a unique identifier per time series.
• TSDS get sorted by [_tsid ascending, timestamp descending] order. Each time series is thus stored in sequence on disk, with newer data points appearing first. Since the _tsid is calculated over dimension values, the latter are also clustered on disk.
• Shard routing is based on _tsid, with each _tsid value appearing in one shard only.
• Backing indices are time-bound, with no overlap over time between them.

The rest of this post explains how we use these properties to improve storage, indexing, and query performance.

Storage optimizations

TSDS already achieved a very competitive storage footprint, reaching 0.9 bytes per data point, when it is possible to combine many metrics in a single doc, sharing the same dimension values. However, when most data points have a unique set of dimensions (which is typical for OTel or Prometheus metrics), docs end up containing a single data point. In this setup, storage required 25 bytes per data point, with dedicated metrics stores requiring less than 10 bytes per data point.

To further reduce the storage footprint, we applied a series of optimizations over the past year:

Replace inverted indices and BKD trees with doc value skippers

Elasticsearch creates inverted indices (for text values) or BKD trees (for numeric values) by default for all non-metric fields, i.e. for @timestamp and dimensions. These indices improve performance for queries including filters on these fields, but have significant impact to storage — effectively doubling the footprint for each field. More so, they are also processed during segment merging, increasing the cpu, memory and storage overhead and slowing down the system — especially in high ingest throughput scenarios, as is often the case with metrics.

Lucene has been extended with doc value skippers, a form of hierarchical sparse indices that store the minimum and maximum value of blocks of documents. Range queries can check these min and max values and ignore blocks that don't fall into the requested range. Skippers work particularly well on sorted fields. Since TSDS are sorted by [_tsid, timestamp desc], dimension values get also clustered on disk. It's therefore possible to replace indices on @timestamp and dimension fields with doc value skippers that amplify the columnar layout — each field stored in its own files, with no duplicate tracking of each doc for indexing purposes.

Doc value skippers have negligible storage overhead — replacing indices with them led to a reduction of 10 bytes out of the initial 25 bytes per data point in OTel. Moreover, they work very well in practice when queries include filters on time ranges or dimension values (including prefixes and regex) — there was no noticeable regression in query performance in our benchmarks when they replaced separate indices. Doc value skippers are enabled for TSDS by default since version 9.3.

Enable synthetic ids

The _id metadata field was another big contributor to the storage footprint. TSDS has already been extended to trim the doc values once they were no longer needed for replication, but the inverted index was kept around to efficiently support the id-based APIs (Get, Delete, Update).

The id value for TSDS is synthesized by combining the _tsid and @timestamp values that uniquely identify each data point. Since these fields are configured with doc value skippers, it's possible to replace the inverted index on _id with (a) retrieval of the _tsid and @timestamp value from the _id value, and (b) checks for matches using doc value skippers respectively. Care has to be taken to avoid expensive checks for duplicate ids during metric ingestion, with segment-level bloom-filters keeping the overhead at bay.

Supporting synthetic ids in metrics is a first for Elasticsearch. It led to a reduction of 5 bytes out of the initial 25 bytes per data point for OTel metrics, with no loss of functionality. Synthetic ids are enabled for TSDS by default in version 9.4. We plan to extend their uses in logs and other applications after further evaluation.

Trim sequence numbers

Sequence numbers are used as part of replication, but also to provide strong consistency semantics on doc modification operations through Optimistic Concurrency Control (OCC). While such semantics are applicable to certain scenarios, they don't fit in metrics where concurrent updates are very rare, with no practical need for guarding against concurrent operations on data points with matching ids. We therefore decided to disable the use of sequence numbers in all APIs, along with OCC support, for TSDS, in version 9.4. This leads to a substantial storage reduction of 4 bytes out of the initial 25 bytes per data point for OTel data, as there's no inverted index and sequence numbers get trimmed once no longer needed for replication. Update and delete by query operations are still supported, albeit with weaker consistency semantics.

If OCC is still deemed important for a particular metrics application, the old behavior can be restored by setting index.disable_sequence_numbers: false in the index template of the involved TSDS.

Use large numeric codec blocks

TSDS already uses an advanced codec, as explained in an earlier article. The codec works very well in most cases, but has poor performance in case of repeated sequences of keywords and numbers, leading to an inflated storage footprint for dimensions containing IP and MAC addresses. We identified that the existing logic for identifying repeated sequences requires larger codec blocks to work well, especially as the sequence length increases. After experimentation, the numeric block size was increased from 128 to 512 elements in version 9.3, leading to a reduction of 2 bytes out of the initial 25 bytes per data point for an OTel dataset containing IP and MAC addresses as dimensions. We're also working on a more configurable codec layout that will allow more flexibility with block sizes and other parameters, based on field type and cardinality.

Indexing throughput

Elasticsearch has support for bulk ingestion of documents. This entrypoint has long been optimized for leniency, ensuring that all docs get accepted. This flexibility, however, incurs additional processing cost during indexing. Metric applications proved good candidates for using different approaches to reduce this overhead, as explained below.

Introduce OTLP protobuf entrypoint

OTel metrics and Prometheus have established protocols for metrics ingestion, using protocol buffers. In the past, a translation step was required to convert collected protobuf messages to bulk requests that Elasticsearch can consume.

Elasticsearch was recently extended with endpoints accepting messages from OTel metrics collectors and over Prometheus remote write. Parsing and processing these (binary) messages is cheaper, compared to json parsing, while hash operation over dimensions for _tsid calculations get reused and amortized across more data points within a single protobuf message. Furthermore, _tsids get evaluated once per doc in the coordinator nodes and propagated to data nodes for indexing, thus deduplicating an expensive step per indexed doc. These improvements led to up to a 20% speedup in indexing throughput for OTel metrics. The OTLP entrypoint was added in version 9.2 (tech preview) and reached GA in version 9.3. We've added similar entrypoints for Prometheus remote write in version 9.4 (tech preview) and are actively working to cover OTel Logs and Traces.

Reduce indexing CPU with doc value skippers

In addition to a substantial storage footprint, inverted indices require a lot of cpu to build and reconstruct during segment merging. The use of doc value skippers in their place helps also reduce cpu load at ingestion and thus improves indexing throughput by 10%, a welcome bonus on top of the aforementioned storage wins.

Synthetic recovery source

The original source of a document, as provided at index time, is never stored for metrics. Still, Elasticsearch needed to temporarily store it for replication purposes. That changed in version 9.1, where the source gets synthesized on demand for replication purposes. This is known as synthetic recovery source and reduces disk I/O by 50%, with a significant impact to metrics indexing performance. Check out this article for more details.

Query execution

Replacing inverted indices with doc value skippers leads to a pure columnar storage layout for TSDS, with metric and dimension fields stored as Lucene doc values, each field encoded and compressed in their own file. Combined with the introduction of the ES|QL compute engine that uses vectorized execution internally, it became possible to introduce a fully columnar storage and query processing engine for metrics in Elasticsearch. We pushed this idea to the extreme and implemented a columnar metrics processing engine that comfortably outperforms dedicated metrics engines and other columnar stores in query performance.

Time series integration in compute engine

Time series processing is largely based on applying aggregation functions per time series (or _tsid), such as a gauge average or a counter rate. These partial results are then reduced by a secondary function to produce results for the grouping dimensions, e.g. per host and process. Observability dashboards are built on top of this execution model, providing summary views of how metrics evolve over time and allowing for quick deep-dives by filtering on dimension values and time ranges.

To support this execution model, we introduced the TS source command, providing a simple yet powerful syntax for executing such queries that combine an inner aggregation function per time series with an outer aggregation over the partial results of the former. For instance, the following query calculates the hourly sum of rate of search requests per host over the last day:

TS metrics
  | WHERE TRANGE(1d)
  | STATS SUM(RATE(search_requests)) BY TBUCKET(1h), host

To execute this query, the compute engine is aware of how data is stored and applies the inner aggregation function per _tsid value. Since data are sorted by _tsid, time series aggregation functions process metric values as they get fetched, until the _tsid changes or the timestamp belongs to the next time bucket. This leads to vectorized execution of these functions over the fetched columns of metric values, while dimension values are only fetched (once) when the _tsid changes. The evaluation of the secondary aggregation function is also efficient, with partial aggregates stored in arrays of primitive values that get populated when _tsid values change.

The compute engine has inherent support for parallel query evaluation, taking full advantage of the available processing cores. Time series aggregations fully use this feature and process data points in parallel as applicable, reducing response times through improved cpu utilization.

Time series processing in ES|QL was introduced in version 9.2 as tech preview and reaches GA in version 9.4. We expect all metrics applications to adopt it and benefit from the much improved query performance wins.

Zero-copy data decoding and loading

Vectorized processing of time series data delivered immediate performance wins (8x for some queries), compared to aggregations through the /_search API, but the performance was still inferior when compared to competitive metrics stores. Benchmarking and profiling showed that there were too many array copies within the compute engine, between data decoding and evaluation of aggregation functions. To that end, the following optimizations were introduced:

• The codec for TSDS was extended to decode on-disk data directly into primitive arrays inside blocks that the compute engine uses to evaluate time series aggregations. No additional copies required, as the compute engine can bulk-read these blocks and process their arrays, one column at a time.
• Blocks containing a single value N times are represented as constant blocks with these 2 values, as opposed to an array with length N, a form of in-memory run-length encoding. Filtering and aggregation operations were extended to efficiently consume these blocks. This reduced memory pressure and cpu overhead for the _tsid and dimension fields, as their values get clustered due to index sorting.
• Documents with null values for the aggregated metric fields are filtered out at the Lucene level, before they get decoded and copied into blocks.
• All filters and regular expressions on the timestamp and dimension fields get pushed down to Lucene that makes use of doc value skippers to efficiently filter out non-matching docs.

Combined, these optimizations led to query execution speedups exceeding 10x (totaling 80x when combined with the 8x speedup from vectorized execution). They were included since the introduction of the TS source command in version 9.2, and fine-tuned ever since.

Optimized counter rate evaluation

While most time series aggregations can be trivially parallelized and evaluated, rate evaluation of cumulative counters is tricky as it requires processing all data points in order to detect counter resets (e.g. when a host restarts). To address this, the compute engine uses the _tsid prefix to shard time series across threads. Care has been taken to assign in-order ranges of _tsid values to each thread, as opposed to hash-partitioning _tsids, so that each thread can scan on-disk data in order, still making use of efficient decoding and zero-copying into blocks. The performance wins are impressive, with rate evaluation performance far exceeding dedicated metrics stores as we shall see in the next section.

Another interesting problem for cumulative counters is how to properly calculate counter increases for the entire time bucket when there are no data points at the bucket boundary timestamps. Metrics systems often use extrapolation, extending the first and last data points of each time bucket to the boundaries, or calculate the delta between the last data point of adjacent buckets. We posit we can do better, by interpolating between the last data point of each bucket and the first of the next, to get an estimate of the value on each boundary. The delta is then calculated over the interpolated values of the lower and upper boundary of each time bucket.

Sliding window support

Elasticsearch has long supported aggregations bucketed by time, but it was not possible to extend the window of processed data beyond each time bucket. Using windows larger than the time bucket, e.g. a window of 5 minutes for per-minute bucketing, helps smoothen out spikes and observe the underlying trend per time series with reduced noise:

All time-series aggregation functions have been extended with window support, as an optional argument. In case the window is a multiple of the time bucket (e.g. 1h window with TBUCKET(5m)), the compute engine first aggregates data points over intervals matching the time bucket span, and then combines these partial results per window span. This 2-phase approach eliminates repeated scans of data points and makes optimal reuse of intermediate results, improving response times. Window support was introduced as tech preview in version 9.3 and reaches GA in version 9.4.

Efficient datetime rounding

Queries on time-series data commonly include time bucketing. While data points can be trivially assigned to sub-hour time buckets, larger buckets start interfering with issues like time zones, daylight savings, variable days per month etc. Elasticsearch has elaborate logic for datetime rounding that takes these peculiarities into account, but that has relatively high cpu cost when processing time series data.

To mitigate this, the compute engine has been extended to identify cases where simpler logic can be employed to assign data points to time buckets. For instance, it can identify when the buckets are sub-hour or when timezones and daylight savings don't affect a particular query, and switches to simple modulo operations for datetime rounding. This led to a further 30% improvement in response times for certain queries. This change is introduced in version 9.4.

Performance evaluation

To evaluate the performance of our offering and track how it evolves and improves over time, we focused on OTel metrics since (a) Open Telemetry is the industry standard for collecting metrics, with universal adoption by all cloud providers and (b) they lead to a storage layout with 1 metric per doc, a setup that traditionally hurt performance for Elasticsearch.

We rely on Metricsgenreceiver to generate datasets. This tool is inspired by TSBS, producing data simulating the data points collected by the OTel hostmetricreceiver. We used two datasets:

1. A low-cardinality setup, with 100 hosts sending metrics every 1s, containing 14k time series in total
2. A high-cardinality setup, with 10k hosts sending metrics every 10s, containing 1.4M time series in total

We benchmarked on single-node deployments on EC2, using c6i.4xlarge and c8g.8xlarge machines for the low- and high-cardinality datasets respectively.

For competitive comparison, we used Prometheus (v.3.11.1), Mimir (v.3.0.6.) and ClickHouse (v26.3.9.8-lts). Prometheus and Mimir have proper time series processing, e.g. for counter rate, whereas ClickHouse lacks such support and only provides approximate values at best (for instance, it can't track counter resets consistently). We still report response times for ClickHouse to showcase that, once we optimize Elasticsearch for columnar query processing, it can exceed competing columnar engines even when they don't process the data per time series as expected.

We strived to use the default configuration for every system (including Elasticsearch), without tweaking them to optimize performance for the particular workload. This helps understand the user experience when systems are deployed by novice users, without much experience (or time) to tweak before receiving metrics traffic and setting up dashboards. We focused on single-node runs to keep noise low and accommodate all systems (Prometheus doesn't offer a multi-node setup out of the box). Elasticsearch performance provably scales well with the number of nodes; we plan to share scalability results in a future post.

Storage efficiency and indexing throughput

Our efforts to improve storage efficiency paid big dividends. Performance on OTel metrics dropped from 25 to 3.75 bytes per data point, in a year. Such an improvement, on top of an offering already optimized for time series, is really impressive and very rare in the industry:

The competitive picture looks favorable at this point, with Elasticsearch:

• Slightly outperforming Mimir in storage efficiency and indexing throughput
• Outperforming Prometheus by 2.5x in storage efficiency and by a small margin in indexing throughput
• Outperforming ClickHouse by 2x in storage efficiency and by 40% in indexing throughput

Query performance

The novel columnar engine for metrics processing proves very efficient in practice. We used a mix of queries based on gauge averages and counter rates, the most common operations that require different optimization approaches. The queried interval was 4 hours of data, covering all time series per metric.

ClickHouse doesn't support time series aggregations, so the results have limited value and are not directly comparable to Prometheus or Mimir that natively support time series processing. We used the published guidelines to adjust each query to get similar results to the extent possible. The point is to show how our columnar engine compares to generic columnar stores.

Here is a summary of the results:

| Query type | vs Mimir | vs Prometheus | vs ClickHouse † | |---|---|-------------------|-------------------| | Gauge average | up to 30x faster | up to 30x faster | up to 8x faster | | Counter rate | up to 30x faster | up to 30x faster | up to 3.5x faster | | Prefix filter on host name | up to 5x faster | up to 5x faster | up to 3x faster | | Gauge average with window | up to 25x faster | up to 25x faster | up to 4x faster |

†ClickHouse lacks native support for time series aggregations and counter reset detection.

Gauge average

We compared performance of evaluating the per-host hourly average of average memory utilization per time series, using the following queries:

# PromQL
avg by (host.name) (avg_over_time(system.memory.utilization[1h]))

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| STATS AVG(AVG_OVER_TIME(system.memory.utilization)) BY host.name, TBUCKET(1h)

Elasticsearch comfortably outperforms the other systems by up to 30x, in both low and high cardinality datasets:

Counter rate

We next compared performance of evaluating the per-host hourly average of cpu rate, using the following queries:

# PromQL
avg by (host.name) (rate(system.cpu.time[1h]))

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| STATS AVG(RATE(system.cpu.time)) BY host.name, TBUCKET(1h)

Despite processing data points per time series in order, counter rate performance matches calculating gauge average (the involved time series have 6.6x more docs than the query above). Elasticsearch maintains its wide advantage compared to the other systems and outperforms Mimir and Prometheus by 30x in the low cardinality dataset and by 16x in the high cardinality one:

It's really impressive that, for the high cardinality dataset, Elasticsearch is able to process 4 hours of data for half a million time series in less than 2 seconds, while the other systems take more than 30 seconds, leading to unresponsive dashboards for such queries. ClickHouse is also slower, despite having no logic to detect counter resets and extrapolate/interpolate deltas across buckets.

Prefix filter on host name

We next compared performance of filtering on host names based on their prefix, using the following queries:

# PromQL
avg by (host_name)
  (avg_over_time(system_cpu_load_average_1m{host_name=~"host-.*"}[5m]))

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| WHERE host.name LIKE "host-*"
| STATS AVG(AVG_OVER_TIME(system.cpu.load_average.1m)) BY host.name, TBUCKET(5m)

Elasticsearch manages to maintain an advantage of up to 5x compared to the other systems, despite replacing the inverted index on host.name with a doc value skipper:

Gauge average with window

We compared the performance of time series aggregations with a window of 90 minutes and time buckets of 30 minutes, using the following queries:

# PromQL
avg by (host.name) (avg_over_time(system.memory.utilization[90m]))&step=30m

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| STATS AVG(AVG_OVER_TIME(system.memory.utilization, 90m))
    BY host.name, TBUCKET(30m)

Elasticsearch comfortably outperforms the other systems in both low and high cardinality datasets:

Elasticsearch maintains an advantage that reaches 25x for the low cardinality dataset and 8x for the high cardinality one. ClickHouse is outperformed by close to 4x, denoting the efficiency of our approach for windowed query operations.

What's next for Elasticsearch metrics

Elasticsearch has been extended with metrics storage and processing capabilities that outperform Prometheus, Mimir, and ClickHouse. We're making fast progress with supporting PromQL and Prometheus remote write, also available as tech preview in version 9.4. These extensions enable users familiar with Prometheus and relevant systems to switch their applications to Elasticsearch — no need to migrate existing dashboards. Since Prometheus integration reuses the same storage and query engine that has been presented in this article, the same performance wins are also expected for Prometheus. Furthermore, collected metrics can be queried with PromQL and ES|QL, side-by-side or in ES|QL query pipelines, further boosting the analytics capabilities far beyond what was conceivable so far with Prometheus-based systems.

The improvements in storage efficiency, indexing throughput and query performance are already impressive, but we're not done. We'll be introducing more refinements to the codec for time series data, further reducing bytes per data point. Batch processing of ingested metrics will be further improved, reducing synchronization overhead and redundant processing layers that are not needed for well-formatted collected metrics. We're also planning to make wider use of doc value skippers, storing pre-computed aggregates like sum and count per block of values, to shortcut data point loading and processing where applicable, as well as use more cpu-friendly partitioning and grouping operations.

Frequently asked questions

What is a columnar metrics engine and why does it matter? A columnar metrics engine stores each field in its own file rather than row-by-row, then processes queries by reading only the columns needed. For time series data, this means Elasticsearch can decode metric values, dimension fields, and timestamps independently, applying vectorized operations across each column. The result is faster aggregations and lower storage overhead compared to row-oriented stores.

How does Elasticsearch compare to Prometheus for time series metrics storage and querying? Elasticsearch stores OTel metrics at 3.75 bytes per data point in version 9.4, roughly 2.5x less than Prometheus. For queries, Elasticsearch outperforms Prometheus and Mimir by up to 30x in gauge average and counter rate benchmarks. For the high-cardinality dataset (1.4M time series), Elasticsearch processes 4 hours of data in under 2 seconds while Prometheus takes over 30 seconds.

What is Elasticsearch TSDS and when should I use it? TSDS (time-series data streams) is Elasticsearch's storage format for metrics and time series data. It sorts documents by time series identifier (_tsid) and timestamp, stores fields in columnar doc values, and uses specialized codecs for compression. Use TSDS for any metrics workload, particularly OpenTelemetry or Prometheus data, where storage efficiency and query speed matter.

What is the TS source command in ES|QL? TS is an ES|QL source command, GA in version 9.4, that executes time series queries using a two-level model: an inner aggregation per time series (such as RATE() or AVG_OVER_TIME()), then an outer aggregation over the results. The compute engine processes data in time series sort order, enabling vectorized and parallel execution. Example: TS metrics | STATS AVG(RATE(cpu.time)) BY host.name, TBUCKET(1h).

How did Elasticsearch go from 25 bytes to 3.75 bytes per OTel data point? Four storage changes contributed across versions 9.1 through 9.4: replacing inverted indices with doc value skippers (-10 bytes), enabling synthetic IDs (-5 bytes), trimming sequence numbers (-4 bytes), and increasing codec block size from 128 to 512 elements (-2 bytes). The result is a 6.7x reduction in storage footprint in roughly one year.

Can Elasticsearch replace Prometheus without migrating dashboards? Elasticsearch supports Prometheus remote write (tech preview, version 9.4) and PromQL queries (tech preview, version 9.4). Existing Grafana dashboards using PromQL can point to Elasticsearch with minor modifications, and we expect to offer a seamless migration experience when our Prometheus offering reaches GA. The same TSDS storage and ES|QL compute engine power both PromQL and ES|QL queries, so the performance improvements apply to both.

What are doc value skippers and why do they matter for metrics? Doc value skippers are Lucene index structures that store min/max values for blocks of documents. For TSDS, which sorts by _tsid and timestamp, they replace inverted indices on dimension fields and @timestamp. This reduces storage by up to 10 bytes per data point and cuts indexing CPU by about 10%, with no measured regression in query performance for time range and dimension filters.

Prerequisites

• Elastic Cloud hosted (9.3+) or serverless

• Claude Desktop

• An MCP server instrumented with the Elastic Distribution of OpenTelemetry (EDOT). We cover how to instrument one below.

The observability gap in MCP servers

MCP servers ship with no built-in observability, which means tool-call latency, errors, and performance baselines are invisible to developers. MCP (Model Context Protocol) servers are increasingly used as infrastructure for AI-powered applications, giving AI models access to databases, APIs, internal tools, and business data. The MCP SDK does not instrument any of it.

The gap shows up in three concrete ways:

• When a tool call takes 3 seconds, you don't know if the bottleneck is in your business logic, a downstream API, or the data layer.
• When a tool call fails, you get the error message but no context about what the server was doing before it failed.
• When you add a new tool, you have no baseline to compare its performance against.

These are the same problems any backend service faces. The answer is the same one backend developers have used for years: distributed tracing with OpenTelemetry.

MCP servers are standard programmatic processes. There is nothing special about them from an instrumentation perspective. You add the OTel SDK, define spans around your tool handlers, and ship traces to your backend. The only new part is knowing which span names and attributes to use so your traces are meaningful and consistent.

What we built

For this article, we use the @modelcontextprotocol/server-everything package, the official reference MCP server published by Anthropic. It ships with a set of tools that cover the common patterns you will find in real-world MCP servers: simple request/response, parameterized calls, long-running operations, and calls that return structured data.

The server exposes several tools. In this article we use three of them:

• echo: receives a string and returns it unchanged. A minimal request/response tool, useful for verifying that the instrumentation pipeline works end to end.

• get-sum: receives two numbers and returns their sum. Represents a parameterized tool with simple business logic.

• trigger-long-running-operation: starts a multi-step operation that takes several seconds to complete. Simulates tools that call downstream APIs or run expensive computations.

We instrumented it with EDOT Node.js (@elastic/opentelemetry-node), which is Elastic's distribution of the OpenTelemetry SDK. EDOT replaces the five or six individual OTel packages you would otherwise install and adds the elasticapm connector that the Kibana APM UI needs to build its service maps, transaction groupings, and latency charts. Without that connector, raw OTLP data arrives in Elasticsearch but the APM views have nothing to build from.

The architecture looks like this:

This is the loop: Claude executes tools, generates telemetry, and then uses a second MCP to analyze that telemetry. The observability data becomes something the AI can reason about, not just something that sits in a dashboard waiting for a human to check it.

Both MCP servers are active simultaneously in Claude Desktop. The instrumented MCP generates telemetry. The Agent Builder MCP lets us query it.

To connect Claude Desktop to both servers, the claude_desktop_config.json looks like this:

{
  "mcpServers": {
    "everything": {
      "command": "node",
      "args": [
        "--import",
        "/path/to/node_modules/@elastic/opentelemetry-node/import.mjs",
        "/path/to/everything/dist/index.js",
        "stdio"
      ],
      "env": {
        "OTEL_SERVICE_NAME": "everything-mcp-server",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "https://<your-otlp-endpoint>",
        "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=ApiKey <your-api-key>",
        "OTEL_LOG_LEVEL": "none"
      }
    },
    "elastic-agent-builder": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://<your-kibana-url>/api/agent_builder/mcp",
        "--header",
        "Authorization:ApiKey <your-api-key>"
      ]
    }
  }
}

The --import /path/to/@elastic/opentelemetry-node/import.mjs flag is all it takes for zero-code auto-instrumentation. But auto-instrumentation only captures HTTP calls, database queries, and other Node.js instrumented libraries. MCP tool calls are application logic, and application logic needs manual spans.

What traces MCP tool calls generate

The OpenTelemetry specification includes official semantic conventions for MCP. Following them means your traces are consistent, searchable by name across tools and teams, and compatible with any OTel-aware backend, including Elastic APM.

Span naming follows the pattern {mcp.method.name} {target}. For a tool call, this becomes tools/call echo or tools/call get-sum. This is what you will see as the transaction name in Kibana APM.

Key attributes on each span:

| Attribute | Value | Purpose | | ----------------------- | ---------------- | ------------------------- | | mcp.method.name | tools/call | The MCP protocol method | | gen_ai.tool.name | echo | The specific tool invoked | | gen_ai.operation.name | execute_tool | GenAI semantic convention | | error.type | error class name | Set only on failure |

The wrapper pattern that creates these spans looks like this:

const tracer = trace.getTracer('everything-mcp-server', '1.0.0');

function withToolSpan(toolName, fn) {
  return tracer.startActiveSpan(`tools/call ${toolName}`, (span) => {
    span.setAttribute('mcp.method.name', 'tools/call');
    span.setAttribute('gen_ai.tool.name', toolName);
    span.setAttribute('gen_ai.operation.name', 'execute_tool');

    try {
      const result = fn();
      span.end();
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      span.setAttribute('error.type', err.constructor.name);
      span.end();
      throw err;
    }
  });
}

Each tool handler wraps its logic in withToolSpan. The result is a named span in Elastic APM for every tool invocation, with duration, status, and error details attached.

Security note: The OTel spec defines two optional attributes for tool calls: gen_ai.tool.call.arguments and gen_ai.tool.call.result_. Both are flagged as potentially containing sensitive data. The_ get-env tool in the everything server is a good example of why this matters: it returns all environment variables, which may include API keys and credentials. Capture these attributes only if you have confirmed the data is safe to store in your observability backend, and consider masking or filtering at the SDK level before export.

Kibana APM: exploring MCP server performance

After starting Claude Desktop with both MCPs configured and triggering a few tool calls, the everything-mcp-server service appears in Kibana under Observability > Applications > Services Inventory:

Transactions view

Kibana groups traces by transaction name. Because we follow the semantic conventions, each tool gets its own row: tools/call echo, tools/call get-sum, tools/call trigger-long-running-operation. You can immediately see latency, throughput, and error rate per tool without any configuration.

This is where the value of consistent span naming becomes concrete. If you have five different developers adding tools to an MCP server and everyone follows tools/call {toolName}, the APM UI stays organized automatically.

Trace waterfall

Clicking on a specific trace shows the waterfall view. For a single tool call, the waterfall is straightforward: one span covering the full execution. If your tool handler makes downstream HTTP requests or database queries that are auto-instrumented, those appear as child spans. You can see exactly how much time was spent in business logic versus waiting for external calls.

Latency distribution

The latency chart shows p50, p95, and p99 distribution across all executions of a given tool. This makes it easy to distinguish between tools that are consistently fast and those that have occasional outliers. The trigger-long-running-operation tool, for example, shows a wide distribution depending on how many steps were requested: a useful baseline for understanding expected execution time ranges before setting alerts.

Error tracking

Failed tool calls appear in the Errors panel with their full stack trace, the span attributes attached at the time of failure, and a count of how many times the error has occurred. If you record the exception with span.recordException(err), Kibana links the error directly to the trace that produced it.

Closing the loop with the Agent Builder MCP

The Elastic Agent Builder MCP lets Claude query its own trace data from Elasticsearch in the same chat session that produced the traces. The Agent Builder MCP can query any Elasticsearch index the API key has access to, and APM traces are stored under .ds-traces-apm.otel-default-*. Granting the Agent Builder API key read access to those indices is what closes the loop: the agent that executed the tool calls can now reason about how they performed.

Here is what this looks like in practice. To generate traces, let's ask in Claude Desktop: "Use the echo tool to say hello, then use get-sum to add 1337 and 42, then run a long-running operation with 3 steps."

Claude executes three tool calls on the instrumented MCP. Three spans land in Elastic APM.

Now, without leaving the chat, let's try querying the traces by asking: "Search the APM trace data from the last 10 minutes. What tool calls were made, and how long did each one take?"

We can confirm the information against the services data:

Claude uses the Agent Builder MCP to run a query against the traces index. It returns the tool names, durations, and status from the actual trace data, then synthesizes an answer in natural language.

You can go further by asking:

• "Which of those tool calls had the highest p95 latency?"

• "Did any tool calls fail? If so, what was the error message?"

• "Compare the latency of the echo tool vs get-sum across all calls in the last hour."

Each of these questions translates into an ES|QL query via the Agent Builder's platform.core.execute_esql tool, run against the APM trace indices.

Why Elastic for MCP observability

The Agent Builder closes the loop: this is the part that is specific to the Elastic ecosystem. Because APM data lives in Elasticsearch, and Elasticsearch is queryable via the Agent Builder MCP, you can bring your AI agent's own observability data back into the conversation. Your AI can reflect on its own performance and spot anomalies.

APM UI built for distributed tracing: Kibana's APM interface is designed for exactly this kind of data: named transactions, trace waterfalls, latency percentiles, error tracking with stack traces, and service maps.

Managed OTLP endpoint: Elastic APM accepts OTLP directly since Elastic 8.x. You point OTEL_EXPORTER_OTLP_ENDPOINT at your APM server and it works.

EDOT simplifies the setup: the elasticapm connector is included, which means the APM UI views work without any additional configuration.

Conclusion

MCP servers do not need special observability tooling. They are programmatic processes, and OpenTelemetry is the right instrument for processes. The OTel MCP semantic conventions are stable and give you a consistent naming scheme that scales across tools and teams.

What makes the Elastic setup interesting is not the instrumentation itself. It is the second MCP. When your observability data lives in Elasticsearch, you can query it from the same AI session that generated it. That feedback loop is new, and it opens up use cases that dashboards alone cannot cover: real-time anomaly questions, automated triage, and AI-assisted incident investigation from the chat interface your team is already using.

Next steps

• Set up EDOT Node.js

• Configure the Elastic Agent Builder MCP

• OTel semantic conventions for MCP

• Managed OTLP endpoint on Elastic Cloud (now GA)

• Instrumenting Node.js applications with EDOT

Frequently asked questions

How do I add tracing to an MCP server? Use the OpenTelemetry SDK and follow the official OTel MCP semantic conventions. With the Elastic Distribution of OpenTelemetry (EDOT) for Node.js, a single --import flag enables auto-instrumentation for HTTP and database calls. Tool-call spans need to be added manually using a wrapper that sets mcp.method.name, gen_ai.tool.name, and gen_ai.operation.name.

Why are my MCP tool calls slow and how do I find the bottleneck? Without tracing, an MCP tool call is a black box: you see the result but not where the time went. Instrument the server with OpenTelemetry, ship traces to Elastic APM, and use the trace waterfall view in Kibana to see exactly how much time was spent in business logic versus downstream HTTP or database calls.

Can I send MCP server traces to Elastic APM without a custom collector? Yes. Elastic APM accepts OTLP directly. Set OTEL_EXPORTER_OTLP_ENDPOINT to your APM endpoint and OTEL_EXPORTER_OTLP_HEADERS with an API key, and traces flow in. EDOT bundles the elasticapm connector that the Kibana APM UI needs for service maps and transaction grouping.

What span names and attributes should I use for MCP tool calls? Follow the OpenTelemetry MCP semantic conventions: name spans {mcp.method.name} {target} (for example, tools/call echo), and set mcp.method.name, gen_ai.tool.name, and gen_ai.operation.name=execute_tool. On failure, set error.type to the error class name. Consistent naming means the Elastic APM transactions view groups your tool calls automatically.

How is this different from sending MCP traces to Datadog or Grafana? Any OTel-compatible backend can receive the traces. The Elastic-specific part is the Agent Builder MCP, which lets the same AI agent that generated the traces query them back from Elasticsearch in natural language. That feedback loop, where the AI reasons about its own tool-call performance, is not available with backends that do not expose their data through an MCP server.

Should I capture MCP tool call arguments and results in my traces? The OTel spec defines gen_ai.tool.call.arguments and gen_ai.tool.call.result as optional and warns they may contain sensitive data. Tools like get-env, which returns environment variables, illustrate the risk: API keys and credentials can land in your observability backend. Capture these only when the data is safe to store, and consider masking at the SDK level before export.

Does this work for MCP servers written in languages other than Node.js? The OpenTelemetry MCP semantic conventions are language-agnostic. EDOT is available for Node.js, Java, Python, .NET, and other languages, and any of them can send OTLP to Elastic APM. The wrapper pattern shown in this post translates directly: open a span around the tool handler, set the standard attributes, record exceptions on failure.

Problem context

Outages often show up as correlated errors on multiple services like checkout, frontend, and recommendation in Elasticsearch. That pattern can mean a shared dependency, or it can mean Kubernetes is misleading callers: wrong targetPort, empty Endpoints, or pods that never become ready. Observability tools like Elasticsearch tell you that callers fail and which edges look wrong. They generally do not fetch the live Service spec or compare containerPort to targetPort for you.

The Elastic AI Agent in Agent Builder is built for APM, logs, metrics, dependencies, and service maps. It is not a full EKS operations console. You could attach all EKS MCP tools to the same agent, but long tool lists increase wrong-tool calls, slow planning, and widen blast radius if a prompt accidentally asks for mutating actions.

Solution overview

Use Elastic AI Agent as the only agent your SRE chats with. It reasons from Elasticsearch first. When evidence points to cluster config, it calls a workflow tool that invokes the K8s Troubleshooter agent over /api/agent_builder/converse with a structured user_prompt. The K8s Troubleshooter agent carries only the EKS MCP tools, and cluster access stays scoped to one specialist identity, IAM, and RBAC. You can audit like any other integration.

Elasticsearch reaches EKS through an in-cluster bridge, exposed to Kibana as an MCP connector with a shared secret.

Before you start

You need:

• An EKS cluster with kubectl configured.
• An Elasticsearch 9.3+ deployment, an OTLP endpoint, an Elasticsearch API key, Agent Builder, and rights to create agents, MCP tools, and Workflows.
• An AI Connector in Elasticsearch for your chosen LLM.
• Budget two to four hours the first time you run these steps.

Implementation walkthrough

Step 1: deploy the Elastic OpenTelemetry Demo and ship telemetry to Elastic Observability

Follow elastic/opentelemetry-demo for Kubernetes and deploy elastic-opentelemetry-demo application to your EKS cluster. Configure your Elasticsearch OTLP endpoint and API key, confirm workloads are running, and note the namespace. In Kibana (APM, Logs, or Service Map), confirm data for checkout, frontend, recommendation, and product-catalog.

If you see healthy traffic to product-catalog, you are ready for the failure drill.

Step 2: run the EKS MCP bridge, register the connector, and bulk import EKS MCP tools

Complete the steps in aws-eks-mcp-setup end to end. The flow you would be following is:

1. Build and push the EKS MCP Bridge image
2. Create IAM policies
3. Create IRSA Service Account
4. Map IRSA role in aws-auth and apply Kubernetes RBAC
5. Deploy the bridge with a strong API_ACCESS_TOKEN to the EKS cluster
6. Connect Elastic Agent Builder with EKS MCP

A green MCP connector proves Kibana can reach the bridge.

For production, restrict LoadBalancer security groups to known Elasticsearch egress, prefer TLS on real paths, store tokens in Kubernetes Secrets, and use read-only MCP modes when you only diagnose.

Step 3: create a K8s Troubleshooter agent with EKS tools only

In Agent Builder, create an agent with agent ID k8s_troubleshooter, display name K8s Troubleshooter, and custom instructions from k8s_troubleshooter_agent. Attach only EKS MCP tools to this agent.

Chat directly with K8s Troubleshooter agent once and confirm a harmless read (for example list pods in the demo namespace).

Step 4 (Elasticsearch 9.3 only): clone the Observability Agent without EKS tools

Clone the bundled Observability Agent (Agent Builder → Manage Agents → Observability Agent → Clone) and name it Elastic AI Agent so it keeps the stock Observability system instructions and tools. Do not attach EKS MCP tools to this copy.

The parent Elastic AI Agent stays an observability-first interface for whoever chats with it.

Step 5: create the workflow and make it a callable tool

Create a new Elastic Workflow by importing k8s_troubleshooter_workflow.yaml and enable it.

Create a new tool in Agent Builder of type Workflow. Select the k8s_troubleshooter workflow, set tool ID custom.k8s_troubleshooter, and set the description to Tool to triage and troubleshoot kubernetes related issues (or equivalent wording your team standardizes on).

On Elastic AI Agent, attach the custom.k8s_troubleshooter workflow tool that you just created.

The parent’s tool list should show the custom.k8s_troubleshooter workflow tool attached, and K8s Troubleshooter agent should still answer when invoked on its own.

Step 6: inject the product-catalog service misconfiguration

Save the original targetPort, then patch to a wrong value (for example 9999).

kubectl get svc -A | grep product-catalog
kubectl get svc product-catalog -n YOUR_NAMESPACE -o yaml

kubectl patch svc product-catalog -n YOUR_NAMESPACE --type='json' \
  -p='[{"op": "replace", "path": "/spec/ports/0/targetPort", "value": 9999}]'

kubectl rollout restart deployment/checkout deployment/recommendation deployment/frontend -n YOUR_NAMESPACE

Callers still resolve Endpoints, but traffic lands on a port the container does not listen on, so Elasticsearch shows downstream errors on checkout, frontend, and recommendation.

You now have symptoms in Elasticsearch and a clear kubernetes cluster-side fault.

Step 7: run two prompts on the parent agent

Use AI Agent chat on Elastic AI Agent, not on the specialist.

Note: If you are using Elasticsearch 9.3, make sure you use the Elastic AI Agent that you created, not the stock agent.

Prompt 1: Why are failure transactions increasing for services like checkout and frontend?

Expect Elastic AI Agent to narrow the issue to the product-catalog service and note possible configuration issues as one of the probable causes, without yet invoking the custom.k8s_troubleshooter tool.

Prompt 2: Why is product-catalog service not servicing any requests in (insert your k8s cluster name) cluster? Is there any misconfiguration in the service?

Expect Elastic AI Agent to call the custom.k8s_troubleshooter tool, which invokes the K8s Troubleshooter agent and reads Service, Endpoints, and pods, compares targetPort to containerPort, and explains the mismatch with evidence. Expect to also see the recommended remediation steps.

Note: depending on the LLM you are using, the response from the agents may vary.

You get agent-led triage in Elastic Observability and cluster-grounded confirmation in the same thread, along with recommended remediation steps.

Step 8: patch the product-catalog service

Prompt 3: Patch the product-catalog service in (your EKS cluster name) cluster to have 8080 as the targetPort

Expect Elastic AI Agent to call the custom.k8s_troubleshooter tool, which invokes the K8s Troubleshooter agent and patches the product-catalog service.

Prompt 4: Rollout restart upstream services of product-catalog service

Expect Elastic AI Agent to identify all upstream services of product-catalog and call the custom.k8s_troubleshooter tool, which invokes the K8s Troubleshooter agent to roll out restarts for upstream services such as checkout, frontend, and recommendation.

Confirm product-catalog and upstream services recover in Elasticsearch.

Validation and trade-offs

You validated that Elastic AI Agent stays the main surface, that ~20 EKS tools live on one specialist K8s Troubleshooter agent, and that the Workflow plus Agent Builder converse API keeps a clear boundary for audits and reviews.

Trade-offs: MCP bridges need ongoing token and network hygiene, and you should keep mutating tools off or tightly RBAC-scoped until you accept the risk.

Frequently asked questions

Why is Elastic AI Agent not triaging the issues as explained in this article?

There could be two primary reasons. (A) If you are on Elasticsearch 9.3, make sure you chat on the Elastic AI Agent that you created, and not on the stock agent. (B) Make sure to use one of the LLM models rated Excellent or Great in Large language model performance matrix for Observability

Why do my services look unhealthy in Elasticsearch when the app code did not change?

Kubernetes can mislead HTTP clients: a bad Service targetPort, empty Endpoints, or pods that never become ready can fan out as errors on multiple edges in traces and service maps. Elastic Observability shows which dependencies fail; confirming the live Service spec usually needs cluster access.

How do I give Kubernetes access to Elastic AI Agent without putting every EKS tool on it?

Run two Agent Builder agents: keep the stock tools on the parent (Elastic AI Agent), and attach only EKS MCP tools to a specialist agent(K8s Troubleshooter agent). Invoke the specialist through a workflow that calls the Agent Builder converse API so the boundary is explicit and auditable.

Why chain agents with Elastic Workflows instead of one long system prompt?

Workflows give a callable, reviewable step between observability reasoning and cluster actions, which helps with governance and keeps the parent agent’s tool list short. Long unified tool lists often increase mistaken tool use and broaden blast radius if a prompt requests a mutating operation.

How does this compare to kubectl or a cloud console for incident response?

Consoles and kubectl stay the source of truth for live object state. This pattern automates the handoff from Elastic Observability signals to those checks through MCP, while still relying on IAM and Kubernetes RBAC on the specialist identity.

What are the main limitations or risks of an EKS MCP bridge with Agent Builder?

MCP bridges need token rotation, network restrictions, and TLS discipline on real paths. Mutating EKS tools should stay off or tightly RBAC-scoped until you accept operational risk.

Why do we need an EKS MCP bridge?

The managed EKS MCP server authenticates via AWS SigV4 through a stdio-based proxy (mcp-proxy-for-aws). Elastic's MCP connector requires an HTTP/SSE endpoint. The bridge pod runs mcp-proxy to expose the stdio proxy as an SSE/HTTP endpoint.

Can I reuse the same layout on GKE, AKS, or self-managed Kubernetes?

Yes. The separation principle is the same: observability data in Elasticsearch plus a specialist agent with cluster-scoped tools and a workflow-mediated handoff. Swap the MCP server or bridge, adjust RBAC, and parameterize cluster name or region in workflow inputs where needed.

This article is a companion guide to the Logstash Azure Event Hubs to Kafka input plugin migration, covering an alternative path: replacing logstash-input-azure_event_hubs with the OpenTelemetry Collector kafka receiver to consume from the Azure Event Hubs Kafka endpoint. For the reasons to migrate, authentication considerations, and key behavior changes such as offset handling, refer to the original article.

Reference: For detailed OTel Kafka receiver configuration options or parameter default values, see the Kafka Receiver README.

Converting your configuration

TLS configuration

Azure Event Hubs requires TLS for all Kafka connections on port 9093. The tls: {} block enables TLS with default settings (system CA certificates, no client certificate), which is sufficient for Azure Event Hubs. Omitting this block will cause the connection to fail because the broker expects a TLS handshake.

Encoding

The encoding field controls how the receiver interprets each Kafka message payload. For events consumed from Azure Event Hubs, the most common options are:

• text: decodes the payload as text and inserts it as the body of a log record. Uses UTF-8 by default; use text_<ENCODING> (e.g., text_shift_jis) for other character sets.
• raw: inserts the payload bytes as-is into the log record body.
• json: decodes the payload as JSON and inserts it as the log record body.
• azure_resource_logs: converts Azure Resource Logs format to OpenTelemetry format.

Additional encodings such as otlp_proto, otlp_json, and trace-specific formats (jaeger_proto, zipkin_json, etc.) are also available. See the Kafka Receiver README for the full list.

Basic configuration

Minimal configuration to consume logs from one Event Hub with SASL/PLAIN.

receivers:
  kafka:
    brokers:
      - "<NAMESPACE>.servicebus.windows.net:9093"
    group_id: "<CONSUMER_GROUP_NAME>"
    auth:
      sasl:
        username: "$ConnectionString"
        password: "Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<ACCESS_KEY_NAME>;SharedAccessKey=<ACCESS_KEY>"
        mechanism: "PLAIN"
    tls: {}
    logs:
      topics:
        - "<EVENT_HUB_NAME>"
      encoding: text

Advanced configuration

Example with multiple Event Hubs.

receivers:
  kafka/eh1:
    brokers:
      - "<NAMESPACE>.servicebus.windows.net:9093"
    group_id: "<CONSUMER_GROUP_1>"
    auth:
      sasl:
        username: "$ConnectionString"
        password: "Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<KEY_1>;SharedAccessKey=<ACCESS_KEY_1>"
        mechanism: "PLAIN"
    tls: {}
    logs:
      topics:
        - "<EVENT_HUB_1>"
      encoding: text

  kafka/eh2:
    brokers:
      - "<NAMESPACE>.servicebus.windows.net:9093"
    group_id: "<CONSUMER_GROUP_2>"
    auth:
      sasl:
        username: "$ConnectionString"
        password: "Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<KEY_2>;SharedAccessKey=<ACCESS_KEY_2>"
        mechanism: "PLAIN"
    tls: {}
    logs:
      topics:
        - "<EVENT_HUB_2>"
      encoding: text

Configuration parameters mapping

The following section maps each logstash-input-azure_event_hubs parameter to its OpenTelemetry Collector kafka receiver equivalent.

1. checkpoint_interval: Direct mapping to autocommit.interval.

   Units: Azure checkpoint_interval is in seconds. OTel autocommit.interval requires a duration string (e.g., 10s, 500ms).

   Azure config:

   input {
       azure_event_hubs {
           # ... other params ...
           checkpoint_interval => 10 # Default 5
       }
   }
   

   OTel receiver equivalent:

   receivers:
     kafka:
       # ... other params ...
       autocommit:
         interval: 10s # Default 1s
   

2. initial_position: Maps to initial_offset.

   Azure config:

   input {
       azure_event_hubs {
           initial_position => "end"
       }
   }
   

   OTel receiver equivalent:

   receivers:
     kafka:
       initial_offset: latest
   

   Value mapping:

   | Azure value | OTel value | | --- | --- | | beginning | earliest | | end | latest (default) | | look_back | Not directly supported |

   Note: Since the Kafka receiver can't read the old Blob Storage checkpoints, it treats the migration as a first-time connection. To avoid reprocessing data the legacy plugin already handled, set initial_offset: latest for the initial deployment.

3. max_batch_size: No direct 1:1 mapping.

   In OTel, the maximum batch of events processed cannot be directly controlled by the receiver. The receiver only controls how much data is read per fetch request using min_fetch_size, max_fetch_size, and max_fetch_wait.

   The actual event batching happens at the processing layer via the batch processor, which groups telemetry at the configured pipeline stage.

   Units: min_fetch_size and max_fetch_size are in bytes. max_fetch_wait uses duration strings (e.g., 250ms). send_batch_size is the number of records. timeout uses duration strings (e.g., 5s).

   Azure config:

   input {
       azure_event_hubs {
           max_batch_size => 125
       }
   }
   

   OTel receiver example:

   receivers:
     kafka:
       max_fetch_size: 2097152  # bytes (2 MiB)
       max_fetch_wait: 250ms
   
   processors:
     batch:
       send_batch_size: 125  # number of log records
   

4. threads: No direct mapping.

   Event Hubs distribute work by partition. A single Collector Kafka client can read from multiple partitions in parallel because the underlying Kafka client (franz-go) uses internal goroutines to fetch and process partition data concurrently. This concurrency is handled internally and is not configurable via a user-facing threads setting.

5. decorate_events: Not supported by Kafka receiver.

Performance comparison

These results use the same test environment described in the companion article: same Event Hub namespace, same number of partitions, and same batch/thread configuration. The absolute numbers are environment-specific, but the relative difference is what matters.

| Component | Payload | Throughput (events/s) | | ---------------------------------- | ----------- | ------------------------- | | Logstash azure_event_hubs plugin | 100B | ~5700 | | OTel Collector kafka receiver | 100B | ~10900 | | Logstash azure_event_hubs plugin | 1KB | ~1500 | | OTel Collector kafka receiver | 1KB | ~1900 | | Logstash azure_event_hubs plugin | 10KB | ~170 | | OTel Collector kafka receiver | 10KB | ~190 |

Across all payload sizes, the OTel Collector kafka receiver outperforms the Logstash azure_event_hubs plugin, with the largest gain at small payloads (~1.9x at 100B) where protocol overhead dominates, narrowing at larger sizes (~1.3x at 1KB, ~1.1x at 10KB). It does not reach the throughput of the Logstash kafka plugin from the companion article, but it improves on the legacy plugin across all tested payload sizes. Combined with the removal of the Blob Storage and GPv2 dependencies, the OTel Collector path removes two pieces of infrastructure that need to be provisioned, secured, and monitored.

Conclusions

Both migration paths eliminate the Blob Storage checkpoint dependency and improve throughput over the legacy azure_event_hubs plugin. The Logstash kafka plugin is the lower-friction option: the configuration change is minimal, the offset model carries over, and it delivers the highest throughput of the options tested. The OTel Collector kafka receiver is the better fit if you want to remove Logstash from the pipeline entirely and align with OpenTelemetry. It trades a lower peak throughput and no decorate_events equivalent for a vendor-neutral ingestion layer that can run alongside other OTel Collector pipelines in the same Collector.

Next steps

With the GPv1 retirement deadline (October 2026) approaching, starting this migration sooner reduces the time spent managing storage infrastructure that is no longer needed.

If any issues arise during migration:

• Usage questions or help with configuration: Post on the OpenTelemetry Collector GitHub Discussions or the Elastic Discuss forum.

• Bugs or unexpected behavior in the Kafka receiver: Open an issue in the opentelemetry-collector-contrib repository.

Related resources

• Kafka receiver documentation: Full reference for all OTel Collector kafka receiver configuration parameters.
• Azure Event Hubs input plugin documentation: Full reference for the legacy plugin being replaced.
• Logstash Azure Event Hubs to Kafka input plugin migration: Companion guide covering the alternative migration path to the logstash-input-kafka plugin.
• Azure Event Hubs for Apache Kafka overview: Microsoft's documentation on the built-in Kafka endpoint in Event Hubs.
• Event Hubs quotas and tier comparison: Tier requirements for Kafka protocol support.

When cluster CPU spikes and you need to find which workload is responsible, narrow from fleet to namespace to pod, one step at a time.

What you need

• An Observability serverless project or a self-managed or Elastic Cloud Hosted stack at version 9.4 or later, where PromQL is available as a preview query language for metrics.
• Kubernetes metrics flowing into Elasticsearch. For this exercise we have considered OpenTelemetry data.
• One or more clusters with workloads running so group by queries have something to compare.

The scenario

You manage a fleet of Kubernetes clusters:

| Cluster | Region | Role | |---------|--------|------| | prod-us-east-1 | US East | Production: services, ML training | | prod-eu-west-1 | EU West | Production: regional web tier, cache | | staging-us-east-1 | US East | Staging: QA, integration tests | | dev-sandbox | US East | Developer sandbox |

The production cluster in US East runs a mix of services and ML training jobs across several namespaces.

An alert fires: cluster-wide CPU is elevated, but only one team is complaining about slower response times.

You are triaging which cluster, then which namespace, then which pod.

You are not after a full root-cause proof in one query, but enough to name the suspect and hand off.

Your data

The OpenTelemetry Collector's Kubelet Stats Receiver populates data streams like metrics-kubeletstatsreceiver.otel-default. Metrics follow the k8s.* naming convention (for example k8s.pod.cpu.usage) and labels like k8s.cluster.name or k8s.namespace.name let you slice by cluster, namespace, or pod.

To verify the data is there, open Discover, switch to ES|QL mode, run TS metrics-*, and scope the query with WHERE data_stream.dataset == "kubeletstatsreceiver.otel".

Investigation: find the noisy neighbor

Step 1: Which cluster is hot?

When you manage multiple clusters, start at the fleet level.

PROMQL sum by (k8s.cluster.name) (k8s.pod.cpu.usage)

This groups total pod CPU by cluster.

prod-us-east-1 immediately stands out: total pod CPU is an order of magnitude higher than the other clusters.

The EU production cluster, staging, and dev-sandbox are all quiet.

Now you know where the problem is, time to zoom in.

Step 2: Overall CPU in the hot cluster

Filter to prod-us-east-1 and look at total CPU:

PROMQL sum(k8s.pod.cpu.usage{k8s.cluster.name="prod-us-east-1"})

This gives you the cluster-wide pod CPU footprint over time.

If the total is climbing or spiking, something changed, but you don't yet know what.

Step 3: Break down by namespace

The fastest way to isolate which team is responsible: group by namespace.

PROMQL sum by (k8s.namespace.name) (k8s.pod.cpu.usage{k8s.cluster.name="prod-us-east-1"})

Set the time picker in Kibana to cover your incident window.

ml-training dominates at ~2.0 cores while every other namespace stays well below 0.2 cores.

Step 4: Drill down to the pod

Now that you know the namespace, identify the specific pod:

PROMQL sum by (k8s.pod.name) (k8s.pod.cpu.usage{k8s.cluster.name="prod-us-east-1", k8s.namespace.name="ml-training"})

That ranks pods in the namespace by total CPU.

The chart should make the outlier obvious.

Pod model-train-v2-run-47-d9j67 is consuming the full 2.0 cores. It is a training job saturating its allocation.

Step 5: Check resource utilization ratios

Raw CPU cores tell you how much. Utilization ratios tell you how close to limits.

A pod hitting 100% of its CPU limit is being throttled, and it is both the noisy neighbor and a victim of its own limits.

PROMQL sum by (k8s.namespace.name) (k8s.container.cpu_limit_utilization{k8s.cluster.name="prod-us-east-1"})

ml-training shows ~100% CPU limit utilization (pegged at the 2-core limit), while the other namespaces stay under 20%.

This confirms the training job is saturating its allocation and likely causing scheduling pressure on the shared node.

What happens next

The PromQL query named the suspect: the training job model-train-v2-run-47 in ml-training.

From here:

• Logs: Filter by the pod name in Discover to see what the training job was doing and whether it logged errors or warnings.
• Kube events: Check for OOMKilled, throttling, or eviction events in the same time window.
• Resource policies: Review whether the training job's requests and limits match its actual usage. A large gap between request and limit lets a pod burst past what the scheduler planned for. Consider ResourceQuota or LimitRange on the namespace.

EDOT is now generally available. The collector, the SDKs, the whole stack — production-ready, enterprise-supported, no asterisks.

But here's the thing: getting your data into Elastic is only half the job. The harder half, in practice, is what happens after. Someone still has to build the dashboards, write the alert rules, and figure out which SLOs are worth tracking — before any of it is useful.

That gap is what OpenTelemetry Content Packages are designed to close.

What Are OpenTelemetry Content Packages?

Elastic's traditional Beats-based integrations always bundled data collection and visualizations together — you got curated dashboards and alerts the moment you turned something on. As Elastic moves to an OpenTelemetry-first world, that same philosophy carries over, but the model is cleaner.

OpenTelemetry Content Packs are purely about the observability assets for a given service. No data collection config is bundled in, because in an OTel world, the collector handles that. Each package contains:

• Dashboards — curated, pre-built Kibana visualizations tailored to the service being monitored
• Alert rules — pre-configured alerting rules that fire on meaningful thresholds, helping teams minimize Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR)
• SLO templates — ready-made Service Level Objective definitions you can apply immediately to track reliability targets, error budgets, and burn rates

More asset types are planned for future packages as the content pack model continues to evolve.

How Does It Work?

The core idea is simple: as soon as data arrives in Elastic, the right dashboards, alert rules, and SLO templates are ready to use. The content package activates based on the incoming data, regardless of how that data was collected.

One of the most powerful aspects of this system is automatic installation. When Elastic detects that data for a particular service has started arriving in Elasticsearch, the corresponding content pack is installed automatically — no manual steps, no hunting through the integrations catalog. By the time you open Kibana, your dashboards are already there waiting for you, your alert rules are ready to be enabled, and your SLO templates are pre-loaded.

To get the data flowing in the first place, we need to configure the collector — a YAML file that defines the building blocks of your telemetry pipeline:

• Receivers — define what data to collect and from where. Each service has its own receiver (for example, the MySQL receiver scrapes metrics directly from the database).
• Exporters — define where the collected data is sent. In our case, we use the Elasticsearch exporter, which ships the telemetry data directly into Elasticsearch in OpenTelemetry native format.
• Pipelines — wire the receivers and exporters together, defining the flow of data through the collector.

Once this configuration is in place and the collector is running, data starts flowing into Elasticsearch — and the content pack takes it from there.

Data Sources

OpenTelemetry data can reach Elastic through any of the following:

• EDOT Collector — the Elastic Distribution of the OpenTelemetry Collector, embedded in or used alongside the Elastic Agent
• Upstream OTel Collector — the standard community OpenTelemetry Collector (Contrib or custom builds)
• EDOT Cloud Forwarder (ECF) — a serverless OTel Collector that collects telemetry from AWS, GCP, and Azure (VPC Flow Logs, CloudTrail, CloudWatch, and more) and forwards it directly to Elastic Observability, with no infrastructure to manage

The content pack doesn't care how the data arrived — only that it's there.

Seeing It in Practice: MySQL Monitoring

Take a team running MySQL who wants to track query throughput, connection counts, buffer pool utilization, and slow query rates — and get alerted before small problems turn into 2am incidents. Historically, that means hours of dashboard building, custom alert queries, and a lot of guesswork about which metrics actually matter.

With the MySQL OpenTelemetry Assets Package, that work is already done. Here's how the whole thing comes together.

Step 1: Get the Data In

The data pipeline is driven by a collector configuration that defines receivers (where to scrape data from), processors (how to enrich or transform it), and exporters (where to send it — in this case, Elasticsearch).

Regardless of whether you use the EDOT Collector or the Upstream OTel Collector, the fundamental configuration structure is the same. The configuration below uses separate receivers for the primary and replica instances, because replication metrics are only available on replicas. Replace the placeholders with your actual endpoints, credentials, and Elasticsearch details.

receivers:
  mysql/primary:
    endpoint: <MYSQL_PRIMARY_ENDPOINT>
    username: <MYSQL_USER>
    password: <MYSQL_PASSWORD>
    collection_interval: 10s
    statement_events:
      digest_text_limit: 120
      limit: 250
    query_sample_collection:
      max_rows_per_query: 100
    events:
      db.server.query_sample:
        enabled: true
      db.server.top_query:
        enabled: true
    metrics:
      mysql.client.network.io:
        enabled: true
      mysql.connection.errors:
        enabled: true
      mysql.max_used_connections:
        enabled: true
      mysql.query.client.count:
        enabled: true
      mysql.query.count:
        enabled: true
      mysql.query.slow.count:
        enabled: true
      mysql.table.rows:
        enabled: true
      mysql.table.size:
        enabled: true

processors:
  resourcedetection:
    detectors: [system, env]

exporters:
  elasticsearch/otel:
    endpoint: <ES_ENDPOINT>
    api_key: <ES_API_KEY>
    mapping:
      mode: otel

service:
  pipelines:
    metrics:
      receivers: [mysql/primary, mysql/replica]
      processors: [resourcedetection]
      exporters: [elasticsearch/otel]

The MySQL receiver scrapes metrics and events from the database at the configured interval and emits them as OpenTelemetry metrics. These flow through the pipeline and land in Elasticsearch, ready to be visualized.

Step 2: Open Kibana — Everything's Already There

Dashboards

As soon as the MySQL metrics and events arrive in Elasticsearch, the MySQL OpenTelemetry Assets Package is automatically installed in the background. By the time you navigate to Kibana, the dashboards are already populated and waiting.

Users immediately get visibility into:

• Active and max connections
• Query throughput — statements executed per second
• InnoDB buffer pool hit rate and memory usage
• Slow query count and trends
• Table lock waits and contention
• Bytes sent and received over time
• Replication lag (for replicated setups)

No manual field mapping. No dashboard building from scratch. Just data in, insights out.

Below are some screenshots of the MySQL OpenTelemetry dashboard in Kibana, showing the out-of-the-box visualizations that are automatically available as soon as your data starts flowing in.

Overview Dashboard

Queries Dashboard

Availability Dashboard

Alert Rules, Ready to Enable

The package includes six pre-built alert rules — covering high connection error rates, slow query spikes, thread saturation, replication lag, buffer pool dirty page ratio, and row lock contention — each with recommended thresholds and severity levels. These are available immediately on install and can be enabled, tuned, and extended directly in Kibana without any custom query authoring. Below is an example of one of the alerts.

SLO Templates, Pre-Loaded

Four SLO templates are included out of the box, tracking replication lag, connection exhaustion errors, slow query rate, and connected thread count — each with a pre-configured target and 30-day rolling window. Teams can adopt them as-is or tune the thresholds to match their own reliability requirements.

What's Available Today

The MySQL OpenTelemetry Assets Package is just one example from a growing library of OpenTelemetry Content Packages that Elastic has already built out. Content packs are available for a range of services — and we have also started extending this to the cloud, with initial support for Cloud Service Provider integrations that use the EDOT Cloud Forwarder (ECF) to bring AWS, GCP, and Azure telemetry into Elastic with ready-made dashboards.

The same pattern holds across all of them — data in, and a complete observability package (dashboards, alert rules, SLO templates) instantly ready — whether you're monitoring a self-managed database or cloud-native services from your preferred cloud service provider.

Where This Is Going

The next step worth watching is OTel Integration Packages, which will let you push collector configurations directly from the Kibana UI — making the entire setup experience point-and-click, from data collection through to visualization, with no YAML editing required.

Get Started

Ready to try it? Start with the EDOT Collector documentation and explore the growing library of OpenTelemetry content packages in Kibana's Integrations page.

Traffic grows. Teams want metrics derived from traces. The backend goes down for maintenance and you lose an hour of telemetry. A compliance requirement means PII must be stripped before data leaves the cluster. Suddenly, that single collector isn't enough — and the question becomes: what should the architecture actually look like?

The OpenTelemetry Collector is designed to be composed. It can run in multiple deployment modes, be chained into pipelines, and scaled independently at each stage. But the documentation describes individual components, not how to think about assembling them. That thinking is what this article is about.

What follows is a conceptual framework for reasoning about collector architectures — not a set of rigid templates. The building blocks described here are reference points. In practice, they combine, overlap, and adapt to your constraints. A tail sampling tier might also need Kafka-backed resilience. A gateway might absorb the role of a sampling tier at low volumes. The goal is to understand the concepts well enough to compose the right architecture for your situation, not to pick a pre-built one off a shelf.

Three conceptual layers

It helps to think about collector architectures in three layers: edge, processing, and resilience. These aren't physical tiers that must exist as separate deployments — they're categories of concern. A single collector can address multiple layers. A complex deployment might have several components within one layer. The layers are a thinking tool, not a deployment diagram.

Edge: how telemetry enters the pipeline

The edge layer is about the first hop — how telemetry gets from your applications and infrastructure into the pipeline. At this stage, the collector gathers data in two fundamentally different ways. Pull-based receivers like filelog and hostmetrics actively reach out to collect data — tailing log files on disk or scraping system-level metrics from the host. Push-based receivers like otlp listen for data sent to them — applications instrumented with OpenTelemetry SDKs export traces, metrics, and logs directly to the collector's OTLP endpoint. A single edge collector typically runs both: pull receivers for infrastructure telemetry the application doesn't know about, and push receivers for application telemetry the SDK produces. There are several common deployment patterns, and the right one depends on your environment and what you need to collect.

DaemonSet Agent — One OpenTelemetry Collector per Kubernetes node, deployed as a DaemonSet. Applications export to the agent running on the same node (typically via status.hostIP:4317 using the Kubernetes Downward API). The agent also tails container log files from disk via the filelog receiver and scrapes host-level metrics via the hostmetrics receiver. This is the most common Kubernetes pattern because it handles both application and infrastructure telemetry with a single deployment, and applications only need to know about localhost.

Sidecar Agent — One OpenTelemetry Collector per pod, deployed as a sidecar container. Each service gets its own collector with a custom configuration. This is required on managed container platforms like AWS Fargate or Azure Container Apps where DaemonSets aren't available, and it's useful when services have different processing requirements. When running alongside a DaemonSet, the sidecar handles application telemetry while the DaemonSet independently collects node-level telemetry — applications don't send to both.

Host Agent — A standalone OpenTelemetry Collector running as a systemd service on bare-metal or VM hosts. It serves the same role as the DaemonSet agent but outside Kubernetes: collecting host metrics, tailing log files, and receiving OTLP from local applications.

Direct SDK Export — Applications export directly to the next stage (gateway or backend) with no local collector. This is the simplest option but only works when you don't need infrastructure collection. For log collection, the recommended pattern is still to write to stdout and use a collector with the filelog receiver — even if the SDK is exporting traces and metrics directly.

These patterns aren't mutually exclusive. A Kubernetes cluster might run DaemonSet agents for infrastructure collection alongside sidecars for services that need custom processing. A VM environment might use host agents for some services and direct SDK export for others. The edge layer is about matching the collection pattern to the workload, not picking one pattern for everything.

Processing: central policy, sampling, and transformation

Not every architecture needs a processing layer. If your edge collectors can export directly to your backend and you don't need centralized policy, you can skip it to favour simplicity. But several scenarios push you toward central processing — and the way you address them can range from a single gateway to a multi-stage pipeline.

Centralized policy (Gateway) — A pool of OpenTelemetry Collectors that sits between edge collectors and the backend. This is where you enforce consistent filtering, transformation, and PII redaction across all services. It's also where you manage backend credentials — edge collectors export to the gateway over OTLP, and only the gateway holds the API keys. Credential isolation is often the primary reason teams add a gateway.

Replica count scales with data volume. At low volumes (under 1K events/sec), 2 replicas co-located with workloads is sufficient. At medium volumes, 3–5 replicas on a dedicated node pool. At high volumes, 5–20+ replicas, potentially in a separate cluster. This is a general rule of thumb, and you should adapt it to your specific needs as loads might vary significantly between payload types.

Tail-based sampling — Sampling decisions that consider the complete trace (e.g., "keep all traces with errors, sample 10% of successful traces") require that all spans of a trace reach the same collector instance. This is achieved with the loadbalancingexporter using routing_key: traceID, which consistently routes spans from the same trace to the same downstream collector.

There's a critical subtlety here: if you're deriving span metrics (RED metrics) from traces using the spanmetrics connector, the derivation must happen before sampling. Otherwise, your metrics only reflect the sampled subset, not the true traffic. The correct pattern is a two-step pipeline within the sampling stage:

1. Receive traces, derive spanmetrics from 100% of traffic, forward via a forward connector.
2. Apply tail_sampling to the forwarded traces, export only kept traces.
3. A separate metrics pipeline exports the derived RED metrics.

This ensures accurate metrics regardless of your sampling rate.

The key point about processing is that these capabilities — gateway policy, tail sampling, span metrics derivation — are not separate products or fixed modules. They're configurations of the same OpenTelemetry Collector. At low volumes, a single gateway deployment might handle policy enforcement, sampling, and metrics derivation all at once. At high volumes, you might split them into dedicated stages for independent scaling. The architecture adapts to your scale, not the other way around.

Resilience: what happens when the backend is down

The resilience layer determines how much data you're willing to lose during backend outages or collector restarts. This isn't a separate tier you bolt on — it's a property you apply to any stage of the pipeline.

In-Memory Queues — The default. The collector's sending_queue retries failed exports with exponential backoff. If the collector process crashes or restarts, queued data is lost. This is acceptable for development and for workloads where some data loss during incidents is tolerable.

Persistent Queues (WAL) — The file_storage extension writes queued data to disk before export. If the collector crashes, it resumes from where it left off after restart. In Kubernetes, this requires a PersistentVolumeClaim. This is the right choice for most production workloads — it survives collector restarts and brief backend outages without the operational complexity of an external message bus.

Kafka Buffer — An external Kafka cluster sits between collectors and the backend. Producer collectors write to Kafka topics; consumer collectors read from Kafka and export to the backend. This provides the strongest durability guarantee — Kafka can buffer hours of telemetry during extended outages and enables replay. But it adds significant operational complexity.

The important thing to understand is that resilience is orthogonal to the other layers. You can add persistent queues to an edge agent, a gateway, or a sampling tier. You can put Kafka in front of a gateway, in front of a sampling tier, or in front of the backend. A tail sampling deployment that needs to survive extended outages might use Kafka-backed ingestion — combining what might look like two separate "modules" into a single stage. The building blocks compose freely based on what you need to protect against.

Where to start with your architecture

The Agent + Gateway two-tier pattern is the de facto production standard, used by the vast majority of organizations running OpenTelemetry at scale. DaemonSet agents on every node handle local collection — pulling infrastructure telemetry via filelog and hostmetrics, receiving application telemetry via OTLP — while a centralized gateway pool enforces policy, manages credentials, and exports to the backend. Persistent queues (WAL) on the gateway protect against backend outages without external dependencies.

Every other configuration either simplifies this pattern or extends it. Smaller environments might drop the gateway and export directly from agents. Larger ones might add a tail sampling tier with traceID-based load balancing, a Kafka buffer for extended resilience, or span metrics derivation before sampling. The building blocks described in the previous sections — edge, processing, resilience — are the modules you add or remove from this foundation.

The key is to start with the two-tier pattern and evolve incrementally:

• Need credential isolation or centralized PII redaction? You already have the gateway.
• Need tail-based sampling? Add a load-balancing exporter and a sampling tier between agents and gateway.
• Need hours of buffer during extended outages? Insert Kafka between agents and the processing tier.
• Running on Fargate or Azure Container Apps? Swap DaemonSet agents for sidecars — the rest of the pipeline stays the same.

Start here. Add modules as your needs grow. The architecture adapts to your scale, not the other way around.

Decision points that shape where you need to take your architecture

When designing a collector architecture, these are the questions that determine which patterns you need:

| Question | Impact | |----------|--------| | Do I need infrastructure telemetry (host metrics, disk logs)? | Determines whether you need a local collector or can use direct SDK export | | Am I on a managed container platform (Fargate, ACA)? | Forces sidecar pattern instead of DaemonSet | | Do I need centralized filtering, PII redaction, or credential isolation? | Adds a gateway stage | | Do I need tail-based sampling? | Adds a sampling stage with load-balancing exporter and traceID routing | | Do I want span-derived metrics (RED metrics)? | Requires spanmetrics before sampling in a two-step pipeline | | How much data loss is acceptable during outages? | Determines in-memory queues vs. persistent queues vs. Kafka — applied to whichever stage needs protection | | What is my expected data volume? | Determines whether capabilities can be co-located in a single deployment or need dedicated stages |

The answers to these questions don't map to a single "correct" architecture. They constrain the design space, and within those constraints, you make trade-offs between simplicity and capability.

Exploring these patterns interactively

If you'd rather explore how these building blocks compose than assemble them by hand, OpenTelemetry Blueprints is an open-source tool that generates reference architectures from your requirements.

Toggle your environment, signals, volume, resilience, and processing needs — and get a composed diagram with animated data flow, interactive tooltips, and reference collector configurations you can open directly in OTelBin for validation.

The generated configurations export via OTLP, so they work with any OTLP-compatible backend — including Elastic Observability, which natively accepts and stores OTLP traces, metrics, and logs.

The architectures Blueprints generates are reference compositions — starting points for understanding how the building blocks fit together, not turnkey deployments. Every architecture should be adapted to your organisation's scale, security, networking, and compliance requirements. The patterns might combine or overlap differently in your environment than in anyone else's, and that's the point.

Get started

The architectures described here export over OTLP, so they work with any compatible backend. If you don't have one yet, the fastest way to see your telemetry flowing end-to-end is with Elastic Observability — it natively ingests OTLP traces, metrics, and logs with no additional configuration.

1. Start a free trial on Elastic Cloud Serverless — no credit card required.
2. Point your collector's OTLP exporter at the managed OTLP endpoint.
3. Explore your traces, metrics, and logs in Kibana within minutes.

Check out these resources to go further:

• Elastic's managed OTLP endpoint documentation
• EDOT Collector — Elastic's distribution of the OpenTelemetry Collector
• OpenTelemetry Blueprints — generate reference architectures interactively

• Distributed components: A single request might hit an API gateway, retrieve data from a feature store, evaluate a predictive model in a Python inference service, query a vector database, and call an external LLM.
• Non-determinism: AI agents make autonomous decisions and tool calls. If an agent fails, you need a full trace to understand its reasoning loop and what external tools it tried to invoke.
• Context dependence: You don't just care that an error happened; you need to know what model version was running, what hyperparameters were used, what the input data looked like, what was the commit that made that change. Many of these attributes are custom to your app, and you need an Observability environment that has the flexibility of creating new parameters on the fly and use them to find and fix issues.

On top of that, with the increased use of AI agents to generate code and make autonomous decisions, Observability becomes key to understanding what is working and what is not. It creates a critical feedback loop to quickly fix problems. More than ever, ML and AI applications need to adopt the best practices of mature software engineering systems to succeed.

This guide shows how to use OpenTelemetry and Elastic to correlate traces, logs, and metrics to track runs, compare model behavior, and trace requests across Python and Go services with one shared context.

Problem context: why AI systems are harder to debug

Traditional services already have distributed failure modes, but ML and AI systems add more moving parts:

• notebook experiments and ad hoc jobs
• batch training and evaluation pipelines
• online inference services
• external API calls, including LLM providers
• changing model versions and hyperparameters

When one prediction path gets slower or starts failing, plain isolated logs do not answer enough questions. You need to correlate:

• what ran (run ID, model version, parameters)
• where time was spent (pipeline stage latencies)
• what was the result (model stats, predictions, API calls, compare with other runs)
• what changed (code, data, dependencies)

In a future blog post, we'll show you how to set up automatic RCA and remediations with Elastic Workflows and our AI integrations. But as a first step, ML and AI pipelines need a robust Observability framework, which is very easy to set up with OpenTelemetry and Elastic.

Solution overview

OpenTelemetry gives you a standard way to emit traces, metrics, and logs. Elastic provides full OpenTelemetry ingestion, giving you a single place to store and query that telemetry. Kibana's UI is fully integrated with OpenTelemetry, allowing you to explore your services, service dependencies, service latencies, spans, and metrics out-of-the-box.

You can start with two deployment options:

• Cloud: send OpenTelemetry data directly to Elastic Cloud Managed OTLP Endpoint (mOTLP docs), without the overhead of managing collectors
• Local: run Elastic and the EDOT Collector with start-local, the EDOT Collector will be automatically listening for OTLP data in localhost:4317

Both options let you keep your application code unchanged for the initial implementation.

Step 1: zero-code baseline for Python services

Start by just installing the Elastic Distribution of OpenTelemetry Python (EDOT Python) package and using the opentelemetry-instrument wrapper to run your script. By simply running your script with this wrapper—without modifying your application code—your Python services begin emitting standard telemetry right away. This includes any logs exported via logging, alongside metrics and traces for auto-instrumented libraries. This data can be routed directly to Elastic's managed OTLP endpoint or a local EDOT collector.

pip install elastic-opentelemetry
edot-bootstrap --action=install

Export the OpenTelemetry environment variables, then run opentelemetry-instrument on your script to enable auto-instrumentation.

export OTEL_EXPORTER_OTLP_ENDPOINT="https://<motlp-endpoint>" # No need when using start-local with EDOT
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey <key>" # No need when using start-local with EDOT
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=prod,service.version=1.0.0" # Set the environment and version for your app
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
export ELASTIC_OTEL_SYSTEM_METRICS_ENABLED=true
export OTEL_METRIC_EXPORT_INTERVAL=5000 # Choose the interval for your application metrics

opentelemetry-instrument --service_name=<pipeline-name> python3 <your_python_script>.py # Set your chosen name for your service

With this baseline, you can quickly get:

• Centralized logs with trace context. Any logs exported via logging will be searchable in Elastic and Kibana, with the ability to perform full-text search on your logs
• Set alerting on log errors
• Process and system metrics. System and process metrics from the execution will be automatically exported to Elastic. You can visualize them, and analyse memory usage (leaks, OOM errors), CPU utilization (Bottlenecks / Spikes), thread counts, disk I/O bottlenecks or network I/O saturation.
• Set alerting on metrics
• Spans for auto instrumented libraries
• Service latency baselines and error trends
• Set manual or Anomaly detection alerting on error rates, latencies or throughput
• Correlate logs, metrics, and traces in a single shared context to quickly find the root cause of issues, using OpenTelemetry for instrumentation and Elastic for analysis.

Once ingested, Kibana immediately populates out-of-the-box dashboards. You can explore full-text searchable logs, monitor system and process metrics, investigate auto-instrumented trace waterfalls, map out your ML dependencies with service maps, and easily set up alerts for latency spikes, memory or CPU usage or log errors.

For LLM-specific observability, OpenTelemetry provides official Semantic Conventions for Generative AI to standardize how you track token usage, model names, and prompts. These semantic conventions are still in development and not stable yet. Some instrumentations for the most used libraries in this space are being developed as part of the OpenTelemetry Python Contrib repository. Alternatively you can implement these conventions manually in your custom spans. LLM related OpenTelemetry logs, metrics and traces sent to Elastic will be in context and automatically correlated with the rest of your application or stack of applications.

Step 2: add ML-specific context with custom spans and log fields

Auto-instrumentation is a starting point. For ML and AI Ops, add explicit spans around business stages and attach run metadata. Elastic's schema flexibility and dynamic mappings make it a perfect fit for custom attributes or metrics that are exclusive to your pipelines or specific experiments. There is no need to know what the data will look like before writing it. You have the flexibility of creating new parameters on the fly, Elastic maps them automatically, and you can track them instantly.

Add custom fields and metric-like values as structured log fields so you can chart and alert on them later:

logger.info("training metrics", extra={
    "ml.run_id": run_id,
    "ml.training_accuracy": train_accuracy,
    "ml.validation_accuracy": val_accuracy,
    "ml.drift_detected": drift_detected,
})

Because Elastic handles dynamic mapping, any custom metrics or attributes you log, like model ids, training accuracy or drift detection, are instantly indexed and available to search in Discover or visualize via Dashboards.

This makes dashboards and rules practical:

• alert when ml.validation_accuracy < 0.8
• alert when ml.drift_detected == true
• compare stage latency by ml.model_version

You can use these custom attributes to build targeted visualizations, and trigger alerts when ML-specific metrics like validation accuracy drop below a critical threshold.

Adding custom spans allows you to break down the specific stages of your ML pipelines, such as data loading and model training, wrapping them in their own measurable execution blocks, and analyze average latency or error rates for specific pipeline stages.

from opentelemetry import trace

tracer = trace.get_tracer("ml.pipeline")

with tracer.start_as_current_span("load_data") as span:
    span.set_attribute("ml.run_id", run_id)
    span.set_attribute("ml.dataset", dataset_source)
    load_data()

with tracer.start_as_current_span("train_model") as span:
    span.set_attribute("ml.model_version", model_version)
    span.set_attribute("ml.learning_rate", learning_rate)
    train_model()

Custom spans will be reflected in the APM UI alongside your traces. So you can explore their latency, impact in total execution, stack traces, error rates.

Step 3: trace across Python and Go in production

Real inference paths often cross service boundaries. For example:

In a production environment, a user request might pass through a Go-based API before hitting your Python ML inference service. OpenTelemetry ensures tracing context is preserved seamlessly across these boundaries.

In our example, we have a simple Go HTTP service that acts as the entry point and demonstrates OpenTelemetry instrumentation in Go. This REST API service stores and retrieves ML predictions by querying Elasticsearch based on data IDs from the source dataset. All of its endpoints are natively instrumented with OTel spans.

The full request lifecycle looks like this:

1. The Go API receives the client request.
2. It searches Elasticsearch for an existing prediction or calls the Python model service to run inference.
3. The Python service loads features, runs the model, and returns predictions.

When both services use OpenTelemetry, trace context is propagated automatically through headers. In Elastic, you can inspect one end-to-end trace and locate latency or errors by service and span.

The resulting distributed trace in Elastic pieces the entire journey together. You can see the exact breakdown of time spent in the Go API versus the Python model, and correlate logs from both services in a single unified view.

Validation checklist

After instrumentation, validate with a short runbook:

1. Confirm logs, metrics, and traces arrive for each service.
2. Verify your custom attributes (e.g. run_id, model_version, llm_ground_truth_score) are present in traces and logs.
3. Compare p95 latency per stage (load_data, train_model, predict).
4. Trigger a controlled failure and confirm error traces include stack context.
5. Test one rule for errors, one rule for latency spikes, and one rule for model-quality fields. Set up a connector and attach it to the rule to reach you in Slack, email, or trigger an auto-remediation workflow.

Conclusion and next steps

OpenTelemetry gives ML and AI teams a unified telemetry layer, while Elastic makes that data instantly queryable and actionable across your entire lifecycle—from notebook experiments to production inference. By starting with zero-code instrumentation and incrementally adding ML-specific attributes and cross-language tracing, your team can easily adopt the Observability best practices of mature software engineering systems and succeed in the new era of complex AI operations.

Try this setup in Elastic Cloud, and use mOTLP for a managed ingest path. If you want a local sandbox first, start with Elastic start-local + EDOT Collector.

As a core contributor, Elastic is proud to have accelerated this effort by previously donating its Universal Profiling™ eBPF-based continuous profiling agent to OpenTelemetry. This production-grade agent enables whole-system visibility across all applications, covering a multitude of programming languages and runtimes including third-party libraries and kernel operations with minimal overhead. It allows SREs and developers to quickly identify performance bottlenecks, maximize resource utilization, and optimize cloud spend.

Additionally, over the last two years, Elastic has been heavily contributing to the OpenTelemetry Collector, Semantic Conventions and Profiling Special Interest Groups (SIGs) to lay the technical foundation for the promotion of Profiles to Alpha.

This Alpha milestone not only boosts the standardization of continuous profiling but also accelerates the practical adoption of profiling as the fourth key signal in observability. Customers now have a vendor-agnostic way of collecting profiling data and enabling correlation with existing signals, like logs, metrics and traces, unveiling new potential for observability insights and a more efficient troubleshooting experience.

What is continuous profiling?

Profiling is a technique used to understand the behavior of a software application by collecting information about its execution. This includes tracking the duration of function calls, memory usage, CPU usage, and other system resources.

However, traditional profiling solutions have significant drawbacks limiting adoption in production environments:

• Significant cost and performance overhead due to code instrumentation
• Disruptive service restarts
• Inability to get visibility into third-party libraries

Unlike traditional profiling, which is often done only in a specific development phase or under controlled test conditions, continuous profiling runs in the background with minimal overhead, eliminating the need for service restarts or manual intervention. This provides real-time, actionable insights without replicating issues in separate environments. SREs, DevOps, and developers can see how code affects performance and cost, making code and infrastructure improvements easier.

Elastic's contribution: Powering the Alpha

The Elastic-donated profiler now forms the reference eBPF-based profiler implementation within OpenTelemetry: opentelemetry-ebpf-profiler. With the Alpha release, the eBPF profiler operates as an OpenTelemetry Collector receiver and contains numerous improvements such as automatic Go symbolization and support for new language runtimes. Operating as an OpenTelemetry Collector receiver enables the profiler to seamlessly leverage existing OpenTelemetry processing and filtering pipelines.

For example, the k8sattributesprocessor can use the container.id resource attribute to automatically enrich every profile with its corresponding Kubernetes context. This means you don't just see a raw stack trace; you see exactly which namespace, pod, and deployment produced it.

receivers:
  # Profiling receiver
  profiling: {}

processors:
  k8sattributes:
    passthrough: false 
    pod_association:
      - sources:
          - from: resource_attribute
            name: container.id
    extract:
      metadata:
        - "k8s.namespace.name"
        - "k8s.deployment.name"
        - "k8s.replicaset.name"
        - "k8s.statefulset.name"
        - "k8s.daemonset.name"
        - "k8s.node.name"
        - "k8s.pod.name"
        - "k8s.pod.ip"
        - "k8s.pod.uid"

Besides improvements to the eBPF profiler, Elastic has made significant contributions to:

• Correlating profiles with the information produced by OpenTelemetry eBPF instrumentation (OBI), a powerful auto-instrumentation tool that can enable distributed tracing.
• Process Context Sharing OTEP which is designed to bridge the gap between application SDKs and the profiler. This mechanism will allow OpenTelemetry SDKs to "publish" their resource attributes (like service.name) into a small, standardized memory region. Because this data is stored in the process's own memory map, the eBPF Profiler can instantly discover and associate it with its corresponding Profile.
• Semantic conventions and integration of OpenTelemetry Profiles with Google's pprof format (transparent conversion)
• OpenTelemetry Collector processing pipelines, allowing it to better integrate with the profiling receiver

Elastic's Next-Generation Profiling Development

Elastic remains deeply committed to OpenTelemetry's vision and is pushing the boundaries of what is possible with profiling data. We are dedicating a team of profiling domain experts to co-maintain and advance profiling capabilities within OpenTelemetry, while simultaneously working on groundbreaking features built on this new open standard.

Exciting areas of internal profiling-specific development include:

• OpenTelemetry Profiles derived Metrics: We are developing innovative ways to automatically generate actionable performance metrics directly from the raw OTel Profiles data, providing a new dimension for infrastructure modeling and alerting.
• Rapid Integration with the Elastic Stack: We are making swift progress on first-class support for OTLP Profiles within the Elastic Stack, ensuring seamless ingestion (the ebpf-profiler receiver is already integrated with the Elastic Distributions of OpenTelemetry (EDOT) collector), storage, and visualization of this new signal alongside your existing logs, metrics and traces.
• AI-Powered Workflows: We are leveraging the deep insights provided by continuous profiling data to power new AI-driven workflows, enabling automatic root-cause analysis, anomaly detection, and intelligent optimization suggestions for both code and infrastructure.

While the Alpha release marks a significant milestone, it is just the beginning. We encourage the community to start testing early preview versions of the OTel Profiles integration and contribute to the ongoing profiling work. To get started with an actual, local deployment, you can use the OpenTelemetry eBPF profiler in combination with a self-hosted Elastic Observability Stack or devfiler, a standalone desktop application that acts as an OpenTelemetry Profiles compliant backend aimed at experimentation and development.

OpenTelemetry has won the hearts of the industry. Adoption is accelerating: the CNCF's 2024 Observability survey found OTel to be the fastest-growing project in the foundation's history, with the OTel Collector registering hundreds of millions of downloads. The proposition is compelling: write instrumentation once, ship it anywhere, avoid lock-in.

But here is what every platform team discovers once they cross into production: the collector sprawl problem. Hundreds of collector instances deployed across regions, Kubernetes namespaces, and bare-metal hosts. Configuration drift creeping in. An upgrade that has to be co-ordinated across a fleet of independent processes. A security patch that someone has to manually roll out to each one. And zero visibility into which collectors are running, healthy, or stuck.

This is the gap between "deploying OpenTelemetry" and "operating OpenTelemetry at scale." With Elastic 9.3, Elastic Agent closes that gap entirely. The Elastic Agent is now built on Elastic's Distribution of the OpenTelemetry Collector (EDOT) and, when managed by Fleet, gives platform teams a single control plane for configuring, updating, and monitoring every OTel collector in their estate — all while remaining compatible with the Beats-based integrations they already rely on.

The Collector Sprawl Problem and Why It Matters

OpenTelemetry's success has created a quiet operational debt for many organisations. Individual teams adopt the collector for their services: logs here, metrics there, a custom pipeline for the new microservice. Without a centralised management layer, each of these collectors becomes an independent snowflake: its own config file, its own upgrade cycle, its own failure domain.

The consequences are predictable. Configuration drift means collectors running different versions of the same pipeline, producing subtly incompatible data. Compliance teams ask "show me all the places data is collected and where it goes", and the honest answer is a spreadsheet that's already out of date.

This isn't a niche problem. A Gartner analysis of enterprise observability programmes consistently identifies operational overhead as the top barrier to expanding OTel adoption beyond initial pilots. The technology works. The tooling to manage it at scale is what's been missing.

How Elastic Agent Became an OTel Collector

To understand the significance of this, it helps to understand what Elastic Agent used to be, and what it is now.

Elastic Agent acts as a supervisor process: Before version 9.3, it managed a collection of separate Beats sub-processes (Filebeat, Metricbeat, Winlogbeat and so on), each running its own input/output lifecycle, each consuming its own memory footprint. The agent coordinated them, but the fundamental model was a collection of discrete daemons running under a parent.

With 9.3, that model has been replaced. Elastic Agent is now itself an instance of the EDOT Collector: Elastic's hardened, production-supported distribution of the upstream OTel Collector. The architectural shift has three important consequences.

First, the process model simplifies dramatically. Instead of a supervisor managing multiple sub-process lifecycles, there is a single EDOT Collector process. This means a smaller memory footprint, fewer things that can fail independently, and fewer processes to observe for health and performance.

Second, Beats functionality is preserved, not discarded. Rather than forcing a breaking migration, Elastic has introduced Beats Receivers: beat inputs and processors re-packaged as native OTel receiver components. A Filestream input is enabled by a filebeatreceiver. The same Filebeat configuration YAML you write today is automatically translated into the corresponding EDOT receiver configuration at runtime. Existing integrations, dashboards, and ingest pipelines continue to work without modification.

Third, the agent is now a first-class participant in the OTel ecosystem. It speaks OTLP natively, it runs standard OTel receivers, and it can be configured to sit alongside any other OTel-compatible tool in a modern observability pipeline.

Central Management with Fleet: Configuration, Lifecycle, and Visibility

The architectural shift above would be valuable on its own. But it becomes transformative when combined with Elastic Fleet, the centralised management plane for Elastic Agents.

Fleet gives platform and SRE teams a single console from which to manage every Elastic Agent (and by extension, every EDOT Collector instance) in their estate. The capabilities break into three categories: configuration management, lifecycle management, and fleet-wide observability.

Configuration management at scale

With Fleet, you define an Agent Policy — a declarative description of what a collector should do. What data should it collect? Via which receivers? Where should it export? The policy is authored once in Fleet's UI (or via its API), and pushed automatically to every agent enrolled in that policy. Change the policy, and every affected collector receives the update. No SSH. No Ansible playbook to maintain. No configuration drift.

Fleet pushes policies to enrolled agents across any environment. Agents send heartbeat and health data back, giving a live inventory of every collector in the estate.

Lifecycle management: upgrades, enrolment, and remediation

Perhaps the most operationally significant benefit of Fleet management is lifecycle control. With Fleet, upgrading a collector is a policy action: select the target version, select the scope (all agents, a specific policy group, a canary subset), and click. Fleet orchestrates the rolling upgrade, tracking status per agent and surfacing failures immediately.

This changes the security calculus fundamentally. When a vulnerability is disclosed in the OTel Collector binary, patching is a Fleet operation measured in minutes, not a change-management ceremony measured in days across SSH sessions to individual hosts.

Fleet also handles enrolment and de-enrolment. New hosts added to your infrastructure can be auto-enrolled into the appropriate policy based on tags or deployment tooling. Agents on decommissioned hosts can be removed from Fleet's inventory, ensuring your observability map reflects your actual infrastructure.

Fleet-wide observability of your collectors

Every Fleet-managed Elastic Agent ships monitoring telemetry about itself: CPU and memory consumption, event throughput, error rates, pipeline latency. This data flows into Elastic and is surfaced in the Fleet UI, giving you a live dashboard of every collector in your estate, not just the ones you happen to be watching.

For the first time, "how healthy is my observability pipeline?" becomes a question with a real-time, fleet-wide answer. You can identify agents that have stopped sending data, agents consuming unexpectedly high resources, and agents that have fallen behind on queue processing — before those problems surface as gaps in your monitoring data.

In the near future this capability will be offered to non-Fleet managed agents (aka standalone) and/or 3rd party OTel collectors provided by other vendors. These collectors can be configured via some other means but be monitored in Fleet - from both resource consumption and/or component pipeline health.

The Hybrid Agent: Beats Data and OTel Data, Simultaneously

One of the most practically significant capabilities introduced in 9.3 is what Elastic calls the Hybrid Agent: an Elastic Agent that can run both Beats-based receivers and native OTel receivers in the same pipeline, at the same time. This does not change anything for existing installations.

This matters enormously for real-world adoption. Most organisations arriving at OTel in 2025 and 2026 are not starting from a blank slate. They have years of investment in Beats-based integrations: Filebeat-powered log collection, Metricbeat-powered host metrics, bespoke ingest pipelines in Elasticsearch that normalise and enrich that data into ECS (Elastic Common Schema) format. The business value locked in those integrations (the dashboards, the alerts, the correlation logic) is not something they can afford to throw away in order to "go OTel."

The Hybrid Agent solves this by making the two worlds coexist. For example, in a single agent policy you can simultaneously configure:

• A filebeatreceiver collecting application logs in ECS format, routed through your existing ingest pipeline to its existing data stream
• A native OTel filelog receiver collecting OTel-native telemetry from your new services instrumented with the OTel SDK, stored in OTel-native data streams without touching ingest pipelines
• An OTel hostmetrics receiver collecting system metrics in semantic convention format alongside your existing Metricbeat-derived system metrics

The two lanes are independent. Beats-receiver data travels through ingest pipelines and lands in ECS-formatted data streams, exactly as it always has. Native OTel data follows OTel semantic conventions and is stored directly in OTel-native data streams, bypassing ingest pipelines. Your existing dashboards and alerts continue to work. Your new OTel-native workloads get the full OTel experience. The same agent, the same Fleet policy, the same management console.

This co-existence is the practical answer to the question every platform team eventually faces: "We want to adopt OTel properly but we can't break what we already have." The Hybrid Agent lets you migrate incrementally, service by service, on your timeline.

The Integration Catalogue: Turning Configuration into a One-Click Operation

Configuration management at scale is only as good as the configurations themselves. Elastic's integration catalogue — over 500 packages covering everything from NGINX and PostgreSQL to AWS CloudTrail and Kubernetes — extends naturally to the Hybrid Agent model.

From 9.3 onwards, the catalogue includes OTel integration packages alongside the existing Beats-based ones. Each OTel package contains two components:

• An Input package: the configuration for the corresponding OTel receiver (receivers, processors, pipeline wiring), ready to be applied to a Hybrid Agent policy
• A Content package: the assets associated with the application: pre-built dashboards, alerts, index templates, and saved queries, all calibrated for OTel semantic convention data

When an operator adds an OTel integration to an Agent Policy in Fleet, the receiver configuration is pushed to all enrolled agents. When those agents start ingesting data and it arrives in Elasticsearch, the content package assets are automatically installed based on metadata in the data received. The dashboard is ready before you've had time to wonder where it is.

The same policy can hold both OTel integrations and legacy Beats integrations. A real-world agent policy might simultaneously collect system metrics via the OTel hostmetrics receiver, application logs via filebeat receiver, and APM data via OTLP — all from one policy, all managed from Fleet, all visible in a unified Kibana experience.

A technical walk through of how this is done for NGINX data collection can be found here for reference. Currently management of Elastic Agents is done via existing Fleet protocols, however in the near future this will move over to OPAMP so that Fleet will be able to provide management to 3rd party OTel collectors as well.

For organisations on platforms not yet in Elastic's OS support matrix, 3rd-party OTel Collectors (such as Red Hat's OpenShift-native collector) can send data to Elastic using the OTLP exporter and be observed alongside all other collectors in their fleet.

What This Means in Practice: A Migration Story

Consider a mid-sized platform team operating 200 Linux hosts across three regions, currently running Elastic Agent 8.x with a mix of Filebeat and Metricbeat integrations. Their new services are being instrumented with the OTel SDK and they want to standardise on OTel going forward without disrupting the monitoring coverage they already have.

With a Fleet-managed upgrade to 9.3, their existing agents become Hybrid Agents automatically. Their Filebeat and Metricbeat configurations are internally translated to Beats receiver configurations and continue to run unmodified. Their existing dashboards still populate. Their ingest pipelines still fire. Nothing breaks.

They then add OTel integration packages to their Fleet policies for each new service. The OTel-instrumented microservices start sending OTLP data, received by native OTel receivers in the same agents. OTel-native dashboards appear automatically in Kibana. They now have both data universes in one place, managed from one console, visible in one interface.

Over the following quarters, as Beats-based integrations for their remaining services are superseded by OTel equivalents in the catalogue, they migrate them one by one, updating the Agent Policy in Fleet and watching the transition happen across all 200 hosts simultaneously, without touching a single one directly.

Looking Forward

Elastic has made a clear architectural bet: OpenTelemetry is the future of observability data collection, and the right response to that future is not to build a parallel OTel tool alongside the existing stack — it is to evolve the existing stack into OTel. The Hybrid Agent and EDOT Collector are the result of that bet.

Fleet central management is the operational layer that makes that bet practical at scale. OpenTelemetry gives you standardised, vendor-neutral instrumentation. Fleet gives you the operational control plane to manage those collectors like the production infrastructure they are, not like artisanal YAML files scattered across your estate.

The collector sprawl problem is solvable. The answer is a managed, policy-driven, centrally observable fleet of EDOT Collectors, and in Elastic 9.3, that answer is production-ready today.

Observability for developers has lately been distilled into implementing auto-instrumentation, allowing you to instantly connect your code with the larger observability world. This way of utilizing an upstream SDK is certainly the simplest and most production-ready, and works efficiently with the Elastic Cloud Managed OTLP Endpoint.

But what if you could not only add powerful tracing to your Go service but also truly understand how the magic works, rather than just copy-pasting configuration files or a line of code? In the same way that you build your knowledge of software development systems, observability, modernized by OpenTelemetry (OTel) standardization, is a rich, broad system that is valuable to understand. Here is an in-depth technical breakdown of every piece of simple OTel instrumentation using the Elastic Distributions of OpenTelemetry (EDOT) and Golang, from the ground up.

Telemetry is the automated collection, transmission and analysis of data from your application, which can apply to any observable distributed system. This data can range from regular health check calls with your application to real-time information about user interactions, requests, and transactions. Using the example application repository here, we’ll build a strong observability foundation to start observing our applications with confidence.

Understanding the OpenTelemetry Flow

Below, you will see the basic flow of your data when implementing observability with OTel in your system. Before we dive in, let’s explain some of the key terms within OTel. Let’s go over these base key players that we need to implement observability solutions with OTel:

• Span: This is a single, timed unit of a distributed trace that can represent a specific operation, such as a database query or an HTTP handler.

• Trace: This is a detailed record of a single request’s journey through your system, AKA a hierarchy of your spans.

• Tracer: This is the handle for generating spans. You will typically have one per instrumentation library, for example myapp/http.

• Tracer Provider: This is the cornerstone of the SDK. This creates Tracer instances, and you can configure it on application start up.

• Context Propagation: The mechanism for passing trace context between operations and services, maintaining the relationship between parent and child spans.

• Exporter: This is the part that is responsible for sending your telemetry data to a vendor backend, and you can decide if you are sending it to the OTel Collector, EDOT Collector or an OTLP Endpoint.

Installing the Magic, Instrumentation Style

OpenTelemetry provides instrumentation libraries that handle much of the tracing complexity for you. These libraries wrap common frameworks and libraries (like net/http/otelhttp) and automatically capture telemetry without requiring you to manually create spans for every operation.

However, before you're able to send any telemetry, OTel needs to know who (which service) is sending that data.

A resource represents the specific entity, in this case "simple-go-service", that is producing your telemetry data. Its identity is recorded as resource attributes, and resource attributes can include pod names, service names or instances, deployment environments; Basically anything important to identifying your resource. This resource is your service's identity card that gets attached to every span and metric that it emits with its attributes. Once your trace arrives, these attributes can answer "what version was running?" or "which service is this from?"

func initOTel(ctx context.Context, endpoint string) (func(context.Context) error, error) {
res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName("simple-go-service"),
            semconv.ServiceVersion("1.0.0"),
        ),
    )
    if err != nil {
        return nil, err
    }

In the code above, resource.New() constructs the "identity card" of our Go service. The attributes that will be attached to it will use semantic conventions(semconv), standardized names for common metadata fields. These semantic conventions make sure that every single OTel-compatible observability backend knows their meaning.

Now that we've bootstrapped our application with the initOtel function, we can continue to configure everything else!

Let’s begin instrumenting this application by building all the app components that we will need to implement modern observability tools. Below is our instrumentation using otelhttp, which will handle span creation after calling the specified API routes. 

http.Handle("/hello", otelhttp.NewHandler(http.HandlerFunc(handleHello), "hello"))
http.Handle("/api/data", otelhttp.NewHandler(http.HandlerFunc(handleData), "data"))
http.HandleFunc("/health", handleHealth)

// Example of a tracer within our handleHello() function
tracer = tp.Tracer("simple-go-service")

ctx, span := tracer.Start(ctx, "process-hello")
defer span.End()

The key insight here is that otelhttp.NewHandler handles all the span lifecycle management for HTTP requests. You don't need to manually call tracer.Start()or span.End() for basic HTTP tracing since the library does this for you.

On application start up, the SDK will use the tracer provider set up below in order to create Tracer instances. These instances help create and manage the spans contained within traces.

traceExporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint(endpoint),
        otlptracegrpc.WithInsecure(),
    )

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(traceExporter),
        sdktrace.WithResource(res),
    )
    otel.SetTracerProvider(tp)
    tracer = tp.Tracer("simple-go-service")

Within our initOTel function, we will set up one of our most important signals: logs. First, we initialize the logExporter that will send logs to our OTel Collector using gRPC protocol. Then the LoggerProvider will create the base of the logExporter that batches log entries together before sending those batches to your exporter, attaching metadata about the services along the way. Lastly, the LoggerProvider also creates a standard Go structured logger (slog) that automatically includes trace context (such as span IDs) and batches your log with other logs. These are sent to your observability backend through the exporter along with your metrics and traces. 

logExporter, err := otlploggrpc.New(ctx,
        otlploggrpc.WithEndpoint(endpoint),
        otlploggrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    lp := sdklog.NewLoggerProvider(
        sdklog.WithProcessor(sdklog.NewBatchProcessor(logExporter)),
        sdklog.WithResource(res),
    )
    logger = slog.New(otelslog.NewHandler("simple-go-service", otelslog.WithLoggerProvider(lp)))

Below you can see how you can view your logs through Kibana in the APM UI. These logs are also color - coordinated; Coded warnings are in yellow, errors are in red, and regular logs are in green.

Metrics are set up in the next part of our code. Metrics are telemetry signals that track the quantitative data from your application, such as response times and request counts. The metric exporter is initialized to send metric data to our EDOT Collector then to our observability backend, Elastic Observability in this case, using gRPC. The meter provider in the next portion periodically collects and exports our metrics data and measurements, the same as the tracer provider creates tracers. The only difference between the two providers is that the meter provider works on a timer while the trace provider exports spans as they complete.

metricExporter, err := otlpmetricgrpc.New(ctx,
        otlpmetricgrpc.WithEndpoint(endpoint),
        otlpmetricgrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    mp := metric.NewMeterProvider(
        metric.WithReader(metric.NewPeriodicReader(metricExporter)),
        metric.WithResource(res),
    )
    otel.SetMeterProvider(mp)

    meter := mp.Meter("simple-go-service")
    requestCounter,  = meter.Int64Counter("http.requests")
    requestDuration,  = meter.Float64Histogram("http.duration")

In order to finish initializing OpenTelemetry, we set up our propagators for context propagation. The set text map propagator automatically injects the trace ID and the span ID of your service making an outbound HTTP request to another service, following the W3C Trace Context standard. In short, this maintains the parent-child relationship between spans.

otel.SetTextMapPropagator(propagation.TraceContext{})

    return func(ctx context.Context) error {
        tp.Shutdown(ctx)
        mp.Shutdown(ctx)
        lp.Shutdown(ctx)
        return nil
    }, nil

Now that you know how these pieces work together, try to run the repository linked here, using the readme as your guide.

Sidenote: Adding Custom Spans

For getting an application emitting traces, this instrumentation works great! If you visit localhost:8080/hello after starting the docker containers, the otelhttp middleware automatically creates spans for each HTTP request. However, basic instrumentation only shows essential application telemetry, such as response duration, URL paths, and status codes. You won’t know what happens between the request coming in and request completion. The moment OpenTelemetry truly gains power is when you add custom spans. Unlike auto-instrumentation where spans are created as well as closed automatically, custom spans require you to explicitly start and stop them.

Custom spans can track your application’s logic, such as specific business events or marking expensive operations, using a detailed hierarchy within each trace. In the application for this article, there are several custom spans that were created to track important operations:

• background-work: This traces asynchronous processing that happens with the main request.

• computation: This measures computations and then captures those results, and the computation type.

Custom spans add granular visibility into your application's behavior. For example, in performComputation:

ctx, span := tracer.Start(ctx, "computation")
defer span.End()

result := rand.Float64()
span.SetAttributes(
    attribute.String("comp.type", compType),
    attribute.Float64("comp.result", result),
    )

    logger.InfoContext(ctx, "Computation completed", "type", compType, "result", result)

if result < 0.3 {
span.AddEvent("Low confidence result")
    logger.WarnContext(ctx, "Low confidence computation", "result", result)
}
}

The attributes set above become searchable and filterable in our Elastic Observability backend, allowing for attribute filtering by attribute.result and attribute.compType. If you query your data with “show me all computations where results are less than 0.3,” then you will notice the span event span.AddEvent(“Low confidence result”) tacked on with a timestamped marker. This appears on your trace timeline as well, adding even more visibility to any unusual events. Below is a small example of the filtering that Kibana can accomplish from custom spans.

The Data Pipeline: From Code to IRL

Now that you can export your custom spans and data to OTLP which sends it to the EDOT Collector and then to an observability backend, the best hub for your telemetry data will be the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/. It is a simple, standalone process that is able to receive, process and export all of your telemetry data. Within this project, we use the Elastic Distributions of OpenTelemetry (EDOT) Collector, an optimized Collector for usage within your Elastic Stack. Since this is a self-managed Elastic instance, this article and connected repository utilize the EDOT Collector through elasticapm, but for Elastic Cloud or Serverless projects, you can use the Elastic Managed OpenTelemetry Protocol (OTLP) Endpoint. As noted in the quickstart documentation here, the Elastic Cloud Managed OTLP Endpoint endpoint helps get your data quickly and efficiently into your Elastic Stack through OTLP, without schema translation! This means that your telemetry hits Elastic instantly and your telemetry data remains vendor-neutral.

For most developers and SREs, this Collector is an amazing tool. It allows you to decouple your code from the observability backend. Your application does not need to know its final destination, it can just send the data to the Collector. Your observability backend can change constantly without it even touching your code. The OpenTelemetry Collector also acts as a gateway for multiple streams of data, and is able to accept various formats in order to unify them for exportation. Lastly, the OpenTelemetry Collector is able to offload processing power from your application - tasks such as retries, batching and filtering can happen in the Collector, not your application.

After trying out this article’s repository, try auto-instrumenting your application with Elastic Distributions of OpenTelemetry (EDOT) so that you can utilize the APM UI to its full potential! With the latest version of Elasticsearch and Kibana start-local, you can use Docker to install and run the services and instantly start monitoring your application. 

Understanding the Collector Configuration

The Collector's behavior is defined in a configuration file (otel-collector-config.yaml). Let's break down each component.

Receivers define how the Collector accepts telemetry data. Here, we're listening for both gRPC and HTTP traffic.

receivers:
  # Receives data from other Collectors in Agent mode
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

Connectors are specialized components that sit in between pipelines, and in this case, we are using the elasticapm Connector. This APM Connector exports our metrics, logs, and traces, while simultaneously acting as a receiver for the metrics/aggregated-otel-metrics pipeline (see below). Without it, your raw OTLP data lands in Elasticsearch, but the APM UI has nothing to build its views from.

connectors:
  elasticapm: {} # Elastic APM Connector

Processors transform, filter, or enrich data as it passes through the EDOT Collector. The batch processor aggregates spans before export, reducing network overhead and improving efficiency, as well as limiting batch sizes. The batch/metrics processor does this as well, but for APM metrics. Lastly, there is the Elastic APM processor. This processor ensures that your spans fields are aligned, your traces views are complete, and it overall bridges the gap between Elastic's expectations and OpenTelemetry's formatting of your traces.

processors:
  batch:
    send_batch_size: 1000
    timeout: 1s
    send_batch_max_size: 1500
  batch/metrics:
    send_batch_max_size: 0 # Explicitly set to 0 to avoid splitting metrics requests
    timeout: 1s
  elasticapm: {} # Elastic APM Processor

As mentioned previously in the article, exporters send data to your observability backend. The debug exporter logs telemetry to the console (useful for development), while the Elasticsearch exporter sends traces to your Elastic stack.

exporters:
  debug: {}
  elasticsearch/otel:
    endpoints:
      - ${ELASTIC_ENDPOINT} # Will be populated from environment variable
    user: elastic
    password: ${ELASTIC_PASSWORD}
    tls:
      ca_file: /config/certs/ca/ca.crt
    mapping:
      mode: otel

Pipelines connect receivers, processors, and exporters into a data flow. These EDOT Collector pipelines receive OTLP traces, batches them, and exports to the debug, elasticapm and elasticsearch/otel exporters. It also exports metrics to the debug and elasticsearch/otel exporters.

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch/metrics]
      exporters: [debug, elasticsearch/otel]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, elasticapm, elasticsearch/otel]
    traces:
      receivers: [otlp]
      processors: [batch, elasticapm]
      exporters: [debug, elasticapm, elasticsearch/otel]
    metrics/aggregated-otel-metrics:
      receivers:
        - elasticapm
      processors: [] # No processors defined in the original for this pipeline
      exporters:
        - debug
        - elasticsearch/otel

Debugging Your Code with Confidence in Kibana

Elastic Observability, utilizing Kibana and Streams, has native support for the OTLP Endpoint through the EDOT Collector, which was used in this project. Below, you can see that your data is automatically connected to Streams from the beginning, requiring no extra leg work! You can add conditions or any Grok processors as your data is streaming in, and you'll be able to instantly see your data's schema and data quality.

Elastic also provides the Elastic Cloud Managed Endpoint for even easier storage, data-processing, and scaling. If you use this Managed Endpoint, it means that you can configure OpenTelemetry to send data directly to Elasticsearch, without ANY specialized Collectors. Any way you choose, once your traces are flowing, Kibana’s APM UI provides powerful visualization and analysis capabilities will be everything you need to debug your code. You are able to drill down into individual requests, identify bottlenecks, find anomalies and troubleshoot any issues that arise with confidence.

Here is one span of interest from this repository. Within Kibana, you can immediately filter by the Trace ID, finding other spans with the same Trace ID to visually see the entire trace.

Kibana Discover also allows you to switch indices instantly without losing your filters, ensuring that you can also see the logs that correspond with the same Trace ID.

In addition to the manually checking your traces, you can automatically check them within the APM UI (shown below). This is easy trace visualization using the Kibana APM UI is readily available while using the elasticapm connector. Below is a visualization of a trace comprised of spans within our project. Knowing both methods of correlating spans is beneficial to build the foundation of utilizing Kibana and the APM UI for observability.

Here is a fully built out dashboard built from the repository featured in this article. The possibilities with Elastic Observability are endless!

Congrats, You’re Not “Just” a Developer Anymore!

We’ve broken down the why and how behind OpenTelemetry’s basic components, including the TraceProvider, the span, the exporter and the Collector. Here, you’ve done more than just implement your tracing tool. You now understand the complete data flow from your code to the graphs on your dashboard.

You can now speak the language of observability with confidence, not because you memorized a configuration file, but because you now understand the data flow from your code to the graph on your dashboard. You understand how telemetry moves through your system. You aren’t “just” a developer anymore; you’re now a developer who can truly see.

Try out the code repo above! Included in the [repository]() is a generate-traffic.sh script file. You can run this repeatedly in order to generate logs, traces, and metrics for you to play with within the APM UI. Also, check out our latest releases in our release docs page for exciting Elastic updates.

Since we announcement of the donation, we have been actively working on the project and are about to release a first beta version.

In this post, we will walk through what was donated, why it matters for production PHP systems, how it relates to existing OpenTelemetry PHP projects, and what is the status of the project.

Why PHP observability can still be hard

OpenTelemetry gives us a common standard, but deployment reality still matters. In many PHP environments, the blocker is not instrumentation APIs. The blocker is operations.

At Elastic, we are committed to open standards and to OpenTelemetry as the industry standard for observability data collection. To help the community with these operational constraints, we donated our EDOT PHP distribution to OpenTelemetry.

Common constraints include:

• Shared hosting or hardened servers without build toolchains
• Production images that cannot be rebuilt frequently
• Package-managed PHP runtimes where OS-native install flows are required
• Teams that need adoption without app code changes

This is where an OS-package distribution helps. If you can install a package and restart PHP, you can usually start collecting telemetry.

What the OpenTelemetry PHP distro provides

The project we donated combines native and PHP runtime components into one production path. The key features we announced in our original donation proposal are implemented and we are close to a first beta release. This includes:

• Native extension and loader artifacts so teams can install prebuilt components instead of compiling in restricted environments.
• Runtime/bootstrap logic for auto-instrumentation so applications can emit telemetry with little or no code changes.
• Packaging support for deb, rpm, and apk so rollout fits existing Linux package management and operations workflows.
• Background telemetry sending and automatic root span behavior so trace data is captured consistently without custom bootstrapping logic or blocking of the main flow.
• OTLP protobuf serialization works out of the box, with no need for the ext-protobuf extension. This means teams don't have to install extra dependencies, which is especially important in PHP environments where adding new extensions is difficult or restricted.
• Inferred spans so users get added visibility into work that is not explicitly instrumented in application code.
• URL grouping for transaction/root spans so high-cardinality route data is easier to aggregate and analyze.
• Built-in OpAMP support utilizing an OpAMP client already present in the agent.

For teams running PHP 8.1 through 8.4, this gives a practical onboarding path that fits existing OS package operations.

How installation looks in practice

A typical flow is simple:

1. Install distro package for your Linux platform.
2. Set exporter endpoint and auth headers.
3. Restart PHP processes.
4. Verify traces in your collector or backend.

Example environment variables:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-collector.example:4318"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <token>"

The key idea is predictable rollout through standard operational controls rather than custom build steps in each application pipeline.

For a detailed guide on how to setup the distro, see the setup documentation.

Relationship to existing OpenTelemetry PHP instrumentation

The current message from maintainers is coexistence with clear differentiation:

• Distro path: package-managed, production-first, zero-code onboarding
• Composer-centric path: more manual control and portability where that is needed

This distinction matters for users choosing a starting point. If you control application packaging tightly and can build extensions as part of app install, Composer-oriented paths can still be a fit. If you need an operations-first rollout through OS packages, the distro can reduce adoption friction.

The donation discussion also raised an important usability concern: too many overlapping options can confuse users. That is why compatibility and long-term alignment across projects is a key follow-up topic. The original proposal details are available in the OpenTelemetry community donation issue.

What to validate before broad production rollout

If you want to test this path in your own environment, validate these points early:

• Runtime coverage: verify your PHP version and SAPI mode (PHP-FPM, Apache mod_php, CLI)
• Packaging fit: confirm your distro package format and architecture support
• Telemetry behavior: check span completeness, service naming, and exporter reliability
• Operational safety: verify restart procedures, rollback steps, and version pinning policy

A lightweight validation matrix can save rework later, especially when multiple runtime profiles exist in the same organization.

The current status and what's next

With the completion of the initially-announced, above-mentioned features, we reached a significant milestone and are about to release a first beta version.

However, the work does not stop here. More enhancements and features are planned, including:

Class Shadowing

In some situations it is possible that the PHP distro loads classes that are already loaded by the application itself. This can lead to collisions and unexpected behavior. We are working on implementing shadowing of classes and namespaces of the PHP distro dependencies to avoid collisions. This new feature will increase the stability and reliability of the PHP distro across a wider range of applications.

Declarative Configuration Support

Just recently the OpenTelemetry community announced the stability of the declarative configuration specification. Including native support for the declarative configuration specification in the PHP distro is a feature that is on our roadmap and one of the next major features we will be working on. With declarative configuration support, teams will be able to configure the PHP distro in a more flexible way without having to modify their application code.

We are working on implementing declarative configuration support in the distro to allow for more fine-grained configuration of the agent. This new feature will increase the flexibility and usability of the PHP distro.

PHP 8.5 Support

In November 2025, PHP 8.5 was released with major changes to the language and runtime. We are working on supporting it in the PHP distro which will expand the scope of compatibility and usability of the PHP distro across a wider range of applications and environments.

Central Configuration Support

Both, the declarative configuration as well as the new proposal around telemetry policies are potential and promising enablers for dynamic, central configuration of OTel SDKs and distors. Once the discussion around telemetry policies is resolved, we will be able to implement central configuration support in the PHP distro. This will combine the concepts around declarative configuration, telemetry policies and the OpenTelemetry Agent Management Protocol (OpAMP) to provide a more flexible and powerful way to configure the PHP distro centrally.

Base EDOT on the upstream distribution

With the new OpenTelemetry PHP distro reaching a first beta release, we are working on basing Elastic's OTel PHP distribution (EDOT PHP) on the upstream distribution to increase the compatibility and avoid feature drift. This will ensure that the PHP distro is always up to date with the latest features and bug fixes from the OpenTelemetry community.

Conclusion

Our OpenTelemetry PHP distro donation is mainly about operational accessibility. It gives PHP teams a package-native way to adopt OpenTelemetry where build-time instrumentation is difficult. As community alignment progresses, we expect this to become a clearer and lower-friction option for production PHP observability. Try out the OpenTelemetry PHP distro repository, document gaps, and feed findings back to the maintainers.

In five years working with Enterprise accounts at Elastic, I have heard the same challenge again and again:

"We have several network monitoring tools, and we would love to correlate all of them into one platform."

For many organizations, the barrier to true correlation isn't a lack of data, but where that data lives. Frequently, we see SNMP metrics, flow data, and logs isolated in purpose-built silos or dashboards. Without a unified data store and a proper correlation engine, piecing together the full narrative — from a topology change to a performance degradation — becomes a manual, time-consuming puzzle.

When an incident happens, engineers become human correlation engines — manually jumping between systems, copying timestamps, cross-referencing device names, and trying to piece together what actually happened. A simple question like "Did this interface failure impact application performance?" requires querying multiple tools and mentally correlating the results.

The real cost isn't the tool licenses — it's the time lost during critical incidents.

This lab is my answer to a fundamental question: Can Elastic become the unified foundation that actually correlates network data?

More importantly, it demonstrates that Elastic is fully ready for network operations — capable of ingesting diverse telemetry and using AI to correlate relationships, identify root causes, and resolve issues in seconds instead of hours.

The Problem: Network Observability is Broken

Let me paint a typical scenario I encounter with enterprise network teams:

The Fragmented Reality:

• No single source of truth
• Manual correlation during incidents (15-30 minutes per event)
• Fragmented teams (network vs. platform engineers)
• Limited automation capabilities
• No AI-powered analysis

When a link goes down at 2 AM:

• Notice the alert - 2 minutes
• Log into monitoring tool to see the metric - 3 minutes
• Switch to traffic analyzer to check impact - 5 minutes
• Open log management to search for related messages - 10 minutes
• Manually correlate timestamps across systems - 8 minutes
• Create a ticket and copy context from multiple tools - 8 minutes

Time to initial diagnosis: 36 minutes

This workflow is expensive, error-prone, and doesn't scale.

The Vision: Elastic as a Unified Network Observability Platform

What if you could:

• Collect SNMP metrics, NetFlow, traps, and topology data in one platform
• Correlate network events with application performance automatically
• Generate executive dashboards without separate BI tools
• Use AI to analyze incidents in seconds, not hours
• Trigger alerting from network events

This is what this lab aims to demonstrate.

What I Built: A Production-Grade Network Simulation

To demonstrate how Elastic unifies network data, I needed a realistic environment that generates real-world telemetry. Enter Containerlab — a Docker-based solution that enables us to create a network simulation framework.

Lab Architecture

I simulated a Service Provider core network with:

• 7 FRR routers forming an OSPF Area 0 mesh
• 2 Ubuntu hosts for additional use cases
• 2 Layer 2 switches for access layer segmentation
• 3 telemetry collectors feeding Elastic Cloud

Total containers: 14

Deployment time: 12-15 minutes (fully automated)

Full deployment instructions and topology details are available in the GitHub repository README.

The Three Telemetry Pipelines: Proving Multi-Source Correlation

What makes this lab production-ready is its hybrid observability approach — proving that Elastic can unify disparate network data sources.

| Pipeline | Data Type | Collection Method | Collector | Use Case | | :---- | :---- | :---- | :---- | :---- | | SNMP Metrics | Interface stats, system health, LLDP topology | Active polling | OTEL Collector | Capacity planning, trend analysis | | NetFlow | Traffic flows | Push-based export | Elastic Agent | Top talkers, security investigation | | SNMP Traps | Interface up/down events | Event-driven | Logstash | Real-time incident detection |

This unified architecture proves Elastic can replace multiple specialized network monitoring tools with a single platform.

The Power of Correlation: One Platform, One Query

When a network incident occurs, you need to answer questions like:

• Which interface failed? (SNMP metrics)
• What traffic was affected? (NetFlow)
• What was the sequence of events? (SNMP traps)
• Which devices are downstream? (LLDP topology)

The Problem: modern tools offer separate modules glued together, forcing users to navigate different spaces for different sets of data.

The Reality: You still have to pivot. You see a spike in the Metrics module, but to see why, you have to open the Logs module and manually align the time picker. The data lives in different tables or backends, making true correlation impossible without human intervention.

The Elastic Difference: One Store, One Language, One AI

Elastic makes it simple. Whether it's an SNMP counter (metric), a NetFlow record (flow), or a Syslog message (log), it is all stored in a unified datastore powered by the Elasticsearch engine. This allows users to easily search across multiple datasets in a single query.

FROM logs-*
| WHERE host.name == "csr23" AND interface.name == "eth1"

Time required: 3 seconds

Furthermore, as you will see later, the exact location of the data becomes agnostic to the user when leveraging the AI Assistant.

Data Transformation: From Cryptic OIDs to Actionable Intelligence

Raw SNMP traps are notoriously difficult to interpret at a glance. In our current lab setup, the data arrives looking like this:

OID: 1.3.6.1.6.3.1.1.5.3
ifIndex: 2
ifDescr: eth1

While traditional Network Management Platforms (NMPs) handle OID translation natively, bringing that clarity into Elastic requires a specific configuration.

In this initial lab, we are intentionally working with this raw data to demonstrate how AI assistants can interpret these events even without pre-existing context.

However, the strategy for the next phase of this project is to implement Elasticsearch Ingest Pipelines. This will allow us to map raw OIDs to human-readable names. This step is crucial for bridging the gap between Network tools and Application Observability platforms, allowing network events to be instantly correlated with application errors and infrastructure logs.

The Target State

Once the pipeline is implemented in the next lab, we will transform that raw trap into searchable, meaningful data:

{
  "event.action": "interface-down",
  "host.name": "csr23",
  "interface.name": "eth1",
  "interface.oper_status_text": "Link Down"
}

The result:

• Human-readable fields
• Searchable dimensions for filtering
• Context for automation rules and dashboards
• Correlation keys for joining with metrics and flows

In our next blog post, we will walk through building the ingest pipeline that performs this transformation — step by step.

Intelligent Alerting: From Noise to Actionable Intelligence

Traditional network monitoring relies on simple threshold alerts — "interface down," "high CPU." These alerts flood your inbox but provide zero context about root cause, impact, or remediation.

The Lab's Approach: ES|QL + AI Assistant

1. Semantic Detection with ES|QL

Instead of generic threshold alerts, the lab uses ES|QL to detect specific event patterns:

FROM logs-snmp.trap-prod
| WHERE snmp.trap_oid == "1.3.6.1.6.3.1.1.5.3"
| KEEP @timestamp, host.name, interface.name, message

2. Automatic AI-Powered Investigation

When the alert triggers, it invokes the Observability AI Assistant with a structured investigation prompt that:

• Performs immediate triage (which device, which interface, when)
• Assesses OSPF impact and traffic rerouting
• Correlates with other recent failures
• Generates severity assessment and recommended actions

The Transformation

| Traditional Alerting | Intelligent Alerting (Elastic) | | :---: | :---: | | Email: "Interface down on csr23" | Structured analysis with device context | | Manual investigation: 20-30 min | AI-automated investigation: 90 seconds | | Engineer correlates across tools | Automatic cross-source correlation | | No business impact assessment | Severity + recommended actions included |

Accelerating Incident Response with the Elastic AI Assistant

This is where the Elastic AI Assistant demonstrates its operational value — moving beyond passive data collection to actively interpret and explain network events in real-time

When an engineer views a trap document in Discover and asks:

"Explain this log message"

The AI Assistant provides comprehensive analysis including:

• What happened: Plain-language explanation of the SNMP trap
• Device context: Router role, interface purpose, network position
• Impact analysis: OSPF neighbor status, traffic rerouting assessment
• Root cause possibilities: Physical layer, link layer, administrative causes
• Recommended actions: Immediate steps, investigation queries, validation checks
• Severity assessment: Business and technical impact rating

Manual Triage vs. AI-Assisted Investigation

| Before | After (Elastic AI) | | :---- | :---- | | Google the OID → 5 min | Click "Explain this log" → 20 seconds | | Open network diagram → 3 min | Topology context auto-provided | | Query multiple tools → 10 min | Cross-source correlation instant | | Assess business impact → 5 min | Impact analysis auto-generated | | Total: ~28 minutes | Total: ~20 seconds |

The Value Proposition: One Platform, One Data Model, One AI

What This Lab Demonstrates

Elastic provides:

• One unified platform for metrics, logs, flows
• One data model (SemConv) for consistent correlation
• One search interface (Kibana) for all network data
• One AI assistant that understands all your network telemetry
• AI-powered alerting with automated investigation

Business Impact

Efficiency Gains:

• 85% reduction in MTTR (36 min → 5 min for initial diagnosis)
• 90% reduction in manual correlation time
• Junior engineers gain access to AI-powered expert analysis

Operational Benefits:

• Network engineers focus on strategy, not tool-switching
• Cross-functional collaboration in one platform
• Reduced tool sprawl and management overhead

Lessons Learned

After building this lab, several key insights emerged regarding how network data fits into the broader observability ecosystem:

1. Extending Observability to the Network

Elastic is already the gold standard for high-volume logs and application traces. This lab demonstrates that the same engine seamlessly handles network telemetry without needing a separate, siloed tool.

• Scale: The same architecture that ingests petabytes of application logs easily handles millions of interface counters.
• Structure: Native support for complex nested documents allows for rich SNMP trap data (variable bindings) without flattening or losing context.
• Speed: Real-time search applies equally to network events, enabling sub-second troubleshooting.

2. OpenTelemetry Semantic Conventions (SemConv) as the Universal Translator

The power isn't just in storing the data, but in standardizing it. By mapping SNMP and NetFlow to the OpenTelemetry Semantic Conventions (SemConv), network data finally speaks the same language as the rest of the stack.

• Unified Search: Query across firewall logs, server metrics, and switch telemetry in a single search bar.
• Instant Visualization: Pre-built dashboards work immediately because the field names are standardized.
• Cross-Domain Correlation: Easily correlates a spike in application latency with a specific interface saturation event.

3. AI Assistants Thrive on Context

While the AI in this lab was powerful on its own, the experiment highlighted a critical realization: an AI Assistant becomes exponentially more effective when coupled with a specific Knowledge Base.

Context is King: The AI delivers better root cause analysis when provided with rich metadata, such as device roles and topology maps. Without it, the advice remains generic.

Pro Tip (and What’s Next):

To get organization-specific advice rather than generic suggestions, you need to feed the AI your documentation.

• The Goal: Create a Knowledge Base containing device roles, network topology diagrams, and troubleshooting procedures.
• The Next Step: In my next blog post, I will demonstrate exactly how to do this — connecting a Knowledge Base to the AI Assistant to enable fully context-aware troubleshooting.

Conclusion: Completing the Observability Picture

Elastic is already widely recognized as the standard for Application and Security observability. The goal of this lab wasn't to ask if Elastic can handle networking, but to demonstrate the immense value of bringing network data into that existing ecosystem.

The verdict is clear: Elastic acts as that unified foundation. It effectively breaks down the silo between Network Engineering and the rest of IT.

This isn't just about consolidating dashboards or replacing legacy tools. It is about establishing the Elasticsearch AI Platform as the single source of truth where network telemetry sits right alongside application and infrastructure data.

By treating network data as a first-class citizen in the observability stack, we unlock automated correlation, AI-assisted investigation, and the speed required to resolve incidents before they impact the business. The capabilities are in place, and the foundation is solid — Elastic is ready to unify your network with the rest of your digital business.

Ready to Try It Yourself?

Check out github.com/DeBaker1974/Containerlab-OSPF

The repository includes:

• Complete deployment scripts (12-15 minute automated setup)
• Pre-configured telemetry pipelines
• Kibana dashboards
• Alert rules with AI Assistant integration
• Detailed README

Not ready to build? Try Elastic Serverless: Start a free 14-day trial and explore AI-powered observability with your own data.

Special thanks to the Containerlab and FRRouting communities for their incredible open-source tools, and to Sheriff Lawal (CCIE, CISSP), Sr. Manager, Solutions Architecture at Elastic, for mentoring on this project.

However, the landscape of log management is shifting. By combining the industry-standard ingestion of OpenTelemetry (OTel) with the AI-driven capabilities of Elastic Streams, we can change how we monitor Windows infrastructure. This approach isn't just moving data. We are also using Large Language Models (LLMs) to understand it.

The Challenge with Traditional Windows Logging

Windows generates a massive variety of logs: System, Security, Application, Setup, and Forwarded Events. Within those categories, you have thousands of Event IDs. Historically, getting this data into an observability platform involved installing proprietary agents and configuring complex pipelines to strip out the XML headers and format the messages.

Once the data was ingested, we can try to figure out what "bad" looked like. You had to know in advance that Event ID 7031 indicated a service crash, and then write a specific alert for it. If you missed a specific Event ID or if the format changed, your monitoring went dark.

Step 1: Ingestion via OpenTelemetry

The first step in modernizing this workflow is adopting OpenTelemetry. The OTel collector has matured significantly and now offers robust support for Windows environments. By installing the collector directly on Windows servers, you can configure receivers to tap into the event log subsystems.

The beauty of this approach is standardization. You aren't locked into a vendor-specific shipping agent. The OTel collector acts as a universal router, grabbing the logs and sending them to your observability backend in this case, the Elastic logs index designed to handle high-throughput streams.

The key thing to pay attention to in this configuration is how we add this transform statement:

transform/logs-streams:
  log_statements:
    - context: resource
      statements:
        - set(attributes["elasticsearch.index"], "logs")

This works with the vanilla opentelemetry collector and when the data arrives in Elastic, it tells Elastic to use the new wired streams feature which enables all the downstream AI features we discuss in later steps.

Checkout my example configuration here

Step 2: AI-Driven Partitioning

Once the data arrives, the next challenge is organization. Dumping all Windows logs into a single logs-* index is a recipe for slow queries and confusion. In the past, we split indices based on hardcoded fields. Now, we can use AI to "fingerprint" the data.

This process involves analyzing the incoming stream to identify patterns. The system looks at the structure and content of the logs to determine their origin. For example, it can distinguish between a Windows Security Audit log and a Service Control Manager log purely based on the data shape.

The result is automatic partitioning. The system creates separate, optimized "buckets" or streams for each data type. You get a clean separation of concerns, Security logs go to one stream, File Manager logs to another, without having to write a single conditional routing rule. This partitioning is crucial for performance and for the next phase of the process: analysis.

Step 3: Significant Events and LLM Analysis

Once your data is partitioned (e.g., into a dedicated Service Control Manager stream), you can apply GenAI models to analyze the semantic meaning of that stream.

In a traditional setup, the system sees text strings. In an AI-driven setup, the system understands context. When an LLM analyzes the Service Control Manager stream, it identifies what that system is responsible for. It knows that this specific component manages the starting and stopping of system services.

Because the model understands the purpose of the log stream, it can generate suggestions for what constitutes a "Significant Event." It doesn't need you to tell it to look for crashes; it knows that for a Service Manager, a crash is a critical failure.

From Passive Storage to Proactive Suggestions

The workflow effectively automates the creation of detection rules. The LLM scans the logs and generates a list of potential problems relevant to that specific dataset, such as:

• Service Crashes: High severity anomalies where background processes terminate unexpectedly.
• Startup/Boot Failures: Critical errors preventing the OS from reaching a stable state.
• Permission Denials: Security-relevant events regarding service interactions.

It bubbles these up as suggested observations. You can review a list of potential issues, see the severity the AI has assigned to them (e.g., Critical, Warning), and with a single click, generate the query required to find those logs.

Conclusion

The combination of OpenTelemetry for standardized ingestion and AI-driven Streams for analysis turns the chaotic flood of Windows logs into a structured, actionable intelligence source. We are moving away from the era of "log everything, look at nothing" to an era where our tools understand our infrastructure as well as we do.

The barrier to effective monitoring is no longer technical complexity. Whether you are tracking security audits or debugging boot loops, leveraging LLMs to partition and analyze your streams is the new standard for observability.

Try Streams today

These latest updates, available in v9.3 and in Serverless, introduce significant performance gains, sophisticated time series functions, and native OpenTelemetry exponential histogram support that directly benefit SREs and Observability practitioners.

Query Performance and Storage Optimizations

Speed is paramount when diagnosing incidents. Compared to prior releases, we have achieved a 5x+ improvement in query latency when wildcarding or filtering by dimensions. Additionally, storage efficiency for OpenTelemetry metrics data has improved by approximately 2x, significantly reducing the infrastructure footprint required to retain high-volume observability data. If you’re hungry to learn more about what architectural updates are driving these optimizations, stay tuned… Tech blogs are on their way! 

Expanded Time Series Analytics in ES|QL

The ESQL TS source command, which targets time series indices and enables time series aggregation functions, has been significantly enhanced to support complex analytics capabilities.

We have expanded the library of time series functions to include essential tools for identifying anomalies and trends.

• PERCENTILE_OVER_TIME, STDDEV_OVER_TIME, VARIANCE_OVER_TIME: Calculate the percentile, standard deviation, or variance of a field over time, which is critical for understanding distribution and variability in service latency or resource usage.

Example: Seeing the worst-case latency in 5-minute intervals.

TS metrics*  | STATS MAX(PERCENTILE_OVER_TIME(kafka.consumer.fetch_latency_avg, 99))
  BY TBUCKET(5m)

• DERIV: This command calculates the derivative of a numeric field over time using linear regression, useful for analyzing the rate of change in system metrics.

Example: trending gauge values over time.

TS metrics*  | STATS AVG(DERIV(container.memory.available))
  BY TBUCKET(1 hour)

• CLAMP: To handle noisy data or outliers, this function limits sample values to a specified lower and upper bound.

Example: handling saturation metrics (like CPU or Memory utilization) where spikes or measurement errors can occasionally report values over 100%, making the rest of the data look like a flat line at the bottom of the chart.\

TS metrics*  | STATS AVG(CLAMP(k8s.pod.memory.node.utilization, 0, 100))
  BY k8s.pod.name

• TRANGE: This new filter function allows you to filter data for a specific time range using the @timestamp attribute, simplifying query syntax for time-bound investigations.

Example: Filtering and showing metrics for the last 4 hours.

TS metrics*  | WHERE TRANGE(4h) | STATS AVG(host.cpu.pct)
  BY TBUCKET(5m)

Window Functions To smoothen results over specific periods, ES|QL now introduces window functions. Most time series aggregation functions now accept an optional second argument that specifies a sliding time window. For example, you can calculate a rate over a 10-minute sliding window while bucketing results by minute.

Example: Calculating the average rate of requests per host for every minute, using values over a sliding window of 5 minutes.

TS metrics*  | STATS AVG(RATE(app.frontend.requests, 5m))
  BY TBUCKET(1m)

Accepted window values are currently limited to multiples of the time bucket interval in the BY clause. Windows that are smaller than the time bucket interval or larger but not a multiple of the time bucket interval will be supported in feature releases. 

Native OpenTelemetry Exponential Histograms

Elastic now provides native support for OpenTelemetry exponential histograms, enabling efficient ingest, querying, and downsampling of high-fidelity distribution data.

We have introduced a new exponential_histogram field type designed to capture distributions with fixed, exponentially spaced bucket boundaries. Because these fields are primarily intended for aggregations, the histogram is stored as compact doc values and is not indexed, optimizing storage efficiency. These fields are fully supported in ES|QL aggregation functions such as PERCENTILES, AVG, MIN, MAX, and SUM.

You can index documents with exponential histograms automatically through our OTLP endpoint or manually. For example, let’s create an index with an exponential histogram field and a keyword field:

PUT my-index-000001
{
  "settings": {
    "index": {
      "mode": "time_series",
      "routing_path": ["http.path"],
      "time_series": {
        "start_time": "2026-01-21T00:00:00Z",
        "end_time": "2026-01-25T00:00:00Z"
     }
    }
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "http.path": {
        "type": "keyword",
        "time_series_dimension": true
      },
      "responseTime": {
        "type": "exponential_histogram",
        "time_series_metric": "histogram"
      }
    }
  }
}

Index a document with a full exponential histogram payload:

POST my-index-000001/_doc
{
  "@timestamp": "2026-01-22T21:25:00.000Z",
  "http.path": "/foo",
  "responseTime": {
    "scale":3,
    "sum":73.2,
    "min":3.12,
    "max":7.02,
    "positive": {
      "indices":[13,14,15,16,17,18,19,20,21,22],
      "counts":[1,1,2,2,1,2,1,3,1,1]
    }
  }
}

POST my-index-000001/_doc
{
  "@timestamp": "2026-01-22T21:26:00.000Z",
  "http.path": "/bar",
  "responseTime": {
    "scale":3,
    "sum":45.86,
    "min":2.15,
    "max":5.1,
    "positive": {
      "indices":[8,9,10,11,12,13,14,15,16,17,18],
      "counts":[1,1,1,1,1,1,1,2,1,1,2]
    }
  }
}

And finally, query the time series index using ES|QL and the TS source command:

TS my-index-000001  | STATS MIN(responseTime), MAX(responseTime),
        AVG(responseTime), MEDIAN(responseTime),
        PERCENTILE(responseTime, 90)
  BY http.path

Enhanced Downsampling

Downsampling is essential for long-term data retention. We have introduced a new "last value" downsampling mode. This method exchanges accuracy for storage efficiency and performance by keeping only the last sample value, providing a lightweight alternative to calculating aggregate metrics.

You can configure a time series data stream for last value downsampling in a similar way as regular downsampling, just by setting the downsampling_method to last_value. For example, by using a data stream lifecycle:

PUT _data_stream/my-data-stream/_lifecycle
{
  "data_retention": "7d",
  "downsampling_method": "last_value",
  "downsampling": [
     {
       "after": "1m",
       "fixed_interval": "10m"
      },
      {
        "after": "1d",
        "fixed_interval": "1h"
      }
   ]
}

In Conclusion

These enhancements mark a significant step forward in Elastic's metrics analytics capabilities, delivering 5x+ faster query latency, 2x storage efficiency and specialized commands like DERIV, CLAMP, and PERCENTILE_OVER_TIME. With native support for OpenTelemetry exponential histograms and expanded downsampling options, SREs can now perform richer, more cost-effective analysis on their observability data. This release empowers teams to detect anomalies faster and manage long-term metrics retention with greater efficiency.

We welcome you to try the new features today!

Today, we introduce EDOT Cloud Forwarder, built on OTel Collector, it is the simplest, fastest, and possibly most boring way to connect your Cloud environment to Elastic Observability, and it is now Generally Available on AWS. With EDOT Cloud Forwarder you can get started in seconds, get observability across your entire cloud estate, and easily handle telemetry at any volume.

Deploying Cloud Forwarder from a District Line train

So, I got to work deploying EDOT Cloud Forwarder in my AWS account. I was doing it on the commute, using nothing more than a decent 4G signal and hoping that my 27% battery would be enough.

I hit deploy on the terraform template and waited. I started seeing events flowing into Elastic Observability.

As the train pulled into Putney Bridge, the flow of logs peaked, and one million events per second scrolled across my screen.

Once deployed, I got the best three zeros I could expect:

• Zero Intervention: I watched as traffic ramped up to a full 1M EPS. Lambda functions automatically scaled out from a few instances to the 60-65 concurrent executions needed. There were zero manual adjustments required. The scaling was instant and hands-free.
• Zero Data Loss: It achieved a consistent processing rate, with every single event indexed in Elasticsearch.
• Zero Idle Cost: When there are no events, Cloud Forwarder scales to zero - it has no fixed infrastructure cost. You only pay for the moment data is being processed, not for permanently over-provisioned servers sitting idle.

Right before the train came to a standstill, I looked at the total cost for running Cloud Forwarder for the two minutes between Parsons Green and Putney Bridge - we forwarded about 120GB of telemetry and the total cost was below £0.10. Well, unless you count the £2.50 train ticket!

Making Observability easy at any scale

Getting started observing your infrastructure is hard and once it's observable, deriving actionable value from it requires sifting and winnowing through massive volumes of telemetry data, sometimes millions of events per second.

The new EDOT Cloud Forwarder for AWS (also available in Preview for GCP and Azure) is easy to deploy, with just a Terraform template. To make sure that it was easy to get started with, we designed it to be as close to a single-click deployment as possible:

Just click the link below to launch the CloudFormation stack in your AWS account:

The best part? The fastest way to get started with Elastic Observability scales to any size workload! With EDOT Cloud Forwarder you have one solution which automatically scales down to zero and up to millions of events per second.

So, what is EDOT Cloud Forwarder?

EDOT Cloud Forwarder is a serverless OpenTelemetry Collector that, on AWS, runs as a Lambda function. In AWS, it is triggered by events and processes logs and metrics from services such as VPC Flow Logs, ELB Access Logs, CloudTrail, CloudWatch Logs and CloudWatch Metrics.

It has the following core capabilities:

• Collects observability and security data from Cloud Service Providers
• Parses data into native OpenTelemetry format
• Forwards data over OTLP
• Scales up and down based on traffic

ECF for AWS is a pure serverless solution, no VMs, containers, or Kubernetes control planes to manage.

Off the train, a more controlled scenario

For a more controlled testing scenario, we used synthetic VPC Flow Log data generated to show how easy EDOT Cloud Forwarder can sustain a million events per second reliably and without data loss.

For the configuration, we left all EDOT Cloud Forwarder configuration/settings at their defaults. In AWS, the Lambda max concurrency defaults to 5. If left to the default of 5, up to around 50k EPS could be expected. For our scenario, we bumped this to 100 to ensure we'd have plenty of headroom for our test.

We ran the scenario in 10 minute stages, with each stage resulting in a larger data volume. We held ingest flat during the stage to provide short-term steady state windows for us to grab metrics.

We experienced no errors across all stages of the scenarios, no retries outside expected behavior, no data loss across 5.4 billion ingested events.

Stats for Observability Nerds

Because we know you love them:

Incremental Load Stages

We tested EDOT Cloud Forwarder using incremental load stages, gradually increasing traffic from approximately 300,000 events per second to over 1 million events per second.

The graph below shows the Elasticsearch ingestion rate throughout the entire test duration. You can see the clear progression as we ramped up through six distinct stages, with each plateau representing a 10 minute stabilization period.

The system handled each traffic increase smoothly, culminating in sustained 1 million documents per second ingestion with no bottlenecks or data loss.

Lambda

As seen in CloudWatch metrics (no manual adjustments required). No errors and no throttles were seen during the full duration of the test.

Concurrent executions

60 to 65 instances running at the same time.

Average execution time per Lambda

Each execution is taking about 5 seconds.

Memory Usage

Memory use stabilized at around 450 MB, below the default limit of 512 MB.

Elasticsearch Indexing

Elasticsearch indexed one million documents per second, with events visible in Discover within seconds and no indexing delays or bottlenecks.

Efficient by design: Lambda at 1M EPS from S3

Running ECF for AWS at 1M EPS costs about $3.87 per hour. Around 66 percent ($2.57 per hour) is data transfer (same region), 34 percent ($1.32 per hour) is Lambda compute, and less than 1 percent is S3 requests.

This is fully serverless with no idle cost. You only pay while events are forwarded. With S3, data arrives pre-batched in large objects, which keeps Lambda invocations low and compute costs tightly controlled. At sustained throughput, Lambda costs are comparable to EKS—but without cluster management or idle capacity.

Other options: OTel Collector on EKS at 1M events per second

An OTel Collector on EKS sized for 1M EPS has a baseline cost of about $3.69 per hour. Roughly $0.33 per hour of this is compute related, EC2 nodes, EKS control plane, and EBS. The rest comes from data transfer and SQS, which scale with traffic and do not change with utilization.

Idle compute impact on EKS real costs

Considering EKS is typically provisioned for peak load, the real cost of the compute portion is affected by idle capacity. At 100 percent utilization, total cost is $3.69 per hour. At 50 percent utilization, a common baseline to absorb burstiness, total cost rises to about $4.02 per hour. At 30 percent utilization, it increases further to about $4.46 per hour.

Pay for Work vs Pay for Capacity

ECF for AWS delivers 1M EPS at a cost comparable to EKS at peak utilization, with no idle compute or capacity planning required. EKS can reach the same peak throughput, but total cost increases further as average utilization drops because compute capacity must be provisioned in advance.

Conclusion

The boring truth: to the EDOT Cloud Forwarder, a million events per second is no different from any other workload.

With no infrastructure to deploy, no idle cost and no manual scaling, I guess the best thing to do is to stop overthinking and start forwarding! It's like stepping onto the fastest train on the line: you just get on and you're instantly en route to your destination, effortlessly handling any distance or in this case, any volume.

So, we're shipping it. ECF for AWS is now Generally Available.

Get Started

1. Deploy EDOT Cloud Forwarder via CloudFormation (below) or using the AWS Serverless Application Repository

1. Create an Observability project using an Elastic Cloud free trial or deploy locally with start-local if you don't already have an Elastic project or deployment.

Visit EDOT Cloud Forwarder for AWS Documentation for more details.

Check out these other resources on OpenTelemetry at Elastic

Discover how Elastic is evolving data ingestion with OpenTelemetry

Learn how OpAMP enables centralized configuration of OpenTelemetry SDKs

This article explains why Nginx tracing is important in an application context, and provides a practical guide to enable the Nginx Otel tracing module exporting spans directly to Elastic APM.

Why Nginx Tracing Matters for Modern Observability

Instrumenting only backend services gives you only half the picture.
Nginx sees:

• every incoming request
• client trace context
• TLS negotiation
• upstream errors (502, 504)
• edge-layer latency
• routing decisions

If Nginx is not in your traces, your distributed trace is incomplete.

By adding OpenTelemetry tracing at this ingress layer, you unlock:

1. Full trace continuity : From browser → Nginx → backend → database.

2. Accurate latency attribution : Edge delays vs. backend delays are clearly separated which unlock Elastic APM Latency anomaly detection for proactive detection.

3. Error root-cause clarity : Nginx errors appear as spans instead of backend “mystery gaps”.

4. Complete service topology : Your APM service map finally shows the real architecture.

Integrating Nginx with OpenTelemetry on Debian

This guide provides a comprehensive overview of why, how to install and configure the Nginx OpenTelemetry module on a Debian-based system. The configuration examples are tailored to send telemetry data directly to an Elastic APM endpoint whether it's an EDOT Collector or mOtel in case of our serverless, enabling end-to-end distributed tracing.

Installation on Debian

The Nginx OTEL module is not included in the standard Nginx packages. It must be installed along with a working nginx configuration.

Prerequisites

First, install the necessary tools for compiling software and the Nginx development dependencies.

sudo apt update
sudo apt install -y apt install nginx-module-otel

Load the Module in Nginx

Edit your main /etc/nginx/nginx.conf file to load the new module. This directive must be at the top level, before the http block.

# /etc/nginx/nginx.conf

load_module modules/ngx_otel_module.so;

events {
    # ...
}

http {
    # ...
}

Now, test your configuration and restart Nginx.

sudo nginx -t
sudo systemctl restart nginx

Configuration

Configuration is split between the main nginx.conf file (for global settings) and your site-specific server block files.

Global Configuration (/etc/nginx/nginx.conf)

This configuration sets up the destination for your telemetry data and defines global variables used for CORS and tracing. These settings are placed inside the http block.

http {
    ...

    # --- OpenTelemetry Exporter Configuration ---
    # Defines where Nginx will send its telemetry data directly to Elastic APM or EDOT.
    otel_exporter {
        endpoint https://<ELASTIC_URL>:443;
        header Authorization "Bearer <TOKEN>";
    }

    # --- OpenTelemetry Service Metadata ---
    # These attributes identify Nginx as a unique service in the APM UI.
    otel_service_name nginx;
    otel_resource_attr service.version 1.28.0;
    otel_resource_attr deployment.environment production;
    otel_trace_context propagate; # Needed to propagate the RUM traces to the backend

    # --- Helper Variables for Tracing and CORS ---
    # Creates the $trace_flags variable needed to build the outgoing traceparent header.
    map $otel_parent_sampled $trace_flags {
        default "00"; # Not sampled
        "1"     "01"; # Sampled
    }

    # Creates the $cors_origin variable for secure, multi-origin CORS handling.
    map $http_origin $cors_origin {
        default "";
        "http://<URL_ORIGIN_1>/" $http_origin; # Add your Origin here to allow CORS
        "https://<URL_ORIGIN_2>/" $http_origin; # Add your others Origin here to allow CORS
    }
...
}

Server Block Configuration (/etc/nginx/conf.d/site.conf)

This configuration enables tracing for a specific site, handles CORS preflight requests, and propagates the trace context to the backend service.

server {
    listen 443 ssl;
    server_name <WEBSITE_URL>;

    # --- OpenTelemetry Module Activation ---
    # Enable tracing for this server block.
    otel_trace on;
    otel_trace_context propagate;

    location / {
        # --- CORS Preflight (OPTIONS) Handling ---
        # Intercepts preflight requests and returns the correct CORS headers,
        # allowing the browser to proceed with the actual request.
        if ($request_method = 'OPTIONS') {
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
            add_header 'Access-Control-Allow-Headers' 'Content-Type, traceparent, tracestate' always;
            add_header 'Access-Control-Max-Age' 86400;
            add_header 'Access-Control-Allow-Origin' "$cors_origin" always;
            return 204;
        }

        # --- OpenTelemetry Trace Context Propagation ---
        # Manually constructs the W3C traceparent header and passes the tracestate
        # header to the backend, linking this trace to the upstream service.
        proxy_set_header traceparent      "00-$otel_trace_id-$otel_span_id-$trace_flags";
        proxy_set_header tracestate       $http_tracestate;

        # --- Standard Proxy Headers ---
        proxy_set_header Host             $host;
        proxy_set_header X-Real-IP        $remote_addr;
        proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # --- Forward to Backend ---
        # Passes the request to the actual application (eg. localhost in this example).
        proxy_pass http://<BACKEND_URL>:8080;
    }
}

Test your configuration and restart Nginx.

sudo nginx -t
sudo systemctl restart nginx

Conclusion: Turning Nginx into a First-Class Observability Signal

By enabling OpenTelemetry tracing directly in Nginx and exporting spans to Elastic APM (via EDOT or Elastic’s managed OTLP endpoint), you bring your ingress layer into the same observability model as the rest of your stack. The result is:

• true end-to-end trace continuity from the browser to backend services
• clear separation between edge latency and application latency
• immediate visibility into gateway-level failures and retries
• accurate service maps that reflect real production traffic

Most importantly, this approach aligns Nginx with modern observability standards. It avoids proprietary instrumentation, fits naturally into OpenTelemetry-based architectures, and scales consistently across hybrid and cloud-native environments.

Try it out!

Once Nginx tracing is in place, several natural extensions can further improve your observability posture:

• correlate Nginx traces with application logs and metrics using Elastic’s unified observability
• add Real User Monitoring (RUM) to close the loop from frontend to backend
• introduce sampling and tail-based decisions at the collector level for cost control
• use Elastic APM service maps and anomaly detection to proactively detect edge-related issues

Instrumenting Nginx is often the missing link in distributed tracing strategies. With OpenTelemetry and Elastic, that gap can now be closed in a clean, standards-based, and production-ready way.

If you want to experiment with this setup quickly, Elastic Serverless provides the fastest way to get started. Sign up and try it out in just a few minutes using our trial environment available at https://cloud.elastic.co/ .

OpenTelemetry is quickly becoming the standard foundation for modern observability. Organizations want its open ecosystem, unified model, and vendor-neutral instrumentation—but moving a mature production environment to OTel is rarely straightforward.

Most teams already rely on battle-tested pipelines for logs, metrics, and security signals. They have dashboards tuned over years, operational practices built around existing data flows, and mission-critical systems where disruption simply isn’t an option.

This means the question isn’t "Why OpenTelemetry?" It’s "How do we get there without breaking what already works?"

Elastic Observability introduces a way to ingest telemetry without disrupting existing data and dashboards with Hybrid ingestion. Released in Elastic 9.2, its a low-friction way to adopt OTel receivers alongside existing native Elastic integrations-managed centrally through Fleet.

This hybrid approach offers one of the most pragmatic and operationally safe routes to OTel adoption available today.

The Challenge: Adopting OTel Without Disrupting the Present

For many organizations, the path to OTel adoption is complicated by realities such as:

• Established log pipelines powering critical alerting
• Legacy infrastructure that isn’t easily re-instrumented
• Existing dashboards and visualizations built on Elastic-native datasets
• Teams with different levels of OTel experience
• Risk constraints that make large changes difficult to roll out

Standardizing on OTel is the right long-term direction, but replacing everything at once is neither realistic nor desirable.

Teams need a way to bring OTel into their environment incrementally, while preserving continuity, reliability, and central governance.

Elastic Agent 9.2+: Hybrid Ingestion as a Bridge to the Future

Elastic Agent now supports two fully supported ingestion paths, both running inside the same unified agent:

1. Elastic-native integrations

Perfect for logs and host-level telemetry, with mature dashboards, alerts, and ECS mappings.

1. OpenTelemetry input integrations (OTel receivers)

Powered by upstream OTel Collector components, managed directly from Fleet.

And crucially:

You can use both, simultaneously, on the same agent.

This hybrid ingestion model allows teams to:

• Continue collecting logs using native Elastic integrations
• Begin collecting metrics or traces via OTel receivers
• Maintain full control through Fleet
• Introduce OTel exactly where and when it makes sense
• Avoid running parallel agents or duplicate pipelines

It’s a way to evolve—not replace—your observability strategy.

A Practical Example: Adding OTel Inputs While Keeping Your Existing Pipelines

Imagine a system where NGINX logs are already handled via Elastic-native integrations. These pipelines drive dashboards, audits, and critical alerts. Interrupting them isn’t an option.

At the same time, your platform team wants to standardize metrics and service telemetry using OpenTelemetry.

With Elastic Agent hybrid ingestion, both goals align:

1. Keep your existing log integration in Fleet
2. Add an OTel input integration (e.g., OTel nginxreceiver)
3. Fleet deploys both inside the same Elastic Agent
4. Deployment is done at scale across your infrastructure from a single management console
5. Logs and OTel metrics flow into Elasticsearch side-by-side

No re-instrumentation. No duplicate agents. No loss of historical visibility. No new tooling for operations. No external deployment tool.

Whether the component is a web server, reverse proxy, database, JVM runtime, or custom service already instrumented in OTel, the workflow is the same.

Why This Hybrid Approach Matters Strategically

Hybrid ingestion is not simply a technical capability—it’s an organizational enabler for OpenTelemetry transformation.

Incremental migration without downtime

Teams can begin adopting OTel at the exact pace they’re comfortable with. Existing collection signals remain stable. OTel metrics or logs are added progressively.

Fleet remains your single control plane

Fleet continues to manage:

• agent lifecycle
• policy management
• version upgrades
• diagnostics and monitoring

Even as OTel becomes part of your ingestion strategy.

Consistent semantics across teams

Adopting OTel receivers through EDOT helps harmonize telemetry models across microservices, infrastructure, and applications.

OTel becomes the shared language—Elastic becomes the scalable backend.

Future-proof flexibility

When the day comes that a team needs advanced OTel features, custom pipelines, custom processors, or additional exporters, they can build their own EDOT custom collector flavor and use it in their elastic-agent in hybrid mode.

This allows deep customization without abandoning the Elastic Agent runtime.

No vendor lock-in—full ecosystem alignment

Hybrid ingestion leverages upstream OpenTelemetry components directly. This reinforces the open, vendor-neutral ecosystem organizations prefer when standardizing observability across teams while being supported by Elastic.

What About Standalone Mode? (Advanced Use Cases)

While Fleet-managed hybrid ingestion will meet the needs of most users, Elastic Agent in hybrid mode also support standalone deployment with the same functions as the managed version.

• native integrations support
• full control over Otel receivers, processors, and exporters
• Elasticsearch output as the backend

This is particularly useful for platform teams testing advanced OTel deployments or building custom telemetry strategies.

But it remains optional—the managed experience is still the default path.

Conclusion: A Modern, Flexible Path Toward OpenTelemetry

Migrating to OpenTelemetry is a journey, not a switch. With hybrid ingestion, Elastic provides a realistic, scalable, and low-risk pathway for organizations that want to adopt OTel gradually while maintaining operational continuity.

Elastic Agent 9.2+ enables teams to:

• retain reliable log integrations
• introduce OTel inputs seamlessly
• manage everything from Fleet
• reduce complexity and operational overhead
• expand into OTel at the right pace
• stay aligned with open standards and best practices

It brings the best of both worlds—Elastic-native richness and OTel-standard flexibility—into a single agent and a unified operational model.

Hybrid isn’t a workaround. It’s the strategic bridge between where your observability platform is today and where it needs to go next.

Technical Walkthrough: Deploying Hybrid Elastic Agent + EDOT in Fleet

Before we close, let’s look at what this actually looks like in practice. Conceptual advantages are important, but many teams want to see how hybrid ingestion works when deployed through Fleet.

The example below walks through a simple, production-ready setup using Elastic Agent 9.2, combining a native integration and an OTel input integration inside a single agent, the same approach you can apply to any service across your environment.

Here is a step-by-step guide showing how to deploy Elastic Agent 9.2 in Fleet-managed hybrid mode, using the OTel nginxreceiver as one concrete example. This applies to any service with an OTel receiver (Redis, HAProxy, Kafka, JVM, etc.).

Requirements

• Elastic Stack 9.2+
• Elastic Agent 9.2+
• Fleet configured in Kibana
• A host running your workload (NGINX in this example)
• NGINX stub_status endpoint or any equivalent OTel metrics endpoint
• API key with ingest privileges

1. Create or Select an Agent Policy

1. In Kibana → Management → Fleet → Agent policies
2. Create a new policy: nginx-o11y
3. Enable system monitoring (recommended)
4. Save

2. Enroll Elastic Agent into the Policy

From the policy page:

1. Click Add agent
2. Choose your OS
3. Copy the installation command
4. Run:

sudo elastic-agent install \
  --url=<FLEET_URL> \
  --enrollment-token=<ENROLLMENT_TOKEN>

You should soon see the agent appear as Healthy in Fleet.

3. Add the Native Integration (Logs)

1. In Fleet, go to Integrations.
2. Search for NGINX.
3. Click Add NGINX.
4. Select your nginx-o11y policy.
5. Only enable log collection (access + error logs).
6. Save and deploy.

4. Validate Log Collection

1. In Kibana, go to Analytics → Discover and search for:

data_stream.dataset : "nginx.access" or "nginx.error"

1. Or open the built-in dashboard:

Analytics → Dashboards → [Logs Nginx] Access and error logs

5. Collecting NGINX Metrics via the OTel NGINX Receiver

Elastic Agent 9.2+ allows Fleet to deploy OTel input integrations.
This scenario uses the OpenTelemetry nginxreceiver through a Fleet-managed integration.

5.1. Install the NGINX OpenTelemetry Integration Content

1. In Kibana, go to Management → Fleet → Integrations.
2. Search for NGINX OpenTelemetry Assets.
3. Click Add Integration.

5.2. Install the NGINX OpenTelemetry Input Integration

1. In Kibana, go to Management → Fleet → Integrations.
2. Search for NGINX OpenTelemetry Input Package.
3. Click Add Integration.
4. Assign it to your agent nginx-o11y policy.

Provide the endpoint for the NGINX status page:

• Endpoint: http://localhost/status
• Collection interval: 10s

Click Add integration.

6. Validate OTel Metrics

1. Go to Analytics → Dashboards.
2. Open: [Metrics Nginx OTEL Overview] Dashboard

You should see metrics such as active connections, writes, reads, waiting, and request counts.

7. Closing thoughts

This example highlights how straightforward hybrid ingestion becomes with Elastic Agent 9.2. By combining native integrations and OTel receivers within a single, centrally managed policy, you gain the flexibility to adopt OpenTelemetry where it adds the most value without disrupting existing pipelines or introducing operational overhead.

Whether you extend this pattern to additional services, experiment with other OTel receivers, or scale it across your fleet, the deployment model remains consistent, repeatable, and production-ready.

For more information and other innovations Elastic Observability has made check out:

• Discover how Elastic is evolving data ingestion with OpenTelemetry

• Learn how OpAMP enables centralized configuration of OpenTelemetry SDKs

• Explore how Streams reshape AI-driven log investigation workflows

The challenge is clear: How do you adopt OTel’s unified observability without abandoning your proven ECS-based logging?

In this guide, we’ll demonstrate how to bridge this gap using the Elastic Distribution of OpenTelemetry (EDOT). We will first show you how to leverage the EDOT Collector to ingest your logs into Elasticsearch, ensuring a seamless transition that unlocks the full power of OTel’s distributed tracing without breaking your current workflows.

Once the data is flowing, we will explore how Elasticsearch's underlying mapping architecture to allow that your existing filters and visualizations remain fully functional through two key features:

• Field Aliases: We’ll explain how Elastic uses aliases to ensure that legacy dashboards looking for log.level (ECS) still work perfectly, even as your new telemetry arrives as severity_text (OTel).

• Passthrough Fields: We’ll show how Elastic’s native OTel mapping structures use passthrough fields to handle OTel attributes. This ensures your data remains searchable and performant without the need for complex, manual schema migrations.

By combining EDOT for ingestion with these intelligent mapping structures, you can maintain your existing Java ECS logging while evolving toward a unified, OTel-native future.

The ECS Foundation

We begin with a Java application using Log4j2 and the ecs-java-plugin. This setup generates structured JSON logs in the Elastic Common Schema (ECS) that Elastic handles natively leveraging the ECS logging plugins that easily integrate with common logging libraries across various programming languages.

The following provides a Log4j2 Configuration Extract and this setup assumes prior configuration of Log4j2 dependencies to include the required ECS plugin libraries:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="DEBUG">
    <Appenders>
        <Console name="LogToConsole" target="SYSTEM_OUT">
            <EcsLayout serviceName="logger-app" serviceVersion="v1.0.0"/>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="LogToConsole"/>
        </Root>
    </Loggers>
</Configuration>

Note: <EcsLayout serviceName="logger-app" serviceVersion="v1.0.0"/> we will come back to this setting later in the blog article, as with Kubernets deployments these values can be automatically populated by the EDOT Collector and the setting could be simplified to <EcsLayout/>

Introducing the Elastic Distribution of OpenTelemetry (EDOT)

The Elastic Distribution of OpenTelemetry (EDOT) is more than just a repackaging; it is a curated set of OTel components (Collector and SDKs) optimized for Elastic Observability. Released in v8.15, it allows you to collect traces, metrics, and logs using standard OTel receivers while benefiting from Elastic-contributed enhancements like powerful log parsing and Kubernetes metadata enrichment.

EDOT's Primary Benefits:

Deliver Enhanced Features Earlier: Provides features not yet available in "vanilla" OTel components, which Elastic continuously contributes upstream.

Enhanced OTel Support: Offers enterprise-grade support and maintenance for fixes outside of standard OTel release cycles.

The question then becomes: How can users transition their ingestion architecture to an OTel-native approach while maintaining the ability to collect logs in ECS format?

This involves replacing classic collection and instrumentation components (like Elastic Agent and the Elastic APM Java Agent). Let us show you how this can be done step by step replacing it with the full suite of components provided by EDOT. A comprehensive view of the EDOT architecture components in Kubernets is shown below.

In a Kubernetes environment, EDOT components are typically installed via an OTel Operator and HELM chart. The main components are:

• EDOT Collector Cluster: deployment used to collect cluster-wide metrics.
• EDOT Collector Daemon: daemonset used to collect node metrics, logs, and application telemetry data.
• EDOT Collector Gateway: performs pre-processing, aggregation, and ingestion of data into Elastic.

Elastic provides a curated configuration file for all the EDOT components available as part of the the OpenTelemetry Operator using the opentelemetry-kube-stack Helm chart. Downloadable from here.

Achieving Correlation: SDK + Logging Context

To link a log line to a specific trace, the EDOT Java SDK performs a "handshake" with your logging library. When a trace is active, the SDK extracts the trace_id and span_id and injects them into the Mapped Diagnostic Context (MDC) of Log4j2. Even though your logs are in ECS format, they now carry the OTel DNA required for correlation. While the EDOT SDK can collect logs directly, a generally more resilient approach is to stick to file collection. This is important because if the OTel Collector is down, logs written to a file are buffered locally on the disk, preventing the data loss that can occur if the SDK's in-memory queue reaches its limit and starts discarding new logs. For an in-depth discussion on this topic we refer to the OpenTelemetry Documentation.

Zero-Code Instrumentation

The EDOT Java SDK is a customized version of the OpenTelemetry Java Agent. In Kubernetes, zero-code Java autoinstrumentation is supported by adding an annotation in the pod template configuration in the deployment manifest:

apiVersion: apps/v1
kind: Deployment
...
spec:
  ..
  template:
    metadata:
      # Auto-Instrumentation
      annotations:
        instrumentation.opentelemetry.io/inject-java: "opentelemetry-operator-system/elastic-instrumentation"

Collecting and Processing Logs with the EDOT Collector

This is the most critical step. Our logs are now JSON, they are in the console output, and they contain trace IDs. Now, we need the EDOT Collector to pick them up and map them to the OpenTelemetry Log Data Model.

EDOT Collector Configuration: Dynamic Workload Discovery and filelog receiver

Applications running on containers become moving targets for monitoring systems. To handle this, we rely on Dynamic workload discovery on Kubernetes. This allows the EDOT Collector to track pod lifecycles and dynamically attach log collection configurations based on specific annotations relying on the k8s_observer and the receiver_creator component.

In our example, we have a Deployment with a Pod consisting of one container. We use Kubernetes annotations to:

1. Enable auto-instrumentation (Java).

2. Enable log collection for this pod.

3. Instruct the collector to parse the output as JSON immediately (json-parser configuration).

4. Add custom attributes (e.g. identify the Application souce code)

Deployment Manifest Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: logger-app-deployment
  labels:
    app: logger-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logger-app
  template:
    metadata:
      annotations:
        # 1. Turn on Auto-Instrumentation
        instrumentation.opentelemetry.io/inject-java: "opentelemetry-operator-system/elastic-instrumentation"
        # 2. Enable Log Collection for this pod
        io.opentelemetry.discovery.logs/enabled: "true"
        # 3. Provide the parsing "hint" (Treat logs as JSON)
        io.opentelemetry.discovery.logs.ecs-log-producer/config: |
            operators:
            - type: container
              id: container-parser
            - type: json_parser
              id: json-parser
         # 4. Identify this application as Java (To allow for user interface rendering in Kibana)
        resource.opentelemetry.io/telemetry.sdk.language: "java"
      ...

This setup provides a bare-minimum configuration for ingesting ECS library logs. Crucially, it decouples log collection from application logic. Developers simply need to provide a hint via annotations that their logs are in JSON format (structurally guaranteed by the ECS libraries). We then define the standardized enrichment and processing rules centrally at the processor level in the (Daemon) EDOT Collector.

This centralization ensures consistency across the platform: if we need to update our standard formatting or enrichment strategies later, we apply the change once in the collector, and it automatically propagates to all services without developers needing to touch their manifests.

(Daemon) EDOT Collector Configuration

To enable this, we configure a Receiver Creator in the Daemon Collector. This component uses the k8s_observer extension to monitor the Kubernetes environment and automatically discover the target pods based on the annotations above.

daemon:
  ...
  config:
    ...
    extensions:
      extensions:
        k8s_observer:
          auth_type: serviceAccount
          node: ${env:K8S_NODE_NAME}
          observe_nodes: true
          observe_pods: true
          observe_services: true
          ...
    receivers:
        receiver_creator/logs:
          watch_observers: [k8s_observer]
          discovery:
            enabled: true
    ...
...

Finally, we reference the receiver_creator in the pipeline instead of a static filelog receiver and we make sure to include the k8s_observer extension:

daemon:
  ...
  config:
    ...
    service:
      extensions:
      - k8s_observer
      pipelines:
        # Pipeline for node-level logs
        logs/node:
          receivers:
            # - filelog             # We disable direct filelog receiver
            - receiver_creator/logs # Using the configured receiver_creator instead of filelog
          processors:
            - batch
            - k8sattributes
            - resourcedetection/system
          exporters:
            - otlp/gateway # Forward to the Gateway Collector for ingestion

The Transformation Layer

While the logs are structured, OTel sees them as generic attributes. We use the OpenTelemetry Transformation Language (OTTL) within a transform processor to "promote" ECS fields to top-level OTel fields. To finalize the pipeline, we use the transform processor, which allows us to modify and restructure telemetry signals using the OpenTelemetry Transformation Language (OTTL).

We use the processor to promote specific ECS fields into the top-level OpenTelemetry fields and renaming attributes according to OpenTelemetry Semantic Conventions:

• Promote the message attribute to the top-level Body field.
• Promote the log.level attribute to the OTel SeverityText field.
• Move the @timestamp attribute to the OTel Time field.
• Map trace_id and span_id to the right log context.

The following provides a sample transform configuration:

 processors:
    transform/ecs_handler:
      log_statements:
      - context: log
        conditions:
          - log.attributes["ecs.version"] != nil
        statements:
          # Map ECS fields to OTel Log Model
          - set(log.body, log.attributes["message"])
          - set(log.time, Time(log.attributes["@timestamp"], "%Y-%m-%dT%H:%M:%SZ"))
          - set(log.trace_id.string, log.attributes["trace_id"])
          - set(log.span_id.string, log.attributes["span_id"])
          - set(log.severity_text, log.attributes["log.level"])
          # Cleanup original keys to save space
          - delete_key(log.attributes, "message")
          - delete_key(log.attributes, "trace_id")
          - delete_key(log.attributes, "span_id")

          # Add here additional transformations as needed...

Note: When working with EDOT Collector and the OpenTelemetry Kube-Stack Helm Chart, resource attributes such as service.name and service.version are automatically populated based on a set of well-defined rules by the k8sattributes processor. Thus, on Kubernetes we do not need to extract those fields from the log content itself.

Make sure to use the newly created processor in the logs pipeline for the Daemon Collector:

service:
  pipelines:
    logs/node:
      receivers:
        - receiver_creator/logs
      processors:
        - batch
        - k8sattributes
        - resourcedetection/system
        - transform/ecs_handler          # Newly created transform processor
      exporters:
        - otlp/gateway

The Compatibility Layer: Bridging ECS and OTel

To bridge the gap between the Elastic Common Schema (ECS) and OpenTelemetry (OTel), Elastic provides a "compatibility layer" built directly into its Observability solution relying on existing index templates and mappings. This architecture allows you to send OTel-native data while still using your legacy ECS-based dashboards, saved searches, and other associated objects.

This "bridge" relies on two key features:

• Bridging ECS and OTel with Passthrough: OpenTelemetry (OTel) data often uses deeply nested structures (e.g., resource.attributes.*). Elasticsearch uses the Passthrough object type to "promote" these nested attributes to the top level when performing a search query. Any new metadata added by the OTel collector is automatically searchable without the user needing to know the full JSON path. This creates a "virtual flattening" layer and makes sure that all fields that match in name are automatically compatible, even though thery're stored in different namespaces (attributes/resource.attributes for OTel vs top-level for ECS). To learn more about fields and attributes alignment between ECS and Otel SemanticConvention refer to this page.

• Bridging with Field Aliases: Elastic relies on OTel mapping templates that include Field Aliases. These aliases link OTel semantic names back to their equivalent ECS fields at query to handle fields that do not align with Otel naming convention.

The Benefit: If you have an existing dashboard looking for message (ECS), but your data is now indexed as body.text (OTEL), an alias allows the dashboard to aggregate and visualize data from both sources simultaneously. This ensures that your existing filters and KQL queries also work flawlessly whether the data originated from a Filebeat agent or a modern OTel SDK Agent.

Some more details about field aliases and pass-through objects can be found here.

Here is an example of the provided mapping template:

{
  "mappings": {
    ...
    "properties": {
      "log": {
          "properties": {
            "level": {
              "type": "alias",
              "path": "severity_text"
            }
          }
        },
      "message": {
        "type": "alias",
        "path": "body.text"
      }
    ...
    }
  }
 }

This architectural approach provides three major advantages for teams in transition:

• Zero Reindexing: You don't have to rewrite or migrate old data. Aliases resolve at query time, meaning your old indices and new indices can coexist in the same visualization.

• Future-Proofing: As OTel becomes the primary standard (following the donation of ECS to the OTel project), Elastic is shifting its native UI to look for OTel fields first. These mappings ensure that your legacy ECS-native data still appears in OTel-native views.

• Unified Observability: It enables "Correlation by Default." Because the aliases link trace_id (OTel) and trace.id (ECS), you can jump from a legacy log to a modern OTel trace without losing context or breaking the drill-down path.

Sending data to Elasticsearch

If you are running Elastic Serverless or the latest Elastic Cloud Hosted (ECH) v9.2+, you now have access to a managed OTLP endpoint. This native functionality allows you to route telemetry directly from your Collector Gateway to Elasticsearch using the OTLP protocol.

Because we mapped our ECS fields to the OTel model in the collector, Elasticsearch recognizes the correlation immediately. You get the best of both worlds: Legacy Compatibility: Your old ECS-based dashboards still work (with minor tweaks). Modern Power: You can now click "View Trace" directly from a log entry in Kibana's Observability UI.

Conclusion

Transitioning to OpenTelemetry doesn't have to be a "big bang" migration. By using the EDOT SDK and Collector, you can: Protect your investment in ECS-based logging libraries. Centralize complexity by handling schema translation in the collector rather than the application. Enable full correlation between traces and logs with zero code changes.

For decades, PHP has powered everything from small business websites to large-scale SaaS platforms. Yet observability in PHP has often required manual setup, compilers, custom extensions, or changes to application code — challenges that limited adoption in production environments. This upcoming donation aims to change that, by making OpenTelemetry for PHP as easy to deploy as any other runtime.

What's coming

Once the contribution process is complete, EDOT PHP will become part of the OpenTelemetry project — providing a complete, production-ready distribution that's optimized for performance, simplicity, and scalability.

EDOT PHP introduces a new approach to PHP observability:

• Simple installation - installing OpenTelemetry for PHP will be as straightforward as installing a standard system package. From that point, the agent automatically detects and instruments PHP applications — no code changes, no manual setup.
• Automatic agent loading - works transparently in cloud and container environments without modifying application deployments.
• Zero configuration - ships as a single, self-contained binary; no need to install or compile any external extensions.
• Native C++ performance - a built-in serializer written in C++ reduces telemetry overhead by up to 5×.
• Automatic instrumentation - instruments popular frameworks and libraries out of the box.
• Inferred spans - reveals the behavior of even uninstrumented code paths, providing full trace coverage.
• Automatic root spans - ensures complete traces, even in legacy or partially instrumented applications.
• OpAMP readiness - while the OpenTelemetry community continues to standardize configuration schemas and management workflows, the implementation in EDOT PHP is fully prepared to support these upcoming specifications — ensuring seamless adoption once the OpAMP ecosystem matures.
• Asynchronous backend communication - telemetry data is exported to the OpenTelemetry Collector or backend asynchronously, without blocking the instrumented application. This ensures that span and metric exports do not add latency to user requests or impact response times, even under heavy load.

Together, these features make EDOT PHP the first truly zero-effort observability solution for PHP — from local testing to cloud-scale production systems.

The native C++ serializer and asynchronous export pipeline in EDOT PHP reduce average request time from 49 ms to 23 ms, more than 2× faster than the pure PHP implementation.

Building on the existing foundation

EDOT PHP doesn't replace the existing OpenTelemetry PHP SDK — it extends and strengthens it. It packages the SDK, automatic instrumentation, and native extension into a single, unified agent package that works seamlessly with existing OpenTelemetry specifications and APIs.

By contributing this work, Elastic helps the OpenTelemetry community accelerate PHP adoption, align implementations across languages, and make distributed tracing truly universal.

“This isn't a hand-off — it's a collaboration. We're contributing years of development to help OpenTelemetry for PHP evolve faster, run more efficiently, and reach more users in every environment.”

• Elastic Observability team

Ongoing improvements

Elastic continues to invest in advancing EDOT PHP ahead of its integration into OpenTelemetry. The team is currently focused on reducing resource usage and memory footprint, particularly in multi-worker server environments such as PHP-FPM or Apache prefork. These optimizations aim to make the agent more predictable and efficient under heavy load — ensuring that telemetry remains lightweight even in large-scale production deployments.

Beyond that, we're exploring further improvements that can enhance both performance and interoperability. Areas under investigation include smarter coordination in high-concurrency scenarios, better sharing of telemetry resources across workers, and future alignment with additional OpenTelemetry signals such as metrics and logs.

Together, these efforts will help make EDOT PHP not only faster, but also more adaptable and seamlessly integrated into diverse runtime architectures.

Why it matters

This contribution is about more than performance — it's about removing barriers. By making OpenTelemetry for PHP installable as a simple system package and automatically loaded into running applications, the project opens observability to every PHP developer, operator, and platform provider.

For the OpenTelemetry ecosystem, it fills one of the last major language gaps, extending visibility to a vast portion of the internet — all under open governance and community collaboration.

Looking ahead

In the months ahead, Elastic and the OpenTelemetry PHP SIG will work closely on the technical integration, documentation, and community onboarding process. Once the transition is complete, developers will gain a fully open, community-driven, and production-ready OpenTelemetry agent that “just works” — without friction, configuration, or code changes.

Together, we're building a future where observability just works — for every language, every framework, and every environment.

For more information:

EDOT documentation
Learn about OTLP Endpoint

Instrumentation Score - The Power of Rule-based Data Quality Analysis

When you first hear the term "Instrumentation Score", your initial reaction might be: "OK, there's a single, percentage-like metric that tells me my instrumentation (i.e. OTel data) has a score of 60 out of 100. So what? How does it help me?"

However, the Instrumentation Score is much more than just a single number. Its power lies in the individual rules from which the score is calculated. The rule definitions' rationale, impact level, and criteria provide an evaluation framework that enables you to drill down into data quality issues and identify specific areas for improvement. Also, the Instrumentation Score specification does not mandate specific tools and implementation details for calculating the score and rule evaluations.

As I explored the Instrumentation Score concepts, I developed the following mental model for deriving actionable insights.

The Score

The score itself is an indicator of the quality of your telemetry data. The lower the number, the more room for improvement with your data quality. In general, if a score falls below 75, you should consider fixing your instrumentation and data collection.

Breakdown by Instrumentation Score Rules

Exploring the evaluation results of individual Instrumentation Score rules will give you insights into what is wrong with your data quality. In addition, the rules' rationales explain why the violation of a rule is problematic.

As an example, let's take the SPA-002 rule:

Description:

Traces do not contain orphan spans.

Rationale:

Orphaned spans indicate potential issues in tracing instrumentation or data integrity. This can lead to incomplete or misleading trace data, hindering effective troubleshooting and performance analysis.

If your data violates the SPA-002 rule, you know what is wrong (i.e. you have broken traces), and the rationale explains why that is an issue (i.e. degraded analysis capabilities).

Breakdown by Services

When you have a large system with hundreds or maybe even thousands of entities (such as services, Kubernetes pods, etc.), a binary signal on all of the data — such as "has a certain rule been passed or not" — is not really actionable. Is the data from all services violating a certain rule, or just a small subset of services?

Breaking down rule evaluation by services (and potentially other entity types) may help you to identify where there are issues with data quality. For example, let's assume only one service — the cart-service — (out of your fifty services) is affected by the violation of rule SPA-002. With that information, you can focus on fixing the instrumentation for the cart-service instead of having to check all fifty services.

Once you know which services (or other entities) violate which Instrumentation Score rules, you're very close to actionable insights. However, there are two more things that I found to be extremely useful for data quality analysis when I was experimenting with the Instrumentation Score evaluation: (1) a quantitative indication of the extent, and (2) concrete examples of rule violation occurrences in your data.

Quantifying the Rule Violation Extent

The Instrumentation Score spec already defines an impact level (e.g. NORMAL, IMPORTANT, CRITICAL) per rule. However, this only covers the "importance" of the rule itself, not the extent of a rule violation. For example, if a single trace (out of a million traces) on your service has an orphan span, technically speaking the rule SPA-002 is violated. But is it really a relevant issue if only one out of a million traces is affected? Probably not. It definitely would be if half of your traces were broken.

Hence, having a quantitative indication of the extent of a rule violation per service — e.g. "40% of your traces violate SPA-002" — would provide additional information on how severe a rule violation actually is.

Tangible Examples

Finally, nothing is as meaningful and self-explanatory as tangible, concrete examples from your own data. If the telemetry data of your cart-service violates SPA-002 (i.e., has traces with orphan spans), wouldn't you want to see a concrete trace from that service that demonstrates the rule violation? Analyzing concrete examples may give you hints about the root cause of broken traces — or, more generally, why your data violates Instrumentation Score rules.

Instrumentation Score with Elastic

The Instrumentation Score spec does not prescribe tool usage or implementation details for the calculation of the score and evaluation of the rules. This allows for integrating the Instrumentation Score concept with whatever backend your OpenTelemetry data is being sent to.

With the goal of building a POC for an end-to-end integration of the Instrumentation Score with Elastic Observability, I combined the powerful capabilities of ES|QL with Kibana's task manager and dashboarding features.

Each Instrumentation Score rule can be formulated as an ES|QL query that covers the steps described above:

• rule passed or not
• breakdown by services
• calculation of the extent
• sampling of an example occurrence

Here is an example query for the LOG-002 rule that checks the validity of the severity_number field:

FROM logs-*.otel-* METADATA _id
| WHERE data_stream.type == "logs"
    AND @timestamp > NOW() - 1h
| EVAL no_sev = severity_number IS NULL OR severity_number == 0
| STATS 
    logs_wo_severity = COUNT(*) WHERE no_sev,
    example = SAMPLE(_id, 1) WHERE no_sev,
    total = COUNT(*)
      BY service.name
| EVAL rule_passed = (logs_wo_severity == 0),
    extent = CASE(total != 0, logs_wo_severity / total, 0.0)
| KEEP rule_passed, service.name, example, extent

These rule evaluation queries are wrapped in a Kibana instrumentation-score plugin that utilizes the task manager for regular execution. The instrumentation-score plugin then takes the results from all the evaluation queries for the different rules and calculates the final instrumentation score value (overall and broken down by service) following the Instrumentation Score spec's calculation formula. The resulting instrumentation score values, as well as the rule evaluation results (with the examples and extent) are then stored in separate Elasticsearch indices for consumption.

With the results stored in dedicated Elasticsearch indices, we can build Dashboards to visualize the Instrumentation Score insights and allow users to troubleshoot their data quality issues.

In this POC I implement subet of instrumentation score rules to prove out the approach.

The Instrumentation Score concept accommodates extension with your own custom rules. I did that in my POC as well to test some quality rules that are not yet formalized as rules in the Instrumentation Score spec, but are important for Elastic Observability to provide the maximum value from the OTel data.

Applying the Instrumentation Score on the OpenTelemetry Demo

The OpenTelemetry Demo is the most-used environment to play around with and showcase OpenTelemetry capabilities. Initially, I thought the demo would be the worst environment to test my Instrumentation Score implementation. After all, it's the showcase environment for OpenTelemetry, and I expected it to have an Instrumentation Score close to 100. Surprisingly, that wasn't the case.

Let's start with the overview.

The Overview

This dashboard shows an overview of the Instrumentation Score results for the OpenTelemetry Demo environment. The first thing you might notice is the very low overall score 35 (top-left corner). The table in the bottom-left corner shows a breakdown of the score by services. Somewhat surprisingly, all the service scores are higher than the overall score. How is that possible?

The main reason is that Instrumentation Score rules have, by definition, a binary result — passed or not. So it can happen that each service fails a single but distinct rule. Hence, the service score is not perfect but also not too bad. But, from the overall perspective, many rules have failed (each by a different service), hence, leading to a very low overall score.

In the table on the right, we see the results for the individual rules with their description, impact level, and example occurrences. We see that 7 out of 11 implemented rules have failed. Let's pick our favorite example from earlier — SPA-002 (in row 5), the orphan spans rule.

With the dashboard indicating that the rule SPA-002 has failed, we know that there are orphan spans somewhere in our OTel traces. But where exactly?

For further analysis, we have two ways to drill down: (1) into a specific rule to see which services violate a specific rule, or (2) into a specific service to see which rules are violated by that service.

Rule Drilldown

The following dashboard shows a detailed view into the rule evaluation results for individual rules. In this case we selected rule SPA-002 at the top.

In addition to the rule's meta information, such as its description, rationale, and criteria, we see some statistics on the right. For example, we see that 2 services have failed that rule, 16 passed, and for 19 services this rule is not applicable (e.g., because those don't have tracing data). In the table below, we see which two services are impacted by this rule violation: the frontend and frontend-proxy services. For each service, we also see the extent. In the case of the frontend service, around 20% of traces have orphan spans. This information is crucial as it gives an indication of how severe the rule violation actually is. If it had been under 1%, this problem might have been negligible, but with one trace out of five being broken, it definitely needs to be fixed. Also, for each of the services, we have an example span.id for which no spans could be found but that are referenced in the parent.id by other spans. This allows us to perform further analyses (e.g., by investigating the referring spans in Kibana's Discover) on concrete example cases.

With that view, we now know that the frontend service has a good amount of broken traces. But is that service also violating other rules? And, if yes, which?

Service Drilldown

To answer the above question we can switch to the Per Service Dashboard.

In this dashboard, we see similar information as on the overview dashboard, however, filtered on a single selected service (e.g., frontend service in this example). In the table, we see that the frontend service violates three rules. We already know about SPA-002 from the previous section. In addition, the violation of the custom rule SPA-C-001 shows that around 99% of transaction span names have high cardinality. In Elastic Observability, transactions refer to service-local root spans (i.e., entry points into services). In the example value, we see directly why the span.names (here referred to as transaction.names) have high cardinality. The span name contains unique identifiers (here the session ID) as part of the URL that the span name is constructed from in the instrumentation. As the EDOT Collector derives metrics for transaction-type spans, we also can observe a violation of the MET-001 which requires bound cardinality on metric dimensions.

As you can see, with the Instrumentation Score concept and a few different breakdown views, we were able to pinpoint data quality issues and identify which services and instrumentations need improvement to fix the issues.

Learnings and Observations

My experimentation with the Instrumentation Score was very insightful and showed me the power of this concept — though it's still in its early phase. It is particularly insightful if the implementation and calculation include breakdowns by meaningful entities, such as services, K8s pods, hosts, etc. With such a breakdown, you can narrow down data quality issues to a manageable scope, instead of having to sift through huge amounts of data and entities.

Furthermore, I realized that having some notion of problem extent (per rule and service), as well as concrete examples, helps make the problem more tangible.

Thinking further about the idea of rule violation extent, there might even be a way to incorporate that into the score formula itself. In my humble opinion, this would make the score significantly more comparable and indicative of the actual impact. I proposed this idea in an issue on the Instrumentation Score project.

Conclusion

The Instrumentation Score is a powerful approach to ensuring a high level of data quality with OpenTelemetry.

Thank you to the maintainers — Antoine Toulme, Daniel Gomez Blanco, Juraci Paixão Kröhling, and Michele Mancioppi — for bringing this great project to life, and to all the contributors for their participation!

With proper implementation of the rules and score calculation, users can easily get actionable insights into what they need to fix in their instrumentation and data collection. The Instrumentation Score rules are in an early stage and are steadily improved and extended. I'm looking forward to what the community will build in the scope of this project in the future, and I hope to intensify my contributions as well.

Streams changes that. It introduces a clear, unified way to manage how long your data lives, whether you’re using DSL or ILM. You can visualize ingestion, understand where data sits across tiers, and adjust retention with confidence, applying updates to a single stream without worrying about unintended changes elsewhere, all from a single view.

Walkthrough: Exploring the Retention Tab

Retention management lives in the Retention tab of each stream. This is your control panel for understanding how much data you’re storing, how quickly it’s growing, and how your lifecycle policies are applied. It’s also where you can monitor and configure the Failure store, which tracks and retains documents that failed to be ingested.

Metrics at a glance

At the top of the view, you’ll find an overview of key metrics:

• Storage size: the total data volume currently held by the stream.
• Ingestion averages: calculated from the selected time range, Streams extrapolates both daily and monthly averages to give you a sense of long-term trends.

This combination of near-real-time and projected values helps you quickly spot when ingestion is ramping up and whether your retention policy aligns with it.

Ingestion over time

Below the metrics, a graph shows ingestion volume over time. This information is approximated based on the number of documents over time, multiplied by the average document size in the backing index. 

Visualizing lifecycle phases

When an ILM policy is effective, the retention view becomes more visual. Streams displays a phase breakdown (hot, warm, cold, frozen) showing the data volume stored in each phase. This gives you a clear sense of how your data is distributed across the storage tiers and whether your lifecycle is doing what you expect.

Failure store

A failure store is a secondary set of indices inside a data stream, dedicated to storing documents that failed to be ingested. Within the Retention tab, you can toggle the Failure store on or off, and configure its own retention period. We’ll cover Failure store and Data quality in more detail in this article.

Updating Retention

Beyond visualizing your retention, Streams makes it easy to change how it’s managed.

Switching between DSL and ILM

You can freely switch a stream between DSL and ILM management, or update a DSL retention with just a few clicks. Streams takes care of updating the lifecycle settings at the data stream level, ensuring consistent retention across all existing backing indices, not just new ones.

Whether you prefer the simplicity of DSL or the fine-grained tiering of ILM, you can move between the two seamlessly.

Clicking “Edit data retention” opens a modal that allows you to update the stream’s configuration. From there you can update the ILM policy or set a custom retention period via DSL.

You can set a custom period, or pick an Indefinite retention for your data.

You can also update streams’ lifecycle via the Upsert stream or the Update ingest stream settings Kibana APIs.

Inherit or defer: different strategies for different stream types

Classic streams

For classic streams, you can default to the existing index template’s retention. Retention isn’t managed by Streams in this case, it follows the lifecycle configuration defined in the template just as it normally would.

This option is useful if you’re onboarding existing data streams and want to keep their lifecycle behavior intact while still benefiting from Streams’ visibility and monitoring features.

Wired streams

Wired streams live in a tree structure, and that hierarchy allows an inheritance model.

A child stream can inherit the lifecycle of its nearest ancestor that has a concrete policy (ILM or DSL). This keeps your configuration lean and consistent since you can set a single lifecycle at a higher level in the tree and let Streams automatically apply it to all relevant descendants.

If that ancestor’s lifecycle is later updated, Streams cascades the change down to all children that inherit it, so everything stays in sync.

In the figure below, we set a different retention for logs.prod and logs.staging environments. The child partitions of these environments automatically inherit the configuration.

How it works under the hood

When you apply or update a lifecycle, Streams calls Elasticsearch’s /_data_stream/_settings. This is a new API we’ve added in 8.19 / 9.1 for this purpose. 

This API is key to keeping retention consistent:

1. It applies the lifecycle directly at the data stream level, overriding any configuration from cluster settings or index templates.
2. It propagates the retention update to all existing backing indices, not just new ones, so retention remains uniform across your historical and future data.

By centralizing lifecycle management at the data stream level and applying a consistent configuration across the backing indices, we remove the ambiguity that used to exist between template-level and index-level configurations. You always know which retention policy is actually in effect, and you can see it directly in the UI.

Wrapping Up

With Streams, retention management becomes clear and consistent. You can visualize ingestion, switch between DSL and ILM, or inherit policies across streams, all without diving into templates or manual index settings.

By unifying retention into a single view, Streams turns lifecycle management into something simple, predictable, and transparent.

Sign up for an Elastic trial at cloud.elastic.co, and trial Elastic's Serverless offering which will allow you to play with all of the Streams functionality.

Additionally, check out:

Read about Reimagining streams

Look at the Streams website

Read the Streams documentation

Observability, the broken promise

To see why the current model fails, let’s look at the all-too-familiar challenge every SRE dreads: knowing a problem exists but needing to spend valuable time just trying to find where to even start the investigation.

Imagine you get a Slack message from the support team: "a few high-value customers are reporting their payments are failing." You have no shortage of alerts, but most are just flagging symptoms. You don’t know where to start. You decide to check the logs to see if there is anything obvious, starting with the systems that have the high CPU alert.

You spend a few minutes searching and grep-ing through terabytes of logs for affected customer IDs, trying to piece together the problem. Nothing. You worry that you aren’t getting all the logs to reveal the problem, so you turn on more logging in the application. Now you’re knee-deep in data, desperately trying to find patterns, errors, or other "hints" that will give you a clue as to the why.

Finally, one of the broader log queries hits on an error code associated with an impacted customer ID. This is the first real clue. You pivot your search to this new error code and after an hour of digging, you finally uncover the error message. You've finally found the why, but it was a stressful, manual hunt that took far too much time and impacted dozens more customers.

This incident perfectly illustrates the broken promise of modern observability: The complete failure of the investigation process. Investigations are a manual, reactive process that SREs are forced into every day. At Elastic, we believe metrics, traces, and logs are all essential, but their roles, and the workflow between them, must be fundamentally re-imagined for effective investigations.

Observability is about having the clearest understanding possible of the what, where, and why. Metrics are essential for understanding the what. They are the heartbeat of your system, powering the dashboards and alerts that tell you when a threshold has been breached, like high CPU utilization or error rates. But they are aggregates; they show the symptom, rarely the root cause. Traces are good at identifying the where. They map the journey of a request through a distributed system, pinpointing the specific microservice or function where latency spikes or an error originates. Yet, their effectiveness hinges on complete and consistent code instrumentation, a constant dependency on development teams that can leave you with critical visibility gaps. Logs tell you the why. They contain all the rich, contextual, and unfiltered truth of an event. If we can more proactively and efficiently extract information from logs, we can greatly improve our overall understanding of our environments.

Challenges of logs in modern environments

While logs are in the standard toolbox, they have been neglected. SREs using today’s solutions deal with several major problems:

• First, due to their unstructured nature, it’s very difficult to parse and manage logs so that they’re useful. As a result, many SRE teams spend a lot of time building and maintaining complex pipelines to help manage this process. 

• Second, logs can get expensive at high volume, which leads teams to drop them on the floor to control costs, throwing away valuable information in the process. Consequently, when an incident occurs, you waste precious time hunting for the right logs, and manually correlating across services.

• Finally, nobody has built a log solution that proactively works to find the important signals in logs and to surface those critical whys to you when you need them. As a result, log-based investigations are too painful and slow.

Why are we here? As applications became more complex, log volume became unmanageable. Instead of solving this with automation, the industry took a shortcut: it gave up on getting the most out of logs and prioritized more manageable but less informative signals.

This decision is the origin of the broken, reactive model. It forced observability into a manual loop of 'observing' alerts, rather than building automation that could help us truly understand our systems to improve how we root cause and resolve issues. This has transformed SREs from investigators into full-time data wranglers, wrestling with Grok patterns and fragile ETL scripts instead of solving outages. 

Introducing Streams to rethink how you use logs for investigations

Streams is an agentic AI solution that simplifies working with logs to help SRE teams rapidly understand the why behind an issue for faster resolution. The combination of Elasticsearch and AI is turning manual management of noisy logs into automated workflows that identify patterns, context, and meaning, marking a fundamental shift in observability.

Log everything in any format

By applying the Elasticsearch platform for context engineering to bring together retrieval and AI-driven parsing to keep up with schema changes, we are reimagining the entire log pipeline.  

Streams ingests raw logs from all your sources to a single destination. It then uses AI to partition incoming logs into their logical components and parses them to extract relevant fields for an SRE to validate, approve, or modify. Imagine a world where you simply point your logs to a single endpoint, and everything just works. Less wrestling with Grok patterns, configuring processors, and hunting for the right plugin. All of which significantly reduces the complexity. Streams is a big step towards realizing that vision.

As a result, SREs are freed from managing complex ingestion pipelines, allowing them to spend less time on data wrangling and more time preventing service disruptions.

Solve incidents faster with Significant Events 

Significant Events, a capability within Streams, uses AI to automatically surface major errors and anomalies, enabling you to be proactive in your investigations. So, instead of just combing through endless noise, you can focus on the events that truly matter, such as startup and shutdown messages, out-of-memory errors, internal server failures, and other significant signals of change. These events act as actionable markers, giving SREs early warning and clear focus to begin an investigation before service impact.

With this new foundation, logs will become your primary signal for investigation. The panicked, manual search for a needle in a digital haystack is about to be over. Significant Events acts like a smart metal detector that sifts through the chaos and only beeps when it finds issues, helping you to easily ignore all that hay and find the "needle" faster. 

Now imagine the same scenario we started with. Instead of starting a frantic, time-consuming grep through terabytes of logs. Streams has already done the heavy lifting. Its AI-driven analysis has detected a new, anomalous pattern that began before your support team even knew about it and automatically surfaced it as a significant event. Rather than you hunting for a clue, the clue finds you. 

With a single click, you have the why: a Java out-of-memory error in a specific service component. This is your starting point. You find the root cause in under two minutes and begin remediation. The customer impact is stopped, the dev team gets the specific error, and the problem is contained before it can escalate. In this case, metrics and traces were unhelpful in finding the why. The answer was waiting in the logs all along.

This ideal outcome is possible because you can both afford to keep every log and instantly find the signal within them. Elastic's cost-efficient architecture with powerful compression, searchable snapshots, and data tiering makes full retention a reality. From there, Streams automatically surfaces the significant event, ensuring that the answer is never lost in the noise.

Elastic is the only company that provides an AI-driven log-first approach to elevate your observability signals and make it dramatically faster and easier to get to why. This is built on our decades of leadership in search, relevance, and powerful analytics that provides the foundation for understanding logs at a deep, semantic level.

The vision for Streams 

The partitioning, parsing, and Significant Events you see today is just the starting point. The next step in our vision is to use the Significant Events to automatically generate critical SRE artifacts. Imagine Streams creating intelligent alerts, on-the-fly investigation dashboards, and even data-driven SLOs based only on the events that actually impact service health. From there, the goal is to use AI to drive automated Root Cause Analysis (RCA) directly from log patterns and generate remediation runbooks, turning a multi-hour hunt into an instant resolution recommendation.

Once this AI-drive log foundation is in place, our vision for Streams expands to become a unified intelligence layer that operates across all your telemetry data. It’s not just about making each signal better in isolation, but about understanding the context and relationships between them to solve complex problems. 

For metrics, Streams won’t just alert you to a single metric spike but detect a correlated anomaly across multiple, seemingly unrelated metrics e.g. p99 latency for a specific service, rise in garbage collection time, transaction success rate.

Similarly, for traces it identifies a new, unexpected service call (e.g., a new database or an external API) appears in a critical transaction path after a deployment or identifies specific span is suddenly responsible for a majority of errors across all traces, even if the overall error rate hasn't breached a threshold.

The goal is not to have separate streams for logs, metrics, and traces, but to weave them into a single narrative that automatically correlates all three signals. Ultimately, Streams is about fundamentally changing the goal from human led data gathering exercise to proactive, AI-driven resolution.

For more on Streams:

Read the Streams launch blog

Look at the Streams website

As a core pillar of Observability, metrics offer a highly structured, quantitative view of system performance and health. They provide a crucial symptomatic perspective—revealing what is happening, such as high application latency, increasing service errors, or spiking container CPU utilization, which is essential for initiating alerting and triaging efforts. This capability for effective monitoring, alerting, and triaging is paramount to ensuring robust service delivery and achieving successful business outcomes.

Elastic Observability provides a comprehensive, end-to-end experience for metrics data. Elastic ensures that metrics data can be collected from numerous sources, enriched as needed and shipped to the Elastic Stack. Elastic efficiently stores this time series data, including high-cardinality metrics, utilizing the TSDS index mode (Time Series Data Stream), introduced in prior versions and used across Elastic time series integrations. This foundation ensures comprehensive observability through out-of-the-box dashboards, alerts, SLOs, and streamlined data management.

Elastic Observability 9.2 provides enhancements to metrics exploration and analysis through powerful query language extensions and expanded UI capabilities. These enhancements focus on making analysis on TSDS data via counter rates and common aggregations over time easier and faster than ever before.

The main metrics enhancements center on these key features, offered as Tech Preview:

1. Metrics analytics with TSDS and ES|QL
2. Interactive metrics exploration in Discover
3. OTLP endpoint for metrics

Metrics analytics with TSDS and ES|QL

The introduction of the new TS source command in ES|QL (Elasticsearch Query Language) on TSDS metrics dramatically simplifies time series analysis.

The TS command is specifically designed to target only time series indices, differentiating it from the general FROM command. Its core power lies in enabling a dedicated suite of time series aggregation functions within the STATS command.

This mechanism utilizes a dual aggregation paradigm, which is standard for time series querying. These queries involve two aggregation functions:

• Inner (Time Series) function: Applied implicitly per time series, often over bucketed time intervals.

• Outer (Regular) function: Used to aggregate the results of the inner function across groups. For instance, if you use STATS SUM(RATE(search_requests)) BY TBUCKET(1 hour), host, the RATE() function is the inner function applied per time series in hourly buckets, and SUM() is the outer function, summing these rates for each host and hourly bucket.

If an ES|QL query using the TS command is missing an inner (time series) aggregation function, LAST_OVER_TIME() is implicitly assumed and used. For example, TS metrics | STATS AVG(memory_usage) is equivalent to TS metrics | STATS AVG(LAST_OVER_TIME(memory_usage)).

Key time series aggregation functions available in ES|QL via TS command

These functions allow for powerful analysis on time-series data:

| | | | | :----------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------: | | Function | Description | Example Use Case | | RATE() / IRATE() | Calculates the per-second average rate of increase of a counter (RATE), accounting for non-monotonic breaks like counter resets, making it the most appropriate function for counters, or the per-second rate of increase between the last two data points (IRATE), ignoring all but the last two points for high responsiveness. | Calculating request per second (RPS) or throughput. | | AVG_OVER_TIME() | Calculates the average of a numeric field over the defined time range. | Determining average resource usage over an hour. | | SUM_OVER_TIME() | Calculates the sum of a field over the time range. | Total errors over a specific time window. | | MAX_OVER_TIME() / MIN_OVER_TIME() | Calculates the maximum or minimum value of a field over time. | Identifying peak resource consumption. | | DELTA() / IDELTA() | Calculates the absolute change of a gauge field over a time window (DELTA) or specifically between the last two data points (IDELTA), making IDELTA more responsive to recent changes. | Tracking changes in system gauge metrics (e.g., buffer size). | | INCREASE() | Calculates the absolute increase of a counter (INCREASE). | Analyzing immediate rate changes in fast-moving counters. | | FIRST_OVER_TIME() / LAST_OVER_TIME() | Calculates the earliest or latest recorded value of a field, determined by the @timestamp field. | Inspecting initial and final metric states within a bucket. | | ABSENT_OVER_TIME() / PRESENT_OVER_TIME() | Calculates the absence or presence of a field in the result over the time range. | Identifying monitoring coverage gaps. | | COUNT_OVER_TIME() / COUNT_DISTINCT_OVER_TIME() | Calculates the total count or the count of distinct values of a field over time. | Measuring frequency or cardinality changes. |

These functions, available with the TS command, allow SREs and Ops teams to easily perform rate calculations and other common aggregations, enabling efficient metrics analysis as a routine part of observability workflows. And it’s much faster, too! Internal performance testing has revealed that TS commands outperform other ways of querying metrics data by an order of magnitude or more, and consistently! 

Interactive metrics exploration in Discover

The 9.2 release introduces the capability to explore and analyze metrics directly and interactively within the Discover interface. In addition to exploring and analyzing logs and raw events, Discover now provides a dedicated environment for metrics exploration:

• Easy start: Begin exploration simply by querying metrics ingested via TS metrics-*.

• Grid view and pre-applied aggregations: This command displays all metrics in a grid format at a glance, immediately applying the appropriate aggregations based on the metric type, such as rate versus avg.

• Search and group-by: Quickly search for specific metrics by name. Also easily group and analyze metrics by dimensions (labels) and specific values. This allows narrowing down to metrics and dimensions of choice for targeted analysis.

• Quick access to details: Furthermore, the interface provides access to crucial details, including query and response details, the underlying ES|QL commands, the metric field type, and applicable dimensions, for each metric.

• Easy tweaking and dashboarding: The system automatically populates ES|QL queries, aiding in making easy tweaks, slicing, and dicing the data. Once analyzed, metrics and resulting analyses can be added to new or existing dashboards with ease.

OTLP endpoint for metrics

We are also introducing a native OpenTelemetry Protocol (OTLP) endpoint specifically for metrics ingest directly into Elasticsearch. The endpoint especially benefits self-managed customers, and will be integrated into our Elastic Cloud Managed OTLP Endpoint for Elastic-managed offerings. The native endpoint and related updates improve ingest performance and scalability of OTel metrics, providing up to 60% higher throughput via _otlp, and up to 25% higher throughput when using classic _bulk methods. 

In Conclusion

By merging the power of ES|QL's new time series aggregations with the familiar interactive experience of Discover, Elastic 9.2 enables a potent set of metrics analytics tools. The tools significantly boost the exploration and analysis phase of any observability workflow. And we’re just getting started on unleashing the full power of metrics in Elastic Observability!

We welcome you to try the new features today!

Also learn more about how we provide metrics analytics for AWS, Azure, GCP, Kubernetes, and LLMs on Observability Labs

Ingestion

Let's start at the beginning, ingest. Typically many of you are using older tools for ingestion today. If you want to be more forward thinking here, it’s time to introduce you to OpenTelemetry. OpenTelemetry was once not very mature or capable for logging but things have changed significantly. Elastic has been working particularly hard to improve the log capabilities resident in OpenTelemetry. So let's start by exploring how we can get started bringing logs into Elastic via the OpenTelemetry collector.

Firstly if you want to follow along simply create a host to run the log generator and OpenTelemetry collector.

Follow the instructions here to get the log generator running:

https://github.com/davidgeorgehope/log-generator-bin/

To get the OpenTelemetry collector up and running in Elastic Serverless, you can click on Add Data from the bottom left, then 'host' and finally 'opentelemetry'

Follow the instructions but don’t start the collector just yet.

Our host here is running a 3 tier application with an Nginx frontend, backend and connected to a MySQL database. So let's start by bringing the logs into Elastic.

First we’ll install the Elastic Distributions for OpenTelemetry but before starting it, we will make a small change to the OpenTelemetry configuration file to expand the directories it will search for logs in.  Edit the otel.yml by simply using vi or your favorite editor:

vi otel.yml

Instead of simply /var/log/.log we will add /var/log/*/.log to bring in all our log files.

receivers:
  # Receiver for platform specific log files
  filelog/platformlogs:
    include: [ /var/log/**/*.log ]
    retry_on_failure:
      enabled: true
    start_at: end
    storage: file_storage

Start the otel collector

sudo ./otelcol --config otel.yml

And we can see these are being brought in, in discover

Now one thing that is immediately noticeable is that we automatically without changing anything get a bunch of useful additional information such as the os name and cpu information.

The OpenTelemetry collector has automatically, without any changes, started to enrich our logs, making it useful for additional processing, though we could do significantly better!

To start with we want to give our logs some structure. Lets edit that otel.yml file and add some OTTL to extract some key data from our NGINX logs.

  transform/parse_nginx:
    trace_statements: []
    metric_statements: []
    log_statements:
      - context: log
        conditions:
          - 'attributes["log.file.name"] != nil and IsMatch(attributes["log.file.name"], "access.log")'
        statements:
          - merge_maps(attributes, ExtractPatterns(body, "^(?P<client_ip>\\S+)"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "^\\S+ - (?P<user>\\S+)"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\\[(?P<timestamp_raw>[^\\]]+)\\]"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\"(?P<method>\\S+) "), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\"\\S+ (?P<path>\\S+)\\?"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "req_id=(?P<req_id>[^ ]+)"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\" (?P<status>\\d+) "), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\" \\d+ (?P<size>\\d+)"), "upsert")
.....

   logs/platformlogs:
      receivers: [filelog/platformlogs]
      processors: [transform/parse_nginx,resourcedetection]
      exporters: [elasticsearch/otel]

Now when we start the Otel collector with this new configuration

sudo ./otelcol --config otel.yml

We will see that we now have structured logs!!

Store and Optimize

To ensure you aren’t blowing your budget out with all this additional structured data there are few things you can do to help maximize storage efficiency.

You can use the filter processors in the Otel collector with granular filtering/dropping of irrelevant attributes to control volume going out of the collector for example.

processors:
  filter/drop_logs_without_user_attributes:
    logs:
      log_record:
        - 'attributes["user"] == nil'
  filter/drop_200_logs:
    logs:
      log_record:
        - 'attributes["status"] == "200"'

service:
  pipelines:
    logs/platformlogs:
      receivers: [filelog/platformlogs]
      processors: [transform/parse_nginx, filter/drop_logs_without_user_attributes, filter/drop_200_logs, resourcedetection]
      exporters: [elasticsearch/otel]

The filter processor will help reduce the noise for example if you wanted to drop the debug logs or logs from a noisy service. Great ways to keep a lid on your observability spend.

Additionally for your most critical flows and logs where you don’t want to drop any data, Elastic has you covered. In version 9.x of Elastic you now have LogsDB switched on by default.

With LogsDB, Elastic has reduced the storage footprint of log data in Elasticsearch by up to 65% allowing you to store more observability and security data without exceeding your budget, while keeping all data accessible and searchable.

LogsDB reduces log storage by up to 65%. This dramatically minimizes storage footprints by leveraging advanced compression techniques like ZSTD, delta encoding, and run-length encoding, and it also reconstructs the _source field on demand, saving about 40% more storage by not retaining the original JSON document. Synthetic _source represents the introduction of columnar storage within Elasticsearch.

Analytics

So we have our data in Elastic, it’s structured, it conforms to the idea of a wide-event log since it has lots of good context, user ids, request ids and the data is captured at the start of a request Next we’re going to look at the analytics part of this. First let's take a stab at looking at the number of Errors for each user transaction in our application.

FROM logs-generic.otel-default
| WHERE log.file.name == "access.log"
| WHERE attributes.status >= "400"
| STATS error_count = COUNT(*) BY attributes.user
| SORT error_count DESC

It’s pretty easy now to save this and put it on a dashboard, we just click the save button:

Next let's look at putting something together to show the global impact, first we will update our collector config to enrich our log data with geo location.

Update the OTTL configuration with this new line:

   log_statements:
      - context: log
        conditions:
          - 'attributes["log.file.name"] != nil and IsMatch(attributes["log.file.name"], "access.log")'
        statements:
          - merge_maps(attributes, ExtractPatterns(body, "^(?P<client_ip>\\S+)"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "^\\S+ - (?P<user>\\S+)"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\\[(?P<timestamp_raw>[^\\]]+)\\]"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\"(?P<method>\\S+) "), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\"\\S+ (?P<path>\\S+)\\?"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "req_id=(?P<req_id>[^ ]+)"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\" (?P<status>\\d+) "), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\" \\d+ (?P<size>\\d+)"), "upsert")
          - set(attributes["source.address"], attributes["client_ip"]) where attributes["client_ip"] != nil

Next add a new processor (you will need to download the GeoIP database from MaxMind)

geoip:
  context: record
  source:
    from: attributes
  providers:
    maxmind:
      database_path: /opt/geoip/GeoLite2-City.mmdb

And add this to the log pipeline after the parse_nginx

service:
  pipelines:
    logs/platformlogs:
      receivers: [filelog/platformlogs]
      processors: [transform/parse_nginx, geoip, resourcedetection]
      exporters: [elasticsearch/otel]

Start the otel collector

sudo ./otelcol --config otel.yml

Once the data starts flowing we can add a map visualization:

Add a layer:

Use ES|QL

Use the following ES|QL

And this should give you a map showing the locations of all your NGINX server requests!

As you can see, analytics is a breeze with your new Otel data collection pipeline.

Conclusion: Beyond log aggregation to operational intelligence

The journey from basic log aggregation to structured, enriched observability represents more than a technical upgrade, it's a shift in how organizations approach system understanding and incident response. By adopting OpenTelemetry for ingestion, implementing intelligent filtering to manage costs, and leveraging LogsDB's storage optimizations, you're not just modernizing your ELK stack; you're building the foundation for proactive system management.

The structured logs, geographic enrichment, and analytical capabilities demonstrated here transform raw log data into actionable intelligence with ES|QL. Instead of reactive grepping through logs during incidents, you now have the infrastructure to identify patterns, track user journeys, and correlate issues across your entire stack before they become critical problems.

But here's the key question: Are you prepared to act on these insights? Having rich, structured data is only valuable if your organization can shift from a reactive "find and fix" mentality to a proactive "predict and prevent" approach. The real evolution isn't in your logging stack, it's in your operational culture.

Get started with this today in Elastic Serverless

In this blog, we will discuss the current state of client instrumentation for the browser, along with an example showing how to instrument a simple JavaScript frontend using the OpenTelemetry browser instrumentation. Furthermore, we'll also share how the baggage propagators help us build a full picture of what is going on across the entire application by connecting backend traces with frontend signals. If you want to dive straight into the code, check out the repo here.

Application Overview

The application that we use for this blog is called OTel Record Store, a simple web application written with Svelte and JavaScript (albeit our implementation is compatible with other web frameworks), communicating with a Java backend. Both send telemetry signals to an Elastic backend.

Eagle-eyed readers will noticed that signals from our frontend pass through a proxy and collector. The proxy is required to ensure that the appropriate Cross-Origin headers are populated to allow the signals to pass into Elastic, as well as the traditional reasons such as security, privacy and access control:

events {}

http {

  server {

    listen 8123; 

    # Traces endpoint exposed as example, others available in code repo
    location /v1/traces {
      proxy_pass http://host.docker.internal:4318;
      # Apply CORS headers to ALL responses, including POST
      add_header 'Access-Control-Allow-Origin' 'http://localhost:4173' always;
      add_header 'Access-Control-Allow-Methods' 'POST, OPTIONS' always;
      add_header 'Access-Control-Allow-Headers' 'Content-Type' always;
      add_header 'Access-Control-Allow-Credentials' 'true' always;

      # Preflight requests receive a 204 No Content response
      if ($request_method = OPTIONS) {
        return 204;
      }
    }
  }
}

While collectors can also be used to add headers, we have left this example to perform traditional tasks such as routing and processing.

Prerequisites

This example requires an Elastic cluster, run either locally via start-local, via Elastic Cloud or Serverless. Here we use the Managed OLTP endpoint in Elastic Serverless. Any mechanism requires you to specify several key environment variables, listed in the .env-example file:

ELASTIC_ENDPOINT=https://my-elastic-endpoint:443
ELASTIC_API_KEY=my-api-key

Running the application

To run our example, follow the steps in the project README, summarized below:

# Terminal 1: backend service, proxy and collector
docker-compose build
docker-compose up

# Terminal 2: frontend and sample telemetry data
cd records-ui
npm install
npm run generate

Java Backend Instrumentation

We will not cover the specifics of instrumentation of Java services with EDOT as there is already a great guide to get started in the elastic-otel-java README. The example is here purely for showcasing propagation that is important for investigating UI issues. All you need to know is that we make use of automatic instrumentation, sending logs, metrics and traces via OpenTelemetry Protocol, or OTLP using the below environment variables:

OTEL_RESOURCE_ATTRIBUTES=service.version=1,deployment.environment=dev
OTEL_SERVICE_NAME=record-store-server-java
OTEL_EXPORTER_OTLP_ENDPOINT=$ELASTIC_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey ${ELASTIC_API_KEY}"
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp

The instrumentation is then initialized using the -javaagent option:

ENV JAVA_TOOL_OPTIONS="-javaagent:./elastic-otel-javaagent-1.2.1.jar"

Client Instrumentation

Now that we have established our prerequisites, let's dive into the instrumentation code for our simple web application. Although we'll cover the implementation in sections, the full solution is available here in frontend.tracer.ts.

State of OTel Client Instrumentation

At time of writing, the OpenTelemetry JavaScript SDK has stable support for metrics and traces, with logs currently under development and therefore subject to breaking changes as listed in their documentation:

| Traces | Metrics | Logs | | ------ | ------- | ----------- | | Stable | Stable | Development |

What differs from many other SDKs is the note warning that client instrumentation for the browser is experimental and mostly unspecified. It is subject to breaking change, and many pieces such as plugin support for measuring Google Core Web Vitals are in progress as reflected in the Client Instrumentation SIG project board. In subsequent sections we'll show examples for signal capture, and also browser specific instrumentations including document load, user interaction and Core Web Vitals capture.

Resource Definition

When instrumenting web UIs, we need to establish our UI as an OpenTelemetry Resource. By definition, resources are entites that produce telemetry information. We want to see our UI as an entity in our system that interacts with other entities, which can be specified using the following code:

// Defines a Resource to include metadata like service.name, required by Elastic
import { resourceFromAttributes, detectResources } from '@opentelemetry/resources';

// Experimental detector for browser environment
import { browserDetector } from '@opentelemetry/opentelemetry-browser-detector';

// Provides standard semantic keys for attributes, like service.name
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';

const detectedResources = detectResources({ detectors: [browserDetector] });
let resource = resourceFromAttributes({
    [ATTR_SERVICE_NAME]: 'records-ui-web',
    'service.version': 1,
    'deployment.environment': 'dev'
});
resource = resource.merge(detectedResources);

A unique identifier for the service is required, and is common to all SDKs. What differs from other implementations is the inclusion of the browserDetector which, when merged with our defined resource attributes adds browser attributes such as platform, brands (e.g. Chrome versus Edge) and whether a mobile browser is being used:

Having this information on spans and errors is useful in diagnostic situations in identifying application and dependency compatibility issues with certain browsers (such as Internet Explorer from my time as an engineer 🤦).

Logs

Traditionally, frontend engineers rely on the DevTools console of their favourite browser to examine logs. With UI log messages only being accessible within your browser rather than forwarded to a file somewhere, which is the common pattern with backend services, we lose visibility of this resource when triaging user issues.

OpenTelemetry defines the concept of an exporter that allow us to send signals to a particular destination, such as logs.

// Get logger and severity constant imports
import { logs, SeverityNumber } from '@opentelemetry/api-logs';

// Provider and batch processor for sending logs
import { BatchLogRecordProcessor, LoggerProvider } from '@opentelemetry/sdk-logs';

// Export logs via OTLP
import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-http';

// Configure logging to send to the collector via nginx
const logExporter = new OTLPLogExporter({
    url: 'http://localhost:8123/v1/logs' // nginx proxy
});

const loggerProvider = new LoggerProvider({
    resource: resource, // see resource initialisation above
    processors: [new BatchLogRecordProcessor(logExporter)]
});

logs.setGlobalLoggerProvider(loggerProvider);

Once the provider has been initialized, we need to get a hold of the logger to send our traces to Elastic rather than using good ol' console.log('Help!'):

// Example gets logger and sends a message to Elastic
const logger = logs.getLogger('default', '1.0.0');
logger.emit({
    severityNumber: SeverityNumber.INFO,
    severityText: 'INFO',
    body: 'Logger initialized'
});

They will now be visible in Discover and the Logs views, allowing us to search for relevant outages as part of investigations and incidents:

Traces

The power of traces in diagnosing issues in the UI is in the visibility of not just what is going on within the web application, but seeing the connections and time taken to make calls to the labyrinth of services behind. To instrument a web-based application, we need to make use of the WebTraceProvider using the OTLPTraceExporter in a similar way to how exporters work for logs and metrics:

/* Packages for exporting traces */

// Import the WebTracerProvider, which is the core provider for browser-based tracing
import { WebTracerProvider } from '@opentelemetry/sdk-trace-web';

// BatchSpanProcessor forwards spans to the exporter in batches to prevent flooding
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';

// Import the OTLP HTTP exporter for sending traces to the collector over HTTP
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

// Configure the OTLP exporter to talk to the collector via nginx
const exporter = new OTLPTraceExporter({
    url: 'http://localhost:8123/v1/traces' // nginx proxy
});

// Instantiate the trace provider and inject the resource
const provider = new WebTracerProvider({
    resource: resource,
    spanProcessors: [
        // Send each completed span through the OTLP exporter
        new BatchSpanProcessor(exporter)
    ]
});

Next we need to register our provider. One thing that's slightly different in the web world is how we configure propagation. Context propagation in OpenTelemetry refers to the concept of moving context between services and processes which, in our case, allows us to correlate the web signals with those of backend services. Often this is done automatically. As you will see from the below snippet, there are 3 concepts that help us with propagation:

// This context manager ensures span context is maintained across async boundaries in the browser
import { ZoneContextManager } from '@opentelemetry/context-zone';

// Context Propagation across signals
import {
    CompositePropagator,
    W3CBaggagePropagator,
    W3CTraceContextPropagator
} from '@opentelemetry/core';

// Provider instantiation code omitted

// Register the provider with propagation and set up the async context manager for spans
provider.register({
    contextManager: new ZoneContextManager(),
    propagator: new CompositePropagator({
        propagators: [new W3CBaggagePropagator(), new W3CTraceContextPropagator()]
    })
});

The first is the ZoneContextManager which propagates context such as spans and traces across asynchronous operations. Web developers will be familiar with zone.js, the framework used by many JS frameworks to provide an execution context that persists across async tasks.

Additionally, we have combined the W3CBaggagePropagator and W3CTraceContextPropagator using the CompositePropagator to ensure key value pair attributes are passed between signals as per the W3C specification defined here. In the case of the W3CTraceContextPropagator, it allows the propagation of the traceparent and tracestate HTTP headers as per the specification located here.

Auto Instrumentation

The simplest way to start instrumenting a web application is to register the web auto-instrumentations. At time of writing the documentation states that the following instrumentations can be configured via this approach:

1. @opentelemetry/instrumentation-document-load
2. @opentelemetry/instrumentation-fetch
3. @opentelemetry/instrumentation-user-interaction
4. @opentelemetry/instrumentation-xml-http-request

Configuration for each configuration can be passed as configuration to registerInstrumentations as shown in the below example configuring the fetch and XMLHTTPRequest instrumentations:

// Used to auto-register built-in instrumentations
import { registerInstrumentations } from '@opentelemetry/instrumentation';

// Import the auto-instrumentations for web, which includes common libraries, frameworks and document load
import { getWebAutoInstrumentations } from '@opentelemetry/auto-instrumentations-web';

// Enable automatic span generation for document load and user click interactions
registerInstrumentations({
  instrumentations: [
    getWebAutoInstrumentations({
      '@opentelemetry/instrumentation-fetch': {
        propagateTraceHeaderCorsUrls: /.*/,
        clearTimingResources: true
        },
        '@opentelemetry/instrumentation-xml-http-request': {
          propagateTraceHeaderCorsUrls: /.*/
          }
      })
    ]
});

Taking the @opentelemetry/instrumentation-fetch instrumentation as an example, we are able to see traces for HTTP requests, and the propagators also ensure that the spans can connect with our Java backend services to give a full picture of the amount of time taken to process the request at each stage:

While auto-instrumentations is agreat way to get common instrumentations, we can also instantiate instrumentations directly, as we'll see in the remainder of this article.

Document Load Instrumentation

Another consideration unique to web frontend is the time taken to load assets such as images, JavaScript files and even stylesheets. Such assets taking considerable time to load can impact metrics such as First Contentful Paint, and therefore the user experience. The OTel Document Load instrumentation allows for automatic instrumentation of the time taken to load assets when using the @opentelemetry/sdk-trace-web package.

It is simply a case of adding the instrumentation to the instrumentations array we have provided to our provider using registerInstrumentations:

// Used to auto-register built-in instrumentations like page load and user interaction
import { registerInstrumentations } from '@opentelemetry/instrumentation';

// Document Load Instrumentation automatically creates spans for document load events
import { DocumentLoadInstrumentation } from '@opentelemetry/instrumentation-document-load';

// Configuration discussed above omitted

// Enable automatic span generation for document load and user click interactions
registerInstrumentations({
  instrumentations: [
    // Automatically tracks when the document loads
    new DocumentLoadInstrumentation({
      ignoreNetworkEvents: false,
      ignorePerformancePaintEvents: false
      }),
      // Other instrumentations omitted
  ]
});

This configuration will create a new trace conventiently named documentLoad, that will show us the time taken to load resources within the document, similar to the following:

Each span will have metadata attached to help us identify which resources are taking considerable time to load, such as this image example, where the resource takes 837ms to load:

Click Events

You may wonder why we want to capture user interactions with web applications for diagnostic purposes. Being able to see the trigger points for errors can be useful in incidents to establish a timeline of what happened, and determine if users are indeed being impact as is the case for Real Ueer Monitoring tools. But if we also consider the field of Digital Experience Monitoring, or DEM, software teams need details on usage of application features to understand the user journey and how it could possibly being improved in a data-drive way. Capturing user events is required for both.

The OTel UserInteraction instrumentation for web is how we capture these events. Similar to the document load instrumentation it depends on the @opentelemetry/sdk-trace-web package, and when used with zone-js and the ZoneContextManager it also supports async operations.

Like other instrumentations it is added via registerInstrumentations:

// Used to auto-register built-in instrumentations like page load and user interaction
import { registerInstrumentations } from '@opentelemetry/instrumentation';

// Automatically creates spans for user interactions like clicks
import { UserInteractionInstrumentation } from '@opentelemetry/instrumentation-user-interaction';

// Configuration discussed above omitted

// Enable automatic span generation for document load and user click interactions
registerInstrumentations({
  instrumentations: [
    // User events
    new UserInteractionInstrumentation({
      eventNames: ['click', 'input'] // instrument click and input events only
    }),
    // Other instrumentations omitted
  ]
});

It will capture and label spans for the user events we configure, and leveraging the propagators configured previously can connect spans from other resources to the user event, similar to the below example where we see the service call to get records when the user adds a search term to the input box:

Metrics

There are numerous different measurements that are helpful in capturing useful indicators of availability and performace of web applications, such as latency, throughput or the number of 404 errors. Google Core Web Vitals are a set of standard metrics used by web developers to measure real-world user experience of web sites, including loading performance, reactivity to user input and visual stability. Given at time of writing the Core Web Vitals Plugin for OTel Browser is on the backlog, let's try building our own custom instrumentation using the web-vitals JS library to capture these as OTel metrics.

In OpenTelemetry you can create your own custom instrumentation by extending the InstrumentationBase, overriding the constructor to create the MeterProvider, Meter and OTLPMetricExporter that will allow us to send our Core Web Vital measurements to Elastic via our proxy, as presented in web-vitals.instrumentation.ts. Note that below we show only the LCP meter for succinctness, but the full example here measures all web vitals.

/* OpenTelemetry JS packages */
// Instrumentation base to create a custom Instrumentation for our provider
import {
    InstrumentationBase,
    type InstrumentationConfig,
    type InstrumentationModuleDefinition
} from '@opentelemetry/instrumentation';

// Metrics API
import {
    metrics,
    type ObservableGauge,
    type Meter,
    type Attributes,
    type ObservableResult,

} from '@opentelemetry/api';

export class WebVitalsInstrumentation extends InstrumentationBase {

  // Meter captures measurements at runtime
    private cwvMeter: Meter;

    /* Core Web Vitals Measures, LCP provided, others omitted */
    private lcp: ObservableGauge;

    constructor(config: InstrumentationConfig, resource: Resource) {
        super('WebVitalsInstrumentation', '1.0', config);

    // Create metric reader to process metrics and export using OTLP
        const metricReader = new PeriodicExportingMetricReader({
            exporter: new OTLPMetricExporter({
                url: 'http://localhost:8123/v1/metrics' // nginx proxy
            }),
            // Default is 60000ms (60 seconds).
            // Set to 10 seconds for demo purposes only.
            exportIntervalMillis: 10000
        });

    // Creating Meter Provider factory to send metrics
        const myServiceMeterProvider = new MeterProvider({
            resource: resource,
            readers: [metricReader]
        });
        metrics.setGlobalMeterProvider(myServiceMeterProvider);

    // Create web vitals meter
        this.cwvMeter = metrics.getMeter('core-web-vitals', '1.0.0');

        // Initialising CWV metric gauge instruments (LCP given as example, others omitted here)
        this.lcp = this.cwvMeter.createObservableGauge('lcp', { unit: 'ms', description: 'Largest Contentful Paint' });
    }

    protected init(): InstrumentationModuleDefinition | InstrumentationModuleDefinition[] | void {}

  // Other steps discussed later
}

You'll notice in our LCP example we have created an ObservableGauge to capture the value at the time it is read via a callback function. This can be setup when we enable our custom instrumentation, specifying when the LCP event is triggered the value will be sent via result.observe:

/* Web Vitals Frontend package, LCP shown as example*/
import { onLCP, type LCPMetric } from 'web-vitals';

/* OpenTelemetry JS packages */
// Instrumentation base to create a custom Instrumentation for our provider
import {
    InstrumentationBase,
    type InstrumentationConfig,
    type InstrumentationModuleDefinition
} from '@opentelemetry/instrumentation';

// Metrics API
import {
    metrics,
    type ObservableGauge,
    type Meter,
    type Attributes,
    type ObservableResult,

} from '@opentelemetry/api';

// Other OTel Metrics imports omitted

// Time calculator via performance component
import { hrTime } from '@opentelemetry/core';

type CWVMetric = LCPMetric | CLSMetric | INPMetric | TTFBMetric | FCPMetric;

export class WebVitalsInstrumentation extends InstrumentationBase {

    /* Core Web Vitals Measures */
    private lcp: ObservableGauge;

    // Constructor and Initialization omitted

    enable() {
        // Capture Largest Contentful Paint, other vitals omitted
        onLCP(
            (metric) => {
                this.lcp.addCallback((result) => {
                    this.sendMetric(metric, result);
                });
            },
            { reportAllChanges: true }
        );
    }

  // Callback utility to add attributes and send captured metric
    private sendMetric(metric: CWVMetric, result: ObservableResult<Attributes>): void {
        const now = hrTime();

        const attributes = {
            startTime: now,
            'web_vital.name': metric.name,
            'web_vital.id': metric.id,
            'web_vital.navigationType': metric.navigationType,
            'web_vital.delta': metric.delta,
            'web_vital.value': metric.value,
            'web_vital.rating': metric.rating,
            // metric specific attributes
            'web_vital.entries': JSON.stringify(metric.entries)
        };

        result.observe(metric.value, attributes);
    }
}

To use our own instrumentation, we need to register our instrumentation just like we did in frontend.tracer.ts for the available web instrumentations to capture document and user event instrumentations:

registerInstrumentations({
  instrumentations: [
    // Other web instrumentations omitted
    // Custom Web Vitals instrumentation
    new WebVitalsInstrumentation({}, resource)
    ]
});

The lcp metric, along with the attributes we specified as part of our sendMetric function will be sent to our Elastic cluster:

These metrics will not feed into the User Experience dashboard due to compatibility, but we can create a dashboard leveraging the values to show the trends of each of our vitals:

Summary

In this blog, we presented the current state of client instrumentation for the browser, along with an example showing how to instrument a simple JavaScript frontend using the OpenTelemetry browser instrumentation. To reflect back on the code, check out the repo here. If you have any questions or want to learn from other developers connect with the Elastic Community.

Developer resources:

• OTel Record Store Application
• JavaScript Browser Instrumentation

As SREs, we've grown accustomed to jumping between dashboards, log aggregators, and trace visualizers when troubleshooting issues. But what if there was a better way? What if AI could analyze all your observability data holistically, answering complex questions in natural language, and identifying root causes automatically?

This is the next evolution of observability. But to harness this power, we need to rethink how we collect, store, and analyze our telemetry data.

The problem: siloed data limits AI effectiveness

Traditional observability setups separate data into distinct types:

• Metrics: Numeric measurements over time (CPU, memory, request rates)
• Logs: Detailed event records with timestamps and context
• Traces: Request journeys through distributed systems
• Profiles: Code-level execution patterns showing resource consumption and performance bottlenecks at the function/line level

This separation made sense historically due to the way the industry evolved. Different data types have traditionally had different cardinality, structure, access patterns and volume characteristics. However, this approach creates significant challenges for AI-powered analysis:

Metrics (Prometheus) → "CPU spiked at 09:17:00"
Logs (ELK) → "Exception in checkout service at 09:17:32" 
Traces (Jaeger) → "Slow DB queries in order-service at 09:17:28"
Profiles (pyroscope) -> "calculate_discount() is taking 75% of CPU time"

When these data sources live in separate systems, AI tools must either:

1. Work with an incomplete picture (seeing only metrics but not the related logs)
2. Rely on complex, brittle integrations that often introduce timing skew
3. Force developers to manually correlate information across tools

Imagine asking an AI, "Why did checkout latency spike at 09:17?" To answer comprehensively, it needs access to logs (to see the stack trace), traces (to understand the service path), and metrics (to identify resource strain). With siloed tools, the AI either sees only fragments of the story or requires complex ETL jobs that are slower than the incident itself.

Why traditional machine learning (ML) falls short

Traditional machine learning for observability typically focuses on anomaly detection within a single data dimension. It can tell you when metrics deviate from normal patterns, but struggles to provide context or root cause.

ML models trained on metrics alone might flag a latency spike, but can't connect it to a recent deployment (found in logs) or identify that it only affects requests to a specific database endpoint (found in traces). They behave like humans with extreme tunnel vision, seeing only a fraction of the relevant information and only the information that a specific vendor has given you an opinionated view into.

This limitation becomes particularly problematic in modern microservice architectures where problems frequently cascade across services. Without a unified view, traditional ML can detect symptoms but struggles to identify the underlying cause.

The solution: unified data with enriched logs

The solution is conceptually simple but transformative: unify metrics, logs, and traces into a single data store, ideally with enriched logs that contain all signals about a request in a single JSON document. We're about to see a merging of signals.

Think of traditional logs as simple text lines:

[2025-05-19 09:17:32] ERROR OrderService - Failed to process checkout for user 12345

Now imagine an enriched log that contains not just the error message, but also:

• The complete distributed trace context
• Related metrics at that moment
• System environment details
• Business context (user ID, cart value, etc.)

This approach creates a holistic view where every signal about the same event sits side-by-side, perfect for AI analysis.

How generative AI changes things

Generative AI differs fundamentally from traditional ML in its ability to:

1. Process unstructured data: Understanding free-form log messages and error text
2. Maintain context: Connecting related events across time and services
3. Answer natural language queries: Translating human questions into complex data analysis
4. Generate explanations: Providing reasoning alongside conclusions
5. Surface hidden patterns: Discovering correlations and anomalies in log data that would be impractical to find through manual analysis or traditional querying

With access to unified observability data, GenAI can analyze complete system behavior patterns and correlate across previously disconnected signals.

For example, when asked "Why is our checkout service slow?" a GenAI model with access to unified data can:

• Analyze unified enriched logs to identify which specific operations are slow and to find errors or warnings in those components
• Check attached metrics to understand resource utilization
• Correlate all these signals with deployment events or configuration changes
• Present a coherent explanation in natural language with supporting graphs and visualizations

Implementing unified observability with OpenTelemetry

OpenTelemetry provides the perfect foundation for unified observability with its consistent schema across metrics, logs, and traces. Here's how to implement enriched logs in a Java application:

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.metrics.Meter;
import io.opentelemetry.api.metrics.DoubleHistogram;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
import java.lang.management.ManagementFactory;
import java.lang.management.OperatingSystemMXBean;

public class OrderProcessor {
    private static final Logger logger = LoggerFactory.getLogger(OrderProcessor.class);
    private final Tracer tracer;
    private final DoubleHistogram cpuUsageHistogram;
    private final OperatingSystemMXBean osBean;

    public OrderProcessor(OpenTelemetry openTelemetry) {
        this.tracer = openTelemetry.getTracer("order-processor");
        Meter meter = openTelemetry.getMeter("order-processor");
        this.cpuUsageHistogram = meter.histogramBuilder("system.cpu.load")
                                      .setDescription("System CPU load")
                                      .setUnit("1")
                                      .build();
        this.osBean = ManagementFactory.getOperatingSystemMXBean();
    }

    public void processOrder(String orderId, double amount, String userId) {
        Span span = tracer.spanBuilder("processOrder").startSpan();
        try (Scope scope = span.makeCurrent()) {
            // Add attributes to the span
            span.setAttribute("order.id", orderId);
            span.setAttribute("order.amount", amount);
            span.setAttribute("user.id", userId);
            // Populate MDC for structured logging
            MDC.put("trace_id", span.getSpanContext().getTraceId());
            MDC.put("span_id", span.getSpanContext().getSpanId());
            MDC.put("order_id", orderId);
            MDC.put("order_amount", String.valueOf(amount));
            MDC.put("user_id", userId);
            // Record CPU usage metric associated with the current trace context
            double cpuLoad = osBean.getSystemLoadAverage();
            if (cpuLoad >= 0) {
                cpuUsageHistogram.record(cpuLoad);
                MDC.put("cpu_load", String.valueOf(cpuLoad));
            }
            // Log a structured message
            logger.info("Processing order");
            // Simulate business logic
            // ...
            span.setAttribute("order.status", "completed");
            logger.info("Order processed successfully");
        } catch (Exception e) {
            span.recordException(e);
            span.setAttribute("order.status", "failed");
            logger.error("Order processing failed", e);
        } finally {
            MDC.clear();
            span.end();
        }
    }
}

This code demonstrates how to:

1. Create a span for the operation
2. Add business attributes
3. Add current CPU usage
4. Link everything with consistent IDs
5. Record exceptions and outcomes in the backend system

When configured with an appropriate exporter, this creates enriched logs that contain both application events and their complete context.

Powerful queries across previously separate data

With data that has not yet been enriched, there is still hope. Firstly with GenAI powered ingestion it is possible to extract key fields to help correlate data such as a session id's. This will help you enrich your logs so they get the structure they need to behave like other signals. Below we can see Elastic's Auto Import mechanism that will automatically generate ingest pipelines and pull unstructured information from logs into a structured format perfect for analytics.

Once you have this data in the same data store, you can perform powerful join queries that were previously impossible. For example, finding slow database queries that affected specific API endpoints:

FROM logs-nginx.access-default 
| LOOKUP JOIN .ds-logs-mysql.slowlog-default-2025.05.01-000002 ON request_id 
| KEEP request_id, mysql.slowlog.query, url.query 
| WHERE mysql.slowlog.query IS NOT NULL

This query joins web server logs with database slow query logs, allowing you to directly correlate user-facing performance with database operations.

For GenAI interfaces, these complex queries can be generated automatically from natural language questions:

"Show me all checkout failures that coincided with slow database queries"

The AI translates this into appropriate queries across your unified data store, correlating application errors with database performance.

Real-world applications and use cases

Natural language investigation

Imagine asking your observability system:

"Why did checkout latency spike at 09:17 yesterday?"

A GenAI-powered system with unified data could respond:

"Checkout latency increased by 230% at 09:17:32 following deployment v2.4.1 at 09:15. The root cause appears to be increased MySQL query times in the inventory-service. Specifically, queries to the 'product_availability' table are taking an average of 2300ms compared to the normal 95ms. This coincides with a CPU spike on database host db-03 and 24 'Lock wait timeout' errors in the inventory service logs."

Here's an example of Claude Desktop connected to Elastic's MCP (Model Context Protocol) Server which demonstrates how powerful natural language investigations can be. Here we ask Claude "analyze my web traffic patterns" and as you can see it has correctly identified that this is in our demo environment.

Unknown problem detection

GenAI can identify subtle patterns by correlating signals that would be missed in siloed systems. For example, it might notice that a specific customer ID appears in error logs only when a particular network path is taken through your microservices—indicating a data corruption issue affecting only certain user flows.

Predictive maintenance

By analyzing the unified historical patterns leading up to previous incidents, GenAI can identify emerging problems before they cause outages:

"Warning: Current load pattern on authentication-service combined with increasing error rates in user-profile-service matches 87% of the signature that preceded the April 3rd outage. Recommend scaling user-profile-service pods immediately."

The future: agentic AI for observability

The next frontier is agentic AI, systems that not only analyze but take action automatically.

These AI agents could:

1. Continuously monitor all observability signals
2. Autonomously investigate anomalies
3. Implement fixes for known patterns
4. Learn from the effectiveness of previous interventions

For example, an observability agent might:

• Detect increased error rates in a service
• Analyze logs and traces to identify a memory leak
• Correlate with recent code changes
• Increase the memory limit temporarily
• Create a detailed ticket with the root cause analysis
• Monitor the fix effectiveness

This is about creating systems that understand your application's behavior patterns deeply enough to maintain them proactively. See how this works in Elastic Observability, in the screenshot at the end of the RCA we are sending an email summary but this could trigger any action.

Business outcomes

Unifying observability data for GenAI analysis delivers concrete benefits:

• Faster resolution times: Problems that previously required hours of manual correlation can be diagnosed in seconds
• Fewer escalations: Junior engineers can leverage AI to investigate complex issues before involving specialists
• Improved system reliability: Earlier detection and resolution of emerging issues
• Better developer experience: Less time spent context-switching between tools
• Enhanced capacity planning: More accurate prediction of resource needs

Implementation steps

Ready to start your observability transformation? Here's a practical roadmap:

1. Adopt OpenTelemetry: Standardize on OpenTelemetry for all telemetry data collection and use it to generate enriched logs.
2. Choose a unified storage solution: Select a platform that can efficiently store and query metrics, logs, traces and enriched logs together
3. Enrich your telemetry: Update application instrumentation to include relevant context
4. Create correlation IDs: Ensure every request has identifiers
5. Implement semantic conventions: Follow consistent naming patterns across your telemetry data
6. Start with focused use cases: Begin with high-value scenarios like checkout flows or critical APIs
7. Leverage GenAI tools: Integrate tools that can analyze your unified data and respond to natural language queries

Remember, AI can only be as smart as the data you feed it. The quality and completeness of your telemetry data will determine the effectiveness of your AI-powered observability.

Generative AI: an evolutionary catalyst for observability

The unification of observability data for GenAI analysis represents an evolutionary leap forward comparable to the transition from Internet 1.0 to 2.0. Early adopters will gain a significant competitive advantage through faster problem resolution, improved system reliability, and more efficient operations. GAI is a huge step for increasing observability maturity and moving your team to a more proactive stance.

Think of traditional observability as a doctor trying to diagnose a patient while only able to see their heart rate. Unified observability with GenAI is like giving that doctor a complete health picture, vital signs, lab results, medical history, and genetic data all accessible through natural conversation.

As SREs, we stand at the threshold of a new era in system observability. The asteroid of GenAI isn't a threat to be feared, it's an opportunity to evolve our practices and tools to build more reliable, understandable systems. The question isn't whether this transformation will happen, but who will lead it.

Will you?

Why OpenTelemetry?

OpenTelemetry provides a powerful set of capabilities that make it a compelling choice for open-source-focused users. Elastic is re-architecting its data ingest tools around OpenTelemetry to offer users vendor-agnostic flexibility, performance optimization through OTel's efficient data model for correlating telemetry, and enhanced flexibility and control over data pipelines. This move brings the benefits of open-source telemetry to Elastic users.

Elastic engineers are active contributors to the Otel project in several areas of the project. Demonstrating its commitment to open source, Elastic continues to make significant contributions to OpenTelemetry.

OpenTelemetry as the Core of Elastic's Data Ingestion

Elastic is transforming its data ingestion strategy by basing all ingestion mechanisms on the OpenTelemetry components. Elastic currently supports the following OTel based ingest architecture, which support OTel SDKs and Collectors from OTel or Elastic's Distribution of OpenTelemetry (EDOT).

This marks a fundamental shift, ensuring a more standardized and scalable telemetry pipeline. All the existing Elastic ingest components will become OTel based.

| | | | ---------------------------- | ----------------------------------------------------------------------------------------------- | | Beats | Beats architecture will be based on OTel. | | Elastic Agent | Agent architecture will be based on OTel to support both beats based inputs and OTel receivers. | | Integrations | Integrations catalogue will additionally include OTel based modules for ease of configuration. | | Fleet central management | Fleet will support monitoring of Elastic OTel collectors. |

Let's discuss how each component of Elastic's data ingestion platform will be based on an OpenTelemetry collector whilst still providing the same functionality to the user.

Beats

Elastic's traditional data shippers will be re-architected as OpenTelemetry Collectors, aligning with OTel's extensibility model. Current Beat architecture is essentially made up of a few stages in its pipeline, as shown in the diagram below. It consists of an Input, Processors for enrichments, Queuing of events and Output for batching and writing the data to a specific output.

Beatreceiver Concept

To ensure a smooth transition without major disruptions, a "beatsreceiver" concept is being implemented. These beatreceivers (like filebeatreceiver or metricbeatreceiver) act as dedicated Beat inputs integrated into the OpenTelemetry Collector as native receivers. They support all existing inputs and processors, guaranteeing that the final architecture accepts the user's current configuration and delivers the same functionality as today's Beats, all without introducing any breaking changes.

An OTel based Beats architecture will see the Input phase embedded as an OTel receiver (eg.  filebeatreceiver to represent the functionality of filebeat). This receiver would only be available as part of Elastic's distribution of OTel in support of our current user base and not a functionality that would be available upstream.

All the remaining components of the pipeline will be based on OTel. The new Beat will accept the same filebeat configuration (as an example) and will transform it to an OTel based configuration in order to avoid any deployment disruption. It should be noted that in this architecture the Beats will continue to only support ECS formatted data. In order to keep the Beat functionality inline with what exists today, the Elasticsearch exporter (as an example) will output ECS formatted data only.

The following diagram illustrates the beatreceiver concept by showing how a basic filebeat configuration is automatically translated into an OpenTelemetry-based configuration. This new configuration retains the original inputs and processors but leverages the native OpenTelemetry pipeline and exporter to achieve the same overall filebeat functionality. Existing filebeat configurations will be automatically converted, eliminating the need for manual adjustments or introducing breaking changes.

Elastic Agent

Elastic Agent is a unified agent for data collection, security, and observability. It can also be deployed in an OpenTelemetry only mode, enabling native OTel workflows. Elastic Agent is a supervisor that manages many other Beats as sub-processes in order to provide a more comprehensive data collection tool. It is capable of translating Agent Policy received from Fleet into configuration acceptable by the various sub-processes.

Expanding on the Beat receiver concept described above, the Elastic Agent, which currently can be deployed as an OTel collector (see blog), will be also modified to a much simpler OTel based architecture based on these receivers. As shown below, this architecture will streamline the components within the Elastic Agent and remove duplicated functionality such as queuing and output. Whilst supporting the current functionality, these changes will reduce the agent footprint and also present a reduction in number of connections opened to pipeline elements egress of the agent (such as Elasticsearch clusters, Logstash or Kafka brokers).

By moving to an OTel based architecture Elastic Agent is now able to operate as a truly hybrid Elastic Agent which provides not only the Beat functionality but also allows our users to create OTel native pipelines and take advantage of plethora of functionality available as part of the open source project.

Elastic's commitment to OpenTelemetry will deepen through increased contributions, resulting in OpenTelemetry receivers gradually superseding Beats receiver features. This evolution will eventually reduce the need for a distinct Beats receiver within the Elastic Agent architecture. The envisioned architecture will empower the Elastic Agent to transmit data in OTLP format as well, granting users the flexibility to select any OTLP-compatible backend, thereby upholding the principle of vendor neutrality.

Fleet & Integrations: Managing OpenTelemetry at Scale

Elastic's centralized management system will support OpenTelemetry-based configurations, making large-scale deployments easier to manage. Managing thousands of telemetry agents at scale presents a significant challenge. Elastic's Fleet & Integrations simplify this process by providing robust lifecycle management for these new OpenTelemetry-based Elastic agents.

Key Capabilities Offered:

• Scalability: Manage up to 100K+ agents across distributed environments.

• Automated Upgrades: Staged rollouts and automatic upgrades ensure minimal downtime.

• Monitoring & Diagnostics: Real-time status updates, failure detection, and diagnostic downloads improve system reliability.

• Policy-Based Configuration Management: Enables centralized control over agent configurations, improving consistency across deployments.

• Pre-Built Integrations: Elastic offers a catalog of 470+ pre-built integrations, allowing users to ingest data seamlessly from various sources. These will also include OTel based packages making configuration much more efficient across a large deployment.

The goal is for Fleet to also provide monitoring capabilities for native OTel collectors as well in a vendor agnostic fashion.

Conclusion

Elastic's adoption of OpenTelemetry marks a significant milestone in the evolution of open-source observability. By standardizing on OpenTelemetry, Elastic is ensuring that its data ingestion strategy remains open, scalable, and future-proof.

For open-source users, this shift means:

• Greater interoperability across observability tools.

• Enhanced flexibility in choosing telemetry backends.

• A stronger commitment to community-driven observability standards.

• Existing Beats and Elastic Agent users can seamlessly adopt OpenTelemetry without rearchitecting their pipelines.

• OpenTelemetry users can integrate with Elastic's observability stack without additional complexity.

Stay tuned for more updates as Elastic continues to expand its OpenTelemetry-based data collection capabilities! In the mean time here are some other references:

• Elastic Distributions of OpenTelemetry (EDOT) Now GA

• Dynamic workload discovery on Kubernetes now supported with EDOT Collector

• Introducing the OTTL Playground for OpenTelemetry

Despite the transformative potential, LLMs introduce complex challenges that necessitate a new level of observability as LLMs are notoriously opaque. Enter LLM observability: a crucial component in the lifecycle management of LLMs. This aspect becomes vital for Service Reliability Engineers (SREs) and other key stakeholders tasked with ensuring seamless, error-free operations, cost control, and minimizing the risks associated with the unpredictable nature of LLM generated responses. SREs need insights into performance metrics, error frequencies, latency issues, the cost implications of running these sophisticated models, and the prompt and response exchange with the model. Traditional monitoring tools fall short in this high-stakes environment; what’s needed is a nuanced approach to address the unique observability demands that LLMs introduce.

Elastic's LLM Observability Capabilities Address These Challenges

With Elastic’s end-to-end LLM observability you can cover a wide range of use cases. To achieve this, you can onboard two types of integrations - API-based logs and metrics and via APM instrumentation. Depending on your use case, you can also choose to use of the LLM integrations.

1. High level overview: via API-based logs and metrics. Monitoring LLM services from providers by ingesting a curated set of service metrics and logs like latency, invocation frequency, tokens, errors, and prompts and responses. Each LLM integration comes with out-of-the-box dashboards.

2. Troubleshooting applications: via APM instrumentation. Fully OTel-native tracing and auto-instrumentation for LLM-based applications through Elastic Distributions of OpenTelemetry (EDOT). Additionally, you can use third party libraries (Langtrace, OpenLit, OpenLLMetry) together with Elastic to extend the coverage to additional LLM-related technologies. 

High level overview: LLM Observability for Leading Providers

Elastic offers tailored API-based integrations for four major LLM hosting providers:

• Azure OpenAI

• OpenAI

• Amazon Bedrock

• Google Vertex AI

These integrations bring a curated set of logs and metrics collection tailored to each provider. What this means for SREs is straightforward access to pre-configured dashboards that highlight the prompts and responses, usage patterns, performance metrics, and cost details across different models and providers.

For instance, SREs keen on identifying which LLM generates the most errors or insights about the models in terms of latency, cost, or usage frequency can leverage these integrations. Imagine having the capability to instantly visualize which LLM is slowing down processes or incurring high costs, thus enabling data-driven decisions to optimize operations.

Troubleshooting applications: Tracing and Auto-Instrumentation of OpenAI, Amazon Bedrock and Google Vertex AI models

Elastic supports OTLP tracing capabilities in EDOT for applications using OpenAI models and models hosted on Amazon Bedrock and Google Vertex AI. In addition, Elastic also supports LLM tracing from third party libraries (Langtrace, OpenLIT, OpenLLMetry). 

Tracing offers a comprehensive map of an application's request flow, pinpointing granular details about each call within the system. For each transaction and span of a request, tracing shows critical information such as specific models utilized, request duration, errors encountered, tokens used per request, and the prompts and responses between the LLM.

Tracing helps SREs troubleshoot performance issues with applications developed in languages like Python, Node.js and Java." If an SRE needs to investigate latency or error issues, LLM tracing provides a zoomed-in view into the request lifecycle and allows for profound insights into whether a delay is application-specific, model-specific or systemic across deployments.

Use Cases: Bringing Elastic's Observability Features to Life

Let’s explore some practical scenarios where Elastic’s observability tools shine:

1. Understanding LLM Performance and Reliability

An SRE team looking to optimize a customer support system powered by Azure OpenAI can utilize Elastic’s Azure OpenAI integration to quickly ascertain which model variants incur higher latency or error rates. This enhances decision-making regarding model deployment or even switching providers based on performance metrics.

Similarly SREs can also use in parallel integrations for Google Vertex AI, Amazon Bedrock, and OpenAI for other applications using models hosted on these providers.

2. Troubleshooting OpenAI-Powered Applications

Consider an enterprise utilizing an OpenAI model for real-time user interactions. Encountering unexplained delays, an SRE can use OpenAI tracing to dissect the transaction pathway, identifying if one specific API call or model invocation is the bottleneck. The SRE can also check the out-of-the-box OpenAI integration dashboard to verify if the latency is only affecting this application or all model invocations across the organization.

An engineer troubleshooting the LLM-based application can also check to see what were the prompt and response exchanges with the LLM during this request so they can rule out possible impact on performance due to the input.

3. Addressing Cost and Usage Concerns

SREs are generally acutely aware of which LLM configurations are less cost-effective than required. Elastic’s integration dashboards, pre-configured to display model usage patterns, help mitigate unnecessary spending effectively. You can find out-of-the box dashboards for Azure OpenAI, OpenAI, Amazon Bedrock, and Google VertexAI models. These dashboards show key cost and usage information such as total invocations and tokens, as well as time series breakdown by model and endpoint. In addition, some integrations show more advanced usage information such as provisioned throughput units (PTU) as well as billing cost.

4. Understanding LLM Compliance 

With the Elastic Amazon Bedrock integration for Guardrails, and Azure OpenAI integration for content filtering, SREs can swiftly address security concerns, like verifying if certain user interactions prompt policy violations. Elastic's observability logs clarify whether guardrails rightly blocked potentially harmful responses, bolstering compliance assurance.

Conclusion

As LLMs continue to revolutionize the capabilities of modern applications, the role of observability becomes increasingly paramount. Elastic’s comprehensive observability framework empowers enterprises to harness the full potential of LLMs while maintaining robust operational insight and control. The integration with prominent LLM hosting providers and advanced tracing for OpenAI, Amazon Bedrock and Google Vertex AI models, equips SREs with the necessary arsenal to navigate the complex landscape of LLM-driven applications, ensuring they remain safe, reliable, efficient, and cost-effective.

In this new era of AI, balancing innovation with observability isn't just beneficial—it's essential. Whether optimizing performance, troubleshooting intricacies, or managing costs and compliance, Elastic stands at the forefront, ensuring your LLM journey is as seamless as it is groundbreaking.

OpenTelemetry recently published a blog on how to do Autodiscovery based on Kubernetes Pods' annotations with the OpenTelemetry Collector.

In this blog post, we will talk about how to use this Kubernetes-related feature of the OpenTelemetry Collector, which is already available with the Elastic Distribution of the OpenTelemetry (EDOT) Collector.

In addition to this feature, at Elastic, we heavily invest in making OpenTelemetry the best, standardized ingest solution for Observability. You might already have seen us focusing on:

• Semantic Conventions standardization

• significant log collection improvements

• various other topics around instrumentation

• profiling

Let's walk you through a hands-on journey using the EDOT Collector covering various use cases you might encounter in the real world, highlighting the capabilities of this powerful feature.

Configuring EDOT Collector

The Collector’s configuration is not our main focus here, since based on the nature of this feature it is minimal, letting workloads define how they should be monitored.

To illustrate the point, here is the Collector configuration snippet that enables the feature for both logs and metrics:

receivers:
    receiver_creator/metrics:
      watch_observers: [k8s_observer]
      discovery:
        enabled: true
      receivers:

    receiver_creator/logs:
      watch_observers: [k8s_observer]
      discovery:
        enabled: true
      receivers:

You can include the above in the EDOT’s Collector configuration, specifically the receivers’ section.

Since logs collection in our examples will happen from the discovery feature make sure that the static filelog receiver configuration block is removed and its preset is disabled (i.e. set to false) to avoid having log duplication.

Make sure that the receiver creator is properly added in the pipelines for logs (in addition to removing the filelog receiver completely) and metrics respectively.

Ensure that k8sobserver is enabled as part of the extensions:

extensions:
  k8s_observer:
    observe_nodes: true
    observe_services: true
    observe_ingresses: true

// ...

service:
  extensions: [k8s_observer]

Last but not least, ensure the log files' volume is mounted properly:

volumeMounts:
 - name: varlogpods
   mountPath: /var/log/pods
   readOnly: true

volumes:
  - name: varlogpods
    hostPath:
      path: /var/log/pods

Once the configuration is ready follow the Kubernetes quickstart guides on how to deploy the EDOT Collector. Make sure to replace the values.yaml file linked in the quickstart guide with the file that includes the above-described modifications.

Collecting Metrics from Moving Targets Based on Their Annotations

In this example, we have a Deployment with a Pod spec that consists of two different containers. One container runs a Redis server, while the other runs an NGINX server. Consequently, we want to provide different hints for each of these target containers.

The annotation-based discovery feature supports this, allowing us to specify metrics annotations per exposed container port.

Here is how the complete spec file looks:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    user  nginx;
    worker_processes  1;
    error_log  /dev/stderr warn;
    pid        /var/run/nginx.pid;
    events {
      worker_connections  1024;
    }
    http {
      include       /etc/nginx/mime.types;
      default_type  application/octet-stream;

      log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
      access_log  /dev/stdout main;
      server {
          listen 80;
          server_name localhost;

          location /nginx_status {
              stub_status on;
          }
      }
      include /etc/nginx/conf.d/*;
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deployment
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
      annotations:
        # redis container port hints
        io.opentelemetry.discovery.metrics.6379/enabled: "true"
        io.opentelemetry.discovery.metrics.6379/scraper: redis
        io.opentelemetry.discovery.metrics.6379/config: |
          collection_interval: "20s"
          timeout: "10s"

        # nginx container port hints
        io.opentelemetry.discovery.metrics.80/enabled: "true"
        io.opentelemetry.discovery.metrics.80/scraper: nginx
        io.opentelemetry.discovery.metrics.80/config: |
          endpoint: "http://`endpoint`/nginx_status"
          collection_interval: "30s"
          timeout: "20s"
    spec:
      volumes:
      - name: nginx-conf
        configMap:
          name: nginx-conf
          items:
            - key: nginx.conf
              path: nginx.conf
      containers:
        - name: webserver
          image: nginx:latest
          ports:
            - containerPort: 80
              name: webserver
          volumeMounts:
            - mountPath: /etc/nginx/nginx.conf
              readOnly: true
              subPath: nginx.conf
              name: nginx-conf
        - image: redis
          imagePullPolicy: IfNotPresent
          name: redis
          ports:
            - name: redis
              containerPort: 6379
              protocol: TCP

When this workload is deployed, the Collector will automatically discover it and identify the specific annotations. After this, two different receivers will be started, each one responsible for each of the target containers.

Collecting Logs from Multiple Target Containers

The annotation-based discovery feature also supports log collection based on the provided annotations. In the example below, we again have a Deployment with a Pod consisting of two different containers, where we want to apply different log collection configurations. We can specify annotations that are scoped to individual container names:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-logs-deployment
  labels:
    app: busybox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
      annotations:
        io.opentelemetry.discovery.logs.lazybox/enabled: "true"
        io.opentelemetry.discovery.logs.lazybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints-lazybox
        io.opentelemetry.discovery.logs.busybox/enabled: "true"
        io.opentelemetry.discovery.logs.busybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints-busybox
    spec:
      containers:
        - name: busybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs from busybox at $(date +%H:%M:%S)" && sleep 5s; done
        - name: lazybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs from lazybox at $(date +%H:%M:%S)" && sleep 25s; done

The above configuration enables two different filelog receiver instances, each applying a unique parsing configuration. This is handy when we know how to parse specific technology logs, such as Apache server access logs.

Combining Both Metrics and Logs Collection

In our third example, we illustrate how to define both metrics and log annotations on the same workload. This allows us to collect both signals from the discovered workload. Below is a Deployment with a Pod consisting of a Redis server and a BusyBox container that performs dummy log writing. We can target annotations to the port and container levels to collect metrics from the Redis server using the Redis receiver, and logs from the BusyBox using the filelog receiver. Here’s how:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deployment
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
      annotations:
        io.opentelemetry.discovery.metrics.6379/enabled: "true"
        io.opentelemetry.discovery.metrics.6379/scraper: redis
        io.opentelemetry.discovery.metrics.6379/config: |
          collection_interval: "20s"
          timeout: "10s"

        io.opentelemetry.discovery.logs.busybox/enabled: "true"
        io.opentelemetry.discovery.logs.busybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints
    spec:
      containers:
        - image: redis
          imagePullPolicy: IfNotPresent
          name: redis
          ports:
            - name: redis
              containerPort: 6379
              protocol: TCP
        - name: busybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs at $(date +%H:%M:%S)" && sleep 15s; done

Explore and analyse data coming from dynamic targets in Elastic

Once the target Pods are discovered and the Collector has started collecting telemetry data from them, we can then explore this data in Elastic. In Discover we can search for Redis and NGINX metrics as well as logs collected from the Busybox container. Here is how it looks like:

Summary

The examples above showcase how users of our OpenTelemetry Collector can take advantage of this new feature — one we played a major role in developing.

For this, we leveraged our years of experience with similar features already supported in Metricbeat, Filebeat, and Elastic-Agent. This makes us extremely happy and confident, as it closes the feature gap between Elastic's specific monitoring agents and the OpenTelemetry Collector — making it even better.

Interested in learning more? Visit the documentation and give it a try by following our EDOT quickstart guide.

The investment paradox of observability in 2025

Here's something fascinating: 96% of executives in our survey expect observability to remain a key investment area. Yet almost all of them (97%) are hitting roadblocks in realizing full value. And surprisingly, the primary hurdles for observability are not technical or complicated in nature, can you guess what they might be?

For 2025, IT leaders are challenged with financial hurdles for their observability. I'm seeing this tension play out constantly in conversations with leaders - they know they need to invest, but they're grappling with budget constraints, licensing costs, and proving ROI for their organizations. This creates an interesting dynamic where organizations must carefully balance increasing investment with rigorous cost optimization and business metrics.

What's particularly interesting is how this paradox is forcing organizations to become more strategic about their investments. Leaders are no longer just throwing money at the problem - they're thinking carefully about how to maximize value from every dollar spent.

Why observability maturity is making all the difference

The data really jumps out at me here. The gap between observability experts and newcomers tells a compelling story that I wasn't expecting to see. Expert organizations are significantly outperforming their peers across every key metric:

• 91% of expert organizations are deploying applications and infrastructure faster (compared to just 34% of those in early stages)

• 82% are successfully reducing operational costs (versus 56% of early-stage organizations)

• 71% achieve better MTTR for incidents (while only 40% of early-stage organizations do)

What I find particularly fascinating is how some benefits go beyond just maturity levels. About 80% of organizations report better customer issue response times regardless of their maturity stage. It tells me that even basic observability delivers immediate customer-facing value. This is crucial information for organizations just starting their observability journey - they can expect to see tangible benefits right from the start. But the overarching story may be that observability maturity leads teams from reactive to proactive and allows them to focus on higher level, value-add activities.

Cost management: the new imperative

The numbers around cost management paint a clear picture of where the industry is heading - 97% of IT decision-makers are actively managing observability costs, and 86% feel personally responsible for business outcomes.

I'm seeing a clear trend where leaders are taking concrete steps in their day to day work:

• Consolidating their observability toolset while maintaining capabilities, they don’t want to lose anything
• Implementing usage-based pricing models
• Establishing clear ROI metrics
• Creating cross-functional teams to optimize spending

This isn't just about cutting costs - it's about being smarter with resources. Organizations are learning that more tools don't necessarily mean better observability.

Two technologies reshaping the observability landscape

AI's growing impact

The enthusiasm for AI is remarkable - 94% of respondents see its tremendous potential. What fascinates me is how concerns about Generative AI reliability have actually decreased from 64% to 55% over the past year.

Leaders are particularly excited about:

• Automated correlation of logs, metrics, and traces (72% of respondents)
• Predictive analytics for preventing outages
• Natural language interfaces for querying observability data
• Automated root cause analysis

The key shift I'm seeing for the upcoming year is the move from AI as a buzzword to AI as a practical tool delivering real value in observability workflows.

Generative AI capabilities paired with retrieval augmented generation (RAG) capabilities allow organizations to leverage the power of LLMs and private data (e.g., runbooks, alerts, business data) to deliver relevant and meaningful results and identify and solve problems faster while reducing noise.

OpenTelemetry's continued momentum

Looking at expert organizations, 80% are either experimenting with or have deployed OpenTelemetry. This isn't just about technology adoption - it's about building for the future with open standards. The correlation between OpenTelemetry adoption and overall observability maturity is correlated and unmistakable.

What's particularly interesting is how OpenTelemetry is changing the vendor landscape. Organizations are increasingly demanding OpenTelemetry support from their vendors, seeing it as a way to future-proof their observability investments and avoid vendor lock-in. Thinking back to how Linux shifted the server landscape, can we expect to see the same in the observability domain?

Business integration and insights deepens

Here's what I find most compelling: 64% of expert organizations are frequently correlating operational data with business outcomes, while only 9% of early-stage organizations do the same. This represents a fundamental shift from technical monitoring to business observability.

This isn't just about uptime anymore - organizations are increasingly using observability data to:

• Make informed business decisions
• Improve customer experience
• Optimize resource allocation
• Drive innovation

Looking ahead

As we continue through 2025, I'm seeing observability mature beyond its initial promise. Organizations are focusing less on basic implementation and more on delivering real business value through:

• Deeper business integration, like mapping system performance directly to revenue metrics
• Optimized cost management through new data lake technology, efficient storage and intelligent retention
• AI-enhanced capabilities powered by LLMs and Agentic AI
• Standardized instrumentation through OpenTelemetry, reducing vendor lock-in

The path to success in 2025 isn't just about having the right tools - it's about building mature practices that deliver measurable business value while managing costs effectively. The organizations that can balance these competing demands while maintaining focus on business outcomes are the ones pulling ahead.

What are you seeing in your organization's observability journey? Are these trends aligning with your experience?

If you would like to dig in deeper on emerging observability trends, download our full report or watch the on-demand webinar, 2025 Observability trends: Maturing beyond the hype and delivering results!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Luckily, Elastic has been one of the leading solutions in observability space and a contributor to the OpenTelemetry project. Elastic’s unique position and its extensive observability capabilities allows end-users to monitor applications built with object-oriented programming languages & Framework in a variety of ways.

In this blog we will explore using Elastic APM to investigate C++ traces with the OpenTelemetry client. We will be providing a comprehensive guide on how to implement the OpenTelemetry client for C++ applications and connecting to Elastic APM solutions. While OTel has its libraries, and this blog reviews how to use the OTel CPP library, Elastic also has its own Elastic Distributions of OpenTelemetry, which were developed to provide commercial support, and are completely upstreamed regularly.

Here are some resources to help get you started:

• Use OpenTelemetry with APM

• The OpenTelemetry C++ Client

• OpenTelemetry C++ Docs

Step by Step Guide

Prerequisites

• ### Environment

Choosing an environment is quite important as there is limited support for the OTEL client. We have experimented with using multiple Operating Systems and here are the suggestions:

• Ubuntu 22.04

• Debian 11 Bullseye

• For this guide we are focusing on Ubuntu 22.04.

• Machine: 2 vCPU, 4GB is sufficient.

• Image: Ubuntu 22.04 LTS (x86_64).

• Disk: ~30 GB is enough.

Implementation method 

We have experimented with multiple methods but we found that the most suitable approach is to use a package manager. After extensive testing, It appears that trying to run otel-cpp client could be quite challenging to the users. If practitioners desire to build with tools such as CMake and Bazel that is a viable solution. With that, as we tested both methods it became obvious that we were spending most of our time and effort fixing compatibility and dependencies’ issues for the OS Vs. Focusing on sending data to our APM. Hence we decided to move to a different method.

The main issues that we kept running into as we test are:

• Compatibility of packages.

• Availability of packages.

• Dependencies of libraries and packages.

In this guide we will use vcpkg since it allows us to bring in all the dependencies required to run the Opentelemetry C++ client.

Installing required OS tools

Update package lists

    sudo apt-get update

Install build essentials, cmake, git, and sqlite dev library

    sudo apt-get install -y build-essential cmake git curl zip unzip sqlite3 libsqlite3-dev

sqlite3 and libsqlite3-dev allow us to build/run SQLite queries in our C++ code.

Set Up vcpkg

vcpkg is the C++ package manager that we’ll use to install opentelemetry-cpp client.

    # Clone vcpkg
    cd ~
    git clone https://github.com/microsoft/vcpkg.git

    # Bootstrap
    cd ~/vcpkg
    ./bootstrap-vcpkg.sh

Install OpenTelemetry C++ with OTLP gRPC

In this guide we focus on trace export to Elastic. At time of writing, vcpkg’s opentelemetry-cpp

version 1.18.0 fully supports traces but has limited direct metrics exporting.

Install the package

    cd ~/vcpkg
    ./vcpkg install opentelemetry-cpp[otlp-grpc]:x64-linux

Note

Sometimes when installing opentelemetry-cpp on linux it doesn't install all the required packages. As a workaround if you run into that case, try running again but pass a flag to allow-unsupported:

    ./vcpkg install opentelemetry-cpp[*]:x64-linux --allow-unsupported

Verify

    ./vcpkg list | grep opentelemetry-cpp

The output thould be something like this:

opentelemetry-cpp:x64-linux 1.18.0

Create the C++ Project with Database Spans

We’ll build a sample in ~/otel-app that:

• Uses SQLite to do basic CREATE/INSERT/SELECT queries. This is helpful to showcase capturing transactions for apps that use databases on Elastic APM.

• Generate random traces to showcase how they are captured on Elastic APM.

This app is going to generate random queries where some will contain database transactions and some are just application traces. Each query is contained in a child span, so they appear in APM as separate database transactions.

# Below is the structure of our project

    otel-app/
    ├── main.cpp
    └── CMakeLists.txt

Create App Project

    cd ~
    mkdir otel-app
    cd otel-app

Inside this project we will create two files

• main.cpp

• CMakeLists.txt

Keep in mind that main.cpp is where you are going to pass the otel exporters that are going to send data to the Elastic cluster. So for your tech stack it would be your application's source code.

Sample application code

    main.cpp
    // Below we declare required libraries that we will be using to ship
    // traces to Elastic APM
    #include <opentelemetry/exporters/otlp/otlp_grpc_exporter.h>
    #include <opentelemetry/sdk/trace/tracer_provider.h>
    #include <opentelemetry/sdk/trace/simple_processor.h>
    #include <opentelemetry/trace/provider.h>

    #include <sqlite3.h>
    #include <chrono>
    #include <iostream>
    #include <thread>
    #include <cstdlib>&nbsp; // for rand(), srand()
    #include <ctime>&nbsp; &nbsp; // for time()

    // Namespace aliases
    namespace trace_api = opentelemetry::trace;
    namespace sdktrace&nbsp; = opentelemetry::sdk::trace;
    namespace otlp&nbsp; &nbsp; &nbsp; = opentelemetry::exporter::otlp;

    // Below we are using a helper function to run SQLITE statement inside&nbsp;
    // child span
    bool ExecuteSql(sqlite3 *db, const std::string &sql,
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;trace_api::Tracer &tracer,
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;const std::string &span_name)
    {
    &nbsp;&nbsp;// Starting the child span
    &nbsp;&nbsp;auto db_span = tracer.StartSpan(span_name);
    &nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;auto scope = tracer.WithActiveSpan(db_span);

    &nbsp;&nbsp;&nbsp;&nbsp;// Here we mark Database attributes for clarity in APM
    &nbsp;&nbsp;&nbsp;&nbsp;db_span->SetAttribute("db.system", "sqlite");
    &nbsp;&nbsp;&nbsp;&nbsp;db_span->SetAttribute("db.statement", sql);

    &nbsp;&nbsp;&nbsp;&nbsp;char *errMsg = nullptr;
    &nbsp;&nbsp;&nbsp;&nbsp;int rc = sqlite3_exec(db, sql.c_str(), nullptr, nullptr, &errMsg);
    &nbsp;&nbsp;&nbsp;&nbsp;if (rc != SQLITE_OK)
    &nbsp;&nbsp;&nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;db_span->AddEvent("SQLite error: " + std::string(errMsg ? errMsg : "unknown"));
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sqlite3_free(errMsg);
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;db_span->End();
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return false;
    &nbsp;&nbsp;&nbsp;&nbsp;}
    &nbsp;&nbsp;&nbsp;&nbsp;db_span->AddEvent("Query OK");
    &nbsp;&nbsp;}
    &nbsp;&nbsp;db_span->End();
    &nbsp;&nbsp;return true;
    }

    /**
    &nbsp;* DoNonDbWork - Simulate some other operation
    &nbsp;*/
    void DoNonDbWork(trace_api::Tracer &tracer, const std::string &span_name)
    {
    &nbsp;&nbsp;auto child_span = tracer.StartSpan(span_name);
    &nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;auto scope = tracer.WithActiveSpan(child_span);
    &nbsp;&nbsp;&nbsp;&nbsp;// Just sleep or do some "fake" work
    &nbsp;&nbsp;&nbsp;&nbsp;std::cout << "[TRACE] Doing non-DB work for " << span_name << "...\n";
    &nbsp;&nbsp;&nbsp;&nbsp;std::this_thread::sleep_for(std::chrono::milliseconds(200 + rand() % 300));
    &nbsp;&nbsp;&nbsp;&nbsp;child_span->AddEvent("Finished non-DB work");
    &nbsp;&nbsp;}
    &nbsp;&nbsp;child_span->End();
    }

    int main()
    {
    &nbsp;&nbsp;// Seed random generator for example
    &nbsp;&nbsp;srand(static_cast<unsigned>(time(nullptr)));

    &nbsp;&nbsp;// 1) Create OTLP exporter for traces
    &nbsp;&nbsp;otlp::OtlpGrpcExporterOptions opts;
    &nbsp;&nbsp;auto exporter = std::make_unique<otlp::OtlpGrpcExporter>(opts);

    &nbsp;&nbsp;// 2) Simple Span Processor
    &nbsp;&nbsp;auto processor = std::make_unique<sdktrace::SimpleSpanProcessor>(std::move(exporter));

    &nbsp;&nbsp;// 3) Tracer Provider
    &nbsp;&nbsp;auto sdk_tracer_provider = std::make_shared<sdktrace::TracerProvider>(std::move(processor));
    &nbsp;&nbsp;auto tracer = sdk_tracer_provider->GetTracer("my-cpp-multi-app");

    &nbsp;&nbsp;// Prepare an in-memory SQLite DB (for random DB usage)
    &nbsp;&nbsp;sqlite3 *db = nullptr;
    &nbsp;&nbsp;int rc = sqlite3_open(":memory:", &db);
    &nbsp;&nbsp;if (rc == SQLITE_OK)
    &nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;// Create a table so we can do inserts/reads
    &nbsp;&nbsp;&nbsp;&nbsp;ExecuteSql(db, "CREATE TABLE IF NOT EXISTS items (id INTEGER PRIMARY KEY, info TEXT);",
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;*tracer.get(), "db_create_table");
    &nbsp;&nbsp;}

    &nbsp;&nbsp;// Create the following loop to generate multiple transactions
    &nbsp;&nbsp;int num_transactions = 5;&nbsp; // Change this variable to the desired number of transaction
    &nbsp;&nbsp;for (int i = 1; i <= num_transactions; i++)
    &nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;// Each iteration is a top-level transaction
    &nbsp;&nbsp;&nbsp;&nbsp;std::string transaction_name = "transaction_" + std::to_string(i);
    &nbsp;&nbsp;&nbsp;&nbsp;auto parent_span = tracer->StartSpan(transaction_name);
    &nbsp;&nbsp;&nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;auto scope = tracer->WithActiveSpan(parent_span);

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;std::cout << "\n=== Starting " << transaction_name << " ===\n";

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Randomly select whether a transaction will interact with the database or not.
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;bool doDb = (rand() % 2 == 0); // 50% chance

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (doDb && db)
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Insert random data
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;std::string insert_sql = "INSERT INTO items (info) VALUES ('Item " + std::to_string(i) + "');";
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ExecuteSql(db, insert_sql, *tracer.get(), "db_insert_item");

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Select from DB
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ExecuteSql(db, "SELECT * FROM items;", *tracer.get(), "db_select_items");
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Do some random non-DB tasks
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DoNonDbWork(*tracer.get(), "non_db_task_1");
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DoNonDbWork(*tracer.get(), "non_db_task_2");
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Sleep a little to simulate transaction time
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;std::this_thread::sleep_for(std::chrono::milliseconds(200));
    &nbsp;&nbsp;&nbsp;&nbsp;}
    &nbsp;&nbsp;&nbsp;&nbsp;parent_span->End();
    &nbsp;&nbsp;}

    &nbsp;&nbsp;// Close DB
    &nbsp;&nbsp;sqlite3_close(db);

    &nbsp;&nbsp;// Extra sleep to ensure final flush
    &nbsp;&nbsp;std::cout << "\n[INFO] Sleeping 5 seconds to allow flush...\n";
    &nbsp;&nbsp;std::this_thread::sleep_for(std::chrono::seconds(5));
    &nbsp;&nbsp;std::cout << "[INFO] Exiting.\n";
    &nbsp;&nbsp;return 0;
    }

What does the code do?

We create 5 top-level “transaction_i” spans.

For each transaction, we randomly choose to do DB or non-DB work

- If DB: Insert a row, then select. Each is a child span.

- If non-DB: We do two “fake tasks” (child spans).

Once we finish, we close the database connection and wait 5 seconds for data flush.

Sample instruction file

CMakeLists.txt : This file contains instructions describing the source files and targets.

    cmake_minimum_required(VERSION 3.10)
    project(OtelApp VERSION 1.0)

    set(CMAKE_CXX_STANDARD 11)
    set(CMAKE_CXX_STANDARD_REQUIRED ON)

    # Here we are pointing to use the vcpkg toolchain
    set(CMAKE_TOOLCHAIN_FILE "PATH-TO/vcpkg.cmake" CACHE STRING "Vcpkg toolchain file")

    find_package(opentelemetry-cpp CONFIG REQUIRED)

    add_executable(otel_app main.cpp)

    # Below we are linking the OTLP gRPC exporter, trace library, and sqlite3
    target_link_libraries(otel_app PRIVATE
        opentelemetry-cpp::otlp_grpc_exporter
        opentelemetry-cpp::trace
        sqlite3
    )

Declare Environmental Variables

Here we are going to export our Elastic Cloud endpoints as environmental variables

You can get that information by doing the following:

1. Login into your elastic cloud

2. Go into your deployment

3. On the Left hand side, click on the hamburger menu and scroll down to “Integrations”

4. Go on the search bar inside the integration and type “APM”

1. Click on the APM integration

2. Scroll down and click on the OpenTelemetry Option on the far left side

1. You should be able to see values similar to the screenshot below. Once you copy the values to export, click on launch APM.

As you copy the required values, go ahead and export them.

    export OTEL_EXPORTER_OTLP_ENDPOINT="APM-ENDPOINT"
    export OTEL_EXPORTER_OTLP_HEADERS="KEY"
    export OTEL_RESOURCE_ATTRIBUTES="service.name=my-app,service.version=1.0.0,deployment.environment=dev"

Note that the elastic OTEL_EXPORTER_OTLP_HEADERS value usually starts with “Authorization=Bearer” make sure that you convert the upper case “A” in authorization to a lower case “a”. This is due to the fact that the otel header exporter expects a lower case “a” for authorization.

Build and Run

Once we create the two files we then move to building the application.

cd ~/otel-app
mkdir -p build
cd build

cmake -DCMAKE_TOOLCHAIN_FILE=~/vcpkg/scripts/buildsystems/vcpkg.cmake \
      -DCMAKE_PREFIX_PATH=~/vcpkg/installed/x64-linux/share \
      ..
make

Once make is successful run the the application

./otel-app

You should be able to see the script execute with a similar console output

    Console outcome:
    === Starting transaction_1 ===
    [TRACE] Doing non-DB work for non_db_task_1...
    [TRACE] Doing non-DB work for non_db_task_2...

    === Starting transaction_2 ===
    [TRACE] Doing DB work for doDb_task_1...
    [TRACE] Doing DB work for doDb_task_2...

    === Starting transaction_3 ===
    [TRACE] Doing non-DB work for non_db_task_1...
    [TRACE] Doing non-DB work for non_db_task_2...

    === Starting transaction_4 ===
    [TRACE] Doing non-DB work for non_db_task_1...
    [TRACE] Doing non-DB work for non_db_task_2...

    === Starting transaction_5 ===
    [TRACE] Doing non-DB work for non_db_task_1...
    [TRACE] Doing non-DB work for non_db_task_2...

    [INFO] Sleeping 5 seconds to allow flush...
    [INFO] Exiting.

Once the script executes you should be able to observe those traces on Elastic APM similar to the screenshots below.

Observe in Elastic APM

Go to Elastic Cloud, open your deployment, and navigate to Observability > APM.

Look for the app name in the service list (as defined by OTEL_RESOURCE_ATTRIBUTES).

Inside that service’s Traces tab, you’ll find multiple transactions like “transaction_1”,

“transaction_2”, etc.

Expanding each transaction shows child spans:

- Possibly db_insert_item and db_select_items if random DB path was taken.

- Otherwise, non_db_task_1 and non_db_task_2.

You can see how some transactions do DB calls, some do not, each with different spans.

This variety demonstrates how your real application might produce multiple different

“routes” or “operations.”

Service Map

If everything runs correctly, you should be able to view your services and see service maps for your application.

Services

My Elastic App

App Transactions

Dependencies

Logs

Navigate to your logs window/Discover to see the incoming application logs

Patterns

Log pattern analysis helps you to find patterns in unstructured log messages and makes it easier to examine your data.

Final Recap

Here is a quick summary of what we did:

• Provisioned an Ubuntu 22.04 machine.

• Installed build tools for SQLite, dev libs, and vcpkg.

• Installed the client for opentelemetry-cpp via vcpkg.

• Created a minimal C++ project that executes app traces and captures database operations.

• Connected database sqlite3 in CMakeLists.txt.

• Exported the Elastic OTLP endpoint & token as environment variables (with a lowercase authorization=Bearer key!).

• Ran the application and observed DB interactions and app traces in Elastic APM.

• Observed application logs and patterns on Elastic logs and Discover.

FAQ & Common Issues

• Getting “Could not find package configuration file provided by opentelemetry-cpp”?

Make sure you pass

-DCMAKE_TOOLCHAIN_FILE=... and -DCMAKE_PREFIX_PATH=... 

to cmake, or embed them in CMakeLists.txt.

• Crash: “validate_metadata: INTERNAL:Illegal header key”?

Use all-lowercase in

OTEL_EXPORTER_OTLP_HEADERS, e.g. authorization=Bearer \<token>.

• Missing otlp_grpc_metrics_exporter.h?

Your vcpkg version of opentelemetry-cpp (1.18.0) lacks a direct metrics exporter for OTLP. For metrics, either upgrade the library or consider an OpenTelemetry Collector approach.

• No data in Elastic APM?

Double-check your endpoint URL, Bearer token, firewall rules, or service name in the APM

Additional Resources:

• Sign up for Elastic Cloud free trial
• More Elastic OpenTelemetry Topics
• Introducing Elastic Distributions of OpenTelemetry
• Introducing Elastic Distribution of OpenTelemetry Collector
• Instrumenting your OpenAI- powered Python, Node.js, and Java Applications with EDOT

Elastic provides the ability to ingest OpenTelemetry data with native OTel SDKs, the off the shelf OTel collector, or even Elastic’s Distributions of OpenTelemetry (EDOT). EDOT enables you to bring in logs, metrics and traces for your GenAI application and for K8s. However you will also generally need libraries to help trace specific components in your application. In tracing GenAI applications you can pick from a large set of libraries.

• OpenTelemetry OpenAI Instrumentation-v2 - allows tracing LLM requests and logging of messages made by the OpenAI Python API library. (note v2 is built by OpenTelemetry, the non v2 version is from a specific vendor and not OpenTelemetry)

• OpenTelemetry VertexAI Instrumentation - allows tracing LLM requests and logging of messages made by the VertexAI Python API library

• Langtrace - commercially available library which supports all LLMs in one library, and all traces are also OTel native.

• Elastic’s EDOT - which recently added tracing. See blog.

As you can see OpenTelemetry is the defacto mechanism that is converging to collect and ingest. OpenTelemetry is growing its support for this but it is also early days.

In this blog, we will walk through how to, with minimal code, observe a RAG based chatbot application with tracing using Langtrace. We previously covered Langtrace in a blog to highlight tracing Langchain.

In this blog we used langtrace OpenAI, Amazon Bedrock, Cohere, and others in one library.

Pre-requisites:

In order to follow along, these few pre-requisites are needed

• An Elastic Cloud account — sign up now, and become familiar with Elastic’s OpenTelemetry configuration. With Serverless no version required. With regular cloud minimally 8.17

• Git clone the RAG based Chatbot application and go through the tutorial on how to bring it up and become more familiar.

• An account on your favorite LLM (OpenAI, AzureOpen AI, etc), with API keys

• Be familiar with EDOT to understand how we bring in logs, metrics, and traces from the application through the OTel Collector

• Kubernetes cluster - I’ll be using Amazon EKS

• Look at Langtrace documentation also.

Application OpenTelemetry output in Elastic

Chatbot-rag-app

The first item that you will need to get up and running is the ChatBotApp, and once up you should see the following:

As you select some of the questions you will set a response based on the index that was created in Elasticsearch when the app initializes. Additionally there will be queries that are made to LLMs.

Traces, logs, and metrics from EDOT in Elastic

Once you have OTel Collector with EDOT configuration on your K8s cluster, and Elastic Cloud up and running you should see the following:

Logs:

In Discover you will see logs from the Chatbotapp, and be able to analyze the application logs, any specific log patterns (saves you time in analysis), and view logs from K8s.