What is the Elastic Cloud Managed OTLP Endpoint?

Managed inputs are ingestion frontends that Elastic operates on your behalf. You point a shipper at an Elastic-owned endpoint, authenticate with an API key, and Elastic takes care of receiving, buffering, and indexing the data into your deployment. There is no gateway to size, patch, or monitor, and no backend credentials to distribute to edge agents.

The Managed OTLP Endpoint is the managed input for OpenTelemetry data. It accepts standard OTLP over HTTP and gRPC from any OpenTelemetry SDK or Collector distribution, buffers it durably, and indexes it into your Elasticsearch deployment using the OpenTelemetry data model. It is generally available on both Elastic Cloud Serverless and ECH. For an overview across logs, traces, and metrics, read Now GA: Managed OTLP Endpoint on Elastic Cloud Hosted.

For metrics, clients send OTLP/HTTP requests to the /v1/metrics signal path and authenticate with an Elasticsearch API key that has the event:write privilege for the apm application:

POST https://<managed-otlp-endpoint>/v1/metrics
Authorization: ApiKey <encoded-api-key>

The /v1/metrics client-facing path never changes. This post focuses on what happens after the request is accepted, during the last hop between the managed pipeline and Elasticsearch.

Two ways to ingest OpenTelemetry metrics: the bulk API and the native OTLP endpoint

Elasticsearch offers two ways to receive OpenTelemetry metrics.

The bulk API (/_bulk) is the general-purpose document ingestion API. To use it for OTLP metrics, something upstream must convert each OTLP data point into a JSON document that follows the OpenTelemetry mapping mode, group the documents by target data stream, serialize them as newline-delimited JSON, and send them in bulk requests. In the OpenTelemetry Collector ecosystem, this conversion is handled by the Elasticsearch exporter, and until recently, it was the only option.

The native OTLP endpoint (/_otlp/v1/metrics) accepts OTLP/HTTP Protobuf directly. Elasticsearch decodes the payload, builds the documents itself, and indexes them into time series data streams. Because Elasticsearch sees the original OTLP structure, it can perform these steps directly:

• It reads the aggregation temporality of every data point and stores it as a dimension
• It hashes the shared resource attributes once for all data points in a resource
• It maps exponential histograms to the exponential_histogram field type without an intermediate T-Digest conversion

The following diagram shows the two lanes.

The two paths differ in three ways that matter for users:

• Work distribution. On the bulk path, the managed pipeline converts and serializes documents, then Elasticsearch parses them again. On the native path, the pipeline forwards batched Protobuf and Elasticsearch builds documents once.
• Temporality. The Elasticsearch exporter used by the managed bulk path does not support cumulative histograms and drops those points. The native path preserves temporality and accepts both delta and cumulative histograms.
• Histogram fidelity. The bulk path converts every histogram to a T-Digest style representation. The native path stores exponential histograms natively and converts explicit-boundary histograms into the same exponential representation.

Do not confuse the two paths with the client-facing path. Clients always send to /v1/metrics on the Managed OTLP Endpoint. The service chooses between /_bulk and /_otlp/v1/metrics behind it.

Native OpenTelemetry metrics support by Elastic Stack version

Native OTLP metrics support in Elasticsearch arrived in several steps.

1. Elastic Stack 9.2 introduced the Elasticsearch OTLP/HTTP metrics endpoint as a technical preview.
2. Elastic Stack 9.3 added the exponential_histogram field type as a technical preview.
3. Elastic Stack 9.4 made exponential_histogram generally available and the default mapping for OTLP histograms received on the native endpoint.
4. Elastic Stack 9.5 added metric temporality support as a generally available feature. The native endpoint now stores the temporality of every data point, which is what makes cumulative histograms possible. The same release added OTLP/HTTP endpoints for logs and traces as a technical preview.
5. Elastic Stack 9.5.3 is the current minimum version for native OTLP metrics routing through the Managed OTLP Endpoint on ECH. The routing floor was raised from 9.2.0 because earlier versions can fail native OTLP requests when audit request-body logging processes Protobuf payloads. The 9.5.3 floor includes the required fixes.

The Elasticsearch Prometheus remote write endpoint arrived in Elastic Stack 9.4 and follows the same capability-gating model, but it is out of scope for this post.

Keep two things apart when reading version numbers in this post. The first four milestones are capabilities of Elasticsearch itself, available to anyone who calls /_otlp/v1/metrics directly. The fifth is a routing decision of the Managed OTLP Endpoint, which is stricter than "the endpoint exists" because the managed service has to work reliably for every tenant.

How Elastic Cloud Hosted routes OpenTelemetry metrics

On Serverless, Elastic operates and upgrades the backend, so the Managed OTLP Endpoint always uses the native path and users never see a version matrix.

ECH is different. Users pick and control their Elastic Stack version, and the Managed OTLP Endpoint supports deployments on any version from 9.0 onward. A single multi-tenant service therefore receives metrics for deployments that have no native endpoint at all, deployments that have one but predate the fixes the service relies on, and deployments that are fully ready. It cannot enable the native path unconditionally.

How the Managed OTLP Endpoint routes a metrics request

The following diagram shows what happens to a metrics request for an ECH deployment.

1. The endpoint authenticates the API key and resolves the target ECH deployment.
2. It looks up the deployment's Elasticsearch version from Elastic's own control plane, not from anything the client sends.
3. It derives a small set of capability flags for the target, such as "native OTLP metrics supported", and attaches them to the request as metadata.
4. The request is buffered durably together with its metadata.
5. At the indexing stage, the consumer reads the flags and hands the batch to one of two exporters: the OTLP/HTTP exporter, which forwards Protobuf to /_otlp/v1/metrics, or the Elasticsearch exporter, which converts the batch into documents and posts them to /_bulk.

Because the flags are derived inside Elastic's trust boundary, a client cannot opt a deployment into a path its version does not support. Elastic can also pin specific deployments to the bulk API for compatibility reasons, so the version rule below describes the normal case rather than a guarantee.

Which ingestion path does my deployment use?

| ECH Elastic Stack version | Path the Managed OTLP Endpoint normally uses | | ------------------------- | ------------------------------------------- | | 9.0 to 9.5.2 | Bulk API (/_bulk) | | 9.5.3 or later | Native OTLP endpoint (/_otlp/v1/metrics) |

No client-side change is required. If you already send OTLP metrics to the Managed OTLP Endpoint on ECH, your traffic moves to the native path when you upgrade the deployment to 9.5.3 or later.

What changes when OpenTelemetry metrics move to the native path

Client configuration and data stream routing remain unchanged. Histogram representation and temporality handling change: the native path uses exponential_histogram by default and records temporality as a dimension. Review dashboards, alerts, and queries that depend on the previous histogram representation or aggregate cumulative snapshots. Queries spanning historical histogram fields and newer exponential_histogram fields may need an explicit ::exponential_histogram cast, as described in querying historical data alongside new data.

The distribution of work across the ingestion chain also changes. On the bulk path, the managed pipeline converts every data point into a JSON document, serializes it, and Elasticsearch parses that JSON back into a document and hashes the dimensions of each one to compute its time-series identifier. On the native path, Elasticsearch decodes a compact Protobuf payload once, and because it sees the OTLP resource and scope structure, it can hash the resource attributes once and reuse that partial hash for every data point that shares the resource, instead of re-hashing the full dimension set per document. The pull request that introduced the endpoint names both effects, the binary encoding and the reused partial hashes, as the reasons the native endpoint is more efficient than bulk ingestion of the same data.

There are also capabilities you gain compared with the Elasticsearch exporter currently used by the managed bulk path. The native endpoint records the aggregation temporality of each data point, stores histograms in the exponential_histogram field type, and accepts cumulative histograms. These are differences between the two managed ingestion implementations, not inherent restrictions of the Bulk API. Custom bulk clients can index exponential histogram documents and configure a temporality dimension. The rest of this post is about those differences.

OpenTelemetry histogram types and temporality explained

The most visible difference between the two paths is what happens to histograms, so the rest of this post is a tour through every histogram case.

OpenTelemetry defines two histogram data types. Histogram uses explicit bucket boundaries chosen by the producer. ExponentialHistogram uses exponentially spaced buckets controlled by a scale parameter, so the SDK adapts resolution to the data.

Each type can use one of two temporalities. With delta temporality, every data point covers one collection interval. With cumulative temporality, every data point covers everything since a fixed start time.

That gives four combinations, and the two ingestion paths treat them differently.

| Path | Explicit, delta | Explicit, cumulative | Exponential, delta | Exponential, cumulative | | -------------------------------- | -------------------- | -------------------- | ------------------ | ----------------------- | | Bulk API (ECH 9.0 to 9.5.2) | Supported, converted | Dropped | Supported, converted | Dropped | | Native (ECH 9.5.3 or later) | Supported, converted | Supported, converted | Supported | Supported |

"Converted" means the metric is accepted but stored with a different bucket representation than the one on the wire. On the native path, that representation is the exponential_histogram field type, which is the default mapping for OTLP histograms since 9.4. Cumulative histograms on the native path require that default; if a cluster overrides xpack.otel_data.histogram_field_type to histogram, cumulative histograms are not supported. If Elastic has pinned a deployment to the bulk API for compatibility, the bulk row applies regardless of version.

Each of the four examples below is a complete curl request you can run against your own deployment. The JSON encoding of OTLP/HTTP is verbose but self-documenting, which makes it a good learning tool. Production shippers use Protobuf (application/x-protobuf) instead.

Prerequisites: your endpoint URL and API key

In the Elastic Cloud Console, find your deployment under Hosted deployments and select Manage. In Application endpoints, cluster and component IDs, select Managed OTLP and copy the public endpoint.

Then open the API keys management page in Kibana and create an API key with the event:write privilege for the apm application. Use the Encoded value of the key. It is already base64-encoded in the id:api_key format that the Authorization header expects.

export MANAGED_OTLP_URL="https://your-endpoint.elastic-cloud.com"
export ELASTIC_API_KEY="your-encoded-api-key"

# Nanosecond-precision timestamps: T0 = 2 min ago, T1 = 1 min ago, T2 = now
T0="$(python3 -c 'import time; print(time.time_ns() - 120_000_000_000)')"
T1="$(python3 -c 'import time; print(time.time_ns() -  60_000_000_000)')"
T2="$(python3 -c 'import time; print(time.time_ns())')"

All four examples report the same metric, http.server.request.duration in seconds, for the same service. Each has a distinct example.case resource attribute so the examples form separate time series and can be queried independently. The explicit and exponential examples describe different latency distributions; each cumulative example contains two snapshots whose difference matches its delta counterpart. Run the timestamp setup again before repeating the examples to avoid resending points with the same dimensions and timestamp.

Explicit histogram with delta temporality (case 1)

This is the case that works on every supported ECH version, so it is the right place to learn the shape of an OTLP histogram.

A histogram distributes individual measurements across buckets. For an explicit histogram, you choose the bucket boundaries in advance. For example: 5 ms, 10 ms, and 25 ms, and the SDK counts how many requests fell into each bucket. Each data point carries the bucket counts, the total count, and the sum, which is enough to estimate percentiles at query time.

Percentiles are what you want for latency. An average of 50 ms hides the fact that 1% of requests take 5 seconds, while a histogram shows p50, p95, and p99 side by side.

curl --fail-with-body --silent --show-error \
  -H "Authorization: ApiKey ${ELASTIC_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary '{
    "resourceMetrics": [
      {
        "resource": {
          "attributes": [
            {"key": "service.name", "value": {"stringValue": "my-service"}},
            {"key": "example.case", "value": {"stringValue": "explicit-delta"}}
          ]
        },
        "scopeMetrics": [
          {
            "scope": {"name": "http"},
            "metrics": [
              {
                "name": "http.server.request.duration",
                "unit": "s",
                "histogram": {
                  "aggregationTemporality": 1,
                  "dataPoints": [
                    {
                      "startTimeUnixNano": "'"${T1}"'",
                      "timeUnixNano":      "'"${T2}"'",
                      "count": 100,
                      "sum": 12.5,
                      "explicitBounds": [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
                      "bucketCounts":   [2,     5,    10,    20,   30,  18,   10,  4,   1,  0],
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }' \
  "${MANAGED_OTLP_URL}/v1/metrics"

aggregationTemporality: 1 means delta. The data point covers the one-minute window from startTimeUnixNano to timeUnixNano and nothing before it.

explicitBounds lists nine upper boundaries, so bucketCounts has ten entries. The last entry is the overflow bucket for values above 2.5 seconds.

A successful response is HTTP 200 with an empty or absent partialSuccess object, meaning the managed endpoint durably accepted the data for processing. It does not confirm that Elasticsearch indexed the points: unsupported cumulative histograms can still be dropped on the managed bulk path, and indexing can fail downstream. Use the queries below to verify ingestion. See indexing errors and the failure store for troubleshooting.

How Elastic stores it. Both paths accept this payload, and both convert it. On the bulk path, the Elasticsearch exporter turns the buckets into a T-Digest style histogram field. On the native path, Elasticsearch converts the explicit boundaries into an exponential_histogram. In both cases percentile precision is bounded by the boundaries you chose on the producer side, because the conversion cannot recover detail that the original buckets did not have.

Explicit histogram with cumulative temporality (case 2)

This case uses the same metric and bucket boundaries as case 1, with cumulative counts and a fixed start time. This is the case that separates the two paths most sharply.

Cumulative temporality is the default in most OpenTelemetry SDKs and the model Prometheus users know. Every data point reports the totals since the process started, so the counts only grow and a consumer computes the rate over a window by subtracting two points. It is resilient to lost data points, because the next point still carries the full state.

curl --fail-with-body --silent --show-error \
  -H "Authorization: ApiKey ${ELASTIC_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary '{
    "resourceMetrics": [
      {
        "resource": {
          "attributes": [
            {"key": "service.name", "value": {"stringValue": "my-service"}},
            {"key": "example.case", "value": {"stringValue": "explicit-cumulative"}}
          ]
        },
        "scopeMetrics": [
          {
            "scope": {"name": "http"},
            "metrics": [
              {
                "name": "http.server.request.duration",
                "unit": "s",
                "histogram": {
                  "aggregationTemporality": 2,
                  "dataPoints": [
                    {
                      "startTimeUnixNano": "'"${T0}"'",
                      "timeUnixNano":      "'"${T1}"'",
                      "count": 150,
                      "sum": 18.5,
                      "explicitBounds": [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
                      "bucketCounts":   [3,     7,    15,    30,   45,  27,   15,  6,   2,  0],
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    },
                    {
                      "startTimeUnixNano": "'"${T0}"'",
                      "timeUnixNano":      "'"${T2}"'",
                      "count": 250,
                      "sum": 31.0,
                      "explicitBounds": [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
                      "bucketCounts":   [5,     12,   25,    50,   75,  45,   25,  10,  3,  0],
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }' \
  "${MANAGED_OTLP_URL}/v1/metrics"

aggregationTemporality: 2 means cumulative. Both points retain T0 as the fixed start of the series. The first reports 150 requests through T1; the second reports 250 through T2. Subtracting their counts, sums, and corresponding buckets gives the 100-request distribution for T1 to T2 shown in case 1.

How Elastic stores it. On the native path Elasticsearch stores the point as an exponential_histogram and records cumulative in the temporality dimension. Time series functions in ES|QL use that dimension to compute per-interval rates from consecutive cumulative points.

On the bulk path this data point is dropped. The managed bulk path's Elasticsearch exporter does not support cumulative histograms; treating those buckets as delta would produce incorrect interval distributions. If your ECH deployment is on 9.0 to 9.5.2, configure your SDK or Collector to export histograms with delta temporality, or put a cumulativetodelta processor in front of the export.

Exponential histogram with delta temporality (case 3)

Now switch the data type while keeping delta temporality.

An exponential histogram removes the need to pick boundaries. Its buckets are exponentially spaced and controlled by a single scale parameter, and the SDK lowers the scale automatically when the data range grows. That is why exponential histograms are the recommended default for latency SLOs where p99 and p99.9 accuracy matters: the resolution is relative to the value, so the tail is captured as precisely as the body.

curl --fail-with-body --silent --show-error \
  -H "Authorization: ApiKey ${ELASTIC_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary '{
    "resourceMetrics": [
      {
        "resource": {
          "attributes": [
            {"key": "service.name", "value": {"stringValue": "my-service"}},
            {"key": "example.case", "value": {"stringValue": "exponential-delta"}}
          ]
        },
        "scopeMetrics": [
          {
            "scope": {"name": "http"},
            "metrics": [
              {
                "name": "http.server.request.duration",
                "unit": "s",
                "exponentialHistogram": {
                  "aggregationTemporality": 1,
                  "dataPoints": [
                    {
                      "startTimeUnixNano": "'"${T1}"'",
                      "timeUnixNano":      "'"${T2}"'",
                      "count": 100,
                      "sum": 62.2,
                      "scale": 3,
                      "zeroCount": 0,
                      "positive": {
                        "offset": -10,
                        "bucketCounts": [2, 5, 10, 20, 30, 18, 10, 4, 1]
                      },
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }' \
  "${MANAGED_OTLP_URL}/v1/metrics"

Reading the bucket layout takes a moment the first time.

The base of the histogram is 2^(2^-scale). With scale: 3 that is about 1.09, so each bucket is 9% wider than the previous one.

Bucket i in bucketCounts covers the range (base^(offset+i), base^(offset+i+1)], lower bound excluded and upper bound included. With offset: -10 the first bucket starts at about 0.42 seconds and the ninth ends at about 0.92 seconds.

zeroCount counts measurements that are exactly zero or within the zero threshold, and a negative bucket range exists for negative values, which latency never uses.

How Elastic stores it. On the native path Elasticsearch stores the scale, offset, and counts as an exponential_histogram without conversion. This is the highest-fidelity case: what you measured is what you query.

On the bulk path the Elasticsearch exporter converts the exponential buckets into a T-Digest style histogram. The data point is accepted and percentiles remain usable, but the original scale and bucket layout are not preserved.

Exponential histogram with cumulative temporality (case 4)

The last case combines the adaptive bucket layout of case 3 with the cumulative semantics of case 2. An SDK configured for exponential histogram aggregation and cumulative export temporality produces this shape. The OpenTelemetry SDK default aggregation for histogram instruments is explicit buckets, so selecting exponential aggregation is a separate configuration choice.

curl --fail-with-body --silent --show-error \
  -H "Authorization: ApiKey ${ELASTIC_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary '{
    "resourceMetrics": [
      {
        "resource": {
          "attributes": [
            {"key": "service.name", "value": {"stringValue": "my-service"}},
            {"key": "example.case", "value": {"stringValue": "exponential-cumulative"}}
          ]
        },
        "scopeMetrics": [
          {
            "scope": {"name": "http"},
            "metrics": [
              {
                "name": "http.server.request.duration",
                "unit": "s",
                "exponentialHistogram": {
                  "aggregationTemporality": 2,
                  "dataPoints": [
                    {
                      "startTimeUnixNano": "'"${T0}"'",
                      "timeUnixNano":      "'"${T1}"'",
                      "count": 150,
                      "sum": 93.3,
                      "scale": 3,
                      "zeroCount": 0,
                      "positive": {
                        "offset": -10,
                        "bucketCounts": [3, 7, 15, 30, 45, 27, 15, 6, 2]
                      },
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    },
                    {
                      "startTimeUnixNano": "'"${T0}"'",
                      "timeUnixNano":      "'"${T2}"'",
                      "count": 250,
                      "sum": 155.5,
                      "scale": 3,
                      "zeroCount": 0,
                      "positive": {
                        "offset": -10,
                        "bucketCounts": [5, 12, 25, 50, 75, 45, 25, 10, 3]
                      },
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }' \
  "${MANAGED_OTLP_URL}/v1/metrics"

How Elastic stores it. On the native path this is stored exactly like case 3, as an exponential_histogram, plus the cumulative temporality dimension. Elasticsearch can then downsample and compute rates over the series correctly. The two points share the same start time and bucket layout; their difference is the delta distribution in case 3.

On the bulk path the data point is dropped, for the same reason as case 2.

Requires Elastic Stack 9.5.3 or later on ECH. On ECH 9.0 to 9.5.2, cases 2 and 4 are dropped. Emit delta histograms until you upgrade.

Query OpenTelemetry histogram percentiles with ES|QL

Sending is only half of the showcase. Open Discover in Kibana, switch to ES|QL mode, and first confirm that the documents arrived:

FROM metrics-*.otel-*
| EVAL duration = metrics.http.server.request.duration::exponential_histogram
| WHERE service.name == "my-service" AND duration IS NOT NULL
| KEEP @timestamp, resource.attributes.example.case, metrics.http.server.request.duration, attributes.http.request.method
| SORT @timestamp DESC
| LIMIT 10

On a native-path deployment the histogram field shows the scale, the bucket indices, and the counts you sent. On a bulk-path deployment you see the converted values and counts arrays instead, which is the visible trace of the conversion.

Then ask the question the histogram was collected for, the request latency percentiles:

TS metrics-*.otel-*
| WHERE service.name == "my-service" AND @timestamp >= NOW() - 5 minutes
| STATS p50 = PERCENTILE(metrics.http.server.request.duration::exponential_histogram, 50),
        p95 = PERCENTILE(metrics.http.server.request.duration::exponential_histogram, 95),
        p99 = PERCENTILE(metrics.http.server.request.duration::exponential_histogram, 99)
    BY resource.attributes.example.case, attributes.http.request.method, TBUCKET(5 minutes)

TS merges histograms per time series and respects their temporality before computing percentiles. For cumulative series, it accounts for the change between consecutive snapshots instead of counting their overlapping observations repeatedly. FROM is useful for inspecting individual documents, but it ignores temporality when aggregating them. See how temporality affects queries.

Grouping by resource.attributes.example.case keeps the four examples separate. The explicit examples place the median in the 0.05 to 0.1 second bucket; the exponential examples place it around 0.6 seconds. These are different input distributions, so their percentile results should differ. The cumulative examples describe the same final one-minute distribution as their corresponding delta examples. Refer to the ES|QL aggregation functions reference for the full list of functions that accept exponential histogram fields.

Summary: OpenTelemetry histogram support on Elastic Cloud Hosted

• ECH deployments on Elastic Stack 9.5.3 or later receive OTLP metrics through the native Elasticsearch endpoint, with no client change.
• Those deployments accept all four histogram cases, and exponential histograms are stored without conversion.
• ECH deployments on 9.0 to 9.5.2 continue through the bulk API and should emit delta histograms.
• Serverless deployments always use the native path.

Clients keep sending to the same /v1/metrics path on the Managed OTLP Endpoint in every case.

Learn more about OpenTelemetry metrics on Elastic

• Elastic Cloud Managed OTLP Endpoint documentation
• Elasticsearch OTLP/HTTP endpoint documentation
• Exponential histogram field type
• Now GA: Managed OTLP Endpoint on Elastic Cloud Hosted

Both features follow the OTel GenAI semantic conventions and work with any OTel-instrumented provider. If your framework already emits OTel GenAI span attributes, there is nothing to change.

How OTel GenAI spans are structured

A GenAI span stores everything as span attributes. A typical chat span includes:

• gen_ai.provider.name: the provider (openai, anthropic, aws.bedrock, etc.); gen_ai.system is supported as a fallback for older instrumentation.
• gen_ai.operation.name: the operation type (chat, embeddings, etc.).
• gen_ai.request.model: the model being called.
• gen_ai.usage.input_tokens: tokens consumed by the prompt.
• gen_ai.usage.output_tokens: tokens generated in the response.
• gen_ai.input.messages, gen_ai.output.messages: conversation messages.
• gen_ai.system_instructions: the system prompt.

Both features read from these attributes:

| Feature | What it shows | Where it appears | Attributes it reads | | --- | --- | --- | --- | | GenAI tab | Details (operation type, request model, provider, input and output token counts, response model, response ID) and Conversation (system prompt, user messages, model response) | Span flyout in the APM trace view, and the span flyout in Discover | Appears with any gen_ai.* attribute. Conversation needs gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages | | Token count badges | Input and output token counts for each GenAI span | Every GenAI span row in the trace waterfall | gen_ai.usage.input_tokens, gen_ai.usage.output_tokens |

How to read LLM prompts and responses in the GenAI tab

When any gen_ai.* attribute is present on a span, the span flyout shows a dedicated GenAI tab next to Metadata.

The Details section shows model metadata from the span attributes: operation type, request model, provider, input and output token counts, response model, and response ID. The Conversation section shows the full exchange, populated from gen_ai.system_instructions (system prompt), gen_ai.input.messages (user messages), and gen_ai.output.messages (model response), each with a copy button so you can pull the exact prompt or response out of the trace without scraping text from a formatted table.

All raw span attributes remain accessible on the Metadata tab.

The GenAI tab is also available in the span flyout in Discover, so you can inspect LLM prompts and responses directly alongside your log and trace data without switching to the APM view.

What instrumentation does LLM tracing require?

No Kibana-side configuration is needed. The GenAI tab appears automatically when any gen_ai.* attribute is present on a span. Full Conversation support requires the OTel GenAI v1.37.0 span-attribute model: gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions.

Frameworks that emit the older span-events model (gen_ai.user.message, gen_ai.assistant.message, gen_ai.choice) will show the Details metadata section but will not populate the Conversation section. For a current list of compatible instrumentations, see the OTel GenAI semantic conventions.

To verify, open the span in Discover, check that gen_ai.input.messages and gen_ai.output.messages are present, and confirm the Conversation section renders.

If your application already sends APM data to Elastic from a GenAI workload, open any GenAI span in the trace view and check for the GenAI tab.

LLM token usage in the trace waterfall

Token count badges now appear on each GenAI span row in the waterfall, so you can scan the full trace without drilling in. In agentic traces with ten or twenty LLM calls, this lets you identify which span is driving token consumption before opening any span.

Each row shows input and output token count badges sourced from gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. The row label is the span name, which instrumentation frameworks typically set to something like chat gpt-4o-mini.

Waterfall-level token counts are most useful in agentic traces where a chain of LLM calls uses different models or the same model with varying context sizes.

Which LLM providers does Elastic APM support for GenAI tracing?

Elastic APM's LLM tracing works with any OTel-instrumented provider: the GenAI tab and waterfall token counts use the same OTel attribute schema regardless of which provider your application uses. Provider is read from gen_ai.provider.name, falling back to gen_ai.system for older instrumentation.

The OTel GenAI semantic conventions that enable this provider detection are currently in a Development lifecycle. Check the release notes before upgrading instrumentation.

How to enable LLM tracing in Elastic APM

Availability: Both features are available as a Technical Preview on Elastic Serverless and will be available as a Technical Preview in Elastic Stack 9.6.

To try these features:

1. Instrument your GenAI application with an OTel SDK that follows the OTel GenAI semantic conventions (v1.37.0 or later for full Conversation support).
2. Send traces to Elastic Observability using OTLP, the Elastic APM agent, or an EDOT SDK.
3. Open the APM section in Kibana, navigate to a service that makes LLM calls, and open the trace waterfall for any transaction.

The GenAI tab appears on any span with at least one gen_ai.* attribute set; token count badges appear when gen_ai.usage.input_tokens or gen_ai.usage.output_tokens are present.

If you don't have a GenAI application to test with, the otel-genai-chat-app repository is a minimal OpenAI chat app pre-instrumented with EDOT. Set OPENAI_API_KEY and follow the EDOT commands in the README to send traces to Elastic and see both features in action.

What's next for LLM observability in Elastic APM

We're exploring cost estimation per span (estimated spend based on model pricing and token counts, surfaced in the waterfall) and tool call rendering (structured display of tool/function call inputs and outputs for agentic spans).

If you are building GenAI applications and want early access or to share feedback, reach out through the Elastic community forums or open an issue in the kibana repository.

Forking the agent would fix this, but then you own every upstream merge. You could also use manual instrumentation for complete control, but that requires code changes and ongoing upkeep. OpenTelemetry Java extensions give you a cleaner path: a separate JAR the agent loads at startup, giving you precise control over what gets captured and exported, without touching agent or application code.

For example, the following challenges are very common:

• Health check probes are flooding your trace view.
• Span names reflect generic framework patterns rather than your domain operations.
• Some span names or attributes have high cardinality creating noise in your traces.
• Spans are missing attributes relevant to your business logic.
• Baggage headers are propagating to downstream services when they shouldn't.
• Resource attributes that describe your deployment are not automatically captured because they rely on custom environment variables.

Some of those can be solved through configuration, or by using an intermediate OpenTelemetry Collector for processing. However, this also might add complexity to the telemetry pipeline, and you might prefer to solve this at the source, where the data is captured.

What are OpenTelemetry Java extensions

An extension is a JAR file the agent loads at startup. It hooks into the agent's extension points through Java's Service Provider Interface (SPI) mechanism, the same mechanism the agent uses internally.

The extension mechanism works identically with the upstream OpenTelemetry Java agent and with Elastic's OpenTelemetry distribution. You write the extension once and it works with either.

For reference, the upstream extension documentation provides an exhaustive overview of extension points and a few examples.

This post does not aim to provide a complete reference, but focuses on simple use cases you're likely to reach for in production: renaming spans, filtering noisy traces, or propagating context that the agent doesn't cover in your environment.

Extensions also let you modify and extend the agent instrumentation itself. That goes beyond what this post covers. Here are two starting points:

• Modify instrumentation using instrumenter customizers.
• Modify or add instrumentation using the instrumentation module.

Setting up an OpenTelemetry Java extension project

An extension is a standard Java Gradle project with two requirements: the output must be a shadow JAR (a fat JAR with all extension dependencies bundled), and OpenTelemetry dependencies must be declared compileOnly so you don't bundle the SDK itself.

The shadow JAR requirement exists because the agent loads the extension in its own classloader. If you declare a dependency as implementation, it gets bundled and may conflict with the version already in the agent. Using compileOnly keeps those JARs out of the extension JAR entirely.

Here is a minimal build.gradle.kts for a simple extension that does not customize instrumentation and thus relies only on the OpenTelemetry SDK/API.

plugins {
  id("java")
  id("com.gradleup.shadow")
}

repositories {
  mavenCentral()
}

java {
  toolchain {
    languageVersion.set(JavaLanguageVersion.of(8))
  }
}

dependencies {
  // Use BOM to manage OpenTelemetry dependency versions
  compileOnly(platform("io.opentelemetry:opentelemetry-bom:1.64.0"))
  // OpenTelemetry SDK autoconfiguration SPI (provided by agent)
  compileOnly("io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi")
  // OpenTelemetry SDK
  compileOnly("io.opentelemetry:opentelemetry-sdk")
  // Annotation processor for automatic SPI registration
  compileOnly("com.google.auto.service:auto-service:1.1.1")
  annotationProcessor("com.google.auto.service:auto-service:1.1.1")
}

tasks.assemble {
  dependsOn(tasks.shadowJar)
}

Check Maven Central for the latest version of the BOM before starting.

Extensions only depend at compile-time on the OpenTelemetry SDK and the autoconfiguration SPI. The agent provides the rest of the SDK and instrumentation implementation at runtime.

Loading OpenTelemetry Java extensions at runtime

To load an OpenTelemetry Java extension at runtime, you can use the otel.javaagent.extensions system property or OTEL_JAVAAGENT_EXTENSIONS environment variable. The value is a comma-separated list of paths to extension JARs:

java -Dotel.javaagent.extensions=/path/to/my-extension.jar -javaagent:/path/to/opentelemetry-javaagent.jar -jar myapp.jar

The upstream OpenTelemetry Java agent also lets you embed extensions directly into the agent JAR to simplify deployment.

Filtering and renaming spans with OpenTelemetry Java extensions

You can modify spans in two ways:

• Using a SpanProcessor that is called synchronously when the span starts or ends.
• Using a SpanExporter that is called asynchronously when the span is exported.

Rename spans with a SpanProcessor

SpanProcessor.onStart receives a ReadWriteSpan, which means you can call span.updateName() before the span is exported. This is the right hook for renaming based on attributes that are available at span start.

public class OperationRenamingSpanProcessor implements SpanProcessor {

  @Override
  public void onStart(Context parentContext, ReadWriteSpan span) {
    String operation = span.getAttribute(AttributeKey.stringKey("app.operation"));
    if (operation != null) {
      span.updateName(operation);
    }
  }

  @Override
  public boolean isStartRequired() { return true; }

  @Override
  public void onEnd(ReadableSpan span) {}

  @Override
  public boolean isEndRequired() { return false; }

  @Override
  public CompletableResultCode shutdown() { return CompletableResultCode.ofSuccess(); }

  @Override
  public CompletableResultCode forceFlush() { return CompletableResultCode.ofSuccess(); }
}

Register the SpanProcessor via AutoConfigurationCustomizerProvider, composing it with whatever processor you have already configured:

@AutoService(AutoConfigurationCustomizerProvider.class)
public class RenamingCustomizerProvider implements AutoConfigurationCustomizerProvider {

  @Override
  public void customize(AutoConfigurationCustomizer customizer) {
    customizer.addTracerProviderCustomizer(this::configureSdkTracerProvider);
  }

  private SdkTracerProviderBuilder configureSdkTracerProvider(
      SdkTracerProviderBuilder tracerProvider, ConfigProperties config) {
    return tracerProvider.addSpanProcessor(new OperationRenamingSpanProcessor());
  }

}

The modify-span EDOT Java extension example provides a complete implementation.

Filter spans with a SpanExporter

A SpanExporter wrapper lets you modify or drop spans before they leave the process. This works well for known noisy endpoints like health checks.

public class FilteringSpanExporter implements SpanExporter {

  private final SpanExporter delegate;

  public FilteringSpanExporter(SpanExporter delegate) {
    this.delegate = delegate;
  }

  @Override
  public CompletableResultCode export(Collection<SpanData> spans) {
    List<SpanData> filtered = new ArrayList<>();
    for (SpanData span : spans) {
      if (!"GET /health".equals(span.getName())) {
        filtered.add(span);
      }
    }
    return delegate.export(filtered);
  }

  @Override
  public CompletableResultCode flush() { return delegate.flush(); }

  @Override
  public CompletableResultCode shutdown() { return delegate.shutdown(); }
}

Register the FilteringSpanExporter via addSpanExporterCustomizer:

customizer.addSpanExporterCustomizer((existing, config) -> new FilteringSpanExporter(existing));

The modify-span EDOT Java extension example provides a complete implementation.

The approach has two limitations:

• This won't discard any child span that may have been created, for example, if the healthcheck calls the database.
• Spans filtered at the exporter have already passed through the full processor pipeline and occupied buffer space in the batch processor.

If you're dropping a large fraction of your traffic at this stage, a custom Sampler (shown below) is more efficient because it drops spans before any processing happens and also filters out child spans. Also, when using declarative configuration, the rule-based sampler lets you implement filtering on rules using only configuration.

Adding custom resource attributes with a ResourceProvider

Resource attributes describe what's running: the service name, its version, the host. A ResourceProvider lets you attach additional attributes that the agent doesn't know about, such as deployment metadata your platform injects through environment variables.

The example below uses environment variables, but it could also be a configuration file, a cloud metadata service, or any other source available to the agent at startup.

Because the SDK initialization is synchronous, when querying an external service like a metadata endpoint, this can make the agent (and thus the application) startup slower. If possible, prefer checking environment variables and local config first before calling an external service.

@AutoService(ResourceProvider.class)
public class DeploymentResourceProvider implements ResourceProvider {

  @Override
  public Resource createResource(ConfigProperties config) {
    AttributesBuilder attributes = Attributes.builder();

    String region = System.getenv("DEPLOY_REGION");
    if (region != null) {
      attributes.put(AttributeKey.stringKey("deployment.region"), region);
    }

    String buildVersion = System.getenv("BUILD_VERSION");
    if (buildVersion != null) {
      attributes.put(AttributeKey.stringKey("build.version"), buildVersion);
    }

    return Resource.create(attributes.build());
  }
}

Attributes from a ResourceProvider merge with the agent's own resource. When two providers supply the same key, the one with the higher order() value wins. The agent's built-in providers use order 0, so overriding order() to return a positive integer gives your provider priority.

The resource-attribute EDOT Java extension example provides a complete implementation.

Custom sampling in OpenTelemetry Java

When filtering at the exporter is too late or too expensive, implement a Sampler directly. The sampler runs before any span processing, so dropped spans never touch the batch buffer.

However, the sampling decision can only rely on attributes that are provided when the span starts. For example, the status code of an HTTP response can't be used as it is only available when the span ends.

The key detail: wrap the existing sampler rather than replacing it. That way, your logic composes with whatever you configured, and parent-based decisions from an upstream service are still respected.

public class HealthCheckSampler implements Sampler {

  private final Sampler delegate;

  public HealthCheckSampler(Sampler delegate) {
    this.delegate = delegate;
  }

  @Override
  public SamplingResult shouldSample(
      Context parentContext,
      String traceId,
      String name,
      SpanKind spanKind,
      Attributes attributes,
      List<LinkData> parentLinks) {
    if (spanKind == SpanKind.SERVER && name.contains("health")) {
      return SamplingResult.create(SamplingDecision.DROP);
    }
    return delegate.shouldSample(parentContext, traceId, name, spanKind, attributes, parentLinks);
  }

  @Override
  public String getDescription() {
    return "HealthCheckSampler{" + delegate.getDescription() + "}";
  }
}

Register the HealthCheckSampler via addSamplerCustomizer, which gives you both the existing sampler and the resolved config:

customizer.addSamplerCustomizer((existing, config) -> new HealthCheckSampler(existing));

Community extensions in opentelemetry-java-contrib

The opentelemetry-java-contrib repository contains several community-maintained extensions.

Some of them are already included in the OpenTelemetry Java agent (and inherited in the Elastic distribution), but are opt-in:

• azure-resources
• aws-resources
• gcp-resources
• cloudfoundry-resources
• baggage-processor

Most Elastic distribution features exist as extensions in the contrib repository, so you can use them with the upstream agent in a vendor-neutral way.

• inferred-spans
• span-stacktrace

Further reading and extension examples

The upstream extension examples cover additional extension points not shown here, including custom propagators, ID generators, and ignored-type configurers.

The Elastic baggage example shows the filtering propagator for baggage running end-to-end with a two-service application, it also demonstrates custom instrumentation to add baggage without modifying the application code.

This post covered the project setup and the patterns most likely to come up in production. Both links above go deeper: the upstream examples add extension points not covered here, and the baggage example shows a complete two-service implementation you can run locally.

What changed in Elastic Agent 9.3: A native OTel Collector under the hood

Previously, Elastic Agent acted as a supervisor process, spinning up Beats-based subprocesses, such as Filebeat or Metricbeat. From 9.3 onward, that architecture has been replaced. Elastic Agent itself is now built on the EDOT Collector, turning it into a first-class OTel Collector under the hood while preserving its original functionality.

Key benefits of this architectural shift include:

• Reduced footprint: Fewer subprocesses mean significantly less memory overhead and a simpler deployment model. In future releases, this footprint will be even further reduced.
• Unified telemetry pipeline: Logs and metrics flow through a single, standards-based OTel pipeline, as do traces.
• Ecosystem interoperability: Elastic Agent can now receive data from any OTel-compatible source out of the box. It can also be configured to emit to OTel-compatible destinations.
• Aligned with the OTel ecosystem: As the OTel ecosystem matures with new receivers, processors, and exporters, Elastic Agent deployments gain access to those capabilities automatically.

When you deploy or update Elastic Agent from version 9.3 onward, you're deploying an OpenTelemetry Collector. EDOT is the technology foundation; Elastic Agent is the product.

How existing Beats configurations run inside the OTel Collector pipeline

Elastic has introduced Beats Receivers, which are Beat inputs and processors that execute natively inside the new OTel Collector pipeline. For your teams and customers, this means:

• Existing elastic-agent.yml configurations require no modification.
• Fleet-managed agents automatically translate policy configurations into OTel format internally.
• All integrations, dashboards, ingest pipelines, and alerting rules continue to function exactly as before.
• Data written via Beats Receivers lands in the same data streams as always.

Upgrading to 9.3 is transparent because it uses the same inputs and produces the same outputs.

Running Beats and OTel Collector pipelines in one Elastic Agent

The new Elastic Agent is a collector capable of simultaneously running traditional Beats-based collections alongside native OTel pipelines, all in a single deployment. One agent policy can collect Elastic Common Schema–schematized (ECS-schematized) data via Beats Receivers and ingest native OpenTelemetry Protocol (OTLP) data from OTel-instrumented applications and infrastructure. This same agent policy can apply OTel processing stages across all telemetry before export.

OTel integrations from Elastic's catalog can be added to any agent policy. When native OTel data is ingested, Elastic automatically installs the relevant dashboards and alerts, in addition to necessary content packs, without any manual setup.

What's the relationship between Elastic Agent and EDOT?

You may be familiar with EDOT, the Elastic Distribution of OpenTelemetry Collector, as a stand-alone product. With this architectural change, EDOT is the technology foundation that now powers Elastic Agent, not a separate product that users need to track or deploy independently.

Going forward, Elastic Agent is the supported, Fleet-manageable, fully featured product. A stand-alone deployment remains available for specific niche scenarios (environments where the full version of Elastic Agent cannot be installed), but it isn't the recommended path for the vast majority of users.

Elastic Agent deployment options: Fleet-managed vs. stand-alone

| | Fleet-managed Elastic Agent | Stand-alone Elastic Agent | | :------------------------- | :-----------------------------: | :-----------------------------------------------------------------------------------------------------: | | Fleet lifecycle management | Yes | Can enroll into Fleet in-field without reinstallation | | Beats Receivers | Yes | Yes | | Elastic Defend | Yes | No | | Cloud Security | Yes | No | | Profiler support | Yes | No | | OTel-native pipeline | Yes | Yes | | Best for | Most deployments | Environments where full Elastic Agent cannot be installed or management is handled by other tools |

Do I need to change anything when upgrading to Elastic Agent 9.3?

For users running Elastic Agent today, upgrading to 9.3 requires no changes to configurations or integrations, and no changes to workflows. For customers evaluating OTel adoption, Elastic Agent now provides a fully supported, production-ready OTel Collector with Fleet management and rich integrations, along with Elastic's full support matrix, and none of this requires a separate OTel deployment.

With Elastic Agent 9.3, Elastic's data collection is fully OpenTelemetry-native. Elastic Agent is now an OpenTelemetry Collector. Everything you have today still works, and you also get all the capabilities of OTel.

The gain is in where the intermediate set lives. When you ask what the saturated hosts logged, the list of saturated hosts is computed and consumed inside Elasticsearch. Six host names or 500 trace IDs never land in a clipboard or in an AI agent's context window, and the set is recomputed from current data every time the query runs.

That changes the unit of investigation. You go from "one slow request stalled on a lock" to "most of the slowest requests did," and only the second answer tells you which team to page. It matters more the further apart your signals are: in many observability stacks, logs, metrics, and traces live in three separate systems, each with its own query language, its own time picker, and its own idea of what a host is, so the same question has to be asked two or three times and the answers joined by hand.

In this post we walk through four investigations, each of which is self-contained. For every one of them, we set out the scenario that started it, the query that answers it, the table it returns, and a note on the difficulties you run into when you try to get the same answer any other way.

| Pattern | Starts from | Question it answers | |---|---|---| | Metrics to logs | CPU saturation | Which error patterns show up only on the saturated hosts? | | Logs to metrics | Error logs | Do the erroring hosts look any different from the healthy ones? | | Traces to logs | Slow spans | What did every service log during those specific requests? | | All three signals | Pod memory pressure | Which log lines sit behind the requests that failed under that pressure? |

The data was collected with OpenTelemetry and lands in the logs-*.otel-*, traces-*.otel-*, and metrics-*.otel-* data streams, where fields keep their semantic convention names rather than being rewritten into another schema. The correlation pattern works just as well on Elastic Agent integrations, though the queries need translating rather than just renaming: ECS carries log severity as the text field log.level instead of a numeric severity_number, the System integration reports CPU as separate system.cpu.*.pct fields instead of one metric with a state dimension, and APM records durations in microseconds. All of it lands in the same cluster either way, which is the part the subquery depends on.

In Discover, the time picker already applies the range, so the examples below omit an explicit @timestamp filter. Outside Discover, add a filter by time yourself, either with literal timestamps in the query or with ?_tstart and ?_tend in the query and values in the params array of your _query request. Every result below comes from a one hour window over a synthetic fleet of 300 hosts.

Metrics to logs: what are the saturated hosts complaining about?

An infrastructure alert tells you a handful of hosts in a fleet of a few hundred sat above 90% CPU over the last hour. That tells you which hosts are hot and nothing about why. The question worth answering is whether those hosts share a failure mode, or whether they are busy for unrelated reasons and the alert is a coincidence.

FROM logs-*.otel-*
| WHERE severity_number >= 17
  AND resource.attributes.host.name IN (
      TS metrics-hostmetrics.otel-*
      | WHERE attributes.state == "idle"
      | STATS idle = AVG(AVG_OVER_TIME(metrics.system.cpu.utilization))
          BY resource.attributes.host.name
      | WHERE idle < 0.1
      | KEEP resource.attributes.host.name
    )
| STATS errors = COUNT(*), hosts = COUNT_DISTINCT(resource.attributes.host.name)
    BY pattern = CATEGORIZE(body.text), service = resource.attributes.service.name
| SORT errors DESC

The two halves of the query map onto the two halves of the question. The subquery works out which hosts were saturated, averaging CPU utilization per host and keeping the ones that averaged under 10% idle, which is another way of saying above 90% busy for the window. The outer query then works out what those hosts were complaining about, pulling their error logs and using CATEGORIZE to collapse thousands of individual lines into a handful of error classes.

Two choices in there are worth pausing on.

The subquery uses TS rather than FROM because a host does not report one CPU number. It reports a separate time series per CPU state, and per logical core as well if your collector is configured to break them out, so the reduction has to happen in two stages. AVG_OVER_TIME collapses each series to a single value first, and the outer AVG then combines those into one figure per host.

Naming that inner function matters more than it looks. Write AVG(metrics.system.cpu.utilization) on its own and TS supplies LAST_OVER_TIME for you, averaging each series' final sample rather than its average over the window. In this dataset that one substitution moves a host from 7% idle to 10% idle, which is the difference between appearing in the results and not. Querying metrics with the TS command goes into the two aggregation phases in more depth.

The log filter tests severity_number (17 is the ERROR floor on the OpenTelemetry scale) rather than the severity text, because the numeric scale is fixed by the spec while the text is whatever the emitting library decided to write. That is not a hypothetical distinction here: the error logs in this cluster carry four different labels, including SEVERE from a Java service. Matching on the text alone returns 2,754 of checkout's errors and misses the billing service entirely, while the numeric filter returns all 5,910 of them and keeps billing too.

When you run the query in Discover, the result is a short table of error classes, scoped to the saturated hosts:

The subquery returned six saturated hosts, and on all six the same service is timing out against an upstream dependency and draining its connection pool. The hosts column is what lets you set the other two rows aside without opening anything: the certificate errors reach only two of the six, and the gateway declines amount to five lines in an hour. Neither tracks the cohort the way the checkout patterns do.

Having all three signals in one store already removed the exports from this investigation. The subquery removes the step after that, and closing that last gap matters more than it sounds. By the time you have read six host names off a chart and typed them into a log search, the set has moved: a host that crossed the threshold a minute ago is missing from your list, and one that has since recovered is still in it. Here the host list is derived from current data on every run, so re-running the query during an incident gives you the current cohort.

The stale list is only half of it. A correlation done by hand exists only in the head of the person who did it, so nobody else can check it, save it, or run it again tomorrow.

Logs to metrics: do the erroring hosts look different from the healthy ones?

Filtering metrics by a log-derived host set answers the opposite question. The payments service is throwing errors on some hosts and not others, and you want to know whether resource pressure explains the split before you start reading deploy history.

That is a comparison, so the query needs both cohorts. FORK runs two branches over the same input, and IN and NOT IN against the same log-derived host set divide the fleet between them.

TS metrics-hostmetrics.otel-*
| WHERE attributes.state == "idle"
| STATS idle = AVG(AVG_OVER_TIME(metrics.system.cpu.utilization))
    BY host = resource.attributes.host.name
| EVAL busy = 1 - idle
| FORK
    ( WHERE host IN (
        FROM logs-*.otel-*
        | WHERE severity_number >= 17
          AND resource.attributes.service.name == "payments"
          AND resource.attributes.host.name IS NOT NULL
        | STATS errors = COUNT(*) BY resource.attributes.host.name
        | KEEP resource.attributes.host.name )
      | EVAL cohort = "logging errors" )
    ( WHERE host NOT IN (
        FROM logs-*.otel-*
        | WHERE severity_number >= 17
          AND resource.attributes.service.name == "payments"
          AND resource.attributes.host.name IS NOT NULL
        | STATS errors = COUNT(*) BY resource.attributes.host.name
        | KEEP resource.attributes.host.name )
      | EVAL cohort = "no errors" )
| STATS hosts = COUNT(*), mean_busy = AVG(busy), busiest_host = MAX(busy)
    BY cohort

The query reads top to bottom as three stages. The metrics query runs first and reduces the whole fleet to one busy figure per host. FORK then splits that fleet in two using the same log query in both branches, separating the hosts that appear in it from the hosts that do not. The final STATS summarizes each group, so both cohorts come back as two rows of one table, measured the same way over the same window.

This query is longer than the others, and three parts of it are less obvious than they look.

The natural way to label the two cohorts would be EVAL cohort = CASE(host IN (...), "erroring", "healthy"), and ES|QL rejects it. In 9.5 an IN subquery has to be a top-level predicate in a WHERE condition rather than an argument to a scalar function, which is why the split happens at the command level with FORK.

The STATS ... BY inside each subquery looks redundant, since KEEP alone would return the same host names. It is not: without it, the subquery returns one row per matching log document instead of one row per host, and those rows are all held in memory for the outer query to filter against. Aggregating first turns millions of rows into a few hundred host names.

The IS NOT NULL filter guards the sharpest edge here, and this dataset is a live example rather than a hypothetical. NOT IN follows SQL null semantics, so a single null in the subquery result makes the predicate match nothing at all. Five of the payments error logs in this cluster came through a sidecar that dropped the host name. Remove that one line from both branches and the query still succeeds, but it returns a single row: the nulls quietly delete the entire 286-host "no errors" cohort, and what is left looks like a perfectly plausible answer to a different question.

Run the query in Discover and you get two rows, one per cohort:

CPU does not explain the split, and the numbers say so twice. The 14 erroring hosts run slightly cooler on average than the 286 quiet ones, and the busiest machine among them averaged 50% over the hour while the quiet cohort contains a host that averaged 95%. Whatever is failing on those 14, they had headroom the entire time, and deploy history is a better place to spend the next ten minutes.

A negative result like this is worth as much as a positive one, and it is usually the one people skip. Getting it the long way means running the metrics query twice against two hand-built host lists and lining the numbers up afterwards, which is enough friction that the check often just does not happen. Both cohorts here come from the same log query in the same execution, over identical time windows, so there is nothing to reconcile and no reason not to check.

Traces to logs: what did every service log during the slow requests?

A service level objective (SLO) burn alert fires on checkout latency. Tracing gives you the slow requests and their spans, and the next question is what the services involved were writing to their logs while those specific requests were in flight.

The trace ID is the join key, and there are far too many of them to move by hand.

FROM logs-*.otel-*
| WHERE trace_id IN (
      FROM traces-*.otel-*
      | WHERE kind == "Server"
        AND resource.attributes.service.name == "checkout"
        AND name == "POST /api/orders"
        AND duration > 2000000000
      | SORT duration DESC
      | LIMIT 500
      | KEEP trace_id
    )
| STATS lines = COUNT(*), traces = COUNT_DISTINCT(trace_id)
    BY pattern = CATEGORIZE(body.text),
       service = resource.attributes.service.name,
       severity_text
| SORT traces DESC

The subquery answers "which requests were slow." It looks at the inbound request span for the checkout endpoint rather than the client and internal spans beneath it, then keeps the 500 slowest requests over two seconds. Durations are recorded in nanoseconds, which is why the threshold has so many zeros.

The outer query answers "what got logged while they were running," gathering every log line that shares one of those trace IDs and grouping them into patterns.

Counting distinct traces per pattern is what makes the output readable. A log pattern that appears 30,000 times across four traces is one chatty request, while a pattern that shows up in 470 of the 500 slowest traces is a property of being slow.

From the results above, the top row is there by construction and can be set aside: checkout writes one order submitted line per order, so it appears in all 500 traces and says nothing about why these particular 500 were slow.

From the results above, the third row is the answer, and it points at a service nobody was looking at. A lock wait timeout in inventory, two hops downstream from where the alert fired, shows up in 470 of the 500 slowest requests, and the 947 lines behind those 470 traces mean a good share of them retried more than once. The payment gateway declines are real failures, and at 19 traces out of 500 they are not what is burning the SLO.

Done by hand, this means opening slow traces one at a time and reading the correlated logs for each, which is tedious at ten traces and nobody's idea of a plan at 500. When the spans and the logs are held in different systems, every trace you check is a copied ID and a context switch, and the sample size you can afford drops to about three. Three traces is enough to form a theory and not enough to test one. Treating the slow requests as a population is what turns "this trace had a lock wait" into "470 of the 500 slowest requests had a lock wait," and that difference decides whether you page the inventory team.

All three signals: from pod memory pressure to the log lines behind the failures

IN subqueries nest, so the pattern extends to as many signal types as the question needs.

A node pool starts reporting memory pressure after a rollout. You want the log lines from the requests that actually failed on the pods under pressure, which means going from metrics to traces to logs without stopping in between.

FROM logs-*.otel-*
| WHERE trace_id IN (
      FROM traces-*.otel-*
      | WHERE kind == "Server"
        AND status.code == "Error"
        AND resource.attributes.k8s.pod.uid IN (
            TS metrics-kubeletstats.otel-*
            | STATS peak = MAX(MAX_OVER_TIME(metrics.k8s.pod.memory_limit_utilization))
                BY resource.attributes.k8s.pod.uid
            | WHERE peak > 0.95
            | KEEP resource.attributes.k8s.pod.uid
          )
      | STATS failures = COUNT(*) BY trace_id
      | SORT failures DESC
      | LIMIT 1000
      | KEEP trace_id
    )
| STATS lines = COUNT(*), traces = COUNT_DISTINCT(trace_id)
    BY pattern = CATEGORIZE(body.text), service = resource.attributes.service.name
| SORT traces DESC

Reading the query inside out, you can see each layer answering one part of the question. The innermost subquery identifies the pods whose memory peaked above 95% of their limit, using MAX_OVER_TIME to take each pod's peak rather than its average. The middle one narrows to the requests that actually failed on those pods and reduces them to at most 1,000 trace IDs. The outer query then collects the logs for those traces from every service that took part, including services running on pods that were entirely healthy, and that last part turns out to be where the answer is.

The pods are matched on their UID rather than their name, because names repeat across namespaces and restarts. The query assumes Kubernetes metadata reaches your spans, which is what the collector's k8sattributes processor is for; if it does not, the host or container ID works the same way.

From the results above, the first row sits at exactly 1,000 because that is the subquery's LIMIT, so it describes the size of the sample rather than the size of the incident. Every trace in that sample carries the cart deadline line, which is the symptom you already knew about when you started.

Look at the second row instead. product-catalog is rejecting oversized payloads across 946 of the same 1,000 traces, and it never appeared in the pod subquery at all: its pods peaked at 75% of their memory limit, well under the 95% threshold. The rollout started sending larger payloads, which would account for both cart's memory climb and the failures. Checkout's retry budget gives out in about half of them, which is how the failure became visible to users.

Filtering logs directly by the pressured pods would have shown you the cart line and hidden the product-catalog one, which is to say it would have confirmed the symptom and buried the cause. Doing it without subqueries means three queries and two hand-built lists, and the second list is a thousand trace IDs. That is usually the point at which people stop after the first hop and go with the cart theory. The reason the second hop is cheap here is that all three signals sit in the same store behind the same query language, so widening from pods to traces to every service in the trace is a clause, not a project.

Why do ES|QL subqueries matter for AI agents?

Keeping the intermediate set inside the cluster is convenient for a person and close to essential for an agent querying on your behalf.

Split across two tool calls, the intermediate result has to travel. A list of 500 trace IDs comes back in a tool response and occupies the model's context, and the agent then has to rewrite every one of them into the next query. That costs tokens on every hop, and it is where truncation and transcription errors come from. With a subquery, the intermediate set stays inside Elasticsearch and the agent only ever sees the final table.

The problem compounds when the signals are spread across systems. An agent then needs credentials, a client, and a working knowledge of the query language for each one, plus the judgment to join results that use different names for the same host. One store and one query language reduce that to a single skill the agent has to be good at.

One ES|QL string is also a complete description of the correlation, which makes the investigation reproducible: an agent can put the query in its summary, and a human can paste it into Discover and get the same logic evaluated against current data. A two-call sequence with a hardcoded host list in the middle gives you neither. The one thing to watch is that an agent calling the _query API has to filter @timestamp itself, since nothing is binding a time picker.

Four things to know before writing ES|QL IN subqueries

IN subqueries are in technical preview in Elastic Stack 9.5 and on Elastic Cloud Serverless, while TS and FORK have been generally available since 9.4. CATEGORIZE has been generally available since 9.1 and requires a Platinum license; every query above works without it if you group by an existing field instead.

Four things are worth knowing before you write your own:

• In 9.5 the subquery returns exactly one column, which is what the trailing KEEP does in each example.
• Aggregate the subquery down to distinct values with STATS ... BY before returning them. Its result is materialized for the outer query to filter against, so handing back a few hundred host names instead of a few million rows is both faster and safer.
• Filter nulls out of any NOT IN subquery, because SQL null semantics mean one null makes the predicate match nothing.
• Subqueries are non-correlated. They run independently and cannot reference columns from the outer query, so this is a set filter rather than a row-by-row join. Reach for LOOKUP JOIN when you need per-row enrichment, which we covered in ES|QL joins for richer observability.

Try ES|QL signal correlation on your own data

The pattern under all four examples is the same. You start with a set you can describe in one signal and a question you can only answer in another, and the subquery carries that set across the boundary for you.

The syntax is the smaller part of what makes that work. It works because logs, metrics, and traces sit in one store behind one query engine, under field names they kept on the way in, so crossing from one signal to another is a clause in a query rather than an integration to build and maintain. Where that is not true, the same four investigations turn into a sequence of exports, translations, and manual joins, and that costs more than slower answers. It quietly shrinks the number of questions anyone is willing to ask, and the negative results are the first to go.

Each pattern here replaces two or three queries with one, and no host list or trace ID list has to move between them. Fewer steps mean fewer places to be wrong, and a correlation you can save as a single string and hand to someone else.

To try it:

1. Open an Observability project on Elastic Cloud Serverless, or upgrade to Elastic Stack 9.5.
2. Send data with the Elastic Distributions of OpenTelemetry, or point an existing collector at Elasticsearch.
3. In Discover, switch to ES|QL and start from the metrics to logs query above, swapping in your own data streams and thresholds.
4. Read the IN subquery reference for the full set of commands you can use inside a subquery.

That team still wants a model. So the request lands on an SRE's desk, and it sounds deceptively small: "Can you stand up an open-weight model for the claims team? Sixty people. It has to run on our hardware." They aren't even allowed to use a neocloud. There is a cost associated with this, but we won't explore that part. Just the part that covers running the model and observing the configuration.

Standing it up is the easy half. Four manifests and an afternoon, and you have a model answering questions. The hard part arrives a week later, when someone says "it feels slow" and you realize you have no idea whether the deployment is configured well, badly, or catastrophically — and no obvious way to find out.

This guide shows you how Elastic Observability can help you analyze the metrics from the configuration. It walks through tuning a real vLLM deployment using the metrics vLLM already emits. vLLM exposes these on a /metrics endpoint in Prometheus exposition format — no instrumentation, no sidecar, no code change — which is why every query in this guide starts from a Prometheus scrape. The goal: turn "it feels slow" into a specific, defensible decision.

Test environment: vLLM on a Kubernetes cluster using NVIDIA A10G with dcgm-exporter and Prometheus metrics

Every figure in this guide was measured on the following stack — one replica, one GPU, no autoscaling.

• Workload — Kubernetes-native load generator, scaled from 8 to 32 concurrent requests.
• Model — Qwen/Qwen2.5-3B-Instruct, bf16, --max-model-len 4096
• Engine — vLLM v0.23.0, OpenAI-compatible server, Prometheus /metrics on :8000
• GPU — NVIDIA A10G, 24 GB — an AWS g5.xlarge
• Cluster — Amazon EKS 1.30, tainted GPU node pool with minSize: 0
• Telemetry — Prometheus scraping every 15s, plus dcgm-exporter on :9400, shipped via remote_write
• Analysis — Elastic Observability, queried with ES|QL and PromQL
• Measured — 2026-07-27

Why is it hard for an SRE to self-host and tune an open-weight LLM?

The difficulty is not the deployment, it's the tuning which has no feedback loop. vLLM starts, serves, and reports success whether it's configured brilliantly or wastefully. Nothing tells you which.

When loading up the model, your manifest would have this configuration:

      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.23.0  # pin an exact release — metric names shift between versions
          args:
            - "--model=Qwen/Qwen2.5-3B-Instruct"
            - "--max-model-len=4096"         # cap context → predictable KV-cache size
            # A10G has native bf16 — do NOT add --dtype=half (T4-only).
          ports:
            - name: http
              containerPort: 8000            # OpenAI API + /metrics

But you can run into specific issues, such as: Hugging Face downloads take minutes. If your cluster expects a server to start in 30 seconds, it will assume the app is dead and kill it mid-download, putting you in an infinite crash loop.

Or you could have a hardware mismatch, and potentially degrade your model’s speed or precision because hardware architectures vary

or a bevy of other issues.

Once the model is finally running, optimizing performance is complete guesswork because default metrics don't tell you if you're being efficient.

You could use nvidia-smi, but this only understands raw hardware state, not application software logic.

Now that you have it running, a few hours to maybe even a day in, the team says "it feels slow." You are, functionally, tuning blind.

How do you tune a self-hosted vLLM deployment?

You're not an inference engineer, you own forty other services besides this one, and you don't have a forward-deployed engineer from a model vendor on call. But tuning an LLM server turns out to need exactly one skill you already have: reading telemetry and reasoning about saturation. The only missing piece is telemetry that exists and means something.

It does. vLLM emits a rich Prometheus endpoint out of the box — latency decomposed by inference phase, cache hit rates, batch occupancy, token accounting, completion outcomes. Almost nobody looks at it. The rest of this guide is how to read it.

Defining the workload: sixty users, short prompts, streaming responses

With the slowness detected and reported, you gather the usage profile of the users. Their usage pattern is as follows:

• ~60 users, but not concurrent. Realistic peak is 8–12 simultaneous in-flight requests; sustained is lower.
• Short prompts, long answers. The user pastes a paragraph and asks for a structured summary. Prompts run ~50 tokens; useful answers run 500–1,000.
• Interactive, streaming UI. Perceived speed is dominated by time to first token (TTFT), not total time — the same psychology as a chat interface.
• Heavy prompt reuse. Every request carries the same system prompt and the same policy-language boilerplate.

From that, you write down actual service objectives — the step most self-hosted LLM projects skip:

• TTFT p95 < 300 ms
• inter-token latency < 50 ms (≥ 20 tokens/sec, faster than reading speed)
• zero queueing at 12 concurrent
• error + abort rate < 0.5%

Those four numbers are the point of everything that follows. Without them, "it feels slow" has no answer. With them, every metric below passes or fails a stated bar.

How do you get Prometheus metrics out of vLLM and the GPU?

There are two sources, and you need both.

• vLLM reports on itself — latency by phase, cache hit rates, batch occupancy, token counts — on /metrics at its serving port, with no adapter and no instrumentation work.
• The GPU reports separately, through NVIDIA's dcgm-exporter on :9400 (NVIDIA Data Center GPU Manager (DCGM) is a suite of tools and libraries designed to comprehensively manage, monitor, and diagnose enterprise-grade NVIDIA GPUs in clusters and data centers). vLLM tells you what the engine thinks is happening; DCGM tells you what the card is actually doing. Step 6 is built entirely on the gap between those two answers.

On this EKS cluster that means three things running side by side:

• vLLM as a plain Deployment on the tainted g5.xlarge GPU node pool. For one model on one card, a Deployment and a Service is the whole architecture.
• dcgm-exporter as a DaemonSet, pinned to the same GPU nodes.
• A Prometheus server on a CPU node, scraping both endpoints every 15 seconds.

Nothing here is AWS-specific. The production version is the same manifests on an on-prem cluster with L40S or H100 nodes — which is the point of doing this on Kubernetes rather than on a vendor's platform.

What about KServe and llm-d?

Neither was run for this guide, and neither changes where the metrics come from. KServe and llm-d sit on top of vLLM rather than replacing it — vLLM is still the engine, so /metrics is still the source of every number here. Each adds its own layer on top (KServe: autoscaler and revision metrics; llm-d: router and cache-routing metrics), but the inference telemetry underneath is identical.

What they change is when you need them — and each promotion is triggered by a metric you're already collecting:

How do you ship the vLLM metrics and DCGM metrics to Observability

A Prometheus scraping inside the cluster only holds hours of data — the metrics have to reach a store you can still query next week. There are two paths for that, and they are not equivalent.

Path A — OpenTelemetry Collector. Puts inference metrics into the same pipeline as your traces and logs. One collector, one auth path, one mental model. The cost is that Prometheus metrics sent through OTLP get normalized: the stored schema ends up neither purely Prometheus nor purely OTel, and metric names shift.

Path B — native Prometheus remote_write. Stands up a small Prometheus that scrapes both endpoints and pushes to a backend speaking the remote-write protocol. Names and labels land untouched, _sum / _count / _bucket histogram parts stay intact, and existing queries keep working.

For a tuning exercise, choose Path B. That's what produced every number in this guide. The reason is narrow but decisive: tuning means comparing against the vLLM documentation and the vLLM community, and both speak in exact metric names. When your chart says vllm:kv_cache_usage_perc, you can search for it.

The deployment is two YAML files — a Prometheus Deployment with two scrape jobs and a remote_write block, plus a Secret holding the backend credential. In this build the destination was an Elastic Serverless project, which exposes a Prometheus remote-write endpoint and lands data in a time-series data stream, metrics-vllm.prometheus-inference.

Two things cost me real time. If your backend has a separate ingest host for OTLP versus its main API, remote-write usually lives on the main API host, not the ingest one — pointing at the wrong one returns a 404 that looks like a path error. And the credential needs index-write privileges, not just ingest authentication; a key that works fine for OTLP can authenticate successfully and then 403 on every sample. Check prometheus_remote_storage_samples_failed_total on the Prometheus itself before looking anywhere else.

How do you confirm vLLM metrics landed in Elastic?

Once the pipeline is up, look at the field list. Roughly 127 metric series arrive from a single vLLM pod plus DCGM:

This screen is more useful than it looks. Scanning the field list is how you confirm your vLLM version's exact metric names — they do shift between major vLLM releases, and a dashboard built against the wrong names fails silently by returning nothing rather than erroring.

Two gotchas when querying vLLM metrics

Both produce results that look like "the metrics aren't working" when the pipeline is perfectly healthy.

vLLM metric names contain a colon (vllm:num_requests_running), so they need escaping in most query languages. More insidiously, if you filter by metric name across several metrics and then aggregate only one of them, you get rows back — full of nulls, with no error. Each Prometheus metric lands in its own field, so naming the field is the filter; you don't need the name predicate at all.

Counters need rate functions, gauges don't. vllm:generation_tokens_total is cumulative and monotonic — taking a max of it gives the pod's lifetime total, not its throughput. Gauges like vllm:num_requests_running, vllm:num_requests_waiting and vllm:kv_cache_usage_perc are instantaneous and want max or average. Mixing these up produces charts that are wrong but plausible, which is considerably worse than charts that are empty.

Which vLLM Prometheus metrics actually matter?

A reference for the metrics used in this guide, what each tells you, and the condition worth watching. Names are as vLLM emits them; the DCGM_FI_* series come from dcgm-exporter.

| Metric | Type | What it tells you | Watch for | |---|---|---|---| | vllm:time_to_first_token_seconds | Histogram | TTFT — how long before the first token streams | p95 above your interactive bar (300 ms here) | | vllm:inter_token_latency_seconds | Histogram | Streaming speed after the first token | Above ~50 ms is slower than reading speed | | vllm:e2e_request_latency_seconds | Histogram | Total request time | Rising while TTFT is flat = decode or workload change | | vllm:request_queue_time_seconds | Histogram | Time waiting for admission | Earliest saturation signal — any sustained rise | | vllm:request_prefill_time_seconds | Histogram | Time processing the prompt | Dominant share = prefill-bound workload | | vllm:request_decode_time_seconds | Histogram | Time generating tokens | Dominant share = memory-bandwidth-bound | | vllm:num_requests_running | Gauge | Requests currently being decoded | Batch occupancy | | vllm:num_requests_waiting | Gauge | Requests queued for admission | Sustained non-zero = add a replica | | vllm:kv_cache_usage_perc | Gauge | Occupancy of the KV block pool — not VRAM | Autoscaling trigger (~60%) | | vllm:prompt_tokens_total + vllm:prompt_tokens_cached_total | Counters | Prefix-cache hit rate | A drop means routing scattered your prefixes | | vllm:generation_tokens_total | Counter | Output throughput in tokens/sec | Headline throughput number | | vllm:request_prompt_tokens + vllm:request_generation_tokens | Histograms | Per-request token sizes; their ratio is the workload's shape | A moving ratio means the workload changed character | | vllm:iteration_tokens_total | Histogram | Tokens advanced per forward pass | Near 1.0 with concurrency = batching broken | | vllm:request_success_total{finished_reason} | Counter | Completion outcomes | error/abort = SLO; length share = truncation | | http_requests_total{status} | Counter | Server-level requests | Catches 4xx and malformed requests vllm:* never sees | | DCGM_FI_DEV_FB_USED / _FB_FREE | Gauge | Physical VRAM | Capacity planning only — never alert on it | | DCGM_FI_DEV_GPU_UTIL | Gauge | "A kernel is resident" | Not a measure of useful work | | DCGM_FI_PROF_PIPE_TENSOR_ACTIVE | Gauge | Tensor-core activity | Low here + high DRAM = memory-bound | | DCGM_FI_PROF_DRAM_ACTIVE | Gauge | Memory-bandwidth activity | High = the bottleneck is bandwidth | | DCGM_FI_DEV_POWER_USAGE | Gauge | Watts drawn | Pairs with throughput for tokens-per-watt |

Step 1: Where does vLLM latency go? TTFT, prefill, and decode decomposed

Decompose total latency into queue, prefill, and decode before optimizing anything. vLLM reports all three separately, and they have completely different fixes. This is the single most valuable chart in the setup.

The query averages each phase's cumulative time by request count in the same window — in Prometheus terms, rate(vllm:request_prefill_time_seconds_sum) / rate(vllm:e2e_request_latency_seconds_count), and the same for queue and decode:

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS reqs = SUM(RATE(`metrics.vllm:e2e_request_latency_seconds_count`)),
        q_s  = SUM(RATE(`metrics.vllm:request_queue_time_seconds_sum`)),
        pf_s = SUM(RATE(`metrics.vllm:request_prefill_time_seconds_sum`)),
        dc_s = SUM(RATE(`metrics.vllm:request_decode_time_seconds_sum`))
    BY minute = BUCKET(@timestamp, 1 minute)
| EVAL queue_ms   = ROUND(q_s  / reqs * 1000, 2),
       prefill_ms = ROUND(pf_s / reqs * 1000, 1),
       decode_ms  = ROUND(dc_s / reqs * 1000, 1)
| KEEP minute, queue_ms, prefill_ms, decode_ms
| SORT minute ASC

At 8 concurrent requests on the A10G:

minute    | queue_ms | prefill_ms | decode_ms | e2e_ms  | ttft_ms | inter_token_ms
22:55:00  | 0.01     | 37.98      | 1664.69   | 1715.18 | 50.81   | 16.23
22:56:00  | 0.01     | 40.99      | 1670.44   | 1723.78 | 53.50   | 16.20

Read it against the objectives:

• TTFT is 51 ms against a 300 ms target. Passing, with almost 6× headroom. Perceived responsiveness is not the problem, whatever was said in the meeting.
• Inter-token latency is 16 ms — about 62 tokens/sec against a 50 ms / 20 tok-s bar. Text arrives roughly three times faster than a person reads it.
• Queue time is 0.01 ms. Nothing is waiting for admission; the engine has capacity to spare at this concurrency.
• Decode is 1,665 ms against 38 ms of prefill — 97% of the time is decode.

That last line is the finding. Every optimization aimed at prefill is worthless for this workload. Chunked prefill, prompt compression, a faster attention kernel for long contexts — all real techniques, all irrelevant when prefill is 2% of the time. Decode is memory-bandwidth-bound, so the levers that would actually move it are quantization, tensor parallelism across two cards, or a smaller model. A single chart eliminated the wrong shopping list.

Watch vllm:request_queue_time_seconds specifically. It is the earliest saturation signal in the entire vLLM metric set — queue time climbs before vllm:num_requests_waiting becomes visibly non-zero, because a request can wait milliseconds for admission without ever registering as queued at scrape time. If you alert on one thing from this section, alert on queue time crossing a small absolute threshold.

Step 2: Is vLLM using the GPU efficiently? KV cache, prefix caching, and batch occupancy

Four metrics answer this: prefix-cache hit rate, tokens per iteration, KV-cache occupancy, and running-vs-waiting requests. Latency tells you the experience is good; these tell you whether you're overpaying for it.

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS ptok    = SUM(RATE(`metrics.vllm:prompt_tokens_total`)),
        cached  = SUM(RATE(`metrics.vllm:prompt_tokens_cached_total`)),
        gen     = SUM(RATE(`metrics.vllm:generation_tokens_total`)),
        it_s    = SUM(RATE(`metrics.vllm:iteration_tokens_total_sum`)),
        it_c    = SUM(RATE(`metrics.vllm:iteration_tokens_total_count`)),
        running = MAX(`metrics.vllm:num_requests_running`),
        waiting = MAX(`metrics.vllm:num_requests_waiting`),
        kv      = MAX(`metrics.vllm:kv_cache_usage_perc`)
    BY minute = BUCKET(@timestamp, 1 minute)
| EVAL prefix_cache_hit_pct = ROUND(cached / ptok * 100, 1),
       tokens_per_iteration = ROUND(it_s / it_c, 2),
       gen_tokens_per_sec   = ROUND(gen, 1),
       kv_cache_pct         = ROUND(kv * 100, 3)
| KEEP minute, prefix_cache_hit_pct, tokens_per_iteration,
       gen_tokens_per_sec, running, waiting, kv_cache_pct
| SORT minute ASC

minute   | prefix_cache_hit_pct | tokens_per_iteration | gen_tok/s | running | waiting | kv_cache_pct
22:55:00 | 32.5                 | 10.36                | 479.1     | 8.0     | 0.0     | 0.255
22:59:00 | 32.3                 | 10.34                | 480.0     | 8.0     | 0.0     | 0.121

A 32% prefix-cache hit rate is a third of all prefill work simply not done. The claims team's requests share a system prompt and policy boilerplate, and vLLM's automatic prefix caching recognizes that. This is a direct argument for raising prompt standardization: the more the application puts shared context in a consistent leading position, the higher this climbs and the cheaper every request gets. It is also the number that will crater the day a naive round-robin load balancer sits in front of two replicas — precisely the condition that justifies llm-d's cache-aware routing.

tokens_per_iteration ≈ 10.3 with 8 concurrent requests is continuous batching working correctly. Each forward pass through the model advances about ten sequences at once. If this sat near 1.0 with multiple requests in flight, batching would be broken and you'd be paying full model-forward cost per token per user. This metric proves you're getting vLLM's core value.

What does vllm:kv_cache_usage_perc actually measure?

vllm:kv_cache_usage_perc reports occupancy of vLLM's pre-allocated KV block pool — not physical GPU memory. At startup, vLLM reserves a fraction of VRAM (governed by --gpu-memory-utilization, default 0.9) and carves a KV block pool out of that reservation. This gauge reports how full that pool is.

That's why it read 0.25% here. Eight concurrent requests holding ~150 tokens each barely touch an A10G's block budget. The pool is large, and correctly so. Push the same server to 32 concurrent requests with 512–1,024 token generations and it moves — to about 2.8%. Still small.

The instinct is to read a number that low as "the cache is broken" or "I've massively over-provisioned." Both are wrong. Treating this gauge as a VRAM proxy is the most common self-hosted vLLM configuration error I see. Step 6 shows exactly how far apart the two are.

Step 3: What shape is your vLLM inference workload?

Confirm the workload is what you think it is before tuning anything. This is the panel that explains a latency "regression" that isn't your fault.

minute   | requests_per_min | avg_prompt_tokens | avg_generated_tokens | avg_max_tokens | gen_to_prompt_ratio
22:55:00 | 282.2            | 49.6              | 103.6                | 103.6          | 2.09
23:01:00 | 276.0            | 50.6              | 103.0                | 103.0          | 2.04

The generation-to-prompt ratio is 2.04 — this workload writes twice as much as it reads. That single ratio is the explanation for Step 1's 97%-decode finding, and it holds for any summarize-and-draft use case. If the team later adds a long-document RAG feature, prompts jump to thousands of tokens, the ratio inverts, the workload becomes prefill-bound, and the correct tuning changes completely. Watching this ratio is how you learn your workload changed character before someone files a ticket.

Now the column that should bother you: avg_generated_tokens equals avg_max_tokens exactly. Every request is stopping because it hit its token ceiling, not because the model finished its thought. The screenshot shows the same pattern holding as generation lengths scale to ~685 tokens against a ~777 ceiling.

In a load test that's an artifact of the generator. In production, that number is users getting cut off mid-sentence — and it is invisible in every latency metric you have. Which brings us to the metric that catches it.

Step 4: Are vLLM requests actually succeeding? Checking finished_reason

Break vllm:request_success_total down by its finished_reason label. This is the closest thing self-hosted inference has to an application-level SLI, and it catches a failure mode no latency chart can.

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS completions_per_min = ROUND(SUM(RATE(`metrics.vllm:request_success_total`)) * 60, 2)
    BY minute = BUCKET(@timestamp, 1 minute), finish_reason = labels.finished_reason
| SORT minute ASC, finish_reason

Five outcomes, each meaning something different operationally:

| finished_reason | What it means | What to do about it | |---|---|---| | stop | The model finished naturally | This is the number you want large | | length | Truncated at max_tokens | High share means users are cut off — raise the ceiling, or shorten the ask | | abort | The client disconnected first | Users giving up, or a proxy timeout shorter than your generations | | error | The engine failed | Your hard SLO signal. Should be flat zero | | repetition | Degenerate looping output | A sampling-parameter problem, not an infrastructure one |

Under the small load generator the split was 100% length — expected, since it requested a fixed ceiling. In the screenshot, at a mixed load, stop and length run side by side at roughly 51 and 32 completions/min. That mix is the healthy shape: most requests finishing on their own, a minority hitting the ceiling.

The lesson generalizes. error and abort are what you page on. But the stop-to-length ratio is what you review weekly, because drift toward length means answers are being truncated and no latency dashboard on earth will tell you.

One blind spot to close: vllm:* metrics only count requests the engine accepted. Malformed JSON, 4xx, auth failures and dropped connections never reach it. Those live in http_requests_total with status and handler labels — worth a panel beside this one, because "the model is broken" reports frequently turn out to be the gateway in front of it.

Step 5: What NVIDIA DCGM metrics say the GPU is actually doing

Use NVIDIA DCGM as an independent witness to vLLM's own account. Everything so far is the engine describing itself; DCGM describes the silicon.

FROM metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS gpu_util_pct    = MAX(`metrics.DCGM_FI_DEV_GPU_UTIL`),
        mem_bw_util_pct = MAX(`metrics.DCGM_FI_DEV_MEM_COPY_UTIL`),
        vram_used_mib   = MAX(`metrics.DCGM_FI_DEV_FB_USED`),
        vram_free_mib   = MIN(`metrics.DCGM_FI_DEV_FB_FREE`),
        power_w         = ROUND(MAX(`metrics.DCGM_FI_DEV_POWER_USAGE`), 1),
        temp_c          = MAX(`metrics.DCGM_FI_DEV_GPU_TEMP`)
    BY minute = BUCKET(@timestamp, 1 minute)
| SORT minute ASC

→ 100% GPU util · 21,483 MiB used / 1,352 MiB free · 240 W · 73 °C

100% utilization and 94% VRAM. Under the conventional reading, this card is maxed out and it's time to ask for more hardware. That reading is wrong.

Why is GPU utilization a misleading metric for LLM inference?

DCGM_FI_DEV_GPU_UTIL means "a kernel is resident on the device," not "the device is doing useful work." It reads 100% for a perfectly-tuned server and 100% for a badly-tuned one, so it cannot distinguish them. The DCGM profiling counters can:

gr_engine_active 99.8%  ·  tensor_active 16.6%  ·  dram_active 80.4%

Read those three together. The GPU's compute engine is busy essentially all the time — but its tensor cores, the units that do the actual matrix math, are active only 16.6%, while DRAM is active 80.4%. The card is not computing. It is waiting on memory.

This is independent, hardware-level confirmation of what Step 1 inferred purely from timings: decode is memory-bandwidth-bound. Two entirely different instruments, two different layers of the stack, one conclusion — the difference between a hypothesis and a finding.

It also permanently retires GPU utilization as a capacity metric for LLM inference. If your GPU capacity planning rests on DCGM_FI_DEV_GPU_UTIL — and most does — it rests on nothing.

Step 6: vLLM KV cache vs GPU VRAM, and why they disagree

Put vllm:kv_cache_usage_perc and physical VRAM usage on one 0–100% axis. They describe the same GPU memory, they sit at opposite ends of the chart, and both are correct.

minute   | running | gen_tok_s | kv_cache_pct | vram_used_pct | gpu_util | dram_active_pct | tensor_active_pct | tokens_per_watt
00:05:00 | 31      | 1627.5    | 2.82         | 94.1          | 100.0    | 80.4            | 16.6              | 6.80

KV cache at 2.8%. VRAM at 94.1%.

vLLM pre-allocates a large fraction of VRAM at startup — governed by --gpu-memory-utilization, default 0.9 — and carves its KV block pool out of that reservation. vllm:kv_cache_usage_perc reports occupancy of the pool. DCGM reports what the driver sees, which is the whole reservation, whether or not it's holding anything.

The operational consequences are precise, and they're the practical payoff of the entire exercise:

• Autoscale on vllm:kv_cache_usage_perc and vllm:num_requests_waiting. These describe admission capacity — whether the engine can take another request right now.
• Capacity-plan on VRAM. This describes physical space — whether a second model could ever fit on this card. (It can't. 1.3 GB free.)
• Never alert on VRAM. It will page you at 3 a.m. for a healthy, mostly-idle server, every single night, forever.

And tokens_per_watt — generated tokens divided by power draw, 6.8 here — is a genuine cost-efficiency metric. It's comparable across GPU models, batch settings and quantization levels in a way that neither latency nor utilization is. When you go back to Finance for card number two, this is the number that makes the argument: at 32 concurrent we sustain 1,627 tokens/sec at 240 watts, and here's what that becomes on an L40S.

vLLM tuning decisions: what the SRE does with these Prometheus metrics

Six steps, thirty minutes, one server. The verdict against the stated objectives:

| Objective | Measured | Verdict | |---|---|---| | TTFT p95 < 300 ms | 51 ms | Pass, 6× headroom | | Inter-token < 50 ms | 16 ms (≈62 tok/s) | Pass | | Zero queueing at 12 concurrent | queue_ms 0.01, waiting 0 at 8; still 0 at 32 | Pass, large margin | | Error + abort < 0.5% | 0% | Pass |

The configuration is correct for this department, and the department is over-provisioned rather than under-provisioned. That's a defensible, evidence-backed answer to "it feels slow" — and it redirects the investigation to the app, the gateway, or the prompt, which is where the problem actually is.

The concrete follow-ups, each tied to a metric rather than a hunch:

1. Stop optimizing prefill. — vllm:request_decode_time_seconds vs vllm:request_prefill_time_seconds. Decode is 97% of the time against prefill's 2%, confirmed twice. Chunked prefill and prompt compression are off the table for this workload.
2. If more throughput is needed, quantize before buying hardware. — DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (16.6%) vs DCGM_FI_PROF_DRAM_ACTIVE (80.4%). The bottleneck is memory bandwidth, not compute, so an FP8 or AWQ build of the same model is the highest-leverage single change: it moves fewer bytes per token, which is exactly the constrained resource.
3. Raise the client-side max_tokens ceiling. — vllm:request_success_total{finished_reason}. Every request finishing on length rather than stop is a user getting cut off mid-answer. This is the one finding that's a live user-experience defect. You need to increase the prompt max_token limit.
4. Standardize the prompt prefix. — vllm:prompt_tokens_cached_total over vllm:prompt_tokens_total, 32% today. But it should be better (more like 70%) More shared boilerplate in a consistent leading position raises it, and it's free.
5. Set the autoscaling trigger now, before it's needed. — vllm:kv_cache_usage_perc and vllm:num_requests_waiting. Scale when the first crosses ~60% or the second stays above zero. Do not scale on DCGM_FI_DEV_GPU_UTIL — it's pinned at 100% regardless.
6. Alert on queue time, not on VRAM. — vllm:request_queue_time_seconds is the earliest true saturation signal; DCGM_FI_DEV_FB_USED is a constant that looks like an emergency.
7. Revisit when the workload changes shape. — vllm:request_generation_tokens over vllm:request_prompt_tokens, 2.04 today. When the RAG feature ships that ratio inverts, the workload becomes prefill-bound, and half of this analysis needs redoing. The chart tells you the day it happens.

Notice that most of the metrics are looking at the vLLM metrics not the GPU metrics in helping optimize. These are still within the limit of a single service, but when you get vllm:num_requests_waiting to persistently non-zero, then you need to run KServe or you can use KEDA and HPA autoscaling. But you get the metrics to help you determine or allow KServe to scale. So you can see that understanding these metrics are crucial to tuning the inference service.

Elastic Observability can provide this to you.

Why self-hosted LLM tuning is an SRE problem, not an ML problem

Self-hosting an open-weight model is not primarily an ML problem. It is a capacity and saturation problem — something SREs have been extremely good at for twenty years. The blocker was never skill. It was that the telemetry sat unexamined on a /metrics endpoint nobody scraped, in a schema nobody had mapped to the questions they actually had.

Once it's collected, the reasoning is familiar work in unfamiliar clothes:

• Decompose latency by phase before optimizing anything (queue / prefill / decode).
• Distinguish the logical resource from the physical one (KV block pool ≠ VRAM), and know which each metric describes.
• Never trust a single-source utilization number — corroborate the engine's account with the hardware's.
• Tie every knob to a metric and every metric to a stated objective, so tuning converges instead of wandering.

For a team that isn't allowed to send its data anywhere, that difference — between running a model and operating one — is the whole ballgame. The department gets a capability it's otherwise locked out of, and the SRE gets to answer questions about it with numbers.

Frequently asked questions

What does vllm:kv_cache_usage_perc measure? It measures occupancy of vLLM's pre-allocated KV block pool, not physical GPU memory. vLLM reserves a fraction of VRAM at startup (--gpu-memory-utilization, default 0.9) and carves the KV pool from that reservation. In this deployment it read 2.8% while DCGM reported 94.1% VRAM used on the same card at the same moment. Use it as an autoscaling signal; use VRAM for capacity planning.

Why is my vLLM deployment decode-bound? Because the workload generates more tokens than it reads. Compare vllm:request_decode_time_seconds against vllm:request_prefill_time_seconds, and check the generation-to-prompt token ratio. In this deployment the ratio was 2.04 — twice as many output tokens as input — which produced 1,665 ms of decode against 38 ms of prefill. Decode is memory-bandwidth-bound, so quantization, tensor parallelism, or a smaller model help; prefill optimizations do not.

Should I autoscale vLLM on GPU utilization? No. DCGM_FI_DEV_GPU_UTIL means a kernel is resident on the device, not that the device is doing useful work — it reads 100% for both a well-tuned and a badly-tuned server. Autoscale on vllm:kv_cache_usage_perc (around 60%) or on vllm:num_requests_waiting staying above zero, since those describe whether the engine can admit another request.

Why does my GPU show 100% utilization when it isn't fully used? Because GPU utilization only reports kernel residency. Check the DCGM profiling counters instead: in this deployment DCGM_FI_PROF_GR_ENGINE_ACTIVE was 99.8% while DCGM_FI_PROF_PIPE_TENSOR_ACTIVE was only 16.6% and DCGM_FI_PROF_DRAM_ACTIVE was 80.4%. That combination means the GPU is waiting on memory bandwidth rather than computing.

How do I get vLLM metrics into Prometheus? vLLM already exposes Prometheus exposition format on /metrics at its serving port — no adapter or instrumentation needed. Point a Prometheus scrape job at the vLLM Service, add a second job for dcgm-exporter on :9400, and use remote_write to ship to long-term storage. Sending through an OpenTelemetry Collector also works but normalizes the metric names, which makes them harder to match against vLLM documentation.

What TTFT should I target for an interactive LLM application? For a streaming chat-style interface, a p95 time-to-first-token under 300 ms feels immediate, and inter-token latency under 50 ms (about 20 tokens/sec) outpaces reading speed. This deployment measured 51 ms TTFT and 16 ms inter-token latency on a 3B model on a single NVIDIA A10G, leaving roughly 6× headroom.

Why are all my vLLM requests finishing with length? Because they're hitting the max_tokens ceiling instead of the model choosing to stop. Break vllm:request_success_total down by its finished_reason label: a high length share means answers are being truncated mid-sentence. This is invisible in every latency metric, so review the stop-to-length ratio regularly and raise the client-side ceiling if it drifts.

When should I move from a plain vLLM Deployment to KServe or llm-d? Move to KServe when vllm:num_requests_waiting is persistently non-zero at peak and you need replicas to appear without human intervention. Move to llm-d when your prefix-cache hit rate collapses across replicas — a sign that load balancing scattered conversations that shared a prefix — or when prefill time starts stealing measurably from decode.

The challenge deepens when something goes wrong. A crash on the device, a slow screen, or an error response might originate in the Android app itself, in a backend service, or somewhere in the network path between them. Debugging these problems without a connected, E2E view from the mobile client to the backend is time-consuming and frustrating. And without a standard instrumentation format, mobile teams often end up maintaining separate tooling that doesn't integrate with what the backend and infrastructure teams already use.

OpenTelemetry offers a way out: a unified, open-standard instrumentation model that works across platforms and languages, backed by a large community. The Elastic Distribution of OpenTelemetry Android, or EDOT Android, is an APM agent for native Android applications built on top of OpenTelemetry. It gives Android teams a practical path to observe mobile app behavior in Elastic, providing them with distributed tracing, crash reporting, session tracking, disk buffering, and automatic instrumentation, with as little code as possible while staying grounded in open standards.

To see what it all looks like, we will instrument a demo Android weather application end to end. You will run Elasticsearch, Kibana, and the Elastic Agent locally. The Elastic Agent provides the OTLP endpoint that receives telemetry from the Android app and backend. You will then generate distributed traces, custom spans, logs, and Android crashes from the app, and explore the results in Kibana using the Android OpenTelemetry dashboards.

This article focuses on a hands-on experiment to explore the E2E experience of observing Android apps with Elastic, using the EDOT Android agent. For more specific details on the EDOT Android agent, such as a list of supported features and a setup guide for your own Android project, take a look at EDOT Android docs.

Setting up EDOT Android with Elasticsearch and Kibana

We will use the EDOT Android demo application. The demo is intentionally small but covers the main workflows you need when evaluating mobile observability with Elastic.

The demo has two main components: an Android app, and a Spring Boot backend. Additionally, you'll need an Elastic Stack environment up and running; we'll explain more about how to get one later in this guide.

Prerequisites

• Java 17 or higher.
• Docker.
• Android Studio.
• An Android emulator.
• On Windows, use Windows Subsystem for Linux (WSL) to run the demo scripts.

Step 1: Clone the demo app's repository

We'll start by cloning the EDOT Android demo application:

git clone git@github.com:elastic/android-agent-demo.git

Step 2: Start the Elastic Stack

The demo uses start-local to run Elasticsearch, Kibana, and the Elastic Agent with a single command. In this setup, the Elastic Agent provides the OTLP endpoint that receives telemetry from the application and backend. Run this from the directory where you want the local Elastic files to be created:

curl -fsSL https://elastic.co/start-local | sh -s -- --edot

For more information on this step, take a look at the demo app's instructions.

Step 3: Start the local backend

The demo backend is a Spring Boot service instrumented with the EDOT Java agent. It handles the app's weather requests and calls the Open-Meteo public API for weather data.

./backend-manager start

For more information on managing the backend service, take a look at the demo app's instructions.

Step 4: Launch the Android application

Use Android Studio to open up the EDOT Android demo application repo and run the application in your emulator. More info on how to run Android apps from Android Studio here.

Generating distributed traces, errors and crashes from an Android app

The Android app has two screens: a city selector and a weather display screen that shows the current weather for the selected city on the previous screen. It includes two intentional failure paths: the first one is reached by selecting New York, which causes the backend to reject the request (the demo backend only supports European cities), and tapping the floating crash button intentionally crashes the app so you can review crash reporting in Kibana after relaunch. We'll take a look at those use cases in more detail below.

Tracing a successful request end to end

In the EDOT Android demo app, selecting "Paris" as the city triggers a successful backend request on the second screen, for which a span will be automatically generated using EDOT Android's OkHttp auto-instrumentation, which supports all OkHttp-generated HTTP requests and tools using it, such as Retrofit. Aside from the Android HTTP span, the successful city request continues e2e and creates a backend HTTP client span to Open-Meteo.

How backend errors appear in the Android distributed trace

The demo backend only supports European cities, so selecting "New York" causes it to fail, which in turn automatically creates an error associated with our Android app's HTTP span. This is done automatically. We'll see later how to find and inspect these issues from Kibana.

How EDOT Android captures and reports app crashes

The crash button creates a crash event that appears in Kibana after the app is reopened. This event contains session information that will help us narrow down its root cause from Kibana, as we'll see later.

Note: EDOT Android automatically attaches Android session context to spans and logs. That means that any span or log created before the crash can be reviewed together with the crash event and nearby spans from the same session, giving you a complete picture of what the user was doing. This even applies to manually created spans and logs.

Visualizing Android application monitoring data in Kibana

To see the whole story from our Android app in a single place, we'll install Kibana's Android OpenTelemetry Assets package by following the steps below.

1. In Kibana, search for "Android OpenTelemetry Assets" in the global search field.
2. Open it and click Install to add the Android dashboards to your Kibana instance.

Exploring Android application monitoring dashboards in Kibana

Once the content package is installed, open the [Android OTel] Application Overview dashboard:

1. In Kibana, search for Dashboards in the global search field or in Kibana's menu.
2. In Dashboards, search for Android OTel and open the "[Android OTel] Application Overview" dashboard.
3. Select your application from the Applications panel at the top of the dashboard.

The dashboard provides a set of metric panels for an overview of your app's health, performance, and RUM, as well as a set of panels that can be further explored either in Discover or the Exception dashboard, as explained below.

How to inspect the distributed tracing waterfall in Kibana

From the Application Overview dashboard, go to one of the span tables (either All spans or Failed spans) and click its Explore in Discover button.

In Discover, click the expand icon on the left side of any span row to open its details panel. The trace waterfall UI appears inside, showing the full span hierarchy and timing for that trace. You can expand the waterfall to full screen and drill down from there.

Analyzing failed spans and backend errors in Kibana

While you can find all kinds of spans in the dashboard's All spans panel, you can narrow them down to failed ones only by exploring the Failed spans panel instead.

For the New York path, find a failed span and expand it. The trace waterfall highlights the backend error, and the exception details show the intentional backend rule that only supports European cities.

Reviewing crash details and stacktraces in the exception dashboard

Crash reporting is provided by the crash automatic instrumentation plugin. When an unhandled exception crashes the app, EDOT Android stores the crash event on disk. The event is exported the next time the app starts. Disk buffering ensures the crash event is not lost even if the network was unavailable at the time of the crash.

In the Application Overview dashboard, scroll to the Crashes section. You will see crash groups listed by a computed stacktrace group ID. Select a group and click View crash details to open the Exception Details dashboard.

The Exception Details dashboard shows a set of metrics to better understand the impact of the selected crash, as well as its full stacktrace. Crash events are grouped based on their stacktrace, which helps ensure that the same crash is counted and aggregated in this dashboard to better understand a single crash's impact on your application.

For this demo, the stacktrace points to the intentional crash in MainActivity. The nearby session events should also include the custom Crash button click log created just before the crash, which helps explain how the crash was triggered. We'll take a look at how to inspect a session to get an idea of the user's journey within your application that led them to a crash.

Using sessions to understand user flow in EDOT Android

Mobile troubleshooting often starts with a single bad outcome (a crash, an error, a slow UX), but the useful question is what happened before that outcome. EDOT Android helps answer that by attaching session.id to every span and log emitted by the application, even for manually created ones.

A session is meant to cover a single user interaction with your application. A new one is created when there is no previous active session or when the previous session has expired. Sessions expire after 30 minutes of inactivity. If the app stays active, a session can last up to 4 hours.

This lets you query all the telemetry from a single session and review it in order. After finding a crash group, drill into one affected session from the Top affected sessions panel and review the event timeline. You can see the custom logs, app startup spans, HTTP request spans, and crash data together in one investigation path.

The Event timeline panel on the Application Overview dashboard is also useful here: select a session from the dashboard's top filters, and the timeline shows the full sequence of spans and logs in that session chronologically.

What EDOT Android adds as a mobile APM for production apps

The demo uses a local stack and simple code, but the same agent features apply to production apps.

Disk buffering stores telemetry locally before export. This reduces data loss when the device has poor connectivity or the app is temporarily offline.

Automatic instrumentation creates telemetry for supported targets without adding code around every call. Today that includes OkHttp, crash reporting, and an adapter for OpenTelemetry Android instrumentation.

Manual instrumentation lets you add spans, logs, and metrics for app-specific workflows. This is useful for screen loading times, checkout flows, login steps, feature usage, or any area where framework-level telemetry alone is not enough.

Central configuration can remotely adjust selected EDOT Android behavior through Kibana when the OpAMP endpoint is configured. At the time of writing, central configuration for EDOT Android is in preview and supports settings such as recording and session sample rate.

Distributed tracing connects Android app requests to backend service spans so you can trace the full path of any user action, from the tap on the screen to the database query on the server. EDOT Android ensures that your application's telemetry timestamps are in sync with the coordinated universal time. This ensures a proper trace waterfall hierarchy later on in Kibana, where different components are properly coordinated in time.

Clean up the demo

When you are finished, stop the backend in case you're planning to restart it later, or uninstall it otherwise:

./backend-manager stop
# ./backend-manager uninstall

Then stop or uninstall the local Elastic Stack:

cd elastic-start-local
./stop.sh
# ./uninstall.sh

Getting started with EDOT Android in your own app

EDOT Android gives native Android teams an OpenTelemetry-based path for mobile APM in Elastic. With a small Gradle setup and one early initialization call, you get automatic HTTP spans, crash reporting, session tracking, and direct access to the OpenTelemetry SDK for custom telemetry, and you can see it all tied together in Kibana's Android dashboards.

Observability is a crucial part of modern mobile development. Crashes, slow screens, and backend errors all impact real users, and the sooner you can identify root causes across the full request path, from the device to the database, the better. The demo app is a good first step because it exercises the complete workflow without requiring a production deployment. After that, the same setup model applies to your own app with production endpoints, API key authentication, and custom spans and logs tailored to your use cases.

Developer resources:

• EDOT Android demo application
• EDOT Android documentation
• EDOT Android getting started guide
• EDOT Android automatic instrumentation
• EDOT Android manual instrumentation
• EDOT Android configuration
• EDOT Android troubleshooting
• Android OpenTelemetry Assets dashboard docs

Don't have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out mobile observability with EDOT Android as described in this guide. We'd love to hear about your experience gaining visibility into your Android application stack with Elastic.

What you need

• An Observability serverless project, or an Elastic Cloud Hosted or self-managed stack at version 9.4 or later. PromQL is generally available in Elastic Cloud Serverless and Elastic Stack 9.5, and available as a technical preview in Elastic Stack 9.4.
• Kubernetes state and container memory metrics in Elasticsearch.

What is the alert telling us?

This is the alert that opened the investigation:

It comes from this waiting-reason query:

PROMQL
  max by (namespace, pod, container) (
    max_over_time(
      kube_pod_container_status_waiting_reason{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api",
        reason="CrashLoopBackOff"
      }[2m]
    )
  ) == 1

A result of 1 means Kubernetes is delaying another start because the container has failed repeatedly. That is the correct paging signal here because the checkout path has a single replica: when that replica restarts, requests fail.

The max_over_time(...[2m]) range keeps the alert tied to recent samples. Without it, the last observed value of 1 can outlive the pod, and the rule keeps matching after that pod is gone.

That PromQL query ran every minute over a two-minute window and created an alert after one matching run.

Why did the last container stop?

The alert shows what Kubernetes is doing now. It does not show how the previous container ended:

PROMQL
  max by (namespace, pod, container) (
    kube_pod_container_status_last_terminated_reason{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      reason="OOMKilled"
    }
  ) == 1

A result of 1 for the same namespace, pod, and container means the last recorded exit was out of memory. Kube-state-metrics keeps that last reason as a gauge, so the value can stay visible after recovery. It points the investigation at memory; it does not prove that every restart in the window was an OOM kill.

Is the failure repeating?

A single restart can still be transient. The restart counter shows whether the failure keeps happening:

PROMQL
  max by (namespace, pod, container) (
    increase(
      kube_pod_container_status_restarts_total{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api"
      }[10m]
    )
  )

increase() shows how much the restart counter rose over the selected range. Repeated increases during the incident window explain why Kubernetes entered backoff.

How close is memory to the limit?

We need to know how close the container is to its memory limit, and whether that gap collapses right before each restart. This deployment allows only 128MiB, so the next query divides working-set memory by that configured limit:

PROMQL
  max by (namespace, pod, container) (
    container_memory_working_set_bytes{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      image!=""
    }
  )
  /
  max by (namespace, pod, container) (
    container_spec_memory_limit_bytes{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      image!=""
    }
  )

The chart shows a repeating sawtooth: memory approaches 90% of the limit, drops when the process stops, and climbs again after each restart.

Working set is the better signal here than total usage. Total usage includes reclaimable file cache, so it can sit near the limit without a kill. Working set is closer to the memory that triggers OOMKilled for this workload.

Is the node under memory pressure?

OOMKilled can mean the container hit its own limit, or the node ran low on memory and Kubernetes started reclaiming. To separate those cases, first find which node runs the pod, then check whether that node (or any peer) reported MemoryPressure.

PROMQL
  max by (namespace, pod, node) (
    kube_pod_info{
      namespace="checkout",
      pod=~"checkout-api-.*"
    }
  ) == 1

The pod sits on ip-10-0-2-18.ec2.internal. That is the node whose MemoryPressure result matters most for this incident:

PROMQL
  max by (node) (
    max_over_time(
      kube_node_status_condition{
        condition="MemoryPressure",
        status="true"
      }[30m]
    )
  )

Every node returns 0, including ip-10-0-2-18.ec2.internal. So the host was not under node-wide memory pressure. The kill came from the container limit itself.

Does raising the limit clear the alert?

checkout-api was healthy, then began building an in-memory cache that grows to 200MiB in 10MiB steps. The container only allows 128MiB, so the process is killed with OOMKilled before that cache is fully allocated.

We will raise the memory limit to 512MiB so the 200MiB cache fits with room for the runtime, then check whether CrashLoopBackOff clears:

kubectl set resources deployment/checkout-api -n checkout --limits=memory=512Mi

The same waiting-reason query then stops matching. CrashLoopBackOff drops off:

PROMQL
  max by (namespace, pod, container) (
    max_over_time(
      kube_pod_container_status_waiting_reason{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api",
        reason="CrashLoopBackOff"
      }[2m]
    )
  ) == 1

And the alert that started this investigation? Gone.

That is how you detect and investigate a Kubernetes CrashLoopBackOff with PromQL: from the firing alert, through OOMKilled and the limit mismatch, to a recovered alert. Elasticsearch holds the metrics; Kibana runs the same PromQL queries you already know from Prometheus.

Try it

1. Open an Observability project on Elastic Cloud Serverless, or use Elastic Stack 9.4 or later.
2. In the ES|QL editor in Kibana, run the waiting-reason query against a workload you care about.
3. Follow the same path from that alert to termination reason, restarts, memory versus the limit, and recovery.

For more PromQL in Elastic, see Elasticsearch supports PromQL, PromQL queries in Kibana, and Investigate Kubernetes infrastructure with PromQL.

If you've been an SRE for any length of time, you know what usually comes next. You acknowledge the page, open your observability tool, and start the process: pull up the cluster dashboard, find the namespace, find the pod, check restart counts, pivot to logs, check whether an upstream dependency is degraded, compare against last week, and slowly assemble a picture from a dozen tabs. It works, but it's a process, and the process is where the minutes go.

Elastic's new Kubernetes Experience changes the starting point. When the CrashLoopBackOff alert fires, an Investigation Workflow runs automatically alongside it. By the time you open the alert, the evidence has already been gathered and a root-cause hypothesis is waiting for you. Instead of a blank dashboard, you open the alert to an answer. Or at minimum, a strong starting point that tells you exactly where to look next.

This post walks through a typical CrashLoopBackOff scenario end to end. The sections that follow break down what Elastic's UI shows at each step and why it saves you time.

The manual CrashLoopBackOff investigation: six steps every SRE runs

Here's the shape of a normal CrashLoopBackOff investigation, minus Elastic's workflow:

1. You get paged. Something restarted too many times.
2. You orient. Which pod? Which namespace? Which deployment owns it?
3. You characterize. How many restarts? What was the last termination reason — OOMKilled, a failed liveness probe, a bad exit code?
4. You classify. Is this a memory problem, a config problem, a dependency problem, a scheduling problem?
5. You corroborate. Pull Kubernetes events, read the pod logs, check whether an upstream service started erroring first, compare current behavior against a healthy baseline.
6. You conclude. Only now can you form a hypothesis and act.

Every one of those steps is a query, a click, or a context switch. None of them is hard on its own. Together, on a bad night, they're twenty minutes you don't have and they're twenty minutes of the same steps you ran during the last incident, and the one before that.

The insight behind Elastic's Kubernetes Experience is simple: that sequence is deterministic enough to automate. So Elastic automated it.

Alert to Root cause in minutes

Elastic's Kubernetes integration ships pre-built alert rule templates for states that are wrong by definition; no baseline or warmup required. A pod in CrashLoopBackOff is always a problem, so the rule fires the moment the restart count crosses your configured threshold within a rolling window.

Here's the alert firing in this scenario, both the [Kubernetes OTel] Pod CrashLoopBackOff and OOMKilled containers rules light up for the recommendation group in the otel-demo namespace:

The alert itself is defined by an ES|QL query, so it's transparent and tunable; you can read exactly what triggers it and adjust the threshold to match your environment. And its action is what makes the rest of this post possible: the rule runs the K8s CrashLoopBackOff Investigation (OTel) workflow per alert, the instant a new alert fires. (For a refresher on the alert library and how these templates work, see Part 1.)

In this scenario the rule fires on the recommendation pod (recommendation-788dc88c6c-56w74) in the otel-demo namespace. When you open the alert, Elastic's AI Agent is already summarizing what happened; no rule-details spelunking required:

But the alert firing is only half the story. Attached to it is a Kubernetes Investigation Workflow (technical preview), a directed graph of steps that triggers the instant the alert does. While your phone is still buzzing, the workflow is already querying your cluster, branching on what it finds, and synthesizing an answer.

Inside the investigation: what the workflow actually does

The workflow mirrors the exact sequence an experienced SRE would run by hand, except it runs in seconds and writes nothing down that it can't back up with evidence. For our CrashLoopBackOff scenario, here's the path it takes.

Step 1: What does the workflow check first on the crashing pod?

The workflow queries Kubernetes metrics for the restart count, the last termination reason, and utilization against the pod's declared limits.

Result: last termination reason OOMKilled, restart count 7. Memory utilization data happened to be unavailable at query time, but the OOMKilled reason is definitive: the container is being killed by the kernel for exceeding its memory limit on each startup, then immediately restarting.

The OOMKilled termination reason determines everything that follows. The workflow branches down the memory-investigation path rather than the log-investigation path it would take for a non-memory crash.

Step 2: How does the workflow tell a memory leak from a load spike?

The ML anomaly check is the step that separates a good investigation from a fast-but-wrong one. OOMKilled does not automatically mean "memory leak." Rather than recompute memory trends from scratch, the workflow queries the ML anomaly index for an active k8s_pod_memory_growth anomaly on this pod.

Result: no anomaly. The memory spike is flagged as load-driven, not a suspected leak. The ML baseline — established over the preceding days — didn't see the slow, creeping growth trajectory that characterizes a leak. It saw a jump consistent with real traffic.

Distinguishing a load-driven memory spike from a genuine leak would take a human several minutes and a good deal of judgment. The workflow reaches it because Part 1's anomaly detection jobs were already learning the workload's baseline in the background.

Step 3: Is the failure spreading to other Kubernetes services?

A crashing pod is often a symptom, not a cause, and it can also cause problems downstream. So the workflow enumerates the pod's dependencies from APM service_destination aggregates and compares the current error rate and latency against baseline. An AI classification step decides whether the failure is spreading.

Result: the sole direct caller is the frontend service, which is absorbing the impact with just a 0.14% error rate against the recommendation service and no other service exceeds its degradation threshold relative to baseline. The blast radius is isolated to the recommendation service; there's no significant downstream cascade. The problem is local to this pod.

Step 4: Did a recent change in the namespace cause the crash loop?

Finally, the workflow scans the namespace event log. It finds a continuous Pulled → Created → Started → Killing → BackOff cycle running from roughly 18:51 to 18:54 UTC, the textbook signature of an active crash loop at the time the alert fired. Nothing changed operationally; this is a steady-state resource problem.

What the CrashLoopBackOff root-cause hypothesis looks like

When you open the alert, this is what greets you:

ROOT CAUSE HYPOTHESIS (confidence: high)

The recommendation service pod (recommendation-788dc88c6c-56w74) is in a
crash-loop caused by repeated OOMKilled terminations. The pod has restarted
7 times and Kubernetes events confirm a continuous BackOff/restart cycle
since at least 18:51 UTC. Memory utilization data was unavailable at query
time, but the OOMKilled termination reason is definitive: the container is
exceeding its configured memory limit on each startup, being killed by the
kernel, and immediately restarting. No memory leak was detected by ML
anomaly analysis, indicating the memory pressure is load-driven — the
container's memory limit is simply insufficient for the current request
volume. The frontend service (the sole direct caller) is absorbing the
impact with a 0.14% error rate on the recommendation service itself, but
no significant downstream cascade is observed.

EVIDENCE
- Pod restarted 7 times; last termination reason: OOMKilled — container is
  consistently exceeding its memory limit
- ML memory anomaly check: no anomaly found; memory spike assessed as
  load-driven, not a leak
- Blast radius is isolated to the recommendation service; no other service
  exceeds degradation thresholds relative to baseline
- Continuous BackOff events from 18:51–18:54 UTC confirm active crash-loop
  at alert time

PROBABLE CAUSE: The recommendation container's memory limit is too low for
current traffic load, causing repeated OOMKilled terminations and a
crash-loop backoff.

RECOMMENDED NEXT STEPS
1. Immediately increase the memory limit (and request) for the
   recommendation container in its Deployment spec to provide headroom
   above the observed peak usage, then redeploy to break the crash-loop.
2. Profile the recommendation service under representative load to
   determine the actual memory working set and set a right-sized limit
   with a safe buffer (e.g., 20–30% above peak observed).
3. Add a Kubernetes HorizontalPodAutoscaler or VPA policy for the
   recommendation service so memory resources scale with traffic rather
   than requiring manual intervention.

Read that again from the perspective of the on-call engineer. You were paged. You opened the alert. And the alert didn't hand you a pile of logs and a dashboard; it handed you a calibrated root-cause hypothesis with the evidence attached and the next actions spelled out. The six manual steps from the "old way" are done. Your job is now to decide, not to dig.

That's the time savings: not shaving a few seconds off each query, but removing the entire investigative scavenger hunt from the critical path.

How does Elastic Observability avoid misdiagnosing OOMKilled as a memory leak?

Speed is worthless if the answer is wrong, so it's worth noting how the workflow avoids the classic misdiagnoses. It encodes the reasoning an experienced SRE applies instinctively:

• OOMKilled is not automatically a leak. It compares against a 7-day baseline before ever claiming one. Here, that check is what turned "the app is leaking memory" into the correct "the limit is undersized for real load."
• Co-symptoms are not causes. It explicitly checks whether the upstream degraded first before blaming or clearing it.
• Absence of evidence is not evidence. If a query returns zero rows, it reports "no data available" rather than inventing a failure mode.
• It's honest about confidence. The hypothesis is labeled high, medium, or low. When two failure modes fit the evidence, the workflow names both and says which it believes is causal and why. Manufacturing false confidence is treated as a failure of the investigation itself.

This is the same diagnostic protocol encoded in Elastic's observability-k8s-investigation Skill, a failure-mode taxonomy covering 16 distinct Kubernetes failure patterns, from OOMKilled and CPU throttling through scheduling and networking issues. (More on that in Part 2.)

Still want to double-check? Dashboards, Discover, and APM

The workflow gives you the answer. But a good root-cause tool should also make it trivial to verify that answer because sometimes you want to see it with your own eyes, and sometimes the workflow returns medium confidence and you need to close the gap yourself. Everything the workflow reasoned over is available to you directly.

Dashboards — confirm the restart cascade visually. The Kubernetes dashboards are built for drill-down. Start at the cluster Overview, where "top namespaces by container restarts" surfaces the problem at a glance. Click into the flagged namespace, then the pod driving the restarts. The Pods view flags container restarts on the recommendation pod and plots memory against requests and limits over time — you'll see the working set pressing against the limit exactly as the workflow described. It's roughly four clicks from cluster to container.

Discover — read the raw evidence. The pod detail dashboard links directly to correlated pod logs and events in Discover. Here you can confirm the OOMKilled events and the restart cadence in the raw log and event stream, and run your own ES|QL queries if you want to slice the data differently.

APM — verify the blast radius is really contained. The workflow found the impact isolated to the recommendation service, with frontend (the sole caller) absorbing it at a 0.14% error rate. You can confirm that independently in the APM UI: open the service map, check the caller's latency and error rate over the incident window, and compare against the weekly baseline yourself.

The point isn't that you have to do any of this; it's that the workflow's conclusion is fully auditable. Fast when you trust it, transparent when you want to check.

The same investigation from your IDE: the MCP App

Not every investigation starts from a Kibana alert. Sometimes a developer just asks, "why is this service crashing?" from their editor. Elastic's Observability MCP App (technical preview) exposes the same telemetry (and the same investigation workflow) as AI-callable tools that render interactive views inline in your chat or IDE, no context switch to Kibana required.

For our CrashLoopBackOff scenario, the flow looks like this from an MCP-compatible client such as Claude Desktop or VS Code:

"What's broken?" → the cluster health rollup returns an overall health badge, degraded services, top memory consumers, and a Kubernetes breakdown (CPU, memory, restarts, nodes) in one inline view. Here it flags a critical cluster with payment, cart, frontend, and frontend-proxy degraded.

"Is anything anomalous in the recommendation pod?" → the memory analysis view confirms the spike is load-driven, not a leak — the crashing 788dc88c6c pods report null memory (they die too fast to emit a sample) while the healthy pods sit flat at 45MB, exactly the ML result the workflow used.

"Why is the recommendation service crashing?" → the agent returns the same structured root-cause reasoning you'd see on the alert, rendered inline: memory limit set below what the container needs to boot, the ReplicaSet has been intermittently OOMing for weeks, and a concrete mitigation (roll back, then fix the limit), all without leaving the editor.

Same evidence, same root cause, delivered wherever you happen to be working. (For the full set of MCP App views and architecture, see Part 2.)

From paged to decided: skipping the Kubernetes investigation entirely

The change here is not "a better dashboard." It's a shift in what you do when you get paged.

Before, the alert was the start of the investigation. You were notified, and then you went and found the answer. Now, the alert arrives with the investigation already run: evidence gathered, dead ends eliminated, a calibrated hypothesis, and next steps in hand. You go straight from "notified" to "deciding," and you keep the full trail of dashboards, Discover, and APM for whenever you want to verify or dig deeper.

For a single incident that's a few minutes saved. Across a quarter of on-call rotations, across every engineer who no longer re-runs the same six steps at 3 a.m., it's real time back and a lot less alert fatigue.

Try it yourself

You don't need a production incident to see this. The OpenTelemetry Astronomy Shop demo environment ships with a feature-flag service that lets you trigger failure scenarios on demand. Enable a cart/checkout failure, watch the restart cascade unfold, and the CrashLoopBackOff alert rule fires with the investigation workflow running right behind it.

To get set up:

1. Install the Kubernetes integration — dashboards are available immediately. (Part 1: Getting started)
2. Deploy data collection via the EDOT Collector (OpenTelemetry) or standalone Elastic Agent, both Helm-based.
3. Enable the alert rule templates in Observability > Alerts, including CrashLoopBackOff, and connect your notification channel.
4. Let the ML modules warm up for 24–48 hours so anomaly baselines are ready when you need them.
5. Enable the Investigation Workflow (technical preview) — import the Kubernetes Crashloop Investigation Workflow from the Workflows page and configure it to trigger on the alert. (Part 2: Getting started)
6. Install the MCP App (technical preview) on your favorite agentic client to bring investigations into your IDE.

Running Kubernetes on Elastic today? Tell us which investigation steps you still repeat by hand on every incident, and which remediations you'd trust a workflow to propose. Join the Elastic Community Discussion.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The closest option today is to hand-code the mapping in the transform processor:

# otel.yml
processors:
  transform:
    log_statements:
      - context: log
        statements:
          - set(attributes["user.name"], "Alice") where attributes["user.id"] == "user001"
          - set(attributes["user.name"], "Bob") where attributes["user.id"] == "user002"
          - set(attributes["user.name"], "Carol") where attributes["user.id"] == "user003"
          # ...and one more line for every user

That works for a handful of entries, but it does not scale beyond 10 to 20 items. Every new mapping means another statement, the lookup data lives in the same file as your pipeline config, and there is no way to point at reference data that already exists. It proves the need is real, but it only covers the simplest, smallest cases.

If you run the Collector and you have been reaching for the transform processor, a sidecar script, or a downstream ingest pipeline just to add reference data, this component is for you.

Why the OpenTelemetry Collector needed a lookup processor

The Collector was already good at a couple of enrichment patterns. The transform processor reshapes and derives data from what is already on a record. The k8sattributes and resourcedetection processors attach system and environment metadata, like Kubernetes pod details or cloud host information.

What it could not do was look up related data by a key you already have. Three patterns in particular had no home:

• File-based lookups from static reference data in JSON, YAML, or CSV
• HTTP and API-based enrichment from an external service
• DNS lookups, such as resolving an IP to a hostname

These are everyday tasks in other data collectors and transformation tools. Mapping an internal service ID to a friendly name, attaching business metadata by customer ID, or resolving an IP all fall into this category. Without a native component, teams built brittle workarounds or pushed the work downstream where it is harder to reuse.

That gap is what the new lookup processor closes. Elastic's Data Processing team proposed it, the community accepted it, and it was built in partnership with Grafana, thanks to Sam DeHaan (GH: dehaansa). The lookup processor takes a value from your telemetry, uses OTTL to build a lookup key, queries a source such as a YAML file or DNS, and writes the result back as new attributes.

How the OpenTelemetry lookup processor is designed

The design grew out of repeated requests from the community for richer enrichment. A few examples that fed into it:

• Enrich attributes based on key matching from YAML or CSV definition (#40526)
• Enrich telemetry with resource metadata from an inventory datasource (#40936)
• Generic resource detector (#29627)
• Alert Manager receiver and exporter (#18526)
• gRPC processor / connector (#20888)

Rather than build a one-off component for each request, the goal was a single processor flexible enough to cover them. Four ideas shaped it:

• Multiple lookups per processor, so one instance can enrich several attributes in a single pass.
• Caching, so external sources like DNS do not get queried for the same key over and over.
• OTTL for key extraction, so you get a full expression language for pulling the lookup key off a record, including converters.
• Extensible sources, so you are not limited to the built-in set. You can register a custom source for your own data.

Extensible sources matter most for the processor's long-term value. A source is a small, well-defined interface, so the processor is a foundation for enrichment rather than a fixed list of features. YAML and CSV cover static reference data today, DNS covers dynamic resolution, and the same interface leaves the door open for HTTP APIs, key-value stores, or anything specific to your environment.

How the lookup processor evaluates keys with OTTL

Whatever source you configure, the processor runs the same three steps for every record.

First, it evaluates an OTTL expression to produce a lookup key from the record. Second, it hands that key to the configured source and gets a value back. Third, it writes the value to the attributes you name, on the record or on its parent resource. When a key has no match, the processor writes a configurable default so downstream queries stay predictable.

The next two sections walk through the two sources available today.

File-based lookups with YAML in the OpenTelemetry Collector

The most common case is static reference data. You keep a mapping file next to the Collector and enrich records against it. Here the processor reads a YAML file and adds user.name to each log based on its user.id.

# otel.yml
processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/mappings.yaml
    lookups:
      - key: log.attributes["user.id"]
        attributes:
          - destination: user.name
            default: "Unknown User"

The mapping file is a plain set of key-value pairs:

# /etc/otel/mappings.yaml
user001: "Alice"
user002: "Bob"

The key field is an OTTL value expression, so log.attributes["user.id"] reads the user.id attribute off the log record. The destination is where the looked-up value lands, and default is what gets written when the key is missing from the file.

Given this incoming log record:

{
  "body": "User logged in",
  "attributes": {
    "user.id": "user001",
    "http.method": "POST"
  }
}

The processor looks up user001, finds Alice, and produces:

{
  "body": "User logged in",
  "attributes": {
    "user.id": "user001",
    "user.name": "Alice",
    "http.method": "POST"
  }
}

The original attributes stay intact and the enriched value is added alongside them. The CSV source works the same way for teams that keep reference data in spreadsheets or exports rather than YAML.

DNS lookup enrichment inside the OpenTelemetry Collector

Static files are one thing, but some enrichment requires live, changing data. Resolving an IP address to a hostname is the classic example, and it is the first dynamic source the processor supports. Instead of a file, you point it at a DNS server.

# otel.yml
processors:
  lookup:
    source:
      type: dns
    lookups:
      - key: log.attributes["client.ip"]
        attributes:
          - destination: client.hostname
            default: "Not found"

The shape of the config is identical to the YAML example. Only the source changed. The processor pulls client.ip off the record, asks a DNS resolver to reverse-resolve it, and writes the hostname back.

Given this incoming record:

{
  "body": "User logged in",
  "attributes": {
    "client.ip": "8.8.8.8",
    "http.method": "POST"
  }
}

The DNS source resolves 8.8.8.8 and produces:

{
  "body": "User logged in",
  "attributes": {
    "client.ip": "8.8.8.8",
    "client.hostname": "dns.google",
    "http.method": "POST"
  }
}

Because DNS queries hit an external system, this is where caching earns its place. The processor keeps results in an in-memory cache so a stream of records sharing the same IP does not turn into a stream of identical DNS queries. That keeps latency down and avoids hammering your resolver.

Writing a custom lookup source

The built-in sources cover common cases, but the real design goal was extensibility. A source is a small contract: you implement a lookup function that takes a string key and returns a value. The processor takes care of OTTL key evaluation, caching, defaults, and writing attributes, so a custom source only has to answer the question "what value goes with this key?"

That means if you already run an internal metadata API, a Redis cache, or a custom database, you can wire it in as a source and reuse everything else the processor provides. HTTP-based sources and key-value stores are natural fits, and they are on the roadmap precisely because the interface makes them straightforward to add.

How Elastic plans to use this lookup processor

Elastic builds its Collector distributions on OpenTelemetry Collector Contrib. The plan is to include lookup processor so the enrichment work teams do today with brittle workarounds can run inside the pipeline instead of downstream.

Two patterns stand out. The first is reference-data enrichment: turning an internal service ID into a friendly name, or attaching metadata such as a team or customer by ID, so telemetry arrives already labeled for search and correlation. The second is DNS resolution: turning a client.ip into a hostname before the data lands, which matters for network and security telemetry where you want the name rather than the raw address.

Doing this in the Collector keeps reference data close to where telemetry is processed and avoids duplicating the logic in separate ingest steps. As the source interface grows to cover HTTP APIs and key-value stores, the same processor can back richer enrichment without changing how pipelines are configured.

Lookup processor roadmap and how to contribute

The main implementation of the lookup processor has merged upstream into OpenTelemetry Collector Contrib. The core processor and YAML source landed first, and the DNS source followed as the first dynamic lookup.

There is a healthy backlog of work ahead, and contributions are welcome:

• An HTTP lookup source for enrichment from external APIs.
• More DNS capabilities, including A and AAAA queries and support for multiple DNS servers.
• Component telemetry, so you can observe cache hit and miss rates, lookup latency, and error rates.
• Performance improvements as real-world usage grows.

If you want to try it, add the processor to a Collector build that includes Contrib, point a YAML source at a mapping file, and enrich a real record. Then open an issue or pull request on the OpenTelemetry Collector Contrib repository. The component is community-owned, and the more sources and feedback it gets, the more useful it becomes.

To go deeper, read the lookup processor README and the enrichment tracking issue. For more on how Elastic builds on OpenTelemetry, browse the OpenTelemetry articles on Elastic Observability Labs.

Availability

This is available in serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

What you need for RED metrics breakdown in Elastic Observability

You need trace data from a service instrumented with any method Elastic APM supports.

• Application instrumentation: one of the following:
• Elastic APM agents for Java, .NET, Node.js, Python, PHP, Ruby, Go, and other supported languages.
• OpenTelemetry SDKs sending OTLP via Elastic Agent or an upstream OpenTelemetry Collector with the elasticapm connector under connectors (not processors).
• Useful attributes: breakdown works best when spans include the dimensions you want to compare (k8s.pod.name, k8s.deployment.name, service.version, and others). You can also declare custom attributes on spans: add labels to transactions and spans with Elastic APM agents, or set OpenTelemetry attributes on spans and resources with OpenTelemetry SDKs. Those custom fields work as breakdown dimensions too.
• Backend: Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

How to go from a RED metrics alert to the root-cause error log

Step 1: Review RED metrics on the alert detail page

When you receive a notification for a RED metric threshold breach, if you open the alert detail page, you can review the symptoms for the impacted service on one page.

In our example, failed transactions have clearly increased for the cart service, so we want to understand what is driving that RED metric change:

Step 2: Break down RED metrics by span attributes in Discover

To investigate why a RED metric changed, open Traces in Discover and use the new breakdown feature to split RED metrics by any attribute on your spans. In our example, we're going to check Kubernetes attributes and service version, but you could break down by any span attribute you send (e.g. cloud.region, cloud.availability_zone, or container.id).

Each breakdown shows which attribute values moved the metric, so you can see whether the problem is isolated to one pod, deployment, version, or whatever dimension you split on.

In our example, error rate clusters on a single Kubernetes deployment, which points the investigation at a release. We will break down by service.version to validate our hypothesis:

Step 3: Open the trace waterfall and read the linked error log

Once trace breakdown has identified a specific service version as the likely cause, we can filter by that service.version and look at sample failing spans to see if they explain why the version is causing failures.

Open the trace waterfall for one failing span and follow through to the linked error log.

In our example, the error log points to bad configuration that could be causing the issue. Either way, we have narrowed the investigation to a solid hypothesis we can act on in just a few clicks:

From RED metrics alert to error log in Elastic Observability

From a RED metric alert, you can review the symptomatic service, break down Traces in Discover by any attribute on your spans, and open a failing span's trace waterfall to reach the error log in just a few clicks.

What is Tail Sampling?

Distributed tracing is useful for debugging, but at production scale it comes with processing overhead and storage costs, at which point sampling becomes a natural way to maintain the value of tracing while keeping costs under control. Tail-based sampling, or tail sampling, is a technique that makes a sampling decision conditionally at a later stage, so that high-value traces like errors or slow transactions are more likely to be sampled. The opposite is head sampling, which makes the decision at the start of a trace, before any such information is available.

How does the tail sampling processor work?

The tail sampling processor buffers 100% of incoming traces (or spans, used interchangeably), then forwards the sampled subset after applying the sampling policies. Buffering is a major source of memory usage, and it scales proportionally to the volume of spans, a well known pain point in the community.

Memory usage is bounded by configuration parameters like decision_wait and num_traces. Setting decision_wait to 1 minute means a sampling decision is made for a trace after 1 minute, during which all spans for that trace are expected to have arrived. If a trace is slower than 1 minute, the decision is made with some spans missing.

As a side note, scaling out the tail sampling setup involves using the loadbalancingexporter to satisfy the requirement that all spans for a trace must be routed to the same collector. This introduces some operational complexity and potentially data loss during collector restarts. But this post focuses on the memory usage of a single tail sampling processor instance, regardless of horizontal scaling.

Why does tail sampling cause memory pressure?

These parameters introduce a tradeoff between data loss and memory usage, and they require assumptions about the shape of traces: how slow they can be, how many spans they contain, how large each span is. These assumptions can become stale as instrumentation evolves.

How much data loss is acceptable to limit memory usage, and can the tradeoff be improved? The following two contributions aim to give operators more flexibility.

How span-ingest reduces tail sampling memory by releasing spans early

sampling_strategy is a new configuration option added to the tail sampling processor in v0.149.0.

sampling_strategy defaults to trace-complete, which matches the original behavior: sampling policies are only evaluated when decision_wait has elapsed, at which point the trace is considered complete. (There is a similar config, decision_wait_after_root_received, for optimization, but it is excluded from this discussion for simplicity.) This means all spans are buffered in memory for roughly decision_wait before being released, regardless of whether a decision could have been made earlier. For example, health check spans that should always be dropped are still held in memory until policy evaluation time.

Alternatively, sampling_strategy can be set to span-ingest, where spans are evaluated individually at ingest time. This allows terminal decisions, specifically drop or sampled, to be made earlier, freeing memory by dropping or exporting all spans buffered so far for that trace before decision_wait elapses. In the health check example, a policy can be configured to drop the entire trace as soon as the root span belongs to a health check. It is worth noting that an unsampled decision, unlike an explicit drop, is not terminal, as it can be overruled by a sampled or drop decision from another span in the same trace, so unsampled traces cannot be released early.

Switching from trace-complete to span-ingest will require policy adjustments, as policies can no longer assume all spans are available at evaluation time. Moreover, not all policy types are supported with the span-ingest strategy.

Disk-backed tail sampling storage with Pebble

Even with span-ingest, all spans are still buffered in memory. As decision_wait is increased to accommodate slow traces and num_traces is increased to limit data loss, the collector will eventually hit its memory limit and get OOM killed, resulting in further data loss.

What if traces were buffered on disk instead, where there is an order of magnitude more capacity? The main drawback is performance: disk throughput and latency, even with SSDs, are at least an order of magnitude slower than memory, so disk writes need to be efficient. For this reason, Pebble, an LSM database, was chosen as the storage backend for its fast write performance. Read performance is less of a concern, as reads only happen for the sampled subset of traces when sampling_strategy is set to span-ingest.

The implementation introduces a TailStorage interface for trace storage operations, and a new tail_storage option in v0.150.0 (behind feature gate processor.tailsamplingprocessor.tailstorageextension) to configure the storage backend. The default in-memory behavior is unchanged, but it is now possible to swap in a different storage backend, like the new pebbletailstorageextension contributed to Collector Contrib.

Tail sampling memory benchmarks: trace-complete vs span-ingest with Pebble

Benchmark setup: OpenTelemetry Demo with fan-out collectors

The following benchmarks were produced by running OpenTelemetry Demo with increased load against a pipe collector, which receives all spans and fans them out to two identical collectors under observation (CUO-A and CUO-B), differing only in their tail sampling configuration. Measurements include pipe collector throughput, spans received, spans sent (sampled), CPU usage, and memory usage.

Benchmark setup diagram

                      demo ns
     +----------------------------------------+
     |  opentelemetry-demo                     |
     |    loadgenerator (locust)               |
     |    services: frontend, cart, ...        |
     |    demo-collector                       |
     +----------------------------------------+
                          |  OTLP/gRPC
                          v
                     chamber ns
     +----------------------------------------+
     |             pipe-collector             |
     |         receive once, fan out          |
     |      exporters: [otlp/a, otlp/b]       |
     +----------------------------------------+
              | OTLP                  | OTLP
              v                       v
     +----------------+      +----------------+
     |     CUO-A      |      |     CUO-B      |
     | tail_sampling  |      | tail_sampling  |
     |   (config A)   |      |   (config B)   |
     +----------------+      +----------------+

Tail sampling processor configurations

CUO-A

config:
  processors:
    tail_sampling:
      sampling_strategy: trace-complete
      decision_wait: 5m
      num_traces: 5000000
      block_on_overflow: true
      decision_cache:
        sampled_cache_size: 10000
        non_sampled_cache_size: 200000
      policies:
        - name: root_1pct
          type: and
          and:
            and_sub_policy:
              - name: root_span_only
                type: ottl_condition
                ottl_condition:
                  error_mode: ignore
                  span:
                    - "IsRootSpan()"
              - name: root_probabilistic
                type: probabilistic
                probabilistic:
                  sampling_percentage: 1.0

CUO-B

CUO-B uses the same tail sampling processor configuration as CUO-A, except it sets sampling_strategy: span-ingest and tail_storage: pebble_tail_storage/main, along with its corresponding pebbletailstorageextension configuration.

extensions:
  pebble_tail_storage/main:
    directory: /var/lib/otelcol/pebble

Memory, CPU and throughput results

The following tables compare trace-complete (CUO-A) against span-ingest with Pebble disk storage (CUO-B) across memory, CPU and throughput. The process RSS, Go heap allocation, and per-process CPU measurements come from OpenTelemetry Collector internal process and runtime metrics, while container working set and container CPU come from Kubernetes cgroup metrics scraped by kubelet/cAdvisor.

Memory (peak over the window)

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | process RSS | 916.4 MiB | 442.7 MiB | -51.7% | | Go heap alloc | 699.3 MiB | 241.7 MiB | -65.4% | | container working set | 763.0 MiB | 282.9 MiB | -62.9% |

CPU (total over the window)

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | per-process CPU | 11.9 core-s | 22.7 core-s | +90.7% | | container CPU | 11.9 core-s | 22.7 core-s | +90.1% |

Throughput (total over the window)

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | spans received | 257,804 | 257,804 | 0.0% | | spans sent | 2,477 | 2,477 | 0.0% |

Tail sampling

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | traces in memory peak | 29,284 | 29,245 | -0.1% | | traces sampled by root_1pct policy | 496 | 496 | 0.0% |

• cuo-a = trace-complete, cuo-b = span-ingest-pebble
• Window: 15m 39s (t+0:00 start, t+10:06 drain start, t+15:39 drain end)

The results show a significant memory reduction when using pebbletailstorageextension with span-ingest, at the cost of increased CPU usage from event serialization and database overhead.

What's next for OpenTelemetry tail sampling

Both sampling_strategy and pebbletailstorageextension are still in their early stages at the time of writing. Feedback and contributions are welcome in the OpenTelemetry Collector Contrib repo. Stay tuned for more improvements.

Elasticsearch now stores Prometheus metrics in the same columnar backend as your logs and traces, at 3.75 bytes per datapoint, with no custom-metric penalty. The graph, the logs, and the traces answer to one query language.

Using an existing Kubernetes cluster (AWS EKS in this example): point Prometheus at Elastic with one config block, see every metric render in Discover with no dashboard to build, run most of your existing PromQL unchanged, find where ES|QL takes you past what PromQL can express, and finish by reading your logs with the same query language.

Nothing about your collection changes. Your scrape configs, relabeling rules, and service discovery carry over as-is. Most of your PromQL comes with you too — see the coverage note in Step 5 for the gaps.

Step 1: get a Prometheus endpoint and an API key from Elastic Observability

You need Elastic Cloud Serverless or Elastic Cloud Hosted. Both expose the Prometheus endpoints with no configuration.

Serverless is the fastest start. Sign in at cloud.elastic.co and create an Observability project. There is nothing to size and nothing to provision.

Elastic Cloud Hosted works the same way for everything in this post, and is the right choice when you need a specific stack version, a specific region topology, or the deployment-level controls that come with a managed cluster.

Find the Prometheus endpoint and create an API key in the UI

For Prometheus endpoint and the API key go to Kibana, click Add data in the left nav.

For Prometheus, scroll to Connect directly to the endpoint at the bottom and select the Prometheus tab.

Two things to copy:

The endpoint. It looks like https://my-observability-project-xxx.ingest.us-west-2.aws.elastic.cloud. Note the .ingest. host. This is not the same host as your Elasticsearch search endpoint or your Kibana URL.

The API key. Click Create key. Copy the value before you close the dialog, because it is not retrievable afterward.

If you want to scope it by hand instead, Open in API keys takes you to the full editor, and the minimum privilege for metrics ingest is:

{
  "ingest": {
    "indices": [
      {
        "names": ["metrics-*"],
        "privileges": ["auto_configure", "create_doc"]
      }
    ]
  }
}

Keep both values. Every remaining step uses them.

Step 2: make sure Prometheus is scraping your Kubernetes cluster

Prometheus should already be scraping your cluster.

If it is not, the shortest path is the kube-prometheus-stack Helm chart, which installs the Prometheus Operator, Prometheus itself, kube-state-metrics, and node-exporter in one command:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

That gives you the metrics you see for the rest of this post:

• cAdvisor container metrics (container_cpu_usage_seconds_total, container_memory_working_set_bytes)
• kube-state-metrics cluster objects (kube_deployment_spec_replicas, kube_daemonset_status_number_ready).

Step 3: configure Prometheus remote write to the Prometheus endpoint in Step 1

Elasticsearch implements the Prometheus Remote Write protocol natively. There is no adapter, no sidecar, and no translation layer. You add one block and the data flows on the next scrape interval.

If you run the Prometheus Operator

The Operator does not read a prometheus.yml you write by hand. It generates one from the Prometheus custom resource, and authorization.credentials there is a reference to a Kubernetes Secret, not an inline value. Create the secret first:

kubectl create secret generic elastic-prometheus \
  --namespace monitoring \
  --from-literal=api_key='YOUR_API_KEY'

Then reference it from values.yaml:

prometheus:
  prometheusSpec:
    remoteWrite:
      - url: "https://my-observability-project-xxxx.ingest.us-west-2.aws.elastic.cloud:443/api/v1/write"
        authorization:
          type: ApiKey
          credentials:
            name: elastic-prometheus
            key: YOUR_API_KEY

And apply it:

helm upgrade prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  -f values.yaml

Configure Prometheus remote write with prometheus.yml

Same thing, in prometheus.yml:

remote_write:
  - url: "https://YOUR_ES_ENDPOINT/_prometheus/metrics/node/eks/api/v1/write"
    authorization:
      type: ApiKey
      credentials: YOUR_API_KEY

If you run Grafana Alloy uss the following configuration

prometheus.remote_write "elasticsearch" {
  endpoint {
    url = "https://YOUR_ES_ENDPOINT/_prometheus/metrics/node/eks/api/v1/write"
    headers = {"Authorization" = "ApiKey YOUR_API_KEY"}
  }
}

How the remote write URL maps to Elasticsearch data streams

You do not name an index anywhere. There is no index field in the payload and no data stream in your remote_write config. Elasticsearch derives the target data stream from the write path and creates it on the first sample. The two path segments after /metrics/ are the dataset and the namespace:

| URL | Data stream | |---|---| | /_prometheus/api/v1/write | metrics-generic.prometheus-default | | /_prometheus/metrics/{dataset}/api/v1/write | metrics-{dataset}.prometheus-default | | /_prometheus/metrics/{dataset}/{namespace}/api/v1/write | metrics-{dataset}.prometheus-{namespace} |

The examples above use /metrics/node/eks/, which writes to metrics-node.prometheus-eks. That is the data stream you will see in Discover in the next step. Use dataset and namespace to separate production from staging, or to give each cluster and each team a data stream with its own retention and downsampling policy.

If you would rather keep a bare /api/v1/write URL, you can route per time series instead: attach data_stream_dataset and data_stream_namespace labels to the series, and they take precedence over the URL path. These two are control fields, so they route the document without being stored in its labels object.

Elasticsearch installs the index template for metrics-*.prometheus-* itself. You do not create templates or mappings.

What Prometheus metrics look like in Elastic Observability

Every Prometheus sample becomes a document. Labels become keyword fields that serve as time series dimensions. The value goes under metrics.<metric_name>:

{
  "@timestamp": "2026-07-02T10:30:00.000Z",
  "data_stream": {
    "type": "metrics",
    "dataset": "node.prometheus",
    "namespace": "eks"
  },
  "labels": {
    "__name__": "container_memory_working_set_bytes",
    "pod": "checkout-7d9f6c4b8-x2kqp",
    "namespace": "oteldemo",
    "container": "checkout",
    "node": "ip-10-0-3-14.us-west-2.compute.internal"
  },
  "metrics": {
    "container_memory_working_set_bytes": 36700160
  }
}

The one gotcha to know now. Elasticsearch infers whether a metric is a counter or a gauge from its name. Names ending in _total, _sum, _count, or _bucket are counters. Everything else is a gauge. That inference is correct for container_cpu_usage_seconds_total and correct for container_memory_working_set_bytes. It is wrong for any metric in your estate that does not follow Prometheus naming convention, and a misclassified metric gets rejected by the function that should accept it: RATE(my_metric::counter) works on counters only. Step 6 shows how to override the inference.

Current limits. Remote Write v1 only. Classic histograms and summaries are supported through their _bucket, _sum, and _count series, each mapped to the right metric type. Native (sparse) histograms and exemplars arrive with Remote Write v2, which is on the roadmap. Staleness markers are not stored or respected, and non-finite values (NaN, Infinity) are dropped silently. See the Prometheus remote write ingest docs for the full list.

Step 4: verify Prometheus metrics are flowing into Elasticsearch

Do not go build a dashboard. Confirm the pipeline first, and Elastic makes that a single command.

In Kibana, open Discover, switch the query editor to ES|QL, and type the name of the data stream you just wrote to:

TS metrics-node.prometheus-eks

That is the whole query. Discover reads the data stream, finds every metric in it, and renders each one as its own time series chart. Fifty metrics, fifty charts, no dashboard to build and no query to write per metric. It reads the metric type and charts each one correctly: gauges as averages, counters as rates, histograms as p95 distributions.

You can also get here without typing anything. Observability → Streams lists every data stream in the cluster. A Time series badge means it is a time series data stream. Click View in Discover and the TS query is filled in for you.

This is your ingest health check. Three things to look for.

• Data is flowing. Recent, continuous values. Not gaps, and not a line that stops an hour ago.
• Values are plausible. Memory in the tens of megabytes for a small container. CPU as a fraction of a core. Network bytes tracking real traffic.
• Coverage is what you expected. If kube_pod_container_status_restarts_total is missing, your kube-state-metrics scrape config is wrong, and you want to know that now rather than when you are building an alert on it.

Widen the time picker before you conclude anything is broken. A 15-minute window over a quiet period makes healthy data look flat.

To list what actually has data rather than what the mapping declares:

TS metrics-node.prometheus-eks | METRICS_INFO | SORT metric_name

The No dimensions selected control above the charts lets you break every chart out by a label: select pod and each chart splits into one series per pod.

Step 5: run PromQL queries on Prometheus metrics in Elastic Observability

If your team writes PromQL, keep writing PromQL. PROMQL is a source command in ES|QL, alongside FROM and TS, and it runs anywhere ES|QL runs: Discover, dashboard panels, and alert rules.

It does not run a separate engine. It parses the expression, resolves each function to its ES|QL equivalent, and builds a TS execution plan, so your PromQL gets the same vectorized, parallel execution as native ES|QL.

Current limits. PROMQL is generally available on Elastic Cloud Serverless and a tech preview on Elastic Stack 9.4, benchmarked at over 80% query coverage against popular Grafana OSS dashboards. The gaps worth knowing before you paste a dashboard in: histogram_quantile is not yet implemented, which matters most because it is how nearly every latency dashboard computes p95; group modifiers (on(...) group_left(...)) and the set operators or, and, and unless are unsupported; and predict_linear, label_join, and label_replace are not yet available. Time buckets also align to fixed calendar boundaries rather than the query start time, so short ranges or large steps can differ slightly from Prometheus. See the PROMQL command reference for the current list.

CPU: per-pod CPU rate with PromQL

The per-second CPU rate across containers, grouped by pod. This is the first thing you look at when something is hot:

PROMQL sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

Broken out by namespace instead, to find which team is burning the cluster:

PROMQL sum by (namespace) (rate(container_cpu_usage_seconds_total[5m]))

Note what came back: a pod column, a step column, and the value column named after the expression itself. That is a normal ES|QL table, which is the whole point and the thing Step 6 builds on.

Memory: working set and RSS with PromQL

Working set is the number that matters for OOM risk. It is what the kernel counts against the limit, and it is not the same as total allocated memory:

PROMQL sum by (pod) (container_memory_working_set_bytes)

Resident set, for comparison, when you are trying to tell a real leak from page cache:

PROMQL sum by (pod) (container_memory_rss)

Network: receive rate by pod with PromQL

PROMQL sum by (pod) (rate(container_network_receive_bytes_total[5m]))

Cluster objects: replica counts with PromQL

Deployments that are not running the replica count they declare:

PROMQL kube_deployment_spec_replicas

One nicety worth calling out: in Prometheus every query needs an explicit start, end, and step. In Kibana you drop all three. The date picker supplies the range and Kibana derives the step, which is why every query above is a single line.

Build a Kibana dashboard from PromQL queries

Every query above is a dashboard panel. In Discover, click Save, or go to Dashboards → Create → Add panel → ES|QL and paste the query in. The date picker drives start, end, and step, so a panel written once works at every zoom level.

Four panels, four one-line PromQL queries, no translation step. If you are coming from Grafana, this is the same dashboard you already have, rebuilt in about five minutes. If you would rather not rebuild it at all, keep Grafana and point its existing Prometheus datasource at Elasticsearch. That is covered at the end of the post.

Step 6: query Prometheus metrics with ES|QL, beyond what PromQL can express

The PROMQL command in ES|QL compiles to TS. These two queries are equivalent:

PROMQL sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

TS metrics-node.prometheus-eks
| WHERE TRANGE(1h)
| STATS SUM(RATE(metrics.container_cpu_usage_seconds_total, 5m)) BY labels.pod, TBUCKET(1m)

The second form is where metrics stop being a separate world from logs and traces — and where the real joins happen.

Top-N and filtering

A TS query returns a normal ES|QL table, so SORT, LIMIT, and WHERE all work downstream. Top-N needs no special function, and no topk. The ten pods using the most memory are a SORT and a LIMIT:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL
| STATS mem = SUM(metrics.container_memory_working_set_bytes) BY `labels.pod`
| SORT mem DESC
| LIMIT 10

Filtering is the same move, a WHERE on the aggregated column. Only the pods over 50 MB:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL
| STATS mem = SUM(metrics.container_memory_working_set_bytes) BY `labels.pod`
| WHERE mem > 50000000
| SORT mem DESC

You actually can pipe the results of a PROMQL query and post-process with regular ES|QL.

Memory usage against the request

A useful question during a memory scare: which pods are using more than they requested, and by how much. That combines two metrics, and ES|QL expresses it in a single pass with filtered aggregations, one MAX per metric:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL AND `labels.namespace` IS NOT NULL
| STATS
    used_memory      = MAX(metrics.container_memory_working_set_bytes),
    requested_memory = MAX(metrics.kube_pod_container_resource_requests)
                       WHERE `labels.resource` == "memory"
  BY `labels.pod`, `labels.namespace`, time_bucket = TBUCKET(5 minute)
| EVAL pct_of_request = 100 * used_memory / requested_memory
| WHERE pct_of_request > 80
| SORT pct_of_request DESC
| LIMIT 100

The WHERE attached to requested_memory is a filtered aggregation: kube_pod_container_resource_requests carries both CPU and memory under a labels.resource dimension, and the filter keeps only the memory rows, so the ratio is memory over memory. Each metric lives in its own documents; the shared BY key lands both aggregates on one row.

Step 7: query Prometheus metrics and logs together with ES|QL

Over the last few sections we used ES|QL and PromQL to explore metrics. ES|QL reads logs too. Here is a quick query against the OpenTelemetry demo running on this cluster:

FROM logs-*
| WHERE TRANGE(30m) AND kubernetes.pod.name == "checkout-9656cbd88-fsr9v"
| STATS events = COUNT(*) BY log.level, TBUCKET(1m)
| SORT events DESC

Same editor, same date picker you used for metrics. Only the source command changed, from TS to FROM, and now you are reading log volume by level per minute. One query language across metrics and logs, no tab switch and no second tool.

What's running in Elastic Observability now

Prometheus is writing to Elastic with one config block. Every metric renders in Discover with no dashboard built. Most of your existing PromQL runs unchanged, and ES|QL takes you further: top-N, filtering, and cross-metric ratios you can pipe into the rest of the language. Metrics and logs answer to the same query in the same window.

You did not drop a single label to get here, and you are not being billed for cardinality.

Next steps: Grafana, dashboard migration, and OpenTelemetry

1. Keep Grafana if you want it. Elasticsearch exposes a Prometheus-compatible read API at <endpoint>/_prometheus. Point Grafana's existing Prometheus datasource at it, set httpMethod: GET on the datasource, and your PromQL dashboards keep working. Keep Grafana, replace Mimir.
2. Migrate your Grafana dashboards. When you are ready to move off Grafana rather than point it at Elasticsearch, the Observability Migration Platform translates Grafana dashboards, panels, and alert rules into Kibana-native equivalents. It is a source-agnostic CLI (obs-migrate) that converts what it can and flags what needs a human, with a migration report showing what translated cleanly and where semantic gaps remain, so nothing is silently dropped. The walkthrough covers a Grafana and Datadog migration end to end.
3. Want to add OpenTelemetry? Read Part 2 if you are also running OpenTelemetry, or if you would rather collect with an OTel Collector than with Prometheus. Both land in the same store and the same queries read across both.

Start a free trial: cloud.elastic.co/registration Docs: elastic.co/docs/solutions/observability

Frequently asked questions

How do I send Prometheus metrics to Elasticsearch? Add a remote_write block to your Prometheus configuration pointing to https://<YOUR_ES_ENDPOINT>/_prometheus/metrics/{dataset}/{namespace}/api/v1/write with an Authorization: ApiKey <YOUR_KEY> header. Elasticsearch implements the Prometheus Remote Write v1 protocol natively — no adapter or sidecar required.

Can I run existing PromQL queries in Elasticsearch? Most of them, yes. ES|QL includes a PROMQL source command that accepts standard PromQL expressions, benchmarked at over 80% coverage against popular Grafana OSS dashboards. It is generally available on Elastic Cloud Serverless and a tech preview on Elastic Stack 9.4. histogram_quantile, predict_linear, label_join, and label_replace are not yet implemented, and group modifiers and the set operators or, and, and unless are unsupported.

Does Elasticsearch charge based on Prometheus metric cardinality? No. Elasticsearch stores Prometheus metrics at 3.75 bytes per datapoint with no cardinality-based billing and no custom-metric penalty, regardless of how many unique label combinations your metrics produce.

How do I route Prometheus metrics to different data streams in Elasticsearch? The two path segments after /metrics/ in the Remote Write URL set the dataset and namespace. /metrics/node/eks/api/v1/write writes to metrics-node.prometheus-eks. You can also route per time series using data_stream_dataset and data_stream_namespace labels, which take precedence over the URL path.

What Prometheus metric types does Elasticsearch support? Elasticsearch supports Remote Write v1, including counters, gauges, classic histograms, and summaries. Counter vs. gauge classification is inferred from the metric name: names ending in _total, _sum, _count, or _bucket are counters; everything else is a gauge. Native histograms and exemplars require Remote Write v2, which is on the roadmap.

Can I query Prometheus metrics and logs together in Elasticsearch? Yes. ES|QL reads both time series metrics and logs in the same query editor with the same date picker. Switch from TS metrics-node.prometheus-eks to FROM logs-* — same syntax, same window, no second tool.

Related reading

• Elasticsearch: best-in-class for logs, now best-in-class for metrics
• Ship Prometheus metrics to Elasticsearch with Remote Write
• Query Prometheus metrics in Elasticsearch with native PromQL support
• Don't leave metrics on the table: query them with the ES|QL TS command
• Elasticsearch as a backend for Grafana
• How we rebuilt Elasticsearch as a columnar metrics engine

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.