What is the Elastic Cloud Managed OTLP Endpoint?

Managed inputs are ingestion frontends that Elastic operates on your behalf. You point a shipper at an Elastic-owned endpoint, authenticate with an API key, and Elastic takes care of receiving, buffering, and indexing the data into your deployment. There is no gateway to size, patch, or monitor, and no backend credentials to distribute to edge agents.

The Managed OTLP Endpoint is the managed input for OpenTelemetry data. It accepts standard OTLP over HTTP and gRPC from any OpenTelemetry SDK or Collector distribution, buffers it durably, and indexes it into your Elasticsearch deployment using the OpenTelemetry data model. It is generally available on both Elastic Cloud Serverless and ECH. For an overview across logs, traces, and metrics, read Now GA: Managed OTLP Endpoint on Elastic Cloud Hosted.

For metrics, clients send OTLP/HTTP requests to the /v1/metrics signal path and authenticate with an Elasticsearch API key that has the event:write privilege for the apm application:

POST https://<managed-otlp-endpoint>/v1/metrics
Authorization: ApiKey <encoded-api-key>

The /v1/metrics client-facing path never changes. This post focuses on what happens after the request is accepted, during the last hop between the managed pipeline and Elasticsearch.

Two ways to ingest OpenTelemetry metrics: the bulk API and the native OTLP endpoint

Elasticsearch offers two ways to receive OpenTelemetry metrics.

The bulk API (/_bulk) is the general-purpose document ingestion API. To use it for OTLP metrics, something upstream must convert each OTLP data point into a JSON document that follows the OpenTelemetry mapping mode, group the documents by target data stream, serialize them as newline-delimited JSON, and send them in bulk requests. In the OpenTelemetry Collector ecosystem, this conversion is handled by the Elasticsearch exporter, and until recently, it was the only option.

The native OTLP endpoint (/_otlp/v1/metrics) accepts OTLP/HTTP Protobuf directly. Elasticsearch decodes the payload, builds the documents itself, and indexes them into time series data streams. Because Elasticsearch sees the original OTLP structure, it can perform these steps directly:

• It reads the aggregation temporality of every data point and stores it as a dimension
• It hashes the shared resource attributes once for all data points in a resource
• It maps exponential histograms to the exponential_histogram field type without an intermediate T-Digest conversion

The following diagram shows the two lanes.

The two paths differ in three ways that matter for users:

• Work distribution. On the bulk path, the managed pipeline converts and serializes documents, then Elasticsearch parses them again. On the native path, the pipeline forwards batched Protobuf and Elasticsearch builds documents once.
• Temporality. The Elasticsearch exporter used by the managed bulk path does not support cumulative histograms and drops those points. The native path preserves temporality and accepts both delta and cumulative histograms.
• Histogram fidelity. The bulk path converts every histogram to a T-Digest style representation. The native path stores exponential histograms natively and converts explicit-boundary histograms into the same exponential representation.

Do not confuse the two paths with the client-facing path. Clients always send to /v1/metrics on the Managed OTLP Endpoint. The service chooses between /_bulk and /_otlp/v1/metrics behind it.

Native OpenTelemetry metrics support by Elastic Stack version

Native OTLP metrics support in Elasticsearch arrived in several steps.

1. Elastic Stack 9.2 introduced the Elasticsearch OTLP/HTTP metrics endpoint as a technical preview.
2. Elastic Stack 9.3 added the exponential_histogram field type as a technical preview.
3. Elastic Stack 9.4 made exponential_histogram generally available and the default mapping for OTLP histograms received on the native endpoint.
4. Elastic Stack 9.5 added metric temporality support as a generally available feature. The native endpoint now stores the temporality of every data point, which is what makes cumulative histograms possible. The same release added OTLP/HTTP endpoints for logs and traces as a technical preview.
5. Elastic Stack 9.5.3 is the current minimum version for native OTLP metrics routing through the Managed OTLP Endpoint on ECH. The routing floor was raised from 9.2.0 because earlier versions can fail native OTLP requests when audit request-body logging processes Protobuf payloads. The 9.5.3 floor includes the required fixes.

The Elasticsearch Prometheus remote write endpoint arrived in Elastic Stack 9.4 and follows the same capability-gating model, but it is out of scope for this post.

Keep two things apart when reading version numbers in this post. The first four milestones are capabilities of Elasticsearch itself, available to anyone who calls /_otlp/v1/metrics directly. The fifth is a routing decision of the Managed OTLP Endpoint, which is stricter than "the endpoint exists" because the managed service has to work reliably for every tenant.

How Elastic Cloud Hosted routes OpenTelemetry metrics

On Serverless, Elastic operates and upgrades the backend, so the Managed OTLP Endpoint always uses the native path and users never see a version matrix.

ECH is different. Users pick and control their Elastic Stack version, and the Managed OTLP Endpoint supports deployments on any version from 9.0 onward. A single multi-tenant service therefore receives metrics for deployments that have no native endpoint at all, deployments that have one but predate the fixes the service relies on, and deployments that are fully ready. It cannot enable the native path unconditionally.

How the Managed OTLP Endpoint routes a metrics request

The following diagram shows what happens to a metrics request for an ECH deployment.

1. The endpoint authenticates the API key and resolves the target ECH deployment.
2. It looks up the deployment's Elasticsearch version from Elastic's own control plane, not from anything the client sends.
3. It derives a small set of capability flags for the target, such as "native OTLP metrics supported", and attaches them to the request as metadata.
4. The request is buffered durably together with its metadata.
5. At the indexing stage, the consumer reads the flags and hands the batch to one of two exporters: the OTLP/HTTP exporter, which forwards Protobuf to /_otlp/v1/metrics, or the Elasticsearch exporter, which converts the batch into documents and posts them to /_bulk.

Because the flags are derived inside Elastic's trust boundary, a client cannot opt a deployment into a path its version does not support. Elastic can also pin specific deployments to the bulk API for compatibility reasons, so the version rule below describes the normal case rather than a guarantee.

Which ingestion path does my deployment use?

| ECH Elastic Stack version | Path the Managed OTLP Endpoint normally uses | | ------------------------- | ------------------------------------------- | | 9.0 to 9.5.2 | Bulk API (/_bulk) | | 9.5.3 or later | Native OTLP endpoint (/_otlp/v1/metrics) |

No client-side change is required. If you already send OTLP metrics to the Managed OTLP Endpoint on ECH, your traffic moves to the native path when you upgrade the deployment to 9.5.3 or later.

What changes when OpenTelemetry metrics move to the native path

Client configuration and data stream routing remain unchanged. Histogram representation and temporality handling change: the native path uses exponential_histogram by default and records temporality as a dimension. Review dashboards, alerts, and queries that depend on the previous histogram representation or aggregate cumulative snapshots. Queries spanning historical histogram fields and newer exponential_histogram fields may need an explicit ::exponential_histogram cast, as described in querying historical data alongside new data.

The distribution of work across the ingestion chain also changes. On the bulk path, the managed pipeline converts every data point into a JSON document, serializes it, and Elasticsearch parses that JSON back into a document and hashes the dimensions of each one to compute its time-series identifier. On the native path, Elasticsearch decodes a compact Protobuf payload once, and because it sees the OTLP resource and scope structure, it can hash the resource attributes once and reuse that partial hash for every data point that shares the resource, instead of re-hashing the full dimension set per document. The pull request that introduced the endpoint names both effects, the binary encoding and the reused partial hashes, as the reasons the native endpoint is more efficient than bulk ingestion of the same data.

There are also capabilities you gain compared with the Elasticsearch exporter currently used by the managed bulk path. The native endpoint records the aggregation temporality of each data point, stores histograms in the exponential_histogram field type, and accepts cumulative histograms. These are differences between the two managed ingestion implementations, not inherent restrictions of the Bulk API. Custom bulk clients can index exponential histogram documents and configure a temporality dimension. The rest of this post is about those differences.

OpenTelemetry histogram types and temporality explained

The most visible difference between the two paths is what happens to histograms, so the rest of this post is a tour through every histogram case.

OpenTelemetry defines two histogram data types. Histogram uses explicit bucket boundaries chosen by the producer. ExponentialHistogram uses exponentially spaced buckets controlled by a scale parameter, so the SDK adapts resolution to the data.

Each type can use one of two temporalities. With delta temporality, every data point covers one collection interval. With cumulative temporality, every data point covers everything since a fixed start time.

That gives four combinations, and the two ingestion paths treat them differently.

| Path | Explicit, delta | Explicit, cumulative | Exponential, delta | Exponential, cumulative | | -------------------------------- | -------------------- | -------------------- | ------------------ | ----------------------- | | Bulk API (ECH 9.0 to 9.5.2) | Supported, converted | Dropped | Supported, converted | Dropped | | Native (ECH 9.5.3 or later) | Supported, converted | Supported, converted | Supported | Supported |

"Converted" means the metric is accepted but stored with a different bucket representation than the one on the wire. On the native path, that representation is the exponential_histogram field type, which is the default mapping for OTLP histograms since 9.4. Cumulative histograms on the native path require that default; if a cluster overrides xpack.otel_data.histogram_field_type to histogram, cumulative histograms are not supported. If Elastic has pinned a deployment to the bulk API for compatibility, the bulk row applies regardless of version.

Each of the four examples below is a complete curl request you can run against your own deployment. The JSON encoding of OTLP/HTTP is verbose but self-documenting, which makes it a good learning tool. Production shippers use Protobuf (application/x-protobuf) instead.

Prerequisites: your endpoint URL and API key

In the Elastic Cloud Console, find your deployment under Hosted deployments and select Manage. In Application endpoints, cluster and component IDs, select Managed OTLP and copy the public endpoint.

Then open the API keys management page in Kibana and create an API key with the event:write privilege for the apm application. Use the Encoded value of the key. It is already base64-encoded in the id:api_key format that the Authorization header expects.

export MANAGED_OTLP_URL="https://your-endpoint.elastic-cloud.com"
export ELASTIC_API_KEY="your-encoded-api-key"

# Nanosecond-precision timestamps: T0 = 2 min ago, T1 = 1 min ago, T2 = now
T0="$(python3 -c 'import time; print(time.time_ns() - 120_000_000_000)')"
T1="$(python3 -c 'import time; print(time.time_ns() -  60_000_000_000)')"
T2="$(python3 -c 'import time; print(time.time_ns())')"

All four examples report the same metric, http.server.request.duration in seconds, for the same service. Each has a distinct example.case resource attribute so the examples form separate time series and can be queried independently. The explicit and exponential examples describe different latency distributions; each cumulative example contains two snapshots whose difference matches its delta counterpart. Run the timestamp setup again before repeating the examples to avoid resending points with the same dimensions and timestamp.

Explicit histogram with delta temporality (case 1)

This is the case that works on every supported ECH version, so it is the right place to learn the shape of an OTLP histogram.

A histogram distributes individual measurements across buckets. For an explicit histogram, you choose the bucket boundaries in advance. For example: 5 ms, 10 ms, and 25 ms, and the SDK counts how many requests fell into each bucket. Each data point carries the bucket counts, the total count, and the sum, which is enough to estimate percentiles at query time.

Percentiles are what you want for latency. An average of 50 ms hides the fact that 1% of requests take 5 seconds, while a histogram shows p50, p95, and p99 side by side.

curl --fail-with-body --silent --show-error \
  -H "Authorization: ApiKey ${ELASTIC_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary '{
    "resourceMetrics": [
      {
        "resource": {
          "attributes": [
            {"key": "service.name", "value": {"stringValue": "my-service"}},
            {"key": "example.case", "value": {"stringValue": "explicit-delta"}}
          ]
        },
        "scopeMetrics": [
          {
            "scope": {"name": "http"},
            "metrics": [
              {
                "name": "http.server.request.duration",
                "unit": "s",
                "histogram": {
                  "aggregationTemporality": 1,
                  "dataPoints": [
                    {
                      "startTimeUnixNano": "'"${T1}"'",
                      "timeUnixNano":      "'"${T2}"'",
                      "count": 100,
                      "sum": 12.5,
                      "explicitBounds": [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
                      "bucketCounts":   [2,     5,    10,    20,   30,  18,   10,  4,   1,  0],
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }' \
  "${MANAGED_OTLP_URL}/v1/metrics"

aggregationTemporality: 1 means delta. The data point covers the one-minute window from startTimeUnixNano to timeUnixNano and nothing before it.

explicitBounds lists nine upper boundaries, so bucketCounts has ten entries. The last entry is the overflow bucket for values above 2.5 seconds.

A successful response is HTTP 200 with an empty or absent partialSuccess object, meaning the managed endpoint durably accepted the data for processing. It does not confirm that Elasticsearch indexed the points: unsupported cumulative histograms can still be dropped on the managed bulk path, and indexing can fail downstream. Use the queries below to verify ingestion. See indexing errors and the failure store for troubleshooting.

How Elastic stores it. Both paths accept this payload, and both convert it. On the bulk path, the Elasticsearch exporter turns the buckets into a T-Digest style histogram field. On the native path, Elasticsearch converts the explicit boundaries into an exponential_histogram. In both cases percentile precision is bounded by the boundaries you chose on the producer side, because the conversion cannot recover detail that the original buckets did not have.

Explicit histogram with cumulative temporality (case 2)

This case uses the same metric and bucket boundaries as case 1, with cumulative counts and a fixed start time. This is the case that separates the two paths most sharply.

Cumulative temporality is the default in most OpenTelemetry SDKs and the model Prometheus users know. Every data point reports the totals since the process started, so the counts only grow and a consumer computes the rate over a window by subtracting two points. It is resilient to lost data points, because the next point still carries the full state.

curl --fail-with-body --silent --show-error \
  -H "Authorization: ApiKey ${ELASTIC_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary '{
    "resourceMetrics": [
      {
        "resource": {
          "attributes": [
            {"key": "service.name", "value": {"stringValue": "my-service"}},
            {"key": "example.case", "value": {"stringValue": "explicit-cumulative"}}
          ]
        },
        "scopeMetrics": [
          {
            "scope": {"name": "http"},
            "metrics": [
              {
                "name": "http.server.request.duration",
                "unit": "s",
                "histogram": {
                  "aggregationTemporality": 2,
                  "dataPoints": [
                    {
                      "startTimeUnixNano": "'"${T0}"'",
                      "timeUnixNano":      "'"${T1}"'",
                      "count": 150,
                      "sum": 18.5,
                      "explicitBounds": [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
                      "bucketCounts":   [3,     7,    15,    30,   45,  27,   15,  6,   2,  0],
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    },
                    {
                      "startTimeUnixNano": "'"${T0}"'",
                      "timeUnixNano":      "'"${T2}"'",
                      "count": 250,
                      "sum": 31.0,
                      "explicitBounds": [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
                      "bucketCounts":   [5,     12,   25,    50,   75,  45,   25,  10,  3,  0],
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }' \
  "${MANAGED_OTLP_URL}/v1/metrics"

aggregationTemporality: 2 means cumulative. Both points retain T0 as the fixed start of the series. The first reports 150 requests through T1; the second reports 250 through T2. Subtracting their counts, sums, and corresponding buckets gives the 100-request distribution for T1 to T2 shown in case 1.

How Elastic stores it. On the native path Elasticsearch stores the point as an exponential_histogram and records cumulative in the temporality dimension. Time series functions in ES|QL use that dimension to compute per-interval rates from consecutive cumulative points.

On the bulk path this data point is dropped. The managed bulk path's Elasticsearch exporter does not support cumulative histograms; treating those buckets as delta would produce incorrect interval distributions. If your ECH deployment is on 9.0 to 9.5.2, configure your SDK or Collector to export histograms with delta temporality, or put a cumulativetodelta processor in front of the export.

Exponential histogram with delta temporality (case 3)

Now switch the data type while keeping delta temporality.

An exponential histogram removes the need to pick boundaries. Its buckets are exponentially spaced and controlled by a single scale parameter, and the SDK lowers the scale automatically when the data range grows. That is why exponential histograms are the recommended default for latency SLOs where p99 and p99.9 accuracy matters: the resolution is relative to the value, so the tail is captured as precisely as the body.

curl --fail-with-body --silent --show-error \
  -H "Authorization: ApiKey ${ELASTIC_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary '{
    "resourceMetrics": [
      {
        "resource": {
          "attributes": [
            {"key": "service.name", "value": {"stringValue": "my-service"}},
            {"key": "example.case", "value": {"stringValue": "exponential-delta"}}
          ]
        },
        "scopeMetrics": [
          {
            "scope": {"name": "http"},
            "metrics": [
              {
                "name": "http.server.request.duration",
                "unit": "s",
                "exponentialHistogram": {
                  "aggregationTemporality": 1,
                  "dataPoints": [
                    {
                      "startTimeUnixNano": "'"${T1}"'",
                      "timeUnixNano":      "'"${T2}"'",
                      "count": 100,
                      "sum": 62.2,
                      "scale": 3,
                      "zeroCount": 0,
                      "positive": {
                        "offset": -10,
                        "bucketCounts": [2, 5, 10, 20, 30, 18, 10, 4, 1]
                      },
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }' \
  "${MANAGED_OTLP_URL}/v1/metrics"

Reading the bucket layout takes a moment the first time.

The base of the histogram is 2^(2^-scale). With scale: 3 that is about 1.09, so each bucket is 9% wider than the previous one.

Bucket i in bucketCounts covers the range (base^(offset+i), base^(offset+i+1)], lower bound excluded and upper bound included. With offset: -10 the first bucket starts at about 0.42 seconds and the ninth ends at about 0.92 seconds.

zeroCount counts measurements that are exactly zero or within the zero threshold, and a negative bucket range exists for negative values, which latency never uses.

How Elastic stores it. On the native path Elasticsearch stores the scale, offset, and counts as an exponential_histogram without conversion. This is the highest-fidelity case: what you measured is what you query.

On the bulk path the Elasticsearch exporter converts the exponential buckets into a T-Digest style histogram. The data point is accepted and percentiles remain usable, but the original scale and bucket layout are not preserved.

Exponential histogram with cumulative temporality (case 4)

The last case combines the adaptive bucket layout of case 3 with the cumulative semantics of case 2. An SDK configured for exponential histogram aggregation and cumulative export temporality produces this shape. The OpenTelemetry SDK default aggregation for histogram instruments is explicit buckets, so selecting exponential aggregation is a separate configuration choice.

curl --fail-with-body --silent --show-error \
  -H "Authorization: ApiKey ${ELASTIC_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary '{
    "resourceMetrics": [
      {
        "resource": {
          "attributes": [
            {"key": "service.name", "value": {"stringValue": "my-service"}},
            {"key": "example.case", "value": {"stringValue": "exponential-cumulative"}}
          ]
        },
        "scopeMetrics": [
          {
            "scope": {"name": "http"},
            "metrics": [
              {
                "name": "http.server.request.duration",
                "unit": "s",
                "exponentialHistogram": {
                  "aggregationTemporality": 2,
                  "dataPoints": [
                    {
                      "startTimeUnixNano": "'"${T0}"'",
                      "timeUnixNano":      "'"${T1}"'",
                      "count": 150,
                      "sum": 93.3,
                      "scale": 3,
                      "zeroCount": 0,
                      "positive": {
                        "offset": -10,
                        "bucketCounts": [3, 7, 15, 30, 45, 27, 15, 6, 2]
                      },
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    },
                    {
                      "startTimeUnixNano": "'"${T0}"'",
                      "timeUnixNano":      "'"${T2}"'",
                      "count": 250,
                      "sum": 155.5,
                      "scale": 3,
                      "zeroCount": 0,
                      "positive": {
                        "offset": -10,
                        "bucketCounts": [5, 12, 25, 50, 75, 45, 25, 10, 3]
                      },
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }' \
  "${MANAGED_OTLP_URL}/v1/metrics"

How Elastic stores it. On the native path this is stored exactly like case 3, as an exponential_histogram, plus the cumulative temporality dimension. Elasticsearch can then downsample and compute rates over the series correctly. The two points share the same start time and bucket layout; their difference is the delta distribution in case 3.

On the bulk path the data point is dropped, for the same reason as case 2.

Requires Elastic Stack 9.5.3 or later on ECH. On ECH 9.0 to 9.5.2, cases 2 and 4 are dropped. Emit delta histograms until you upgrade.

Query OpenTelemetry histogram percentiles with ES|QL

Sending is only half of the showcase. Open Discover in Kibana, switch to ES|QL mode, and first confirm that the documents arrived:

FROM metrics-*.otel-*
| EVAL duration = metrics.http.server.request.duration::exponential_histogram
| WHERE service.name == "my-service" AND duration IS NOT NULL
| KEEP @timestamp, resource.attributes.example.case, metrics.http.server.request.duration, attributes.http.request.method
| SORT @timestamp DESC
| LIMIT 10

On a native-path deployment the histogram field shows the scale, the bucket indices, and the counts you sent. On a bulk-path deployment you see the converted values and counts arrays instead, which is the visible trace of the conversion.

Then ask the question the histogram was collected for, the request latency percentiles:

TS metrics-*.otel-*
| WHERE service.name == "my-service" AND @timestamp >= NOW() - 5 minutes
| STATS p50 = PERCENTILE(metrics.http.server.request.duration::exponential_histogram, 50),
        p95 = PERCENTILE(metrics.http.server.request.duration::exponential_histogram, 95),
        p99 = PERCENTILE(metrics.http.server.request.duration::exponential_histogram, 99)
    BY resource.attributes.example.case, attributes.http.request.method, TBUCKET(5 minutes)

TS merges histograms per time series and respects their temporality before computing percentiles. For cumulative series, it accounts for the change between consecutive snapshots instead of counting their overlapping observations repeatedly. FROM is useful for inspecting individual documents, but it ignores temporality when aggregating them. See how temporality affects queries.

Grouping by resource.attributes.example.case keeps the four examples separate. The explicit examples place the median in the 0.05 to 0.1 second bucket; the exponential examples place it around 0.6 seconds. These are different input distributions, so their percentile results should differ. The cumulative examples describe the same final one-minute distribution as their corresponding delta examples. Refer to the ES|QL aggregation functions reference for the full list of functions that accept exponential histogram fields.

Summary: OpenTelemetry histogram support on Elastic Cloud Hosted

• ECH deployments on Elastic Stack 9.5.3 or later receive OTLP metrics through the native Elasticsearch endpoint, with no client change.
• Those deployments accept all four histogram cases, and exponential histograms are stored without conversion.
• ECH deployments on 9.0 to 9.5.2 continue through the bulk API and should emit delta histograms.
• Serverless deployments always use the native path.

Clients keep sending to the same /v1/metrics path on the Managed OTLP Endpoint in every case.

Learn more about OpenTelemetry metrics on Elastic

• Elastic Cloud Managed OTLP Endpoint documentation
• Elasticsearch OTLP/HTTP endpoint documentation
• Exponential histogram field type
• Now GA: Managed OTLP Endpoint on Elastic Cloud Hosted

METRICS_INFO, an ES|QL command, reports which metrics and time series exist in the scope of your query, one row each. Discover appends it to your query behind the scenes, builds every metric chart from that single response, and can split those charts by any dimension your data exposes. Gauges are averaged, counters use SUM(RATE()), histograms take a percentile. You never need to know which aggregation a metric requires.

For the design and internals, including the per-series sibling command TS_INFO, see the METRICS_INFO and TS_INFO deep dive.

What METRICS_INFO returns

METRICS_INFO retrieves information about the metrics available in your time series data streams, together with applicable dimensions and other metadata, all scoped to the current TS query.

TS metrics-* | METRICS_INFO

You get one row describing each metric in the query scope:

| metric_name | data_stream | unit | metric_type | field_type | dimension_fields | | ----------------------------------- | --------------------------------- | --------- | ----------- | ---------- | ---------------------------------- | | system.cpu.user.pct | metrics-system.cpu-default | percent | gauge | double | [host.name, cloud.region] | | activemq.broker.connections.count | metrics-activemq.broker-default | null | counter | long | [activemq.broker.mbean,agent.id] |

For syntax, see the METRICS_INFO command. For the design and internals, including the per-series sibling command TS_INFO, see the METRICS_INFO and TS_INFO deep dive.

How Discover turns a TS query into metric charts

Discover provides a dedicated experience for exploring metrics data. When it detects a TS query, it automatically builds an inventory of charts for the metrics available in your data.

Your original query remains unchanged and continues to run as usual. Behind the scenes, Discover derives a second request from it by appending | METRICS_INFO.

For example, if you run:

TS metrics-*
| WHERE `cloud.provider` == "gcp" AND `cloud.region` == "us-central1"

Discover derives a second request behind the scenes:

TS metrics-*
| WHERE `cloud.provider` == "gcp" AND `cloud.region` == "us-central1"
| METRICS_INFO

The METRICS_INFO response is parsed once and becomes the source for the dedicated metrics experience in Discover. From that inventory, Discover can generate charts and provide capabilities such as searching and filtering metrics, breaking them down by dimensions, inspecting the ES|QL query behind each chart, and adding metrics to dashboards.

What each METRICS_INFO column controls

Discover uses one METRICS_INFO response to generate every chart in the inventory. The response tells Discover which metrics to render, where to find their data, how to aggregate their values and how to display them.

Each column in the response plays a specific role:

| Column name | What it controls in the inventory | | ------------------ | ------------------------------------------------------- | | metric_name | Which metric panels are rendered | | data_stream | Where each panel gets its data | | metric_type | How each metric is aggregated | | field_type | The type of the field, e.g., double, long | | unit | How values are formatted | | dimension_fields | Which dimensions can be used to filter and split charts |

Which metric charts get rendered: metric_name

Each entry in the METRICS_INFO response represents a metric that Discover can visualize.

Discover iterates over the parsed response and creates a chart panel for each metric. The inventory therefore reflects the contents of the response directly.

Where each chart gets its data: data_stream

Each chart queries its data independently, even though all metric metadata comes from the same METRICS_INFO request.

The data_stream value determines the source used to construct the chart's ES|QL query.

For example:

TS metrics-system.cpu-default
| STATS AVG(system.cpu.user.pct) BY TBUCKET(100)

Querying each data stream separately matters most for searches that span multiple projects.

When the same metric_name appears in multiple data_stream values, Discover creates a separate panel for each stream. For example, in a serverless environment with cross-project search, each panel queries its own backing data and identifies the stream it represents.

This metric exists in multiple data streams. This chart shows data from metrics-system.cpu-default only.

Discover separates the panels intentionally. Combining results across streams could hide differences between them, which may be important when investigating metrics across projects or environments.

How counter, gauge, and histogram metrics are aggregated: metric_type, field_type

The metric_type tells Lens charts how to aggregate the metric, while the field_type describes the field.

Three metric types are currently supported:

| metric_type | What the chart computes | Example | | ------------- | -------------------------------------------------- | --------------------- | | gauge | AVG(field), representing a point-in-time level | CPU usage | | counter | SUM(RATE(field)), representing a rate of change | Bytes sent per second | | histogram | PERCENTILE(field, p), summarizing a distribution | p99 latency |

How values are displayed: unit

The unit column controls how values are formatted on the Y axis. Lens, the charting library behind each panel, applies the formatting.

For example:

• bytes: 1,024 is displayed as 1 KB.
• percent: 0.75 is displayed as 75%.
• No unit: the raw value is displayed without unit-specific formatting.

How charts are filtered and split by dimension: dimension_fields

The dimension_fields column identifies the dimensions associated with each metric, such as host.name, cloud.region, or service.name.

Discover combines these values across the METRICS_INFO response to populate the dimensions dropdown in the inventory toolbar.

Selecting a dimension affects the inventory in two ways:

1. It filters the inventory. Discover re-runs METRICS_INFO with a condition such as WHERE MV_CONTAINS(dimension_fields, "host.name"), removing metrics that do not support the selected dimension.

2. It splits each chart. Discover adds the selected dimension to the BY clause of each chart's ES|QL query, producing one series for each dimension value.

For example:

TS metrics-system.cpu-default
| STATS AVG(system.cpu.user.pct)
  BY TBUCKET(100), host.name

The result is a chart with a separate series for each host.name, while the inventory ensures that only metrics supporting that dimension are included in the inventory.

When you query a different data stream, any previously selected dimensions that the new stream does not expose are automatically cleared, so the per-chart queries never reference fields that do not exist there.

Try it in Kibana Discover

Visualizing your metrics requires no configuration, no dashboard setup, and no per-metric query to write. One TS query is enough.

Step 1: Ingest metrics data

If you are starting from scratch, you can send Prometheus metrics to Elasticsearch using Prometheus Remote Write. Any Elastic integration that collects system or application metrics works the same way. Once data lands in a TSDB-backed metrics-* data stream, Discover picks it up without any extra setup.

Step 2: Open Discover and run a TS query

1. Open Kibana -> Discover.
2. Switch to ES|QL mode.
3. Type TS metrics-* and run the query.

Step 3: Explore your metric charts

Discover builds the inventory automatically. From there, you can:

• Search for a metric by name to narrow the inventory.
• Select a dimension from the toolbar to split every chart by host.name, cloud.region, or any dimension your data exposes.
• Click a chart panel to open the full ES|QL query behind it.
• Add individual panels to a dashboard.

Your metrics are ready to explore immediately, so you can start investigating your data as soon as you run the query.

See the documentation for how to Explore metrics data with Discover in Kibana.

Prerequisites

To follow along, you need:

• An Elasticsearch cluster with Kibana. Everything up to the ES|QL section works on any recent version; LOOKUP JOIN is generally available in Elasticsearch 9.1 and was a technical preview in 9.0, so use 9.1 or later for the last section.
• No special license tier. Everything in this article, including LOOKUP JOIN, works on the free basic license.
• The two small sample indices created in the next section.

Why checkout latency increased in production

The example starts with a common operations question:

Why did checkout latency increase in production, and which team owns the service?

The metrics index contains 15-minute service measurements for four services across three regions. One service, checkout-api, has higher p95 latency in us-central1 during the investigation window. The goal is to get from all metrics to the small set of documents that explain the issue.

The walkthrough follows these steps:

1. Select the right data view and time range.
2. Use UI filters to include, exclude, disable, and pin criteria.
3. Use KQL for the main field and range search.
4. Switch to Lucene when regular expression syntax is useful.
5. Use ES|QL mode with LOOKUP JOIN to enrich metrics with service catalog data.

Set up the sample metrics index

The walkthrough searches a metrics index named o11y-labs-discover-service-metrics. Create it with keyword fields for the service dimensions and numeric fields for the measurements:

PUT o11y-labs-discover-service-metrics
{
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "service": {
        "properties": {
          "name": { "type": "keyword" },
          "environment": { "type": "keyword" },
          "version": { "type": "keyword" }
        }
      },
      "cloud": { "properties": { "region": { "type": "keyword" } } },
      "host": { "properties": { "name": { "type": "keyword" } } },
      "metrics": {
        "properties": {
          "latency": { "properties": { "p95_ms": { "type": "float" } } },
          "cpu": { "properties": { "pct": { "type": "float" } } },
          "error": { "properties": { "rate": { "type": "float" } } }
        }
      }
    }
  }
}

Each document is one 15-minute measurement for one service in one region:

POST o11y-labs-discover-service-metrics/_bulk
{ "index": {} }
{ "@timestamp": "2026-06-30T16:15:00.000Z", "service": { "name": "checkout-api", "environment": "production", "version": "2026.06.30-1" }, "cloud": { "region": "us-central1" }, "host": { "name": "checkout-api-us-central1-01" }, "metrics": { "latency": { "p95_ms": 582.6 }, "cpu": { "pct": 0.81 }, "error": { "rate": 0.041 } } }
{ "index": {} }
{ "@timestamp": "2026-06-30T16:15:00.000Z", "service": { "name": "payments-api", "environment": "production", "version": "2026.06.29-7" }, "cloud": { "region": "us-central1" }, "host": { "name": "payments-api-us-central1-01" }, "metrics": { "latency": { "p95_ms": 231.4 }, "cpu": { "pct": 0.31 }, "error": { "rate": 0.008 } } }

To reproduce the screenshots, index one document per service, region, and 15-minute interval:

• Services: checkout-api, checkout-worker, payments-api, inventory-api
• Regions: us-central1, us-east4, europe-west1
• Window: 14:00 to 19:45 UTC on June 30, 2026, giving 24 intervals of 15 minutes
• Documents per interval: 12 production, plus two staging (checkout-api and payments-api, both in us-central1)
• Total: 24 intervals × 14 documents = 336 documents

The exact values do not matter, as long as checkout-api in us-central1 reports metrics.latency.p95_ms above 500 between 15:45 and 18:45 UTC and stays well under 500 ms everywhere else.

Instead of indexing everything by hand, you can run the supporting notebook, which generates the full 336-document dataset, creates both indices, and verifies the final ES|QL query.

The ES|QL section also uses a second, four-document lookup index for service catalog data. We will create it when we get there.

Choose a data view in Kibana Discover

The data view is the first filter in Discover. It decides which Elasticsearch indices are searched, which time field drives the histogram, and which fields are available in the left field list.

For this walkthrough, the Discover data view points to:

o11y-labs-discover-service-metrics

The time field is @timestamp. That matters because the time picker limits the documents before you add a query, a filter pill, or a selected field.

Use a narrow data view when you can. For example, a data view that targets only service metrics makes Discover easier to scan than a broad logs-*,metrics-* data view when you already know the question is about metrics.

Once the data view is selected, add the fields that support the investigation:

• service.name
• service.environment
• cloud.region
• metrics.latency.p95_ms
• metrics.cpu.pct
• metrics.error.rate

Filter pills in Discover: include, exclude, disable, and pin

UI filters are useful when you want a visible, editable list of constraints. They are also helpful when you are exploring fields from the document table and want Discover to write the field syntax for you.

In the document table, use the field actions (the +/- icons that appear when you hover over a value) to include or exclude it. For example:

service.environment: production
NOT cloud.region: us-east4
service.version: 2026.06.29-7  (disabled)

These three filters show the main filter controls:

• Include a value when you want only matching documents.
• Exclude a value when a dimension is not part of the problem.
• Temporarily disable a filter when you want to keep it nearby but remove it from the current query.
• Pin a filter when it should stay active as you move between Kibana apps.

Pinned filters are useful for investigations that cross app boundaries. For example, you can pin service.environment: production before moving from Discover to dashboards, Lens, or another view. Disabled filters are useful for testing a theory without deleting the context that got you there.

The key habit is to keep the filters readable. If a query has a long search expression and many hidden assumptions, another engineer has to reconstruct your thinking. Filter pills make the major scope decisions visible.

KQL query syntax for field, range, and boolean searches

KQL, the Kibana Query Language, is a good default for Discover searches. It supports field names, exact values, ranges, wildcards, and boolean logic in a readable form.

For the checkout latency example, this KQL query narrows the view to one service, one region, and high p95 latency:

service.name : "checkout-api" and cloud.region : "us-central1" and metrics.latency.p95_ms >= 500

Read it from left to right:

• service.name : "checkout-api" keeps one service.
• cloud.region : "us-central1" keeps one cloud region.
• metrics.latency.p95_ms >= 500 keeps latency samples at or above 500 ms.

You can add the environment in KQL:

service.environment : "production" and service.name : "checkout-api" and cloud.region : "us-central1" and metrics.latency.p95_ms >= 500

Or you can keep service.environment: production as a UI filter. Both approaches are valid. For shared investigations, we prefer stable scope, such as environment and service, as filter pills, and the active hypothesis, such as a latency threshold, in the search bar.

KQL also works well for combining fields:

service.environment : "production" and
(service.name : "checkout-api" or service.name : "payments-api") and
metrics.error.rate > 0.02

This is useful when a user-facing flow crosses multiple services. You can compare a small group of services without switching data views or creating a dashboard first.

Lucene query syntax in Kibana: searching with regular expressions

Lucene query syntax is the option in Kibana that supports regular expressions. KQL does not, so when you need a regex in the search bar, open the query menu at the right of the search bar and switch the language to Lucene.

For example, this Lucene query searches production services whose names start with checkout- and whose p95 latency is above 500 ms:

service.name:/checkout-.*/ AND service.environment:production AND metrics.latency.p95_ms:>500

Lucene syntax is more compact, but it is also easier to misread. Use it when it gives you something you cannot express as clearly in KQL, such as a regex pattern over a field. For everyday field, value, and range filtering, KQL is usually easier for a teammate to review.

How to join two indices in Discover with ES|QL LOOKUP JOIN

Classic Discover mode is good when you want to search, filter, inspect fields, and look at raw documents. ES|QL in Discover is better when the question needs transformation before the result is useful. Use the Query in ES|QL button in the Discover toolbar to switch modes.

In this example, raw metrics tell us that checkout-api latency is high. They do not tell us who owns that service or what latency target the service is expected to meet. That data lives in a small service catalog lookup index.

Create a lookup index for service catalog data

PUT o11y-labs-service-catalog-lookup
{
  "settings": {
    "index.mode": "lookup"
  },
  "mappings": {
    "properties": {
      "service": {
        "properties": {
          "name": {
            "type": "keyword"
          }
        }
      },
      "owner": {
        "properties": {
          "team": {
            "type": "keyword"
          }
        }
      },
      "slo": {
        "properties": {
          "latency_target_ms": {
            "type": "long"
          }
        }
      },
      "runbook": {
        "properties": {
          "url": {
            "type": "keyword"
          }
        }
      }
    }
  }
}

One catalog document can attach ownership and an SLO target to the service:

POST o11y-labs-service-catalog-lookup/_doc/checkout-api
{
  "service": {
    "name": "checkout-api"
  },
  "owner": {
    "team": "checkout-platform"
  },
  "slo": {
    "latency_target_ms": 350
  },
  "runbook": {
    "url": "https://runbooks.example.com/checkout-api/latency"
  }
}

Run the LOOKUP JOIN query

Now Discover can run an ES|QL query that joins the metric documents with that catalog metadata using LOOKUP JOIN. Remember that this command needs Elasticsearch 9.1 or later, that the lookup index must be created with index.mode: lookup, and that the join field, service.name here, must be mapped as keyword in the lookup index.

FROM o11y-labs-discover-service-metrics
| WHERE @timestamp >= "2026-06-30T15:00:00.000Z" AND @timestamp <= "2026-06-30T18:45:00.000Z"
| WHERE service.environment == "production"
| LOOKUP JOIN o11y-labs-service-catalog-lookup ON service.name
| WHERE owner.team == "checkout-platform" AND metrics.latency.p95_ms > slo.latency_target_ms
| KEEP @timestamp, service.name, cloud.region, metrics.latency.p95_ms, slo.latency_target_ms, owner.team
| SORT @timestamp DESC

This is the part classic mode does not cover. Classic Discover can filter the metric documents, but ES|QL can enrich those rows with data from another index before displaying the result.

The result table answers a more operational question than the original search. It shows the affected service, the region, the latency value, the target, and the owning team in one view.

This pattern is useful for more than ownership. You can keep small lookup indices for service tier, deployment ring, escalation channel, business capability, or runbook URL. Then you can join that context into metric searches at investigation time.

How to choose the right Discover search method

The most useful workflow is not one search language for everything. It is a progression from broad scope to specific evidence.

| Use case | Discover feature | Why it helps | | :---- | :---- | :---- | | Limit the searchable data | Data view and time picker | Removes irrelevant indices and old documents before the query runs | | Keep scope visible | UI filters | Makes include, exclude, disabled, and pinned criteria easy to review | | Search exact fields and ranges | KQL | Keeps common metric searches readable | | Match field values with regex | Lucene mode | Adds regular expression syntax when the search needs it | | Enrich or reshape results | ES|QL mode | Adds joins, projections, sorting, and transformations |

For a real investigation, start with the smallest data view that still contains the data you need. Add filter pills for stable scope. Use KQL for the active search. Switch to Lucene only when regex syntax is worth the extra complexity. Move to ES|QL when the question needs enrichment, aggregation, or reshaping.

Field naming conventions that make metrics easier to search

Metric search works best when the field names carry enough context. The examples above use Elastic Common Schema-style fields where possible:

• service.name for the monitored service.
• service.environment for production, staging, or development.
• cloud.region for the deployment region.
• host.name for host-level drill-down.
• Numeric metric fields under metrics.*.

You do not need this exact schema to use Discover, but predictable field names make the search bar and filter pills much easier to use. They also make saved searches and screenshots easier to understand during a handoff.

For service catalog data, keep the lookup index small and stable. Fields like service owner, tier, SLO target, and runbook URL change less often than raw metrics. That makes them good candidates for LOOKUP JOIN during analysis.

Run the walkthrough on your own cluster

Use Discover as a drill-down path, not only as a document table. In this walkthrough, we:

• Scoped the search with a narrow data view and the time picker before writing any query.
• Made the investigation scope visible and shareable with include, exclude, disabled, and pinned filter pills.
• Used KQL for readable field, range, and boolean searches.
• Switched to Lucene only for the regex case KQL cannot express.
• Enriched metric documents with ownership and SLO data from a lookup index using ES|QL LOOKUP JOIN.

To try the full flow on your own cluster, run the supporting notebook, which creates both indices and the incident data used in every example.

Related documentation:

• Discover
• Data views
• Filtering
• KQL
• Lucene query syntax
• ES|QL in Discover
• LOOKUP JOIN

Related Observability Labs articles:

• Exploring metrics from a new time series data stream in Discover
• Explore and analyze metrics with ease in Elastic Observability
• Traces in Discover for deeper application insights in Elastic Observability
• Connecting the dots: ES|QL joins for richer observability insights
• Common ES|QL queries for Kubernetes monitoring

The Kubernetes OpenTelemetry Assets package now ships four Kubernetes SLO templates for Deployments, StatefulSets, DaemonSets, and Jobs on those OTel metrics. If you already followed Part 1 and have the dashboards and alert rules, create an SLO from a template and you get SLI, remaining budget, and burn rate without new instrumentation.

The diagram above extends the stack from Part 1. The Kubernetes OpenTelemetry Assets package (kubernetes_otel) 2.3.0 includes:

• Dashboards designed for drill-down (Part 1)
• Alert rule templates that fire on known bad states (Part 1)
• ML anomaly detection jobs with workload baselines (Part 1)
• SLO templates for rolling 30-day budgets (this post)

All four use the same OTel metrics. Burn rate alerts on an SLO send you back into Overview, Workloads, and Deployment Details when the number alone is not enough.

Why Kubernetes observability needs SLO monitoring alongside alerts

Part 1 built the reactive stack for the engineer who gets paged at 3 AM. SLOs serve the planning conversation on a 30-day horizon: Are we meeting our reliability commitments? They give platform and engineering leaders a number for prioritisation: how much error budget remains and which workload is burning it fastest. The table later in this post maps each SLO template to its Part 1 alert counterpart.

The SLO templates in this post are part of the Kubernetes OpenTelemetry Assets package (kubernetes_otel). Install the Kubernetes OpenTelemetry Assets package and confirm your cluster is already sending Kubernetes metrics through OpenTelemetry (the same pipeline from Part 1). No additional instrumentation is required.

Four SLO templates for Kubernetes Deployments, StatefulSets, DaemonSets and Jobs

In Integrations → Kubernetes OpenTelemetry → Assets, enable any of the four templates below. Names match Kibana; each includes the [Kubernetes OTel] prefix in the UI.

| Template | Rolling objective | Package description | | ----------------------------------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Deployment Replica Availability 99.5% Rolling 30 Days | 99.5% / 30d | Tracks Deployment availability from OTel metrics: 99.5% of intervals should have each Deployment at its desired replica count. When k8s.deployment.available < k8s.deployment.desired, the workload has fewer healthy replicas than configured. | | StatefulSet Replica Availability 99.5% Rolling 30 Days | 99.5% / 30d | Same pattern for StatefulSets, where pod identity and ordering matter for databases, queues, and caches. | | DaemonSet Scheduling Availability 99.0% Rolling 30 Days | 99.0% / 30d | Tracks whether each DaemonSet runs on all eligible nodes. Covers node-level agents such as log collectors, monitoring, security, and CNI plugins. | | Job Completion Success Rate 99.0% Rolling 30 Days | 99.0% / 30d | Tracks batch Jobs (ETL, backups, pipelines, scheduled tasks) completing without failed pods over the rolling window. |

Each is a timeslice-metric SLO: Elastic marks every five-minute window good or bad, then rolls those results into a 30-day rolling objective per namespace and workload.

Reliability is scored at two levels. Each five-minute slice gets one verdict: Elastic aggregates OTel metrics in that window, evaluates the template equation, and compares the result to the metric threshold. For Deployments, that is sum(available) / sum(desired) >= 1. At a ~30-second OTel scrape cadence, that is roughly ten measurements per slice, and the slice passes or fails on the aggregated result. The SLO target (99.5% or 99.0%) is the share of slices that must pass across the rolling window. Over 30 days at five-minute slices, that is 8,640 possible slices per workload (30 × 24 × 12). After you create an SLO from a template, the SLO detail view shows how many slices passed and how much error budget remains.

At 99.5%, a workload can miss roughly 43 of those slices (~3.6 hours of bad slices) before breach. At 99.0%, about 86 slices (~7.2 hours).

How to set SLO targets by Kubernetes workload type

We picked defaults per workload type, not one number for the whole cluster.

Deployments and StatefulSets at 99.5%: We considered 99.9% (~43 minutes per month), which fits a single critical API or a formal SLA buffer. For a default integration template across many Deployments, 99.5% (~3.6 hours) leaves room for normal rollout churn: a 20-minute bad image tag is roughly half the monthly budget at 99.9%, but a small fraction at 99.5%. Tune per workload; payment paths often warrant 99.9% or higher.

DaemonSets and Jobs at 99.0%: We considered 99.5% for DaemonSets, but node additions, replacements, and rolling updates often leave ready_nodes below desired_scheduled_nodes for several minutes per event. At 99.5%, that normal platform churn would burn error budget on infrastructure agents (log collectors, monitoring, CNI) as if they were user-facing outages. 99.0% (~7.2 hours) absorbs that lifecycle noise. Jobs get 99.0% for a different reason: a failed ETL run usually hurts data freshness, not live request availability, and failures can sit unnoticed until downstream teams see stale reports.

Deployment replica availability (99.5%)

Metric:     sum(k8s.deployment.available) / sum(k8s.deployment.desired) >= 1
Target:     99.5% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.deployment.name

When available < desired, the application runs fewer healthy replicas than configured. Failed rollouts, crash loops, and node loss all show up here. 99.5% leaves roughly 3.6 hours of degradation per deployment per month before breach.

Grouping by namespace and deployment name creates one SLO per workload. A cluster-wide average would let a healthy frontend mask a burning checkout. Linked dashboards (Overview and Workloads) connect the SLO view to investigation context in one click; from Workloads you drill into Deployment Detail for the failing deployment.

StatefulSet replica availability (99.5%)

Metric:     sum(k8s.statefulset.ready_pods) / sum(k8s.statefulset.desired_pods) >= 1
Target:     99.5% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.statefulset.name

When ready_pods < desired_pods, the StatefulSet reports fewer Ready replicas than configured. Ordered rollouts, stuck pods, and node loss show up here too. Rollouts proceed in order, and each pod keeps its name and volume, so a missing replica can stay below desired longer than a stateless pod would.

Grouping by namespace and StatefulSet name avoids a healthy workload masking another that is burning the SLO budget. 

DaemonSet scheduling availability (99.0%)

Metric:     sum(k8s.daemonset.ready_nodes) / sum(k8s.daemonset.desired_scheduled_nodes) >= 1
Target:     99.0% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.daemonset.name

DaemonSets run node-level infrastructure: log collectors, monitoring agents, security agents, and network plugins. When ready_nodes < desired_scheduled_nodes, an eligible node lacks a Ready pod, which can leave that node without logs or metrics from that agent. Rolling updates and new nodes drive most gaps; pods that never become Ready show the same signal. Cordoned nodes often still run DaemonSet pods. 99.0% (~7.2 hours per month) reflects that churn. 

Group by namespace and DaemonSet name so a healthy fluentd does not mask a broken node-exporter on the same SLO budget.

Job completion success rate (99.0%)

Metric:     max(k8s.job.failed_pods) <= 0
Target:     99.0% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.job.name

Jobs cover batch workloads: ETL pipelines, backups, database migrations, and scheduled reports. When failed_pods > 0, at least one pod created by the Job reached the Failed phase. Application errors, timeouts, and missing dependencies drive many failures; when retries reach the configured backoffLimit, Kubernetes marks the Job as Failed. Missed runs often surface as stale or delayed data, not as a serving outage. 99.0% (~7.2 hours per month) reflects that occasional batch failure is less time-sensitive than a Deployment or StatefulSet breach. 

How do SLOs and alerts work together in Kubernetes observability?

The SLO templates and the alert rules from Part 1 serve different people asking different questions at different times.

| SLO Template | Alert rule (Part 1) | Failure consequence | Monthly budget (30d) | | --------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------- | ------------------------ | | Deployment Replica Availability | Deployment below the desired replicas | Reduced throughput, degraded UX | ~3.6 hours at 99.5% | | StatefulSet Replica Availability | No dedicated rule. Covered by CrashLoopBackOff / OOMKilled at pod level | Split-brain risk, degraded durability | ~3.6 hours at 99.5% | | DaemonSet Scheduling Availability | Pod stuck in Pending / node disk pressure | Blind spots: unmonitored nodes and gaps in node-level coverage | ~7.2 hours at 99.0% | | Job Completion Success Rate | CrashLoopBackOff / OOMKilled | Stale or incomplete data | ~7.2 hours at 99.0% |

Alert rules answer: Is something broken right now? They fire within minutes, page the on-call engineer, and expect immediate action.

SLO templates answer: Are we meeting our reliability commitments over time? They accumulate signal across weeks and turn prioritisation debates into a number tied to remaining budget.

From incident to error budget burn: a Kubernetes walkthrough

A deployment drops from 3/3 to 2/3 available replicas during a rolling update. The new pod fails its readiness probe. Here is what happened in our test cluster, from dashboard signal through alert, SLO impact, and root cause.

Rollout begins. The Deployment dashboard shows available: 2, desired: 3. The Part 1 Deployment unavailable replicas rule has a 5-minute grace period, so on-call is not paged yet during a short rollout gap. The Deployment Detail view for web-frontend shows available replicas dropping while desired stays at 3. The Deployment replicas over time chart marks where the rollout started to fail.

Alert fires, then root cause. After the grace period, the alert rule triggers: Deployment unavailable replicas. The on-call engineer opens the Workloads dashboard, finds web-frontend at available: 2, desired: 3, and drills into Deployment Detail. The replicas-over-time chart confirms when availability dropped. 

In Discover, filter Kubernetes events for that pod with k8s.object.name: "web-frontend-796fcd55b9-jmlkh". The event stream shows ImagePullBackOff and Back-off pulling image "nginx:nonexistent-tag-999". The rollout references an image tag that does not exist.

Rollback and recovery. The engineer runs kubectl rollout undo deployment/web-frontend. Replicas return to 3/3.

How two rollouts consumed 88% of a 30-day error budget

The rollback fixed availability. The SLO still counted the day's failures.

Two rollout failures left web-frontend with 39 failed timeslices where available < desired. That consumed 88.0% of the 30-day error budget. The SLI still read 99.56%, above the 99.5% target, but only 12% of the monthly allowance remained.

The burn rate alert fired next, even though replicas were healthy again. Over the past day the deployment consumed budget at 26× the rate a 99.5% SLO can sustain long term. Each failed 5-minute timeslice uses roughly 2.3% of the monthly budget (about 43 failures allowed per 30 days). Thirty-nine failures across two rollouts is worth a reliability review, not a one-line postmortem. The burn rate alert often matters more than the raw SLI mid-month because it fires while you still have budget left to spend deliberately.

Try it yourself: trigger an error budget burn on a test Deployment

If you already have Kubernetes OpenTelemetry Assets installed, the SLO templates live under Integrations → Kubernetes OpenTelemetry → Assets.

Create a Deployment replica availability SLO for the deployment you use below. Open the SLO and note the baseline: current SLI, remaining error budget, and existing timeslice history.

Create an isolated namespace and a small deployment so the exercise does not affect production workloads. Wait a few minutes for the OTel collector to scrape metrics before you create the SLO.

kubectl create namespace blog-demo
kubectl create deployment web-frontend --namespace blog-demo --image=nginx:latest --replicas=3

Trigger a bad rollout with a non-existent image tag:

kubectl get deployment web-frontend -n blog-demo
kubectl set image deployment/web-frontend nginx=nginx:nonexistent-tag-999 --namespace blog-demo

Within a few minutes a new pod enters ImagePullBackOff, available replicas drop below desired, and the SLO records failed timeslices. Roll back to recover:

kubectl rollout undo deployment/web-frontend -n blog-demo

Refresh the SLO view. You should see new failed timeslices in the 30-day history and a reduction in remaining error budget.

One failed timeslice consumes about 2.3% of the monthly error budget at 99.5%. Repeat that across deployments in a week and the burn rate alert becomes the prioritisation signal.

When you are done, delete the test namespace with:

kubectl delete namespace blog-demo

What's next: from SLO monitoring to agentic remediation

Alerts tell you replicas dropped. SLOs tell you how much monthly budget that cost. In the walkthrough above, the same ImagePullBackOff showed up in Deployment Detail, the unavailable-replicas alert, and failed timeslices on the replica-availability SLO, all from the OTel pipeline you installed in Part 1. The SLI still read 99.56% while 88% of the monthly error budget was gone.

Part 1 closed by previewing Agentic Investigations: investigation workflows that run when an alert fires, with skills, tools, and MCP views. This post adds the SLO layer on those same metrics so you can quantify reliability debt before automating runbooks. A follow-up post will cover that agentic workflow and propose remediations you review before applying.

Which remediations would you trust a workflow to suggest on a Kubernetes incident, and which would you keep manual? Join the Elastic Community discussion.

The gain is in where the intermediate set lives. When you ask what the saturated hosts logged, the list of saturated hosts is computed and consumed inside Elasticsearch. Six host names or 500 trace IDs never land in a clipboard or in an AI agent's context window, and the set is recomputed from current data every time the query runs.

That changes the unit of investigation. You go from "one slow request stalled on a lock" to "most of the slowest requests did," and only the second answer tells you which team to page. It matters more the further apart your signals are: in many observability stacks, logs, metrics, and traces live in three separate systems, each with its own query language, its own time picker, and its own idea of what a host is, so the same question has to be asked two or three times and the answers joined by hand.

In this post we walk through four investigations, each of which is self-contained. For every one of them, we set out the scenario that started it, the query that answers it, the table it returns, and a note on the difficulties you run into when you try to get the same answer any other way.

| Pattern | Starts from | Question it answers | |---|---|---| | Metrics to logs | CPU saturation | Which error patterns show up only on the saturated hosts? | | Logs to metrics | Error logs | Do the erroring hosts look any different from the healthy ones? | | Traces to logs | Slow spans | What did every service log during those specific requests? | | All three signals | Pod memory pressure | Which log lines sit behind the requests that failed under that pressure? |

The data was collected with OpenTelemetry and lands in the logs-*.otel-*, traces-*.otel-*, and metrics-*.otel-* data streams, where fields keep their semantic convention names rather than being rewritten into another schema. The correlation pattern works just as well on Elastic Agent integrations, though the queries need translating rather than just renaming: ECS carries log severity as the text field log.level instead of a numeric severity_number, the System integration reports CPU as separate system.cpu.*.pct fields instead of one metric with a state dimension, and APM records durations in microseconds. All of it lands in the same cluster either way, which is the part the subquery depends on.

In Discover, the time picker already applies the range, so the examples below omit an explicit @timestamp filter. Outside Discover, add a filter by time yourself, either with literal timestamps in the query or with ?_tstart and ?_tend in the query and values in the params array of your _query request. Every result below comes from a one hour window over a synthetic fleet of 300 hosts.

Metrics to logs: what are the saturated hosts complaining about?

An infrastructure alert tells you a handful of hosts in a fleet of a few hundred sat above 90% CPU over the last hour. That tells you which hosts are hot and nothing about why. The question worth answering is whether those hosts share a failure mode, or whether they are busy for unrelated reasons and the alert is a coincidence.

FROM logs-*.otel-*
| WHERE severity_number >= 17
  AND resource.attributes.host.name IN (
      TS metrics-hostmetrics.otel-*
      | WHERE attributes.state == "idle"
      | STATS idle = AVG(AVG_OVER_TIME(metrics.system.cpu.utilization))
          BY resource.attributes.host.name
      | WHERE idle < 0.1
      | KEEP resource.attributes.host.name
    )
| STATS errors = COUNT(*), hosts = COUNT_DISTINCT(resource.attributes.host.name)
    BY pattern = CATEGORIZE(body.text), service = resource.attributes.service.name
| SORT errors DESC

The two halves of the query map onto the two halves of the question. The subquery works out which hosts were saturated, averaging CPU utilization per host and keeping the ones that averaged under 10% idle, which is another way of saying above 90% busy for the window. The outer query then works out what those hosts were complaining about, pulling their error logs and using CATEGORIZE to collapse thousands of individual lines into a handful of error classes.

Two choices in there are worth pausing on.

The subquery uses TS rather than FROM because a host does not report one CPU number. It reports a separate time series per CPU state, and per logical core as well if your collector is configured to break them out, so the reduction has to happen in two stages. AVG_OVER_TIME collapses each series to a single value first, and the outer AVG then combines those into one figure per host.

Naming that inner function matters more than it looks. Write AVG(metrics.system.cpu.utilization) on its own and TS supplies LAST_OVER_TIME for you, averaging each series' final sample rather than its average over the window. In this dataset that one substitution moves a host from 7% idle to 10% idle, which is the difference between appearing in the results and not. Querying metrics with the TS command goes into the two aggregation phases in more depth.

The log filter tests severity_number (17 is the ERROR floor on the OpenTelemetry scale) rather than the severity text, because the numeric scale is fixed by the spec while the text is whatever the emitting library decided to write. That is not a hypothetical distinction here: the error logs in this cluster carry four different labels, including SEVERE from a Java service. Matching on the text alone returns 2,754 of checkout's errors and misses the billing service entirely, while the numeric filter returns all 5,910 of them and keeps billing too.

When you run the query in Discover, the result is a short table of error classes, scoped to the saturated hosts:

The subquery returned six saturated hosts, and on all six the same service is timing out against an upstream dependency and draining its connection pool. The hosts column is what lets you set the other two rows aside without opening anything: the certificate errors reach only two of the six, and the gateway declines amount to five lines in an hour. Neither tracks the cohort the way the checkout patterns do.

Having all three signals in one store already removed the exports from this investigation. The subquery removes the step after that, and closing that last gap matters more than it sounds. By the time you have read six host names off a chart and typed them into a log search, the set has moved: a host that crossed the threshold a minute ago is missing from your list, and one that has since recovered is still in it. Here the host list is derived from current data on every run, so re-running the query during an incident gives you the current cohort.

The stale list is only half of it. A correlation done by hand exists only in the head of the person who did it, so nobody else can check it, save it, or run it again tomorrow.

Logs to metrics: do the erroring hosts look different from the healthy ones?

Filtering metrics by a log-derived host set answers the opposite question. The payments service is throwing errors on some hosts and not others, and you want to know whether resource pressure explains the split before you start reading deploy history.

That is a comparison, so the query needs both cohorts. FORK runs two branches over the same input, and IN and NOT IN against the same log-derived host set divide the fleet between them.

TS metrics-hostmetrics.otel-*
| WHERE attributes.state == "idle"
| STATS idle = AVG(AVG_OVER_TIME(metrics.system.cpu.utilization))
    BY host = resource.attributes.host.name
| EVAL busy = 1 - idle
| FORK
    ( WHERE host IN (
        FROM logs-*.otel-*
        | WHERE severity_number >= 17
          AND resource.attributes.service.name == "payments"
          AND resource.attributes.host.name IS NOT NULL
        | STATS errors = COUNT(*) BY resource.attributes.host.name
        | KEEP resource.attributes.host.name )
      | EVAL cohort = "logging errors" )
    ( WHERE host NOT IN (
        FROM logs-*.otel-*
        | WHERE severity_number >= 17
          AND resource.attributes.service.name == "payments"
          AND resource.attributes.host.name IS NOT NULL
        | STATS errors = COUNT(*) BY resource.attributes.host.name
        | KEEP resource.attributes.host.name )
      | EVAL cohort = "no errors" )
| STATS hosts = COUNT(*), mean_busy = AVG(busy), busiest_host = MAX(busy)
    BY cohort

The query reads top to bottom as three stages. The metrics query runs first and reduces the whole fleet to one busy figure per host. FORK then splits that fleet in two using the same log query in both branches, separating the hosts that appear in it from the hosts that do not. The final STATS summarizes each group, so both cohorts come back as two rows of one table, measured the same way over the same window.

This query is longer than the others, and three parts of it are less obvious than they look.

The natural way to label the two cohorts would be EVAL cohort = CASE(host IN (...), "erroring", "healthy"), and ES|QL rejects it. In 9.5 an IN subquery has to be a top-level predicate in a WHERE condition rather than an argument to a scalar function, which is why the split happens at the command level with FORK.

The STATS ... BY inside each subquery looks redundant, since KEEP alone would return the same host names. It is not: without it, the subquery returns one row per matching log document instead of one row per host, and those rows are all held in memory for the outer query to filter against. Aggregating first turns millions of rows into a few hundred host names.

The IS NOT NULL filter guards the sharpest edge here, and this dataset is a live example rather than a hypothetical. NOT IN follows SQL null semantics, so a single null in the subquery result makes the predicate match nothing at all. Five of the payments error logs in this cluster came through a sidecar that dropped the host name. Remove that one line from both branches and the query still succeeds, but it returns a single row: the nulls quietly delete the entire 286-host "no errors" cohort, and what is left looks like a perfectly plausible answer to a different question.

Run the query in Discover and you get two rows, one per cohort:

CPU does not explain the split, and the numbers say so twice. The 14 erroring hosts run slightly cooler on average than the 286 quiet ones, and the busiest machine among them averaged 50% over the hour while the quiet cohort contains a host that averaged 95%. Whatever is failing on those 14, they had headroom the entire time, and deploy history is a better place to spend the next ten minutes.

A negative result like this is worth as much as a positive one, and it is usually the one people skip. Getting it the long way means running the metrics query twice against two hand-built host lists and lining the numbers up afterwards, which is enough friction that the check often just does not happen. Both cohorts here come from the same log query in the same execution, over identical time windows, so there is nothing to reconcile and no reason not to check.

Traces to logs: what did every service log during the slow requests?

A service level objective (SLO) burn alert fires on checkout latency. Tracing gives you the slow requests and their spans, and the next question is what the services involved were writing to their logs while those specific requests were in flight.

The trace ID is the join key, and there are far too many of them to move by hand.

FROM logs-*.otel-*
| WHERE trace_id IN (
      FROM traces-*.otel-*
      | WHERE kind == "Server"
        AND resource.attributes.service.name == "checkout"
        AND name == "POST /api/orders"
        AND duration > 2000000000
      | SORT duration DESC
      | LIMIT 500
      | KEEP trace_id
    )
| STATS lines = COUNT(*), traces = COUNT_DISTINCT(trace_id)
    BY pattern = CATEGORIZE(body.text),
       service = resource.attributes.service.name,
       severity_text
| SORT traces DESC

The subquery answers "which requests were slow." It looks at the inbound request span for the checkout endpoint rather than the client and internal spans beneath it, then keeps the 500 slowest requests over two seconds. Durations are recorded in nanoseconds, which is why the threshold has so many zeros.

The outer query answers "what got logged while they were running," gathering every log line that shares one of those trace IDs and grouping them into patterns.

Counting distinct traces per pattern is what makes the output readable. A log pattern that appears 30,000 times across four traces is one chatty request, while a pattern that shows up in 470 of the 500 slowest traces is a property of being slow.

From the results above, the top row is there by construction and can be set aside: checkout writes one order submitted line per order, so it appears in all 500 traces and says nothing about why these particular 500 were slow.

From the results above, the third row is the answer, and it points at a service nobody was looking at. A lock wait timeout in inventory, two hops downstream from where the alert fired, shows up in 470 of the 500 slowest requests, and the 947 lines behind those 470 traces mean a good share of them retried more than once. The payment gateway declines are real failures, and at 19 traces out of 500 they are not what is burning the SLO.

Done by hand, this means opening slow traces one at a time and reading the correlated logs for each, which is tedious at ten traces and nobody's idea of a plan at 500. When the spans and the logs are held in different systems, every trace you check is a copied ID and a context switch, and the sample size you can afford drops to about three. Three traces is enough to form a theory and not enough to test one. Treating the slow requests as a population is what turns "this trace had a lock wait" into "470 of the 500 slowest requests had a lock wait," and that difference decides whether you page the inventory team.

All three signals: from pod memory pressure to the log lines behind the failures

IN subqueries nest, so the pattern extends to as many signal types as the question needs.

A node pool starts reporting memory pressure after a rollout. You want the log lines from the requests that actually failed on the pods under pressure, which means going from metrics to traces to logs without stopping in between.

FROM logs-*.otel-*
| WHERE trace_id IN (
      FROM traces-*.otel-*
      | WHERE kind == "Server"
        AND status.code == "Error"
        AND resource.attributes.k8s.pod.uid IN (
            TS metrics-kubeletstats.otel-*
            | STATS peak = MAX(MAX_OVER_TIME(metrics.k8s.pod.memory_limit_utilization))
                BY resource.attributes.k8s.pod.uid
            | WHERE peak > 0.95
            | KEEP resource.attributes.k8s.pod.uid
          )
      | STATS failures = COUNT(*) BY trace_id
      | SORT failures DESC
      | LIMIT 1000
      | KEEP trace_id
    )
| STATS lines = COUNT(*), traces = COUNT_DISTINCT(trace_id)
    BY pattern = CATEGORIZE(body.text), service = resource.attributes.service.name
| SORT traces DESC

Reading the query inside out, you can see each layer answering one part of the question. The innermost subquery identifies the pods whose memory peaked above 95% of their limit, using MAX_OVER_TIME to take each pod's peak rather than its average. The middle one narrows to the requests that actually failed on those pods and reduces them to at most 1,000 trace IDs. The outer query then collects the logs for those traces from every service that took part, including services running on pods that were entirely healthy, and that last part turns out to be where the answer is.

The pods are matched on their UID rather than their name, because names repeat across namespaces and restarts. The query assumes Kubernetes metadata reaches your spans, which is what the collector's k8sattributes processor is for; if it does not, the host or container ID works the same way.

From the results above, the first row sits at exactly 1,000 because that is the subquery's LIMIT, so it describes the size of the sample rather than the size of the incident. Every trace in that sample carries the cart deadline line, which is the symptom you already knew about when you started.

Look at the second row instead. product-catalog is rejecting oversized payloads across 946 of the same 1,000 traces, and it never appeared in the pod subquery at all: its pods peaked at 75% of their memory limit, well under the 95% threshold. The rollout started sending larger payloads, which would account for both cart's memory climb and the failures. Checkout's retry budget gives out in about half of them, which is how the failure became visible to users.

Filtering logs directly by the pressured pods would have shown you the cart line and hidden the product-catalog one, which is to say it would have confirmed the symptom and buried the cause. Doing it without subqueries means three queries and two hand-built lists, and the second list is a thousand trace IDs. That is usually the point at which people stop after the first hop and go with the cart theory. The reason the second hop is cheap here is that all three signals sit in the same store behind the same query language, so widening from pods to traces to every service in the trace is a clause, not a project.

Why do ES|QL subqueries matter for AI agents?

Keeping the intermediate set inside the cluster is convenient for a person and close to essential for an agent querying on your behalf.

Split across two tool calls, the intermediate result has to travel. A list of 500 trace IDs comes back in a tool response and occupies the model's context, and the agent then has to rewrite every one of them into the next query. That costs tokens on every hop, and it is where truncation and transcription errors come from. With a subquery, the intermediate set stays inside Elasticsearch and the agent only ever sees the final table.

The problem compounds when the signals are spread across systems. An agent then needs credentials, a client, and a working knowledge of the query language for each one, plus the judgment to join results that use different names for the same host. One store and one query language reduce that to a single skill the agent has to be good at.

One ES|QL string is also a complete description of the correlation, which makes the investigation reproducible: an agent can put the query in its summary, and a human can paste it into Discover and get the same logic evaluated against current data. A two-call sequence with a hardcoded host list in the middle gives you neither. The one thing to watch is that an agent calling the _query API has to filter @timestamp itself, since nothing is binding a time picker.

Four things to know before writing ES|QL IN subqueries

IN subqueries are in technical preview in Elastic Stack 9.5 and on Elastic Cloud Serverless, while TS and FORK have been generally available since 9.4. CATEGORIZE has been generally available since 9.1 and requires a Platinum license; every query above works without it if you group by an existing field instead.

Four things are worth knowing before you write your own:

• In 9.5 the subquery returns exactly one column, which is what the trailing KEEP does in each example.
• Aggregate the subquery down to distinct values with STATS ... BY before returning them. Its result is materialized for the outer query to filter against, so handing back a few hundred host names instead of a few million rows is both faster and safer.
• Filter nulls out of any NOT IN subquery, because SQL null semantics mean one null makes the predicate match nothing.
• Subqueries are non-correlated. They run independently and cannot reference columns from the outer query, so this is a set filter rather than a row-by-row join. Reach for LOOKUP JOIN when you need per-row enrichment, which we covered in ES|QL joins for richer observability.

Try ES|QL signal correlation on your own data

The pattern under all four examples is the same. You start with a set you can describe in one signal and a question you can only answer in another, and the subquery carries that set across the boundary for you.

The syntax is the smaller part of what makes that work. It works because logs, metrics, and traces sit in one store behind one query engine, under field names they kept on the way in, so crossing from one signal to another is a clause in a query rather than an integration to build and maintain. Where that is not true, the same four investigations turn into a sequence of exports, translations, and manual joins, and that costs more than slower answers. It quietly shrinks the number of questions anyone is willing to ask, and the negative results are the first to go.

Each pattern here replaces two or three queries with one, and no host list or trace ID list has to move between them. Fewer steps mean fewer places to be wrong, and a correlation you can save as a single string and hand to someone else.

To try it:

1. Open an Observability project on Elastic Cloud Serverless, or upgrade to Elastic Stack 9.5.
2. Send data with the Elastic Distributions of OpenTelemetry, or point an existing collector at Elasticsearch.
3. In Discover, switch to ES|QL and start from the metrics to logs query above, swapping in your own data streams and thresholds.
4. Read the IN subquery reference for the full set of commands you can use inside a subquery.

That team still wants a model. So the request lands on an SRE's desk, and it sounds deceptively small: "Can you stand up an open-weight model for the claims team? Sixty people. It has to run on our hardware." They aren't even allowed to use a neocloud. There is a cost associated with this, but we won't explore that part. Just the part that covers running the model and observing the configuration.

Standing it up is the easy half. Four manifests and an afternoon, and you have a model answering questions. The hard part arrives a week later, when someone says "it feels slow" and you realize you have no idea whether the deployment is configured well, badly, or catastrophically — and no obvious way to find out.

This guide shows you how Elastic Observability can help you analyze the metrics from the configuration. It walks through tuning a real vLLM deployment using the metrics vLLM already emits. vLLM exposes these on a /metrics endpoint in Prometheus exposition format — no instrumentation, no sidecar, no code change — which is why every query in this guide starts from a Prometheus scrape. The goal: turn "it feels slow" into a specific, defensible decision.

Test environment: vLLM on a Kubernetes cluster using NVIDIA A10G with dcgm-exporter and Prometheus metrics

Every figure in this guide was measured on the following stack — one replica, one GPU, no autoscaling.

• Workload — Kubernetes-native load generator, scaled from 8 to 32 concurrent requests.
• Model — Qwen/Qwen2.5-3B-Instruct, bf16, --max-model-len 4096
• Engine — vLLM v0.23.0, OpenAI-compatible server, Prometheus /metrics on :8000
• GPU — NVIDIA A10G, 24 GB — an AWS g5.xlarge
• Cluster — Amazon EKS 1.30, tainted GPU node pool with minSize: 0
• Telemetry — Prometheus scraping every 15s, plus dcgm-exporter on :9400, shipped via remote_write
• Analysis — Elastic Observability, queried with ES|QL and PromQL
• Measured — 2026-07-27

Why is it hard for an SRE to self-host and tune an open-weight LLM?

The difficulty is not the deployment, it's the tuning which has no feedback loop. vLLM starts, serves, and reports success whether it's configured brilliantly or wastefully. Nothing tells you which.

When loading up the model, your manifest would have this configuration:

      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.23.0  # pin an exact release — metric names shift between versions
          args:
            - "--model=Qwen/Qwen2.5-3B-Instruct"
            - "--max-model-len=4096"         # cap context → predictable KV-cache size
            # A10G has native bf16 — do NOT add --dtype=half (T4-only).
          ports:
            - name: http
              containerPort: 8000            # OpenAI API + /metrics

But you can run into specific issues, such as: Hugging Face downloads take minutes. If your cluster expects a server to start in 30 seconds, it will assume the app is dead and kill it mid-download, putting you in an infinite crash loop.

Or you could have a hardware mismatch, and potentially degrade your model’s speed or precision because hardware architectures vary

or a bevy of other issues.

Once the model is finally running, optimizing performance is complete guesswork because default metrics don't tell you if you're being efficient.

You could use nvidia-smi, but this only understands raw hardware state, not application software logic.

Now that you have it running, a few hours to maybe even a day in, the team says "it feels slow." You are, functionally, tuning blind.

How do you tune a self-hosted vLLM deployment?

You're not an inference engineer, you own forty other services besides this one, and you don't have a forward-deployed engineer from a model vendor on call. But tuning an LLM server turns out to need exactly one skill you already have: reading telemetry and reasoning about saturation. The only missing piece is telemetry that exists and means something.

It does. vLLM emits a rich Prometheus endpoint out of the box — latency decomposed by inference phase, cache hit rates, batch occupancy, token accounting, completion outcomes. Almost nobody looks at it. The rest of this guide is how to read it.

Defining the workload: sixty users, short prompts, streaming responses

With the slowness detected and reported, you gather the usage profile of the users. Their usage pattern is as follows:

• ~60 users, but not concurrent. Realistic peak is 8–12 simultaneous in-flight requests; sustained is lower.
• Short prompts, long answers. The user pastes a paragraph and asks for a structured summary. Prompts run ~50 tokens; useful answers run 500–1,000.
• Interactive, streaming UI. Perceived speed is dominated by time to first token (TTFT), not total time — the same psychology as a chat interface.
• Heavy prompt reuse. Every request carries the same system prompt and the same policy-language boilerplate.

From that, you write down actual service objectives — the step most self-hosted LLM projects skip:

• TTFT p95 < 300 ms
• inter-token latency < 50 ms (≥ 20 tokens/sec, faster than reading speed)
• zero queueing at 12 concurrent
• error + abort rate < 0.5%

Those four numbers are the point of everything that follows. Without them, "it feels slow" has no answer. With them, every metric below passes or fails a stated bar.

How do you get Prometheus metrics out of vLLM and the GPU?

There are two sources, and you need both.

• vLLM reports on itself — latency by phase, cache hit rates, batch occupancy, token counts — on /metrics at its serving port, with no adapter and no instrumentation work.
• The GPU reports separately, through NVIDIA's dcgm-exporter on :9400 (NVIDIA Data Center GPU Manager (DCGM) is a suite of tools and libraries designed to comprehensively manage, monitor, and diagnose enterprise-grade NVIDIA GPUs in clusters and data centers). vLLM tells you what the engine thinks is happening; DCGM tells you what the card is actually doing. Step 6 is built entirely on the gap between those two answers.

On this EKS cluster that means three things running side by side:

• vLLM as a plain Deployment on the tainted g5.xlarge GPU node pool. For one model on one card, a Deployment and a Service is the whole architecture.
• dcgm-exporter as a DaemonSet, pinned to the same GPU nodes.
• A Prometheus server on a CPU node, scraping both endpoints every 15 seconds.

Nothing here is AWS-specific. The production version is the same manifests on an on-prem cluster with L40S or H100 nodes — which is the point of doing this on Kubernetes rather than on a vendor's platform.

What about KServe and llm-d?

Neither was run for this guide, and neither changes where the metrics come from. KServe and llm-d sit on top of vLLM rather than replacing it — vLLM is still the engine, so /metrics is still the source of every number here. Each adds its own layer on top (KServe: autoscaler and revision metrics; llm-d: router and cache-routing metrics), but the inference telemetry underneath is identical.

What they change is when you need them — and each promotion is triggered by a metric you're already collecting:

How do you ship the vLLM metrics and DCGM metrics to Observability

A Prometheus scraping inside the cluster only holds hours of data — the metrics have to reach a store you can still query next week. There are two paths for that, and they are not equivalent.

Path A — OpenTelemetry Collector. Puts inference metrics into the same pipeline as your traces and logs. One collector, one auth path, one mental model. The cost is that Prometheus metrics sent through OTLP get normalized: the stored schema ends up neither purely Prometheus nor purely OTel, and metric names shift.

Path B — native Prometheus remote_write. Stands up a small Prometheus that scrapes both endpoints and pushes to a backend speaking the remote-write protocol. Names and labels land untouched, _sum / _count / _bucket histogram parts stay intact, and existing queries keep working.

For a tuning exercise, choose Path B. That's what produced every number in this guide. The reason is narrow but decisive: tuning means comparing against the vLLM documentation and the vLLM community, and both speak in exact metric names. When your chart says vllm:kv_cache_usage_perc, you can search for it.

The deployment is two YAML files — a Prometheus Deployment with two scrape jobs and a remote_write block, plus a Secret holding the backend credential. In this build the destination was an Elastic Serverless project, which exposes a Prometheus remote-write endpoint and lands data in a time-series data stream, metrics-vllm.prometheus-inference.

Two things cost me real time. If your backend has a separate ingest host for OTLP versus its main API, remote-write usually lives on the main API host, not the ingest one — pointing at the wrong one returns a 404 that looks like a path error. And the credential needs index-write privileges, not just ingest authentication; a key that works fine for OTLP can authenticate successfully and then 403 on every sample. Check prometheus_remote_storage_samples_failed_total on the Prometheus itself before looking anywhere else.

How do you confirm vLLM metrics landed in Elastic?

Once the pipeline is up, look at the field list. Roughly 127 metric series arrive from a single vLLM pod plus DCGM:

This screen is more useful than it looks. Scanning the field list is how you confirm your vLLM version's exact metric names — they do shift between major vLLM releases, and a dashboard built against the wrong names fails silently by returning nothing rather than erroring.

Two gotchas when querying vLLM metrics

Both produce results that look like "the metrics aren't working" when the pipeline is perfectly healthy.

vLLM metric names contain a colon (vllm:num_requests_running), so they need escaping in most query languages. More insidiously, if you filter by metric name across several metrics and then aggregate only one of them, you get rows back — full of nulls, with no error. Each Prometheus metric lands in its own field, so naming the field is the filter; you don't need the name predicate at all.

Counters need rate functions, gauges don't. vllm:generation_tokens_total is cumulative and monotonic — taking a max of it gives the pod's lifetime total, not its throughput. Gauges like vllm:num_requests_running, vllm:num_requests_waiting and vllm:kv_cache_usage_perc are instantaneous and want max or average. Mixing these up produces charts that are wrong but plausible, which is considerably worse than charts that are empty.

Which vLLM Prometheus metrics actually matter?

A reference for the metrics used in this guide, what each tells you, and the condition worth watching. Names are as vLLM emits them; the DCGM_FI_* series come from dcgm-exporter.

| Metric | Type | What it tells you | Watch for | |---|---|---|---| | vllm:time_to_first_token_seconds | Histogram | TTFT — how long before the first token streams | p95 above your interactive bar (300 ms here) | | vllm:inter_token_latency_seconds | Histogram | Streaming speed after the first token | Above ~50 ms is slower than reading speed | | vllm:e2e_request_latency_seconds | Histogram | Total request time | Rising while TTFT is flat = decode or workload change | | vllm:request_queue_time_seconds | Histogram | Time waiting for admission | Earliest saturation signal — any sustained rise | | vllm:request_prefill_time_seconds | Histogram | Time processing the prompt | Dominant share = prefill-bound workload | | vllm:request_decode_time_seconds | Histogram | Time generating tokens | Dominant share = memory-bandwidth-bound | | vllm:num_requests_running | Gauge | Requests currently being decoded | Batch occupancy | | vllm:num_requests_waiting | Gauge | Requests queued for admission | Sustained non-zero = add a replica | | vllm:kv_cache_usage_perc | Gauge | Occupancy of the KV block pool — not VRAM | Autoscaling trigger (~60%) | | vllm:prompt_tokens_total + vllm:prompt_tokens_cached_total | Counters | Prefix-cache hit rate | A drop means routing scattered your prefixes | | vllm:generation_tokens_total | Counter | Output throughput in tokens/sec | Headline throughput number | | vllm:request_prompt_tokens + vllm:request_generation_tokens | Histograms | Per-request token sizes; their ratio is the workload's shape | A moving ratio means the workload changed character | | vllm:iteration_tokens_total | Histogram | Tokens advanced per forward pass | Near 1.0 with concurrency = batching broken | | vllm:request_success_total{finished_reason} | Counter | Completion outcomes | error/abort = SLO; length share = truncation | | http_requests_total{status} | Counter | Server-level requests | Catches 4xx and malformed requests vllm:* never sees | | DCGM_FI_DEV_FB_USED / _FB_FREE | Gauge | Physical VRAM | Capacity planning only — never alert on it | | DCGM_FI_DEV_GPU_UTIL | Gauge | "A kernel is resident" | Not a measure of useful work | | DCGM_FI_PROF_PIPE_TENSOR_ACTIVE | Gauge | Tensor-core activity | Low here + high DRAM = memory-bound | | DCGM_FI_PROF_DRAM_ACTIVE | Gauge | Memory-bandwidth activity | High = the bottleneck is bandwidth | | DCGM_FI_DEV_POWER_USAGE | Gauge | Watts drawn | Pairs with throughput for tokens-per-watt |

Step 1: Where does vLLM latency go? TTFT, prefill, and decode decomposed

Decompose total latency into queue, prefill, and decode before optimizing anything. vLLM reports all three separately, and they have completely different fixes. This is the single most valuable chart in the setup.

The query averages each phase's cumulative time by request count in the same window — in Prometheus terms, rate(vllm:request_prefill_time_seconds_sum) / rate(vllm:e2e_request_latency_seconds_count), and the same for queue and decode:

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS reqs = SUM(RATE(`metrics.vllm:e2e_request_latency_seconds_count`)),
        q_s  = SUM(RATE(`metrics.vllm:request_queue_time_seconds_sum`)),
        pf_s = SUM(RATE(`metrics.vllm:request_prefill_time_seconds_sum`)),
        dc_s = SUM(RATE(`metrics.vllm:request_decode_time_seconds_sum`))
    BY minute = BUCKET(@timestamp, 1 minute)
| EVAL queue_ms   = ROUND(q_s  / reqs * 1000, 2),
       prefill_ms = ROUND(pf_s / reqs * 1000, 1),
       decode_ms  = ROUND(dc_s / reqs * 1000, 1)
| KEEP minute, queue_ms, prefill_ms, decode_ms
| SORT minute ASC

At 8 concurrent requests on the A10G:

minute    | queue_ms | prefill_ms | decode_ms | e2e_ms  | ttft_ms | inter_token_ms
22:55:00  | 0.01     | 37.98      | 1664.69   | 1715.18 | 50.81   | 16.23
22:56:00  | 0.01     | 40.99      | 1670.44   | 1723.78 | 53.50   | 16.20

Read it against the objectives:

• TTFT is 51 ms against a 300 ms target. Passing, with almost 6× headroom. Perceived responsiveness is not the problem, whatever was said in the meeting.
• Inter-token latency is 16 ms — about 62 tokens/sec against a 50 ms / 20 tok-s bar. Text arrives roughly three times faster than a person reads it.
• Queue time is 0.01 ms. Nothing is waiting for admission; the engine has capacity to spare at this concurrency.
• Decode is 1,665 ms against 38 ms of prefill — 97% of the time is decode.

That last line is the finding. Every optimization aimed at prefill is worthless for this workload. Chunked prefill, prompt compression, a faster attention kernel for long contexts — all real techniques, all irrelevant when prefill is 2% of the time. Decode is memory-bandwidth-bound, so the levers that would actually move it are quantization, tensor parallelism across two cards, or a smaller model. A single chart eliminated the wrong shopping list.

Watch vllm:request_queue_time_seconds specifically. It is the earliest saturation signal in the entire vLLM metric set — queue time climbs before vllm:num_requests_waiting becomes visibly non-zero, because a request can wait milliseconds for admission without ever registering as queued at scrape time. If you alert on one thing from this section, alert on queue time crossing a small absolute threshold.

Step 2: Is vLLM using the GPU efficiently? KV cache, prefix caching, and batch occupancy

Four metrics answer this: prefix-cache hit rate, tokens per iteration, KV-cache occupancy, and running-vs-waiting requests. Latency tells you the experience is good; these tell you whether you're overpaying for it.

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS ptok    = SUM(RATE(`metrics.vllm:prompt_tokens_total`)),
        cached  = SUM(RATE(`metrics.vllm:prompt_tokens_cached_total`)),
        gen     = SUM(RATE(`metrics.vllm:generation_tokens_total`)),
        it_s    = SUM(RATE(`metrics.vllm:iteration_tokens_total_sum`)),
        it_c    = SUM(RATE(`metrics.vllm:iteration_tokens_total_count`)),
        running = MAX(`metrics.vllm:num_requests_running`),
        waiting = MAX(`metrics.vllm:num_requests_waiting`),
        kv      = MAX(`metrics.vllm:kv_cache_usage_perc`)
    BY minute = BUCKET(@timestamp, 1 minute)
| EVAL prefix_cache_hit_pct = ROUND(cached / ptok * 100, 1),
       tokens_per_iteration = ROUND(it_s / it_c, 2),
       gen_tokens_per_sec   = ROUND(gen, 1),
       kv_cache_pct         = ROUND(kv * 100, 3)
| KEEP minute, prefix_cache_hit_pct, tokens_per_iteration,
       gen_tokens_per_sec, running, waiting, kv_cache_pct
| SORT minute ASC

minute   | prefix_cache_hit_pct | tokens_per_iteration | gen_tok/s | running | waiting | kv_cache_pct
22:55:00 | 32.5                 | 10.36                | 479.1     | 8.0     | 0.0     | 0.255
22:59:00 | 32.3                 | 10.34                | 480.0     | 8.0     | 0.0     | 0.121

A 32% prefix-cache hit rate is a third of all prefill work simply not done. The claims team's requests share a system prompt and policy boilerplate, and vLLM's automatic prefix caching recognizes that. This is a direct argument for raising prompt standardization: the more the application puts shared context in a consistent leading position, the higher this climbs and the cheaper every request gets. It is also the number that will crater the day a naive round-robin load balancer sits in front of two replicas — precisely the condition that justifies llm-d's cache-aware routing.

tokens_per_iteration ≈ 10.3 with 8 concurrent requests is continuous batching working correctly. Each forward pass through the model advances about ten sequences at once. If this sat near 1.0 with multiple requests in flight, batching would be broken and you'd be paying full model-forward cost per token per user. This metric proves you're getting vLLM's core value.

What does vllm:kv_cache_usage_perc actually measure?

vllm:kv_cache_usage_perc reports occupancy of vLLM's pre-allocated KV block pool — not physical GPU memory. At startup, vLLM reserves a fraction of VRAM (governed by --gpu-memory-utilization, default 0.9) and carves a KV block pool out of that reservation. This gauge reports how full that pool is.

That's why it read 0.25% here. Eight concurrent requests holding ~150 tokens each barely touch an A10G's block budget. The pool is large, and correctly so. Push the same server to 32 concurrent requests with 512–1,024 token generations and it moves — to about 2.8%. Still small.

The instinct is to read a number that low as "the cache is broken" or "I've massively over-provisioned." Both are wrong. Treating this gauge as a VRAM proxy is the most common self-hosted vLLM configuration error I see. Step 6 shows exactly how far apart the two are.

Step 3: What shape is your vLLM inference workload?

Confirm the workload is what you think it is before tuning anything. This is the panel that explains a latency "regression" that isn't your fault.

minute   | requests_per_min | avg_prompt_tokens | avg_generated_tokens | avg_max_tokens | gen_to_prompt_ratio
22:55:00 | 282.2            | 49.6              | 103.6                | 103.6          | 2.09
23:01:00 | 276.0            | 50.6              | 103.0                | 103.0          | 2.04

The generation-to-prompt ratio is 2.04 — this workload writes twice as much as it reads. That single ratio is the explanation for Step 1's 97%-decode finding, and it holds for any summarize-and-draft use case. If the team later adds a long-document RAG feature, prompts jump to thousands of tokens, the ratio inverts, the workload becomes prefill-bound, and the correct tuning changes completely. Watching this ratio is how you learn your workload changed character before someone files a ticket.

Now the column that should bother you: avg_generated_tokens equals avg_max_tokens exactly. Every request is stopping because it hit its token ceiling, not because the model finished its thought. The screenshot shows the same pattern holding as generation lengths scale to ~685 tokens against a ~777 ceiling.

In a load test that's an artifact of the generator. In production, that number is users getting cut off mid-sentence — and it is invisible in every latency metric you have. Which brings us to the metric that catches it.

Step 4: Are vLLM requests actually succeeding? Checking finished_reason

Break vllm:request_success_total down by its finished_reason label. This is the closest thing self-hosted inference has to an application-level SLI, and it catches a failure mode no latency chart can.

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS completions_per_min = ROUND(SUM(RATE(`metrics.vllm:request_success_total`)) * 60, 2)
    BY minute = BUCKET(@timestamp, 1 minute), finish_reason = labels.finished_reason
| SORT minute ASC, finish_reason

Five outcomes, each meaning something different operationally:

| finished_reason | What it means | What to do about it | |---|---|---| | stop | The model finished naturally | This is the number you want large | | length | Truncated at max_tokens | High share means users are cut off — raise the ceiling, or shorten the ask | | abort | The client disconnected first | Users giving up, or a proxy timeout shorter than your generations | | error | The engine failed | Your hard SLO signal. Should be flat zero | | repetition | Degenerate looping output | A sampling-parameter problem, not an infrastructure one |

Under the small load generator the split was 100% length — expected, since it requested a fixed ceiling. In the screenshot, at a mixed load, stop and length run side by side at roughly 51 and 32 completions/min. That mix is the healthy shape: most requests finishing on their own, a minority hitting the ceiling.

The lesson generalizes. error and abort are what you page on. But the stop-to-length ratio is what you review weekly, because drift toward length means answers are being truncated and no latency dashboard on earth will tell you.

One blind spot to close: vllm:* metrics only count requests the engine accepted. Malformed JSON, 4xx, auth failures and dropped connections never reach it. Those live in http_requests_total with status and handler labels — worth a panel beside this one, because "the model is broken" reports frequently turn out to be the gateway in front of it.

Step 5: What NVIDIA DCGM metrics say the GPU is actually doing

Use NVIDIA DCGM as an independent witness to vLLM's own account. Everything so far is the engine describing itself; DCGM describes the silicon.

FROM metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS gpu_util_pct    = MAX(`metrics.DCGM_FI_DEV_GPU_UTIL`),
        mem_bw_util_pct = MAX(`metrics.DCGM_FI_DEV_MEM_COPY_UTIL`),
        vram_used_mib   = MAX(`metrics.DCGM_FI_DEV_FB_USED`),
        vram_free_mib   = MIN(`metrics.DCGM_FI_DEV_FB_FREE`),
        power_w         = ROUND(MAX(`metrics.DCGM_FI_DEV_POWER_USAGE`), 1),
        temp_c          = MAX(`metrics.DCGM_FI_DEV_GPU_TEMP`)
    BY minute = BUCKET(@timestamp, 1 minute)
| SORT minute ASC

→ 100% GPU util · 21,483 MiB used / 1,352 MiB free · 240 W · 73 °C

100% utilization and 94% VRAM. Under the conventional reading, this card is maxed out and it's time to ask for more hardware. That reading is wrong.

Why is GPU utilization a misleading metric for LLM inference?

DCGM_FI_DEV_GPU_UTIL means "a kernel is resident on the device," not "the device is doing useful work." It reads 100% for a perfectly-tuned server and 100% for a badly-tuned one, so it cannot distinguish them. The DCGM profiling counters can:

gr_engine_active 99.8%  ·  tensor_active 16.6%  ·  dram_active 80.4%

Read those three together. The GPU's compute engine is busy essentially all the time — but its tensor cores, the units that do the actual matrix math, are active only 16.6%, while DRAM is active 80.4%. The card is not computing. It is waiting on memory.

This is independent, hardware-level confirmation of what Step 1 inferred purely from timings: decode is memory-bandwidth-bound. Two entirely different instruments, two different layers of the stack, one conclusion — the difference between a hypothesis and a finding.

It also permanently retires GPU utilization as a capacity metric for LLM inference. If your GPU capacity planning rests on DCGM_FI_DEV_GPU_UTIL — and most does — it rests on nothing.

Step 6: vLLM KV cache vs GPU VRAM, and why they disagree

Put vllm:kv_cache_usage_perc and physical VRAM usage on one 0–100% axis. They describe the same GPU memory, they sit at opposite ends of the chart, and both are correct.

minute   | running | gen_tok_s | kv_cache_pct | vram_used_pct | gpu_util | dram_active_pct | tensor_active_pct | tokens_per_watt
00:05:00 | 31      | 1627.5    | 2.82         | 94.1          | 100.0    | 80.4            | 16.6              | 6.80

KV cache at 2.8%. VRAM at 94.1%.

vLLM pre-allocates a large fraction of VRAM at startup — governed by --gpu-memory-utilization, default 0.9 — and carves its KV block pool out of that reservation. vllm:kv_cache_usage_perc reports occupancy of the pool. DCGM reports what the driver sees, which is the whole reservation, whether or not it's holding anything.

The operational consequences are precise, and they're the practical payoff of the entire exercise:

• Autoscale on vllm:kv_cache_usage_perc and vllm:num_requests_waiting. These describe admission capacity — whether the engine can take another request right now.
• Capacity-plan on VRAM. This describes physical space — whether a second model could ever fit on this card. (It can't. 1.3 GB free.)
• Never alert on VRAM. It will page you at 3 a.m. for a healthy, mostly-idle server, every single night, forever.

And tokens_per_watt — generated tokens divided by power draw, 6.8 here — is a genuine cost-efficiency metric. It's comparable across GPU models, batch settings and quantization levels in a way that neither latency nor utilization is. When you go back to Finance for card number two, this is the number that makes the argument: at 32 concurrent we sustain 1,627 tokens/sec at 240 watts, and here's what that becomes on an L40S.

vLLM tuning decisions: what the SRE does with these Prometheus metrics

Six steps, thirty minutes, one server. The verdict against the stated objectives:

| Objective | Measured | Verdict | |---|---|---| | TTFT p95 < 300 ms | 51 ms | Pass, 6× headroom | | Inter-token < 50 ms | 16 ms (≈62 tok/s) | Pass | | Zero queueing at 12 concurrent | queue_ms 0.01, waiting 0 at 8; still 0 at 32 | Pass, large margin | | Error + abort < 0.5% | 0% | Pass |

The configuration is correct for this department, and the department is over-provisioned rather than under-provisioned. That's a defensible, evidence-backed answer to "it feels slow" — and it redirects the investigation to the app, the gateway, or the prompt, which is where the problem actually is.

The concrete follow-ups, each tied to a metric rather than a hunch:

1. Stop optimizing prefill. — vllm:request_decode_time_seconds vs vllm:request_prefill_time_seconds. Decode is 97% of the time against prefill's 2%, confirmed twice. Chunked prefill and prompt compression are off the table for this workload.
2. If more throughput is needed, quantize before buying hardware. — DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (16.6%) vs DCGM_FI_PROF_DRAM_ACTIVE (80.4%). The bottleneck is memory bandwidth, not compute, so an FP8 or AWQ build of the same model is the highest-leverage single change: it moves fewer bytes per token, which is exactly the constrained resource.
3. Raise the client-side max_tokens ceiling. — vllm:request_success_total{finished_reason}. Every request finishing on length rather than stop is a user getting cut off mid-answer. This is the one finding that's a live user-experience defect. You need to increase the prompt max_token limit.
4. Standardize the prompt prefix. — vllm:prompt_tokens_cached_total over vllm:prompt_tokens_total, 32% today. But it should be better (more like 70%) More shared boilerplate in a consistent leading position raises it, and it's free.
5. Set the autoscaling trigger now, before it's needed. — vllm:kv_cache_usage_perc and vllm:num_requests_waiting. Scale when the first crosses ~60% or the second stays above zero. Do not scale on DCGM_FI_DEV_GPU_UTIL — it's pinned at 100% regardless.
6. Alert on queue time, not on VRAM. — vllm:request_queue_time_seconds is the earliest true saturation signal; DCGM_FI_DEV_FB_USED is a constant that looks like an emergency.
7. Revisit when the workload changes shape. — vllm:request_generation_tokens over vllm:request_prompt_tokens, 2.04 today. When the RAG feature ships that ratio inverts, the workload becomes prefill-bound, and half of this analysis needs redoing. The chart tells you the day it happens.

Notice that most of the metrics are looking at the vLLM metrics not the GPU metrics in helping optimize. These are still within the limit of a single service, but when you get vllm:num_requests_waiting to persistently non-zero, then you need to run KServe or you can use KEDA and HPA autoscaling. But you get the metrics to help you determine or allow KServe to scale. So you can see that understanding these metrics are crucial to tuning the inference service.

Elastic Observability can provide this to you.

Why self-hosted LLM tuning is an SRE problem, not an ML problem

Self-hosting an open-weight model is not primarily an ML problem. It is a capacity and saturation problem — something SREs have been extremely good at for twenty years. The blocker was never skill. It was that the telemetry sat unexamined on a /metrics endpoint nobody scraped, in a schema nobody had mapped to the questions they actually had.

Once it's collected, the reasoning is familiar work in unfamiliar clothes:

• Decompose latency by phase before optimizing anything (queue / prefill / decode).
• Distinguish the logical resource from the physical one (KV block pool ≠ VRAM), and know which each metric describes.
• Never trust a single-source utilization number — corroborate the engine's account with the hardware's.
• Tie every knob to a metric and every metric to a stated objective, so tuning converges instead of wandering.

For a team that isn't allowed to send its data anywhere, that difference — between running a model and operating one — is the whole ballgame. The department gets a capability it's otherwise locked out of, and the SRE gets to answer questions about it with numbers.

Frequently asked questions

What does vllm:kv_cache_usage_perc measure? It measures occupancy of vLLM's pre-allocated KV block pool, not physical GPU memory. vLLM reserves a fraction of VRAM at startup (--gpu-memory-utilization, default 0.9) and carves the KV pool from that reservation. In this deployment it read 2.8% while DCGM reported 94.1% VRAM used on the same card at the same moment. Use it as an autoscaling signal; use VRAM for capacity planning.

Why is my vLLM deployment decode-bound? Because the workload generates more tokens than it reads. Compare vllm:request_decode_time_seconds against vllm:request_prefill_time_seconds, and check the generation-to-prompt token ratio. In this deployment the ratio was 2.04 — twice as many output tokens as input — which produced 1,665 ms of decode against 38 ms of prefill. Decode is memory-bandwidth-bound, so quantization, tensor parallelism, or a smaller model help; prefill optimizations do not.

Should I autoscale vLLM on GPU utilization? No. DCGM_FI_DEV_GPU_UTIL means a kernel is resident on the device, not that the device is doing useful work — it reads 100% for both a well-tuned and a badly-tuned server. Autoscale on vllm:kv_cache_usage_perc (around 60%) or on vllm:num_requests_waiting staying above zero, since those describe whether the engine can admit another request.

Why does my GPU show 100% utilization when it isn't fully used? Because GPU utilization only reports kernel residency. Check the DCGM profiling counters instead: in this deployment DCGM_FI_PROF_GR_ENGINE_ACTIVE was 99.8% while DCGM_FI_PROF_PIPE_TENSOR_ACTIVE was only 16.6% and DCGM_FI_PROF_DRAM_ACTIVE was 80.4%. That combination means the GPU is waiting on memory bandwidth rather than computing.

How do I get vLLM metrics into Prometheus? vLLM already exposes Prometheus exposition format on /metrics at its serving port — no adapter or instrumentation needed. Point a Prometheus scrape job at the vLLM Service, add a second job for dcgm-exporter on :9400, and use remote_write to ship to long-term storage. Sending through an OpenTelemetry Collector also works but normalizes the metric names, which makes them harder to match against vLLM documentation.

What TTFT should I target for an interactive LLM application? For a streaming chat-style interface, a p95 time-to-first-token under 300 ms feels immediate, and inter-token latency under 50 ms (about 20 tokens/sec) outpaces reading speed. This deployment measured 51 ms TTFT and 16 ms inter-token latency on a 3B model on a single NVIDIA A10G, leaving roughly 6× headroom.

Why are all my vLLM requests finishing with length? Because they're hitting the max_tokens ceiling instead of the model choosing to stop. Break vllm:request_success_total down by its finished_reason label: a high length share means answers are being truncated mid-sentence. This is invisible in every latency metric, so review the stop-to-length ratio regularly and raise the client-side ceiling if it drifts.

When should I move from a plain vLLM Deployment to KServe or llm-d? Move to KServe when vllm:num_requests_waiting is persistently non-zero at peak and you need replicas to appear without human intervention. Move to llm-d when your prefix-cache hit rate collapses across replicas — a sign that load balancing scattered conversations that shared a prefix — or when prefill time starts stealing measurably from decode.

What you need

• An Observability serverless project, or an Elastic Cloud Hosted or self-managed stack at version 9.4 or later. PromQL is generally available in Elastic Cloud Serverless and Elastic Stack 9.5, and available as a technical preview in Elastic Stack 9.4.
• Kubernetes state and container memory metrics in Elasticsearch.

What is the alert telling us?

This is the alert that opened the investigation:

It comes from this waiting-reason query:

PROMQL
  max by (namespace, pod, container) (
    max_over_time(
      kube_pod_container_status_waiting_reason{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api",
        reason="CrashLoopBackOff"
      }[2m]
    )
  ) == 1

A result of 1 means Kubernetes is delaying another start because the container has failed repeatedly. That is the correct paging signal here because the checkout path has a single replica: when that replica restarts, requests fail.

The max_over_time(...[2m]) range keeps the alert tied to recent samples. Without it, the last observed value of 1 can outlive the pod, and the rule keeps matching after that pod is gone.

That PromQL query ran every minute over a two-minute window and created an alert after one matching run.

Why did the last container stop?

The alert shows what Kubernetes is doing now. It does not show how the previous container ended:

PROMQL
  max by (namespace, pod, container) (
    kube_pod_container_status_last_terminated_reason{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      reason="OOMKilled"
    }
  ) == 1

A result of 1 for the same namespace, pod, and container means the last recorded exit was out of memory. Kube-state-metrics keeps that last reason as a gauge, so the value can stay visible after recovery. It points the investigation at memory; it does not prove that every restart in the window was an OOM kill.

Is the failure repeating?

A single restart can still be transient. The restart counter shows whether the failure keeps happening:

PROMQL
  max by (namespace, pod, container) (
    increase(
      kube_pod_container_status_restarts_total{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api"
      }[10m]
    )
  )

increase() shows how much the restart counter rose over the selected range. Repeated increases during the incident window explain why Kubernetes entered backoff.

How close is memory to the limit?

We need to know how close the container is to its memory limit, and whether that gap collapses right before each restart. This deployment allows only 128MiB, so the next query divides working-set memory by that configured limit:

PROMQL
  max by (namespace, pod, container) (
    container_memory_working_set_bytes{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      image!=""
    }
  )
  /
  max by (namespace, pod, container) (
    container_spec_memory_limit_bytes{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      image!=""
    }
  )

The chart shows a repeating sawtooth: memory approaches 90% of the limit, drops when the process stops, and climbs again after each restart.

Working set is the better signal here than total usage. Total usage includes reclaimable file cache, so it can sit near the limit without a kill. Working set is closer to the memory that triggers OOMKilled for this workload.

Is the node under memory pressure?

OOMKilled can mean the container hit its own limit, or the node ran low on memory and Kubernetes started reclaiming. To separate those cases, first find which node runs the pod, then check whether that node (or any peer) reported MemoryPressure.

PROMQL
  max by (namespace, pod, node) (
    kube_pod_info{
      namespace="checkout",
      pod=~"checkout-api-.*"
    }
  ) == 1

The pod sits on ip-10-0-2-18.ec2.internal. That is the node whose MemoryPressure result matters most for this incident:

PROMQL
  max by (node) (
    max_over_time(
      kube_node_status_condition{
        condition="MemoryPressure",
        status="true"
      }[30m]
    )
  )

Every node returns 0, including ip-10-0-2-18.ec2.internal. So the host was not under node-wide memory pressure. The kill came from the container limit itself.

Does raising the limit clear the alert?

checkout-api was healthy, then began building an in-memory cache that grows to 200MiB in 10MiB steps. The container only allows 128MiB, so the process is killed with OOMKilled before that cache is fully allocated.

We will raise the memory limit to 512MiB so the 200MiB cache fits with room for the runtime, then check whether CrashLoopBackOff clears:

kubectl set resources deployment/checkout-api -n checkout --limits=memory=512Mi

The same waiting-reason query then stops matching. CrashLoopBackOff drops off:

PROMQL
  max by (namespace, pod, container) (
    max_over_time(
      kube_pod_container_status_waiting_reason{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api",
        reason="CrashLoopBackOff"
      }[2m]
    )
  ) == 1

And the alert that started this investigation? Gone.

That is how you detect and investigate a Kubernetes CrashLoopBackOff with PromQL: from the firing alert, through OOMKilled and the limit mismatch, to a recovered alert. Elasticsearch holds the metrics; Kibana runs the same PromQL queries you already know from Prometheus.

Try it

1. Open an Observability project on Elastic Cloud Serverless, or use Elastic Stack 9.4 or later.
2. In the ES|QL editor in Kibana, run the waiting-reason query against a workload you care about.
3. Follow the same path from that alert to termination reason, restarts, memory versus the limit, and recovery.

For more PromQL in Elastic, see Elasticsearch supports PromQL, PromQL queries in Kibana, and Investigate Kubernetes infrastructure with PromQL.

Why managing observability dashboards by hand breaks down at scale

Large organizations often end up with hundreds of dashboards. Teams build similar panels and maintain them using the Kibana UI.

When a small change comes in (a panel rename, a field fix, a new error threshold), there is no easy way to apply it across all of them. You either open each dashboard and edit it in the UI one by one, or you export the NDJSON, run a string replace, and re-import it.

Dashboards are code now

Elastic ships a typed Kibana Dashboards API and a native elasticstack_kibana_dashboard Terraform resource. You define a dashboard in an HCL file and then manage versions and changes as if it was regular code.

Golden signals dashboard: one definition for every team

The platform team owns a standard dashboard built on the four golden signals: latency, traffic, errors, and saturation. Every team should get that standard, and some teams add a panel or two of their own.

We want one definition of the standard, each team's dashboard generated from it, and a single change that reaches every team.

Prerequisites

• An Elastic Cloud deployment or self-managed cluster running Elastic 9.4 or newer, or an Elastic Cloud Serverless project
• Terraform installed
• An Elasticsearch API key

The full Terraform configuration, the seed script, and the captured terraform plan outputs used in this article are available in the companion repo.

Configure the Elastic Terraform provider

Create a provider.tf next to the rest of your Terraform files:

terraform {
  required_providers {
    elasticstack = {
      source  = "elastic/elasticstack"
      version = "~> 0.11"
    }
  }
}

variable "elasticsearch_endpoint" {
  type = string
}

variable "elasticsearch_api_key" {
  type      = string
  sensitive = true
}

variable "kibana_endpoint" {
  type = string
}

variable "kibana_api_key" {
  type      = string
  sensitive = true
}

provider "elasticstack" {
  elasticsearch {
    endpoints = [var.elasticsearch_endpoint]
    api_key   = var.elasticsearch_api_key
  }
  kibana {
    endpoints = [var.kibana_endpoint]
    api_key   = var.kibana_api_key
  }
}

Provide your credentials through a local terraform.tfvars file (and add it to .gitignore so the keys never reach the repo):

elasticsearch_endpoint = "https://...es.region.cloud.es.io"
elasticsearch_api_key  = "..."
kibana_endpoint        = "https://...kb.region.cloud.es.io"
kibana_api_key         = "..."

You can use the same API key for both elasticsearch_api_key and kibana_api_key as long as it has dashboard write privileges in the target space.

Then initialize the working directory:

terraform init

Define a single-team Kibana dashboard in HCL

Start with a baseline dashboard for a single team. Panels sit on a 48-column grid, and each one is a Lens visualization configured inline. Use config_json for KPI tiles (it exposes secondary metrics and value coloring) and xy_chart_config for time-series charts.

Add the baseline resource to a new dashboards.tf:

resource "elasticstack_kibana_dashboard" "golden_signals" {
  title            = "Golden Signals - payments"
  description      = "Latency, traffic, errors"
  query            = { language = "kql", text = "" }
  refresh_interval = { pause = false, value = 60000 }
  time_range       = { from = "now-15m", to = "now" }

  panels = [
    {
      type = "vis"
      grid = { x = 0, y = 0, w = 12, h = 5 }
      config_json = jsonencode({
        type        = "metric"
        data_source = {
          type  = "esql"
          query = "FROM logs-payments-* | STATS `5xx errors` = COUNT(CASE(status >= 500, 1, null))"
        }
        metrics = [{ type = "primary", column = "5xx errors" }]
      })
    },
    # More panels follow the same shape: other metric tiles, xy_chart_config line charts, and a breakdown datatable. See the companion repo for the full file.
  ]
}

Each panel sets a type and grid position, then picks one chart kind. KPI tiles serialize the whole Lens config into config_json; the ES|QL query lives under data_source and the metric column is referenced by name in metrics[*].column. The dashboard time picker already scopes ES|QL panels, so the query needs no explicit @timestamp range filter.

Preview Kibana dashboard changes with terraform plan

Run terraform plan to see what Terraform will create:

terraform plan

The plan output lists the new elasticstack_kibana_dashboard.golden_signals resource and every attribute it will set: the top-level dashboard fields and one entry per panel with its grid position, chart kind, and data source.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # elasticstack_kibana_dashboard.golden_signals will be created
  + resource "elasticstack_kibana_dashboard" "golden_signals" {
      + description      = "Latency, traffic, errors"
      + title            = "Golden Signals - payments"
      + query            = { language = "kql", text = "" }
      + refresh_interval = { pause = false, value = 60000 }
      + time_range       = { from = "now-15m", to = "now" }
      + panels           = [
          # Every panel described in full: KPI tiles (config_json),
          # line charts (xy_chart_config), and the breakdown datatable.
        ]
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Reviewing the plan is your last check before anything ships to Kibana.

Don't apply yet. The next section extends the file with per-team dashboards, and then a single terraform apply ships everything.

Generate per-team observability dashboards from a shared panel library

On top of the baseline, each team gets the standard set of panels, with the option to add a few of their own. Hardcoding one resource per team does not scale. Instead, define a panel library and a teams map as locals, then build the dashboards with for_each. Each library entry describes a chart kind, a title, and the data it needs; the resource emits the right Lens block (config_json for metric tiles, xy_chart_config for line charts) based on chart_type.

Replace the contents of dashboards.tf with:

locals {
  panel_library = {
    errors = {
      chart_type     = "metric"
      title          = "Error rate"
      esql_query_tpl = "FROM {idx} | STATS `5xx errors` = COUNT(CASE(status >= 500, 1, null))"
      esql_column    = "5xx errors"
    }
    saturation = {
      chart_type     = "metric"
      title          = "Saturation (CPU)"
      # Saturation reads from the metrics TSDB, so this query is not parameterized by {idx}.
      esql_query_tpl = "TS metrics-payments-* | STATS avg_cpu = AVG(cpu.pct)"
      esql_column    = "avg_cpu"
    }
    latency = {
      chart_type = "xy"
      title      = "Latency p95"
      x_json     = jsonencode({
        operation          = "date_histogram"
        field              = "@timestamp"
        suggested_interval = "auto"
      })
      y_json = jsonencode({
        operation  = "percentile"
        field      = "duration_ms"
        percentile = 95
      })
    }
    # ... more entries (traffic, cart_value) in the companion repo.
  }

  teams = {
    payments = {
      index  = "logs-payments-*"
      panels = ["errors", "saturation", "latency", "traffic"]
    }
    checkout = {
      index  = "logs-checkout-*"
      panels = ["errors", "cart_value", "latency", "traffic"]
    }
  }
}

resource "elasticstack_kibana_dashboard" "golden_signals" {
  for_each         = local.teams
  title            = "Golden Signals - ${each.key}"
  description      = "Latency, traffic, and errors for the ${each.key} service"
  query            = { language = "kql", text = "" }
  refresh_interval = { pause = false, value = 60000 }
  time_range       = { from = "now-15m", to = "now" }

  sections = [
    {
      title     = "KPIs"
      grid      = { y = 0 }
      collapsed = false
      panels = [
        for i, p in [for q in each.value.panels : q if local.panel_library[q].chart_type == "metric"] : {
          type        = "vis"
          grid        = { x = (i % 4) * 12, y = 0, w = 12, h = 5 }
          config_json = jsonencode({ ... }) # one metric tile per panel; see the companion repo for the full config
        }
      ]
    },
    {
      title     = "Trends"
      grid      = { y = 1 }
      collapsed = false
      panels = [
        for i, p in [for q in each.value.panels : q if local.panel_library[q].chart_type == "xy"] : {
          type       = "vis"
          grid       = { x = (i % 3) * 16, y = 0, w = 16, h = 10 }
          vis_config = { by_value = { xy_chart_config = { ... } } }
        }
      ]
    },
    # A third "Breakdown" section holds the request-by-status datatable. See the companion repo.
  ]
}

Adding a team is one entry in teams. Adding a panel to every team is one entry in panel_library and one reference per team. The full config (data source ES|QL queries, metrics, layers, axis defaults, and legend placement) lives in dashboards.tf.

The saturation panel queries the metrics data stream with the ES|QL TS command, which is designed for TSDB. For the query to work, data streams matching metrics-payments-* must use time_series mode, so the configuration also ships an index template (metrics_tsdb.tf) that enables that.

Apply dashboards as code to Kibana with terraform apply

Run terraform plan to confirm both team dashboards (payments and checkout) will be created then apply:

terraform apply

Open Kibana and you'll see one Golden Signals dashboard per team, each backed by its own index pattern.

Dashboards as code in the GitOps loop: review changes in pull requests

Dashboards are now an artifact in version control, like the rest of your infrastructure.

You edit the library or a team's selection, open a pull request, your reviewer reads the terraform plan diff and sees which dashboards change.

For example, say you tighten the "critical error" threshold from status >= 500 to status >= 503 in panel_library.errors.esql_query_tpl. Running terraform plan shows the change reaching both teams at once:

Note: Full output in terraform-plan-update.txt.

A single edit to panel_library.errors propagates to every team that references it. After the PR merges, it's time to run terraform apply.

After the apply finishes, refresh the dashboards in Kibana and the new threshold is in effect:

Detect dashboard drift and roll back with git

If someone edits a dashboard using the UI, the next terraform plan shows the difference, because the code and the live state no longer match.

To see this in action, open Golden Signals - payments in Kibana, rename the Latency p95 panel to Latency p95 (EDITED), and save the dashboard.

Then run terraform plan:

Terraform reads the panel title from the live dashboard, compares it against the code, and proposes reverting the UI rename. You decide whether to keep the change (update the code to match) or revert it by running terraform apply.

You can commit the new version, or rollback one or many versions using git.

Replaying the earlier example: if you reopen the PR that changed panel_library.errors to broaden the error threshold and add a clearer title, git diff dashboards.tf shows the entire intent in two lines:

Every team that references errors picks up the new threshold on the next terraform apply, and reverting that commit rolls the change back across all of them at once.

Wrap up

Managing Kibana observability dashboards by hand does not scale past a few teams. With the Kibana Dashboards API and Terraform, you define a standard once, compose each team's dashboard from a shared library, and review every change in a pull request. One edit reaches every team, and you can roll back by reverting a commit.

The proposed file structure is only one of many ways you can organize your dashboards depending on how much information they share.

Next steps

• Kibana Dashboards as code with Terraform
• elasticstack_kibana_dashboard resource reference
• Elastic Stack Terraform provider documentation
• Kibana Dashboards API documentation
• ES|QL reference

If you've been an SRE for any length of time, you know what usually comes next. You acknowledge the page, open your observability tool, and start the process: pull up the cluster dashboard, find the namespace, find the pod, check restart counts, pivot to logs, check whether an upstream dependency is degraded, compare against last week, and slowly assemble a picture from a dozen tabs. It works, but it's a process, and the process is where the minutes go.

Elastic's new Kubernetes Experience changes the starting point. When the CrashLoopBackOff alert fires, an Investigation Workflow runs automatically alongside it. By the time you open the alert, the evidence has already been gathered and a root-cause hypothesis is waiting for you. Instead of a blank dashboard, you open the alert to an answer. Or at minimum, a strong starting point that tells you exactly where to look next.

This post walks through a typical CrashLoopBackOff scenario end to end. The sections that follow break down what Elastic's UI shows at each step and why it saves you time.

The manual CrashLoopBackOff investigation: six steps every SRE runs

Here's the shape of a normal CrashLoopBackOff investigation, minus Elastic's workflow:

1. You get paged. Something restarted too many times.
2. You orient. Which pod? Which namespace? Which deployment owns it?
3. You characterize. How many restarts? What was the last termination reason — OOMKilled, a failed liveness probe, a bad exit code?
4. You classify. Is this a memory problem, a config problem, a dependency problem, a scheduling problem?
5. You corroborate. Pull Kubernetes events, read the pod logs, check whether an upstream service started erroring first, compare current behavior against a healthy baseline.
6. You conclude. Only now can you form a hypothesis and act.

Every one of those steps is a query, a click, or a context switch. None of them is hard on its own. Together, on a bad night, they're twenty minutes you don't have and they're twenty minutes of the same steps you ran during the last incident, and the one before that.

The insight behind Elastic's Kubernetes Experience is simple: that sequence is deterministic enough to automate. So Elastic automated it.

Alert to Root cause in minutes

Elastic's Kubernetes integration ships pre-built alert rule templates for states that are wrong by definition; no baseline or warmup required. A pod in CrashLoopBackOff is always a problem, so the rule fires the moment the restart count crosses your configured threshold within a rolling window.

Here's the alert firing in this scenario, both the [Kubernetes OTel] Pod CrashLoopBackOff and OOMKilled containers rules light up for the recommendation group in the otel-demo namespace:

The alert itself is defined by an ES|QL query, so it's transparent and tunable; you can read exactly what triggers it and adjust the threshold to match your environment. And its action is what makes the rest of this post possible: the rule runs the K8s CrashLoopBackOff Investigation (OTel) workflow per alert, the instant a new alert fires. (For a refresher on the alert library and how these templates work, see Part 1.)

In this scenario the rule fires on the recommendation pod (recommendation-788dc88c6c-56w74) in the otel-demo namespace. When you open the alert, Elastic's AI Agent is already summarizing what happened; no rule-details spelunking required:

But the alert firing is only half the story. Attached to it is a Kubernetes Investigation Workflow (technical preview), a directed graph of steps that triggers the instant the alert does. While your phone is still buzzing, the workflow is already querying your cluster, branching on what it finds, and synthesizing an answer.

Inside the investigation: what the workflow actually does

The workflow mirrors the exact sequence an experienced SRE would run by hand, except it runs in seconds and writes nothing down that it can't back up with evidence. For our CrashLoopBackOff scenario, here's the path it takes.

Step 1: What does the workflow check first on the crashing pod?

The workflow queries Kubernetes metrics for the restart count, the last termination reason, and utilization against the pod's declared limits.

Result: last termination reason OOMKilled, restart count 7. Memory utilization data happened to be unavailable at query time, but the OOMKilled reason is definitive: the container is being killed by the kernel for exceeding its memory limit on each startup, then immediately restarting.

The OOMKilled termination reason determines everything that follows. The workflow branches down the memory-investigation path rather than the log-investigation path it would take for a non-memory crash.

Step 2: How does the workflow tell a memory leak from a load spike?

The ML anomaly check is the step that separates a good investigation from a fast-but-wrong one. OOMKilled does not automatically mean "memory leak." Rather than recompute memory trends from scratch, the workflow queries the ML anomaly index for an active k8s_pod_memory_growth anomaly on this pod.

Result: no anomaly. The memory spike is flagged as load-driven, not a suspected leak. The ML baseline — established over the preceding days — didn't see the slow, creeping growth trajectory that characterizes a leak. It saw a jump consistent with real traffic.

Distinguishing a load-driven memory spike from a genuine leak would take a human several minutes and a good deal of judgment. The workflow reaches it because Part 1's anomaly detection jobs were already learning the workload's baseline in the background.

Step 3: Is the failure spreading to other Kubernetes services?

A crashing pod is often a symptom, not a cause, and it can also cause problems downstream. So the workflow enumerates the pod's dependencies from APM service_destination aggregates and compares the current error rate and latency against baseline. An AI classification step decides whether the failure is spreading.

Result: the sole direct caller is the frontend service, which is absorbing the impact with just a 0.14% error rate against the recommendation service and no other service exceeds its degradation threshold relative to baseline. The blast radius is isolated to the recommendation service; there's no significant downstream cascade. The problem is local to this pod.

Step 4: Did a recent change in the namespace cause the crash loop?

Finally, the workflow scans the namespace event log. It finds a continuous Pulled → Created → Started → Killing → BackOff cycle running from roughly 18:51 to 18:54 UTC, the textbook signature of an active crash loop at the time the alert fired. Nothing changed operationally; this is a steady-state resource problem.

What the CrashLoopBackOff root-cause hypothesis looks like

When you open the alert, this is what greets you:

ROOT CAUSE HYPOTHESIS (confidence: high)

The recommendation service pod (recommendation-788dc88c6c-56w74) is in a
crash-loop caused by repeated OOMKilled terminations. The pod has restarted
7 times and Kubernetes events confirm a continuous BackOff/restart cycle
since at least 18:51 UTC. Memory utilization data was unavailable at query
time, but the OOMKilled termination reason is definitive: the container is
exceeding its configured memory limit on each startup, being killed by the
kernel, and immediately restarting. No memory leak was detected by ML
anomaly analysis, indicating the memory pressure is load-driven — the
container's memory limit is simply insufficient for the current request
volume. The frontend service (the sole direct caller) is absorbing the
impact with a 0.14% error rate on the recommendation service itself, but
no significant downstream cascade is observed.

EVIDENCE
- Pod restarted 7 times; last termination reason: OOMKilled — container is
  consistently exceeding its memory limit
- ML memory anomaly check: no anomaly found; memory spike assessed as
  load-driven, not a leak
- Blast radius is isolated to the recommendation service; no other service
  exceeds degradation thresholds relative to baseline
- Continuous BackOff events from 18:51–18:54 UTC confirm active crash-loop
  at alert time

PROBABLE CAUSE: The recommendation container's memory limit is too low for
current traffic load, causing repeated OOMKilled terminations and a
crash-loop backoff.

RECOMMENDED NEXT STEPS
1. Immediately increase the memory limit (and request) for the
   recommendation container in its Deployment spec to provide headroom
   above the observed peak usage, then redeploy to break the crash-loop.
2. Profile the recommendation service under representative load to
   determine the actual memory working set and set a right-sized limit
   with a safe buffer (e.g., 20–30% above peak observed).
3. Add a Kubernetes HorizontalPodAutoscaler or VPA policy for the
   recommendation service so memory resources scale with traffic rather
   than requiring manual intervention.

Read that again from the perspective of the on-call engineer. You were paged. You opened the alert. And the alert didn't hand you a pile of logs and a dashboard; it handed you a calibrated root-cause hypothesis with the evidence attached and the next actions spelled out. The six manual steps from the "old way" are done. Your job is now to decide, not to dig.

That's the time savings: not shaving a few seconds off each query, but removing the entire investigative scavenger hunt from the critical path.

How does Elastic Observability avoid misdiagnosing OOMKilled as a memory leak?

Speed is worthless if the answer is wrong, so it's worth noting how the workflow avoids the classic misdiagnoses. It encodes the reasoning an experienced SRE applies instinctively:

• OOMKilled is not automatically a leak. It compares against a 7-day baseline before ever claiming one. Here, that check is what turned "the app is leaking memory" into the correct "the limit is undersized for real load."
• Co-symptoms are not causes. It explicitly checks whether the upstream degraded first before blaming or clearing it.
• Absence of evidence is not evidence. If a query returns zero rows, it reports "no data available" rather than inventing a failure mode.
• It's honest about confidence. The hypothesis is labeled high, medium, or low. When two failure modes fit the evidence, the workflow names both and says which it believes is causal and why. Manufacturing false confidence is treated as a failure of the investigation itself.

This is the same diagnostic protocol encoded in Elastic's observability-k8s-investigation Skill, a failure-mode taxonomy covering 16 distinct Kubernetes failure patterns, from OOMKilled and CPU throttling through scheduling and networking issues. (More on that in Part 2.)

Still want to double-check? Dashboards, Discover, and APM

The workflow gives you the answer. But a good root-cause tool should also make it trivial to verify that answer because sometimes you want to see it with your own eyes, and sometimes the workflow returns medium confidence and you need to close the gap yourself. Everything the workflow reasoned over is available to you directly.

Dashboards — confirm the restart cascade visually. The Kubernetes dashboards are built for drill-down. Start at the cluster Overview, where "top namespaces by container restarts" surfaces the problem at a glance. Click into the flagged namespace, then the pod driving the restarts. The Pods view flags container restarts on the recommendation pod and plots memory against requests and limits over time — you'll see the working set pressing against the limit exactly as the workflow described. It's roughly four clicks from cluster to container.

Discover — read the raw evidence. The pod detail dashboard links directly to correlated pod logs and events in Discover. Here you can confirm the OOMKilled events and the restart cadence in the raw log and event stream, and run your own ES|QL queries if you want to slice the data differently.

APM — verify the blast radius is really contained. The workflow found the impact isolated to the recommendation service, with frontend (the sole caller) absorbing it at a 0.14% error rate. You can confirm that independently in the APM UI: open the service map, check the caller's latency and error rate over the incident window, and compare against the weekly baseline yourself.

The point isn't that you have to do any of this; it's that the workflow's conclusion is fully auditable. Fast when you trust it, transparent when you want to check.

The same investigation from your IDE: the MCP App

Not every investigation starts from a Kibana alert. Sometimes a developer just asks, "why is this service crashing?" from their editor. Elastic's Observability MCP App (technical preview) exposes the same telemetry (and the same investigation workflow) as AI-callable tools that render interactive views inline in your chat or IDE, no context switch to Kibana required.

For our CrashLoopBackOff scenario, the flow looks like this from an MCP-compatible client such as Claude Desktop or VS Code:

"What's broken?" → the cluster health rollup returns an overall health badge, degraded services, top memory consumers, and a Kubernetes breakdown (CPU, memory, restarts, nodes) in one inline view. Here it flags a critical cluster with payment, cart, frontend, and frontend-proxy degraded.

"Is anything anomalous in the recommendation pod?" → the memory analysis view confirms the spike is load-driven, not a leak — the crashing 788dc88c6c pods report null memory (they die too fast to emit a sample) while the healthy pods sit flat at 45MB, exactly the ML result the workflow used.

"Why is the recommendation service crashing?" → the agent returns the same structured root-cause reasoning you'd see on the alert, rendered inline: memory limit set below what the container needs to boot, the ReplicaSet has been intermittently OOMing for weeks, and a concrete mitigation (roll back, then fix the limit), all without leaving the editor.

Same evidence, same root cause, delivered wherever you happen to be working. (For the full set of MCP App views and architecture, see Part 2.)

From paged to decided: skipping the Kubernetes investigation entirely

The change here is not "a better dashboard." It's a shift in what you do when you get paged.

Before, the alert was the start of the investigation. You were notified, and then you went and found the answer. Now, the alert arrives with the investigation already run: evidence gathered, dead ends eliminated, a calibrated hypothesis, and next steps in hand. You go straight from "notified" to "deciding," and you keep the full trail of dashboards, Discover, and APM for whenever you want to verify or dig deeper.

For a single incident that's a few minutes saved. Across a quarter of on-call rotations, across every engineer who no longer re-runs the same six steps at 3 a.m., it's real time back and a lot less alert fatigue.

Try it yourself

You don't need a production incident to see this. The OpenTelemetry Astronomy Shop demo environment ships with a feature-flag service that lets you trigger failure scenarios on demand. Enable a cart/checkout failure, watch the restart cascade unfold, and the CrashLoopBackOff alert rule fires with the investigation workflow running right behind it.

To get set up:

1. Install the Kubernetes integration — dashboards are available immediately. (Part 1: Getting started)
2. Deploy data collection via the EDOT Collector (OpenTelemetry) or standalone Elastic Agent, both Helm-based.
3. Enable the alert rule templates in Observability > Alerts, including CrashLoopBackOff, and connect your notification channel.
4. Let the ML modules warm up for 24–48 hours so anomaly baselines are ready when you need them.
5. Enable the Investigation Workflow (technical preview) — import the Kubernetes Crashloop Investigation Workflow from the Workflows page and configure it to trigger on the alert. (Part 2: Getting started)
6. Install the MCP App (technical preview) on your favorite agentic client to bring investigations into your IDE.

Running Kubernetes on Elastic today? Tell us which investigation steps you still repeat by hand on every incident, and which remediations you'd trust a workflow to propose. Join the Elastic Community Discussion.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The Datadog Kubernetes dashboard used in this migration

The walkthrough uses Kubernetes - Overview from infra/datadog/dashboards/integrations/kubernetes.json in the migration repository. It is a cluster-wide board with the signals operators check during an incident: pod counts, CPU and memory by host or pod, non-running pods, and containers stuck in CrashLoopBackOff.

Below are representative queries from the source dashboard:

# Pod CPU by host
sum:kubernetes.cpu.usage.total{$scope,$cluster,$label,$node} by {host}

# Pod memory by pod
sum:kubernetes.memory.usage{$scope,$deployment,$statefulset,$replicaset,$daemonset,$cluster,$namespace,!pod_name:no_pod,$label,$service,$node} by {pod_name}

# Pods not running (pressure / scheduling signal)
sum:kubernetes_state.pod.status_phase{$scope,$cluster,$namespace,$deployment,$statefulset,$replicaset,$daemonset,!pod_phase:running,!pod_phase:succeeded,$label,$node,$service} by {kube_cluster_name,kube_namespace,pod_phase}

# CrashLoopBackOff
sum:kubernetes_state.container.status_report.count.waiting{$cluster,$namespace,$deployment,$statefulset,$replicaset,$daemonset,reason:crashloopbackoff,$scope,$daemonset,$label,$node,$service} by {pod_name}

If this board translates cleanly, most production Datadog Kubernetes folders are worth testing with the same workflow.

Why Datadog-to-Elastic migration is faster now

The migration platform automates the query translation and panel rebuilds that used to dominate Datadog moves. Elasticsearch stores Kubernetes metrics efficiently and runs the ES|QL queries those panels use. See Elasticsearch as a metrics backend for benchmark context and storage comparisons.

The platform maps Datadog queries to Kibana panels, validates ES|QL against live data, and writes artifacts you can inspect before anything goes to production.

Prerequisites

You need an Elastic Observability Serverless project, a project API key, and the migration CLI installed from the Observability Migration Platform repository.

Export your endpoints and API key:

export ELASTICSEARCH_ENDPOINT="https://YOUR_ES_ENDPOINT"
export KIBANA_ENDPOINT="https://YOUR_KIBANA_ENDPOINT"
export KEY="YOUR_API_KEY"

Install the CLI and confirm the toolchain:

python3 -m venv .venv
.venv/bin/pip install ".[all]"
.venv/bin/obs-migrate doctor

The doctor command checks compile and lint dependencies. Resolve any errors before you migrate production dashboards. Pin a release tag if you plan to run this in CI.

To pull dashboards from the Datadog API instead of JSON files, copy datadog_creds.env.example to datadog_creds.env and set DD_API_KEY, DD_APP_KEY, and DD_SITE.

Ingest Kubernetes metrics first

Empty panels after upload usually mean Elasticsearch does not yet have the series the Datadog queries reference. Make sure to confirm ingest before you run the migration.

There are two common paths to do so:

1. OpenTelemetry into managed OTLP with Kubernetes receivers (kubeletstats, k8s_cluster), then explore in Discover
2. Existing Prometheus or agent pipelines that already write pod and node metrics to metrics-*

The migration CLI accepts --field-profile otel to map Datadog tags such as pod_name, kube_namespace, and kube_cluster_name to OpenTelemetry fields like kubernetes.pod.name and kubernetes.namespace. If panels are empty after migration, verify field mapping and the selected time range before you change translator settings.

Run the Datadog dashboard migration CLI

Export the Datadog dashboard JSON from the UI, or copy the sample kubernetes.json from infra/datadog/dashboards/integrations/ in the migration repo. Place files in a directory such as ./datadog_k8s_exports/.

Run the migration from that directory:

datadog-migrate \
  --source files \
  --input-dir ./datadog_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --field-profile otel \
  --data-view "metrics-*" \
  --logs-index "logs-*" \
  --upload \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --ensure-data-views \
  --create-alert-rules \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY"

These flags matter for Kubernetes boards:

• --field-profile otel maps Datadog Kubernetes fields to OpenTelemetry field names in Elasticsearch
• --assets all includes dashboards and Datadog monitor definitions when present
• --validate runs emitted ES|QL against your cluster before upload
• --create-alert-rules creates Kibana rules in a disabled state

The unified CLI performs the same work:

obs-migrate migrate \
  --source datadog \
  --input-mode files \
  --input-dir ./datadog_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --field-profile otel \
  --data-view "metrics-*" \
  --logs-index "logs-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload \
  --create-alert-rules

To fetch a dashboard from Datadog directly:

datadog-migrate \
  --source api \
  --env-file datadog_creds.env \
  --dashboard-ids YOUR_DASHBOARD_ID \
  --output-dir ./migration_output \
  --assets all \
  --field-profile otel \
  --data-view "metrics-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload

Validate the migrated Datadog dashboard in Kibana

Open Kibana → Dashboards and locate the migrated Kubernetes - Overview board. Confirm that cluster and namespace pod counts, CPU and memory series, pod phase panels, CrashLoopBackOff widgets, and deployment replica charts return data for your selected time range.

If you migrated monitors, open Observability → Rules. Imported rules remain disabled until you enable them after reviewing thresholds.

The CLI also writes local artifacts under ./migration_output/:

• dashboards/yaml/ contains the translated dashboard definition.
• dashboards/migration_report.json lists panels that translated automatically and panels flagged for manual review.
• alerts/ contains monitor translations when monitors were included in the export.

Handle manual-review panels

Some Datadog widget types do not translate on the first pass. Exotic formulas, log-only panels, and unsupported widgets appear as manual-review entries in the migration report rather than as silently broken charts.

| Result | Recommended action | | --- | --- | | Panel returns data | Accept the translation and continue | | Panel is empty | Confirm metric names and data_stream.dataset values in metrics-*, then widen or shift the time range | | Manual-review marker | Open the original Datadog query and simplify or redesign the panel | | Monitor never fires | Confirm the rule is enabled and thresholds match your environment |

Datadog coverage is narrower than Grafana in some areas. Read the migration report before you commit to full parity with leadership. The platform prefers conservative failures over uploading panels that look correct but query the wrong fields.

Related Datadog and Grafana migration guides

For the Grafana PromQL version of this workflow, see Migrate your Grafana Kubernetes dashboard to Elastic Observability. For platform-level context, see Migrating Datadog and Grafana dashboards and alerts to Kibana. Review known limitations before you migrate every production folder.

The migration tool keeps panel queries in PromQL instead of rewriting them into a new dialect, validates them when you pass --validate, and uploads compiled dashboards to Kibana. You still review the migration report and enable alerts on your schedule.

Sample Grafana Kubernetes dashboard used in this migration

The walkthrough uses Kubernetes / Views / Global, a community-style Grafana dashboard in the migration repository. It includes the signals operators check during an incident: namespace CPU and memory, throttling pressure, and restart counts.

Below are representative PromQL queries from the source dashboard:

# Pod / container CPU by namespace
sum(rate(container_cpu_usage_seconds_total{image!="", cluster="$cluster"}[$__rate_interval])) by (namespace)

# Memory working set by namespace
sum(container_memory_working_set_bytes{image!="", cluster="$cluster"}) by (namespace)

# Node / CPU pressure style signal: throttled seconds
sum(rate(container_cpu_cfs_throttled_seconds_total{image!="", cluster="$cluster"}[$__rate_interval])) by (namespace) > 0

# Container restart counts
sum(increase(kube_pod_container_status_restarts_total{cluster="$cluster"}[$__rate_interval])) by (namespace) > 0

If this board translates cleanly, most production Grafana Kubernetes folders are worth testing with the same workflow.

Why Grafana-to-Elastic migration is faster now

The migration platform automates the query translation and panel rebuilds that used to dominate Grafana moves. Native Prometheus Remote Write and PromQL in Kibana let you keep the dialect your on-call team already uses. The --native-promql flag passes PromQL through unchanged. See Elasticsearch as a metrics engine for storage and query context.

Prerequisites

You need an Elastic Observability Serverless project, a project API key, and the migration CLI installed from the observability-migration-platform repository.

Export your endpoints and API key:

export ELASTICSEARCH_ENDPOINT="https://YOUR_ES_ENDPOINT"
export KIBANA_ENDPOINT="https://YOUR_KIBANA_ENDPOINT"
export KEY="YOUR_API_KEY"

Install the CLI and confirm the toolchain:

python3 -m venv .venv
.venv/bin/pip install ".[all]"
.venv/bin/obs-migrate doctor

The doctor command checks compile and lint dependencies. Resolve any errors before you migrate production dashboards. Pin a release tag if you plan to run this in CI.

How do you get Kubernetes metrics into Elasticsearch?

Empty panels after upload usually mean Elasticsearch does not yet have the series the PromQL references. Make sure to confirm ingest before you run the migration.

There are two common paths to do so:

1. Prometheus Remote Write into Elasticsearch from kube-state-metrics, cAdvisor or kubelet metrics, and node exporters.
2. OpenTelemetry into managed OTLP for Kubernetes receivers, then explore in Discover.

Test with a query such as sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace) against Elastic. If that returns data, continue. If it does not, fix ingest first.

Run the Grafana dashboard migration CLI

Export your Grafana dashboard JSON, or copy the sample k8s-views-global.json from infra/grafana/dashboards/ in the migration repo. Place files in a directory such as ./grafana_k8s_exports/.

Run the migration from that directory:

grafana-migrate \
  --source files \
  --input-dir ./grafana_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --native-promql \
  --data-view "metrics-*" \
  --esql-index "metrics-*" \
  --upload \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --ensure-data-views \
  --create-alert-rules \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY"

These flags matter for Kubernetes boards:

• --native-promql keeps pod CPU, memory, throttling, and restart queries in PromQL
• --assets all includes dashboards and Grafana PromQL alert definitions when present
• --validate runs emitted queries against Elasticsearch before upload
• --create-alert-rules creates Kibana rules in a disabled state

The unified CLI performs the same work:

obs-migrate migrate \
  --source grafana \
  --input-mode files \
  --input-dir ./grafana_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --native-promql \
  --data-view "metrics-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload \
  --create-alert-rules

Validate the migrated Grafana dashboard in Kibana

Open Kibana → Dashboards and locate Kubernetes / Views / Global. Confirm that namespace or pod CPU utilization, memory working set panels, throttling or pressure widgets, and restart charts return data for your selected time range.

If you migrated alerts, open Observability → Rules. Imported rules remain disabled until you enable them after reviewing thresholds.

The CLI also writes local artifacts under ./migration_output/:

• dashboards/yaml/ contains the translated dashboard definition.
• dashboards/migration_report.json lists panels that translated automatically and panels flagged for manual review.
• alerts/ contains alert translations when alert definitions were included in the export.

What do you do when Grafana panels don't migrate automatically?

The Observability Migration Platform flags PromQL expressions that do not translate automatically. Hard joins, unusual arithmetic, and a few Alertmanager-era edge cases appear as manual-review entries in the migration report rather than as silently broken charts.

| Result | Recommended action | | --- | --- | | Panel returns data | Accept the translation and continue | | Panel is empty | Confirm metric names exist in metrics-*, then widen or shift the time range | | Manual-review marker | Open the original PromQL and simplify or redesign the panel | | Alert never fires | Confirm the rule is enabled and thresholds match your environment |

This path migrates Grafana PromQL dashboards and Grafana unified PromQL alert definitions into Kibana. It does not ingest a raw alertmanager.yml. The goal is to keep the PromQL your pager already trusts instead of rebuilding the Kubernetes board from zero.

Related guides

For platform-level context, see Migrating Datadog and Grafana dashboards and alerts to Kibana. Review known limitations before you migrate every production folder.

Elasticsearch now stores Prometheus metrics in the same columnar backend as your logs and traces, at 3.75 bytes per datapoint, with no custom-metric penalty. The graph, the logs, and the traces answer to one query language.

Using an existing Kubernetes cluster (AWS EKS in this example): point Prometheus at Elastic with one config block, see every metric render in Discover with no dashboard to build, run most of your existing PromQL unchanged, find where ES|QL takes you past what PromQL can express, and finish by reading your logs with the same query language.

Nothing about your collection changes. Your scrape configs, relabeling rules, and service discovery carry over as-is. Most of your PromQL comes with you too — see the coverage note in Step 5 for the gaps.

Step 1: get a Prometheus endpoint and an API key from Elastic Observability

You need Elastic Cloud Serverless or Elastic Cloud Hosted. Both expose the Prometheus endpoints with no configuration.

Serverless is the fastest start. Sign in at cloud.elastic.co and create an Observability project. There is nothing to size and nothing to provision.

Elastic Cloud Hosted works the same way for everything in this post, and is the right choice when you need a specific stack version, a specific region topology, or the deployment-level controls that come with a managed cluster.

Find the Prometheus endpoint and create an API key in the UI

For Prometheus endpoint and the API key go to Kibana, click Add data in the left nav.

For Prometheus, scroll to Connect directly to the endpoint at the bottom and select the Prometheus tab.

Two things to copy:

The endpoint. It looks like https://my-observability-project-xxx.ingest.us-west-2.aws.elastic.cloud. Note the .ingest. host. This is not the same host as your Elasticsearch search endpoint or your Kibana URL.

The API key. Click Create key. Copy the value before you close the dialog, because it is not retrievable afterward.

If you want to scope it by hand instead, Open in API keys takes you to the full editor, and the minimum privilege for metrics ingest is:

{
  "ingest": {
    "indices": [
      {
        "names": ["metrics-*"],
        "privileges": ["auto_configure", "create_doc"]
      }
    ]
  }
}

Keep both values. Every remaining step uses them.

Step 2: make sure Prometheus is scraping your Kubernetes cluster

Prometheus should already be scraping your cluster.

If it is not, the shortest path is the kube-prometheus-stack Helm chart, which installs the Prometheus Operator, Prometheus itself, kube-state-metrics, and node-exporter in one command:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

That gives you the metrics you see for the rest of this post:

• cAdvisor container metrics (container_cpu_usage_seconds_total, container_memory_working_set_bytes)
• kube-state-metrics cluster objects (kube_deployment_spec_replicas, kube_daemonset_status_number_ready).

Step 3: configure Prometheus remote write to the Prometheus endpoint in Step 1

Elasticsearch implements the Prometheus Remote Write protocol natively. There is no adapter, no sidecar, and no translation layer. You add one block and the data flows on the next scrape interval.

If you run the Prometheus Operator

The Operator does not read a prometheus.yml you write by hand. It generates one from the Prometheus custom resource, and authorization.credentials there is a reference to a Kubernetes Secret, not an inline value. Create the secret first:

kubectl create secret generic elastic-prometheus \
  --namespace monitoring \
  --from-literal=api_key='YOUR_API_KEY'

Then reference it from values.yaml:

prometheus:
  prometheusSpec:
    remoteWrite:
      - url: "https://my-observability-project-xxxx.ingest.us-west-2.aws.elastic.cloud:443/api/v1/write"
        authorization:
          type: ApiKey
          credentials:
            name: elastic-prometheus
            key: YOUR_API_KEY

And apply it:

helm upgrade prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  -f values.yaml

Configure Prometheus remote write with prometheus.yml

Same thing, in prometheus.yml:

remote_write:
  - url: "https://YOUR_ES_ENDPOINT/_prometheus/metrics/node/eks/api/v1/write"
    authorization:
      type: ApiKey
      credentials: YOUR_API_KEY

If you run Grafana Alloy uss the following configuration

prometheus.remote_write "elasticsearch" {
  endpoint {
    url = "https://YOUR_ES_ENDPOINT/_prometheus/metrics/node/eks/api/v1/write"
    headers = {"Authorization" = "ApiKey YOUR_API_KEY"}
  }
}

How the remote write URL maps to Elasticsearch data streams

You do not name an index anywhere. There is no index field in the payload and no data stream in your remote_write config. Elasticsearch derives the target data stream from the write path and creates it on the first sample. The two path segments after /metrics/ are the dataset and the namespace:

| URL | Data stream | |---|---| | /_prometheus/api/v1/write | metrics-generic.prometheus-default | | /_prometheus/metrics/{dataset}/api/v1/write | metrics-{dataset}.prometheus-default | | /_prometheus/metrics/{dataset}/{namespace}/api/v1/write | metrics-{dataset}.prometheus-{namespace} |

The examples above use /metrics/node/eks/, which writes to metrics-node.prometheus-eks. That is the data stream you will see in Discover in the next step. Use dataset and namespace to separate production from staging, or to give each cluster and each team a data stream with its own retention and downsampling policy.

If you would rather keep a bare /api/v1/write URL, you can route per time series instead: attach data_stream_dataset and data_stream_namespace labels to the series, and they take precedence over the URL path. These two are control fields, so they route the document without being stored in its labels object.

Elasticsearch installs the index template for metrics-*.prometheus-* itself. You do not create templates or mappings.

What Prometheus metrics look like in Elastic Observability

Every Prometheus sample becomes a document. Labels become keyword fields that serve as time series dimensions. The value goes under metrics.<metric_name>:

{
  "@timestamp": "2026-07-02T10:30:00.000Z",
  "data_stream": {
    "type": "metrics",
    "dataset": "node.prometheus",
    "namespace": "eks"
  },
  "labels": {
    "__name__": "container_memory_working_set_bytes",
    "pod": "checkout-7d9f6c4b8-x2kqp",
    "namespace": "oteldemo",
    "container": "checkout",
    "node": "ip-10-0-3-14.us-west-2.compute.internal"
  },
  "metrics": {
    "container_memory_working_set_bytes": 36700160
  }
}

The one gotcha to know now. Elasticsearch infers whether a metric is a counter or a gauge from its name. Names ending in _total, _sum, _count, or _bucket are counters. Everything else is a gauge. That inference is correct for container_cpu_usage_seconds_total and correct for container_memory_working_set_bytes. It is wrong for any metric in your estate that does not follow Prometheus naming convention, and a misclassified metric gets rejected by the function that should accept it: RATE(my_metric::counter) works on counters only. Step 6 shows how to override the inference.

Current limits. Remote Write v1 only. Classic histograms and summaries are supported through their _bucket, _sum, and _count series, each mapped to the right metric type. Native (sparse) histograms and exemplars arrive with Remote Write v2, which is on the roadmap. Staleness markers are not stored or respected, and non-finite values (NaN, Infinity) are dropped silently. See the Prometheus remote write ingest docs for the full list.

Step 4: verify Prometheus metrics are flowing into Elasticsearch

Do not go build a dashboard. Confirm the pipeline first, and Elastic makes that a single command.

In Kibana, open Discover, switch the query editor to ES|QL, and type the name of the data stream you just wrote to:

TS metrics-node.prometheus-eks

That is the whole query. Discover reads the data stream, finds every metric in it, and renders each one as its own time series chart. Fifty metrics, fifty charts, no dashboard to build and no query to write per metric. It reads the metric type and charts each one correctly: gauges as averages, counters as rates, histograms as p95 distributions.

You can also get here without typing anything. Observability → Streams lists every data stream in the cluster. A Time series badge means it is a time series data stream. Click View in Discover and the TS query is filled in for you.

This is your ingest health check. Three things to look for.

• Data is flowing. Recent, continuous values. Not gaps, and not a line that stops an hour ago.
• Values are plausible. Memory in the tens of megabytes for a small container. CPU as a fraction of a core. Network bytes tracking real traffic.
• Coverage is what you expected. If kube_pod_container_status_restarts_total is missing, your kube-state-metrics scrape config is wrong, and you want to know that now rather than when you are building an alert on it.

Widen the time picker before you conclude anything is broken. A 15-minute window over a quiet period makes healthy data look flat.

To list what actually has data rather than what the mapping declares:

TS metrics-node.prometheus-eks | METRICS_INFO | SORT metric_name

The No dimensions selected control above the charts lets you break every chart out by a label: select pod and each chart splits into one series per pod.

Step 5: run PromQL queries on Prometheus metrics in Elastic Observability

If your team writes PromQL, keep writing PromQL. PROMQL is a source command in ES|QL, alongside FROM and TS, and it runs anywhere ES|QL runs: Discover, dashboard panels, and alert rules.

It does not run a separate engine. It parses the expression, resolves each function to its ES|QL equivalent, and builds a TS execution plan, so your PromQL gets the same vectorized, parallel execution as native ES|QL.

Current limits. PROMQL is generally available on Elastic Cloud Serverless and a tech preview on Elastic Stack 9.4, benchmarked at over 80% query coverage against popular Grafana OSS dashboards. The gaps worth knowing before you paste a dashboard in: histogram_quantile is not yet implemented, which matters most because it is how nearly every latency dashboard computes p95; group modifiers (on(...) group_left(...)) and the set operators or, and, and unless are unsupported; and predict_linear, label_join, and label_replace are not yet available. Time buckets also align to fixed calendar boundaries rather than the query start time, so short ranges or large steps can differ slightly from Prometheus. See the PROMQL command reference for the current list.

CPU: per-pod CPU rate with PromQL

The per-second CPU rate across containers, grouped by pod. This is the first thing you look at when something is hot:

PROMQL sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

Broken out by namespace instead, to find which team is burning the cluster:

PROMQL sum by (namespace) (rate(container_cpu_usage_seconds_total[5m]))

Note what came back: a pod column, a step column, and the value column named after the expression itself. That is a normal ES|QL table, which is the whole point and the thing Step 6 builds on.

Memory: working set and RSS with PromQL

Working set is the number that matters for OOM risk. It is what the kernel counts against the limit, and it is not the same as total allocated memory:

PROMQL sum by (pod) (container_memory_working_set_bytes)

Resident set, for comparison, when you are trying to tell a real leak from page cache:

PROMQL sum by (pod) (container_memory_rss)

Network: receive rate by pod with PromQL

PROMQL sum by (pod) (rate(container_network_receive_bytes_total[5m]))

Cluster objects: replica counts with PromQL

Deployments that are not running the replica count they declare:

PROMQL kube_deployment_spec_replicas

One nicety worth calling out: in Prometheus every query needs an explicit start, end, and step. In Kibana you drop all three. The date picker supplies the range and Kibana derives the step, which is why every query above is a single line.

Build a Kibana dashboard from PromQL queries

Every query above is a dashboard panel. In Discover, click Save, or go to Dashboards → Create → Add panel → ES|QL and paste the query in. The date picker drives start, end, and step, so a panel written once works at every zoom level.

Four panels, four one-line PromQL queries, no translation step. If you are coming from Grafana, this is the same dashboard you already have, rebuilt in about five minutes. If you would rather not rebuild it at all, keep Grafana and point its existing Prometheus datasource at Elasticsearch. That is covered at the end of the post.

Step 6: query Prometheus metrics with ES|QL, beyond what PromQL can express

The PROMQL command in ES|QL compiles to TS. These two queries are equivalent:

PROMQL sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

TS metrics-node.prometheus-eks
| WHERE TRANGE(1h)
| STATS SUM(RATE(metrics.container_cpu_usage_seconds_total, 5m)) BY labels.pod, TBUCKET(1m)

The second form is where metrics stop being a separate world from logs and traces — and where the real joins happen.

Top-N and filtering

A TS query returns a normal ES|QL table, so SORT, LIMIT, and WHERE all work downstream. Top-N needs no special function, and no topk. The ten pods using the most memory are a SORT and a LIMIT:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL
| STATS mem = SUM(metrics.container_memory_working_set_bytes) BY `labels.pod`
| SORT mem DESC
| LIMIT 10

Filtering is the same move, a WHERE on the aggregated column. Only the pods over 50 MB:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL
| STATS mem = SUM(metrics.container_memory_working_set_bytes) BY `labels.pod`
| WHERE mem > 50000000
| SORT mem DESC

You actually can pipe the results of a PROMQL query and post-process with regular ES|QL.

Memory usage against the request

A useful question during a memory scare: which pods are using more than they requested, and by how much. That combines two metrics, and ES|QL expresses it in a single pass with filtered aggregations, one MAX per metric:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL AND `labels.namespace` IS NOT NULL
| STATS
    used_memory      = MAX(metrics.container_memory_working_set_bytes),
    requested_memory = MAX(metrics.kube_pod_container_resource_requests)
                       WHERE `labels.resource` == "memory"
  BY `labels.pod`, `labels.namespace`, time_bucket = TBUCKET(5 minute)
| EVAL pct_of_request = 100 * used_memory / requested_memory
| WHERE pct_of_request > 80
| SORT pct_of_request DESC
| LIMIT 100

The WHERE attached to requested_memory is a filtered aggregation: kube_pod_container_resource_requests carries both CPU and memory under a labels.resource dimension, and the filter keeps only the memory rows, so the ratio is memory over memory. Each metric lives in its own documents; the shared BY key lands both aggregates on one row.

Step 7: query Prometheus metrics and logs together with ES|QL

Over the last few sections we used ES|QL and PromQL to explore metrics. ES|QL reads logs too. Here is a quick query against the OpenTelemetry demo running on this cluster:

FROM logs-*
| WHERE TRANGE(30m) AND kubernetes.pod.name == "checkout-9656cbd88-fsr9v"
| STATS events = COUNT(*) BY log.level, TBUCKET(1m)
| SORT events DESC

Same editor, same date picker you used for metrics. Only the source command changed, from TS to FROM, and now you are reading log volume by level per minute. One query language across metrics and logs, no tab switch and no second tool.

What's running in Elastic Observability now

Prometheus is writing to Elastic with one config block. Every metric renders in Discover with no dashboard built. Most of your existing PromQL runs unchanged, and ES|QL takes you further: top-N, filtering, and cross-metric ratios you can pipe into the rest of the language. Metrics and logs answer to the same query in the same window.

You did not drop a single label to get here, and you are not being billed for cardinality.

Next steps: Grafana, dashboard migration, and OpenTelemetry

1. Keep Grafana if you want it. Elasticsearch exposes a Prometheus-compatible read API at <endpoint>/_prometheus. Point Grafana's existing Prometheus datasource at it, set httpMethod: GET on the datasource, and your PromQL dashboards keep working. Keep Grafana, replace Mimir.
2. Migrate your Grafana dashboards. When you are ready to move off Grafana rather than point it at Elasticsearch, the Observability Migration Platform translates Grafana dashboards, panels, and alert rules into Kibana-native equivalents. It is a source-agnostic CLI (obs-migrate) that converts what it can and flags what needs a human, with a migration report showing what translated cleanly and where semantic gaps remain, so nothing is silently dropped. The walkthrough covers a Grafana and Datadog migration end to end.
3. Want to add OpenTelemetry? Read Part 2 if you are also running OpenTelemetry, or if you would rather collect with an OTel Collector than with Prometheus. Both land in the same store and the same queries read across both.

Start a free trial: cloud.elastic.co/registration Docs: elastic.co/docs/solutions/observability

Frequently asked questions

How do I send Prometheus metrics to Elasticsearch? Add a remote_write block to your Prometheus configuration pointing to https://<YOUR_ES_ENDPOINT>/_prometheus/metrics/{dataset}/{namespace}/api/v1/write with an Authorization: ApiKey <YOUR_KEY> header. Elasticsearch implements the Prometheus Remote Write v1 protocol natively — no adapter or sidecar required.

Can I run existing PromQL queries in Elasticsearch? Most of them, yes. ES|QL includes a PROMQL source command that accepts standard PromQL expressions, benchmarked at over 80% coverage against popular Grafana OSS dashboards. It is generally available on Elastic Cloud Serverless and a tech preview on Elastic Stack 9.4. histogram_quantile, predict_linear, label_join, and label_replace are not yet implemented, and group modifiers and the set operators or, and, and unless are unsupported.

Does Elasticsearch charge based on Prometheus metric cardinality? No. Elasticsearch stores Prometheus metrics at 3.75 bytes per datapoint with no cardinality-based billing and no custom-metric penalty, regardless of how many unique label combinations your metrics produce.

How do I route Prometheus metrics to different data streams in Elasticsearch? The two path segments after /metrics/ in the Remote Write URL set the dataset and namespace. /metrics/node/eks/api/v1/write writes to metrics-node.prometheus-eks. You can also route per time series using data_stream_dataset and data_stream_namespace labels, which take precedence over the URL path.

What Prometheus metric types does Elasticsearch support? Elasticsearch supports Remote Write v1, including counters, gauges, classic histograms, and summaries. Counter vs. gauge classification is inferred from the metric name: names ending in _total, _sum, _count, or _bucket are counters; everything else is a gauge. Native histograms and exemplars require Remote Write v2, which is on the roadmap.

Can I query Prometheus metrics and logs together in Elasticsearch? Yes. ES|QL reads both time series metrics and logs in the same query editor with the same date picker. Switch from TS metrics-node.prometheus-eks to FROM logs-* — same syntax, same window, no second tool.

Related reading

• Elasticsearch: best-in-class for logs, now best-in-class for metrics
• Ship Prometheus metrics to Elasticsearch with Remote Write
• Query Prometheus metrics in Elasticsearch with native PromQL support
• Don't leave metrics on the table: query them with the ES|QL TS command
• Elasticsearch as a backend for Grafana
• How we rebuilt Elasticsearch as a columnar metrics engine

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

A customer described it to me this way: A single transaction passes through three products on its way through the mainframe, and each one names the same field differently (system name, program name, user). Getting the data into Elastic isn't the hard part; getting it to correlate across products, so that Elastic's dashboards and ML jobs recognize it as the same data, is where most projects fall short.

Done right, Elastic becomes the speed layer that mainframe environments have never had: a near–real-time view across operational, transactional, and security data, while the authoritative systems of record stay exactly where they are.

This is the onboarding process I use with mainframe customers, built from architectures running in production at large financial institutions. It covers the ingest patterns that actually work, how Elastic Common Schema (ECS) alignment makes the data usable, and whether the data quality holds up or fails quietly.

Validate your mainframe source data before you write a pipeline

The most expensive failures I've seen in mainframe ingest projects are the ones that don't fail loudly. Pipelines run, data lands, dashboards render, and weeks later, someone notices that half the events from one logical partition (LPAR) never parsed or a quiet typing change has been silently corrupting a field.

Two mistakes come up frequently:

1. Format inconsistency across LPARs and time windows: Log formats vary across LPARs, between batch and online windows, and across shift changes. A format that parses cleanly in a dev LPAR may not match what production emits during peak batch. This is the single most common cause of partial parse failures I run into.

2. Sample configurations treated as production configurations: A common cause of "it broke overnight" incidents: The upstream collector configuration was based on a sample structure shipped by the vendor and then never replaced with a deliberate production configuration. When the vendor pushed an update, naming and typing changed (fields renamed, types shifted) and the downstream pipeline started rejecting records mid-flight. Treat sample configurations as exactly that, and replace them with a deliberate production configuration that doesn't move under you.

Before any pipeline development begins, walk through 24–48 hours of raw samples from each source with the mainframe team. This review should be treated as a recurring requirement rather than a one-off event, because the conditions that produce format drift (vendor updates, configuration changes, new message types) keep happening after the project goes live.

Worth knowing first: For mainframe environments, the Elastic integrations catalog is short. The IBM MQ integration is the most complete option (Queue Manager error logs and performance metrics, ECS-aligned, with out-of-the-box dashboards), though the metrics data stream requires the containerized MQ distribution rather than native z/OS MQ. If your architecture includes Customer Information Control System (CICS) workloads or you need end-to-end distributed tracing, assess IBM Z Observability Connect before building custom pipelines: It's the native OpenTelemetry (OTel) path and covers more ground than the architectures below. For everything else, read on.

ECS alignment: from data in Elastic to data Elastic can use

Before choosing an ingest strategy, it's worth understanding why ECS alignment comes first in practice, even if the pipeline gets built later. It's the decision that determines whether everything else pays off.

A mainframe team's core mission: Trace a single transaction from a REST call into z/OS Connect, through to an Information Management System (IMS) application, and back. That flow touches three products, each emitting telemetry with its own field names for the same concepts (system name, program name, user, transaction ID). Without normalization, correlating that transaction means writing queries that explicitly union three different field names per concept. That’s expensive to write and fragile when any product changes its schema.

ECS solves this. It defines a consistent target schema (host.name, process.name, user.name, event.code) that every source maps into. Once z/OS Connect, IMS Connect, and IMS data all land in the same ECS fields for the same logical concepts, that cross-product transaction trace becomes a single query.

There's a second reason this matters. Elastic's OOTB dashboards, alerting rules, anomaly detection, and ML jobs are all built against ECS field paths. A job_name field that Logstash extracted from a JES log is invisible to them. A process.name field carrying the same value is immediately recognized and processed. ECS alignment is what makes the platform's built-in capabilities recognize your data.

Skipping this step is the most common reason that ingest projects fall short, despite the data being technically present.

Map what fits to core ECS

The mapping below is a starting point drawn from what I've seen work across customer environments. Field names in your source data will vary, but the ECS targets are stable:

| z/OS concept | ECS field | Notes | | :---- | :---- | :---- | | Job name | process.name | | | Return code | process.exit_code | Ensure integer type; hex strings are a common mapping mistake | | Program name | process.executable | | | Elapsed time | event.duration | Nanoseconds in ECS; z/OS typically reports in hundredths of a second or milliseconds, so convert at the pipeline stage; unit mismatches silently break ML anomaly detection on latency | | Message ID | event.code | | | Timestamp | @timestamp | Normalize from z/OS format to ISO 8601 in the pipeline | | LPAR name | host.name | | | System ID (SMFID) | host.hostname | | | User ID | user.name | |

Reference: ECS process fields and ECS event fields.

Extend strategically with custom ECS fields

Mainframe-specific concepts have no ECS equivalent: job class, ASID, SMF record type and subtype, sysplex name, WTO routing codes, CICS transaction ID. Flattening these into labels.* as untyped strings destroys type information and makes them effectively unusable for queries and aggregations.

Define a zos.* custom namespace using ECS's documented extension mechanism. It keeps your core telemetry ECS-compliant while retaining the operational context your mainframe team needs for incident response.

Use ecs@mappings to stay current

Include ecs@mappings as a component template in your index template (available from Elasticsearch 8.9 for custom index templates and from 8.13 for Elastic Agent integration templates). It provides Elastic-maintained ECS field definitions automatically and keeps them current with each Elasticsearch release. For custom pipelines, this is what keeps your ECS alignment from drifting over time without manual upkeep.

One important caveat from the field: ecs@mappings provides the field definitions but doesn't enforce types at ingest. A return code arriving as a string is accepted and mapped as a string. Monitoring these discrepancies is critical, and they can be identified using the Data Quality dashboard. And because Elastic donated ECS to OpenTelemetry, the zos.* mappings you define here remain valid as OTel semantic conventions and ECS converge. The alignment work is the same whether data arrives via Logstash or OpenTelemetry Protocol (OTLP).

Choose the right architecture for the source

Most environments I work with run more than one of these ingest architectures, and different data sources have different latency, throughput, and licensing characteristics. A single architecture rarely covers everything. The table below maps common z/OS data sources to the architectures that work well for them.

Kafka is commonly added when there's a network resilience requirement between the mainframe and the Elastic cluster. If Kafka isn't already in your estate, the operational overhead of running Kafka should be weighed against the resilience benefits. From IBM MQ 9.4.3, Kafka Connect can run natively in z/OS UNIX System Services for MQ connector use cases, reducing the need for an off-platform Kafka Connect cluster.

When Kafka is used, Logstash is the recommended downstream consumer for the ingest paths in this guide. The Confluent Elasticsearch sink connector is an alternative; the self-managed v1 connector supports Elasticsearch 7.x and 8.x but is deprecated with end of life (EOL) in April 2027; and the v2 connector is Confluent Cloud only, making it unsuitable for on-premises and air-gapped environments.

Reference architecture: Kafka as middleware.

| Data source | Collector | Notes | | :---- | :---- | :---- | | SMF type 30 job accounting | IBM Z Common Data Provider (CDP) | Binary SMF records need preprocessing before ingestion | | z/OS SYSLOG | IBM Z CDP | | | JES job logs | IBM Z CDP | Batch export is an alternative for historical / proof of concept (PoC) work | | Resource Access Control Facility (RACF) audit events | IBM Z CDP | ECS-aligned RACF data works with Elastic SIEM out of the box | | RMF performance data | IBM Z CDP | Consider time series data stream (TSDS) for the index template | | IMS statistical records | IBM Z CDP | | | OMEGAMON agent metrics (CICS, IMS, Db2, z/OS, network, storage) | IBM OMEGAMON Data Provider (ODP) | Outputs JSON natively; no binary preprocessing needed | | IMS transaction data | IMS Connect Extension (Rocket Software) | JSON output bypasses SMF binary parsing; requires Rocket Software licensing | | CICS transaction traces | IBM Z Observability Connect | Native OTel; covered in detail in the End-to-End Observability from Cloud Native to Mainframe deep -dive | | Linux on IBM Z (zLinux) | Standard Elastic Agent | Full integration catalog available; different problem from z/OS onboarding | | Historical analysis / PoC | Batch export (CSV / FTP) | Not suitable as a long-term operational solution |

Flow diagram of the ingest paths from z/OS to Elastic.

IBM Z CDP: The workhorse for z/OS operational data

IBM Z CDP is the most widely deployed first-mile collector for z/OS operational data. It reads from SMF datasets in near-real time and forwards off-platform, handling the genuinely difficult part of getting data off z/OS without burdening performance-critical paths. In the environments I work with, it's the standard path for SMF type 30 job accounting, IMS statistical records, and z/OS SYSLOG.

CDP forwards to Logstash, which handles parsing, field extraction, and routing into Elasticsearch. Kafka is an optional middleware message queue:

• CDP → (Kafka →) Logstash → Elasticsearch

The trade-offs: CDP is a separately licensed IBM product, binary SMF records need preprocessing before Logstash can parse them, and the architecture isn't suited to sub-minute latency requirements.

Worth noting alongside CDP:

• IBM ODP plays the same collector role for performance and availability metrics from whichever OMEGAMON monitors are in your stack: CICS, IMS, Db2, z/OS, network, and storage. Unlike CDP's binary SMF output, ODP converts to JSON natively, so there's no preprocessing step. ODP consists of two components: OMEGAMON Data Broker (a Zowe cross-memory server plugin running on z/OS that collects attributes from OMEGAMON monitoring agents and forwards them to Data Connect); and OMEGAMON Data Connect (a Java application running on or off z/OS that receives data from Data Broker and forwards it to destinations including Elasticsearch; the destination settings are configured here). If OMEGAMON is already in your monitoring stack, ODP is the natural path for getting that telemetry into Elastic.
• IBM Z Operational Log and Data Analytics (IZLDA) packages CDP's data streaming capabilities alongside analytics and dashboarding into a single licensed product — I haven't encountered it in production yet, but it's the direction IBM is heading. If your organization is evaluating or has recently licensed IZLDA, the CDP ingest path described above remains the same — IZLDA uses CDP as its underlying collection engine, with Elastic Stack as one of its supported destinations. Gruppo BCC ICCREA's deployment is a published example of IZLDA feeding an Elasticsearch-based monitoring stack.

Reference architecture: Logstash to Elasticsearch.

IMS Connect Extension: For IMS workloads that can bypass CDP

Rocket Software's IMS Connect Extension journals IMS transaction activity directly as JSON, bypassing the SMF layer entirely. Events publish to Kafka, and Logstash consumes and indexes. Some organizations standardize all log streams through Kafka (rsyslog → Kafka → Logstash) as an optional resilience pattern.

• IMS Connect Extension → (Kafka →) Logstash → Elasticsearch

This works well for IMS transaction performance data and application-level event streams. JSON output removes the binary parsing problem. Kafka gives you decoupling, replay, and a buffer for downstream maintenance.

The trade-offs: IMS Connect Extension licensing, Kafka infrastructure to operate, and IMS-specific coverage that doesn't help with z/OS SYSLOG or other SMF types.

One thing I always validate before committing to this pattern is Kafka topic naming. Banks and regulated environments typically have strict topic naming policies, and IMS Connect Extension's default behavior of creating topics itself can clash with those policies. It’s cheaper to discover this before architecture commitment than after.

Batch export: For historical analysis and PoC

Export from IMS Problem Investigator or similar tooling to CSV, transfer off-platform, and ingest via Logstash or Elastic Agent file input. This approach has no real-time capability, and it doesn’t require any new z/OS software.

• Batch export → CSV/FTP → Logstash/Elastic Agent → Elasticsearch

This works well for historical analysis, initial PoC work, and demonstrating value before committing to a real-time pipeline. I also use this to get ECS mapping right before the production architecture is in place. It isn’t suitable as a long-term operational observability solution.

Linux on IBM Z: A separate and easier path

This path is often overlooked. Linux on IBM Z workloads can run standard Elastic Agent (Elastic Agent doesn’t run on native z/OS), no z/OS-specific tooling, no custom pipeline and the full Elastic integration catalog is available.

If you have Linux on IBM Z workloads in your estate, treat them as a separate (and considerably easier) onboarding path.

For all options, please see the reference architectures with Elastic Agent.

Throughput and air-gapped: The two questions every mainframe team asks

Throughput impact: Anything that touches z/OS performance-critical paths is a nonstarter for mainframe teams running thousands of transactions per second. All four architectures above use off-platform collection deliberately: CDP, ODP, Kafka/Logstash, batch export, or standard Linux agent. This is the right design for the environment, not a workaround.

Air-gapped environments: Most mainframe estates I work with are network-restricted to some degree. Elastic's air-gapped reference architecture is a documented, supported deployment path.

Build the pipeline, and don't start from scratch

For the CDP and IMS Connect Extension architectures, log data lands in Elastic reflecting the limited structure of its source. Mainframe log formats are installation-specific and partially structured at best; no off-the-shelf parser covers them, and writing a pipeline from scratch has historically been the largest time sink in any onboarding project. AI has changed that. For mainframe estates that can't call out to a hosted model (which is most of them), both tools below work with self-managed local large language models (LLMs), so the capability is available in air-gapped and network-restricted environments. See the local LLMs overview for supported options.

Streams: For data already landing in Elastic (available from 9.2).
Open the Processing tab for a stream in Kibana, and click Suggest pipeline. Within seconds, you're looking at a complete, validated pipeline (Grok or Dissect pattern, date normalization, type conversions, field cleanup) with a live preview of how your actual documents parse through it. Nothing writes to the stream until you confirm. Under the hood, generation runs in two stages: First, deterministic fingerprinting groups your log formats and picks the best parsing approach; second, a reasoning agent iterates to add normalization and cleanup, validating against hard thresholds before handing control to you. The result is a working pipeline you refine, not a starting point you rewrite. The technical detail is in How Streams Generates a Log Pipeline in Seconds.

Automatic Import: For building a new custom integration from the ground up (available from 8.18/9.0). Automatic Import takes a different path. You upload sample data, and it generates a complete, deployable Elastic Agent integration package (ingest pipeline, ECS field mappings, event categorization, and related.* field population), which you review and approve before it installs. Where the Streams Suggest Pipeline structures data already arriving in a stream, Automatic Import builds the entire collection path from scratch. Supported input formats include JSON, NDJSON, CSV, and syslog, which covers z/OS SYSLOG directly. Supported collection methods include Kafka, File Stream, TCP, and HTTP Endpoint, making it a natural fit for shops already routing data through Kafka or receiving ODP output over TCP. For mainframe shops adopting Elastic Agent, this removes what was previously weeks of custom integration work.

A less obvious benefit that applies to both tools is continuity. The engineer who wrote your custom GROK pattern eventually moves to another team. A tool that can regenerate a pipeline or integration from sample data is operational resilience.

In terms of scope, Streams works on text. Binary SMF records need to be converted to text or JSON upstream (via CDP or IBM-supplied utilities) before either tool can do anything with them. That conversion happens before Elastic is involved.

Configure the dead-letter queue from day one

Mainframe teams know the dead-letter queue pattern from MQ: When a message can't be delivered or processed, it goes to a holding queue rather than being silently dropped. Elasticsearch has the same concept for ingest pipelines, called the failure store. Configure it from day one, not after your first production incident.

Format drift is a recurring failure mode: vendor updates, sample-config-as-production, new message types appearing in batch windows. The failure store is how you find out about it before your dashboards lie to you. When a log line arrives in an unexpected format and the pipeline can't parse it, the failure store captures the original document with metadata about why it failed. You can query it, alert on its growth rate, and use the captured documents to fix the pipeline.

Without it, parse failures either fall to default handling (records indexed with raw message fields, expected query fields simply absent) or get dropped entirely. Either way, you don't know it's happening.

Configure retention based on how long it takes your team to triage drift, typically days to a couple of weeks. Pair it with an alert on document count or growth rate so the queue itself is the early warning, not something someone has to remember to check.

Verify mainframe data quality before you build on it

Don't build dashboards or alerting rules on data you haven't verified. The Data Quality dashboard tells you whether your ECS alignment is real or aspirational.

For mainframe data, silent type mismatches are common: return codes in hex mapped as keywords, elapsed times stored as strings, timestamps that never coerced to @timestamp. None of these fail at ingest. All of them silently break queries and alerting conditions.

Run the checker against real production data, not synthetic samples. z/OS log variation across batch windows and message types means edge cases only surface under real conditions. Expect to iterate: Find the mismatch, fix the pipeline, and run again. Two or three passes is normal for a new mainframe data stream.

Source quality validation, the failure store, and the Data Quality dashboard are three points on the same loop. Together they give you confidence that the dashboards reflect what's actually happening on the mainframe, not what you hoped your pipeline was producing.

Getting started with mainframe data onboarding

The mainframe is a first-class observability target, and the path there is more concrete than it was a few years ago. Managed integrations cover IBM MQ. CDP and Kafka-based architectures have well-understood deployment patterns. Streams and Automatic Import remove the blank-page problem for custom pipelines, including in restricted environments through local LLMs. IBM Z Observability Connect is there when the OTel path is in reach.

Recommended order of operations:

1. Validate the source data.
2. Use OOTB integrations where they exist.
3. Align to ECS early.
4. Choose architectures source by source.
5. Generate pipelines rather than write them from scratch.
6. Configure the failure store from day one.
7. Verify before building anything on top.

If your organization is working through this and you'd like to compare notes, or if you're hitting a specific blocker, reach out to your Elastic account team. For the OTel-native path, the End-to-End Observability from Cloud Native to Mainframe deep dive is the next read. To try the building blocks in your own environment, start a free Elastic Cloud trial.

The first time most teams learn that their OpenAI project is close to a rate limit is when production traffic starts getting throttled with HTTP 429 responses. OpenAI enforces rate limits at the project level, not at the organization level, so a single noisy workload in one project can saturate that project's RPM or TPM ceiling while the rest of the organization still has plenty of room. Without OpenAI rate limit monitoring that compares configured limits against actual consumption, headroom is invisible until it runs out.

OpenAI API monitoring in Elastic: what's new

We're pleased to announce updates to the Elastic OpenAI integration. On top of the existing token usage and audit log coverage, the integration now polls OpenAI's List project rate limits Admin API per project and rolls the results up into both per-project and org-wide views. A new openai.rate_limits dataset feeds two new dashboard panels and a prebuilt threshold alert rule, so teams can see how close each project and model is to being throttled before users experience production impact.

What the integration polls: Usage, Audit Logs and Rate Limits APIs

The Elastic OpenAI integration is built for teams running applications on the OpenAI API platform. The people accountable for it are the developers shipping those services, the platform and SRE teams keeping them running, and the finance and FinOps owners answering "how much is our software consuming, and are we within our capacity envelopes?"

The integration collects from three OpenAI Admin API surfaces:

• Usage API for usage counts across tokens, characters, seconds, sessions, bytes, and images, with project, model, user, and API key attribution where that Usage API surface provides it.
• Audit Logs API for organization audit events such as API key creation, project changes, and user activity.
• Rate Limits API for configured RPM, TPM, and IPM ceilings per project and per model, plus other limit dimensions where available; the new headroom views compare the per-minute request, token, and image limits against actual consumption.

Because everything is pulled from the Admin API at the organization level, platform teams get a unified view across every project, model and API key, alongside the rest of the telemetry they already monitor in Elastic, without touching application code or installing SDKs in every service.

What teams need to monitor when running on the OpenAI API

Four operational needs come up over and over for teams running production workloads on the OpenAI API.

Token usage attribution

A single OpenAI organization usually serves many internal teams and products, each with its own project, its own mix of models (GPT-5.5 Pro for the hardest reasoning tasks, GPT-5.4 for everyday traffic, GPT-5.4 nano for high-volume low-cost requests, and specialized models for images, audio and embeddings) and its own user and API key footprint. When usage patterns shift, the platform team needs to know which project, model and key is driving the change so they can attribute consumption back to the right team and decide which workloads should move to a cheaper model.

Rate limit headroom

OpenAI enforces per-model rate limits on requests per minute (RPM) and tokens per minute (TPM) at the project level, not at the organization level. The first time a team learns they're close to the ceiling is usually when production traffic starts being throttled. Surfacing configured limits alongside actual consumption, per project and per model, lets platform teams see headroom in advance, plan capacity, and request limit increases before users feel the impact.

Audit visibility

Security and compliance teams need to know who created API keys, who changed project settings, who invited or removed users, and when. The integration ingests OpenAI's organization audit log so those events land in the same Elastic deployment as the usage data, ready for correlation, alerting and long-term retention. Audit log ingestion has two prerequisites: audit logging must be enabled in your OpenAI organization, and the Admin API key used by the integration must belong to an Organization Owner, because OpenAI restricts audit-log access to that role. Without both, the openai.audit dataset stays empty.

Granularity for every audience

The same data needs to serve different cadences. SREs want one-minute resolution to catch spikes and trigger throttling alerts. Platform engineers want hourly views for capacity planning. Finance and FinOps owners want daily totals that roll up cleanly for reporting. A single integration that exposes all three granularities removes the need to maintain separate pipelines for each audience.

How does Elastic poll the OpenAI Admin API?

The integration runs on Elastic Agent and uses the CEL input to poll OpenAI's Admin API on a schedule. Authentication uses a single Admin API key, stored as an encrypted Fleet secret and redacted from agent logs. From a single configuration, the integration ingests datasets from three Admin API sources:

Usage API datasets (per project, model, user and API key, with each dataset tracking the unit OpenAI exposes for that workload):

• openai.completions for chat and completion token counts (input, output, cached, audio input/output).
• openai.embeddings for embedding token counts.
• openai.moderations for moderation token counts.
• openai.images for image counts and size dimensions.
• openai.audio_speeches for text-to-speech character counts.
• openai.audio_transcriptions for speech-to-text duration in seconds.
• openai.code_interpreter_sessions for code interpreter session counts.
• openai.vector_stores for vector store byte counts.

Audit Logs API dataset:

• openai.audit for organization audit events such as API key creation, project changes and user activity.

Rate Limits API dataset:

• openai.rate_limits (new) for snapshots of configured rate limits per project and per model, including RPM, TPM, IPM, and other limit fields where OpenAI returns them, paged across all active projects on each poll.

Ingest pipelines handle parsing and field mapping so the data lands queryable, dashboard-ready, and aligned with the rest of Elastic Observability. Because the data is pulled from the Admin API at the organization level, you get this visibility without any application-side instrumentation or SDK changes.

What you need to set up OpenAI monitoring in Elastic

To get started with the Elastic OpenAI integration, you will need:

• An Elastic deployment:
• Elastic Cloud Hosted (ECH) running a recent supported version, or
• Elastic Cloud Serverless, no version requirement, works out of the box.
• An OpenAI organization with Admin API access.
• An Admin API key provisioned by an Organization Owner from the OpenAI platform settings under Settings → Admin keys. Owner-level keys are required if you want the openai.audit dataset to populate.
• Audit logging enabled in your OpenAI organization, if you want audit data.
• Elastic Agent installed on a host with outbound HTTPS access to api.openai.com, or the agentless deployment option.

How to set up the OpenAI integration

1. Generate an Admin API key in the OpenAI platform settings.
2. In Kibana, go to Management → Integrations, search for OpenAI and click Add.
3. Choose your deployment mode: agentless for a zero-install experience, or Elastic Agent on your own host.
4. Tune the defaults if you need to. Each dataset has sensible defaults:

• Usage datasets poll every 5 minutes with 1-minute buckets. Each dataset exposes a finalization_grace setting that controls when a per-minute usage bucket is considered final. The default 0s favors freshness: a bucket is ingested as soon as its minute closes. The observed behavior (which OpenAI does not document, but the integration team measured against the live API) is that bucket counts can keep rising for some time after that point, so per-minute totals at 0s can undercount during heavy bursts. Setting finalization_grace to 15m, the recommended value for accurate counts, holds a bucket back until the grace window has elapsed and brings counts much closer to the Usage API, though a small residual undercount can remain during very high-volume bursts because OpenAI's per-minute counts can be revised upward beyond any fixed grace window. The cost is delaying dashboards and the rate limit headroom alert by the grace period.
• Rate limits polls every 5 minutes. Each poll captures the full set of configured RPM, TPM and IPM limits per project and per model.
• Audit logs polls on a separate cadence and ingests all org-level audit events. Remember the prerequisites: audit logging enabled in your OpenAI organization and an Organization-Owner Admin API key.

1. Open the integration assets in Kibana. Within minutes, usage, audit, and rate-limit data starts flowing, and the prebuilt dashboards and alert rule are ready to use.

For the full configuration reference, see the OpenAI integration documentation.

What do the OpenAI rate limit dashboards show?

The integration ships with a pre-built Kibana dashboard that gives you an immediate, queryable view of your organization's OpenAI API consumption. The overview pulls headline numbers (total tokens, total invocations, top models, top projects, top users and top API keys) into one place for a quick read on the state of your OpenAI usage. The screenshot below shows the OpenAI usage overview dashboard:

From the overview, you can drill into the views that answer the operational needs introduced earlier.

Token usage by model, project and user

The token metrics panels break down token consumption (input, output, cached input, audio input/output) for the token-based datasets (openai.completions, openai.embeddings, openai.moderations) by model and over time. This is the view that tells you where your token budget is actually going, which workloads are getting the most out of prompt caching, and which projects, users or API keys are driving the bulk of your token consumption. Filter by project or model to scope the view to a single team or product. Image, audio and vector-store consumption (measured in images, characters, seconds, sessions and bytes rather than tokens) is reported in dedicated sections of the same dashboard. The token metrics panels look like this:

Rate limit headroom: per project and model (new)

The new rate limit headroom panel joins the configured limits from openai.rate_limits against actual consumption from the usage datasets, per project_id and model. For each row it reports peak one-minute used, the configured limit, and utilization percentage for requests (RPM), tokens (TPM), and images (IPM). Rows are sorted by highest TPM utilization first, with RPM and IPM utilization as tie-breakers, so the highest token-pressure rows appear at the top of the list. Utilization is computed against the peak one-minute bucket in the lookback window, never a 5- or 15-minute sum against a one-minute ceiling, so the panel reflects peak-minute pressure against the one-minute ceiling instead of averaging it away. The per-project rate limit headroom panel is shown below:

Rate limit headroom: org-wide rollup by model (new)

Because OpenAI enforces limits per project, a single per-project view doesn't answer "how much total capacity do we have for gpt-image-2 across the organization?" The new org-wide rollup panel reports the same RPM, TPM and IPM metrics summed across all active projects for each model. Both the limit and the usage figures are indicative upper bounds rather than exact org-wide numbers (the limit is a sum of per-project ceilings; the usage is a sum of each project's peak minute, which may fall in different minutes across projects), but together they give platform teams a single number to plan against when they're sizing a new workload or deciding which project should absorb a new use case. The org-wide rollup panel is shown below:

Behind the scenes, version 2.3.0 also normalizes request and token counts into shared openai.base.usage_tokens and openai.base.usage_images fields across the usage datasets, so the headroom panels render correctly even when only a subset of usage datasets is enabled.

OpenAI audit log activity in Elastic

The audit panels surface organization audit events (API key creations, project changes, user invitations and login activity) so security and compliance teams can review who did what, when, alongside the usage data. The audit dashboard is shown below:

Out-of-the-box alert for rate limit headroom (new)

The integration ships with a pre-built threshold alert rule template, [OpenAI] Rate limit headroom low, ready to install in one click from the integration's Assets tab.

The default behavior is tuned to be useful out of the box:

• Runs every 5 minutes.
• Looks back over the last 15 minutes.
• Fires after 3 consecutive matches.
• Triggers when peak one-minute TPM utilization reaches or exceeds 80% of the configured project/model limit.
• Groups alerts by project_id::model, so an incident in one project on one model doesn't get lost in an org-wide aggregate.

The 80% threshold and other parameters are editable in Kibana after you install the rule, so each team can tune the alert to its own risk tolerance.

Custom OpenAI alerts and SLOs in Elastic Observability

As with every other Elastic integration, all the OpenAI metrics and audit data is fully available to leverage in every capability in Elastic Observability, including SLOs, alerting, custom dashboards and in-depth logs exploration.

For example, to keep token consumption under control across a single project, create a custom threshold rule that sums tokens from the relevant usage dataset and fires when the daily or hourly total crosses your budget. To track model mix, define an SLO in Elastic Observability that treats OpenAI requests on your approved lower-cost model families as the "good events" (the ones that count as meeting the target) and all OpenAI requests as the "total events", grouped by openai.base.project_id and openai.base.user_id. The ratio becomes your SLI; a 7-day rolling 80% target quickly surfaces projects and users overusing more expensive models.

Choosing OpenAI usage data granularity

OpenAI usage data collected by the integration powers different cadences, with a fidelity-versus-freshness tradeoff to be aware of. One-minute usage buckets feed the rate limit headroom alert and near-real-time throttling notifications when a project approaches its ceiling: with finalization_grace set to 0s (the default), per-minute counts arrive within minutes but can undercount during heavy bursts; raising finalization_grace to 15m brings counts much closer to reconciled at the cost of delaying the dashboards and alert by the grace period; a small residual undercount can still remain for the busiest buckets. Hourly views support operational monitoring and capacity planning across projects and models. Daily aggregates roll up cleanly for FinOps reporting and reconciliation. An out-of-the-box alert ships for rate limit headroom ([OpenAI] Rate limit headroom low), and the same data can be reused for custom usage and budget thresholds without building anything from scratch.

Get started with OpenAI monitoring in Elastic

The Elastic OpenAI integration is available today in Elastic Cloud, including Elastic Cloud Hosted and Elastic Cloud Serverless. To get started, sign up for a free Elastic Cloud trial, provision an Admin API key in the OpenAI platform settings, and add the OpenAI integration from Kibana under Management → Integrations.

Within minutes you'll have token usage, audit activity, and rate limit headroom data flowing into Elasticsearch, with the prebuilt dashboards and the new throttling alert ready to use.

• Elasticsearch is a Prometheus-compatible metrics backend — Prometheus Remote Write and PromQL now works natively in Kibana, no translation layer required.

• Metrics land in Elasticsearch's columnar TSDS architecture storing data up to 2.5× more efficient than Prometheus and 2× more efficient than ClickHouse.

• ES|QL time series queries run up to 30× faster than Prometheus on gauge averages and counter rates, including high-cardinality workloads.

• Elastic costs approximately 50% less than Datadog, with no custom metric classification and no cardinality-based billing.

• Grafana can query Elasticsearch directly through the native Prometheus API, keeping your visualization layer while replacing the backend.

• Kubernetes and AWS monitoring ship with pre-built dashboards, alert templates, ML anomaly jobs, and agentic investigation content ready at ingest. Additionally skills and MCP apps are available.

• Unified backend for Metrics, logs, and traces enabling agentic investigations without stitching context across tools.

• Metrics exploration in Discover lets anyone start querying and analyzing metrics immediately, no query language expertise required.

• Custom dashboarding is fast and flexible — dashboards-as-code, AI-assisted dashboard creation, variable controls, and collapsible panels mean less time building and more time investigating.

• Migration tooling to help easily migrate dashboards and alerting rules / monitors from Datadog and Grafana.

Elasticsearch metrics now competes on every dimension that matters to SREs: you can afford to keep every metric at full resolution, query it up to 30x faster than Prometheus, pay 50% less than Datadog, migrate dashboards and alerting rules from Grafana or Datadog easily, and go from alert to root cause without stitching context across disconnected tools. The rest of this post walks through each of these in detail.

Elasticsearch metrics performance: 30× faster than Prometheus and Mimir

Datadog and Prometheus force the same tradeoff: drop high-cardinality data or watch costs spiral. SREs managing Kubernetes, AWS, or any high-cardinality infrastructure know the specific shape of this problem. The Kubernetes labels, ephemeral pod data, and fine-grained OTel dimensions that matter most during an incident are the first to go when budgets tighten.

Elastic rebuilt the time series data store and ES|QL compute engine into a fully columnar metrics engine. Adding a new Kubernetes label, a new AWS instance tag, or a new application dimension doesn't strain the system; it adds far less cost than systems that index every label. OTel, Prometheus, and application-defined metrics all land in the same columnar backend at full resolution, with logs, traces, and metrics in a single store. No data dropped, no retention shortened.

Elasticsearch stores metrics up to 2.5× more efficiently than Prometheus (results may differ due to factors like compaction), and 2× more efficiently than ClickHouse. Query performance via ES|QL runs up to 30× faster than Prometheus on gauge averages and counter rates, including high-cardinality workloads where competitors stall. The architecture post covers how TSDS is organized and why the columnar layout produces these results.

| | | | | | :------------------------: | :----------------: | :--------------: | :----------------: | | Dimension | vs. Prometheus | vs. Mimir | vs. ClickHouse | | Query performance (ES|QL) | Up to 30× faster | Up to 30× faster | Up to 8× faster | | Storage efficiency | Up to 2.5× better | On par | 2× better |

The key architectural difference is that Elasticsearch metrics does not maintain a per-series in-memory state that scales with cardinality, so adding thousands of new Kubernetes pod labels or OTel dimensions doesn't drive up memory pressure.

OTel, Prometheus-native, and application-defined metrics are all stored the same way at full resolution, queried fast, at half the cost of Datadog.

Elastic Observability metrics pricing without the Datadog custom metric penalties

Observability cost is the #1 reason teams switch platforms. For Datadog customers, the pain comes down to one pricing mechanic: custom metrics. Any user-defined value outside of Datadog's built-in integrations is classified as a custom metric and billed at a premium rate. That includes the high-cardinality data that Kubernetes, OpenTelemetry, and cloud-native workloads generate by default. The more granular your instrumentation, the faster the bill compounds. Teams running modern infrastructure hit this ceiling quickly, and the response is predictable: drop data, shorten retention, lose the context that matters most when an incident happens.

Elasticsearch metrics removes that classification. Every metric is priced the same, with no per-metric penalties, no cardinality-based billing, and no forced rollups. You keep every metric at full resolution without a surprise invoice at the end of the month. And because Elastic is 50% the cost of Datadog, the conversation with finance changes: not what data you had to drop to stay on budget, but what you found because you kept everything. It's also why the AI investigation works. Unlike Grafana's fragmented LGTM stack, the context is already unified when the alert fires, not assembled by hand across disconnected tools.

Native Prometheus and PromQL support in Elasticsearch

Most SRE teams aren't running a clean, single-format telemetry pipeline. Prometheus is deeply embedded in applications, services, platforms, and automations. Migrating metrics backends historically meant rewriting queries, rebuilding dashboards, and retraining engineers — enough friction that teams stay on platforms they've outgrown rather than go through it.

Elasticsearch metrics has removed most of that friction. Prometheus metrics arrive via Prometheus Remote Write and land in the same columnar store without semantic changes, preserving full metric fidelity end to end. Point them at Elasticsearch instead of Mimir and the data flows. No translation layer, no changes to existing scrape configs.

PromQL now works natively in Kibana, so engineers who live in PromQL don't have to change how they work. Existing PromQL queries, dashboards, and alert rules migrate into Kibana directly. 

PromQL queries work unchanged on Elasticsearch

If your team already writes PromQL, nothing needs to change. These queries run as-is against Elasticsearch as your backend — copy, paste, and go.

CPU usage rate (container-level) The per-second CPU rate across containers, grouped by pod. Useful for spotting which pods are burning CPU during an incident.

PROMQL sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

Memory working set (container-level) Current memory in active use per container — the number that matters for OOM risk, not total allocated memory.

PROMQL sum by (container) (avg_over_time(container_memory_working_set_bytes[5m]))

HTTP request rate (application-level) Per-second request throughput grouped by instance. A standard first signal when investigating latency or error spikes.

PROMQL sum by (instance) (rate(http_requests_total[5m]))

All three follow standard PromQL syntax. If you use Elasticsearch as your backend, they run without modification. For the full syntax reference and what's covered, see the PromQL support documentation.

The native Prometheus API makes Elasticsearch a fully Prometheus-compatible backend. Any Prometheus-compatible frontend (Grafana included) can query Elasticsearch directly, so teams that want to keep Grafana as their visualization layer while consolidating onto Elasticsearch can do exactly that without modifying existing dashboards or alert rules.

When SREs need to go deeper than PromQL allows, ES|QL works across metrics, logs, and traces in a single interface. The TS command handles the time series specifics: counter rates, gauge averages, window functions, and multilevel aggregations across high-cardinality dimensions. The same query that pulls a CPU counter rate can join against logs from the same host and surface the deployment event that preceded the spike. No tool switching, no new query language. The query language, the dashboards, the alert rules, the visualization layer — all of it carries over. The only thing that changes is that Elasticsearch is the single backend powering everything.

Elastic Observability: out-of-the-box dashboards, alerts, and infrastructure content

Most Observability vendors require you to build everything from scratch. Elastic Observability has reduced this need across three areas:

Metrics exploration in Discover. The new Elasticsearch metrics exploration experience lets SREs explore metrics in the same interface used for logs — no tab switching, no duplicate queries. Connect an OTel pipeline or Prometheus scrape config, open Streams, and every metric in the data stream renders as a time series chart immediately. No dashboard to build, no query to write. This is where teams can validate data, spot patterns, and start building alerts and SLOs from a live view of what's flowing and cross correlate with logs, traces and other indexed data in Elasticsearch.

Dashboards. Kibana dashboards have gained collapsible panels with lazy loading, so panels that aren't immediately visible don't generate queries until they're needed and ES|QL control variables that let SREs manipulate visualizations through dropdowns without writing new queries. Dashboards-as-code is also shipping, enabling version-controlled dashboard definitions that can be templated, shared, and deployed programmatically across environments.

Out-of-the-box infrastructure content.  Elastic is shipping with two new infrastructure OOTB experiences:

• The new Kubernetes integration ships with hierarchical dashboards, alert rule templates, ML anomaly detection jobs, and the context and prompts needed for AI-assisted root cause analysis — all pre-configured and ready the moment data starts flowing. 

• AWS infrastructure monitoring follows the same pattern: OOTB content for core AWS services activates at ingest, so teams aren't starting from scratch every time a new service or account comes online. The same approach extends to databases and other core infrastructure — the platform arrives opinionated, not blank.

Agentic investigations across your infrastructure with Elastic Observability

Elasticsearch correlates metrics, logs, and traces in a single backend, so the investigation context is assembled before an engineer is paged.

The hard part is 2am. An RDS instance hitting connection limits, starving services upstream. An Auto Scaling group failing health checks for a reason buried in application logs. A pod restart cascading across a namespace.

In a Grafana LGTM stack, you're opening three tabs before you have enough context to form a hypothesis.

In Datadog, the context is unified but the AI is a black box: no BYO-LLM, no data residency options.

In Elastic, metrics, logs, and traces share a single backend and a common schema, so the investigation context is already assembled when the alert fires — no manual correlation across tools, no context lost in translation between query languages. ML anomaly detection runs automatically against infrastructure metrics (Kubernetes, AWS, databases), so the investigation starts from a scored anomaly with context about what's typical, what changed, and how severe the deviation is, not just a raw threshold breach.

When an alert fires, Elastic's investigation workflow correlates signals, assembles root cause context, and surfaces recommended next steps before anyone is paged. The agentic Kubernetes observability post walks through a complete example end to end. The EKS troubleshooting walkthrough shows how Agent Builder and MCP work together for a full root cause loop across EC2, EKS, and related AWS services.

In addition to investigating issues in Elastic Observability, you can use Claude, Cursor, VS Code, or your favorite tool to analyze issues using MCP Apps and agent skills from Elastic. The Observability MCP App extends the analysis to wherever your team already works. If your team investigates in Claude, Cursor, or VS Code, the same investigation capabilities (infrastructure health rollup, service dependency graph, anomaly detail, blast radius analysis) render as interactive views directly in the conversation. Neither Grafana nor Datadog offer this.

• Observability MCP App — Connects Claude, Cursor, VS Code, or any MCP-compatible tool directly to your Elasticsearch data, so infrastructure health, service dependencies, and anomaly context surface as interactive views inside the conversation without leaving your tool of choice. See how it works with Kubernetes.

• Agent Skills — Pre-built skills for Kubernetes, AWS, and other core infrastructure let any agent — in Elastic or your own — run structured investigations against your observability data without custom prompt engineering. Drop them into Claude, Cursor, or your own agent pipeline and they work out of the box. Explore the observability skills or browse the skills library on GitHub.

Migrating from Datadog or Grafana to Elastic Observability

The most common reason SRE teams don't switch observability platforms is migration. Moving years of alert rules, hundreds of dashboards, and runbook-embedded PromQL queries is a daunting operational task, and the cost of maintaining parallel stacks while doing it compounds every day.

The Observability Migration Platform handles the translation automatically. Point the CLI or Claude/Cursor (with Elastic’s agent skills) at your Datadog org or Grafana instance and it converts supported dashboards, alert rules, and PromQL queries into Kibana-native outputs. The tool allows you to see what was fully migrated, what needed tweaks and what is needed from you to migrate everything. You move what you've already built.

On the ingest side, Prometheus Remote Write means the pipeline requires no changes. Scrape configs point to Elasticsearch instead of another Prometheus-compatible backend and the data lands in the same columnar store. Workflows, queries, and alert configurations carry over without change. For teams that want to keep Grafana as a visualization layer during or after migration, the native Prometheus API and PromQL support in Kibana mean the transition can be phased rather than cut over all at once.

Elasticsearch as a backend for Grafana

For teams not ready to leave Grafana, replacing the backend is a migration path in its own right, and there are two ways to do it depending on your workflow.

If your team runs Prometheus today, the lowest-friction path is Grafana's Prometheus data source. Elasticsearch now exposes a native Prometheus-compatible API, so you can point Grafana's existing Prometheus plugin directly at Elasticsearch. No sidecars, no adapters, no pipeline changes required. Existing PromQL dashboards, alert rules, and variable dropdowns work without modification, including Grafana's Metrics Drilldown explorer. Add Elasticsearch as a remote_write target in your Prometheus config and swap the data source URL. That's the full migration for most teams. See the end-to-end setup guide.

For teams that want to go further and query logs, metrics, and traces together from a single Grafana query editor, the official Grafana Elasticsearch plugin now ships with ES|QL support. This unlocks cross-signal correlation directly in Grafana, with Elasticsearch handling all three data types in a unified columnar backend. See how to set it up.

Either way, keep Grafana, replace Mimir and Loki, and gain the full benefit of Elasticsearch's columnar storage and query performance underneath. Years of operational work, preserved. The migration that teams have been putting off becomes a backend swap.

What's GA and what's in tech preview

| Capability | Status | | ----------------------------------------- | ------------ | | Columnar metrics engine (TSDS) | GA | | ES|QL time series support | GA | | PromQL support in Kibana | GA | | Prometheus Remote Write ingest | GA | | Kubernetes infrastructure OOTB experience | GA | | AWS infrastructure OOTB experience | Tech Preview | | Observability MCP App | Tech Preview | | Agent skills | Tech Preview | | Observability Migration Platform | Tech Preview |

The individual posts linked throughout cover GA versus preview specifics and known limitations.

All of this (the columnar metrics engine, native PromQL, agentic investigations, and migration tooling) runs across Elastic's three deployment modes: serverless, Elastic Cloud, and self-managed. Datadog has no on-prem option; Grafana Cloud limits its highest-value features to hosted deployments. With Elastic, you choose where your data lives.

Elastic Observability: lower cost without dropping data

Modern cloud infrastructure broke the observability model built around separate tools for separate signals. The cost is real: duplicate tooling bills, manual correlation during incidents, and data dropped just to stay on budget.

A single backend that stores every signal efficiently means you keep what you need without the bill that usually comes with it. That's a different kind of conversation to have with finance: not "we had to drop data to stay on budget," but "here's what we found." The AI gets the full picture because there's only one picture, and the platform arrives with enough pre-built content to be useful on day one, not after weeks of dashboard toil.

That's possible because of how Elasticsearch is built differently from the platforms you're likely replacing:

• Columnar metrics storage stores stores metrics data highly efficiently in TSDS index mode. 

• Native Prometheus compatibility means existing scrape configs, PromQL queries, and dashboards work without rewriting.

• Unified metrics, logs, and traces in a single backend means investigation context is assembled at query time, not manually across tabs.

• Search and analytics in the same engine — an inverted index for logs, a columnar index for metrics, queried together with ES|QL.

• Agentic investigations that correlate signals, surface anomalies, and suggest remediation before anyone is paged.

• Serverless, Elastic Cloud, or self-managed — you choose where your data lives, which Datadog cannot offer.

The cost conversation with finance becomes about what you found, not what you spent.

Get started

• Start a free trial

• Elastic Observability documentation

• Elastic Observability Labs

Frequently asked questions

Is Elasticsearch now a production-ready metrics platform?

Yes. As of June 2026, Elasticsearch ships a rebuilt columnar storage engine purpose-built for time series data, native Prometheus Remote Write ingest, PromQL support in Kibana, ES|QL time series querying, and out-of-the-box infrastructure dashboards for Kubernetes and AWS. The columnar metrics engine, ES|QL time series support, PromQL, and Prometheus ingest are all generally available in Elastic Serverless and soon GA in Elastic Cloud Hosted.

How does Elasticsearch compare to Datadog for metrics cost?

In comparable metrics workloads, Elastic Observability Serverless costs significantly less than Datadog — in illustrative examples based on published list pricing, more than 50% less, and often closer to two-thirds less. The gap is structural: Datadog bills primarily per host, then adds charges for custom metrics and containers as instrumentation grows. The cost difference is largest for exactly the workloads where Datadog bills most: high-cardinality, densely instrumented environments like Kubernetes and OTel.

How does Elasticsearch metrics performance compare to Prometheus and Grafana Mimir?

ES|QL queries on Elasticsearch run up to 30× faster than Prometheus and Mimir on gauge averages and counter rates, including high-cardinality workloads. Elasticsearch stores OTel metrics at 3.75 bytes per data point; up to 2.5× more efficiently than Prometheus and 2× more efficiently than ClickHouse.

Can teams migrate from Datadog or Grafana to Elasticsearch without rebuilding everything?

Yes. Elastic's Observability Migration Platform converts Datadog and Grafana dashboards, alert rules, and migrates PromQL queries into Kibana as-is. Teams can also keep Grafana as a visualization layer while replacing the backend with Elasticsearch, using the native Prometheus API and PromQL support in Kibana.

What makes Elasticsearch different from Grafana for metrics observability?

Elasticsearch stores metrics, logs, and traces in a single unified backend with one query language (ES|QL), while Grafana's LGTM stack splits metrics (Mimir/Prometheus) and logs (Loki) across separate backends requiring separate query languages. Elasticsearch also ships agentic investigation capabilities, which includes AI Agent, Workflows, MCP App, and Agent skills, a more comprehensive set of capabilities than Grafana. 

Does Elasticsearch support Prometheus and PromQL natively?

Yes, in two distinct ways. First, Elasticsearch accepts Prometheus metrics via Prometheus Remote Write and exposes a native Prometheus-compatible API, so it can serve as a backend for any Prometheus-compatible frontend, including Grafana. Second, Kibana supports PromQL natively, meaning existing queries, dashboards, and alert rules run directly in Kibana without a translation layer or modification.

What infrastructure monitoring content ships out of the box with Elastic Observability?

Elastic ships pre-built dashboards, alert templates, and ML anomaly detection jobs across hundreds of infrastructure integrations covering hosts, containers, cloud services, databases, network devices, and more. For Kubernetes and AWS specifically, the platform also includes agentic investigation content such as agent skills and an Observability MCP App that lets teams run investigations directly from Claude, Cursor, or VS Code. All of this is available at ingest with no configuration required.

Elasticsearch now implements a native Prometheus-compatible API layer, which covers ingestion via Remote Write and querying via PromQL. This post shows the Grafana setup end to end. Companion posts also cover PromQL in Kibana and the Remote Write architecture.

Why use Elasticsearch as a Prometheus backend?

Over the last year, Elasticsearch has become a state-of-the-art metrics store: time series data streams, ES|QL's TS command, and storage and query optimizations that deliver strong metrics performance for Prometheus-style workloads. The Prometheus-compatible API layer makes that engine reachable through the tools your team already uses.

Many teams have invested heavily in Prometheus-based tooling: dashboards, runbooks that reference PromQL queries, on-call workflows built around Grafana panels. Elasticsearch's Prometheus-compatible endpoints let you move metrics storage while keeping those Grafana workflows.

This is particularly relevant if you already use Elasticsearch for logs or traces and want to consolidate your observability data into a single platform, while keeping your Grafana-based workflows intact.

What the Elasticsearch Prometheus API includes

The Elasticsearch Prometheus API exposes three endpoint groups.

Query APIs

The core query endpoints allow Grafana to evaluate PromQL expressions against data stored in Elasticsearch:

• GET and POST /_prometheus/api/v1/query_range evaluate a PromQL expression over a time window and return matrix results. This is what powers most Grafana dashboard panels.
• GET and POST /_prometheus/api/v1/query evaluate a PromQL expression at a single point in time and return vector results.

Both endpoints implement the standard Prometheus response envelope, including result types (vector, matrix, scalar, string), status codes, and error handling. For POST, send parameters in an application/x-www-form-urlencoded body, matching Prometheus client behavior.

Metadata APIs

Grafana's metric explorer, autocomplete, and variable dropdowns rely on metadata endpoints to discover what's available. Elasticsearch supports:

• GET and POST /_prometheus/api/v1/series return time series matching label selectors.
• GET and POST /_prometheus/api/v1/labels return all available label names.
• GET /_prometheus/api/v1/label/{name}/values returns all values for a given label.
• GET /_prometheus/api/v1/metadata returns type and help text for each metric name.

These endpoints power autocomplete and the metric browser in Grafana. The /metadata endpoint additionally enables Grafana's Metrics Drilldown: an interactive metric explorer that displays all available metrics as a grid of live sparklines and lets you drill into any metric without writing a PromQL query.

Index pre-filtering

All query and metadata endpoints accept an optional {index} path segment immediately after /_prometheus/, for example:

GET /_prometheus/metrics-prod-*/api/v1/query_range

This pre-filters the Elasticsearch indices that the PromQL query runs against before any expression evaluation happens. Scoping queries to the relevant data can reduce query work for dashboards that span large volumes of metrics across different data streams.

You can configure a separate Grafana data source per index pattern to give teams scoped access to their own metrics.

Remote Write ingestion

Elasticsearch also implements the Prometheus Remote Write protocol, which lets you ship metrics from Prometheus to Elasticsearch using the standard remote_write configuration. Adding Elasticsearch as a remote write destination requires a single block in your existing Prometheus config:

remote_write:
  - url: "<es_endpoint>/_prometheus/api/v1/write"
    authorization:
      type: ApiKey
      credentials: <api_key>

Metrics are stored in the metrics-generic.prometheus-default data stream by default. You can route metrics from different Prometheus instances or environments into separate data streams using the dataset and namespace path segments:

• POST /_prometheus/metrics/{dataset}/api/v1/write stores metrics in metrics-{dataset}.prometheus-default
• POST /_prometheus/metrics/{dataset}/{namespace}/api/v1/write stores metrics in metrics-{dataset}.prometheus-{namespace}

How to connect Grafana to Elasticsearch

Step 1: Create a serverless project

Sign in to cloud.elastic.co and create a new Observability serverless project. Once the project is ready, you will land directly in Kibana. To find the Elasticsearch endpoint, go back to the Elastic Cloud console, open Manage > Application endpoints, cluster and component IDs, and click the copy icon next to Elasticsearch. The endpoint looks like:

https://<project-id>.es.<region>.<provider>.elastic.cloud

Step 2: Create API keys

Create two API keys with scoped privileges: one for ingestion, one for querying. Using separate keys means a leaked Grafana key cannot be used to write data, and a leaked ingest key cannot be used to read it.

In your project, open Admin and settings (the ⚙️ icon at the bottom left of the side nav), go to API keys, and create the first key.

Ingest key (prometheus-remote-write): restricts access to writing metrics data streams only. In the Control security privileges section, paste the following role descriptor:

{
  "ingest": {
    "indices": [
      {
        "names": ["metrics-*"],
        "privileges": ["auto_configure", "create_doc"]
      }
    ]
  }
}

Create a second key for Grafana in the same section.

Query key (prometheus-grafana): restricts access to reading metrics data streams only.

{
  "query": {
    "indices": [
      {
        "names": ["metrics-*"],
        "privileges": ["read", "view_index_metadata"]
      }
    ]
  }
}

Copy both key values before closing. You will not be able to retrieve them again.

Step 3: Run Prometheus and Grafana

Create a prometheus.yml that scrapes Prometheus itself and forwards those metrics to Elasticsearch. Replace <es_endpoint> with the endpoint from Step 1 and <ingest_api_key> with the ingest key from Step 2:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

remote_write:
  - url: "<es_endpoint>/_prometheus/api/v1/write"
    authorization:
      type: ApiKey
      credentials: <ingest_api_key>

Next, create the Grafana provisioning directories:

mkdir -p grafana/provisioning/datasources grafana/provisioning/dashboards

Then create a Grafana data source configuration that points at the Elasticsearch Prometheus API. Create grafana/provisioning/datasources/datasource.yml, replacing <es_endpoint> and <query_api_key> with the values from Steps 1 and 2:

apiVersion: 1

datasources:
  - name: Elasticsearch
    type: prometheus
    access: proxy
    url: "<es_endpoint>/_prometheus"
    uid: elasticsearch-prometheus
    isDefault: true
    jsonData:
      httpHeaderName1: Authorization
    secureJsonData:
      httpHeaderValue1: "ApiKey <query_api_key>"

This configures a Prometheus-type data source backed by Elasticsearch. Grafana sends Prometheus queries with POST by default, which Elasticsearch accepts on authenticated HTTPS endpoints such as Serverless.

Create grafana/provisioning/dashboards/dashboards.yml to tell Grafana where to find provisioned dashboards:

apiVersion: 1

providers:
  - name: default
    type: file
    options:
      path: /var/lib/grafana/dashboards

Finally, create a docker-compose.yml to start everything:

services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

  download-dashboard:
    image: curlimages/curl:latest
    user: root
    volumes:
      - dashboards:/dashboards
    command: >
      sh -c 'curl -fsSL https://grafana.com/api/dashboards/3662/revisions/2/download
      | sed "s/\$${DS_THEMIS}/elasticsearch-prometheus/g"
      > /dashboards/prometheus-overview.json'

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    depends_on:
      download-dashboard:
        condition: service_completed_successfully
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=grafana
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - dashboards:/var/lib/grafana/dashboards:ro

volumes:
  dashboards:

The download-dashboard service fetches the Prometheus 2.0 Overview dashboard from the Grafana marketplace and patches it to use the Elasticsearch data source. The sed replaces the dashboard's ${DS_THEMIS} data source placeholder with our data source UID. This is needed because Grafana's provisioning does not resolve these placeholders on its own (grafana#10786). Grafana waits for the download to finish before starting.

Start both with:

docker compose up -d

Prometheus will start scraping its own metrics and shipping them to Elasticsearch every 15 seconds. Give it one or two scrape intervals before opening the dashboard.

Step 4: Open the dashboard

Open Grafana at http://localhost:3000 and log in with admin / grafana. Go to Dashboards and open Prometheus 2.0 Overview.

The dashboard shows your Prometheus self-monitoring metrics, pulled from Elasticsearch via PromQL queries.

Step 5: Explore metrics with Grafana's Metrics Drilldown

Because Elasticsearch implements the Prometheus metadata and discovery endpoints, Grafana's Metrics Drilldown works out of the box.

In Grafana, go to Drilldown > Metrics in the left-hand navigation and select Elasticsearch as the data source. Grafana loads all available metrics from Elasticsearch and displays them as a grid of live sparklines. From there you can filter by label, search by name, and drill into any metric without writing PromQL.

Current limitations and what's next

This is the first implementation and updates should be expected. All of the following are actively being worked on:

PromQL coverage is not yet complete

Queries using group modifiers (for example, on(instance, job)), set operators (or, and, unless), and certain functions like topk are not yet supported.

Form-encoded POST has deployment requirements

POST requests with application/x-www-form-urlencoded bodies require security enabled, TLS on the Elasticsearch HTTP interface, and an authenticated request. Serverless meets these requirements out of the box. If TLS terminates before Elasticsearch and the node sees plain HTTP, use GET with query-string parameters instead.

Only Remote Write v1 is supported

Remote Write v2 support is planned.

Instant queries are not point-in-time yet

The instant query endpoint currently runs a short range query under the hood and returns the last sample. It will be replaced with a proper point-in-time evaluation.

Coming next: broader PromQL function and operator coverage, Remote Write v2, and exemplar endpoints.

Frequently asked questions

Can Grafana query Prometheus metrics stored in Elasticsearch? Yes. Grafana can use Elasticsearch as a Prometheus data source when the URL points to /_prometheus. Queries use PromQL and return the standard Prometheus response format for Grafana dashboards, variables, Metrics Drilldown, and alerting.

Do I need to change Prometheus or Grafana dashboards to use Elasticsearch? You do not need to rewrite PromQL queries or dashboard panels for common Grafana use cases. Configure Prometheus Remote Write to send metrics to Elasticsearch, then point Grafana's Prometheus data source at the Elasticsearch /_prometheus endpoint.

Why use Elasticsearch instead of a separate Prometheus long term storage backend? Using Elasticsearch as a Prometheus backend lets you store metrics with logs and traces under the same access controls and retention model. Recent work on the Elasticsearch metrics engine also delivers strong performance for Prometheus-style workloads. For the benchmark details, see the Elasticsearch metrics performance post.

What PromQL features are supported in Elasticsearch today? Elasticsearch supports common PromQL query patterns used by Grafana dashboards. Advanced group modifiers, set operators, and topk are not yet supported.

Can I limit Grafana queries to specific Elasticsearch indices? Yes. Add an index pattern after /_prometheus/, such as /_prometheus/metrics-prod-*/api/v1/query_range. This pre-filters the Elasticsearch indices before PromQL evaluation and can reduce query work for large metrics deployments.

Prometheus API availability

The Prometheus-compatible API is available now on Elasticsearch Serverless with no additional configuration.

If you run into issues or have feedback, open an issue on the Elasticsearch repository.

Getting SNMP data into Elasticsearch unlocks rich visibility into your network — interface utilization, routing health, L2 forwarding, and more. The path there involves a few familiar steps: choosing which MIBs to walk, mapping OIDs to human-readable field names, configuring SNMP v2c or v3 authentication, accommodating vendor-specific MIB extensions, and tuning the pipeline to gracefully handle device timeouts across large inventories. With a solid template in place, what used to be a bespoke Logstash project becomes a repeatable, shareable setup that any engineer on the team can pick up and extend.

The plugin includes a Logstash pipeline template that handles the common cases out of the box. It walks IF-MIB (interface counters and status), IP-MIB (ARP tables and IP address assignments), BRIDGE-MIB (MAC address forwarding tables), BGP4-MIB (BGP peer sessions), and OSPF-MIB (OSPF neighbor adjacencies) per target device on a configurable poll interval. You add your device list and authentication details, start Logstash, and data begins flowing into Elasticsearch.

The template also handles the operational annoyances that trip people up: poll timeouts, missing OID branches on devices that don't support a given MIB, and batching walks across large device inventories.

Structuring SNMP data in Elasticsearch: schema design

Once SNMP data lands in Elasticsearch, the next problem is structure. Interface counters like ifInOctets and ifOperStatus map to ECS concepts reasonably well. They're host-level metrics with direct equivalents in host.network.* fields. But the data network engineers actually need for troubleshooting is relational, and this plugin offers a way to view those relationships.

A BGP peer session has a state, a remote AS number, an uptime, and an update count. An OSPF adjacency has a neighbor router ID, an area, a priority, and a state machine position. A MAC table entry records which physical switch port a given MAC address was learned on. None of these have ECS field definitions, and stuffing them into generic event.* or observer.* fields loses the semantic meaning that makes the data useful.

The plugin takes an opinionated approach: use ECS where it fits, extend with clear namespaces where it doesn't. Interface data maps to ECS-aligned fields. Routing protocol and L2 forwarding data goes into purpose-built namespaces (bgp_peer.*, ospf_neighbor.*, arp.*, mac_table.*) with field names that match the concepts operators already think in. If you know what bgpPeerState means on a router CLI, bgp_peer.state in Elasticsearch is immediately familiar. If you already collect SNMP data in a homegrown schema, the plugin's templates and ingest pipeline will complement it rather than replace it. The new fields are additive, so you can adopt them at your own pace!

| Data Type | Key Fields | ECS Namespace | | --- | --- | --- | | BGP Peer Session | State, Remote AS, Uptime, Update Count | bgp_peer.* | | OSPF Adjacency | Neighbor Router ID, Area, Priority, State | ospf_neighbor.* | | MAC Table Entry | Switch Port, Learned MAC Address | mac_table.* | | ARP Entry | IP-to-MAC Mapping | arp_table.* |

An ingest pipeline (snmp-device-enrichment) handles classification at index time. It parses each device's sysDescr string to assign a normalized device.type (router, switch, firewall, access point) and device.vendor, so every downstream consumer (dashboards, ES|QL queries, alerting rules, the topology view) gets consistent device metadata without custom parsing. The pipeline recognizes common vendors out of the box and is extensible for environments with less common hardware.

The result is SNMP data you can query like any other structured data in Elasticsearch. "Show me every BGP session not in Established state" is a filter, not a regex exercise. "Which Cisco switches have interfaces that are admin-up but oper-down" is a KQL query, not a script.

Visualising SNMP network topology in Kibana

Dashboards excel at answering "what are the numbers?" A topological view answers a complementary question: "what's connected to what?" Network engineers think in topology: upstream and downstream relationships, path diversity, and blast radius of a link failure. A spatial, graph-based view brings that mental model directly into Kibana, sitting alongside the charts and data tables operators already rely on.

The plugin adds an interactive topology graph to Kibana's Observability navigation. It reads the structured SNMP data from Elasticsearch, builds an adjacency graph from ARP, MAC table, BGP, and OSPF relationships, and renders it as a force-directed layout you can zoom, pan, and rearrange. Nodes are devices, edges are discovered relationships, and clicking any device opens a flyout with its interface table, ARP neighbors, and routing protocol sessions.

How do you set up SNMP network topology monitoring in Kibana?

The plugin is nearly ready to go out of the box, only a few assets need installation. Here's what the setup looks like:

1. Install the plugin zip on a self-managed Kibana instance (bin/kibana-plugin install file:///path/to/networkTopology-<version>.zip).

2. Apply the index templates and ingest pipeline. Click through the template installation in the plugin's Setup tab. A few button clicks and the schema is in place.

3. Deploy the Logstash pipeline. Add your device targets, authentication details, or other configuration to the included template and start it. If you're using Logstash Centralized Pipeline Management, push it from Kibana, no SSH required.

Data hits Elasticsearch on the next poll cycle, the ingest pipeline classifies and enriches them, and the topology view populates. Start to finish, you're looking at minutes, not hours or days of trial and error.

A sample data generator is included for teams that want to evaluate the plugin before connecting to live infrastructure; spin up a Docker development environment and explore the full feature set with a realistic multi-site network.

The processing pipeline is straightforward: The profiler samples every CPU core on the system at a fixed rate (19Hz by default), unwinds execution stacks, symbolizes the resulting stacktraces and ships the profiles to a backend like Elasticsearch.

And then… the user has to figure out what to do with them.

That last step is where continuous profiling has historically faced adoption challenges, as the path from "profiling is on" to "profiling is useful" is steeper than it should be.

Four barriers to adoption

• Storage cost: Full stacktraces, even after deduplication and clever storage schemas, are expensive to store at fleet scale. That cost makes continuous profiling an opt-in feature in practice: a lot of potential users never enable it, and the ones who do, tend to enable it only on a subset of hosts.

• Query friction: A normalized stacktrace schema is optimized for ingestion and storage but complicates ad-hoc questions. "How much CPU time does my service spend in TLS?" is a simple question that may require intricate ES|QL or custom code in order to be answered.

• AI-hostile data: Normalized stacktrace data (typically involving multiple levels of indirection) resists straightforward algorithmic analysis. LLMs in particular struggle with it and necessitate further data transformations into representations more amenable to LLM processing.

• UX barrier: Flamegraphs are extremely useful when you know how to read them but intimidating when you don't.

These four barriers compound: storage cost limits coverage, the UX barrier limits who benefits from coverage, query friction limits what questions users can ask and the AI-hostile data representation limits what the system can do when users don't know what questions to ask.

How profiling-derived metrics work: classify at the edge

The core idea is simple: instead of sending full stacktraces all the way to a backend and asking the user to make sense of them there, we classify and count at the edge, inside an OpenTelemetry Collector pipeline, and emit ordinary OpenTelemetry time-series counters. The profiling logic itself doesn't change; it's still the OpenTelemetry eBPF profiler running inside the OpenTelemetry Collector. All the new work happens in a stateless connector inside the Collector: the connector inspects each stacktrace produced by the profiler, classifies its frames into one or more categories and increments counters.

We've released profilingmetricsconnector as part of Elastic's opentelemetry-collector-components repository. It sits between the OpenTelemetry eBPF profiler receiver and any metrics exporter, and turns symbolized stacktraces into named, aggregated counters with attributes.

Because the profilingmetricsconnector lives inside the standard OpenTelemetry Collector pipeline, every metric it produces flows through the same processors as the rest of your telemetry. In the following example, the resourcedetectionprocessor enriches each counter with host-derived attributes.

connectors:
  profilingmetrics:
    flush_interval: 30s

receivers:
  profiling: {}

exporters:
  elasticsearch:
    endpoints:
      - # ENDPOINT
    api_key: # API_KEY
    mapping:
      mode: otel

processors:
  resourcedetection:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: false
        host.arch:
          enabled: true
        os.description:
          enabled: true
        os.type:
          enabled: true

service:
  pipelines:
    profiles:
      receivers: [ profiling ]
      exporters: [ profilingmetrics ]
    metrics:
      receivers: [ profilingmetrics ]
      processors: [resourcedetection]
      exporters: [ elasticsearch ]

Profiling-derived CPU metrics: what gets emitted

The connector ships with a set of pre-baked counters built from useful classification rules. Each metric is a count of stacktrace samples whose leaf frame matched a particular category, with the frequency value standing in for CPU consumption.

| Metric | Classifies | Attached metadata | |---|---|---| | kernel.count | Kernel leaf frames | syscall, category (disk/rw, ipc/rw, network/{tcp,udp,other}/rw, memory, synchronization, …) | | native.count | Native C/C++/Rust leaf frames | shared library name (libcrypto, libclrjit, libsystemd, …) | | hotspot.count, go.count, python.count, … | Runtime-specific leaf frames | runtime-specific attributes |

The kernel categorization is worth a closer look as a modern Linux kernel has more than 400 system calls. However, most of what shows up in CPU stacktraces falls into a handful of subsystems: filesystem read/write, network read/write, memory management, scheduling, synchronization. Some syscalls (e.g. read, write) are ambiguous on their own and only become specific when one examines more frames down the stack: ext4_file_read_iter points to filesystem, tcp_v4_rcv to network. The connector handles this disambiguation as part of frame iteration.

Native frames typically lack symbolic information beyond shared library names, but those names are still informative: libssl and libcrypto mean cryptographic work as part of OpenSSL or one of its variants; libz means compression; libclrjit means the .NET JIT is busy. We don't need to enumerate libraries statically as the connector dynamically generates shlib_name attribute values using the trimmed library name (e.g. libssl not libssl.so.3) for clean cardinality.

Currently, for each stacktrace, the connector computes a Self CPU count (the leaf frame matched the category) corresponding to exclusive CPU usage. A complication exists for fine-grained kernel categories like network/tcp/write where the actual leaf frame is usually a device-driver call that we can't meaningfully match. We deal with that by trying to match frames further up the stack (e.g. tcp_sendmsg is enough to correctly classify the sample).

Users can also add their own categories by specifying a frame pattern (e.g. a function or package) and the connector will generate counters for them.

Benefits of profiling-derived metrics for observability

This shift looks small from the outside — "we're emitting counters" — but it changes almost everything about how profiling fits into an observability stack.

• Orders of magnitude less storage: A counter aggregated over a 5-second (or 30-second or one-minute) window is dramatically cheaper than the full stacktraces it distills. The pre-aggregation interval is configurable with the trade-off being time resolution rather than categorization fidelity. For most "where is my CPU being spent?" questions, 30 seconds is plenty.

• On by default: Because the storage cost is now in line with regular metrics, profiling-derived metrics can be on for everyone, on every host, from the moment the profiler is deployed. Users get a CPU breakdown by runtime, syscall, kernel category and shared library on day one.

• Standard dashboards: These are ordinary OpenTelemetry time-series counters and can be visualized ad-hoc using stacked bar graphs, pie charts, top-N panels or any other visualization Kibana supports out of the box. The same Lens and TSDB-backed views for application metrics work here.

• AI and query-friendly: Standard time-series data is trivially consumable by ES|QL, ML jobs, anomaly detectors and by LLMs. "Show me the top services by network/udp/write time, filtered to the payments namespace, over the last six hours" is one query that is not only simple for the system to answer but also simple for an LLM to generate.

• Cross-signal correlation: Because the metrics flow through the standard OpenTelemetry Collector pipeline, they pick up the same resource attributes (e.g. service.name, k8s.pod.name, host.name, deployment.environment) that logs, other metrics and traces already carry.

• Instant value, with a path to more detail: A user who just wants to know "what's burning my CPU?" gets a meaningful answer without ever opening a flamegraph. A user who wants to dig deeper still has the full eBPF profiler underneath, ready to hand back complete stacktraces when they're warranted.

User-programmable profiling and adaptive sampling

The longer-term direction is for the profiler to stop being something users consume and start being something they program. User-defined metrics are the first step in this direction, complemented by on-demand (full) profiling and adaptive sampling.

Profiling-derived metrics or other signals can act as a trigger for on-demand profiling where the system enables full profiling on a specific host or service to capture complete stacktraces. In that way, the full profiling processing and storage cost is paid only when it matters.

We can apply the same idea to the sampling rate. 19Hz is a sensible baseline for steady state but when the metrics signal an interesting event or an anomaly, the system can automatically ramp to 100Hz or higher to capture high-fidelity data for the time window during which it's relevant. It can then ramp down to baseline.

How profiling-derived metrics enable self-driving observability

Most observability stacks today use an open-loop model: the profiler emits data with a fixed configuration. Then a human looks at flamegraphs and dashboards, potentially correlates with logs, other metrics and traces, forms a hypothesis and triggers a deeper investigation. Every link in this chain requires a human decision. Nothing feeds back into the profiler at speed and the system cannot act on its own observations.

Profiling-derived metrics close that loop.

1. A "significant host events" metric, an anomaly on network/udp/write or a spike in native.count/libz: something crosses a threshold.

2. The profiler adjusts in response: sampling rate increases, full profiling turns on for the affected hosts.

3. The richer data is correlated against logs, traces, and other metrics by an LLM, by a human or both. The same resource attributes that make cross-signal correlation easy for the user make it easy for the system.

4. A root cause is identified. A remediation is suggested or applied. The metric returns to baseline and the loop continues.

This is what we mean when we talk about self-driving observability. The profiler is no longer just an instrument that someone wields. It is the sensory organ of an autonomous feedback loop: a system that observes itself, decides what to look at more closely and adjusts its own configuration in response to what it sees.

What's next: inclusive CPU, off-CPU metrics, and runtime-specific profiling

Any piece of data visible in a stacktrace can be a metric source and several extensions are already on the roadmap.

• Inclusive-CPU metrics: Today's pre-baked counters attribute CPU at the leaf frame (exclusive-CPU). Inclusive-CPU metrics will attribute the entire call chain which is useful when you care about the total cost of a function call — the function plus everything it transitively calls — not just the work done directly in its own body.

• Runtime-specific metrics: GC time per runtime, JSON/Protobuf serialization, RPC frameworks, FFI boundaries. The kinds of questions every team eventually asks about their language runtime, answered by default.

• Off-CPU metrics: On-CPU profiling tells you where you're spending CPU but Off-CPU profiling tells you where you're not (e.g. blocked on I/O, locks). The same classification logic applies, with the only change being the source signal.

Profiling-derived metrics are an active area of work within Elastic and the profilingmetricsconnector is the place to start if you want to play with this today. A ready-made Kibana integration ships dashboards for all the metrics described above.

If you're already using Elastic's continuous profiling, expect these metrics to show up as first-class citizens in the Elastic stack. If you're not, this is a very low-friction way in as no flamegraph expertise is required and storage cost is minimal.

The flamegraph isn't going anywhere, but for the first time, it isn't the only way profiling yields results.

Elasticsearch has supported storing metrics in time-series data streams (TSDS) since version 8.7. This offering mainly focused on storage gains as explained in an earlier blog post. Still, performance was not on par with specialized systems for storing and querying metrics, in terms of storage efficiency, indexing throughput and query latency.

In the past year, we revisited the storage layer, optimized ingestion for OTel metrics and extended the ES|QL compute engine with vectorized processing for time series data. These efforts led to substantial performance wins across the board, compared to earlier versions of TSDS:

1. Up to 6.6x improvement in storage efficiency, reaching 3.75 bytes per data point in OTel metrics
2. Up to 50% improvement in indexing throughput for OTel data
3. Up to 160x improvement in query latency, including blazing fast counter rate evaluation and window support in time series aggregations

Elasticsearch has thus become a leading columnar metrics engine, matching or exceeding the competition (like Prometheus, Mimir, and ClickHouse) in indexing throughput and exceeding it by up to 2.5x in storage efficiency and 30x in query performance. All while maintaining the ability to store logs and other data and fully use the rich querying capabilities of ES|QL (e.g. inline stats, lookup join) — which other PromQL-based systems lack. Elasticsearch can thus serve as a unified storage and query engine for all user data, with no compromises for metrics and observability applications.

How TSDS is organized

TSDS has the following properties that help improve the performance of time-series codecs and produce correct results when aggregating data points per time series:

• The metric name and the dimension names and values are used to calculate the _tsid, a unique identifier per time series.
• TSDS get sorted by [_tsid ascending, timestamp descending] order. Each time series is thus stored in sequence on disk, with newer data points appearing first. Since the _tsid is calculated over dimension values, the latter are also clustered on disk.
• Shard routing is based on _tsid, with each _tsid value appearing in one shard only.
• Backing indices are time-bound, with no overlap over time between them.

The rest of this post explains how we use these properties to improve storage, indexing, and query performance.

Storage optimizations

TSDS already achieved a very competitive storage footprint, reaching 0.9 bytes per data point, when it is possible to combine many metrics in a single doc, sharing the same dimension values. However, when most data points have a unique set of dimensions (which is typical for OTel or Prometheus metrics), docs end up containing a single data point. In this setup, storage required 25 bytes per data point, with dedicated metrics stores requiring less than 10 bytes per data point.

To further reduce the storage footprint, we applied a series of optimizations over the past year:

Replace inverted indices and BKD trees with doc value skippers

Elasticsearch creates inverted indices (for text values) or BKD trees (for numeric values) by default for all non-metric fields, i.e. for @timestamp and dimensions. These indices improve performance for queries including filters on these fields, but have significant impact to storage — effectively doubling the footprint for each field. More so, they are also processed during segment merging, increasing the cpu, memory and storage overhead and slowing down the system — especially in high ingest throughput scenarios, as is often the case with metrics.

Lucene has been extended with doc value skippers, a form of hierarchical sparse indices that store the minimum and maximum value of blocks of documents. Range queries can check these min and max values and ignore blocks that don't fall into the requested range. Skippers work particularly well on sorted fields. Since TSDS are sorted by [_tsid, timestamp desc], dimension values get also clustered on disk. It's therefore possible to replace indices on @timestamp and dimension fields with doc value skippers that amplify the columnar layout — each field stored in its own files, with no duplicate tracking of each doc for indexing purposes.

Doc value skippers have negligible storage overhead — replacing indices with them led to a reduction of 10 bytes out of the initial 25 bytes per data point in OTel. Moreover, they work very well in practice when queries include filters on time ranges or dimension values (including prefixes and regex) — there was no noticeable regression in query performance in our benchmarks when they replaced separate indices. Doc value skippers are enabled for TSDS by default since version 9.3.

Enable synthetic ids

The _id metadata field was another big contributor to the storage footprint. TSDS has already been extended to trim the doc values once they were no longer needed for replication, but the inverted index was kept around to efficiently support the id-based APIs (Get, Delete, Update).

The id value for TSDS is synthesized by combining the _tsid and @timestamp values that uniquely identify each data point. Since these fields are configured with doc value skippers, it's possible to replace the inverted index on _id with (a) retrieval of the _tsid and @timestamp value from the _id value, and (b) checks for matches using doc value skippers respectively. Care has to be taken to avoid expensive checks for duplicate ids during metric ingestion, with segment-level bloom-filters keeping the overhead at bay.

Supporting synthetic ids in metrics is a first for Elasticsearch. It led to a reduction of 5 bytes out of the initial 25 bytes per data point for OTel metrics, with no loss of functionality. Synthetic ids are enabled for TSDS by default in version 9.4. We plan to extend their uses in logs and other applications after further evaluation.

Trim sequence numbers

Sequence numbers are used as part of replication, but also to provide strong consistency semantics on doc modification operations through Optimistic Concurrency Control (OCC). While such semantics are applicable to certain scenarios, they don't fit in metrics where concurrent updates are very rare, with no practical need for guarding against concurrent operations on data points with matching ids. We therefore decided to disable the use of sequence numbers in all APIs, along with OCC support, for TSDS, in version 9.4. This leads to a substantial storage reduction of 4 bytes out of the initial 25 bytes per data point for OTel data, as there's no inverted index and sequence numbers get trimmed once no longer needed for replication. Update and delete by query operations are still supported, albeit with weaker consistency semantics.

If OCC is still deemed important for a particular metrics application, the old behavior can be restored by setting index.disable_sequence_numbers: false in the index template of the involved TSDS.

Use large numeric codec blocks

TSDS already uses an advanced codec, as explained in an earlier article. The codec works very well in most cases, but has poor performance in case of repeated sequences of keywords and numbers, leading to an inflated storage footprint for dimensions containing IP and MAC addresses. We identified that the existing logic for identifying repeated sequences requires larger codec blocks to work well, especially as the sequence length increases. After experimentation, the numeric block size was increased from 128 to 512 elements in version 9.3, leading to a reduction of 2 bytes out of the initial 25 bytes per data point for an OTel dataset containing IP and MAC addresses as dimensions. We're also working on a more configurable codec layout that will allow more flexibility with block sizes and other parameters, based on field type and cardinality.

Indexing throughput

Elasticsearch has support for bulk ingestion of documents. This entrypoint has long been optimized for leniency, ensuring that all docs get accepted. This flexibility, however, incurs additional processing cost during indexing. Metric applications proved good candidates for using different approaches to reduce this overhead, as explained below.

Introduce OTLP protobuf entrypoint

OTel metrics and Prometheus have established protocols for metrics ingestion, using protocol buffers. In the past, a translation step was required to convert collected protobuf messages to bulk requests that Elasticsearch can consume.

Elasticsearch was recently extended with endpoints accepting messages from OTel metrics collectors and over Prometheus remote write. Parsing and processing these (binary) messages is cheaper, compared to json parsing, while hash operation over dimensions for _tsid calculations get reused and amortized across more data points within a single protobuf message. Furthermore, _tsids get evaluated once per doc in the coordinator nodes and propagated to data nodes for indexing, thus deduplicating an expensive step per indexed doc. These improvements led to up to a 20% speedup in indexing throughput for OTel metrics. The OTLP entrypoint was added in version 9.2 (tech preview) and reached GA in version 9.3. We've added similar entrypoints for Prometheus remote write in version 9.4 (tech preview) and are actively working to cover OTel Logs and Traces.

Reduce indexing CPU with doc value skippers

In addition to a substantial storage footprint, inverted indices require a lot of cpu to build and reconstruct during segment merging. The use of doc value skippers in their place helps also reduce cpu load at ingestion and thus improves indexing throughput by 10%, a welcome bonus on top of the aforementioned storage wins.

Synthetic recovery source

The original source of a document, as provided at index time, is never stored for metrics. Still, Elasticsearch needed to temporarily store it for replication purposes. That changed in version 9.1, where the source gets synthesized on demand for replication purposes. This is known as synthetic recovery source and reduces disk I/O by 50%, with a significant impact to metrics indexing performance. Check out this article for more details.

Query execution

Replacing inverted indices with doc value skippers leads to a pure columnar storage layout for TSDS, with metric and dimension fields stored as Lucene doc values, each field encoded and compressed in their own file. Combined with the introduction of the ES|QL compute engine that uses vectorized execution internally, it became possible to introduce a fully columnar storage and query processing engine for metrics in Elasticsearch. We pushed this idea to the extreme and implemented a columnar metrics processing engine that comfortably outperforms dedicated metrics engines and other columnar stores in query performance.

Time series integration in compute engine

Time series processing is largely based on applying aggregation functions per time series (or _tsid), such as a gauge average or a counter rate. These partial results are then reduced by a secondary function to produce results for the grouping dimensions, e.g. per host and process. Observability dashboards are built on top of this execution model, providing summary views of how metrics evolve over time and allowing for quick deep-dives by filtering on dimension values and time ranges.

To support this execution model, we introduced the TS source command, providing a simple yet powerful syntax for executing such queries that combine an inner aggregation function per time series with an outer aggregation over the partial results of the former. For instance, the following query calculates the hourly sum of rate of search requests per host over the last day:

TS metrics
  | WHERE TRANGE(1d)
  | STATS SUM(RATE(search_requests)) BY TBUCKET(1h), host

To execute this query, the compute engine is aware of how data is stored and applies the inner aggregation function per _tsid value. Since data are sorted by _tsid, time series aggregation functions process metric values as they get fetched, until the _tsid changes or the timestamp belongs to the next time bucket. This leads to vectorized execution of these functions over the fetched columns of metric values, while dimension values are only fetched (once) when the _tsid changes. The evaluation of the secondary aggregation function is also efficient, with partial aggregates stored in arrays of primitive values that get populated when _tsid values change.

The compute engine has inherent support for parallel query evaluation, taking full advantage of the available processing cores. Time series aggregations fully use this feature and process data points in parallel as applicable, reducing response times through improved cpu utilization.

Time series processing in ES|QL was introduced in version 9.2 as tech preview and reaches GA in version 9.4. We expect all metrics applications to adopt it and benefit from the much improved query performance wins.

Zero-copy data decoding and loading

Vectorized processing of time series data delivered immediate performance wins (8x for some queries), compared to aggregations through the /_search API, but the performance was still inferior when compared to competitive metrics stores. Benchmarking and profiling showed that there were too many array copies within the compute engine, between data decoding and evaluation of aggregation functions. To that end, the following optimizations were introduced:

• The codec for TSDS was extended to decode on-disk data directly into primitive arrays inside blocks that the compute engine uses to evaluate time series aggregations. No additional copies required, as the compute engine can bulk-read these blocks and process their arrays, one column at a time.
• Blocks containing a single value N times are represented as constant blocks with these 2 values, as opposed to an array with length N, a form of in-memory run-length encoding. Filtering and aggregation operations were extended to efficiently consume these blocks. This reduced memory pressure and cpu overhead for the _tsid and dimension fields, as their values get clustered due to index sorting.
• Documents with null values for the aggregated metric fields are filtered out at the Lucene level, before they get decoded and copied into blocks.
• All filters and regular expressions on the timestamp and dimension fields get pushed down to Lucene that makes use of doc value skippers to efficiently filter out non-matching docs.

Combined, these optimizations led to query execution speedups exceeding 10x (totaling 80x when combined with the 8x speedup from vectorized execution). They were included since the introduction of the TS source command in version 9.2, and fine-tuned ever since.

Optimized counter rate evaluation

While most time series aggregations can be trivially parallelized and evaluated, rate evaluation of cumulative counters is tricky as it requires processing all data points in order to detect counter resets (e.g. when a host restarts). To address this, the compute engine uses the _tsid prefix to shard time series across threads. Care has been taken to assign in-order ranges of _tsid values to each thread, as opposed to hash-partitioning _tsids, so that each thread can scan on-disk data in order, still making use of efficient decoding and zero-copying into blocks. The performance wins are impressive, with rate evaluation performance far exceeding dedicated metrics stores as we shall see in the next section.

Another interesting problem for cumulative counters is how to properly calculate counter increases for the entire time bucket when there are no data points at the bucket boundary timestamps. Metrics systems often use extrapolation, extending the first and last data points of each time bucket to the boundaries, or calculate the delta between the last data point of adjacent buckets. We posit we can do better, by interpolating between the last data point of each bucket and the first of the next, to get an estimate of the value on each boundary. The delta is then calculated over the interpolated values of the lower and upper boundary of each time bucket.

Sliding window support

Elasticsearch has long supported aggregations bucketed by time, but it was not possible to extend the window of processed data beyond each time bucket. Using windows larger than the time bucket, e.g. a window of 5 minutes for per-minute bucketing, helps smoothen out spikes and observe the underlying trend per time series with reduced noise:

All time-series aggregation functions have been extended with window support, as an optional argument. In case the window is a multiple of the time bucket (e.g. 1h window with TBUCKET(5m)), the compute engine first aggregates data points over intervals matching the time bucket span, and then combines these partial results per window span. This 2-phase approach eliminates repeated scans of data points and makes optimal reuse of intermediate results, improving response times. Window support was introduced as tech preview in version 9.3 and reaches GA in version 9.4.

Efficient datetime rounding

Queries on time-series data commonly include time bucketing. While data points can be trivially assigned to sub-hour time buckets, larger buckets start interfering with issues like time zones, daylight savings, variable days per month etc. Elasticsearch has elaborate logic for datetime rounding that takes these peculiarities into account, but that has relatively high cpu cost when processing time series data.

To mitigate this, the compute engine has been extended to identify cases where simpler logic can be employed to assign data points to time buckets. For instance, it can identify when the buckets are sub-hour or when timezones and daylight savings don't affect a particular query, and switches to simple modulo operations for datetime rounding. This led to a further 30% improvement in response times for certain queries. This change is introduced in version 9.4.

Performance evaluation

To evaluate the performance of our offering and track how it evolves and improves over time, we focused on OTel metrics since (a) Open Telemetry is the industry standard for collecting metrics, with universal adoption by all cloud providers and (b) they lead to a storage layout with 1 metric per doc, a setup that traditionally hurt performance for Elasticsearch.

We rely on Metricsgenreceiver to generate datasets. This tool is inspired by TSBS, producing data simulating the data points collected by the OTel hostmetricreceiver. We used two datasets:

1. A low-cardinality setup, with 100 hosts sending metrics every 1s, containing 14k time series in total
2. A high-cardinality setup, with 10k hosts sending metrics every 10s, containing 1.4M time series in total

We benchmarked on single-node deployments on EC2, using c6i.4xlarge and c8g.8xlarge machines for the low- and high-cardinality datasets respectively.

For competitive comparison, we used Prometheus (v.3.11.1), Mimir (v.3.0.6.) and ClickHouse (v26.3.9.8-lts). Prometheus and Mimir have proper time series processing, e.g. for counter rate, whereas ClickHouse lacks such support and only provides approximate values at best (for instance, it can't track counter resets consistently). We still report response times for ClickHouse to showcase that, once we optimize Elasticsearch for columnar query processing, it can exceed competing columnar engines even when they don't process the data per time series as expected.

We strived to use the default configuration for every system (including Elasticsearch), without tweaking them to optimize performance for the particular workload. This helps understand the user experience when systems are deployed by novice users, without much experience (or time) to tweak before receiving metrics traffic and setting up dashboards. We focused on single-node runs to keep noise low and accommodate all systems (Prometheus doesn't offer a multi-node setup out of the box). Elasticsearch performance provably scales well with the number of nodes; we plan to share scalability results in a future post.

Storage efficiency and indexing throughput

Our efforts to improve storage efficiency paid big dividends. Performance on OTel metrics dropped from 25 to 3.75 bytes per data point, in a year. Such an improvement, on top of an offering already optimized for time series, is really impressive and very rare in the industry:

The competitive picture looks favorable at this point, with Elasticsearch:

• Slightly outperforming Mimir in storage efficiency and indexing throughput
• Outperforming Prometheus by 2.5x in storage efficiency and by a small margin in indexing throughput
• Outperforming ClickHouse by 2x in storage efficiency and by 40% in indexing throughput

Query performance

The novel columnar engine for metrics processing proves very efficient in practice. We used a mix of queries based on gauge averages and counter rates, the most common operations that require different optimization approaches. The queried interval was 4 hours of data, covering all time series per metric.

ClickHouse doesn't support time series aggregations, so the results have limited value and are not directly comparable to Prometheus or Mimir that natively support time series processing. We used the published guidelines to adjust each query to get similar results to the extent possible. The point is to show how our columnar engine compares to generic columnar stores.

Here is a summary of the results:

| Query type | vs Mimir | vs Prometheus | vs ClickHouse † | |---|---|-------------------|-------------------| | Gauge average | up to 30x faster | up to 30x faster | up to 8x faster | | Counter rate | up to 30x faster | up to 30x faster | up to 3.5x faster | | Prefix filter on host name | up to 5x faster | up to 5x faster | up to 3x faster | | Gauge average with window | up to 25x faster | up to 25x faster | up to 4x faster |

†ClickHouse lacks native support for time series aggregations and counter reset detection.

Gauge average

We compared performance of evaluating the per-host hourly average of average memory utilization per time series, using the following queries:

# PromQL
avg by (host.name) (avg_over_time(system.memory.utilization[1h]))

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| STATS AVG(AVG_OVER_TIME(system.memory.utilization)) BY host.name, TBUCKET(1h)

Elasticsearch comfortably outperforms the other systems by up to 30x, in both low and high cardinality datasets:

Counter rate

We next compared performance of evaluating the per-host hourly average of cpu rate, using the following queries:

# PromQL
avg by (host.name) (rate(system.cpu.time[1h]))

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| STATS AVG(RATE(system.cpu.time)) BY host.name, TBUCKET(1h)

Despite processing data points per time series in order, counter rate performance matches calculating gauge average (the involved time series have 6.6x more docs than the query above). Elasticsearch maintains its wide advantage compared to the other systems and outperforms Mimir and Prometheus by 30x in the low cardinality dataset and by 16x in the high cardinality one:

It's really impressive that, for the high cardinality dataset, Elasticsearch is able to process 4 hours of data for half a million time series in less than 2 seconds, while the other systems take more than 30 seconds, leading to unresponsive dashboards for such queries. ClickHouse is also slower, despite having no logic to detect counter resets and extrapolate/interpolate deltas across buckets.

Prefix filter on host name

We next compared performance of filtering on host names based on their prefix, using the following queries:

# PromQL
avg by (host_name)
  (avg_over_time(system_cpu_load_average_1m{host_name=~"host-.*"}[5m]))

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| WHERE host.name LIKE "host-*"
| STATS AVG(AVG_OVER_TIME(system.cpu.load_average.1m)) BY host.name, TBUCKET(5m)

Elasticsearch manages to maintain an advantage of up to 5x compared to the other systems, despite replacing the inverted index on host.name with a doc value skipper:

Gauge average with window

We compared the performance of time series aggregations with a window of 90 minutes and time buckets of 30 minutes, using the following queries:

# PromQL
avg by (host.name) (avg_over_time(system.memory.utilization[90m]))&step=30m

# ES|QL
TS metrics-hostmetricsreceiver.otel-default
| STATS AVG(AVG_OVER_TIME(system.memory.utilization, 90m))
    BY host.name, TBUCKET(30m)

Elasticsearch comfortably outperforms the other systems in both low and high cardinality datasets:

Elasticsearch maintains an advantage that reaches 25x for the low cardinality dataset and 8x for the high cardinality one. ClickHouse is outperformed by close to 4x, denoting the efficiency of our approach for windowed query operations.

What's next for Elasticsearch metrics

Elasticsearch has been extended with metrics storage and processing capabilities that outperform Prometheus, Mimir, and ClickHouse. We're making fast progress with supporting PromQL and Prometheus remote write, also available as tech preview in version 9.4. These extensions enable users familiar with Prometheus and relevant systems to switch their applications to Elasticsearch — no need to migrate existing dashboards. Since Prometheus integration reuses the same storage and query engine that has been presented in this article, the same performance wins are also expected for Prometheus. Furthermore, collected metrics can be queried with PromQL and ES|QL, side-by-side or in ES|QL query pipelines, further boosting the analytics capabilities far beyond what was conceivable so far with Prometheus-based systems.

The improvements in storage efficiency, indexing throughput and query performance are already impressive, but we're not done. We'll be introducing more refinements to the codec for time series data, further reducing bytes per data point. Batch processing of ingested metrics will be further improved, reducing synchronization overhead and redundant processing layers that are not needed for well-formatted collected metrics. We're also planning to make wider use of doc value skippers, storing pre-computed aggregates like sum and count per block of values, to shortcut data point loading and processing where applicable, as well as use more cpu-friendly partitioning and grouping operations.

Frequently asked questions

What is a columnar metrics engine and why does it matter? A columnar metrics engine stores each field in its own file rather than row-by-row, then processes queries by reading only the columns needed. For time series data, this means Elasticsearch can decode metric values, dimension fields, and timestamps independently, applying vectorized operations across each column. The result is faster aggregations and lower storage overhead compared to row-oriented stores.

How does Elasticsearch compare to Prometheus for time series metrics storage and querying? Elasticsearch stores OTel metrics at 3.75 bytes per data point in version 9.4, roughly 2.5x less than Prometheus. For queries, Elasticsearch outperforms Prometheus and Mimir by up to 30x in gauge average and counter rate benchmarks. For the high-cardinality dataset (1.4M time series), Elasticsearch processes 4 hours of data in under 2 seconds while Prometheus takes over 30 seconds.

What is Elasticsearch TSDS and when should I use it? TSDS (time-series data streams) is Elasticsearch's storage format for metrics and time series data. It sorts documents by time series identifier (_tsid) and timestamp, stores fields in columnar doc values, and uses specialized codecs for compression. Use TSDS for any metrics workload, particularly OpenTelemetry or Prometheus data, where storage efficiency and query speed matter.

What is the TS source command in ES|QL? TS is an ES|QL source command, GA in version 9.4, that executes time series queries using a two-level model: an inner aggregation per time series (such as RATE() or AVG_OVER_TIME()), then an outer aggregation over the results. The compute engine processes data in time series sort order, enabling vectorized and parallel execution. Example: TS metrics | STATS AVG(RATE(cpu.time)) BY host.name, TBUCKET(1h).

How did Elasticsearch go from 25 bytes to 3.75 bytes per OTel data point? Four storage changes contributed across versions 9.1 through 9.4: replacing inverted indices with doc value skippers (-10 bytes), enabling synthetic IDs (-5 bytes), trimming sequence numbers (-4 bytes), and increasing codec block size from 128 to 512 elements (-2 bytes). The result is a 6.7x reduction in storage footprint in roughly one year.

Can Elasticsearch replace Prometheus without migrating dashboards? Elasticsearch supports Prometheus remote write (tech preview, version 9.4) and PromQL queries (tech preview, version 9.4). Existing Grafana dashboards using PromQL can point to Elasticsearch with minor modifications, and we expect to offer a seamless migration experience when our Prometheus offering reaches GA. The same TSDS storage and ES|QL compute engine power both PromQL and ES|QL queries, so the performance improvements apply to both.

What are doc value skippers and why do they matter for metrics? Doc value skippers are Lucene index structures that store min/max values for blocks of documents. For TSDS, which sorts by _tsid and timestamp, they replace inverted indices on dimension fields and @timestamp. This reduces storage by up to 10 bytes per data point and cuts indexing CPU by about 10%, with no measured regression in query performance for time range and dimension filters.

When cluster CPU spikes and you need to find which workload is responsible, narrow from fleet to namespace to pod, one step at a time.

What you need

• An Observability serverless project or a self-managed or Elastic Cloud Hosted stack at version 9.4 or later, where PromQL is available as a preview query language for metrics.
• Kubernetes metrics flowing into Elasticsearch. For this exercise we have considered OpenTelemetry data.
• One or more clusters with workloads running so group by queries have something to compare.

The scenario

You manage a fleet of Kubernetes clusters:

| Cluster | Region | Role | |---------|--------|------| | prod-us-east-1 | US East | Production: services, ML training | | prod-eu-west-1 | EU West | Production: regional web tier, cache | | staging-us-east-1 | US East | Staging: QA, integration tests | | dev-sandbox | US East | Developer sandbox |

The production cluster in US East runs a mix of services and ML training jobs across several namespaces.

An alert fires: cluster-wide CPU is elevated, but only one team is complaining about slower response times.

You are triaging which cluster, then which namespace, then which pod.

You are not after a full root-cause proof in one query, but enough to name the suspect and hand off.

Your data

The OpenTelemetry Collector's Kubelet Stats Receiver populates data streams like metrics-kubeletstatsreceiver.otel-default. Metrics follow the k8s.* naming convention (for example k8s.pod.cpu.usage) and labels like k8s.cluster.name or k8s.namespace.name let you slice by cluster, namespace, or pod.

To verify the data is there, open Discover, switch to ES|QL mode, run TS metrics-*, and scope the query with WHERE data_stream.dataset == "kubeletstatsreceiver.otel".

Investigation: find the noisy neighbor

Step 1: Which cluster is hot?

When you manage multiple clusters, start at the fleet level.

PROMQL sum by (k8s.cluster.name) (k8s.pod.cpu.usage)

This groups total pod CPU by cluster.

prod-us-east-1 immediately stands out: total pod CPU is an order of magnitude higher than the other clusters.

The EU production cluster, staging, and dev-sandbox are all quiet.

Now you know where the problem is, time to zoom in.

Step 2: Overall CPU in the hot cluster

Filter to prod-us-east-1 and look at total CPU:

PROMQL sum(k8s.pod.cpu.usage{k8s.cluster.name="prod-us-east-1"})

This gives you the cluster-wide pod CPU footprint over time.

If the total is climbing or spiking, something changed, but you don't yet know what.

Step 3: Break down by namespace

The fastest way to isolate which team is responsible: group by namespace.

PROMQL sum by (k8s.namespace.name) (k8s.pod.cpu.usage{k8s.cluster.name="prod-us-east-1"})

Set the time picker in Kibana to cover your incident window.

ml-training dominates at ~2.0 cores while every other namespace stays well below 0.2 cores.

Step 4: Drill down to the pod

Now that you know the namespace, identify the specific pod:

PROMQL sum by (k8s.pod.name) (k8s.pod.cpu.usage{k8s.cluster.name="prod-us-east-1", k8s.namespace.name="ml-training"})

That ranks pods in the namespace by total CPU.

The chart should make the outlier obvious.

Pod model-train-v2-run-47-d9j67 is consuming the full 2.0 cores. It is a training job saturating its allocation.

Step 5: Check resource utilization ratios

Raw CPU cores tell you how much. Utilization ratios tell you how close to limits.

A pod hitting 100% of its CPU limit is being throttled, and it is both the noisy neighbor and a victim of its own limits.

PROMQL sum by (k8s.namespace.name) (k8s.container.cpu_limit_utilization{k8s.cluster.name="prod-us-east-1"})

ml-training shows ~100% CPU limit utilization (pegged at the 2-core limit), while the other namespaces stay under 20%.

This confirms the training job is saturating its allocation and likely causing scheduling pressure on the shared node.

What happens next

The PromQL query named the suspect: the training job model-train-v2-run-47 in ml-training.

From here:

• Logs: Filter by the pod name in Discover to see what the training job was doing and whether it logged errors or warnings.
• Kube events: Check for OOMKilled, throttling, or eviction events in the same time window.
• Resource policies: Review whether the training job's requests and limits match its actual usage. A large gap between request and limit lets a pod burst past what the scheduler planned for. Consider ResourceQuota or LimitRange on the namespace.

Migrations covered by the Observability Migration Platform

The current scope covers Datadog and Grafana. The platform can work from exported assets or live APIs, and it focuses on dashboards and alerting content on the Datadog and Grafana paths it currently covers.

Support is not identical across the two sources. Datadog has end-to-end extraction, validation, compile, upload, smoke, and verification workflows, but it currently covers a narrower slice of widgets and monitors. Grafana coverage is broader. The platform provides a practical translation pipeline for the supported paths.

The screenshots below show examples of dashboards after migration.

How the Observability Migration Platform works

At a high level, the workflow has two halves: source-aware translation on the way in and target-aware validation and delivery on the way out. That split matters because Grafana and Datadog differ not only in JSON shape, but also in query languages, panel types, controls, and alerting models.

A run starts with exported assets or live source APIs. From there, the workflow normalizes source-specific objects, chooses a translation path for each supported dashboard, panel, and alerting artifact, and emits Kibana-native output. This is where most of the source-specific logic lives: translating queries or Datadog formulas, mapping panel semantics, carrying forward controls and links where possible, and deciding when an exact translation is not the right answer.

The second half is target-aware. The emitted output can be validated against an Elastic target, compiled, and uploaded to Kibana through the shared runtime. In the happy path, that yields a working translated dashboard. In rougher cases, validation may show that a panel cannot run safely as emitted. When that happens, the workflow is designed to fail conservatively: it can mark the panel for manual review or replace it with an upload-safe placeholder instead of shipping a broken runtime panel.

Just as important, the outcome is not simply "a dashboard showed up in Kibana." The workflow also produces reviewer-facing evidence such as a migration report, manifest, verification packets, and rollout plan so you can see what translated cleanly, what was downgraded or manualized, and what still needs human judgment. Those artifacts are what make the process operationally credible: they give teams something concrete to inspect, compare, and act on.

Running the migration

The platform is CLI-driven, and a good fit for migration work that needs to be repeatable, reviewable, and easy to automate. Users can start with a representative slice of dashboards and alerting content from Grafana or Datadog, point the workflow at an Elastic target, and use that first run to understand translation quality, validation results, and how much follow-up review is required.

To run the full path against Elastic, create an Elastic Observability Serverless project, generate a Serverless project API key, and point the CLI at your Elasticsearch and Kibana endpoints:

obs-migrate migrate \
  --source grafana \
  --input-mode files \
  --input-dir ./grafana_exports \
  --output-dir ./migration_output \
  --assets all \
  --native-promql \
  --data-view "metrics-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload

The run validates the emitted queries against Elastic, compiles the generated dashboards, uploads them to Kibana, and produces the standard migration artifacts for review.

A typical run looks like this:

1. Start with exported assets or live source APIs from Grafana or Datadog.
2. Choose the asset scope with --assets dashboards, --assets alerts, or --assets all.
3. Translate the supported dashboards, queries, controls, and alerting artifacts into Kibana-native output.
4. Validate the emitted content against an Elastic target (if configured), then compile and upload the translated dashboards for dashboard-capable runs.
5. Review the migration evidence, including migration_report.json, verification_packets.json, run_summary.json, etc., to understand what translated cleanly, where semantic gaps remain, and which dashboards, panels, or alert rules still require human review.
6. If alert rule creation is enabled, review the migrated rules (which are disabled by default) in Kibana before deciding which ones to enable or redesign.

What's next

The platform is still evolving, and will continue to gain depth and self-service capabilities. The biggest open areas are stronger measured source-to-target semantic verification, further coverage for Datadog, deeper coverage for harder query families and non-dashboard surfaces, and cleaner shared runtime contracts across the workflow.

It is also built to grow over time. The source and target boundaries are explicit by design, which gives the platform room to expand coverage and support additional source paths in the future.

In conclusion

If you are planning a move into Elastic, a good starting point is to create an Elastic Observability Serverless project. That gives you the target environment where translated dashboards and alerting content can be validated and reviewed.

To learn more about the migration workflow, talk to your Elastic representative about current access, supported coverage, and how it can help with your migration needs.

The new Elastic Kubernetes integration is built for that operator. It includes dashboards with drilldowns, alert rule templates, and ML anomaly detection jobs. Additionally Elastic also offers Agentic Investigations, that drives investigations automatically.

This blog will cover the foundational observability components (dashboards, drilldowns, alert templates, etc), while a part 2 covering the agentic investigations will cover workflows, agent skills, and MCP tools and views

The new Kubernetes integration content in this post is generally available across Elastic Cloud Hosted, Serverless, and self-managed deployments.

Dashboards designed for drill-down, not just display

The new Kubernetes dashboards are organized around a three-tier design: a cluster Overview that surfaces what needs attention at a glance, object summary pages for clusters, nodes, namespaces, workloads, and pods, and object detail pages that give you the full picture for any single entity.

Every layer connects to the next: click any entity in a summary table and choose: apply it as a filter on the current view, or open its dedicated detail page.

Here's what that looks like when something's actually wrong:

Following a restart cascade from overview to container

Overview: The Overview surfaces what needs attention across your cluster. You can see top pods by CPU, top namespaces by container restarts, and top nodes by memory utilization in one screen. When the "container restarts" panel starts climbing, you know where to look.

Namespaces Overview: Click into the flagged namespace with 1232 restarts and CPU limit utilization at 116%. The detail view plots CPU and memory against requests and limits over time. This shows both the size and duration of the overage.

Namespace Details: We can get more info on the various pods in this namespace here. Click the pod driving the restarts.

Pod Details: The pod detail dashboard is organized into capacity, metrics, and containers sections. Container restarts are flagged in red at the top of the page. Most panels are metric-driven, and the dashboard also links to correlated pod logs in Discover.

It takes four clicks to move from the Cluster Overview to container logs that explain the failure. These dashboards are starting points for your team. You can copy and customize them with ESQL visualizations.

Alert rules that fire on day one

The integration ships with pre-built alerting rule templates for states that are wrong by definition. No historical baseline or warmup period is required. Enable them during setup and they work immediately.

These rules do not ask, "Is this abnormal for this service?" They ask, "Is this a known bad state in Kubernetes?" A pod in CrashLoopBackOff is always a problem. A container killed by the kernel for exceeding its memory limit is always a problem.

Like the Kubernetes dashboards, these alerts are built on ES|QL queries. You can see that in the CrashLoopBackOff definition below. If you are new to ES|QL, you can start with the ES|QL docs.

The alert templates cover:

• CrashLoopBackOff detection - Fires when a pod's restart count exceeds a configurable threshold within a rolling window. The default catches a real restart cycle without triggering on routine restarts during a rolling deployment.
• Container OOMKilled - Surfaces kernel-level container terminations due to memory limits. These events are easy to miss in dashboards and often precede wider failures. This rule fires on any occurrence.
• Deployment below desired replicas - Fires when a deployment runs fewer replicas than declared for longer than a grace period. This catches scaling failures and partially failed rollouts.
• Pod stuck in Pending - Fires when a pod cannot be scheduled past a configurable time threshold. This surfaces node capacity problems, missing resources, and affinity failures before availability drops.
• Node disk pressure - Fires immediately when the Kubernetes DiskPressure node condition is True. A node condition is a direct state signal, not a statistical threshold.
• Persistent volume near capacity - Alerts when storage utilization crosses a configurable threshold before writes start failing.

Each template is parameterized. Adjust thresholds in the ES|QL query to match your environment. Connect notifications to PagerDuty, Slack, or another destination in your runbook.

Anomaly detection jobs with ML baselines

Alert rules catch what is definitively wrong. ML anomaly detection catches patterns that often precede failures. If you are new to this area, see the Elastic anomaly detection overview.

A pod that always runs at 85% memory utilization might be healthy. A pod that grew from 40% to 85% over twelve hours is usually not healthy. A static threshold often catches this only after an OOM kill. The ML module should catch the trajectory earlier.

The integration ships with ML module configurations that learn workload baselines and flag meaningful deviations. These jobs need 24 to 48 hours of data before results become useful. Results become more reliable as jobs continue to run.

The included modules

1. Pod memory growth anomalies

• What it learns: per-pod memory consumption pattern over time
• What it flags: Growth trajectories that are inconsistent with baseline behavior, such as a slow leak that never crosses the hard limit.
• Why ML (not alert rule): The alert rule catches the OOMKill after the fact. The ML job catches the trajectory that leads there.

2. Network I/O anomalies

• What it learns: per-pod network transmit/receive byte rate patterns
• What it flags: Unusual spikes or drops relative to the pod baseline. A spike can indicate a runaway process or unexpected load. A drop can indicate a network partition that causes the pod to go idle.
• Why ML (not alert rule): Normal network traffic varies by time of day and workload type. A batch job pod at high throughput during its normal window is expected. The same throughput outside that window can be anomalous.

3. Pod restart frequency

• What it learns: Per-workload restart rate patterns during deployments, scaling events, and routine operations.
• What it flags: Restart patterns that are anomalous relative to each workload's own history. This is distinct from the CrashLoopBackOff alert rule, which fires on a fixed threshold regardless of context.
• Why ML (not alert rule): A deployment that restarts twice during every rollout can be healthy. The same deployment restarting twice on a Tuesday afternoon may be unhealthy. The alert rule cannot distinguish these cases without workload history.

Here's our Single Metric Viewer showing anomalies triggered against a specific pod, for the memory growth job:

And here's the multi-series Anomaly Explorer view of the same job, showing detections firing across a variety of pods:

Try it yourself: the OTel Astronomy Shop

If you do not have a Kubernetes cluster ready, you can use the OpenTelemetry Astronomy Shop demo environment. It uses the same commands as Getting Started Step 2, Path A, but points to demo services. Create the namespace and secret, then run the Helm install. All 16 services, Kafka, and PostgreSQL start flowing into Elastic without instrumentation changes.

The demo ships with a built-in feature flag service, flagd, that lets you activate failure scenarios. Enable cartServiceFailure and watch the checkout-service restart cascade unfold in real time. The CrashLoopBackOff alert rule fires. The ML modules begin establishing baselines. If you have the investigation workflow enabled, it runs automatically when the alert fires.

Getting started

Step 1 - Install the Kubernetes integration. Dashboards are available immediately. No additional configuration is required.

Step 2 - Deploy data collection. There are two supported paths, both based on Helm. Choose the one that fits your deployment model.

Path A - OpenTelemetry (EDOT collector): This path uses the opentelemetry-kube-stack Helm chart with the Elastic Distribution of OpenTelemetry (EDOT) collector. Create a namespace and a secret with your endpoint and API key, then install:

kubectl create namespace opentelemetry-operator-system

kubectl create secret generic elastic-secret-otel \
  --namespace opentelemetry-operator-system \
  --from-literal=elastic_otlp_endpoint='https://<your-endpoint>.elastic.cloud:443' \
  --from-literal=elastic_api_key='<your-api-key>'

helm upgrade --install opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack \
  --namespace opentelemetry-operator-system \
  --values 'https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v9.3.2/deploy/helm/edot-collector/kube-stack/managed_otlp/values.yaml' \
  --version '0.12.4'

Path B - Elastic Agent (standalone): This path uses the elastic/elastic-agent Helm chart. The default manifest includes resource limits that may not be appropriate for production. Review the Scaling Elastic Agent on Kubernetes guide before deploying.

helm repo add elastic https://helm.elastic.co/ && \
helm install elastic-agent elastic/elastic-agent \
  --version 9.3.2 \
  -n kube-system \
  --set outputs.default.url=https://<your-endpoint>.es.elastic.cloud:443 \
  --set outputs.default.type=ESPlainAuthAPI \
  --set outputs.default.api_key=$(echo "<your-base64-api-key>" | base64 -d) \
  --set kubernetes.enabled=true

Step 3 - Enable the alert rule templates. Go to Observability > Alerts in Kibana. The Kubernetes templates are in the rule library. Enable the templates relevant to your environment, set thresholds, and connect your notification channel.

Step 4 - Let the ML modules warm up. After 24 to 48 hours, anomaly detection modules establish baselines and begin surfacing pattern-based deviations. Longer running jobs usually produce better baselines. Find results in the ML Anomaly Explorer, linked from the Kubernetes dashboards.

Steps 5, 6, and 7 - Agentic content will be covered in Part 2 (forthcoming), Kubernetes observability with Elastic: Agentic Investigations.

What's next

The next step is the layer that runs investigation workflows when an alert fires. That includes skills that encode investigation logic, tools that expose facts like ML state and topology, and MCP apps that render outputs in places like Claude Desktop or VS Code. These technical preview capabilities are available today and will be covered in Part 2 (forthcoming), Kubernetes observability with Elastic: Agentic Investigations.

If you are running Kubernetes on Elastic today, tell us which investigation steps you repeat manually on every incident. Tell us which remediations you would trust a workflow to propose. You can join the Elastic Community Discussion here.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Running PromQL queries in Kibana

In the ES|QL editor in Kibana, enter the PROMQL command, and type your PromQL in that block. PROMQL marks that segment so Elasticsearch parses it as PromQL inside the wider ES|QL request Kibana sends.

What you can query

Here are a few patterns to get started.

Raw metric

PROMQL container.cpu.usage

Average across all containers

PROMQL avg(container.cpu.usage)

rate() on a counter

PROMQL rate(docker.network.inbound.bytes)

Aggregated rate

PROMQL sum(rate(docker.network.inbound.bytes))

Group by a label

PROMQL sum by (agent.id) (rate(docker.network.inbound.bytes))

You may notice that none of these examples include start, end, step, or a lookback window on every rate(). Those parameters are optional: the time picker and Kibana defaults handle most of it for you.

Optionally, you can include the data stream name using the index= parameter. For example: PROMQL index=metrics-docker.cpu-default container.cpu.usage. Adding the parameter helps narrow down the scope of what data the query scans.

The current release of PromQL tech preview has over 80% query coverage benchmarked against top Grafana dashboards. Advanced modifiers and specific functions are in consideration for future releases.

Find your streams and metric names

If you have existing PromQL queries, you can use them directly in the PROMQL command without changes. If you are writing a query from scratch and need to find the exact field names, run TS metrics-* in Discover to see every metrics data stream. Each metric appears as a small chart so you can tell at a glance what is active. Hover over a metric and click the "View details" action to see the field name and the data stream it belongs to.

For a deeper walkthrough, see Explore metrics data with Discover in Kibana.

Time picker and query time handling

The time picker in Kibana sets the time window for the query. Dashboard panels and Alerting rules work the same way using their own time range, so you do not need to write start= or end= in the query itself.

Step is the gap between two consecutive data points on the chart. A smaller step means more data points across the same span. If you do not set step= or buckets=, the default is buckets=100. You can set step= to a fixed width such as 1m, or set buckets= to a different target maximum number of data points.

Discover and Dashboards

In Discover, switch to ES|QL mode and run your PROMQL query so you can see how the metric behaves over the range you pick, as a time-series chart. When you want to save that visualization, choose "Save visualization to dashboard" and add it to a new or existing dashboard.

Or go to Dashboards directly: add a panel, choose ES|QL, and write your PROMQL query.

Alerting

You can create alert rules using PromQL. Go to Alerts, open Manage rules, and create a rule. Search for Elasticsearch query and select it. Choose ES|QL as the query type.

Write your PROMQL query, but assign the metric to a variable so you can use it in a WHERE clause for the alert condition:

PROMQL metric_value=(sum by (agent.id) (rate(docker.network.inbound.bytes)))
| WHERE metric_value >= 500

Select @timestamp for the time field and continue defining the rest of the rule configuration.

Try it

1. Open an Observability project on Elastic Cloud Serverless, or use Elastic Stack 9.4.
2. Write your query: in the ES|QL editor in Kibana, run your PromQL via PROMQL. You can also go to Dashboards, add a panel, choose ES|QL, and write the query there.
3. If you are writing from scratch and need to find metric names, run TS metrics-* in Discover (see "Find your streams and metric names" above).
4. Check the results and adapt the query if needed.

PromQL support in Elasticsearch and Kibana will continue to evolve. Follow the Observability Labs feed for follow-up posts as coverage and ergonomics improve.

This lets you bring your Prometheus metrics into the same cluster which can also store logs, traces, and other data. One storage backend, one set of access controls, one place to query.

Why store Prometheus metrics in Elasticsearch?

Prometheus local storage is designed for short retention, typically 15 to 30 days. For anything beyond that, you need a remote storage backend.

Elasticsearch's time series data streams (TSDS) are built for highly efficient long term metrics storage: automatic rollover, time-based partitioning, compression via index sorting, and downsampling to reduce storage costs as data ages. Your Prometheus scrape configs stay the same.

Recent Elasticsearch releases have significantly reduced the storage footprint for metrics. A dedicated post with the numbers is coming soon.

On the query side, ES|QL embraces PromQL: a built-in PROMQL function lets your existing queries run unchanged, while the rest of ES|QL is available when you want joins, aggregations, or transformations that span multiple datasets.

And because metrics land in the same store as your logs, traces, and profiling data, correlating signals across types becomes a single query rather than a cross-system investigation.

How it works

For a detailed look at what happens inside Elasticsearch when a Remote Write request arrives — protobuf parsing, metric type inference, TSDS mapping, and data stream routing — see How Prometheus Remote Write Ingestion Works in Elasticsearch.

Prometheus sends metrics to Elasticsearch via the standard Remote Write protocol (v1). The endpoint accepts protobuf-encoded, snappy-compressed WriteRequest payloads.

Each sample becomes an Elasticsearch document in a pre-defined time series data stream. Prometheus labels become TSDS dimensions. The metric value is stored in a typed field under metrics.<metric_name>.

Elasticsearch infers the metric type (counter vs gauge) from naming conventions. Names ending in _total, _sum, _count, or _bucket are treated as counters. Everything else is treated as a gauge.

Setting it up

Step 1: Get an Elasticsearch endpoint

You need an Elasticsearch cluster with the Prometheus endpoints enabled. The simplest option is Elastic Cloud Serverless, where this works out of the box.

For serverless: sign in to cloud.elastic.co, create an Observability project, and copy the Elasticsearch endpoint from the project settings page. The endpoint looks like https://<project-id>.es.<region>.<provider>.elastic.cloud.

Step 2: Create an API key

Create an API key scoped to writing metrics data streams only. In your Elastic Cloud Serverless project, go to Admin and settings (the gear icon at the bottom left of the side nav), then API keys.

Use the following role descriptor in the Control security privileges section:

{
  "ingest": {
    "indices": [
      {
        "names": ["metrics-*"],
        "privileges": ["auto_configure", "create_doc"]
      }
    ]
  }
}

Copy the key value before closing the dialog. You will not be able to retrieve it again.

Step 3: Configure Prometheus

Add the following remote_write block to your prometheus.yml:

remote_write:
  - url: "https://YOUR_ES_ENDPOINT/_prometheus/api/v1/write"
    authorization:
      type: ApiKey
      credentials: YOUR_API_KEY

That's it. Prometheus will start shipping metrics to Elasticsearch on the next scrape interval.

If you use Grafana Alloy instead of Prometheus, the equivalent configuration is:

prometheus.remote_write "elasticsearch" {
  endpoint {
    url = "https://YOUR_ES_ENDPOINT/_prometheus/api/v1/write"
    headers = {"Authorization" = "ApiKey YOUR_API_KEY"}
  }
}

Routing metrics to separate data streams

By default, all metrics land in metrics-generic.prometheus-default. You can route metrics from different environments or teams into separate data streams using the dataset and namespace path segments in the URL.

The three URL patterns are:

• /_prometheus/api/v1/write routes to metrics-generic.prometheus-default
• /_prometheus/metrics/{dataset}/api/v1/write routes to metrics-{dataset}.prometheus-default
• /_prometheus/metrics/{dataset}/{namespace}/api/v1/write routes to metrics-{dataset}.prometheus-{namespace}

For example, using /_prometheus/metrics/infrastructure/production/api/v1/write routes data to metrics-infrastructure.prometheus-production.

This is useful for separating production from staging metrics, or giving different teams their own data streams with independent lifecycle policies.

What gets stored

Here is what a sample document looks like in Elasticsearch:

{
  "@timestamp": "2026-04-02T10:30:00.000Z",
  "data_stream": {
    "type": "metrics",
    "dataset": "generic.prometheus",
    "namespace": "default"
  },
  "labels": {
    "__name__": "prometheus_http_requests_total",
    "handler": "/api/v1/query",
    "code": "200",
    "instance": "localhost:9090",
    "job": "prometheus"
  },
  "metrics": {
    "prometheus_http_requests_total": 42
  }
}

Labels map to keyword fields that serve as TSDS dimensions. The metric value is stored under metrics.<metric_name> with the inferred time_series_metric type (counter or gauge).

Elasticsearch installs a built-in index template matching metrics-*.prometheus-* that configures TSDS mode, passthrough dimension container objects, and a 10,000 field limit. The field limit is configurable via a custom component template (see the custom metric type inference section below for how to use one). You do not need to create any templates or mappings yourself.

Custom metric type inference

Metric type inference is based on naming conventions. Metrics that don't follow Prometheus naming best practices may be classified incorrectly. You can override the defaults by creating a metrics-prometheus@custom component template with your own dynamic templates. For example, to mark all *_counter metrics as counters:

{
  "template": {
    "mappings": {
      "dynamic_templates": [
        {
          "counter": {
            "path_match": "metrics.*_counter",
            "mapping": {
              "type": "double",
              "time_series_metric": "counter"
            }
          }
        }
      ]
    }
  }
}

Custom rules are merged with the built-in patterns, so the defaults still apply for metrics you don't override.

Current limitations

Only Remote Write v1 is supported. v2, which brings native histograms and exemplars, is planned.

Staleness markers (special NaN values Prometheus uses to signal a series has disappeared) are not yet stored or respected in queries.

Non-finite values (NaN, Infinity) are silently dropped.

Get started

The Prometheus Remote Write endpoint is available now on Elasticsearch Serverless with no configuration needed. To get started with a local cluster, start-local gets you a single-node cluster in minutes.

Once metrics are flowing, you can query them with ES|QL using the built-in PROMQL function for PromQL compatibility, or write native ES|QL queries to join metrics with logs and traces in the same store.

This post looks at what happens inside Elasticsearch when a Remote Write request arrives.

If you want to understand the implementation, evaluate how Elasticsearch compares to other Prometheus-compatible backends, or contribute, this is the post for you. A companion post, Ship Prometheus Metrics to Elasticsearch with Remote Write, covers the setup and configuration side.

Request lifecycle: from HTTP to indexed documents

A quick note on the Prometheus data model before we dive in: Prometheus stores all metric values as 64-bit floats and treats the metric name as just another label (__name__). The storage engine itself is agnostic of whether a value is a counter or a gauge. Keep this in mind as we walk through how Elasticsearch maps these concepts.

Here is the full path of a Remote Write request through Elasticsearch:

1. HTTP layer — The endpoint receives a compressed protobuf payload, checks indexing pressure, decompresses with Snappy, and parses the protobuf WriteRequest.
2. Document construction — Each sample in each time series becomes an Elasticsearch document with @timestamp, labels.*, and metrics.* fields.
3. Bulk indexing — All documents from a single request are written to the target data stream via a single bulk call.

The sections below walk through each stage in detail.

HTTP layer

The endpoint accepts application/x-protobuf POST requests. The incoming request body is tracked against the same indexing pressure limits that protect the bulk indexing API. If the cluster is already under heavy indexing load, the request gets rejected with a 429 before any parsing happens.

Prometheus compresses Remote Write payloads with Snappy. Elasticsearch decompresses the body in a streaming fashion without materializing it into a single contiguous allocation, and validates the declared uncompressed size against a configurable maximum to guard against decompression bombs.

The decompressed body is then deserialized as a protobuf WriteRequest. Each WriteRequest contains a list of TimeSeries entries, and each TimeSeries contains a set of labels (key-value pairs) and a list of samples (timestamp + float64 value).

Document construction

For each sample in each time series, Elasticsearch builds an index request. Here is what a single document looks like:

{
  "@timestamp": "2026-04-01T12:00:00.000Z",
  "data_stream": {
    "type": "metrics",
    "dataset": "generic.prometheus",
    "namespace": "default"
  },
  "labels": {
    "__name__": "http_requests_total",
    "job": "prometheus",
    "instance": "localhost:9090",
    "method": "GET",
    "status": "200"
  },
  "metrics": {
    "http_requests_total": 1027.0
  }
}

All labels from the Prometheus time series (including __name__) end up in the labels.* fields. The metric value goes into metrics.<metric_name>, where <metric_name> is the value of the __name__ label.

Time series without a __name__ label are dropped entirely, and the samples are counted as failures. Non-finite values (NaN, Infinity, negative Infinity) are silently skipped. This includes Prometheus staleness markers, which use a special NaN bit pattern (0x7ff0000000000002) to signal that a series has disappeared.

One sample, one document

You might wonder whether storing each individual sample as its own document creates significant storage overhead, especially for labels. A common pattern to reduce that overhead was to group all metrics sharing the same labels and timestamp into a single document.

With recent TSDB improvements, that optimization is no longer necessary. Elasticsearch has trimmed the per-document storage overhead to the point where there is negligible difference between packing many metrics in a single document and writing each sample separately. A dedicated post covering these TSDB storage improvements in detail is coming soon.

Bulk indexing

All documents from a single Remote Write request are sent to Elasticsearch via a single bulk request. Each document targets the data stream metrics-{dataset}.prometheus-{namespace} and is indexed as an append-only create operation.

Metric type inference

Remote Write v1 does not reliably transmit metric types alongside samples. Prometheus sends metadata (type, help text, unit) in separate requests roughly once per minute, and those requests may land on a different node than the samples. Buffering samples until metadata arrives is not practical in a distributed system, so Elasticsearch infers the type from naming conventions instead.

Metric names ending in _total, _sum, _count, or _bucket are mapped as counters. Everything else defaults to gauge. This is a well-established convention that other Prometheus-compatible backends use as well.

http_requests_total             → counter
request_duration_seconds_sum    → counter
request_duration_seconds_count  → counter
request_duration_seconds_bucket → counter
process_resident_memory_bytes   → gauge
go_goroutines                   → gauge

The heuristic can be wrong. A metric like temperature_total (if someone named a gauge that way) would be misclassified as a counter. The main consequence today is that some ES|QL functions like rate() require the metric type to be a counter and will reject a misclassified gauge. For PromQL, we plan to lift this restriction so that rate() works regardless of the declared type, which will make incorrect inference less consequential.

You can override the inference by creating a metrics-prometheus@custom component template with custom dynamic templates. For example, to treat all *_counter fields as counters:

PUT /_component_template/metrics-prometheus@custom
{
  "template": {
    "mappings": {
      "dynamic_templates": [
        {
          "counter": {
            "path_match": "metrics.*_counter",
            "mapping": {
              "type": "double",
              "time_series_metric": "counter"
            }
          }
        }
      ]
    }
  }
}

Custom dynamic templates are merged with the built-in ones, so the default naming-convention rules still apply for metrics you don't explicitly override.

The index template

Elasticsearch installs a built-in index template that matches metrics-*.prometheus-*. This template is what makes field type inference work without manual mapping configuration.

TSDS mode is enabled, which gives you time-based partitioning, optimized storage, deduplication, and the ability to downsample data as it ages.

Passthrough object fields are used for both the labels and metrics namespaces. This serves three purposes:

1. Namespace isolation: Labels and metrics live in separate object namespaces (labels.* and metrics.*), so a label named status and a metric named status cannot conflict with each other.

2. Dimension identification: The labels passthrough object is configured with time_series_dimension: true, which means every field under labels.* is automatically treated as a TSDS dimension. When Prometheus sends a time series with a label you have never seen before, it becomes a dimension without any explicit field mapping.

3. Transparent queries: You don't need to write the labels. or metrics. prefix in ES|QL or PromQL. A query can reference job instead of labels.job, or http_requests_total instead of metrics.http_requests_total. The passthrough mapping handles the resolution.

Dynamic inference for metrics applies the naming-convention heuristics described above. When a new metric name appears for the first time, its field mapping is created automatically under metrics.* with the correct time_series_metric annotation.

Failure store is enabled. Documents that fail indexing (for example, due to a mapping conflict where the same metric name appears with incompatible types) are routed to a separate failure store instead of being dropped silently.

Data stream routing

The three URL patterns map directly to data stream names:

| URL pattern | Data stream | |---|---| | /_prometheus/api/v1/write | metrics-generic.prometheus-default | | /_prometheus/metrics/{dataset}/api/v1/write | metrics-{dataset}.prometheus-default | | /_prometheus/metrics/{dataset}/{namespace}/api/v1/write | metrics-{dataset}.prometheus-{namespace} |

This lets you separate metrics from different Prometheus instances or environments into different data streams. That separation is useful for a few reasons.

Lifecycle isolation: you can apply different retention policies per data stream. Production metrics might be kept for 90 days, while dev metrics might expire after 7 days.

Access control: you can scope API keys to specific data streams. A team's Prometheus instance writes to metrics-teamA.prometheus-prod, and their API key only has access to that stream.

Query performance: PromQL queries and Grafana dashboards can be scoped to a specific index pattern, avoiding scans of unrelated data.

Error handling and the Remote Write spec

The Remote Write spec defines two response classes: retryable (5xx, 429) and non-retryable (4xx). Prometheus uses this distinction to decide whether to retry or drop a failed request.

Elasticsearch returns 429 (Too Many Requests) if any sample in the bulk request was rejected due to indexing pressure. This signals Prometheus to back off and retry with exponential backoff.

For partial failures (some samples indexed, others rejected), the response includes a summary. It reports how many samples failed, grouped by target index and status code, along with a sample error message from each group.

Time series without a __name__ label result in a 400 error for those samples. Non-finite values (NaN, Infinity) are silently dropped: Prometheus receives a success response and will not retry.

NaN appears most commonly for summary quantiles when no observations have been recorded (for example, a p99 latency metric before any requests arrive) and for staleness markers. The practical impact of dropping these is limited today: for most queries, a missing sample behaves similarly to a NaN one, since PromQL's lookback window fills the gap with the last known value either way. The more significant gap is staleness markers, which are covered below.

What's next: Remote Write v2 and beyond

Remote Write v2 is still experimental, which is why the current implementation starts with v1. But v2 addresses several of v1's shortcomings.

Metadata alongside samples: v2 sends metric type, unit, and description with each time series in the same request. This eliminates the need for naming-convention heuristics entirely.

Native histograms: v2 supports Prometheus native histograms, which map naturally to Elasticsearch's exponential_histogram field type. Classic histograms (one counter per bucket boundary) are verbose and lose precision at query time. Native histograms are more compact and more accurate.

Dictionary encoding: v2 replaces repeated label strings with integer references, reducing payload size significantly for high-cardinality label sets.

Created timestamps: counters in v2 include a "created" timestamp that marks when the counter was initialized. This allows backends to detect counter resets more accurately than the current heuristic (value decreased since last sample).

Beyond v2, there are two other items in consideration for future enhancements.

Staleness marker support: currently, staleness markers (the special NaN that Prometheus writes when a scrape target disappears) are dropped. Supporting them would allow correct PromQL lookback behavior and avoid the 5-minute "trailing data" artifact where a disappeared series still appears in query results.

Shared metric field: the current layout creates a separate field for each metric name (metrics.http_requests_total, metrics.go_goroutines, etc.). This works, but it means the number of field mappings grows with the number of distinct metric names, which is why the field limit is set to 10,000 for Prometheus data streams. A different approach we're considering is to store the metric name only in the __name__ label and write the metric value to a single shared field. This eliminates the field explosion problem entirely and more closely matches how Prometheus stores data internally. This direction is part of the broader effort to make Elasticsearch's metrics storage more efficient and more compatible with Prometheus conventions.

Availability

The Prometheus Remote Write endpoint is available now on Elasticsearch Serverless with no additional configuration.

For self-managed clusters, check out start-local to get up and running quickly.

If you run into issues or have feedback, open an issue on the Elasticsearch repository.

EDOT is now generally available. The collector, the SDKs, the whole stack — production-ready, enterprise-supported, no asterisks.

But here's the thing: getting your data into Elastic is only half the job. The harder half, in practice, is what happens after. Someone still has to build the dashboards, write the alert rules, and figure out which SLOs are worth tracking — before any of it is useful.

That gap is what OpenTelemetry Content Packages are designed to close.

What Are OpenTelemetry Content Packages?

Elastic's traditional Beats-based integrations always bundled data collection and visualizations together — you got curated dashboards and alerts the moment you turned something on. As Elastic moves to an OpenTelemetry-first world, that same philosophy carries over, but the model is cleaner.

OpenTelemetry Content Packs are purely about the observability assets for a given service. No data collection config is bundled in, because in an OTel world, the collector handles that. Each package contains:

• Dashboards — curated, pre-built Kibana visualizations tailored to the service being monitored
• Alert rules — pre-configured alerting rules that fire on meaningful thresholds, helping teams minimize Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR)
• SLO templates — ready-made Service Level Objective definitions you can apply immediately to track reliability targets, error budgets, and burn rates

More asset types are planned for future packages as the content pack model continues to evolve.

How Does It Work?

The core idea is simple: as soon as data arrives in Elastic, the right dashboards, alert rules, and SLO templates are ready to use. The content package activates based on the incoming data, regardless of how that data was collected.

One of the most powerful aspects of this system is automatic installation. When Elastic detects that data for a particular service has started arriving in Elasticsearch, the corresponding content pack is installed automatically — no manual steps, no hunting through the integrations catalog. By the time you open Kibana, your dashboards are already there waiting for you, your alert rules are ready to be enabled, and your SLO templates are pre-loaded.

To get the data flowing in the first place, we need to configure the collector — a YAML file that defines the building blocks of your telemetry pipeline:

• Receivers — define what data to collect and from where. Each service has its own receiver (for example, the MySQL receiver scrapes metrics directly from the database).
• Exporters — define where the collected data is sent. In our case, we use the Elasticsearch exporter, which ships the telemetry data directly into Elasticsearch in OpenTelemetry native format.
• Pipelines — wire the receivers and exporters together, defining the flow of data through the collector.

Once this configuration is in place and the collector is running, data starts flowing into Elasticsearch — and the content pack takes it from there.

Data Sources

OpenTelemetry data can reach Elastic through any of the following:

• EDOT Collector — the Elastic Distribution of the OpenTelemetry Collector, embedded in or used alongside the Elastic Agent
• Upstream OTel Collector — the standard community OpenTelemetry Collector (Contrib or custom builds)
• EDOT Cloud Forwarder (ECF) — a serverless OTel Collector that collects telemetry from AWS, GCP, and Azure (VPC Flow Logs, CloudTrail, CloudWatch, and more) and forwards it directly to Elastic Observability, with no infrastructure to manage

The content pack doesn't care how the data arrived — only that it's there.

Seeing It in Practice: MySQL Monitoring

Take a team running MySQL who wants to track query throughput, connection counts, buffer pool utilization, and slow query rates — and get alerted before small problems turn into 2am incidents. Historically, that means hours of dashboard building, custom alert queries, and a lot of guesswork about which metrics actually matter.

With the MySQL OpenTelemetry Assets Package, that work is already done. Here's how the whole thing comes together.

Step 1: Get the Data In

The data pipeline is driven by a collector configuration that defines receivers (where to scrape data from), processors (how to enrich or transform it), and exporters (where to send it — in this case, Elasticsearch).

Regardless of whether you use the EDOT Collector or the Upstream OTel Collector, the fundamental configuration structure is the same. The configuration below uses separate receivers for the primary and replica instances, because replication metrics are only available on replicas. Replace the placeholders with your actual endpoints, credentials, and Elasticsearch details.

receivers:
  mysql/primary:
    endpoint: <MYSQL_PRIMARY_ENDPOINT>
    username: <MYSQL_USER>
    password: <MYSQL_PASSWORD>
    collection_interval: 10s
    statement_events:
      digest_text_limit: 120
      limit: 250
    query_sample_collection:
      max_rows_per_query: 100
    events:
      db.server.query_sample:
        enabled: true
      db.server.top_query:
        enabled: true
    metrics:
      mysql.client.network.io:
        enabled: true
      mysql.connection.errors:
        enabled: true
      mysql.max_used_connections:
        enabled: true
      mysql.query.client.count:
        enabled: true
      mysql.query.count:
        enabled: true
      mysql.query.slow.count:
        enabled: true
      mysql.table.rows:
        enabled: true
      mysql.table.size:
        enabled: true

processors:
  resourcedetection:
    detectors: [system, env]

exporters:
  elasticsearch/otel:
    endpoint: <ES_ENDPOINT>
    api_key: <ES_API_KEY>
    mapping:
      mode: otel

service:
  pipelines:
    metrics:
      receivers: [mysql/primary, mysql/replica]
      processors: [resourcedetection]
      exporters: [elasticsearch/otel]

The MySQL receiver scrapes metrics and events from the database at the configured interval and emits them as OpenTelemetry metrics. These flow through the pipeline and land in Elasticsearch, ready to be visualized.

Step 2: Open Kibana — Everything's Already There

Dashboards

As soon as the MySQL metrics and events arrive in Elasticsearch, the MySQL OpenTelemetry Assets Package is automatically installed in the background. By the time you navigate to Kibana, the dashboards are already populated and waiting.

Users immediately get visibility into:

• Active and max connections
• Query throughput — statements executed per second
• InnoDB buffer pool hit rate and memory usage
• Slow query count and trends
• Table lock waits and contention
• Bytes sent and received over time
• Replication lag (for replicated setups)

No manual field mapping. No dashboard building from scratch. Just data in, insights out.

Below are some screenshots of the MySQL OpenTelemetry dashboard in Kibana, showing the out-of-the-box visualizations that are automatically available as soon as your data starts flowing in.

Overview Dashboard

Queries Dashboard

Availability Dashboard

Alert Rules, Ready to Enable

The package includes six pre-built alert rules — covering high connection error rates, slow query spikes, thread saturation, replication lag, buffer pool dirty page ratio, and row lock contention — each with recommended thresholds and severity levels. These are available immediately on install and can be enabled, tuned, and extended directly in Kibana without any custom query authoring. Below is an example of one of the alerts.

SLO Templates, Pre-Loaded

Four SLO templates are included out of the box, tracking replication lag, connection exhaustion errors, slow query rate, and connected thread count — each with a pre-configured target and 30-day rolling window. Teams can adopt them as-is or tune the thresholds to match their own reliability requirements.

What's Available Today

The MySQL OpenTelemetry Assets Package is just one example from a growing library of OpenTelemetry Content Packages that Elastic has already built out. Content packs are available for a range of services — and we have also started extending this to the cloud, with initial support for Cloud Service Provider integrations that use the EDOT Cloud Forwarder (ECF) to bring AWS, GCP, and Azure telemetry into Elastic with ready-made dashboards.

The same pattern holds across all of them — data in, and a complete observability package (dashboards, alert rules, SLO templates) instantly ready — whether you're monitoring a self-managed database or cloud-native services from your preferred cloud service provider.

Where This Is Going

The next step worth watching is OTel Integration Packages, which will let you push collector configurations directly from the Kibana UI — making the entire setup experience point-and-click, from data collection through to visualization, with no YAML editing required.

Get Started

Ready to try it? Start with the EDOT Collector documentation and explore the growing library of OpenTelemetry content packages in Kibana's Integrations page.

• Detect Kubernetes pod restarts and OOMKill events using Elastic Agent Builder
• Analyze CPU and memory pressure using ES|QL over Kubernetes metrics
• Generate troubleshooting summaries and remediation guidance

This article explains how to use Elastic Agent Builder to automatically detect, analyze, and remediate Kubernetes pod failures caused by resource pressure (CPU and memory), with a focus on pods experiencing frequent restarts and OOMKilled events. Elastic Agent Builder lets you quickly create precise agents that utilize all your data with powerful tools (such as ES|QL queries), chat interfaces, and custom agents.

Introduction: What is the Elastic Agent Builder?

Elastic has an AI Agent embedded that you can use to get more insights from all of the logs, metrics and traces that you’ve ingested. While that’s great, you can take it one step further and streamline the process by creating tools that the agent can use.

Giving the agent tools means it spends less time ‘thinking’ and quickly gets to assessing what’s important to you. For example, if I have a Kubernetes environment that needs monitoring, and I want to keep an eye on pod restarts and memory and CPU usage without hanging out at the terminal, I can have Elastic alert me if something goes wrong. 

Having an alert is great, but how do I get the bigger picture, faster? You need to know what service is having (or creating) the issues, why, and how to fix it.

Assumptions

This guide assumes:

• A running Kubernetes cluster
• An Elastic Observability deployment
• Kubernetes metrics indexed in Elastic

Step 1: Create a New Elastic Agent

In Elastic Observability, use the top search bar to search for Agents. Create a new agent.

This agent is going to be the Kubernetes Pod Troubleshooter agent, designed to help users troubleshoot pod restarts, OOMKill terminations and evaluate CPU or memory pressure. 

The Kubernetes Pod Troubleshooter agent will:

1. Identify pods that have restarted more than once
2. Filter for pods that are not in a running state
3. Retrieve the container termination reason (e.g., OOMKilled)
4. Analyze CPU and memory utilization for affected services
5. Flag resource utilization above 60% (warning) and 80% (critical)
6. Provide remediation recommendations

The agent requires instructions to guide how the agent behaves when interacting with tools or responding to queries. This description can set tone, priorities or special behaviours. The instructions below tell the agent to execute the steps outlined above. 

You will help users troubleshoot problematic pods by searching the metrics for pods that have restarted more than once and the status is not running. Pods that have the highest number of restarts will be returned to the user.
Once the containers that are not running and have restarted multiple times are found you will use their container ID or image name to to look up the container status reason and reason for the last termination. You will return that reason to the user.
You will also begin basic troubleshooting steps, such as checking  for insufficient cluster resources (CPU or memory) from the metrics and tools available.
Any CPU or memory utilization percentages over 60%, and definitely over 80% should be flagged to the user with remediation steps.

Getting answers quickly is critical when troubleshooting high-value systems and environments. Using Tools ensures that the workflow is repeatable and that you can trust the results. You also get complete oversight of the process, as the Elastic Agent outlines every step and query that it took and you can explore the results in Discover.

You will create custom tools that the agent will run to complete the Kubernetes troubleshooting tasks that the custom instructions references such as: look up the container status reason and reason for the last termination and checking  for insufficient cluster resources (CPU or memory).

Step 2: Create Tools - Pod Restarts

The first tool takes the Kubernetes metrics and assesses if the pod has restarted and it has a last terminated reason, and if it has the agent will present that information to the user.

This pod-restarts tool uses a custom ES|QL query that interrogates the Kubernetes metrics data coming from OTel.

The ES|QL query:

1. Filters for containers that have restarted and have a reason for termination; then
2. Calculates the number of restarts; then
3. Returns the number of restarts and termination reason per service.

FROM metrics-k8sclusterreceiver.otel-default
| WHERE metrics.k8s.container.restarts > 0
| WHERE resource.attributes.k8s.container.status.last_terminated_reason IS NOT NULL
| STATS total_restarts = SUM(metrics.k8s.container.restarts),
        reasons = VALUES(resource.attributes.k8s.container.status.last_terminated_reason) 
  BY resource.attributes.service.name
| SORT total_restarts DESC

Step 3: Create Tools - Service Memory

The custom tools can take input variables, which increases speed and accuracy of the results.

Common reasons for pods not scheduling, or restarting often, is due to the cluster or nodes being under-resourced. The pod-restarts tool returns services that have many restarts and OOMKill termination reasons, which indicate memory pressure.

The eval-pod-memory tool is a custom ES|QL that:

1. Filters for metrics data that match the service name returned from the pod-restarts tool within the last 12 hours; then
2. Converts memory usage, requests, limits and utilization into megabytes; then
3. Calculates the average of each of those metrics; then
4. Groups them into 1 minute groupings and sorts them.

FROM metrics-*
| WHERE resource.attributes.service.name == ?servicename
| WHERE @timestamp >= NOW() - 12 hours
| EVAL
  memory_usage_mb = metrics.container.memory.usage / 1024 / 1024,
   memory_request_mb = metrics.k8s.container.memory_request / 1024 / 1024,
   memory_limit_mb = metrics.k8s.container.memory_limit / 1024 / 1024,
   memory_utilization_pct = metrics.k8s.container.memory_limit_utilization * 100
| STATS
   avg_memory_usage = AVG(memory_usage_mb),
   avg_memory_request = AVG(memory_request_mb),
   avg_memory_limit = AVG(memory_limit_mb),
   avg_memory_utilization = AVG(memory_utilization_pct)
   BY bucket = BUCKET(@timestamp, 1 minute)
| SORT bucket ASC

Step 4: Create Tools: Service CPU

As CPU usage is another common reason for pods to fail scheduling or be stuck in endless restart loops, the next tool will evaluate CPU usage, requests and limits.

The eval-pod-cpu tool is a custom ES|QL that:

1. Filters for metrics data that match the service name returned from the pod-restarts tool within the last 12 hours; then
2. Calculates the average for CPU usage, CPU request utilization and CPU limit utilization.

FROM metrics-kubeletstatsreceiver.otel-default
| WHERE k8s.container.name == ?servicename OR resource.attributes.k8s.container.name == ?servicename
| STATS
  avg_cpu_usage = AVG(container.cpu.usage),
  avg_cpu_request_utilization = AVG(k8s.container.cpu_request_utilization) * 100,
  avg_cpu_limit_utilization = AVG(k8s.container.cpu_limit_utilization) * 100
| LIMIT 100

Step 5: Assign Tools to Kubernetes Pod Troubleshooter Agent

Once all of the tools are built you need to assign them to the agent.

This image shows the Kubernetes Pod Troubleshooter agent with the three tools: pod-restarts, eval-pod-cpu and eval-pod-memory assigned to it and active.

Step 6: Test the Kubernetes Pod Troubleshooter Agent

To simulate memory pressure the Open Telemetry demo is running inside the cluster. Artificially lowering the memory requests and limits and increasing the service load will cause pods to restart.

To do this to the open telemetry demo in your cluster, follow these steps. 

Reduce the cart service to one replica by scaling the deployment. Once that is complete, change the resources on the deployment by lowering the memory requests and limits as shown in this command:

kubectl -n otel-demo scale deploy/cart --replicas=1
kubectl -n otel-demo set resources deploy/cart -c cart --requests=memory=50Mi --limits=memory=60Mi

The OpenTelemetry demo application comes with a load-generator. This is used to simulate requests to the demo site by modifying the users and spawn rate in the load generator deployment, as shown in this command:

kubectl -n otel-demo set env deploy/load-generator LOCUST_USERS=800 LOCUST_SPAWN_RATE=200 LOCUST_BROWSER_TRAFFIC_ENABLED=false

If you list all of your pods in the cluster or namespace, you should begin to see restarts.

You can now chat with the Kubernetes Pod Troubleshooter agent and ask “Are any of my Kubernetes pods having issues?”.

The screenshot shows the final response from the Kubernetes Pod Troubleshooter agent. It provides a problem summary of its findings from each tool, showing which services were experiencing the most restarts and memory and CPU utilization. 

The threshold interpretations were described in the initial agent instructions, where >60% utilization is a warning (sustained pressure) and >80% utilization is critical (high likelihood of restarts or throttling). This aligns with findings presented by the Kubernetes Pod Troubleshooter agent, where the services that had the highest restarts were all above 90% memory utilization. The agent needs clearly defined threshold values to correctly assess the returned memory and CPU utilization values. 

Problem summary returned by the Kubernetes Pod Troubleshooter agent:

Conclusion and Final Thoughts

Elastic Agent Builder enables fast, repeatable Kubernetes troubleshooting by combining ES|QL-driven analysis with constrained AI reasoning.

The creation of custom tools that use specific ES|QL queries combined with downstream queries that take input variables from the output of previous tools eliminates or reduces error propagation and hallucinations. In comparison to generic AI troubleshooting without purpose-built tools, you run the risk of it analyzing too many services (that aren’t relevant to the issue at hand). This will slow down the thinking process and generate longer responses, increasing the likelihood of error propagation and hallucinations. 

With the Elastic Agent Builder, you can inspect the output of every tool if you need to, to explore and verify the outputs.

Having a succinct problem summary is a game-changer, bringing your attention straight to the most affected services.

Reasoning returned by the Kubernetes Pod Troubleshooter agent:

Not only that, but the agent can go one step further and offer recommendations for remediation based on what outputs the tools delivered.

Remediation recommendation returned by the Kubernetes Pod Troubleshooter agent:

Sign up for Elastic Cloud Serverless and try this out with your Kubernetes clusters.

Frequently Asked Questions

1. When to use the Elastic Agent Builder for Troubleshooting

Use the Elastic Agent Builder for Troubleshooting that works best if:

• You need repeatable, auditable troubleshooting workflows

• You want deterministic analysis instead of free-form AI responses

• You’re investigating something that is reported in the logs or metrics (i.e. pod restarts, OOMKills, or resource pressure)

• You want to reduce mean time to resolution (MTTR)

2. Do I need OpenTelemetry to use Elastic Agent Builder for Kubernetes troubleshooting? 

No, you don’t need to use OpenTelemetry. You have two options:

• You can collect logs and metrics from Kubernetes using the Elastic Agent; or 

• You can collect logs, traces and metrics with the Elastic Distro for OTel (EDOT) Collector

When following the steps above, this would change the field names that are used in the tools above. For example, kubernetes.container.memory.usage.bytes vs metrics.container.memory.usage.

3. Can this agent be adapted for node-level failures? 

Yes, Elastic has hundreds of integrations, including AWS (for EKS), Azure (for AKS), Google Cloud (for GKE), as well as host operating system monitoring.

The queries shown above would be modified to use the correct field.

4. Can these tools be reused in automation workflows? 

Yes, Elastic Workflows can reuse the same scripted automations and AI agents you build in Elastic. An agent can handle the initial analysis and investigation (reducing manual effort), and the workflow can continue with structured steps, such as running Elasticsearch queries, transforming data, branching on conditions and calling external APIs or tools like Slack, Jira and PagerDuty. Workflows can also be exposed to Agent Builder as reusable tools, just like the tool created in this guide.

For more advanced automation from a similar scenario as described in this guide, learn how to integrate AI agents into GitHub Actions to monitor K8s health and improve deployment reliability via Observability.

5. Can these tools be triggered by alerts? 

Yes, alerts can trigger Elastic Workflows, and pass the alert context to the workflow. This workflow may be integrated with an Elastic Agent, as described above.

Additionally, Elastic Alerts allow you to publish investigation guides alongside alerts so an SRE has all of the information they need to begin investigating. Any troubleshooting or investigative agents can be linked to from the investigation guide, meaning the SRE doesn’t have to follow manual processes outlined in an investigation guide and instead let the agent handle the manual, repetitive investigations.

6. How can I get started with Agent Builder?

Sign up for Elastic Cloud Serverless, a new fully managed, stateless architecture that auto-scales no matter your data, usage, and performance needs.

In five years working with Enterprise accounts at Elastic, I have heard the same challenge again and again:

"We have several network monitoring tools, and we would love to correlate all of them into one platform."

For many organizations, the barrier to true correlation isn't a lack of data, but where that data lives. Frequently, we see SNMP metrics, flow data, and logs isolated in purpose-built silos or dashboards. Without a unified data store and a proper correlation engine, piecing together the full narrative — from a topology change to a performance degradation — becomes a manual, time-consuming puzzle.

When an incident happens, engineers become human correlation engines — manually jumping between systems, copying timestamps, cross-referencing device names, and trying to piece together what actually happened. A simple question like "Did this interface failure impact application performance?" requires querying multiple tools and mentally correlating the results.

The real cost isn't the tool licenses — it's the time lost during critical incidents.

This lab is my answer to a fundamental question: Can Elastic become the unified foundation that actually correlates network data?

More importantly, it demonstrates that Elastic is fully ready for network operations — capable of ingesting diverse telemetry and using AI to correlate relationships, identify root causes, and resolve issues in seconds instead of hours.

The Problem: Network Observability is Broken

Let me paint a typical scenario I encounter with enterprise network teams:

The Fragmented Reality:

• No single source of truth
• Manual correlation during incidents (15-30 minutes per event)
• Fragmented teams (network vs. platform engineers)
• Limited automation capabilities
• No AI-powered analysis

When a link goes down at 2 AM:

• Notice the alert - 2 minutes
• Log into monitoring tool to see the metric - 3 minutes
• Switch to traffic analyzer to check impact - 5 minutes
• Open log management to search for related messages - 10 minutes
• Manually correlate timestamps across systems - 8 minutes
• Create a ticket and copy context from multiple tools - 8 minutes

Time to initial diagnosis: 36 minutes

This workflow is expensive, error-prone, and doesn't scale.

The Vision: Elastic as a Unified Network Observability Platform

What if you could:

• Collect SNMP metrics, NetFlow, traps, and topology data in one platform
• Correlate network events with application performance automatically
• Generate executive dashboards without separate BI tools
• Use AI to analyze incidents in seconds, not hours
• Trigger alerting from network events

This is what this lab aims to demonstrate.

What I Built: A Production-Grade Network Simulation

To demonstrate how Elastic unifies network data, I needed a realistic environment that generates real-world telemetry. Enter Containerlab — a Docker-based solution that enables us to create a network simulation framework.

Lab Architecture

I simulated a Service Provider core network with:

• 7 FRR routers forming an OSPF Area 0 mesh
• 2 Ubuntu hosts for additional use cases
• 2 Layer 2 switches for access layer segmentation
• 3 telemetry collectors feeding Elastic Cloud

Total containers: 14

Deployment time: 12-15 minutes (fully automated)

Full deployment instructions and topology details are available in the GitHub repository README.

The Three Telemetry Pipelines: Proving Multi-Source Correlation

What makes this lab production-ready is its hybrid observability approach — proving that Elastic can unify disparate network data sources.

| Pipeline | Data Type | Collection Method | Collector | Use Case | | :---- | :---- | :---- | :---- | :---- | | SNMP Metrics | Interface stats, system health, LLDP topology | Active polling | OTEL Collector | Capacity planning, trend analysis | | NetFlow | Traffic flows | Push-based export | Elastic Agent | Top talkers, security investigation | | SNMP Traps | Interface up/down events | Event-driven | Logstash | Real-time incident detection |

This unified architecture proves Elastic can replace multiple specialized network monitoring tools with a single platform.

The Power of Correlation: One Platform, One Query

When a network incident occurs, you need to answer questions like:

• Which interface failed? (SNMP metrics)
• What traffic was affected? (NetFlow)
• What was the sequence of events? (SNMP traps)
• Which devices are downstream? (LLDP topology)

The Problem: modern tools offer separate modules glued together, forcing users to navigate different spaces for different sets of data.

The Reality: You still have to pivot. You see a spike in the Metrics module, but to see why, you have to open the Logs module and manually align the time picker. The data lives in different tables or backends, making true correlation impossible without human intervention.

The Elastic Difference: One Store, One Language, One AI

Elastic makes it simple. Whether it's an SNMP counter (metric), a NetFlow record (flow), or a Syslog message (log), it is all stored in a unified datastore powered by the Elasticsearch engine. This allows users to easily search across multiple datasets in a single query.

FROM logs-*
| WHERE host.name == "csr23" AND interface.name == "eth1"

Time required: 3 seconds

Furthermore, as you will see later, the exact location of the data becomes agnostic to the user when leveraging the AI Assistant.

Data Transformation: From Cryptic OIDs to Actionable Intelligence

Raw SNMP traps are notoriously difficult to interpret at a glance. In our current lab setup, the data arrives looking like this:

OID: 1.3.6.1.6.3.1.1.5.3
ifIndex: 2
ifDescr: eth1

While traditional Network Management Platforms (NMPs) handle OID translation natively, bringing that clarity into Elastic requires a specific configuration.

In this initial lab, we are intentionally working with this raw data to demonstrate how AI assistants can interpret these events even without pre-existing context.

However, the strategy for the next phase of this project is to implement Elasticsearch Ingest Pipelines. This will allow us to map raw OIDs to human-readable names. This step is crucial for bridging the gap between Network tools and Application Observability platforms, allowing network events to be instantly correlated with application errors and infrastructure logs.

The Target State

Once the pipeline is implemented in the next lab, we will transform that raw trap into searchable, meaningful data:

{
  "event.action": "interface-down",
  "host.name": "csr23",
  "interface.name": "eth1",
  "interface.oper_status_text": "Link Down"
}

The result:

• Human-readable fields
• Searchable dimensions for filtering
• Context for automation rules and dashboards
• Correlation keys for joining with metrics and flows

In our next blog post, we will walk through building the ingest pipeline that performs this transformation — step by step.

Intelligent Alerting: From Noise to Actionable Intelligence

Traditional network monitoring relies on simple threshold alerts — "interface down," "high CPU." These alerts flood your inbox but provide zero context about root cause, impact, or remediation.

The Lab's Approach: ES|QL + AI Assistant

1. Semantic Detection with ES|QL

Instead of generic threshold alerts, the lab uses ES|QL to detect specific event patterns:

FROM logs-snmp.trap-prod
| WHERE snmp.trap_oid == "1.3.6.1.6.3.1.1.5.3"
| KEEP @timestamp, host.name, interface.name, message

2. Automatic AI-Powered Investigation

When the alert triggers, it invokes the Observability AI Assistant with a structured investigation prompt that:

• Performs immediate triage (which device, which interface, when)
• Assesses OSPF impact and traffic rerouting
• Correlates with other recent failures
• Generates severity assessment and recommended actions

The Transformation

| Traditional Alerting | Intelligent Alerting (Elastic) | | :---: | :---: | | Email: "Interface down on csr23" | Structured analysis with device context | | Manual investigation: 20-30 min | AI-automated investigation: 90 seconds | | Engineer correlates across tools | Automatic cross-source correlation | | No business impact assessment | Severity + recommended actions included |

Accelerating Incident Response with the Elastic AI Assistant

This is where the Elastic AI Assistant demonstrates its operational value — moving beyond passive data collection to actively interpret and explain network events in real-time

When an engineer views a trap document in Discover and asks:

"Explain this log message"

The AI Assistant provides comprehensive analysis including:

• What happened: Plain-language explanation of the SNMP trap
• Device context: Router role, interface purpose, network position
• Impact analysis: OSPF neighbor status, traffic rerouting assessment
• Root cause possibilities: Physical layer, link layer, administrative causes
• Recommended actions: Immediate steps, investigation queries, validation checks
• Severity assessment: Business and technical impact rating

Manual Triage vs. AI-Assisted Investigation

| Before | After (Elastic AI) | | :---- | :---- | | Google the OID → 5 min | Click "Explain this log" → 20 seconds | | Open network diagram → 3 min | Topology context auto-provided | | Query multiple tools → 10 min | Cross-source correlation instant | | Assess business impact → 5 min | Impact analysis auto-generated | | Total: ~28 minutes | Total: ~20 seconds |

The Value Proposition: One Platform, One Data Model, One AI

What This Lab Demonstrates

Elastic provides:

• One unified platform for metrics, logs, flows
• One data model (SemConv) for consistent correlation
• One search interface (Kibana) for all network data
• One AI assistant that understands all your network telemetry
• AI-powered alerting with automated investigation

Business Impact

Efficiency Gains:

• 85% reduction in MTTR (36 min → 5 min for initial diagnosis)
• 90% reduction in manual correlation time
• Junior engineers gain access to AI-powered expert analysis

Operational Benefits:

• Network engineers focus on strategy, not tool-switching
• Cross-functional collaboration in one platform
• Reduced tool sprawl and management overhead

Lessons Learned

After building this lab, several key insights emerged regarding how network data fits into the broader observability ecosystem:

1. Extending Observability to the Network

Elastic is already the gold standard for high-volume logs and application traces. This lab demonstrates that the same engine seamlessly handles network telemetry without needing a separate, siloed tool.

• Scale: The same architecture that ingests petabytes of application logs easily handles millions of interface counters.
• Structure: Native support for complex nested documents allows for rich SNMP trap data (variable bindings) without flattening or losing context.
• Speed: Real-time search applies equally to network events, enabling sub-second troubleshooting.

2. OpenTelemetry Semantic Conventions (SemConv) as the Universal Translator

The power isn't just in storing the data, but in standardizing it. By mapping SNMP and NetFlow to the OpenTelemetry Semantic Conventions (SemConv), network data finally speaks the same language as the rest of the stack.

• Unified Search: Query across firewall logs, server metrics, and switch telemetry in a single search bar.
• Instant Visualization: Pre-built dashboards work immediately because the field names are standardized.
• Cross-Domain Correlation: Easily correlates a spike in application latency with a specific interface saturation event.

3. AI Assistants Thrive on Context

While the AI in this lab was powerful on its own, the experiment highlighted a critical realization: an AI Assistant becomes exponentially more effective when coupled with a specific Knowledge Base.

Context is King: The AI delivers better root cause analysis when provided with rich metadata, such as device roles and topology maps. Without it, the advice remains generic.

Pro Tip (and What’s Next):

To get organization-specific advice rather than generic suggestions, you need to feed the AI your documentation.

• The Goal: Create a Knowledge Base containing device roles, network topology diagrams, and troubleshooting procedures.
• The Next Step: In my next blog post, I will demonstrate exactly how to do this — connecting a Knowledge Base to the AI Assistant to enable fully context-aware troubleshooting.

Conclusion: Completing the Observability Picture

Elastic is already widely recognized as the standard for Application and Security observability. The goal of this lab wasn't to ask if Elastic can handle networking, but to demonstrate the immense value of bringing network data into that existing ecosystem.

The verdict is clear: Elastic acts as that unified foundation. It effectively breaks down the silo between Network Engineering and the rest of IT.

This isn't just about consolidating dashboards or replacing legacy tools. It is about establishing the Elasticsearch AI Platform as the single source of truth where network telemetry sits right alongside application and infrastructure data.

By treating network data as a first-class citizen in the observability stack, we unlock automated correlation, AI-assisted investigation, and the speed required to resolve incidents before they impact the business. The capabilities are in place, and the foundation is solid — Elastic is ready to unify your network with the rest of your digital business.

Ready to Try It Yourself?

Check out github.com/DeBaker1974/Containerlab-OSPF

The repository includes:

• Complete deployment scripts (12-15 minute automated setup)
• Pre-configured telemetry pipelines
• Kibana dashboards
• Alert rules with AI Assistant integration
• Detailed README

Not ready to build? Try Elastic Serverless: Start a free 14-day trial and explore AI-powered observability with your own data.

Special thanks to the Containerlab and FRRouting communities for their incredible open-source tools, and to Sheriff Lawal (CCIE, CISSP), Sr. Manager, Solutions Architecture at Elastic, for mentoring on this project.

These latest updates, available in v9.3 and in Serverless, introduce significant performance gains, sophisticated time series functions, and native OpenTelemetry exponential histogram support that directly benefit SREs and Observability practitioners.

Query Performance and Storage Optimizations

Speed is paramount when diagnosing incidents. Compared to prior releases, we have achieved a 5x+ improvement in query latency when wildcarding or filtering by dimensions. Additionally, storage efficiency for OpenTelemetry metrics data has improved by approximately 2x, significantly reducing the infrastructure footprint required to retain high-volume observability data. If you’re hungry to learn more about what architectural updates are driving these optimizations, stay tuned… Tech blogs are on their way! 

Expanded Time Series Analytics in ES|QL

The ESQL TS source command, which targets time series indices and enables time series aggregation functions, has been significantly enhanced to support complex analytics capabilities.

We have expanded the library of time series functions to include essential tools for identifying anomalies and trends.

• PERCENTILE_OVER_TIME, STDDEV_OVER_TIME, VARIANCE_OVER_TIME: Calculate the percentile, standard deviation, or variance of a field over time, which is critical for understanding distribution and variability in service latency or resource usage.

Example: Seeing the worst-case latency in 5-minute intervals.

TS metrics*  | STATS MAX(PERCENTILE_OVER_TIME(kafka.consumer.fetch_latency_avg, 99))
  BY TBUCKET(5m)

• DERIV: This command calculates the derivative of a numeric field over time using linear regression, useful for analyzing the rate of change in system metrics.

Example: trending gauge values over time.

TS metrics*  | STATS AVG(DERIV(container.memory.available))
  BY TBUCKET(1 hour)

• CLAMP: To handle noisy data or outliers, this function limits sample values to a specified lower and upper bound.

Example: handling saturation metrics (like CPU or Memory utilization) where spikes or measurement errors can occasionally report values over 100%, making the rest of the data look like a flat line at the bottom of the chart.\

TS metrics*  | STATS AVG(CLAMP(k8s.pod.memory.node.utilization, 0, 100))
  BY k8s.pod.name

• TRANGE: This new filter function allows you to filter data for a specific time range using the @timestamp attribute, simplifying query syntax for time-bound investigations.

Example: Filtering and showing metrics for the last 4 hours.

TS metrics*  | WHERE TRANGE(4h) | STATS AVG(host.cpu.pct)
  BY TBUCKET(5m)

Window Functions To smoothen results over specific periods, ES|QL now introduces window functions. Most time series aggregation functions now accept an optional second argument that specifies a sliding time window. For example, you can calculate a rate over a 10-minute sliding window while bucketing results by minute.

Example: Calculating the average rate of requests per host for every minute, using values over a sliding window of 5 minutes.

TS metrics*  | STATS AVG(RATE(app.frontend.requests, 5m))
  BY TBUCKET(1m)

Accepted window values are currently limited to multiples of the time bucket interval in the BY clause. Windows that are smaller than the time bucket interval or larger but not a multiple of the time bucket interval will be supported in feature releases. 

Native OpenTelemetry Exponential Histograms

Elastic now provides native support for OpenTelemetry exponential histograms, enabling efficient ingest, querying, and downsampling of high-fidelity distribution data.

We have introduced a new exponential_histogram field type designed to capture distributions with fixed, exponentially spaced bucket boundaries. Because these fields are primarily intended for aggregations, the histogram is stored as compact doc values and is not indexed, optimizing storage efficiency. These fields are fully supported in ES|QL aggregation functions such as PERCENTILES, AVG, MIN, MAX, and SUM.

You can index documents with exponential histograms automatically through our OTLP endpoint or manually. For example, let’s create an index with an exponential histogram field and a keyword field:

PUT my-index-000001
{
  "settings": {
    "index": {
      "mode": "time_series",
      "routing_path": ["http.path"],
      "time_series": {
        "start_time": "2026-01-21T00:00:00Z",
        "end_time": "2026-01-25T00:00:00Z"
     }
    }
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "http.path": {
        "type": "keyword",
        "time_series_dimension": true
      },
      "responseTime": {
        "type": "exponential_histogram",
        "time_series_metric": "histogram"
      }
    }
  }
}

Index a document with a full exponential histogram payload:

POST my-index-000001/_doc
{
  "@timestamp": "2026-01-22T21:25:00.000Z",
  "http.path": "/foo",
  "responseTime": {
    "scale":3,
    "sum":73.2,
    "min":3.12,
    "max":7.02,
    "positive": {
      "indices":[13,14,15,16,17,18,19,20,21,22],
      "counts":[1,1,2,2,1,2,1,3,1,1]
    }
  }
}

POST my-index-000001/_doc
{
  "@timestamp": "2026-01-22T21:26:00.000Z",
  "http.path": "/bar",
  "responseTime": {
    "scale":3,
    "sum":45.86,
    "min":2.15,
    "max":5.1,
    "positive": {
      "indices":[8,9,10,11,12,13,14,15,16,17,18],
      "counts":[1,1,1,1,1,1,1,2,1,1,2]
    }
  }
}

And finally, query the time series index using ES|QL and the TS source command:

TS my-index-000001  | STATS MIN(responseTime), MAX(responseTime),
        AVG(responseTime), MEDIAN(responseTime),
        PERCENTILE(responseTime, 90)
  BY http.path

Enhanced Downsampling

Downsampling is essential for long-term data retention. We have introduced a new "last value" downsampling mode. This method exchanges accuracy for storage efficiency and performance by keeping only the last sample value, providing a lightweight alternative to calculating aggregate metrics.

You can configure a time series data stream for last value downsampling in a similar way as regular downsampling, just by setting the downsampling_method to last_value. For example, by using a data stream lifecycle:

PUT _data_stream/my-data-stream/_lifecycle
{
  "data_retention": "7d",
  "downsampling_method": "last_value",
  "downsampling": [
     {
       "after": "1m",
       "fixed_interval": "10m"
      },
      {
        "after": "1d",
        "fixed_interval": "1h"
      }
   ]
}

In Conclusion

These enhancements mark a significant step forward in Elastic's metrics analytics capabilities, delivering 5x+ faster query latency, 2x storage efficiency and specialized commands like DERIV, CLAMP, and PERCENTILE_OVER_TIME. With native support for OpenTelemetry exponential histograms and expanded downsampling options, SREs can now perform richer, more cost-effective analysis on their observability data. This release empowers teams to detect anomalies faster and manage long-term metrics retention with greater efficiency.

We welcome you to try the new features today!

As a core pillar of Observability, metrics offer a highly structured, quantitative view of system performance and health. They provide a crucial symptomatic perspective—revealing what is happening, such as high application latency, increasing service errors, or spiking container CPU utilization, which is essential for initiating alerting and triaging efforts. This capability for effective monitoring, alerting, and triaging is paramount to ensuring robust service delivery and achieving successful business outcomes.

Elastic Observability provides a comprehensive, end-to-end experience for metrics data. Elastic ensures that metrics data can be collected from numerous sources, enriched as needed and shipped to the Elastic Stack. Elastic efficiently stores this time series data, including high-cardinality metrics, utilizing the TSDS index mode (Time Series Data Stream), introduced in prior versions and used across Elastic time series integrations. This foundation ensures comprehensive observability through out-of-the-box dashboards, alerts, SLOs, and streamlined data management.

Elastic Observability 9.2 provides enhancements to metrics exploration and analysis through powerful query language extensions and expanded UI capabilities. These enhancements focus on making analysis on TSDS data via counter rates and common aggregations over time easier and faster than ever before.

The main metrics enhancements center on these key features, offered as Tech Preview:

1. Metrics analytics with TSDS and ES|QL
2. Interactive metrics exploration in Discover
3. OTLP endpoint for metrics

Metrics analytics with TSDS and ES|QL

The introduction of the new TS source command in ES|QL (Elasticsearch Query Language) on TSDS metrics dramatically simplifies time series analysis.

The TS command is specifically designed to target only time series indices, differentiating it from the general FROM command. Its core power lies in enabling a dedicated suite of time series aggregation functions within the STATS command.

This mechanism utilizes a dual aggregation paradigm, which is standard for time series querying. These queries involve two aggregation functions:

• Inner (Time Series) function: Applied implicitly per time series, often over bucketed time intervals.

• Outer (Regular) function: Used to aggregate the results of the inner function across groups. For instance, if you use STATS SUM(RATE(search_requests)) BY TBUCKET(1 hour), host, the RATE() function is the inner function applied per time series in hourly buckets, and SUM() is the outer function, summing these rates for each host and hourly bucket.

If an ES|QL query using the TS command is missing an inner (time series) aggregation function, LAST_OVER_TIME() is implicitly assumed and used. For example, TS metrics | STATS AVG(memory_usage) is equivalent to TS metrics | STATS AVG(LAST_OVER_TIME(memory_usage)).

Key time series aggregation functions available in ES|QL via TS command

These functions allow for powerful analysis on time-series data:

| | | | | :----------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------: | | Function | Description | Example Use Case | | RATE() / IRATE() | Calculates the per-second average rate of increase of a counter (RATE), accounting for non-monotonic breaks like counter resets, making it the most appropriate function for counters, or the per-second rate of increase between the last two data points (IRATE), ignoring all but the last two points for high responsiveness. | Calculating request per second (RPS) or throughput. | | AVG_OVER_TIME() | Calculates the average of a numeric field over the defined time range. | Determining average resource usage over an hour. | | SUM_OVER_TIME() | Calculates the sum of a field over the time range. | Total errors over a specific time window. | | MAX_OVER_TIME() / MIN_OVER_TIME() | Calculates the maximum or minimum value of a field over time. | Identifying peak resource consumption. | | DELTA() / IDELTA() | Calculates the absolute change of a gauge field over a time window (DELTA) or specifically between the last two data points (IDELTA), making IDELTA more responsive to recent changes. | Tracking changes in system gauge metrics (e.g., buffer size). | | INCREASE() | Calculates the absolute increase of a counter (INCREASE). | Analyzing immediate rate changes in fast-moving counters. | | FIRST_OVER_TIME() / LAST_OVER_TIME() | Calculates the earliest or latest recorded value of a field, determined by the @timestamp field. | Inspecting initial and final metric states within a bucket. | | ABSENT_OVER_TIME() / PRESENT_OVER_TIME() | Calculates the absence or presence of a field in the result over the time range. | Identifying monitoring coverage gaps. | | COUNT_OVER_TIME() / COUNT_DISTINCT_OVER_TIME() | Calculates the total count or the count of distinct values of a field over time. | Measuring frequency or cardinality changes. |

These functions, available with the TS command, allow SREs and Ops teams to easily perform rate calculations and other common aggregations, enabling efficient metrics analysis as a routine part of observability workflows. And it’s much faster, too! Internal performance testing has revealed that TS commands outperform other ways of querying metrics data by an order of magnitude or more, and consistently! 

Interactive metrics exploration in Discover

The 9.2 release introduces the capability to explore and analyze metrics directly and interactively within the Discover interface. In addition to exploring and analyzing logs and raw events, Discover now provides a dedicated environment for metrics exploration:

• Easy start: Begin exploration simply by querying metrics ingested via TS metrics-*.

• Grid view and pre-applied aggregations: This command displays all metrics in a grid format at a glance, immediately applying the appropriate aggregations based on the metric type, such as rate versus avg.

• Search and group-by: Quickly search for specific metrics by name. Also easily group and analyze metrics by dimensions (labels) and specific values. This allows narrowing down to metrics and dimensions of choice for targeted analysis.

• Quick access to details: Furthermore, the interface provides access to crucial details, including query and response details, the underlying ES|QL commands, the metric field type, and applicable dimensions, for each metric.

• Easy tweaking and dashboarding: The system automatically populates ES|QL queries, aiding in making easy tweaks, slicing, and dicing the data. Once analyzed, metrics and resulting analyses can be added to new or existing dashboards with ease.

OTLP endpoint for metrics

We are also introducing a native OpenTelemetry Protocol (OTLP) endpoint specifically for metrics ingest directly into Elasticsearch. The endpoint especially benefits self-managed customers, and will be integrated into our Elastic Cloud Managed OTLP Endpoint for Elastic-managed offerings. The native endpoint and related updates improve ingest performance and scalability of OTel metrics, providing up to 60% higher throughput via _otlp, and up to 25% higher throughput when using classic _bulk methods. 

In Conclusion

By merging the power of ES|QL's new time series aggregations with the familiar interactive experience of Discover, Elastic 9.2 enables a potent set of metrics analytics tools. The tools significantly boost the exploration and analysis phase of any observability workflow. And we’re just getting started on unleashing the full power of metrics in Elastic Observability!

We welcome you to try the new features today!

Also learn more about how we provide metrics analytics for AWS, Azure, GCP, Kubernetes, and LLMs on Observability Labs

Amazon MQ is a managed message broker service for Apache ActiveMQ Classic and RabbitMQ that manages the setup, operation, and maintenance of message brokers. Messaging systems like RabbitMQ, managed by Amazon MQ, are pivotal in modern decoupled, event-driven applications. By serving as an intermediary between services, RabbitMQ facilitates asynchronous communication through message queuing, routing, and reliable delivery, making it an ideal fit for microservices, real-time pipelines, and event-driven architectures. However, this flexibility introduces operational challenges, such as retries, processing delays, consumer failures, and queue backlogs, which can gradually impact downstream performance and system reliability.

With Elastic’s Amazon MQ integration, users gain deep visibility into message flow patterns, queue performance, and consumer health. This integration allows for the proactive detection of bottlenecks, helps optimize system behaviour, and ensures reliable message delivery at scale.

In this blog, we'll dive into the operational challenges of RabbitMQ in modern architectures, while also examining the common gaps and strategies for overcoming them.

Why Observability for RabbitMQ on Amazon MQ Matters?

RabbitMQ brokers are integral to distributed systems, handling tasks ranging from order processing to payment workflows and notification delivery. Any disruption can cascade into significant downstream issues. Observability into RabbitMQ helps answer critical operational questions like:​

• Is CPU and memory utilization increasing over time?
• What are the trends in the message publish rate, message confirmation rate?
• Are consumers failing to acknowledge messages?
• Which queues are experiencing abnormal growth?
• Are there an increasing number of messages being dead-lettered over time?

Enhanced Observability with Amazon MQ Integration

Elastic provides a dedicated Amazon MQ integration for RabbitMQ that utilizes Amazon CloudWatch metrics and logs to deliver comprehensive observability data. This integration enables the ingestion of metrics related to connections, nodes, queues, exchanges, and system logs.

By deploying Elastic Agent with this integration, the users can monitor:​

• Queue performance and Dead-letter queue (DLQ) metrics include total message count (MessageCount.max), messages ready for delivery (MessageReadyCount.max), and unacknowledged messages (MessageUnacknowledgedCount.max). MessageCount.max metric tracks the total number of messages in a queue, including those that have been dead-lettered, and monitoring this over time can help identify trends in message accumulation, which may suggest issues leading to dead-lettering.
• Consumer behaviour through metrics like consumer count (ConsumerCount.max) and acknowledgement rate (AckRate.max), which help identify underperforming consumers or potential backlogs.
• Messaging throughput by tracking publish (PublishRate.max), confirm (ConfirmRate.max), and acknowledgement rates in real time. These are crucial for understanding application messaging patterns and flow.
• Broker and node-level health, including memory usage (RabbitMQMemUsed.max), CPU utilization (SystemCpuUtilization.max), disk availability (RabbitMQDiskFree.min), and file descriptor usage (RabbitMQFdUsed.max). These indicators are essential for diagnosing resource saturation and avoiding service disruption.

Integrating Amazon MQ Metrics into Elastic Observability

Elastic's Amazon MQ integration facilitates the ingestion of CloudWatch metrics and logs into Elastic Observability, delivering near real-time insights into RabbitMQ. The prebuilt Amazon MQ dashboard visualizes this data, providing a centralized view of broker health, messaging activity, and resource usage, helping users quickly detect and resolve issues. Elastic's alerting for Observability enables proactive notifications based on custom conditions, while its SLO capabilities allow users to define and track key performance targets, strengthening system reliability and service commitments. 

Elastic brings together logs and metrics from Amazon MQ alongside data from a wide range of other services and applications, whether running in AWS, on-premises, or across multi-cloud environments, offering unified observability from a single platform.

Prerequisites

To follow along, ensure you have:

• An account on Elastic Cloud and a deployed stack in AWS (see instructions here). Ensure you are using version 8.16.5 or higher. Alternatively, you can use Elastic Cloud Serverless, a fully managed solution that eliminates infrastructure management, automatically scales based on usage, and lets you focus entirely on extracting value from your data.
• An AWS account with permissions to pull the necessary data from AWS. See details in our documentation.

Architecture

Tracing Audit Flows from RabbitMQ to AWS Lambda

Consider a financial audit trail use case, where every user action, such as a funds transfer, is published to RabbitMQ. A Python-based AWS Lambda function consumes these messages, deduplicates them using the id field, and logs structured audit events for downstream analysis.

Sample payload sent through RabbitMQ:

{
  "id": "txn-849302",
  "type": "audit",
  "payload": {
    "user_id": "u-10245",
    "event": "funds.transfer",
    "amount": 1200.75,
    "currency": "USD",
    "timestamp": "T14:20:15Z",
    "ip": "192.168.0.8",
    "location": "New York, USA"
  }
}

You can now correlate message publishing activity from RabbitMQ with AWS Lambda invocation logs, track processing latency, and configure alerts for conditions like drops in consumer throughput or an unexpected surge in RabbitMQ queue depth.

AWS Lambda Function: Processing RabbitMQ Messages

This Python-based AWS Lambda function processes audit events received from RabbitMQ. It deduplicates messages based on the id field and logs structured event data for downstream analysis or compliance. Save the code below in a file named app.py.

import json
import logging
import base64
# Configure logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# In-memory set to track processed message IDs for deduplication
processed_ids = set()
def lambda_handler(event, context):
    logger.info("Lambda triggered by RabbitMQ event")
    if 'rmqMessagesByQueue' not in event:
        logger.warning("Invalid event: missing 'rmqMessagesByQueue'")
        return {'statusCode': 400, 'body': 'Invalid RabbitMQ event'}
    for queue_name, messages in event['rmqMessagesByQueue'].items():
        logger.info(f"Processing queue: {queue_name}, Messages count: {len(messages)}")
        for msg in messages:
            try:
                raw_data = msg['data']
                decoded_json = base64.b64decode(raw_data).decode('utf-8')
                message = json.loads(decoded_json)
                logger.info(f"Decoded message: {json.dumps(message)}")
                message_id = message.get('id')
                if not message_id:
                    logger.warning("Message missing 'id', skipping.")
                    continue
                if message_id in processed_ids:
                    logger.warning(f"Duplicate message detected: {message_id}")
                    continue
                payload = message.get('payload', {})
                logger.info(f"Processing message ID: {message_id}")
                logger.info(f"Event Type: {message.get('type')}")
                logger.info(f"User ID: {payload.get('user_id')}")
                logger.info(f"Event: {payload.get('event')}")
                logger.info(f"Amount: {payload.get('amount')} {payload.get('currency')}")
                logger.info(f"Timestamp: {payload.get('timestamp')}")
                logger.info(f"IP Address: {payload.get('ip')}")
                logger.info(f"Location: {payload.get('location')}")
                processed_ids.add(message_id)
            except Exception as e:
                logger.error(f"Error processing message: {str(e)}")
    return {'statusCode': 200, 'body': 'Messages processed successfully'}

Setting up AWS Secrets Manager

To securely store and manage your RabbitMQ credentials, use AWS Secrets Manager.​

1. Create a New Secret:

• Navigate to the AWS Secrets Manager console.
• Choose Store a new secret.
• Select Other type of secret.
• Enter the following key-value pairs:
  • username: Your RabbitMQ username
  • password: Your RabbitMQ password

1. Configure the Secret:

• Provide a meaningful name, such as RabbitMQAccess.
• Optionally, add tags and set rotation if needed.​

1. Store the Secret:

• Review the settings and store the secret. Note the ARN of the secret you have created.

Setting up Amazon MQ for RabbitMQ

To get started with RabbitMQ on Amazon MQ, follow these steps to set up your broker.

• Open the Amazon MQ console.

• Create a new broker with the RabbitMQ engine.

• Choose your preferred deployment option—single-instance or clustered

• Use the same username and password that you previously stored in AWS Secrets Manager.

• Under Additional settings, enable CloudWatch Logs for observability.

• Configure access and security settings, ensuring that the broker is accessible to your AWS Lambda function.

• After the broker is created, note the following important details:

• ARN of the RabbitMQ broker.

• RabbitMQ web console URL.

• You’ll need the RabbitMQ log group ARN to set up Elastic’s Amazon MQ integration for RabbitMQ. Follow these steps to locate it:

• Go to the General – Enabled Logs section of the broker. 

• Copy the CloudWatch log group ARN.

Create a RabbitMQ Queue

Now that the RabbitMQ broker is configured, use the management console to create a queue where messages will be published.

• Access the RabbitMQ management console using the web console URL.
• Create a new queue (example: myQueue) to receive messages.

Build and deploy the AWS Lambda function

In this section, we'll set up the Lambda function using AWS SAM, add the message processing logic, and deploy it to AWS. This Lambda function will be responsible for consuming messages from RabbitMQ and logging audit events.

Before continuing, make sure you have completed the following prerequisites.

• AWS SAM prerequisites

• Install the AWS SAM CLI

Next, follow the steps outlined below to continue with the setup.

1. In your command line, run the command sam init from a directory of your choice.
2. The AWS SAM CLI will walk you through the setup.
   • Select AWS Quick Start Templates.
   • Choose the Hello World Example 
   • Use the Python runtime and zip package type.
   • Proceed with the default options.
   • Name your application as sample-rabbitmq-app.
   • The AWS SAM CLI downloads your starting template and creates the application project directory structure.
3. From your command line, move to the newly created sample-rabbitmq-app directory.
   • Replace the content of the hello_world/app.py file with the lambda function code for rabbitmq message processing.
   • In the template.yaml file, use the values mentioned below to update the file content. yaml Resources: &nbsp;SampleRabbitMQApp: &nbsp;&nbsp;&nbsp;Type: AWS::Serverless::Function &nbsp;&nbsp;&nbsp;Properties: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;CodeUri: hello_world/ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Description: A starter AWS Lambda function. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;MemorySize: 128 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Timeout: 3 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Handler: app.lambda_handler &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Runtime: python3.10 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;PackageType: Zip &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Policies: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- Statement: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- Effect: Allow &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Resource: '*' &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Action: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- mq:DescribeBroker &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- secretsmanager:GetSecretValue &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- ec2:CreateNetworkInterface &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- ec2:DescribeNetworkInterfaces &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- ec2:DescribeVpcs &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- ec2:DeleteNetworkInterface &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- ec2:DescribeSubnets &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- ec2:DescribeSecurityGroups &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Events: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;MQEvent: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Type: MQ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Properties: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Broker: <ARN of the Broker> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Queues: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- myQueue &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;SourceAccessConfigurations: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- Type: BASIC_AUTH &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;URI: <ARN of the secret>
4. Run the command sam deploy --guided and wait for the confirmation message. This deploys all of the resources.

Sending Audit Events to RabbitMQ and Triggering Lambda

To test the end-to-end setup, simulate the flow by publishing audit event data into RabbitMQ using its web UI. Once the message is sent, it triggers the Lambda function. 

1. Navigate to the Amazon MQ console and select your newly created broker.
2. Locate and open the Rabbit web console URL
   
3. Under the Queues and Streams tab, select the target queue (example: myQueue).
4. Enter the message payload, and click Publish message to send it to the queue.
   Here’s a sample payload published via RabbitMQ:

   {
     "id": "txn-849302",
     "type": "audit",
     "payload": {
       "user_id": "u-10245",
       "event": "funds.transfer",
       "amount": 1200.75,
       "currency": "USD",
       "timestamp": "T14:20:15Z",
       "ip": "192.168.0.8",
       "location": "New York, USA"
     }
   }

1. Navigate to the AWS Lambda function created earlier.
2. Under the Monitor tab, click View CloudWatch logs.
3. Check the latest log stream to confirm that the Lambda was triggered by Amazon MQ and that the message was processed successfully.

Configuring Amazon MQ integration for Metrics and Logs collection

Elastic’s Amazon MQ integration simplifies the collection of logs and metrics from RabbitMQ brokers managed by Amazon MQ. Logs are ingested via Amazon CloudWatch Logs, while metrics are fetched from the specified AWS region at a defined interval.

Elastic provides a default configuration for metrics collection. You can accept these defaults or adjust settings such as the Collection Period to better fit your needs.

To enable the collection of logs:

1. Navigate to the Amazon MQ console and select the newly created broker.
2. Click the Logs hyperlink under the General – Enabled Logs section to open the detailed log settings page.
3. From this page, copy the CloudWatch log group ARN.
4. In Elastic, set up the Amazon MQ integration and paste the CloudWatch log group ARN.
5. Accept Defaults or Customize Settings – Elastic provides a default configuration for logs collection. You can accept these defaults or adjust settings such as collection intervals to better fit your needs.

Visualizing RabbitMQ Workloads with the Pre-Built Amazon MQ Dashboard

You can access the RabbitMQ dashboard by:

1. Navigate to the Dashboard Menu – Select the Dashboard menu option in Elastic and search for [Amazon MQ] RabbitMQ Overview to open the dashboard.

2. Navigate to the Integrations Menu – Open the Integrations menu in Elastic, select Amazon MQ, go to the Assets tab, and choose [Amazon MQ] RabbitMQ Overview from the dashboard assets

The Amazon MQ RabbitMQ dashboard in the Elastic integration delivers a comprehensive overview of broker health and messaging activity. It provides real-time insights into broker resource utilization, queue and topic performance, connection trends, and messaging throughput. The dashboard helps users track system behaviour, detect performance bottlenecks, and ensure reliable message delivery across distributed applications.

Broker Metrics

This section provides a centralised view of the overall health and performance of the RabbitMQ broker on Amazon MQ. The visualizations highlights the number of configured exchanges and queues, active broker connections, producers, consumers, and total messages in flight. System-level metrics such as CPU utilization, memory consumption, and free disk space help assess whether the broker has sufficient resources to handle current workloads.

Message flow metrics such as publish rate, confirmation rate, and acknowledgement rate are displayed to provide visibility into how messages are processed through the broker. Monitoring trends in these values helps detect message delivery issues, throughput degradation, or potential saturation of the broker under load.

Node Metrics

Node-level visibility helps identify resource imbalances across nodes in clustered RabbitMQ setups. This section includes per-node CPU usage, memory consumption, and available disk space, offering insight into the underlying infrastructure's ability to support broker operations.

Queue Metrics

Queue-specific insights are critical for understanding message delivery patterns and backlog conditions. This section details total messages, ready messages, and unacknowledged messages, segmented by broker, virtual host, and queue.

By observing how these counts change over time, users can identify slow consumers, message build-ups, or delivery issues that may affect application performance or lead to dropped messages under pressure.

Logs

This section displays log level, process ID, and raw message content. These logs provide immediate visibility into events such as connection failures, resource thresholds being hit, or unexpected queue behaviors.

Detecting Queue Backlogs with Alerting Rules

Elastic’s alert framework allows you to define rules that monitor critical RabbitMQ metrics and automatically trigger actions when specific thresholds are breached. 

Alert: Queue Backlog (Message Ready or Unacknowledged Messages)

This alert helps detect queue backlog in Amazon MQ by evaluating two metrics

• MessageUnacknowledgedCount.max and
• MessageReadyCount.max. 

The alert is triggered if either condition persists for more than 10 minutes:

• MessageUnacknowledgedCount.max exceeds 5,000
• MessageReadyCount.max exceeds 7,000

These thresholds should be adjusted based on typical message volume and consumer throughput. Sustained high values can indicate that consumers are not keeping up or message delivery pipelines are congested, potentially causing delays or dropped messages. Sustained high values may result in processing delays or dropped messages if not addressed.

Tracking Resource Utilization to Maintain RabbitMQ Performance

Elastic’s Service-level objectives (SLOs) capabilities allow you to define and monitor performance targets using key indicators like latency, availability, and error rates. Once configured, Elastic continuously evaluates these SLOs in real time, offering intuitive dashboards, alerts for threshold violations, and insights into error budget consumption. This enables teams to stay ahead of issues, ensuring service reliability and consistent performance.

SLO: Node Resource Health (CPU, Memory, Disk)

This SLO focuses on ensuring RabbitMQ brokers and nodes have sufficient resources to process messages without performance degradation. It tracks CPU, memory, and disk usage across RabbitMQ brokers and nodes to prevent resource exhaustion that could lead to service interruptions.

Target thresholds:

• SystemCpuUtilization.max remains below 85% for 99% of the time.
• RabbitMQMemUsed.max remains below 80% of RabbitMQMemLimit.max for 99% of the time.
• RabbitMQDiskFree.min remains above 25% of RabbitMQDiskFreeLimit.max for 99% of the time.

Sustained high values in CPU or memory usage can signal resource contention, which may result in slower message processing or downtime. Low disk availability may cause the broker to stop accepting messages, risking message loss. These thresholds are designed to catch early signs of resource saturation and ensure smooth, uninterrupted message flow across RabbitMQ deployments.

Conclusion

As RabbitMQ-based messaging architectures scale and become more complex, the need for in-depth visibility into system performance and potential issues deepens. Elastic’s Amazon MQ integration brings that visibility front and center—helping you go beyond basic health checks to understand real-time messaging throughput, queue backlog trends, and resource saturation across your brokers and consumers.

By leveraging the prebuilt dashboards, configuring alerts and SLOs, you can proactively detect anomalies, fine-tune consumer performance, and ensure reliable delivery across your event-driven applications.

OpenTelemetry recently published a blog on how to do Autodiscovery based on Kubernetes Pods' annotations with the OpenTelemetry Collector.

In this blog post, we will talk about how to use this Kubernetes-related feature of the OpenTelemetry Collector, which is already available with the Elastic Distribution of the OpenTelemetry (EDOT) Collector.

In addition to this feature, at Elastic, we heavily invest in making OpenTelemetry the best, standardized ingest solution for Observability. You might already have seen us focusing on:

• Semantic Conventions standardization

• significant log collection improvements

• various other topics around instrumentation

• profiling

Let's walk you through a hands-on journey using the EDOT Collector covering various use cases you might encounter in the real world, highlighting the capabilities of this powerful feature.

Configuring EDOT Collector

The Collector’s configuration is not our main focus here, since based on the nature of this feature it is minimal, letting workloads define how they should be monitored.

To illustrate the point, here is the Collector configuration snippet that enables the feature for both logs and metrics:

receivers:
    receiver_creator/metrics:
      watch_observers: [k8s_observer]
      discovery:
        enabled: true
      receivers:

    receiver_creator/logs:
      watch_observers: [k8s_observer]
      discovery:
        enabled: true
      receivers:

You can include the above in the EDOT’s Collector configuration, specifically the receivers’ section.

Since logs collection in our examples will happen from the discovery feature make sure that the static filelog receiver configuration block is removed and its preset is disabled (i.e. set to false) to avoid having log duplication.

Make sure that the receiver creator is properly added in the pipelines for logs (in addition to removing the filelog receiver completely) and metrics respectively.

Ensure that k8sobserver is enabled as part of the extensions:

extensions:
  k8s_observer:
    observe_nodes: true
    observe_services: true
    observe_ingresses: true

// ...

service:
  extensions: [k8s_observer]

Last but not least, ensure the log files' volume is mounted properly:

volumeMounts:
 - name: varlogpods
   mountPath: /var/log/pods
   readOnly: true

volumes:
  - name: varlogpods
    hostPath:
      path: /var/log/pods

Once the configuration is ready follow the Kubernetes quickstart guides on how to deploy the EDOT Collector. Make sure to replace the values.yaml file linked in the quickstart guide with the file that includes the above-described modifications.

Collecting Metrics from Moving Targets Based on Their Annotations

In this example, we have a Deployment with a Pod spec that consists of two different containers. One container runs a Redis server, while the other runs an NGINX server. Consequently, we want to provide different hints for each of these target containers.

The annotation-based discovery feature supports this, allowing us to specify metrics annotations per exposed container port.

Here is how the complete spec file looks:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    user  nginx;
    worker_processes  1;
    error_log  /dev/stderr warn;
    pid        /var/run/nginx.pid;
    events {
      worker_connections  1024;
    }
    http {
      include       /etc/nginx/mime.types;
      default_type  application/octet-stream;

      log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
      access_log  /dev/stdout main;
      server {
          listen 80;
          server_name localhost;

          location /nginx_status {
              stub_status on;
          }
      }
      include /etc/nginx/conf.d/*;
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deployment
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
      annotations:
        # redis container port hints
        io.opentelemetry.discovery.metrics.6379/enabled: "true"
        io.opentelemetry.discovery.metrics.6379/scraper: redis
        io.opentelemetry.discovery.metrics.6379/config: |
          collection_interval: "20s"
          timeout: "10s"

        # nginx container port hints
        io.opentelemetry.discovery.metrics.80/enabled: "true"
        io.opentelemetry.discovery.metrics.80/scraper: nginx
        io.opentelemetry.discovery.metrics.80/config: |
          endpoint: "http://`endpoint`/nginx_status"
          collection_interval: "30s"
          timeout: "20s"
    spec:
      volumes:
      - name: nginx-conf
        configMap:
          name: nginx-conf
          items:
            - key: nginx.conf
              path: nginx.conf
      containers:
        - name: webserver
          image: nginx:latest
          ports:
            - containerPort: 80
              name: webserver
          volumeMounts:
            - mountPath: /etc/nginx/nginx.conf
              readOnly: true
              subPath: nginx.conf
              name: nginx-conf
        - image: redis
          imagePullPolicy: IfNotPresent
          name: redis
          ports:
            - name: redis
              containerPort: 6379
              protocol: TCP

When this workload is deployed, the Collector will automatically discover it and identify the specific annotations. After this, two different receivers will be started, each one responsible for each of the target containers.

Collecting Logs from Multiple Target Containers

The annotation-based discovery feature also supports log collection based on the provided annotations. In the example below, we again have a Deployment with a Pod consisting of two different containers, where we want to apply different log collection configurations. We can specify annotations that are scoped to individual container names:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-logs-deployment
  labels:
    app: busybox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
      annotations:
        io.opentelemetry.discovery.logs.lazybox/enabled: "true"
        io.opentelemetry.discovery.logs.lazybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints-lazybox
        io.opentelemetry.discovery.logs.busybox/enabled: "true"
        io.opentelemetry.discovery.logs.busybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints-busybox
    spec:
      containers:
        - name: busybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs from busybox at $(date +%H:%M:%S)" && sleep 5s; done
        - name: lazybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs from lazybox at $(date +%H:%M:%S)" && sleep 25s; done

The above configuration enables two different filelog receiver instances, each applying a unique parsing configuration. This is handy when we know how to parse specific technology logs, such as Apache server access logs.

Combining Both Metrics and Logs Collection

In our third example, we illustrate how to define both metrics and log annotations on the same workload. This allows us to collect both signals from the discovered workload. Below is a Deployment with a Pod consisting of a Redis server and a BusyBox container that performs dummy log writing. We can target annotations to the port and container levels to collect metrics from the Redis server using the Redis receiver, and logs from the BusyBox using the filelog receiver. Here’s how:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deployment
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
      annotations:
        io.opentelemetry.discovery.metrics.6379/enabled: "true"
        io.opentelemetry.discovery.metrics.6379/scraper: redis
        io.opentelemetry.discovery.metrics.6379/config: |
          collection_interval: "20s"
          timeout: "10s"

        io.opentelemetry.discovery.logs.busybox/enabled: "true"
        io.opentelemetry.discovery.logs.busybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints
    spec:
      containers:
        - image: redis
          imagePullPolicy: IfNotPresent
          name: redis
          ports:
            - name: redis
              containerPort: 6379
              protocol: TCP
        - name: busybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs at $(date +%H:%M:%S)" && sleep 15s; done

Explore and analyse data coming from dynamic targets in Elastic

Once the target Pods are discovered and the Collector has started collecting telemetry data from them, we can then explore this data in Elastic. In Discover we can search for Redis and NGINX metrics as well as logs collected from the Busybox container. Here is how it looks like:

Summary

The examples above showcase how users of our OpenTelemetry Collector can take advantage of this new feature — one we played a major role in developing.

For this, we leveraged our years of experience with similar features already supported in Metricbeat, Filebeat, and Elastic-Agent. This makes us extremely happy and confident, as it closes the feature gap between Elastic's specific monitoring agents and the OpenTelemetry Collector — making it even better.

Interested in learning more? Visit the documentation and give it a try by following our EDOT quickstart guide.

Elastic provides the ability to ingest OpenTelemetry data with native OTel SDKs, the off the shelf OTel collector, or even Elastic’s Distributions of OpenTelemetry (EDOT). EDOT enables you to bring in logs, metrics and traces for your GenAI application and for K8s. However you will also generally need libraries to help trace specific components in your application. In tracing GenAI applications you can pick from a large set of libraries.

• OpenTelemetry OpenAI Instrumentation-v2 - allows tracing LLM requests and logging of messages made by the OpenAI Python API library. (note v2 is built by OpenTelemetry, the non v2 version is from a specific vendor and not OpenTelemetry)

• OpenTelemetry VertexAI Instrumentation - allows tracing LLM requests and logging of messages made by the VertexAI Python API library

• Langtrace - commercially available library which supports all LLMs in one library, and all traces are also OTel native.

• Elastic’s EDOT - which recently added tracing. See blog.

As you can see OpenTelemetry is the defacto mechanism that is converging to collect and ingest. OpenTelemetry is growing its support for this but it is also early days.

In this blog, we will walk through how to, with minimal code, observe a RAG based chatbot application with tracing using Langtrace. We previously covered Langtrace in a blog to highlight tracing Langchain.

In this blog we used langtrace OpenAI, Amazon Bedrock, Cohere, and others in one library.

Pre-requisites:

In order to follow along, these few pre-requisites are needed

• An Elastic Cloud account — sign up now, and become familiar with Elastic’s OpenTelemetry configuration. With Serverless no version required. With regular cloud minimally 8.17

• Git clone the RAG based Chatbot application and go through the tutorial on how to bring it up and become more familiar.

• An account on your favorite LLM (OpenAI, AzureOpen AI, etc), with API keys

• Be familiar with EDOT to understand how we bring in logs, metrics, and traces from the application through the OTel Collector

• Kubernetes cluster - I’ll be using Amazon EKS

• Look at Langtrace documentation also.

Application OpenTelemetry output in Elastic

Chatbot-rag-app

The first item that you will need to get up and running is the ChatBotApp, and once up you should see the following:

As you select some of the questions you will set a response based on the index that was created in Elasticsearch when the app initializes. Additionally there will be queries that are made to LLMs.

Traces, logs, and metrics from EDOT in Elastic

Once you have OTel Collector with EDOT configuration on your K8s cluster, and Elastic Cloud up and running you should see the following:

Logs:

In Discover you will see logs from the Chatbotapp, and be able to analyze the application logs, any specific log patterns (saves you time in analysis), and view logs from K8s.

Traces:

In Elastic Observability APM, you can also see tha chatbot details, which include transactions, dependencies, logs, errors, etc.

When you look at traces, you will be able to see the chatbot interactions in the trace.

1. You will see the end to end http call

2. Individual calls to elasticsearch

3. Specific calls such as invoke actions, and calls to the LLM

You can also get individual details of the traces, and look at related logs, and metrics related to that trace,

Metrics:

In addition to logs, and traces, any instrumented metrics will also get ingested into Elastic.

Setting it all up

In order to properly set up the Chatbot-app on K8s with telemetry sent over to Elastic, a few things must be set up:

1. Git clone the chatbot-rag-app, and modify one of the python files.

2. Next create a docker container that can be used in Kubernetes. The Docker build here in the Chatbot-app is good to use.

3. Collect all needed env variables. In this example we are using OpenAI, but the files can be modified for any of the LLMs. Hence you will have to get a few environmental variables loaded into the cluster. In the github repo there is a env.example for docker. You can pick and chose what is needed or not needed and adjust appropriately in the K8s file below.

4. Set up your K8s Cluster, and then install the OpenTelemetry collector with the appropriate yaml file and credentials. This will help collect K8s cluster logs and metrics also.

5. Utilize the two yaml files listed below to ensure you can run it on Kubernetes.

• Init-index-job.yaml - Initiates the index in elasticsearch with the local corporate information

• k8s-deployment-chatbot-rag-app.yaml - initializes the application frontend and backend.

1. Open the app on the load balancer URL against the chatbot-app service in K8s

2. Go to Elasticsearch and look at Discover for logs, go to APM and look for your chatbot-app and review the traces, and finally.

Modify the code for tracing with Langtrace

Once you curl the app and untar, go to the chatbot-rag-app directory:

curl https://codeload.github.com/elastic/elasticsearch-labs/tar.gz/main | 
tar -xz --strip=2 elasticsearch-labs-main/example-apps/chatbot-rag-app
cd elasticsearch-labs-main/example-apps/chatbot-rag-app

Next open the app.py file in the api directory and add the following

from opentelemetry.instrumentation.flask import FlaskInstrumentor

from langtrace_python_sdk import langtrace

langtrace.init(batch=False)

FlaskInstrumentor().instrument_app(app)

into the code:

import os
import sys
from uuid import uuid4

from chat import ask_question
from flask import Flask, Response, jsonify, request
from flask_cors import CORS

from opentelemetry.instrumentation.flask import FlaskInstrumentor

from langtrace_python_sdk import langtrace

langtrace.init(batch=False)

app = Flask(__name__, static_folder="../frontend/build", static_url_path="/")
CORS(app)

FlaskInstrumentor().instrument_app(app)

@app.route("/")

See the items in BOLD which will add in the langtrace library, and the opentelemetry flask instrumentation. This combination will provide and end to end trace for the https call all the way down to the calls to Elasticsearch, and to OpenAI (or other LLMs).

Create the docker container

Use the Dockerfile that is in the chatbot-rag-app directory as is and add the following line:

RUN pip3 install --no-cache-dir langtrace-python-sdk

into the Dockerfile:

COPY requirements.txt ./requirements.txt
RUN pip3 install -r ./requirements.txt
RUN pip3 install --no-cache-dir langtrace-python-sdk
COPY api ./api
COPY data ./data

EXPOSE 4000

This enables the langtrace-python-sdk to be installed into the docker container so the langtrace libraries can be used properly.

Collecting the proper env variables:

First collect the env variables from Elastic:

Envs for index initialization in Elastic:

ELASTICSEARCH_URL=https://aws.us-west-2.aws.found.io
ELASTICSEARCH_USER=elastic
ELASTICSEARCH_PASSWORD=elastic

# The name of the Elasticsearch indexes
ES_INDEX=workplace-app-docs
ES_INDEX_CHAT_HISTORY=workplace-app-docs-chat-history

The ELASTICSEARCH_URL can be found in cloud.elastic.co when you bring up your instance. The user and password, you will need to setup in Elastic.

Envs for sending the OTel instrumentation you will need the following:

OTEL_EXPORTER_OTLP_ENDPOINT="https://123456789.apm.us-west-2.aws.cloud.es.io:443"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer xxxxx"

These credentials are found in Elastic under APM integration and under OpenTelemetry

Envs for LLMs

In this example we’re using OpenAI, hence only three variables are needed.

LLM_TYPE=openai
OPENAI_API_KEY=XXXX
CHAT_MODEL=gpt-4o-mini

All these variables will be needed in the Kubernetes yamls in the next step

Setup K8s cluster and load up OTel Collector with EDOT

This step is outlined in the following Blog. It’s a simple three step process.

This step will bring in all the K8s cluster logs and metrics and setup the OTel collector.

Setup secrets, initialize indices, and start the app

Now that the cluster is up, and you have your environmental variables, you will need to

1. Install and run the k8s-deployments.yaml with the variables

2. Initialize the index

Essentially run the following:

kubectl create -f k8s-deployment.yaml
kubectl create -f init-index-job.yaml

Here are the two yamls you should use. Also found here

k8s-deployment.yaml

apiVersion: v1
kind: Secret
metadata:
  name: genai-chatbot-langtrace-secrets
type: Opaque
stringData:
  OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer%20xxxx"
  OTEL_EXPORTER_OTLP_ENDPOINT: "https://1234567.apm.us-west-2.aws.cloud.es.io:443"
 ELASTICSEARCH_URL: "YOUR_ELASTIC_SEARCH_URL"
  ELASTICSEARCH_USER: "elastic"
  ELASTICSEARCH_PASSWORD: "elastic"
  OPENAI_API_KEY: "XXXXXXX"  

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: genai-chatbot-langtrace
spec:
  replicas: 2
  selector:
    matchLabels:
      app: genai-chatbot-langtrace
  template:
    metadata:
      labels:
        app: genai-chatbot-langtrace
    spec:
      containers:
      - name: genai-chatbot-langtrace
        image:65765.amazonaws.com/genai-chatbot-langtrace2:latest
        ports:
        - containerPort: 4000
        env:
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: OTEL_SDK_DISABLED
          value: "false"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=genai-chatbot-langtrace,service.version=0.0.1,deployment.environment=dev"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "http/protobuf"
        envFrom:
        - secretRef:
            name: genai-chatbot-langtrace-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

---
apiVersion: v1
kind: Service
metadata:
  name: genai-chatbot-langtrace-service
spec:
  selector:
    app: genai-chatbot-langtrace
  ports:
  - port: 80
    targetPort: 4000
  type: LoadBalancer

Init-index-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: init-elasticsearch-index-test
spec:
  template:
    spec:
      containers:
      - name: init-index
#update your image location for chatbot rag app
        image: your-image-location:latest
        workingDir: /app/api
        command: ["python3", "-m", "flask", "--app", "app", "create-index"]
        env:
        - name: FLASK_APP
          value: "app"
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: ES_INDEX
          value: "workplace-app-docs"
        - name: ES_INDEX_CHAT_HISTORY
          value: "workplace-app-docs-chat-history"
        - name: ELASTICSEARCH_URL
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_URL
        - name: ELASTICSEARCH_USER
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_USER
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_PASSWORD
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
      restartPolicy: Never
  backoffLimit: 4

Open App with LoadBalancer URL

Run the kubectl get services command and get the URL for the chatbot app

% kubectl get services
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                                                                     AGE
chatbot-langtrace-service            LoadBalancer   10.100.130.44    xxxxxxxxx-1515488226.us-west-2.elb.amazonaws.com   80:30748/TCP                                                                6d23h

Play with app and review telemetry in Elastic

Once you go to the URL, you should see all the screens we described earlier in the beginning of this blog.

Conclusion

With Elastic's Chatbot-rag-app you have an example of how to build out a OpenAI driven RAG based chat application. However, you still need to understand how well it performs, whether its working properly, etc. Using OTel, Elastic’s EDOT and Langtrace gives you the ability to achieve this. Additionally, you will generally run this application on Kubernetes. Hopefully this blog provides the outline of how to achieve this.

Here are the other Tracing blogs:

App Observability with LLM (Tracing)-

• Observing LangChain with Langtrace and OpenTelemetry

• Observing LangChain with OpenLit Tracing

• Instrumenting LangChain with OpenTelemetry

LLM Observability -

• Elevate LLM Observability with GCP Vertex AI Integration

• LLM Observability on AWS Bedrock

• LLM Observability for Azure OpenAI

• LLM Observability for Azure OpenAI v2

This app is also now insturmented with EDOT, and you can visualize the Chatbot's traces to OpenAI, as well as relevant logs, and metrics from the application. By running the app as instructed in the github repo with Docker you can see these traces on a local stack. But how about running it against serverless, Elastic cloud or even with Kubernetes?

In this blog we will walk through how to set up Elastic's RAG Based Chatbot application with Elastic cloud and Kubernetes.

Prerequisites:

In order to follow along, these few pre-requisites are needed

• An Elastic Cloud account — sign up now, and become familiar with Elastic's OpenTelemetry configuration. With Serverless no version required. With regular cloud minimally 8.17

• Git clone the RAG based Chatbot application and go through the tutorial on how to bring it up and become more familiar and how to bring up the application using Docker.

• An account on OpenAI with API keys

• Kubernetes cluster to run the RAG based Chatbot app

• The instructions in this blog are also found in observability-examples in github.

Application OpenTelemetry output in Elastic

Chatbot-rag-app

The first item that you will need to get up and running is the ChatBotApp, and once up you should see the following:

As you select some of the questions you will set a response based on the index that was created in Elasticsearch when the app initializes. Additionally there will be queries that are made to LLMs.

Traces, logs, and metrics from EDOT in Elastic

Once you have the application running on your K8s cluster or with Docker, and Elastic Cloud up and running you should see the following:

Logs:

In Discover you will see logs from the Chatbotapp, and be able to analyze the application logs, any specific log patterns, which saves you time in analysis.

Traces:

In Elastic Observability APM, you can also see tha chatbot details, which include transactions, dependencies, logs, errors, etc.

When you look at traces, you will be able to see the chatbot interactions in the trace.

1. You will see the end to end http call

2. Individual calls to elasticsearch

3. Specific calls such as invoke actions, and calls to the LLM

You can also get individual details of the traces, and look at related logs, and metrics related to that trace,

Metrics:

In addition to logs, and traces, any instrumented metrics will also get ingested into Elastic.

Setting it all up with Docker

In order to properly set up the Chatbot-app on Docker with telemetry sent over to Elastic, a few things must be set up:

1. Git clone the chatbot-rag-app

2. Modify the env file as noted in the github README with the following exception:

Use your Elastic cloud's OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADER instead.

You can find these in the Elastic Cloud under integrations->APM

Envs for sending the OTel instrumentation you will need the following:

OTEL_EXPORTER_OTLP_ENDPOINT="https://123456789.apm.us-west-2.aws.cloud.es.io:443"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20xxxxx"

Notice the %20 in the headers. This will be needed to account for the space in credentials.

1. Set the following to false - OTEL_SDK_DISABLED=false

2. Set the envs for LLMs

In this example we're using OpenAI, hence only three variables are needed.

LLM_TYPE=openai
OPENAI_API_KEY=XXXX
CHAT_MODEL=gpt-4o-mini

1. Run the docker container as noted

docker compose up --build --force-recreate

1. Play with the app at localhost:4000

2. Then log into Elastic cloud and see the output as shown previously.

Run chatbot-rag-app on Kubernetes

In order to set this up, you can follow the following repo on Observability-examples which has the Kubernetes yaml files being used. These will also point to Elastic Cloud.

1. Set up the Kubernetes Cluster (we're using EKS)

2. Get the appropriate ENV variables:

• Find the OTEL_EXPORTER_OTLP_ENDPOINT/HEADER variables as noted in the pervious for Docker.

• Get your OpenAI Key

• Elasticsearch URL, and username and password.

1. Follow the instructions in the following github repo in observability examples to run two Kubernetes yaml files.

Essentially you need only replace the secret variables in k8s-deployment.yaml, and run

kubectl create -f k8s-deployment.yaml
kubectl create -f init-index-job.yaml

The app needs to be running first, then we use the app to initialize Elasticsearch with indices for the app.

Init-index-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: init-elasticsearch-index-test
spec:
  template:
    spec:
      containers:
      - name: init-index
        image: ghcr.io/elastic/elasticsearch-labs/chatbot-rag-app:latest
        workingDir: /app/api
        command: ["python3", "-m", "flask", "--app", "app", "create-index"]
        env:
        - name: FLASK_APP
          value: "app"
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: ES_INDEX
          value: "workplace-app-docs"
        - name: ES_INDEX_CHAT_HISTORY
          value: "workplace-app-docs-chat-history"
        - name: ELASTICSEARCH_URL
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_URL
        - name: ELASTICSEARCH_USER
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_USER
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_PASSWORD
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
      restartPolicy: Never
  backoffLimit: 4

k8s-deployment.yaml

apiVersion: v1
kind: Secret
metadata:
  name: chatbot-regular-secrets
type: Opaque
stringData:
  ELASTICSEARCH_URL: "https://yourelasticcloud.es.us-west-2.aws.found.io"
  ELASTICSEARCH_USER: "elastic"
  ELASTICSEARCH_PASSWORD: "elastic"
  OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer%20xxxx"
  OTEL_EXPORTER_OTLP_ENDPOINT: "https://12345.apm.us-west-2.aws.cloud.es.io:443"
  OPENAI_API_KEY: "YYYYYYYY"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chatbot-regular
spec:
  replicas: 2
  selector:
    matchLabels:
      app: chatbot-regular
  template:
    metadata:
      labels:
        app: chatbot-regular
    spec:
      containers:
      - name: chatbot-regular
        image: ghcr.io/elastic/elasticsearch-labs/chatbot-rag-app:latest
        ports:
        - containerPort: 4000
        env:
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=chatbot-regular,service.version=0.0.1,deployment.environment=dev"
        - name: OTEL_SDK_DISABLED
          value: "false"
        - name: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT
          value: "true"
        - name: OTEL_EXPERIMENTAL_RESOURCE_DETECTORS
          value: "process_runtime,os,otel,telemetry_distro"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "http/protobuf"
        - name: OTEL_METRIC_EXPORT_INTERVAL
          value: "3000"
        - name: OTEL_BSP_SCHEDULE_DELAY
          value: "3000"
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

---
apiVersion: v1
kind: Service
metadata:
  name: chatbot-regular-service
spec:
  selector:
    app: chatbot-regular
  ports:
  - port: 80
    targetPort: 4000
  type: LoadBalancer

Open App with LoadBalancer URL

Run the kubectl get services command and get the URL for the chatbot app

% kubectl get services
NAME                                 TYPE           CLUSTER-IP    EXTERNAL-IP                                                               PORT(S)                                                                     AGE
chatbot-regular-service            LoadBalancer   10.100.130.44    xxxxxxxxx-1515488226.us-west-2.elb.amazonaws.com   80:30748/TCP                                                                6d23h

1. Play with app and review telemetry in Elastic

2. Once you go to the URL, you should see all the screens we described earlier in the beginning of this blog.

Conclusion

With Elastic's Chatbot-rag-app you have an example of how to build out a OpenAI driven RAG based chat application. However, you still need to understand how well it performs, whether its working properly, etc. Using OTel and Elastic’s EDOT gives you the ability to achieve this. Additionally, you will generally run this application on Kubernetes. Hopefully this blog provides the outline of how to achieve this. Here are the other Tracing blogs:

App Observability with LLM (Tracing)-

• Observing LangChain with Langtrace and OpenTelemetry

• Observing LangChain with OpenLit Tracing

• Instrumenting LangChain with OpenTelemetry

LLM Observability -

• Elevate LLM Observability with GCP Vertex AI Integration

• LLM Observability on AWS Bedrock

• LLM Observability for Azure OpenAI

• LLM Observability for Azure OpenAI v2

Last year, we announced Elastic Distribution of OpenTelemetry (a.k.a. EDOT) language SDKs, which collect logs, traces and metrics from applications. When this was announced, we didn’t yet support Large Language Model (LLM) providers such as OpenAI. This limited insight developers had into Generative AI (GenAI) applications.

In a prior post, we reviewed LLM observability focus, such as token usage, chat latency and knowing which tools (like DuckDuckGo) your application uses. With the right logs, traces and metrics, developers can answer questions like "Which version of a model generated this response?" or "What was the exact chat prompt created by my RAG application?"

In the last six months, Elastic invested a lot of energy alongside others in the OpenTelemetry community towards shared specifications on these areas, including code to collect LLM related logs, metrics and traces. Our goal was to extend the zero code (agent) approach EDOT brings to GenAI use cases.

Today, we announce our first GenAI instrumentation capability in the EDOT language SDKs: OpenAI. Below, you’ll see how to observe GenAI applications using our Python, Node.js and Java EDOT SDKs.

Example application

Many of us may be familiar with ChatGPT, which is frontend for OpenAI’s GPT model family. Using this, you can ask a question and the assistant might reply correctly depending on what you ask and text the LLM was trained on.

Here’s an example of an esoteric question answered by ChatGPT:

Our example application will simply ask this predefined question and print the result. We’ll write it in three languages: Python, JavaScript and Java.

We’ll execute each with a "zero code" (agent) approach, so that logs, metrics and traces are captured and visible in an Elastic Stack configured with Kibana and APM server. If you don’t have a stack running, use instructions from Elasticsearch Labs to set one up.

Regardless of programming language, three variables are needed: the OpenAI API key, the location of your Elastic APM server, and the service name of the application. You’ll write these to a file named .env.

OPENAI_API_KEY=sk-YOUR_API_KEY
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8200
OTEL_SERVICE_NAME=openai-example

By default instrumentations does not capture the content sent to the OpenAI API in the GenAI events sent to logs, if you want to capture it add the following:

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

Each time the application is run, it sends logs, traces and metrics to the APM server, which you can find by querying Kibana like this for the application "openai-example"

http://localhost:5601/app/apm/services/openai-example/transactions

When you choose a trace, you’ll see the LLM request made by the OpenAI SDK, and HTTP traffic caused by it:

Select the logs tab to see the exact request and response to OpenAI. This data is critical for Q/A and evaluation use cases.

You can also go to the Metrics Explorer and make a graph of "gen_ai.client.token.usage" or "gen_ai.client.operation.duration" over all the times you ran the application:

http://localhost:5601/app/metrics/explorer

Continue to see exactly how this application looks and is run, in Python, Java and Node.js. Those already using our EDOT language SDKs will be familiar with how this works.

Python

Assuming you have python installed, the first thing would be to setup a virtual environment and install the required packages: the OpenAI client, a helper tool to read the .env file and our EDOT Python package:

python3 -m venv .venv
source .venv/bin/activate
pip install openai "python-dotenv[cli]" elastic-opentelemetry

Next, run edot-bootstrap which analyzes the code to install any relevant instrumentation available:

edot-bootstrap —-action=install

Now, create your .envfile, as described earlier in this article, and the below source code in chat.py

import os

import openai

CHAT_MODEL = os.environ.get("CHAT_MODEL", "gpt-4o-mini")


def main():
  client = openai.Client()

  messages = [
    {
      "role": "user",
        "content": "Answer in up to 3 words: Which ocean contains Bouvet Island?",
    }
  ]

  chat_completion = client.chat.completions.create(model=CHAT_MODEL, messages=messages)
  print(chat_completion.choices[0].message.content)

if __name__ == "__main__":
  main()

Now you can run everything with:

dotenv run -- opentelemetry-instrument python chat.py

Finally, look for a trace for the service named "openai-example" in Kibana. You should see a transaction named "chat gpt-4o-mini".

Rather than copy/pasting above, you can find a working copy of this example (along with the instructions) in the Python EDOT repository here.

Finally, if you would like to try a more comprehensive example, take a look at chatbot-rag-app which uses OpenAI with Elasticsearch’s Elser retrieval model.

Java

There are multiple popular ways to initialize a Java project. Since we are using OpenAI, the first step is to configure the dependency com.openai:openai-java and write the below source as Chat.java.

package openai.example;

import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.*;


final class Chat {

  public static void main(String[] args) {
    String chatModel = System.getenv().getOrDefault("CHAT_MODEL", "gpt-4o-mini");

    OpenAIClient client = OpenAIOkHttpClient.fromEnv();

    String message = "Answer in up to 3 words: Which ocean contains Bouvet Island?";
    ChatCompletionCreateParams params = ChatCompletionCreateParams.builder()
        .addMessage(ChatCompletionUserMessageParam.builder()
          .content(message)
          .build())
        .model(chatModel)
        .build();

    ChatCompletion chatCompletion = client.chat().completions().create(params);
    System.out.println(chatCompletion.choices().get(0).message().content().get());
  }
}

Build the project such that all dependencies are in a single jar. For example, if using Gradle, you would use the com.gradleup.shadowplugin.

Next, create your .envfile, as described earlier, and download shdotenv which we’ll use to load it.

curl -O -L https://github.com/ko1nksm/shdotenv/releases/download/v0.14.0/shdotenv
chmod +x ./shdotenv

At this point, you have a jar and configuration you can use to run the OpenAI example. The next step is to download the EDOT Java javaagent binary. This is the part that records and exports logs, metrics and traces.

curl -o elastic-otel-javaagent.jar -L 'https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=co.elastic.otel&a=elastic-otel-javaagent&v=LATEST'

Assuming you assembled a file named openai-example-all.jar, run it with EDOT like this:

./shdotenv java -javaagent:elastic-otel-javaagent.jar -jar openai-example-all.jar

Finally, look for a trace for the service named "openai-example" in Kibana. You should see a transaction named "chat gpt-4o-mini".

Rather than copy/pasting above, you can find a working copy of this example in the EDOT Java source repository here.

Node.js

Assuming you already have npm installed and configured, run the following commands to initialize a project for the example. This includes the openai package and @elastic/opentelemetry-node (EDOT Node.js)

npm init -y
npm install openai @elastic/opentelemetry-node

Next, create your .env file, as described earlier in this article and the below source code in index.js

const {OpenAI} = require('openai');

let chatModel = process.env.CHAT_MODEL ?? 'gpt-4o-mini';

async function main() {
 const client = new OpenAI();
 const completion = await client.chat.completions.create({
  model: chatModel,
  messages: [
   {
    role: 'user',
    content: 'Answer in up to 3 words: Which ocean contains Bouvet Island?',
   },
  ],
 });
 console.log(completion.choices[0].message.content);
}

main();

With this in place, run the above source with EDOT like this:

node --env-file .env --require @elastic/opentelemetry-node index.js

Finally, look for a trace for the service named "openai-example" in Kibana. You should see a transaction named "chat gpt-4o-mini".

Rather than copy/pasting above, you can find a working copy of this example in the EDOT Node.js source repository here.

Finally, if you would like to try a more comprehensive example, take a look at openai-embeddings which uses OpenAI with Elasticsearch as a vector database!

Closing Notes

Above you’ve seen how to observe the official OpenAI SDK in three different languages, using Elastic Distribution of OpenTelemetry (EDOT).

It is important to note that some of the OpenAI SDKs and also OpenTelemetry specifications around generative AI are experimental. If you find this helps you, or find glitches, please join our slack and let us know about it.

Several LLM platforms accept requests from the OpenAI client SDK, by setting OPENAI_BASE_URL and choosing relevant models. During development, we tested against OpenAI Platform and Azure OpenAI Service. We also ran integration tests against Ollama, contributing improvements its OpenAI support released in v0.5.12. Whatever your choice of OpenAI compatible platform, we hope this new tooling helps you understand your LLM usage.

Finally, while the first Generative AI SDK instrumented with EDOT is OpenAI, you’ll see more soon. We are already working on Bedrock, and collaborating with others in the OpenTelemetry community for other platforms. Keep watching this blog for exciting updates.

With the release of an upgraded vSphere integration we now support an enhanced set of metrics and datastreams. Package version 1.15.0 onwards introduces new datastreams that significantly improve the collection of performance metrics, providing deeper insights into your vSphere environment.

This enhanced version includes a total of seven datastreams, featuring critical new metrics such as disk performance, memory utilization, and network status. Additionally, these datastreams now offer detailed visibility into associated resources like hosts, clusters, and resource pools. To make the most of these insights, we’ve also introduced prebuilt dashboards, helping teams monitor and troubleshoot their vSphere environments with ease and precision.

We have expanded the performance metrics to encompass a broader range of insights across all datastreams, while also introducing new datastreams for clusters, resource pools, and networks. This enhanced integration version now includes a total of seven datastreams, featuring critical new metrics such as disk performance, memory utilization, and network status. Additionally, these datastreams now offer detailed visibility into associated resources like hosts, clusters, and resource pools.

Each datastream also includes detailed alarm information, such as the alarm name, description, status (e.g. critical or warning), and the affected entity's name. To make the most of these insights, we’ve also introduced prebuilt dashboards, helping teams monitor and troubleshoot their vSphere environments with ease and precision.

Overview of the Datastreams

• Host Datastream: This datastream monitors the disk performance of the host, including metrics such as disk latency, average read/write bytes, uptime, and status. It also captures network metrics, such as packet information, network bandwidth, and utilization, as well as CPU and memory usage of the host. Additionally, it lists associated datastores, virtual machines, and networks within vSphere.

• Virtual Machine Datastream: This datastream tracks the used and available CPU and memory resources of virtual machines, along with the uptime and status of each VM. It includes information about the host on which the VM is running, as well as detailed snapshot metrics like the number of snapshots, creation dates, and descriptions. Additionally, it provides insights into associated hosts and datastores.

• Datastore Datastream: This datastream provides information on the total, used, and available capacity of datastores, along with their overall status. It also captures metrics such as the average read/write rate and lists the hosts and virtual machines connected to each datastore.

• Datastore Cluster: A datastore cluster in vSphere is a collection of datastores grouped together for efficient storage management. This datastream provides details on the total capacity and free space in the storage pod, along with the list of datastores within the cluster.

• Resource Pool: Resource pools in vSphere serve as logical abstractions that allow flexible allocation of CPU and memory resources. This datastream captures memory metrics, including swapped, ballooned, and shared memory, as well as CPU metrics like distributed and static CPU entitlement. It also lists the virtual machines associated with each resource pool.

• Network Datastream: This datastream captures the overall configuration and status of the network, including network types (e.g., vSS, vDS). It also lists the hosts and virtual machines connected to each network.

• Cluster Datastream: A Cluster in vSphere is a collection of ESXi hosts and their associated virtual machines that function as a unified resource pool. Clustering in vSphere allows administrators to manage multiple hosts and resources centrally, providing high availability, load balancing, and scalability to the virtual environment. This datastream includes metrics indicating whether HA or admission control is enabled and lists the hosts, networks, and datastores associated with the cluster.

Alarms support in vSphere Integration

Alarms are a vital part of the vSphere integration, providing real-time insights into critical events across your virtual environment. In the updated Elastic’s vSphere integration, alarms are now reported for all the entities. They include detailed information such as the alarm name, description, severity (e.g., critical or warning), affected entity, and triggered time. These alarms are seamlessly integrated into datastreams, helping administrators and SREs quickly identify and resolve issues like resource shortages or performance bottlenecks.

Example Alarm

"triggered_alarms": [
  {
    "description": "Default alarm to monitor host memory usage",
    "entity_name": "host_us",
    "id": "alarm-4.host-12",
    "name": "Host memory usage",
    "status": "red",
    "triggered_time": "2024-08-28T10:31:26.621Z"
  }
]

This example highlights a triggered alarm for monitoring host memory usage, indicating a critical status (red) for the host "host_us." Such alarms empower teams to act swiftly and maintain the stability of their vSphere environment.

Lets Try It Out!

The new vSphere integration in Elastic Cloud is more than just a monitoring tool; it’s a comprehensive solution that empowers you to manage and optimize your virtual environments effectively. With deeper insights and enhanced data granularity, you can ensure high availability, improved load balancing, and smarter resource allocation. Spin up an Elastic Cloud, and start monitoring your vSphere infrastructure.

• Native OTel Integrity: Elastic is now 100% OTel-native, retaining OTel data natively without requiring data translation This eliminates the need for SREs to handle tedious schema conversions and develop customized views. All Elastic Observability capabilities—such as entity discovery, entity-centric insights, APM, infrastructure monitoring, and AI-driven issue analysis— now seamlessly work with native OTel data.

• Powerful end to end OTel based Kubernetes observability with Elastic Distributions of OpenTelemetry (EDOT): Elastic now supports EDOT deployment and management on Kubernetes via the OTel Operator, enabling streamlined EDOT collector deployment, application auto-instrumentation, and lifecycle management. With out-of-the-box OTel-based Kubernetes integration and dashboards, SREs gain instant, real-time visibility into cluster and application metrics, logs, and traces—with no manual configuration needed.

For organizations, it signals our commitment to open standards, streamlined data collection, and delivering insights from native OpenTelemetry data. Bring the power of Elastic Observability to your Kubernetes and OpenTelemetry deployments for maximum visibility and performance. 

Fully native OTel architecture with in-depth data analysis

Elastic’s OpenTelemetry-first architecture is 100% OTel-native, fully retaining the OTel data model, including OTel Semantic Conventions and Resource attributes, so your observability data remains in OpenTelemetry standards. OTel data in Elastic is also backward compatible with the Elastic Common Schema (ECS).

SREs now gain a holistic view of resources, as Elastic accurately identifies entities through OTel resource attributes. For example, in a Kubernetes environment, Elastic identifies containers, hosts, and services and connects these entities to logs, metrics, and traces.

Once OTel data is in Elastic’s scalable vector datastore, Elastic’s capabilities such as the AI Assistant, zero-config machine learning-based anomaly detection, pattern analysis, and latency correlation empower SREs to quickly analyze and pinpoint potential issues in production environments.

Kubernetes insights with Elastic Distributions of OpenTelemetry (EDOT)

EDOT reduces manual effort through automated onboarding and pre-configured dashboards. With EDOT and OpenTelemetry, Elastic makes Kubernetes monitoring straightforward and accessible for organizations of any size.

EDOT paired with Elasticsearch,  enables storage for all signal types—logs, metrics, traces, and soon profiling—while maintaining essential resource attributes and semantic conventions.

Elastic’s OpenTelemetry-native solution enables customers to quickly extract insights from their data rather than manage complex infrastructure to ingest data. Elastic automates the deployment and configuration of observability components to deliver a user experience focused on ease and scalability, making it well-suited for large-scale environments and diverse industry needs.

Let’s take a look at how Elastic’s EDOT enables visibility into Kubernetes environments.

1. Simple 3-step OTel ingest with lifecycle management and auto-instrumentation 

Elastic leverages the upstream OpenTelemetry Operator to automate its EDOT lifecycle management—including deployment, scaling, and updates—allowing customers to focus on visibility into their Kubernetes infrastructure and applications instead of their observability infrastructure for data collection.

The Operator integrates with the EDOT Collector and language SDKs to provide a consistent, vendor-agnostic experience. For instance, when customers deploy a new application, they don’t need to manually configure instrumentation for various languages; the OpenTelemetry Operator manages this through auto-instrumentation, as supported by the upstream OpenTelemetry project.

This integration simplifies observability by ensuring consistent application instrumentation across the Kubernetes environment. Elastic’s collaboration with the upstream OpenTelemetry project strengthens this automation, enabling users to benefit from the latest updates and improvements in the OpenTelemetry ecosystem. By relying on open source tools like the OpenTelemetry Operator, Elastic ensures that its solutions stay aligned with the latest advancements in the OpenTelemetry project, reinforcing its commitment to open standards and community-driven development.

The diagram above shows how the operator can deploy multiple OTel collectors, helping SREs deploy individual EDOT Collectors for specific applications and infrastructure. This configuration improves availability for OTel ingest and the telemetry is sent directly to Elasticsearch servers via OTLP.

Check out our recent blog on how to set this up.

2. Out-of-the-box OTel-based Kubernetes integration with dashboards

Elastic delivers an OTel-based Kubernetes configuration for the OTel collector by packaging all necessary receivers, processors, and configurations for Kubernetes observability. This enables users to automatically collect, process, and analyze Kubernetes metrics, logs, and traces without the need to configure each component individually.

The OpenTelemetry Kubernetes Collector components provide essential building blocks, including receivers like the Kubernetes Receiver for cluster metrics, Kubeletstats Receiver for detailed node and container metrics, along with processors for data transformation and enrichment. By packaging these components, Elastic offers a turnkey solution that simplifies Kubernetes observability and eliminates the need for users to set up and configure individual collectors or processors.

This pre-packaged approach, which includes OTel-native Kibana assets such as dashboards, allows users to focus on analyzing their observability data rather than managing configuration details. Elastic’s Unified OpenTelemetry Experience ensures that users can harness OpenTelemetry’s full potential without needing deep expertise. Whether you’re monitoring resource usage, container health, or API server metrics, users gain comprehensive observability through EDOT.

For more details on OpenTelemetry Kubernetes Collector components, visit OpenTelemetry Collector Components.

3. Streamlined ingest architecture with OTel data and Elasticsearch

Elastic’s ingest architecture minimizes infrastructure overhead by enabling users to forward trace data directly into Elasticsearch with the EDOT Collector, removing the need for the Elastic APM server. This approach:

• Reduces the costs and complexity associated with maintaining additional infrastructure, allowing users to deploy, scale, and manage their observability solutions with fewer resources.

• Allows all OTel data, metrics, logs, and traces to be ingested and stored in Elastic’s singular vector database store enabling further analysis with Elastic’s AI-driven capabilities.

SREs can now reduce operational burdens while also gaining high performance analytics and observability insights provided by Elastic.

Elastic’s ongoing commitment to open source and OpenTelemetry

With Elasticsearch fully open source once again under the AGPL license,  this change reinforces our deep commitment to open standards and the open source community. This aligns with Elastic’s OpenTelemetry-first approach to observability, where Elastic Distributions of OpenTelemetry (EDOT) streamline OTel ingestion and schema auto-detection, providing real-time insights for Kubernetes and application telemetry.

As users increasingly adopt OTel as their schema and data collection architecture for observability, Elastic’s Distribution of OpenTelemetry (EDOT), currently in tech preview, enhances standard OpenTelemetry capabilities and improves troubleshooting while also serving as a commercially supported OTel distribution. EDOT, together with Elastic’s recent contributions of the Elastic Profiling Agent and Elastic Common Schema (ECS) to OpenTelemetry, reinforces Elastic’s commitment to establishing OpenTelemetry as the industry standard.

Customers can now embrace open standards and enjoy the advantages of an open, extensible platform that integrates seamlessly with their environment. End result?  Reduced costs, greater visibility, and vendor independence.

Getting hands-on with Elastic Observability and EDOT

Ready to try out the OTel Operator with EDOT collector and SDKs to see how Elastic utilizes ingested OTel data in APM, Discover, Analysis, and out-of-the-box dashboards? 

• Get an account on Elastic Cloud

• Learn about Elastic Distributions of OpenTelemetry Overview

• Utilize the OpenTelemetry Demo with EDOT

• Understand how you can monitor Kubernetes with EDOT

• Utilize the EDOT Operator and the EDOT OTel collector

If you have your own application and want to configure EDOT the application with auto-instrumentation, read the following blogs on Go, Java, PHP, Python

• Auto-Instrumenting Go Applications with OpenTelemetry

• Elastic Distribution OpenTelemetry Java Agent

• Elastic OpenTelemetry Distribution for PHP

• Elastic OpenTelemetry Distribution for Python

What is APM?

Application performance monitoring lets you see where your applications spend their time, what they are doing, what other applications or services they are calling, and what errors or exceptions they are encountering.

In addition, APM also lets you see history and trends for key performance indicators, such as latency and throughput, as well as transaction and dependency information:

Whether you're setting up alerts for SLA breaches, trying to gauge the impact of your latest release, or deciding where to make the next improvement, APM can help with your root-cause analysis to help improve your users' experience and drive your mean time to resolution (MTTR) toward zero.

Logical architecture

Elastic APM relies on the APM Integration inside Elastic Agent, which forwards application trace and metric data from applications instrumented with APM agents to an Elastic Observability cluster. Elastic APM supports multiple agent flavors:

• Native Elastic APM Agents, available for multiple languages, including Java, .NET, Go, Ruby, Python, Node.js, PHP, and client-side JavaScript
• Code instrumented with OpenTelemetry
• Code instrumented with OpenTracing
• Code instrumented with Jaeger

In this blog, we'll provide a quick example of how to instrument code with the native Elastic APM Python agent, but the overall steps are similar for other languages.

Please note that there is a strong distinction between the Elastic APM Agent and the Elastic Agent. These are very different components, as you can see in the diagram above, so it's important not to confuse them.

Install the Elastic Agent

The first step is to install the Elastic Agent. You either need Fleet installed first, or you can install the Elastic Agent standalone. Install the Elastic Agent somewhere by following this guide. This will give you an APM Integration endpoint you can hit. Note that this step is not necessary in Elastic Cloud, as we host the APM Integration for you. Check Elastic Agent is up by running:

curl <ELASTIC_AGENT_HOSTNAME>:8200

Instrumenting sample code with an Elastic APM agent

The instructions for the various language agents differ based on the programming language, but at a high level they have a similar flow. First, you add the dependency for the agent in the language's native spec, then you configure the agent to let it know how to find the APM Integration.

You can try out any flavor you'd like, but I am going to walk through the Python instructions using this Python example that I created.

Get the sample code (or use your own)

To get started, I clone the GitHub repository then change to the directory:

git clone https://github.com/davidgeorgehope/PythonElasticAPMExample
cd PythonElasticAPMExample

How to add the dependency

Adding the Elastic APM Dependency is simple — check the app.py file from the github repo and you will notice the following lines of code.

import elasticapm
from elasticapm import Client

app = Flask(__name__)
app.config["ELASTIC_APM"] = {    "SERVICE_NAME": os.environ.get("APM_SERVICE_NAME", "flask-app"),    "SECRET_TOKEN": os.environ.get("APM_SECRET_TOKEN", ""),    "SERVER_URL": os.environ.get("APM_SERVER_URL", "http://localhost:8200"),}
elasticapm.instrumentation.control.instrument()
client = Client(app.config["ELASTIC_APM"])

The Python library for Flask is capable of auto detecting transactions, but you can also start transactions in code as per the following, as we have done in this example:

@app.route("/")
def hello():
    client.begin_transaction('demo-transaction')
    client.end_transaction('demo-transaction', 'success')

Configure the agent

The agents need to send application trace data to the APM Integration, and to do this it has to be reachable. I configured the Elastic Agent to listen on my local host's IP, so anything in my subnet can send data to it. As you can see from the code below, we use docker-compose.yml to pass in the config via environment variables. Please edit these variables for your own Elastic installation.

# docker-compose.yml
version: "3.9"
services:
  flask_app:
    build: .
    ports:
      - "5001:5001"
    environment:
      - PORT=5001
      - APM_SERVICE_NAME=flask-app
      - APM_SECRET_TOKEN=your_secret_token
      - APM_SERVER_URL=http://host.docker.internal:8200

Some commentary on the above:

• service_name: If you leave this out it will just default to the application's name, but you can override that here.
• secret_token: Secret tokens allow you to authorize requests to the APM Server, but they require that the APM Server is set up with SSL/TLS and that a secret token has been set up. We're not using HTTPS between the agents and the APM Server, so we'll comment this one out.
• server_url: This is how the agent can reach the APM Integration inside Elastic Agent. Replace this with the name or IP of your host running Elastic Agent.

Now that the Elastic APM side of the configuration is done, we simply follow the steps from the README to start up.

docker-compose up --build -d

The build step will take several minutes.

You can navigate to the running sample application by visiting http://localhost:5001. There's not a lot to the sample, but it does generate some APM data. To generate a bit of a load, you can reload them a few times or run a quick little script:

#!/bin/bash
# load_test.sh
url="http://localhost:5001"
for i in {1..1000}
do
  curl -s -o /dev/null $url
  sleep 1
done

This will just reload the pages every second.

Back in Kibana, navigate back to the APM app (hamburger icon, then select APM ) and you should see our new flask-app service (I let mine run so it shows a bit more history):

The Service Overview page provides an at-a-glance summary of the health of a service in one place. If you're a developer or an SRE, this is the page that will help you answer questions like:

• How did a new deployment impact performance?
• What are the top impacted transactions?
• How does performance correlate with underlying infrastructure?

This view provides a list of all of the applications that have sent application trace data to Elastic APM in the specified period of time (in this case, the last 15 minutes). There are also sparklines showing mini graphs of latency, throughput, and error rate. Clicking on flask-app takes us to the service overview page, which shows the various transactions within the service (recall that my script is hitting the / endpoint, as seen in the Transactions section). We get bigger graphs for Latency , Throughput , Errors , and Error Rates.

When you're instrumenting real applications, under real load, you'll see a lot more connectivity (and errors!)

Clicking on a transaction in the transaction view, in this case, our sample app's demo-transaction transaction, we can see exactly what operations were called:

This includes detailed information about calls to external services, such as database queries:

What's next?

Now that you've got your Elastic Observability cluster up and running and collecting out-of-the-box application trace data, explore the public APIs for the languages that your applications are using, which allow you to take your APM data to the next level. The APIs allow you to add custom metadata, define business transactions, create custom spans, and more. You can find the public API specs for the various APM agents (such as Java, Ruby, Python, and more) on the APM agent documentation pages.

If you'd like to learn more about Elastic APM, check out our webinar on Elastic APM in the shift to cloud native to see other ways that Elastic APM can help you in your ecosystem.

If you decide that you'd rather have us host your observability cluster, you can sign up for a free trial of the Elasticsearch Service on Elastic Cloud and change your agents to point to your new cluster.

Originally published May 5, 2021; updated April 6, 2023.

Site reliability engineers (SREs) face the challenging task of sifting through vast quantities of data, not just from the applications themselves but also from the underlying infrastructure. In addition to data analysis, they are responsible for ensuring the effective use and development of operational tools. The growing volume of data, the daily resolution of issues, and the continuous evolution of tools and processes can detract from the focus on business performance.

Elastic Observability offers a solution to this challenge. It enables SREs to integrate and examine all telemetry data (logs, metrics, traces, and profiling) in conjunction with business metrics. This comprehensive approach to data analysis fosters operational excellence, boosts productivity, and yields critical insights, all of which are integral to maintaining high-performing applications in a demanding digital environment.

To help manage operations and business metrics, Elastic Observability's SLO (Service Level Objectives) feature was introduced in 8.12. This feature enables setting measurable performance targets for services, such as availability, latency, traffic, errors, and saturation or define your own. Key components include:

• Defining and monitoring SLIs (Service Level Indicators)

• Monitoring error budgets indicating permissible performance shortfalls

• Alerting on burn rates showing error budget consumption

Users can monitor SLOs in real-time with dashboards, track historical performance, and receive alerts for potential issues. Additionally, SLO dashboard panels offer customized visualizations.

Service Level Objectives (SLOs) are generally available for our Platinum and Enterprise subscription customers.

In this blog, we will outline the following:

• What are SLOs? A Google SRE perspective

• Several scenarios of defining and managing SLOs

Service Level Objective overview

Service Level Objectives (SLOs) are a crucial component for Site Reliability Engineering (SRE), as detailed in Google's SRE Handbook. They provide a framework for quantifying and managing the reliability of a service. The key elements of SLOs include:

• Service Level Indicators (SLIs): These are carefully selected metrics, such as uptime, latency, throughput, error rates, or other important metrics, that represent the aspects of the service and are important from an operations or business perspective. Hence, an SLI is a measure of the service level provided (latency, uptime, etc.), and it is defined as a ratio of good over total events, with a range between 0% and 100%.

• Service Level Objective (SLO): An SLO is the target value for a service level measured as a percentage by an SLI. Above the threshold, the service is compliant. As an example, if we want to use service availability as an SLI, with the number of successful responses at 99.9%, then any time the number of failed responses is > .1%, the SLO will be out of compliance.

• Error budget: This represents the threshold of acceptable errors, balancing the need for reliability with practical limits. It is defined as 100% minus the SLO quantity of errors that is tolerated.

• Burn rate: This concept relates to how quickly the service is consuming its error budget, which is the acceptable threshold for unreliability agreed upon by the service providers and its users.

Understanding these concepts and effectively implementing them is essential for maintaining a balance between innovation and reliability in service delivery. For more detailed information, you can refer to Google's SRE Handbook.

One main thing to remember is that SLO monitoring is not incident monitoring. SLO monitoring is a proactive, strategic approach designed to ensure that services meet established performance standards and user expectations. It involves tracking Service Level Objectives, error budgets, and the overall reliability of a service over time. This predictive method helps in preventing issues that could impact users and aligns service performance with business objectives.

In contrast, incident monitoring is a reactive process focused on detecting, responding to, and mitigating service incidents as they occur. It aims to address unexpected disruptions or failures in real time, minimizing downtime and impact on service. This includes monitoring system health, errors, and response times during incidents, with a focus on rapid response to minimize disruption and preserve the service's reputation.

Elastic®’s SLO capability is based directly off the Google SRE Handbook. All the definitions and semantics are utilized as described in Google’s SRE handbook. Hence users can perform the following on SLOs in Elastic:

• Define an SLO on an SLI such as KQL (log based query), service availability, service latency, custom metric, histogram metric, or a timeslice metric. Additionally, set the appropriate threshold.

• Utilize occurrence versus time slice based budgeting. Occurrences is the number of good events over the number of total events to compute the SLO. Timeslices break the overall time window into slammer slices of a defined duration and compute the number of good slices over the total slices to compute the SLO. Timeslice targets are more accurate and useful when calculating things like a service’s SLO when trying to meet agreed upon customer targets.

• Manage all the SLOs in a singular location.

• Trigger alerts from the defined SLO, whether the SLI is off, burn rate is used up, or the error rate is X.

• Create unique service level dashboards with SLO information for a more comprehensive view of the service.

SREs need to be able to manage business metrics.

SLOs based on logs: NGINX availability

Defining SLOs does not always mean metrics need to be used. Logs are a rich form of information, even when they have metrics embedded in them. Hence it’s useful to understand your business and operations status based on logs.

Elastic allows you to create an SLO based on specific fields in the log message, which don’t have to be metrics. A simple example is a simple multi-tier app that has a web server layer (nginx), a processing layer, and a database layer.

Let’s say that your processing layer is managing a significant number of requests. You want to ensure that the service is up properly. The best way is to ensure that all http.response.status_code are less than 500. Anything less ensures the service is up and any errors (like 404) are all user or client errors versus server errors.

If we use Discover in Elastic, we see that there are close to 2M log messages over a seven-day time frame.

Additionally, the number of messages with http.response.status_code > 500 is minimal, like 17K.

Rather than creating an alert, we can create an SLO with this query:

We chose to use occurrences as the budgeting method to keep things simple.

Once defined, we can see how well our SLO is performing over a seven-day time frame. We can see not only the SLO, but also the burn rate, the historical SLI, and error budget, and any specific alerts against the SLO.

Not only do we get information about the violation, but we also get:

• Historical SLI (7 days)

• Error budget burn down

• Good vs. bad events (24 hours)

We can see how we’ve easily burned through our error budget.

Hence something must be going on with nginx. To investigate, all we need to do is utilize the AI Assistant, and use its natural language interface to ask questions to help analyze the situation.

Let’s use Elastic’s AI Assistant to analyze the breakdown of http.response.status_code across all the logs from the past seven days. This helps us understand how many 50X errors we are getting.

As we can see, the number of 502s is minimal compared to the number of overall messages, but it is affecting our SLO.

However, it seems like Nginx is having an issue. In order to reduce the issue, we also ask the AI Assistant how to work on this error. Specifically, we ask if there is an internal runbook the SRE team has created.

AI Assistant gets a runbook the team has added to its knowledge base. I can now analyze and try to resolve or reduce the issue with nginx.

While this is a simple example, there are an endless number of possibilities that can be defined based on KQL. Some other simple examples:

• 99% of requests occur under 200ms

• 99% of log message are not errors

Application SLOs: OpenTelemetry demo cartservice

A common application developers and SREs use to learn about OpenTelemetry and test out Observability features is the OpenTelemetry demo.

This demo has feature flags to simulate issues. With Elastic’s alerting and SLO capability, you can also determine how well the entire application is performing and how well your customer experience is holding up when these feature flags are used.

Elastic supports OpenTelemetry by taking OTLP directly with no need for an Elastic specific agent. You can send in OpenTelemetry data directly from the application (through OTel libraries) and through the collector.

We’ve brought up the OpenTelemetry demo on a K8S cluster (AWS EKS) and turned on the cartservice feature flag. This inserts errors into the cartservice. We’ve also created two SLOs to monitor the cartservice’s availability and latency.

We can see that the cartservice’s availability is violated. As we drill down, we see that there aren’t as many successful transactions, which is affecting the SLO.

As we drill into the service, we can see in Elastic APM that there is a higher than normal failure rate of about 5.5% for the emptyCart service.

We can investigate this further in APM, but that is a discussion for another blog. Stay tuned to see how we can use Elastic’s machine learning, AIOps, and AI Assistant to understand the issue.

Conclusion

SLOs allow you to set clear, measurable targets for your service performance, based on factors like availability, response times, error rates, and other key metrics. Hopefully with the overview we’ve provided in this blog, you can see that:

• SLOs can be based on logs. In Elastic, you can use KQL to essentially find and filter on specific logs and log fields to monitor and trigger SLOs.

• AI Assistant is a valuable, easy-to-use capability to analyze, troubleshoot, and even potentially resolve SLO issues.

• APM Service based SLOs are easy to create and manage with integration to Elastic APM. We also use OTel telemetry to help monitor SLOs.

For more information on SLOs in Elastic, check out Elastic documentation and the following resources:

• What’s new in Elastic Observability 8.12

• Introducing the Elastic AI Assistant

• Elastic OpenTelemetry support

Ready to get started? Sign up for Elastic Cloud and try out the features and capabilities I’ve outlined above to get the most value and visibility out of your SLOs.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Elastic Observability offers over 25 out-of-the-box integrations for Microsoft Azure services with more on the way. A full list of Azure integrations can be found in our online documentation.

Elastic Observability aggregates not only logs but also metrics for Azure services and the applications running on Azure compute services (Virtual Machines, Functions, Kubernetes Service, etc.). All this data can be analyzed visually and more intuitively using Elastic®’s advanced machine learning (ML) capabilities, which help detect performance issues and surface root causes before end users are affected.

For more details on how Elastic Observability provides application performance monitoring (APM) capabilities such as service maps, tracing, dependencies, and ML-based metrics correlations, read APM correlations in Elastic Observability: Automatically identifying probable causes of slow or failed transactions.

That’s right, Elastic offers capabilities to collect, aggregate, and analyze metrics for Microsoft Azure services and applications running on Azure. Elastic Observability is for more than just capturing logs — it offers a unified observability solution for Microsoft Azure workloads.

In this blog, we’ll review how Elastic Observability can monitor metrics for a three-tier web application running on Microsoft Azure and leveraging:

• Microsoft Azure Virtual Machines
• Microsoft Azure SQL database
• Microsoft Azure Virtual Network

As you will see, once the integration is installed, metrics will arrive instantly and you can immediately start deriving insights from metrics.

Prerequisites and config

Here are some of the components and details we used to set up this demonstration:

• Ensure you have a Microsoft Azure account and an Azure service principal with permission to read monitoring data from Microsoft Azure (see details in our documentation).
• This post does not cover application monitoring; instead, we will focus on how Microsoft Azure services can be easily monitored. If you want to get started with examples of application monitoring, see our Hello World observability code samples.
• In order to see metrics, you will need to load the application. We’ve also created a Playwright script to drive traffic to the application.

Three-tier application overview

Before we dive into the Elastic deployment setup and configuration, let's review what we are monitoring. If you follow the Microsoft Learn N-tier example app instructions for deploying the "What's for Lunch?" app, you will have the following deployed.

What’s deployed:

• Microsoft Azure VM presentation tier that renders an HTML client in the user's browser and enables user requests to be sent to the “What’s for Lunch?” app
• Microsoft Azure VM application tier that communicates with the presentation and the database tier
• Microsoft Azure SQL instance in the database tier, handling requests from the application tier to store and serve data

At the end of the blog, we will also provide a Playwright script that can be run to send requests to this app in order to load it with example data and exercise its functionality. This will help drive metrics to “light up” the dashboards.

Setting it all up

Let’s walk through the details of how to deploy the example three-tier application, Azure integration on Elastic and visualize what gets ingested in Elastic’s Kibana® dashboards.

Step 0: Get an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 1: Deploy the Microsoft Azure three-tier application

From the Azure portal, click the Cloud Shell icon at the top of the portal to open Cloud Shell…

… and when the Cloud Shell first opens, select Bash as the shell type to use.

If you’re prompted that “You have no storage mounted,” then click the Create storage button to create a file store to be used for saving and editing files from Cloud Shell.

You should now see the open Cloud Shell terminal.

Run the following command in Cloud Shell to define the environment variables that we’ll be using in the Cloud Shell commands required to deploy and view the sample application.

Be sure to specify a valid RESOURCE_GROUP from your available Resource Groups listed in the Azure portal. Also specify a new password to replace the SpecifyNewPasswordHere placeholder text before running the command. See the Microsoft password policy documentation for password requirements.

RESOURCE_GROUP="test"
APP_PASSWORD="SpecifyNewPasswordHere"

Run the following az deployment group create command, which will deploy the example three-tier web app in around five minutes.

az deployment group create --resource-group $RESOURCE_GROUP --template-uri https://raw.githubusercontent.com/MicrosoftDocs/mslearn-n-tier-architecture/master/Deployment/azuredeploy.json --parameters password=$APP_PASSWORD

After the deployment has completed, run the following command, which returns the URL for the app.

az deployment group show --output table --resource-group $RESOURCE_GROUP --name azuredeploy --query properties.outputs.webSiteUrl

Copy the web app URL and paste it into a browser to view the example “What’s for Lunch?” web app.

Step 2: Create an Azure service principal and grant access permission

Go to the Microsoft Azure Portal. Search for active directory and select Microsoft Entra ID.

Copy the Tenant ID for use in a later step in this blog post. This ID is required to configure Elastic Agent to connect to your Azure account.

In the navigation pane, select App registrations.

Then click New registration.

Type the name of your application (this tutorial uses three-tier-app-azure) and click Register (accept the default values for other settings).

Copy the Application (client) ID and save it for later. This ID is required to configure Elastic Agent to connect to your Azure account.

In the navigation pane, select Certificates & secrets , and then click New client secret to create a new security key.

Type a description of the secret and select an expiration. Click Add to create the client secret. Under Value , copy the secret value and save it (along with your client ID) for later.

After creating the Azure service principal, you need to grant it the correct permissions. In the Azure Portal, search for and select Subscriptions.

In the Subscriptions page, click the name of your subscription. On the subscription details page, copy your Subscription ID and save it for a later step.

In the navigation pane, select Access control (IAM).

Click Add and select Add role assignment.

On the Role tab, select the Monitoring Reader role and then click Next.

On the Members tab, select the option to assign access to User, group, or service principal. Click Select members , and then search for and select the principal you created earlier. For the description, enter the name of your service principal. Click Next to review the role assignment.

Click Review + assign to grant the service principal access to your subscription.

Step 3: Create an Azure VM instance

In the Azure Portal, search for and select Virtual machines.

On the Virtual machines page, click + Create and select Azure virtual machine.

On the Virtual machine creation page, enter a name like “metrics-vm” for the virtual machine name and select VM Size to be “Standard_D2s_v3 - 2 vcpus, 8 GiB memory.” Click the Next : Disks button.

On the Disks page, keep the default settings and click the Next : Networking button.

On the Networking page, demo-vnet should be selected for Virtual network and demo-biz-subnet should be selected for Subnet. These resources are created as part of the three-tier example app’s deployment that was done in Step 1.

Click the Review + create button.

On the Review page, click the Create button.

Step 4: Install the Azure Resource Metrics integration

In your Elastic Cloud deployment, navigate to the Elastic Azure integrations by selecting Integrations from the top-level menu. Search for azure resource and click the Azure Resource Metrics tile.

Click Add Azure Resource Metrics.

Click Add integration only (skip agent installation).

Enter the values that you saved previously for Client ID, Client Secret, Tenant ID, and Subscription ID.

As you can see, the Azure Resource Metrics integration will collect a significant amount of data from eight Azure services. Click Save and continue.

You’ll be presented with a confirmation dialog window. Click Add Elastic Agent to your hosts.

This will display the instructions required to install the Elastic agent. Copy the command under the Linux Tar tab.

Next you will need to use SSH to log in to the Azure VM instance and run the commands copied from Linux Tar tab. Go to Azure Virtual Machines in the Azure portal. Then click the name of the VM instance that you created in Step 3.

Click the Select button in the SSH Using Azure CLI section.

Select the “I understand …” checkbox and then click the Configure + connect button.

Once you are SSH’d inside the VM instance terminal window, run the commands copied previously from Linux Tar tab in the Install Elastic Agent on your host instructions. When the installation completes, you’ll see a confirmation message in the Install Elastic Agent on your host form.

Super! The Elastic agent is sending data to Elastic Cloud. Now let’s observe some metrics.

Step 5: Run traffic against the application

While getting the application running is fairly easy, there is nothing to monitor or observe with Elastic unless you add a load on the application.

Here is a simple script you can also run using Playwright to add traffic and exercise the functionality of the Azure three-tier application:

import { test, expect } from "@playwright/test";

test("homepage for Microsoft Azure three tier app", async ({ page }) => {
  // Load web app
  await page.goto("http://20.172.198.231/");
  // Add lunch suggestions
  await page.fill("id=txtAdd", "tacos");
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=txtAdd", "sushi");
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=txtAdd", "pizza");
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=txtAdd", "burgers");
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=txtAdd", "salad");
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=txtAdd", "sandwiches");
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  // Click vote buttons
  await page.getByRole("button").nth(1).click();
  await page.getByRole("button").nth(3).click();
  await page.getByRole("button").nth(5).click();
  await page.getByRole("button").nth(7).click();
  await page.getByRole("button").nth(9).click();
  await page.getByRole("button").nth(11).click();
  // Click remove buttons
  await page.getByRole("button").nth(12).click();
  await page.getByRole("button").nth(10).click();
  await page.getByRole("button").nth(8).click();
  await page.getByRole("button").nth(6).click();
  await page.getByRole("button").nth(4).click();
  await page.getByRole("button").nth(2).click();
});

Step 6: View Azure dashboards in Elastic

With Elastic Agent running, you can go to Elastic Dashboards to view what’s being ingested. Simply search for “dashboard” in Elastic and choose Dashboard.

This will open the Elastic Dashboards page. In the Dashboards search box, search for azure vm and click the [Azure Metrics] Compute VMs Overview dashboard, one of the many out-of-the-box dashboards available.

You will see a Dashboard populated with your deployed application’s VM metrics.

On the Azure Compute VM dashboard, we can see the following sampling of some of the many available metrics:

• CPU utilization
• Available memory
• Network sent and received bytes
• Disk writes and reads metrics

For metrics not covered by out-of-the-box dashboards, custom dashboards can be easily created to visualize metrics that are important to you.

Congratulations, you have now started monitoring metrics from Microsoft Azure services for your application!

Analyze your data with Elastic AI Assistant

Once metrics and logs (or either one) are in Elastic, start analyzing your data with context-aware insights using the Elastic AI Assistant for Observability.

Conclusion: Monitoring Microsoft Azure service metrics with Elastic Observability is easy!

We hope you’ve gotten an appreciation for how Elastic Observability can help you monitor Azure service metrics. Here’s a quick recap of what you learned:

• Elastic Observability supports ingest and analysis of Azure service metrics.
• It’s easy to set up ingest from Azure services via the Elastic Agent.
• Elastic Observability has multiple out-of-the-box Azure service dashboards you can use to preliminarily review information and then modify for your needs.

Try it out for yourself by signing up via Microsoft Azure Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on Microsoft Azure around the world. Your Azure Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with Microsoft Azure.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic Observability offers 16 out-of-the-box integrations for Google Cloud services with more on the way. A full list of Google Cloud integrations can be found in our online documentation.

In addition to our native Google Cloud integrations, Elastic Observability aggregates not only logs but also metrics for Google Cloud services and the applications running on Google Cloud compute services (Compute Engine, Cloud Run, Cloud Functions, Kubernetes Engine). All this data can be analyzed visually and more intuitively using Elastic®’s advanced machine learning (ML) capabilities, which help detect performance issues and surface root causes before end users are affected.

For more details on how Elastic Observability provides application performance monitoring (APM) capabilities such as service maps, tracing, dependencies, and ML based metrics correlations, read: APM correlations in Elastic Observability: Automatically identifying probable causes of slow or failed transactions.

That’s right, Elastic offers metrics ingest, aggregation, and analysis for Google Cloud services and applications on Google Cloud compute services. Elastic is more than logs — it offers a unified observability solution for Google Cloud environments.

In this blog, I’ll review how Elastic Observability can monitor metrics for a three-tier web application running on Google Cloud services, which include:

• Google Cloud Run
• Google Cloud SQL for PostgreSQL
• Google Cloud Memorystore for Redis
• Google Cloud VPC Network

As you will see, once the integration is installed, metrics will arrive instantly and you can immediately start reviewing metrics.

Prerequisites and config

Here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• Ensure you have a Google Cloud project and a Service Account with permissions to pull the necessary data from Google Cloud (see details in our documentation).
• We used Google Cloud’s three-tier app and deployed it using the Google Cloud console.
• We’ll walk through installing the general Elastic Google Cloud Platform Integration, which covers the services we want to collect metrics for.
• We will not cover application monitoring; instead, we will focus on how Google Cloud services can be easily monitored.
• In order to see metrics, you will need to load the application. We’ve also created a playwright script to drive traffic to the application.

Three-tier application overview

Before we dive into the Elastic configuration, let's review what we are monitoring. If you follow the Jump Start Solution: Three-tier web app instructions fordeploying the task-tracking app, you will have the following deployed.

What’s deployed:

• Cloud Run frontend tier that renders an HTML client in the user's browser and enables user requests to be sent to the task-tracking app
• Cloud Run middle tier API layer that communicates with the frontend and the database tier
• Memorystore for Redis instance in the database tier, caching and serving data that is read frequently
• Cloud SQL for PostgreSQL instance in the database tier, handling requests that can't be served from the in-memory Redis cache

At the end of the blog, we will also provide a Playwright script that can be run to send requests to this app in order to load it with example data and exercise its functionality. This will help drive metrics to “light up” the dashboards.

Setting it all up

Let’s walk through the details of how to get the application, Google Cloud integration on Elastic, and what gets ingested.

Step 0: Get an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 1: Deploy the Google Cloud three-tier application

Follow the instructions listed out in Jump Start Solution: Three-tier web app choosing the Deploy through the console option for deployment.

Step 2: Create a Google Cloud Service Account and download credentials file

Once you’ve installed the app, the next step is to create a Service Account with a Role and a Service Account Key that will be used by Elastic’s integration to access data in your Google Cloud project.

Go to Google Cloud IAM Roles to create a Role with the necessary permissions. Click the CREATE ROLE button.

Give the Role a Title and an ID. Then add the 10 assigned permissions listed here.

• cloudsql.instances.list
• compute.instances.list
• monitoring.metricDescriptors.list
• monitoring.timeSeries.list
• pubsub.subscriptions.consume
• pubsub.subscriptions.create
• pubsub.subscriptions.get
• pubsub.topics.attachSubscription
• redis.instances.list
• run.services.list

These permissions are a minimal set of what’s required for this blog post. You should add permissions for all the services for which you would like to collect metrics. If you need to add or remove permissions in the future, the Role’s permissions can be updated as many times as necessary.

Click the CREATE button.

Go to Google Cloud IAM Service Accounts to create a Service Account that will be used by the Elastic integration for access to Google Cloud. Click the CREATE SERVICE ACCOUNT button.

Enter a Service account name and a Service account ID. Click the CREATE AND CONTINUE button.

Then select the Role that you created previously and click the CONTINUE button.

Click the DONE button to complete the Service Account creation process.

Next select the Service Account you just created to see its details page. Under the KEYS tab, click the ADD KEY dropdown and select Create new key.

In the Create private key dialog window, with the Key type set as JSON, click the CREATE button.

The JSON credentials file key will be automatically downloaded to your local computer’s Downloads folder. The credentials file will be named something like:

your-project-id-12a1234b1234.json

You can rename the file to be something else. For the purpose of this blog, we’ll rename it to:

credentials.json

Step 3: Create a Google Cloud VM instance

To create the Compute Engine VM instance in Google Cloud, go to Compute Engine. Then select CREATE INSTANCE.

Enter the following values for the VM instance details:

• Enter a Name of your choice for the VM instance.
• Expand the Advanced Options section and the Networking sub-section.
• Enter allow-ssh as the Networking tag.
• Select the Network Interface to use the tiered-web-app-private-network , which is the network on which the Google Cloud three-tier web app is deployed.

Click the CREATE button to create the VM instance.

Step 4: SSH in to the Google Cloud VM instance and upload the credentials file

In order to SSH into the Google Cloud VM instance you just created in the previous step, you’ll need to create a Firewall rule in tiered-web-app-private-network , which is the network where the VM instance resides.

Go to the Google Cloud Firewall policies page. Click the CREATE FIREWALL RULE button.

Enter the following values for the Firewall Rule.

• Enter a firewall rule Name.
• Select tiered-web-app-private-network for the Network.
• Enter allow-ssh for Target Tags.
• Enter 0.0.0.0/0 for the Source IPv4 ranges.Click TCP and set the Ports to 22.

Click CREATE to create the firewall rule.

After the new Firewall rule is created, you can now SSH into your VM instance. Go to the Google Cloud VM instances and select the VM instance you created in the previous step to see its details page. Click the SSH button.

Once you are SSH’d inside the VM instance terminal window, click the UPLOAD FILE button.

Select the credentials.json file located on your local computer and click the Upload Files button to upload the file.

In the VM instance’s SSH terminal, run the following command to get the full path to your Google Cloud Service Account credentials file.

realpath credentials.json

This should return the full path to your Google Cloud Service Account credentials file.

Copy the credentials file’s full path and save it in a handy location to be used in a later step.

Step 5: Add the Elastic Google Cloud integration

Navigate to the Google Cloud Platform integration in Elastic by selecting Integrations from the top-level menu. Search for google and click the Google Cloud Platform tile.

Click Add Google Cloud Platform.

Click Add integration only (skip agent installation).

Update the Project Id input text box to be your Google Cloud Project ID. Next, paste in the credentials file’s full path into the Credentials File input text box.

As you can see, the general Elastic Google Cloud Platform Integration will collect a significant amount of data from 16 Google Cloud services. If you don’t want to install this general Elastic Google Cloud Platform Integration, you can select individual integrations to install. Click Save and continue.

You’ll be presented with a confirmation dialog window. Click Add Elastic Agent to your hosts.

This will display the instructions required to install the Elastic agent. Copy the command under the Linux Tar tab.

Next you will need to use SSH to log in to the Google Cloud VM instance and run the commands copied from Linux Tar tab. Go to Compute Engine. Then click the name of the VM instance that you created in Step 2. Log in to the VM by clicking the SSH button.

Once you are SSH’d inside the VM instance terminal window, run the commands copied previously from Linux Tar tab in the Install Elastic Agent on your host instructions.

When the installation completes, you’ll see a confirmation message in the Install Elastic Agent on your host form. Click the Add the integration button.

Excellent! The Elastic agent is sending data to Elastic Cloud. Now let’s observe some metrics.

Step 6: Run traffic against the application

While getting the application running is fairly easy, there is nothing to monitor or observe with Elastic unless you add a load on the application.

Here is a simple script you can also run using Playwright to add traffic and exercise the functionality of the Google Cloud three-tier application:

import { test, expect } from "@playwright/test";

test("homepage for Google Cloud Threetierapp", async ({ page }) => {
  await page.goto("https://tiered-web-app-fe-zg62dali3a-uc.a.run.app");
  // Insert 2 todo items
  await page.fill("id=todo-new", (Math.random() * 100).toString());
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=todo-new", (Math.random() * 100).toString());
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  // Click one todo item
  await page.getByRole("checkbox").nth(0).check();
  await page.waitForTimeout(1000);
  // Delete one todo item
  const deleteButton = page.getByText("delete").nth(0);
  await deleteButton.dispatchEvent("click");
  await page.waitForTimeout(4000);
});

Step 7: Go to Google Cloud dashboards in Elastic

With Elastic Agent running, you can go to Elastic Dashboards to view what’s being ingested. Simply search for “dashboard” in Elastic and choose Dashboards.

This will open the Elastic Dashboards page.

In the Dashboards search box, search for GCP and click the [Metrics GCP] CloudSQL PostgreSQL Overview dashboard, one of the many out-of-the-box dashboards available. Let’s see what comes up.

On the Cloud SQL dashboard, we can see the following sampling of some of the many available metrics:

• Disk write ops
• CPU utilization
• Network sent and received bytes
• Transaction count
• Disk bytes used
• Disk quota
• Memory usage
• Disk read ops

Next let’s take a look at metrics for Cloud Run.

We’ve created a custom dashboard using the Create dashboard button on the Elastic Dashboards page. Here we see a few of the numerous available metrics:

• Container instance count
• CPU utilization for the three-tier app frontend and API
• Request count for the three-tier app frontend and API
• Bytes in and out of the API

This is a custom dashboard created for MemoryStore where we can see the following sampling of the available metrics:

• Network traffic to the Memorystore Redis instance
• Count of the keys stored in Memorystore Redis
• CPU utilization of the Memorystore Redis instance
• Memory usage of the Memorystore Redis instance

Congratulations, you have now started monitoring metrics from key Google Cloud services for your application!

What to monitor on Google Cloud next?

Add logs from Google Cloud Services

Now that metrics are being monitored, you can also now add logging. There are several options for ingesting logs.

The Google Cloud Platform Integration in the Elastic Agent has four separate logs settings: audit logs, firewall logs, VPC Flow logs, and DNS logs. Just ensure you turn on what you wish to receive.

Analyze your data with Elastic machine learning

Once metrics and logs (or either one) are in Elastic, start analyzing your data through Elastic’s ML capabilities. A great review of these features can be found here:

• Correlating APM Telemetry to determine root causes in transactions
• Introduction to Elastic Machine Learning

Conclusion: Monitoring Google Cloud service metrics with Elastic Observability is easy!

I hope you’ve gotten an appreciation for how Elastic Observability can help you monitor Google Cloud service metrics. Here’s a quick recap of lessons and what you learned:

• Elastic Observability supports ingest and analysis of Google Cloud service metrics.
• It’s easy to set up ingest from Google Cloud services via the Elastic Agent.
• Elastic Observability has multiple out-of-the-box Google Cloud service dashboards you can use to preliminarily review information and then modify for your needs.
• For metrics not covered by out-of-the-box dashboards, custom dashboards can be easily created to visualize metrics that are important to you.
• 16 Google Cloud services are supported as part of Google Cloud Platform Integration on Elastic Observability, with more services being added regularly.
• As noted in related blogs, you can analyze your Google Cloud service metrics with Elastic’s machine learning capabilities.

Try it out for yourself by signing up via Google Cloud Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on Google Cloud around the world. Your Google Cloud Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with Google Cloud.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Prometheus is primarily built for short-term metric storage, typically retaining data in-memory or on local disk storage, with a focus on real-time monitoring and alerting rather than historical analysis. While it offers valuable insights into current metric values and trends, it may pose economic challenges and fall short of the robust functionalities and capabilities necessary for in-depth historical analysis, long-term trend detection, and forecasting. This is particularly evident in large environments with a substantial number of targets or high data ingestion rates, where metric data accumulates rapidly.

Numerous organizations assess their unique needs and explore avenues to augment their Prometheus monitoring and observability capabilities. One effective approach is integrating Prometheus with Elastic®. In this blog post, we will showcase the integration of Prometheus with Elastic, emphasizing how Elastic elevates metrics monitoring through extensive historical analytics, anomaly detection, and forecasting, all in a cost-effective manner.

Integrate Prometheus with Elastic seamlessly

Organizations that have configured their cloud-native applications to expose metrics in Prometheus format can seamlessly transmit the metrics to Elastic by using Prometheus integration. Elastic enables organizations to monitor their metrics in conjunction with all other data gathered through Elastic's extensive integrations.

Go to Integrations and find the Prometheus integration.

To gather metrics from Prometheus servers, the Elastic Agent is employed, with central management of Elastic agents handled through the Fleet server.

After enrolling the Elastic Agent in the Fleet, users can choose from the following methods to ingest Prometheus metrics into Elastic.

1. Prometheus collectors

The Prometheus collectors connect to the Prometheus server and pull metrics or scrape metrics from a Prometheus exporter.

2. Prometheus queries

The Prometheus queries execute specific Prometheus queries against Prometheus Query API.

3. Prometheus remote-write

The Prometheus remote_write can receive metrics from a Prometheus server that has configured the remote_write setting.

After your Prometheus metrics are ingested, you have the option to visualize your data graphically within the Metrics Explorer and further segment it based on labels, such as hosts, containers, and more.

You can also query your metrics data in Discover and explore the fields of your individual documents within the details panel.

Storing historical metrics with Elastic’s data tiering mechanism

By exporting Prometheus metrics to Elasticsearch, organizations can extend the retention period and gain the ability to analyze metrics historically. Elastic optimizes data storage and access based on the frequency of data usage and the performance requirements of different data sets. The goal is to efficiently manage and store data, ensuring that it remains accessible when needed while keeping storage costs in check.

After ingesting Prometheus metrics data, you have various retention options. You can set the duration for data to reside in the hot tier, which utilizes high IO hardware (SSD) and is more expensive. Alternatively, you can move the Prometheus metrics to the warm tier, employing cost-effective hardware like spinning disks (HDD) while maintaining consistent and efficient search performance. The cold tier mirrors the infrastructure of the warm tier for primary data but utilizes S3 for replica storage. Elastic automatically recovers replica indices from S3 in case of node or disk failure, ensuring search performance comparable to the warm tier while reducing disk cost.

The frozen tier allows direct searching of data stored in S3 or an object store, without the need for rehydration. The purpose is to further reduce storage costs for Prometheus metrics data that is less frequently accessed. By moving historical data into the frozen tier, organizations can optimize their storage infrastructure, ensuring that the recent, critical data remains in higher-performance tiers while less frequently accessed data is stored economically in the frozen tier. This way, organizations can perform historical analysis and trend detection, identify patterns and make informed decisions, and maintain compliance with regulatory standards in a cost-effective manner.

An alternative way to store your cloud-native metrics more efficiently is to use Elastic Time Series Data Stream (TSDS). TSDS can store your metrics data more efficiently with ~70% less disk space than a regular data stream. The downsampling functionality will further reduce the storage required by rolling up metrics within a fixed time interval into a single summary metric. This not only assists organizations in cutting down on storage expenses for metric data but also simplifies the metric infrastructure, making it easier for users to correlate metrics with logs and traces through a unified interface.

Advanced analytics

Besides Metrics Explorer and Discover, Elasticsearch® provides more advanced analytics capabilities and empowers organizations to gain deeper, more valuable insights into their Prometheus metrics data.

Out of the box, Prometheus integration provides a default overview dashboard.

From Metrics Explorer or Discover, users can also easily edit their Prometheus metrics visualization in Elastic Lens or create new visualizations from Lens.

Elastic Lens enables users to explore and visualize data intuitively through dynamic visualizations. This user-friendly interface eliminates the need for complex query languages, making data analysis accessible to a broader audience. Elasticsearch also offers other powerful visualization methods with aggregations and filters, enabling users to perform advanced analytics on their Prometheus metrics data, including short-term and historical data. To learn more, check out the how-to series: Kibana.

Anomaly detection and forecasting

When analyzing data, maintaining a constant watch on the screen is simply not feasible, especially when dealing with millions of time series of Prometheus metrics. Engineers frequently encounter the challenge of differentiating normal from abnormal data points, which involves analyzing historical data patterns — a process that can be exceedingly time consuming and often exceeds human capabilities. Thus, there is a pressing need for a more intelligent approach to detect anomalies efficiently.

Setting up alerts may seem like an obvious solution, but relying solely on rule-based alerts with static thresholds can be problematic. What's normal on a Wednesday at 9:00 a.m. might be entirely different from a Sunday at 2:00 a.m. This often leads to complex and hard-to-maintain rules or wide alert ranges that end up missing crucial issues. Moreover, as your business, infrastructure, users, and products evolve, these fixed rules don't keep up, resulting in lots of false positives or, even worse, important issues slipping through the cracks without detection. A more intelligent and adaptable approach is needed to ensure accurate and timely anomaly detection.

Elastic's machine learning anomaly detection excels in such scenarios. It automatically models the normal behavior of your Prometheus data, learning trends, and identifying anomalies, thereby reducing false positives and improving mean time to resolution (MTTR). With over 13 years of development experience in this field, Elastic has emerged as a trusted industry leader.

The key advantage of Elastic's machine learning anomaly detection lies in its unsupervised learning approach. By continuously observing real-time data, it acquires an understanding of the data's behavior over time. This includes grasping daily and weekly patterns, enabling it to establish a normalcy range of expected behavior. Behind the scenes, it constructs statistical models that allow accurate predictions, promptly identifying any unexpected variations. In cases where emerging data exhibits unusual trends, you can seamlessly integrate with alerting systems, operationalizing this valuable insight.

Machine learning's ability to project into the future, forecasting data trends one day, a week, or even a month ahead, equips engineers not only with reporting capabilities but also with pattern recognition and failure prediction based on historical Prometheus data. This plays a crucial role in maintaining mission-critical workloads, offering organizations a proactive monitoring approach. By foreseeing and addressing issues before they escalate, organizations can avert downtime, cut costs, optimize resource utilization, and ensure uninterrupted availability of their vital applications and services.

Creating a machine learning job for your Prometheus data is a straightforward task with a few simple steps. Simply specify the data index and set the desired time range in the single metric view. The machine learning job will then automatically process the historical data, building statistical models behind the scenes. These models will enable the system to predict trends and identify anomalies effectively, providing valuable and actionable insights for your monitoring needs.

In essence, Elastic machine learning empowers us to harness the capabilities of data scientists and effectively apply them in monitoring Prometheus metrics. By seamlessly detecting anomalies and predicting potential issues in advance, Elastic machine learning bridges the gap and enables IT professionals to benefit from the insights derived from advanced data analysis. This practical and accessible approach to anomaly detection equips organizations with a proactive stance toward maintaining the reliability of their systems.

Try it out

Start a free trial on Elastic Cloud and ingest your Prometheus metrics into Elastic. Enhance your Prometheus monitoring with Elastic Observability. Stay ahead of potential issues with advanced AI/ML anomaly detection and prediction capabilities. Eliminate data silos, reduce costs, and enhance overall response efficiency.

Elevate your monitoring capabilities with Elastic today!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In an era where APIs serve as the backbone of modern applications, having the means to maintain visibility and control over these vital components is absolutely essential. In this blog post, we dive deep into the comprehensive observability solution offered by Elastic^®, ensuring real-time visibility, advanced analytics, and actionable insights, empowering you to fine-tune your API Gateway for optimal performance.

For application owners and developers, this integration stands as a beacon of empowerment. Elastic's meticulous orchestration of the seamless merging of metrics, logs, and traces, built upon the robust ELK Stack foundation, equips them with potent real-time monitoring and analysis tools. These tools facilitate precise performance optimization and swift issue resolution, all within a secure and dependable environment.

With Elastic's AWS API Gateway integration, application owners and developers unlock the capability to proactively identify and resolve problems, fine-tune resource utilization, and provide extraordinary digital experiences to their users.

Architecture

Why the AWS API Gateway integration matters

API Gateway now serves as the foundation of contemporary application development, simplifying the process of creating and overseeing APIs on a large scale. Yet, monitoring and troubleshooting these API endpoints can be challenging. With the new AWS API Gateway integration introduced by Elastic, you can gain the following:

• Unprecedented visibility: Monitor your API Gateway endpoints' performance, error rates, and usage metrics in real time. Get a comprehensive view of your APIs' health and performance.
• Log analysis: Dive deep into API Gateway logs with ease. Our integration enables you to collect and analyze logs for HTTP, REST, and Websocket API types, helping you troubleshoot issues and gain valuable insights.
• Rapid issue resolution: Identify and resolve issues in your API Gateway workflows faster than ever. Elastic Observability's powerful search and analytics tools help you pinpoint problems with ease.
• Alerting and notifications: Set up custom alerts based on API Gateway metrics and logs. Receive notifications when performance thresholds are breached, ensuring that you can take action promptly.
• Optimized costs: Visualize resource usage and performance metrics for your API Gateway deployments. Use these insights to optimize resource allocation and reduce operational costs.
• Custom dashboards: Create customized dashboards and visualizations tailored to your API Gateway monitoring needs. Stay in control with real-time data and actionable insights.
• Effortless integration: Seamlessly connect your AWS API Gateway to our observability solution. Our intuitive setup process ensures a smooth integration experience.
• Scalability: Whether you have a handful of APIs or a complex API Gateway landscape, our observability solution scales to meet your needs. Grow confidently as your API infrastructure expands.

How to get started

Getting started with the AWS API Gateway integration in Elastic Observability is seamless. Here's a quick overview of the steps:

Prerequisites and configurations

If you intend to follow the steps outlined in this blog post, there are a few prerequisites and configurations that you should have in place beforehand.

1. You will need an account on Elastic Cloud and a deployed stack and agent. Instructions for deploying a stack on AWS can be found here. This is necessary for AWS API Gateway logging and analysis.

2. You will also need an AWS account with the necessary permissions to pull data from AWS. Details on the required permissions can be found in our documentation.

3. You can monitor API execution by using CloudWatch, which collects and processes raw data from API Gateway into readable, near-real-time metrics and logs. Details on the required steps to enable logging can be found here.

Step 1. Create an account with Elastic

Create an account on Elastic Cloud by following the steps provided.

Step 2. Add integration

• Log in to your Elastic Cloud deployment.

• Click on Add integrations. You will be navigated to a catalog of supported integrations.

• Search and select AWS API Gateway.

Step 3. Configure integration

• Click on the Add AWS API Gateway button and provide the required details.
• If this is your first time adding an AWS integration, you’ll need to configure and enroll the Elastic Agent on an AWS instance.

• Then complete the “Configure integration” form, providing all the necessary information required for agents to collect the AWS API Gateway metrics and associated CloudWatch logs. Multiple AWS credential methods are supported, including access keys, temporary security credentials, and IAM role ARN. Please see the IAM security and access documentation for more details. You can choose to collect API Gateway metrics, API Gateway logs via S3, or API Gateway logs via CloudWatch.
• Click on the Save and continue button at the bottom of the page.

Step 4. Analyze and monitor

Explore the data using the out-of-the-box dashboards available for the integration. Select Discover from the Elastic Cloud top-level menu.

Or, create custom dashboards, set up alerts, and gain actionable insights into your API Gateway service performance.

Here are key monitoring metrics collected through this integration across Rest APIs, HTTP APIs, and Websocket APIs:

• 4XXError – The number of client-side errors captured in a given period
• 5XXError – The number of server-side errors captured in a given period
• CacheHitCount – The number of requests served from the API cache in a given period
• CacheMissCount – The number of requests served from the backend in a given period, when API caching is enabled
• Count – The total number of API requests in a given period
• IntegrationLatency – The time between when API Gateway relays a request to the backend and when it receives a response from the backend
• Latency – The time between when API Gateway receives a request from a client and when it returns a response to the client — the latency includes the integration latency and other API Gateway overhead
• DataProcessed – The amount of data processed in bytes
• ConnectCount – The number of messages sent to the $connect route integration
  MessageCount – The number of messages sent to the WebSocket API, either from or to the client

Conclusion

The native integration of AWS API Gateway into Elastic Observability marks a significant advancement in streamlining the monitoring and management of your APIs. With this integration, you gain access to a wealth of insights, real-time visibility, and powerful analytics tools, empowering you to optimize your API performance, enhance security, and troubleshoot with ease. Don't miss out on this opportunity to take your API management to the next level, ensuring your digital assets operate at their best, all while providing a seamless experience for your users. Embrace this integration, and stay at the forefront of API observability in the ever-evolving world of digital technology.

Visit our documentation to learn more about Elastic Observability and the AWS API Gateway integration, or contact our sales team to get started!

Start a free trial today

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

OpenTelemetry is more than just becoming the open ingestion standard for observability. As one of the major Cloud Native Computing Foundation (CNCF) projects, with as many commits as Kubernetes, it is gaining support from major ISVs and cloud providers delivering support for the framework. Many global companies from finance, insurance, tech, and other industries are starting to standardize on OpenTelemetry. With OpenTelemetry, DevOps teams have a consistent approach to collecting and ingesting telemetry data providing a de-facto standard for observability.

Elastic^® is strategically standardizing on OpenTelemetry for the main data collection architecture for observability and security. Additionally, Elastic is making a commitment to help OpenTelemetry become the best de facto data collection infrastructure for the observability ecosystem. Elastic is deepening its relationship with OpenTelemetry beyond the recent contribution of Elastic Common Schema (ECS) to OpenTelemetry (OTel).

Today, Elastic supports OpenTelemetry natively, since Elastic 7.14, by being able to directly ingest OpenTelemetry protocol (OTLP) based traces, metrics, and logs.

In this blog, we’ll review the current OpenTelemetry support provided by Elastic, which includes the following:

• Easy ingest of distributed tracing and metrics for applications configured with OpenTelemetry agents for Python, NodeJS, Java, Go, and .NET
• OpenTelemetry logs instrumentation and ingest using various configurations
• Open semantic conventions for logs and more through ECS, which is not part of OpenTelemetry
• Machine learning based AIOps capabilities, such as latency correlations, failure correlations, anomaly detection, log spike analysis, predictive pattern analysis, Elastic AI Assistant support, and more, all apply to native OTLP telemetry.
• Migrate applications to OpenTelemetry at your own speed. Elastic’s APM capabilities all work seamlessly even with a mix of services using OpenTelemetry and/or Elastic APM agents. You can even combine OpenTelemetry instrumentation with Elastic Agent.
• Integrated views and analysis with Kubernetes clusters, which most OpenTelemetry applications are running on. Elastic can highlight specific pods and containers related to each service when analyzing issues for applications based on OpenTelemetry.

Ingesting OpenTelemetry into Elastic

If you’re interested in seeing how simple it is to ingest OpenTelemetry traces and metrics into Elastic, follow the steps outlined in this blog.

Let’s outline what Elastic provides for ingesting OpenTelemetry data. Here are all your options:

Using the OpenTelemetry Collector

When using the OpenTelemetry Collector, which is the most common configuration option, you simply have to add two key variables.

The instructions utilize a specific opentelemetry-collector configuration for Elastic. Essentially, the Elastic values.yaml file specified in the elastic/opentelemetry-demo configure the opentelemetry-collector to point to the Elastic APM Server using two main values:

OTEL_EXPORTER_OTLP_ENDPOINT is Elastic’s APM Server
OTEL_EXPORTER_OTLP_HEADERS Elastic Authorization

These two values can be found in the OpenTelemetry setup instructions under the APM integration instructions (Integrations->APM) in your Elastic Cloud.

Native OpenTelemetry agents embedded in code

If you are thinking of using OpenTelemetry libraries in your code, you can simply point the service to Elastic’s APM server, because it supports native OLTP protocol. No special Elastic conversion is needed.

To demonstrate this effectively and provide some education on how to use OpenTelemetry, we have two applications you can use to learn from:

• Elastic’s version of OpenTelemetry demo: As with all the other observability vendors, we have our own forked version of the OpenTelemetry demo.
• Elastiflix: This demo application is an example to help you learn how to instrument on various languages and telemetry signals.

Check out our blogs on using the Elastiflix application and instrumenting with OpenTelemetry:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation

We have created YouTube videos on these topics as well:

• How to Manually Instrument Java with OpenTelemetry (Part 1)
• How to Manually Instrument Java with OpenTelemetry (Part 2)
• Custom Java Instrumentation with OpenTelemetry
• Elastic APM - Automatic .NET Instrumentation with OpenTelemetry
• How to Manually Instrument .NET Applications with OpenTelemetry

Given Elastic and OpenTelemetry’s vast user base, these provide a rich source of education for anyone trying to learn the intricacies of instrumenting with OpenTelemetry.

Elastic Agents supporting OpenTelemetry

If you’ve already implemented OpenTelemetry, you can still use them with OpenTelemetry. Elastic APM agents today are able to ship OpenTelemetry spans as part of a trace. This means that if you have any component in your application that emits an OpenTelemetry span, it’ll be part of the trace the Elastic APM agent captures.

OpenTelemetry logs in Elastic

If you look at OpenTelemetry documentation, you will see that a lot of language libraries are still in experimental or not implemented yet state. Java is in stable state, per the documentation. Depending on your service’s language, and your appetite for adventure, there exist several options for exporting logs from your services and applications and marrying them together in your observability backend.

In a previous blog, we discussed 3 different configurations to properly get logging data into Elastic for Java. The blog explores the current state of the art of OpenTelemetry logging and provides guidance on the available approaches with the following tenants in mind:

• Correlation of service logs with OTel-generated tracing where applicable
• Proper capture of exceptions
• Common context across tracing, metrics, and logging
• Support for slf4j key-value pairs (“structured logging”)
• Automatic attachment of metadata carried between services via OTel baggage
• Use of an Elastic Observability backend
• Consistent data fidelity in Elastic regardless of the approach taken

Three models, which are covered in the blog, currently exist for getting your application or service logs to Elastic with correlation to OTel tracing and baggage:

• Output logs from your service (alongside traces and metrics) using an embedded OpenTelemetry Instrumentation library to Elastic via the OTLP protocol
• Write logs from your service to a file scrapped by the OpenTelemetry Collector, which then forwards to Elastic via the OTLP protocol
• Write logs from your service to a file scrapped by Elastic Agent (or Filebeat), which then forwards to Elastic via an Elastic-defined protocol

Note that (1), in contrast to (2) and (3), does not involve writing service logs to a file prior to ingestion into Elastic.

OpenTelemetry is Elastic’s preferred schema

Elastic recently contributed the Elastic Common Schema (ECS) to the OpenTelemetry (OTel) project, enabling a unified data specification for security and observability data within the OTel Semantic Conventions framework.

ECS, an open source specification, was developed with support from the Elastic user community to define a common set of fields to be used when storing event data in Elasticsearch^®. ECS helps reduce management and storage costs stemming from data duplication, improving operational efficiency.

Similarly, OTel’s Semantic Conventions (SemConv) also specify common names for various kinds of operations and data. The benefit of using OTel SemConv is in following a common naming scheme that can be standardized across a codebase, libraries, and platforms for OTel users.

The merging of ECS and OTel SemConv will help advance OTel’s adoption and the continued evolution and convergence of observability and security domains.

Elastic Observability APM and machine learning capabilities

All of Elastic Observability’s APM capabilities are available with OTel data (read more on this in our blog, Independence with OpenTelemetry):

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services
• Transactions (traces)
• ML correlations (specifically for latency)
• Service logs

In addition to Elastic’s APM and unified view of the telemetry data, you will now be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR. Here are some of the ML based AIOps capabilities we have:

• Anomaly detection: Elastic Observability, when turned on (see documentation), automatically detects anomalies by continuously modeling the normal behavior of your OpenTelemetry data — learning trends, periodicity, and more.
• Log categorization: Elastic also identifies patterns in your OpenTelemetry log events quickly, so that you can take action quicker.
• High-latency or erroneous transactions: Elastic Observability’s APM capability helps you discover which attributes are contributing to increased transaction latency and identifies which attributes are most influential in distinguishing between transaction failures and successes.
• Log spike detector helps identify reasons for increases in OpenTelemetry log rates. It makes it easy to find and investigate causes of unusual spikes by using the analysis workflow view.
• Log pattern analysis helps you find patterns in unstructured log messages and makes it easier to examine your data.

Elastic allows you to migrate to OTel on your schedule

Although OpenTelemetry supports many programming languages, the status of its major functional components — metrics, traces, and logs — are still at various stages. Thus migrating applications written in Java, Python, and JavaScript are good choices to start with as their metrics, traces, and logs (for Java) are stable.

For the other languages that are not yet supported, you can easily instrument those using Elastic Agents, therefore running your full stack observability platform in mixed mode (Elastic agents with OpenTelemetry agents).

Here is a simple example:

The above shows a simple variation of our standard Elastic Agent application with one service flipped to OTel — the newsletter-otel service. But we can easily and as needed convert each of these services to OTel as development resources allow.

Hence you can migrate what you need to OpenTelemetry with Elastic as specific languages reach a stable state, and you can then continue your migration to OpenTelemetry agents.

Integrated Kubernetes and OpenTelemetry views in Elastic

Elastic manages your Kubernetes cluster using the Elastic Agent, and you can use it on your Kubernetes cluster where your OpenTelemetry application is running. Hence you can not only use OpenTelemetry for your application, but Elastic can also monitor the corresponding Kubernetes cluster.

There are two configurations for Kubernetes:

1. Simply deploying the Elastic Agent daemon set on the kubernetes cluster. We outline this out in the article entitled Managing your Kubernetes cluster with Elastic Observability. This would also push just the Kubernetes metrics and logs to Elastic.

2. Deploying the Elastic Agent with not only the Kubernetes Daemon set, but also Elastic’s APM integration, the Defend (Security) integration, and Network Packet capture integration to provide more comprehensive Kubernetes cluster observability. We outline this configuration in the following article Modern observability and security on Kubernetes with Elastic and OpenTelemetry.

Both OpenTelemetry visualization examples use the OpenTelemetry demo, and in Elastic, we tie the Kubernetes information with the application to provide you an ability to see Kubernetes information from your traces in APM. This provides a more integrated approach when troubleshooting.

Summary

In essence, Elastic's commitment goes beyond mere support for OpenTelemetry. We are dedicated to ensuring our customers not only adopt OpenTelemetry but thrive with it. Through our solutions, expertise, and resources, we aim to elevate the observability journey for every business, turning data into actionable insights that drive growth and innovation.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Why “Generic SQL”?

Elastic already has metricbeat and integration packages targeted for specific databases. One example is metricbeat for MySQL — and the corresponding integration package. These beats modules and integrations are customized for a specific database, and the metrics are extracted using pre-defined queries from the specific database. The queries used in these integrations and the corresponding metrics are not available for modification.

Whereas the Generic SQL inputs (metricbeat or input package) can be used to scrape metrics from any supported database using the user's SQL queries. The queries are provided by the user depending on specific metrics to be extracted. This enables a much more powerful mechanism for metrics ingestion, where users can choose a specific driver and provide the relevant SQL queries and the results get mapped to one or more Elasticsearch documents, using a structured mapping process (table/variable format explained later).

Generic SQL inputs can be used in conjunction with the existing integration packages, which already extract specific database metrics, to extract additional custom metrics dynamically, making this input very powerful. In this blog, Generic SQL input and Generic SQL are used interchangeably.

Functionalities details

This section covers some of the features that would help with the metrics extraction. We provide a brief description of the response format configuration. Then we dive into the merge_results functionality, which is used to combine results from multiple SQL queries into a single document.

The next key functionality users may be interested in is to collect metrics from all the custom databases, which is now possible with the fetch_from_all_databases feature.

Now let's dive into the specific functionalities:

Different drivers supported

The generic SQL can fetch metrics from the different databases. The current version has the capability to fetch metrics from the following drivers: MySQL, PostgreSQL, Oracle, and Microsoft SQL Server(MSSQL).

Response format

The response format in generic SQL is used to manipulate the data in either table or in variable format. Here’s an overview of the formats and syntax for creating and using the table and variables.

Syntax: response_format: table {{or}} variables

Response format table
This mode generates a single event for each row. The table format has no restrictions on the number of columns in the response. This format can have any number of columns.

Example:

driver: "mssql"
sql_queries:
 - query: "SELECT counter_name, cntr_value FROM sys.dm_os_performance_counters WHERE counter_name= 'User Connections'"
   response_format: table

This query returns a response similar to this:

"sql":{
      "metrics":{
         "counter_name":"User Connections ",
         "cntr_value":7
      },
      "driver":"mssql"
}

The response generated above adds the counter_name as a key in the document.

Response format variables
The variable format supports key:value pairs. This format expects only two columns to fetch in a query.

Example:

driver: "mssql"
sql_queries:
 - query: "SELECT counter_name, cntr_value FROM sys.dm_os_performance_counters WHERE counter_name= 'User Connections'"
   response_format: variables

The variable format takes the first variable in the query above as the key:

"sql":{
      "metrics":{
         "user connections ":7
      },
      "driver":"mssql"
}

In the above response, you can see the value of counter_name is used to generate the key in variable format.

Response optimization: merge_results

We are now supporting merging multiple query responses into a single event. By enabling merge_results , users can significantly optimize the storage space of the metrics ingested to Elasticsearch. This mode enables an efficient compaction of the document generated, where instead of generating multiple documents, a single merged document is generated wherever applicable. The metrics of a similar kind, generated from multiple queries, are combined into a single event.

Syntax: merge_results: true {{or}} false

In the below example, you can see how the data is loaded into Elasticsearch for the below query when the merge_results is disabled.

Example:

In this example, we are using two different queries to fetch metrics from the performance counter.

merge_results: false
driver: "mssql"
sql_queries:
  - query: "SELECT cntr_value As 'user_connections' FROM sys.dm_os_performance_counters WHERE counter_name= 'User Connections'"
    response_format: table
  - query: "SELECT cntr_value As 'buffer_cache_hit_ratio' FROM sys.dm_os_performance_counters WHERE counter_name = 'Buffer cache hit ratio' AND object_name like '%Buffer Manager%'"
    response_format: table

As you can see, the response for the above example generates a single document for each query.

The resulting document from the first query:

"sql":{
      "metrics":{
         "user_connections":7
      },
      "driver":"mssql"
}

And resulting document from the second query:

"sql":{
      "metrics":{
         "buffer_cache_hit_ratio":87
      },
      "driver":"mssql"
}

When we enable the merge_results flag in the query, both the above metrics are combined together and the data gets loaded in a single document.

You can see the merged document in the below example:

"sql":{
      "metrics":{
         "user connections ":7,
         “buffer_cache_hit_ratio”:87
      },
      "driver":"mssql"
}

However, such a merge is possible only if the table queries are merged, and each produces a single row. There is no restriction on variable queries being merged.

Introducing a new capability: fetch_from_all_databases

This is a new functionality to fetch all the database metrics automatically from the system and user databases of the Microsoft SQL Server, by enabling the fetch_from_all_databases flag.

Keep an eye out for the 8.10 release version where you can start using the fetch all database feature. Prior to the 8.10 version, users had to provide the database names manually to fetch metrics from custom/user databases.

Syntax: fetch_from_all_databases: true {{or}} false

Below is the sample query with fetch all databases flag as disabled:

fetch_from_all_databases: false
driver: "mssql"
sql_queries:
  - query: "SELECT @@servername AS server_name, @@servicename AS instance_name, name As 'database_name', database_id FROM sys.databases WHERE name='master';"

The above query fetches metrics only for the provided database name. Here the input database is master, so the metrics are fetched only for the master.

Below is the sample query with the fetch all databases flag as enabled:

fetch_from_all_databases: true
driver: "mssql"
sql_queries:
  - query: SELECT @@servername AS server_name, @@servicename AS instance_name, DB_NAME() AS 'database_name', DB_ID() AS database_id;
    response_format: table

The above query fetches metrics from all available databases. This is useful when the user wants to get data from all the databases.

Please note: currently this feature is supported only for Microsoft SQL Server and will be used by MS SQL integration internally, to support extracting metrics for all user DBs by default.

Using generic SQL: Metricbeat

The generic SQL metricbeat module provides flexibility to execute queries against different database drivers. The metricbeat input is available as GA for any production usage. Here, you can find more information on configuring the generic SQL for different drivers with various examples.

Using generic SQL: Input package

The input package provides a flexible solution to advanced users for customizing their ingestion experience in Elastic. Generic SQL is now also available as an SQLinput package. The input package is currently available for early users as a beta release. Let's take a walk through how users can use generic SQL via the input package.

Configurations of generic SQL input package:

The configuration options for the generic SQL input package are as below:

• Driver** :** This is the SQL database for which you want to use the package. In this case, we will take mysql as an example.
• Hosts: Here the user enters the connection string to connect to the database. It would vary depending on which database/driver is being used. Refer here for examples.
• SQL Queries: Here the user writes the SQL queries they want to fire and the response_format is specified.
• Data set: The user specifies a data set name to which the response fields get mapped.
• Merge results** :** This is an advanced setting, used to merge queries into a single event.

Metrics extensibility with customized SQL queries

Let's say a user is using MYSQL Integration, which provides a fixed set of metrics. Their requirement now extends to retrieving more metrics from the MYSQL database by firing new customized SQL queries.

This can be achieved by adding an instance of SQL input package, writing the customized queries and specifying a new data set name as shown in the screenshot below.

This way users can get any metrics by executing corresponding queries. The resultant metrics of the query will be indexed to the new data set, sql_second_dataset.

When there are multiple queries, users can club them into a single event by enabling the Merge Results toggle.

Customizing user experience

Users can customize their data by writing their own ingest pipelines and providing their customized mappings. Users can also build their own bespoke dashboards.

As we can see above, the SQL input package provides the flexibility to get new metrics by running new queries, which are not supported in the default MYSQL integration (the user gets metrics from a predetermined set of queries).

The SQL input package also supports multiple drivers: mssql, postgresql and oracle. So a single input package can be used to cater to all these databases.

Note: The fetch_from_all_databases feature is not supported in the SQL input package yet.

Try it out!

Now that you know about various use cases and features of generic SQL, get started with Elastic Cloud and try using the SQL input package for your SQL database and get customized experience and metrics. If you are looking for newer metrics for some of our existing SQL based integrations — like Microsoft SQL Server, Oracle, and more — go ahead and give the SQL input package a swirl.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

First, we need an example application that allows us to monitor infrastructure changes effectively. We use a Python Flask application with the Elastic Python APM agent. The application is a simple calculator taking the numbers as a REST request. We utilize Locust — a simple load-testing tool to evaluate performance under varying workloads.

The next step includes obtaining the pricing information associated with the cloud services. Every cloud provider is different. Most of them offer an option to retrieve pricing through an API. But today, we will focus on Google Cloud and will leverage their pricing calculator to retrieve relevant cost information.

The calculator and Google Cloud pricing

To perform a cost analysis, we need to know the cost of the machines in use. Google provides a billing API and Client Library to fetch the necessary data programmatically. In this blog, we are not covering the API approach. Instead, the Google Cloud Pricing Calculator is enough. Select the machine type and region in the calculator and set the count 1 instance. It will then report the total estimated cost for this machine. Doing this for an e2-standard-4 machine type results in 107.7071784 US$ for a runtime of 730 hours.

Now, let’s go to our Kibana® where we will create a new index inside Dev Tools. Since we don’t want to analyze text, we will tell Elasticsearch® to treat every text as a keyword. The index name is cloud-billing. I might want to do the same for Azure and AWS, then I can append it to the same index.

PUT cloud-billing
{
  "mappings": {
    "dynamic_templates": [
      {
        "stringsaskeywords": {
          "match": "*",
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword"
          }
        }
      }
    ]
  }
}

Next up is crafting our billing document:

POST cloud-billing/_doc/e2-standard-4_europe-west4
{
  "machine": {
    "enrichment": "e2-standard-4_europe-west4"
  },
  "cloud": {
    "machine": {
       "type": "e2-standard-4"
    },
    "region": "europe-west4",
    "provider": "google"
  },
  "stats": {
    "cpu": 4,
    "memory": 8
  },
  "price": {
    "minute": 0.002459068,
    "hour": 0.14754408,
    "month": 107.7071784
  }
}

We create a document and set a custom ID. This ID matches the instance name and the region since the machines' costs may differ in each region. Automatic IDs could be problematic because I might want to update what a machine costs regularly. I could use a timestamped index for that and only ever use the latest document matching. But this way, I can update and don’t have to worry about it. I calculated the price down to minute and hour prices as well. The most important thing is the machine.enrichment field, which is the same as the ID. The same instance type can exist in multiple regions, but our enrichment processor is limited to match or range. We create a matching name that can explicitly match as in e2-standard-4_europe-west4. It’s up to you to decide whether you want the cloud provider in there and make it google_e2-standard-4_europ-west-4.

Calculating the cost

There are multiple ways of achieving this in the Elastic Stack. In this case, we will use an enrich policy, ingest pipeline, and transform.

The enrich policy is rather easy to setup:

PUT _enrich/policy/cloud-billing
{
  "match": {
    "indices": "cloud-billing",
    "match_field": "machine.enrichment",
    "enrich_fields": ["price.minute", "price.hour", "price.month"]
  }
}

POST _enrich/policy/cloud-billing/_execute

Don’t forget to run the _execute at the end of it. This is necessary to make the internal indices used by the enrichment in the ingest pipeline. The ingest pipeline is rather minimalistic — it calls the enrichment and renames a field. This is where our machine.enrichment field comes in. One caveat around enrichment is that when you add new documents to the cloud-billing index, you need to rerun the _execute statement. The last bit calculates the total cost with the count of unique machines seen.

PUT _ingest/pipeline/cloud-billing
{
  "processors": [
    {
      "set": {
        "field": "_temp.machine_type",
        "value": "{{cloud.machine.type}}_{{cloud.region}}"
      }
    },
    {
      "enrich": {
        "policy_name": "cloud-billing",
        "field": "_temp.machine_type",
        "target_field": "enrichment"
      }
    },
    {
      "rename": {
        "field": "enrichment.price",
        "target_field": "price"
      }
    },
    {
      "remove": {
        "field": [
          "_temp",
          "enrichment"
        ]
      }
    },
    {
      "script": {
        "source": "ctx.total_price=ctx.count_machines*ctx.price.hour"
      }
    }
  ]
}

Since this is all configured now, we are ready for our Transform. For this, we need a data view that matches the APM data_streams. This is traces-apm*, metrics-apm.*, logs-apm.*. For the Transform, go to the Transform UI in Kibana and configure it in the following way:

We are doing an hourly breakdown, therefore, I get a document per service, per hour, per machine type. The interesting bit is the aggregations. I want to see the average CPU usage and the 75,95,99 percentile, to view the CPU usage on an hourly basis. Allowing me to identify the CPU usage across an hour. At the bottom, give the transform a name and select an index cloud-costs and select the cloud-billing ingest pipeline.

Here is the entire transform as a JSON document:

PUT _transform/cloud-billing
{
  "source": {
    "index": [
      "traces-apm*",
      "metrics-apm.*",
      "logs-apm.*"
    ],
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "should": [
                {
                  "exists": {
                    "field": "cloud.provider"
                  }
                }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  },
  "pivot": {
    "group_by": {
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1h"
        }
      },
      "cloud.provider": {
        "terms": {
          "field": "cloud.provider"
        }
      },
      "cloud.region": {
        "terms": {
          "field": "cloud.region"
        }
      },
      "cloud.machine.type": {
        "terms": {
          "field": "cloud.machine.type"
        }
      },
      "service.name": {
        "terms": {
          "field": "service.name"
        }
      }
    },
    "aggregations": {
      "avg_cpu": {
        "avg": {
          "field": "system.cpu.total.norm.pct"
        }
      },
      "percentiles_cpu": {
        "percentiles": {
          "field": "system.cpu.total.norm.pct",
          "percents": [
            75,
            95,
            99
          ]
        }
      },
      "avg_transaction_duration": {
        "avg": {
          "field": "transaction.duration.us"
        }
      },
      "percentiles_transaction_duration": {
        "percentiles": {
          "field": "transaction.duration.us",
          "percents": [
            75,
            95,
            99
          ]
        }
      },
      "count_machines": {
        "cardinality": {
          "field": "cloud.instance.id"
        }
      }
    }
  },
  "dest": {
    "index": "cloud-costs",
    "pipeline": "cloud-costs"
  },
  "sync": {
    "time": {
      "delay": "120s",
      "field": "@timestamp"
    }
  },
  "settings": {
    "max_page_search_size": 1000
  }
}

Once the transform is created and running, we need a Kibana Data View for the index: cloud-costs. For the transaction, use the custom formatter inside Kibana and set its format to “Duration” in “microseconds.”