Prometheus metrics in Elastic Observability: your PromQL runs unchanged

It is 2:14 AM and an alert fires on your Kubernetes cluster. You open Grafana for the memory graph, then Loki for the container logs, then your APM tool to check whether the upstream service was already degrading. Three tabs and eleven minutes later you have a hypothesis, and the label you needed to confirm it was dropped last quarter to keep the metrics bill down.

Elasticsearch now stores Prometheus metrics in the same columnar backend as your logs and traces, at 3.75 bytes per datapoint, with no custom-metric penalty. The graph, the logs, and the traces answer to one query language.

Using an existing Kubernetes cluster (AWS EKS in this example): point Prometheus at Elastic with one config block, see every metric render in Discover with no dashboard to build, run most of your existing PromQL unchanged, find where ES|QL takes you past what PromQL can express, and finish by reading your logs with the same query language.

Nothing about your collection changes. Your scrape configs, relabeling rules, and service discovery carry over as-is. Most of your PromQL comes with you too — see the coverage note in Step 5 for the gaps.

Step 1: get a Prometheus endpoint and an API key from Elastic Observability

You need Elastic Cloud Serverless or Elastic Cloud Hosted. Both expose the Prometheus endpoints with no configuration.

Serverless is the fastest start. Sign in at cloud.elastic.co and create an Observability project. There is nothing to size and nothing to provision.

Elastic Cloud Hosted works the same way for everything in this post, and is the right choice when you need a specific stack version, a specific region topology, or the deployment-level controls that come with a managed cluster.

Find the Prometheus endpoint and create an API key in the UI

For Prometheus endpoint and the API key go to Kibana, click Add data in the left nav.

For Prometheus, scroll to Connect directly to the endpoint at the bottom and select the Prometheus tab.

Two things to copy:

The endpoint. It looks like https://my-observability-project-xxx.ingest.us-west-2.aws.elastic.cloud. Note the .ingest. host. This is not the same host as your Elasticsearch search endpoint or your Kibana URL.

The API key. Click Create key. Copy the value before you close the dialog, because it is not retrievable afterward.

If you want to scope it by hand instead, Open in API keys takes you to the full editor, and the minimum privilege for metrics ingest is:

{
  "ingest": {
    "indices": [
      {
        "names": ["metrics-*"],
        "privileges": ["auto_configure", "create_doc"]
      }
    ]
  }
}

Keep both values. Every remaining step uses them.

Step 2: make sure Prometheus is scraping your Kubernetes cluster

Prometheus should already be scraping your cluster.

If it is not, the shortest path is the kube-prometheus-stack Helm chart, which installs the Prometheus Operator, Prometheus itself, kube-state-metrics, and node-exporter in one command:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

That gives you the metrics you see for the rest of this post:

cAdvisor container metrics (container_cpu_usage_seconds_total, container_memory_working_set_bytes)
kube-state-metrics cluster objects (kube_deployment_spec_replicas, kube_daemonset_status_number_ready).

Step 3: configure Prometheus remote write to the Prometheus endpoint in Step 1

Elasticsearch implements the Prometheus Remote Write protocol natively. There is no adapter, no sidecar, and no translation layer. You add one block and the data flows on the next scrape interval.

If you run the Prometheus Operator

The Operator does not read a prometheus.yml you write by hand. It generates one from the Prometheus custom resource, and authorization.credentials there is a reference to a Kubernetes Secret, not an inline value. Create the secret first:

kubectl create secret generic elastic-prometheus \
  --namespace monitoring \
  --from-literal=api_key='YOUR_API_KEY'

Then reference it from values.yaml:

prometheus:
  prometheusSpec:
    remoteWrite:
      - url: "https://my-observability-project-xxxx.ingest.us-west-2.aws.elastic.cloud:443/api/v1/write"
        authorization:
          type: ApiKey
          credentials:
            name: elastic-prometheus
            key: YOUR_API_KEY

And apply it:

helm upgrade prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  -f values.yaml

Configure Prometheus remote write with prometheus.yml

Same thing, in prometheus.yml:

remote_write:
  - url: "https://YOUR_ES_ENDPOINT/_prometheus/metrics/node/eks/api/v1/write"
    authorization:
      type: ApiKey
      credentials: YOUR_API_KEY

If you run Grafana Alloy uss the following configuration

prometheus.remote_write "elasticsearch" {
  endpoint {
    url = "https://YOUR_ES_ENDPOINT/_prometheus/metrics/node/eks/api/v1/write"
    headers = {"Authorization" = "ApiKey YOUR_API_KEY"}
  }
}

How the remote write URL maps to Elasticsearch data streams

You do not name an index anywhere. There is no index field in the payload and no data stream in your remote_write config. Elasticsearch derives the target data stream from the write path and creates it on the first sample. The two path segments after /metrics/ are the dataset and the namespace:

URL	Data stream
`/_prometheus/api/v1/write`	`metrics-generic.prometheus-default`
`/_prometheus/metrics/{dataset}/api/v1/write`	`metrics-{dataset}.prometheus-default`
`/_prometheus/metrics/{dataset}/{namespace}/api/v1/write`	`metrics-{dataset}.prometheus-{namespace}`

The examples above use /metrics/node/eks/, which writes to metrics-node.prometheus-eks. That is the data stream you will see in Discover in the next step. Use dataset and namespace to separate production from staging, or to give each cluster and each team a data stream with its own retention and downsampling policy.

If you would rather keep a bare /api/v1/write URL, you can route per time series instead: attach data_stream_dataset and data_stream_namespace labels to the series, and they take precedence over the URL path. These two are control fields, so they route the document without being stored in its labels object.

Elasticsearch installs the index template for metrics-*.prometheus-* itself. You do not create templates or mappings.

What Prometheus metrics look like in Elastic Observability

Every Prometheus sample becomes a document. Labels become keyword fields that serve as time series dimensions. The value goes under metrics.<metric_name>:

{
  "@timestamp": "2026-07-02T10:30:00.000Z",
  "data_stream": {
    "type": "metrics",
    "dataset": "node.prometheus",
    "namespace": "eks"
  },
  "labels": {
    "__name__": "container_memory_working_set_bytes",
    "pod": "checkout-7d9f6c4b8-x2kqp",
    "namespace": "oteldemo",
    "container": "checkout",
    "node": "ip-10-0-3-14.us-west-2.compute.internal"
  },
  "metrics": {
    "container_memory_working_set_bytes": 36700160
  }
}

The one gotcha to know now. Elasticsearch infers whether a metric is a counter or a gauge from its name. Names ending in _total, _sum, _count, or _bucket are counters. Everything else is a gauge. That inference is correct for container_cpu_usage_seconds_total and correct for container_memory_working_set_bytes. It is wrong for any metric in your estate that does not follow Prometheus naming convention, and a misclassified metric gets rejected by the function that should accept it: RATE(my_metric::counter) works on counters only. Step 6 shows how to override the inference.

Current limits. Remote Write v1 only. Classic histograms and summaries are supported through their _bucket, _sum, and _count series, each mapped to the right metric type. Native (sparse) histograms and exemplars arrive with Remote Write v2, which is on the roadmap. Staleness markers are not stored or respected, and non-finite values (NaN, Infinity) are dropped silently. See the Prometheus remote write ingest docs for the full list.

Step 4: verify Prometheus metrics are flowing into Elasticsearch

Do not go build a dashboard. Confirm the pipeline first, and Elastic makes that a single command.

In Kibana, open Discover, switch the query editor to ES|QL, and type the name of the data stream you just wrote to:

TS metrics-node.prometheus-eks

That is the whole query. Discover reads the data stream, finds every metric in it, and renders each one as its own time series chart. Fifty metrics, fifty charts, no dashboard to build and no query to write per metric. It reads the metric type and charts each one correctly: gauges as averages, counters as rates, histograms as p95 distributions.

You can also get here without typing anything. Observability → Streams lists every data stream in the cluster. A Time series badge means it is a time series data stream. Click View in Discover and the TS query is filled in for you.

This is your ingest health check. Three things to look for.

Data is flowing. Recent, continuous values. Not gaps, and not a line that stops an hour ago.
Values are plausible. Memory in the tens of megabytes for a small container. CPU as a fraction of a core. Network bytes tracking real traffic.
Coverage is what you expected. If kube_pod_container_status_restarts_total is missing, your kube-state-metrics scrape config is wrong, and you want to know that now rather than when you are building an alert on it.

Widen the time picker before you conclude anything is broken. A 15-minute window over a quiet period makes healthy data look flat.

To list what actually has data rather than what the mapping declares:

TS metrics-node.prometheus-eks | METRICS_INFO | SORT metric_name

The No dimensions selected control above the charts lets you break every chart out by a label: select pod and each chart splits into one series per pod.

Step 5: run PromQL queries on Prometheus metrics in Elastic Observability

If your team writes PromQL, keep writing PromQL. PROMQL is a source command in ES|QL, alongside FROM and TS, and it runs anywhere ES|QL runs: Discover, dashboard panels, and alert rules.

It does not run a separate engine. It parses the expression, resolves each function to its ES|QL equivalent, and builds a TS execution plan, so your PromQL gets the same vectorized, parallel execution as native ES|QL.

Current limits. PROMQL is generally available on Elastic Cloud Serverless and a tech preview on Elastic Stack 9.4, benchmarked at over 80% query coverage against popular Grafana OSS dashboards. The gaps worth knowing before you paste a dashboard in: histogram_quantile is not yet implemented, which matters most because it is how nearly every latency dashboard computes p95; group modifiers (on(...) group_left(...)) and the set operators or, and, and unless are unsupported; and predict_linear, label_join, and label_replace are not yet available. Time buckets also align to fixed calendar boundaries rather than the query start time, so short ranges or large steps can differ slightly from Prometheus. See the PROMQL command reference for the current list.

CPU: per-pod CPU rate with PromQL

The per-second CPU rate across containers, grouped by pod. This is the first thing you look at when something is hot:

PROMQL sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

Broken out by namespace instead, to find which team is burning the cluster:

PROMQL sum by (namespace) (rate(container_cpu_usage_seconds_total[5m]))

Note what came back: a pod column, a step column, and the value column named after the expression itself. That is a normal ES|QL table, which is the whole point and the thing Step 6 builds on.

Memory: working set and RSS with PromQL

Working set is the number that matters for OOM risk. It is what the kernel counts against the limit, and it is not the same as total allocated memory:

PROMQL sum by (pod) (container_memory_working_set_bytes)

Resident set, for comparison, when you are trying to tell a real leak from page cache:

PROMQL sum by (pod) (container_memory_rss)

Network: receive rate by pod with PromQL

PROMQL sum by (pod) (rate(container_network_receive_bytes_total[5m]))

Cluster objects: replica counts with PromQL

Deployments that are not running the replica count they declare:

PROMQL kube_deployment_spec_replicas

One nicety worth calling out: in Prometheus every query needs an explicit start, end, and step. In Kibana you drop all three. The date picker supplies the range and Kibana derives the step, which is why every query above is a single line.

Build a Kibana dashboard from PromQL queries

Every query above is a dashboard panel. In Discover, click Save, or go to Dashboards → Create → Add panel → ES|QL and paste the query in. The date picker drives start, end, and step, so a panel written once works at every zoom level.

Four panels, four one-line PromQL queries, no translation step. If you are coming from Grafana, this is the same dashboard you already have, rebuilt in about five minutes. If you would rather not rebuild it at all, keep Grafana and point its existing Prometheus datasource at Elasticsearch. That is covered at the end of the post.

Step 6: query Prometheus metrics with ES|QL, beyond what PromQL can express

The PROMQL command in ES|QL compiles to TS. These two queries are equivalent:

PROMQL sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

TS metrics-node.prometheus-eks
| WHERE TRANGE(1h)
| STATS SUM(RATE(metrics.container_cpu_usage_seconds_total, 5m)) BY labels.pod, TBUCKET(1m)

The second form is where metrics stop being a separate world from logs and traces — and where the real joins happen.

Top-N and filtering

A TS query returns a normal ES|QL table, so SORT, LIMIT, and WHERE all work downstream. Top-N needs no special function, and no topk. The ten pods using the most memory are a SORT and a LIMIT:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL
| STATS mem = SUM(metrics.container_memory_working_set_bytes) BY `labels.pod`
| SORT mem DESC
| LIMIT 10

Filtering is the same move, a WHERE on the aggregated column. Only the pods over 50 MB:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL
| STATS mem = SUM(metrics.container_memory_working_set_bytes) BY `labels.pod`
| WHERE mem > 50000000
| SORT mem DESC

You actually can pipe the results of a PROMQL query and post-process with regular ES|QL.

Memory usage against the request

A useful question during a memory scare: which pods are using more than they requested, and by how much. That combines two metrics, and ES|QL expresses it in a single pass with filtered aggregations, one MAX per metric:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL AND `labels.namespace` IS NOT NULL
| STATS
    used_memory      = MAX(metrics.container_memory_working_set_bytes),
    requested_memory = MAX(metrics.kube_pod_container_resource_requests)
                       WHERE `labels.resource` == "memory"
  BY `labels.pod`, `labels.namespace`, time_bucket = TBUCKET(5 minute)
| EVAL pct_of_request = 100 * used_memory / requested_memory
| WHERE pct_of_request > 80
| SORT pct_of_request DESC
| LIMIT 100

The WHERE attached to requested_memory is a filtered aggregation: kube_pod_container_resource_requests carries both CPU and memory under a labels.resource dimension, and the filter keeps only the memory rows, so the ratio is memory over memory. Each metric lives in its own documents; the shared BY key lands both aggregates on one row.

Step 7: query Prometheus metrics and logs together with ES|QL

Over the last few sections we used ES|QL and PromQL to explore metrics. ES|QL reads logs too. Here is a quick query against the OpenTelemetry demo running on this cluster:

FROM logs-*
| WHERE TRANGE(30m) AND kubernetes.pod.name == "checkout-9656cbd88-fsr9v"
| STATS events = COUNT(*) BY log.level, TBUCKET(1m)
| SORT events DESC

Same editor, same date picker you used for metrics. Only the source command changed, from TS to FROM, and now you are reading log volume by level per minute. One query language across metrics and logs, no tab switch and no second tool.

What's running in Elastic Observability now

Prometheus is writing to Elastic with one config block. Every metric renders in Discover with no dashboard built. Most of your existing PromQL runs unchanged, and ES|QL takes you further: top-N, filtering, and cross-metric ratios you can pipe into the rest of the language. Metrics and logs answer to the same query in the same window.

You did not drop a single label to get here, and you are not being billed for cardinality.

Next steps: Grafana, dashboard migration, and OpenTelemetry

Keep Grafana if you want it. Elasticsearch exposes a Prometheus-compatible read API at <endpoint>/_prometheus. Point Grafana's existing Prometheus datasource at it, set httpMethod: GET on the datasource, and your PromQL dashboards keep working. Keep Grafana, replace Mimir.
Migrate your Grafana dashboards. When you are ready to move off Grafana rather than point it at Elasticsearch, the Observability Migration Platform translates Grafana dashboards, panels, and alert rules into Kibana-native equivalents. It is a source-agnostic CLI (obs-migrate) that converts what it can and flags what needs a human, with a migration report showing what translated cleanly and where semantic gaps remain, so nothing is silently dropped. The walkthrough covers a Grafana and Datadog migration end to end.
Want to add OpenTelemetry? Read Part 2 if you are also running OpenTelemetry, or if you would rather collect with an OTel Collector than with Prometheus. Both land in the same store and the same queries read across both.

Start a free trial: cloud.elastic.co/registration Docs: elastic.co/docs/solutions/observability

Frequently asked questions

How do I send Prometheus metrics to Elasticsearch? Add a remote_write block to your Prometheus configuration pointing to https://<YOUR_ES_ENDPOINT>/_prometheus/metrics/{dataset}/{namespace}/api/v1/write with an Authorization: ApiKey <YOUR_KEY> header. Elasticsearch implements the Prometheus Remote Write v1 protocol natively — no adapter or sidecar required.

Can I run existing PromQL queries in Elasticsearch? Most of them, yes. ES|QL includes a PROMQL source command that accepts standard PromQL expressions, benchmarked at over 80% coverage against popular Grafana OSS dashboards. It is generally available on Elastic Cloud Serverless and a tech preview on Elastic Stack 9.4. histogram_quantile, predict_linear, label_join, and label_replace are not yet implemented, and group modifiers and the set operators or, and, and unless are unsupported.

Does Elasticsearch charge based on Prometheus metric cardinality? No. Elasticsearch stores Prometheus metrics at 3.75 bytes per datapoint with no cardinality-based billing and no custom-metric penalty, regardless of how many unique label combinations your metrics produce.

How do I route Prometheus metrics to different data streams in Elasticsearch? The two path segments after /metrics/ in the Remote Write URL set the dataset and namespace. /metrics/node/eks/api/v1/write writes to metrics-node.prometheus-eks. You can also route per time series using data_stream_dataset and data_stream_namespace labels, which take precedence over the URL path.

What Prometheus metric types does Elasticsearch support? Elasticsearch supports Remote Write v1, including counters, gauges, classic histograms, and summaries. Counter vs. gauge classification is inferred from the metric name: names ending in _total, _sum, _count, or _bucket are counters; everything else is a gauge. Native histograms and exemplars require Remote Write v2, which is on the roadmap.

Can I query Prometheus metrics and logs together in Elasticsearch? Yes. ES|QL reads both time series metrics and logs in the same query editor with the same date picker. Switch from TS metrics-node.prometheus-eks to FROM logs-* — same syntax, same window, no second tool.