Use Elasticsearch as a Drop-In Prometheus Backend for Grafana

Elasticsearch is already one of the most popular plugins in the Grafana ecosystem, and we have now made it much more powerful for metrics usage. If you run Prometheus today and use Grafana to visualize your metrics, you can now point Grafana's Prometheus data source directly at Elasticsearch. No sidecars, no adapters, no pipeline changes required.

Elasticsearch now implements a native Prometheus-compatible API layer, which covers ingestion via Remote Write and querying via PromQL. This post shows the Grafana setup end to end. Companion posts also cover PromQL in Kibana and the Remote Write architecture.

Why use Elasticsearch as a Prometheus backend?

Over the last year, Elasticsearch has become a state-of-the-art metrics store: time series data streams, ES|QL's TS command, and storage and query optimizations that deliver strong metrics performance for Prometheus-style workloads. The Prometheus-compatible API layer makes that engine reachable through the tools your team already uses.

Many teams have invested heavily in Prometheus-based tooling: dashboards, runbooks that reference PromQL queries, on-call workflows built around Grafana panels. Elasticsearch's Prometheus-compatible endpoints let you move metrics storage while keeping those Grafana workflows.

This is particularly relevant if you already use Elasticsearch for logs or traces and want to consolidate your observability data into a single platform, while keeping your Grafana-based workflows intact.

What the Elasticsearch Prometheus API includes

The Elasticsearch Prometheus API exposes three endpoint groups.

Query APIs

The core query endpoints allow Grafana to evaluate PromQL expressions against data stored in Elasticsearch:

GET and POST /_prometheus/api/v1/query_range evaluate a PromQL expression over a time window and return matrix results. This is what powers most Grafana dashboard panels.
GET and POST /_prometheus/api/v1/query evaluate a PromQL expression at a single point in time and return vector results.

Both endpoints implement the standard Prometheus response envelope, including result types (vector, matrix, scalar, string), status codes, and error handling. For POST, send parameters in an application/x-www-form-urlencoded body, matching Prometheus client behavior.

Metadata APIs

Grafana's metric explorer, autocomplete, and variable dropdowns rely on metadata endpoints to discover what's available. Elasticsearch supports:

GET and POST /_prometheus/api/v1/series return time series matching label selectors.
GET and POST /_prometheus/api/v1/labels return all available label names.
GET /_prometheus/api/v1/label/{name}/values returns all values for a given label.
GET /_prometheus/api/v1/metadata returns type and help text for each metric name.

These endpoints power autocomplete and the metric browser in Grafana. The /metadata endpoint additionally enables Grafana's Metrics Drilldown: an interactive metric explorer that displays all available metrics as a grid of live sparklines and lets you drill into any metric without writing a PromQL query.

Index pre-filtering

All query and metadata endpoints accept an optional {index} path segment immediately after /_prometheus/, for example:

GET /_prometheus/metrics-prod-*/api/v1/query_range

This pre-filters the Elasticsearch indices that the PromQL query runs against before any expression evaluation happens. Scoping queries to the relevant data can reduce query work for dashboards that span large volumes of metrics across different data streams.

You can configure a separate Grafana data source per index pattern to give teams scoped access to their own metrics.

Remote Write ingestion

Elasticsearch also implements the Prometheus Remote Write protocol, which lets you ship metrics from Prometheus to Elasticsearch using the standard remote_write configuration. Adding Elasticsearch as a remote write destination requires a single block in your existing Prometheus config:

remote_write:
  - url: "<es_endpoint>/_prometheus/api/v1/write"
    authorization:
      type: ApiKey
      credentials: <api_key>

Metrics are stored in the metrics-generic.prometheus-default data stream by default. You can route metrics from different Prometheus instances or environments into separate data streams using the dataset and namespace path segments:

POST /_prometheus/metrics/{dataset}/api/v1/write stores metrics in metrics-{dataset}.prometheus-default
POST /_prometheus/metrics/{dataset}/{namespace}/api/v1/write stores metrics in metrics-{dataset}.prometheus-{namespace}

How to connect Grafana to Elasticsearch

Step 1: Create a serverless project

Sign in to cloud.elastic.co and create a new Observability serverless project. Once the project is ready, you will land directly in Kibana. To find the Elasticsearch endpoint, go back to the Elastic Cloud console, open Manage > Application endpoints, cluster and component IDs, and click the copy icon next to Elasticsearch. The endpoint looks like:

https://<project-id>.es.<region>.<provider>.elastic.cloud

Step 2: Create API keys

Create two API keys with scoped privileges: one for ingestion, one for querying. Using separate keys means a leaked Grafana key cannot be used to write data, and a leaked ingest key cannot be used to read it.

In your project, open Admin and settings (the ⚙️ icon at the bottom left of the side nav), go to API keys, and create the first key.

Ingest key (prometheus-remote-write): restricts access to writing metrics data streams only. In the Control security privileges section, paste the following role descriptor:

{
  "ingest": {
    "indices": [
      {
        "names": ["metrics-*"],
        "privileges": ["auto_configure", "create_doc"]
      }
    ]
  }
}

Create a second key for Grafana in the same section.

Query key (prometheus-grafana): restricts access to reading metrics data streams only.

{
  "query": {
    "indices": [
      {
        "names": ["metrics-*"],
        "privileges": ["read", "view_index_metadata"]
      }
    ]
  }
}

Copy both key values before closing. You will not be able to retrieve them again.

Step 3: Run Prometheus and Grafana

Create a prometheus.yml that scrapes Prometheus itself and forwards those metrics to Elasticsearch. Replace <es_endpoint> with the endpoint from Step 1 and <ingest_api_key> with the ingest key from Step 2:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

remote_write:
  - url: "<es_endpoint>/_prometheus/api/v1/write"
    authorization:
      type: ApiKey
      credentials: <ingest_api_key>

Next, create the Grafana provisioning directories:

mkdir -p grafana/provisioning/datasources grafana/provisioning/dashboards

Then create a Grafana data source configuration that points at the Elasticsearch Prometheus API. Create grafana/provisioning/datasources/datasource.yml, replacing <es_endpoint> and <query_api_key> with the values from Steps 1 and 2:

apiVersion: 1

datasources:
  - name: Elasticsearch
    type: prometheus
    access: proxy
    url: "<es_endpoint>/_prometheus"
    uid: elasticsearch-prometheus
    isDefault: true
    jsonData:
      httpHeaderName1: Authorization
    secureJsonData:
      httpHeaderValue1: "ApiKey <query_api_key>"

This configures a Prometheus-type data source backed by Elasticsearch. Grafana sends Prometheus queries with POST by default, which Elasticsearch accepts on authenticated HTTPS endpoints such as Serverless.

Create grafana/provisioning/dashboards/dashboards.yml to tell Grafana where to find provisioned dashboards:

apiVersion: 1

providers:
  - name: default
    type: file
    options:
      path: /var/lib/grafana/dashboards

Finally, create a docker-compose.yml to start everything:

services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

  download-dashboard:
    image: curlimages/curl:latest
    user: root
    volumes:
      - dashboards:/dashboards
    command: >
      sh -c 'curl -fsSL https://grafana.com/api/dashboards/3662/revisions/2/download
      | sed "s/\$${DS_THEMIS}/elasticsearch-prometheus/g"
      > /dashboards/prometheus-overview.json'

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    depends_on:
      download-dashboard:
        condition: service_completed_successfully
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=grafana
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - dashboards:/var/lib/grafana/dashboards:ro

volumes:
  dashboards:

The download-dashboard service fetches the Prometheus 2.0 Overview dashboard from the Grafana marketplace and patches it to use the Elasticsearch data source. The sed replaces the dashboard's ${DS_THEMIS} data source placeholder with our data source UID. This is needed because Grafana's provisioning does not resolve these placeholders on its own (grafana#10786). Grafana waits for the download to finish before starting.

Start both with:

docker compose up -d

Prometheus will start scraping its own metrics and shipping them to Elasticsearch every 15 seconds. Give it one or two scrape intervals before opening the dashboard.

Step 4: Open the dashboard

Open Grafana at http://localhost:3000 and log in with admin / grafana. Go to Dashboards and open Prometheus 2.0 Overview.

The dashboard shows your Prometheus self-monitoring metrics, pulled from Elasticsearch via PromQL queries.

Step 5: Explore metrics with Grafana's Metrics Drilldown

Because Elasticsearch implements the Prometheus metadata and discovery endpoints, Grafana's Metrics Drilldown works out of the box.

In Grafana, go to Drilldown > Metrics in the left-hand navigation and select Elasticsearch as the data source. Grafana loads all available metrics from Elasticsearch and displays them as a grid of live sparklines. From there you can filter by label, search by name, and drill into any metric without writing PromQL.

Current limitations and what's next

This is the first implementation and updates should be expected. All of the following are actively being worked on:

PromQL coverage is not yet complete

Queries using group modifiers (for example, on(instance, job)), set operators (or, and, unless), and certain functions like topk are not yet supported.

Form-encoded POST has deployment requirements

POST requests with application/x-www-form-urlencoded bodies require security enabled, TLS on the Elasticsearch HTTP interface, and an authenticated request. Serverless meets these requirements out of the box. If TLS terminates before Elasticsearch and the node sees plain HTTP, use GET with query-string parameters instead.

Only Remote Write v1 is supported

Remote Write v2 support is planned.

Instant queries are not point-in-time yet

The instant query endpoint currently runs a short range query under the hood and returns the last sample. It will be replaced with a proper point-in-time evaluation.

Coming next: broader PromQL function and operator coverage, Remote Write v2, and exemplar endpoints.

Frequently asked questions

Can Grafana query Prometheus metrics stored in Elasticsearch? Yes. Grafana can use Elasticsearch as a Prometheus data source when the URL points to /_prometheus. Queries use PromQL and return the standard Prometheus response format for Grafana dashboards, variables, Metrics Drilldown, and alerting.

Do I need to change Prometheus or Grafana dashboards to use Elasticsearch? You do not need to rewrite PromQL queries or dashboard panels for common Grafana use cases. Configure Prometheus Remote Write to send metrics to Elasticsearch, then point Grafana's Prometheus data source at the Elasticsearch /_prometheus endpoint.

Why use Elasticsearch instead of a separate Prometheus long term storage backend? Using Elasticsearch as a Prometheus backend lets you store metrics with logs and traces under the same access controls and retention model. Recent work on the Elasticsearch metrics engine also delivers strong performance for Prometheus-style workloads. For the benchmark details, see the Elasticsearch metrics performance post.

What PromQL features are supported in Elasticsearch today? Elasticsearch supports common PromQL query patterns used by Grafana dashboards. Advanced group modifiers, set operators, and topk are not yet supported.

Can I limit Grafana queries to specific Elasticsearch indices? Yes. Add an index pattern after /_prometheus/, such as /_prometheus/metrics-prod-*/api/v1/query_range. This pre-filters the Elasticsearch indices before PromQL evaluation and can reduce query work for large metrics deployments.

Prometheus API availability

The Prometheus-compatible API is available now on Elasticsearch Serverless with no additional configuration.

If you run into issues or have feedback, open an issue on the Elasticsearch repository.

Use Elasticsearch as a Drop- In Prometheus Backend for Grafana