Prometheus remote write endpoint

In addition to the ingestion of metrics data through the bulk API, Elasticsearch offers an endpoint that natively supports the Prometheus remote write protocol.

The endpoint is available under /_prometheus/api/v1/write.

The Prometheus remote write endpoint allows you to send metrics data directly from Prometheus or any Prometheus remote write-compatible client to Elasticsearch. Data is automatically stored in time series data streams (TSDS).

Ingesting metrics data using the Prometheus remote write endpoint has the following advantages:

• Direct ingestion from Prometheus without needing an intermediate pipeline or converter.
• Simplified index mapping: there's no need to manually create data streams, index templates, or define dimensions and metrics. Metrics are dynamically mapped based on Prometheus naming conventions.
• Prometheus labels are automatically mapped as time series dimensions.
• Metric types are inferred from naming conventions: fields ending in _sum, _count, _total, or _bucket are mapped as counters; all others are mapped as gauges.

Authenticate with an API key or user that has the create_doc and auto_configure index privileges on the data streams the endpoint writes to. create_doc allows writing documents without overwriting existing ones. auto_configure allows the endpoint to create the target data streams on first write.

For example, an API key role descriptor that allows ingesting metrics and exemplars:

{
  "indices": [
    {
      "names": ["metrics-*.prometheus-*", "exemplars-*.prometheus-*"],
      "privileges": ["create_doc", "auto_configure"]
    }
  ]
}
		

To send data from Prometheus to the Elasticsearch remote write endpoint, add a remote_write configuration to your prometheus.yml:

remote_write:
  - url: "https://<es_endpoint>/_prometheus/api/v1/write"
    authorization:
      type: ApiKey
      credentials: <api_key>
    # basic_auth:
    #   username: <user>
    #   password: <password>
		

To send data using Grafana Alloy, use the prometheus.remote_write component:

prometheus.remote_write "elasticsearch" {
  endpoint {
    url = "https://<es_endpoint>/_prometheus/api/v1/write"

    headers = {
      "Authorization" = "ApiKey <api_key>",
    }

    // basic_auth {
    //   username = "<user>"
    //   password = "<password>"
    // }
  }
}
		

The endpoint accepts POST requests with:

• Content type: application/x-protobuf
• Compression: snappy (as mandated by the Prometheus remote write specification) or uncompressed
• Body: Protocol Buffers encoded WriteRequest message as defined by the Prometheus remote write 1.0 specification

By default, metrics are ingested into the metrics-generic.prometheus-default data stream. You can control the target data stream using URL path parameters or per-time-series labels. In both cases, Elasticsearch sanitizes dataset and namespace values, replacing any character that is not alphanumeric, a hyphen, or an underscore with _.

Set the dataset and namespace via URL path segments:

For example, to route infrastructure metrics to a dedicated data stream, set the remote write URL to:

remote_write:
  - url: "https://<es_endpoint>/_prometheus/metrics/infrastructure/production/api/v1/write"
		

This sends data to the metrics-infrastructure.prometheus-production data stream.

You can also route individual time series to different data streams by attaching data_stream_dataset and data_stream_namespace labels to each time series. These labels take precedence over the URL path when set and allow a single remote write endpoint to fan out metrics to multiple data streams.

Elasticsearch treats these as control fields and does not store them in the document's labels object.

If only one of the two labels is present on a time series, the other value falls back to the URL path segment (or the default if no path segment was given).

Use write_relabel_configs to add routing labels before sending:

remote_write:
  - url: "https://<es_endpoint>/_prometheus/api/v1/write"
    write_relabel_configs:
      - target_label: data_stream_dataset
        replacement: myapp
      - target_label: data_stream_namespace
        replacement: production
		

This example attaches data_stream_dataset=myapp and data_stream_namespace=production to every time series in this remote write target and routes all metrics to metrics-myapp.prometheus-production.

Incoming Prometheus time series are mapped as follows:

Metric types are automatically inferred from metric names using dynamic templates:

• Counter: Metric names ending in _sum, _count, _total, or _bucket are mapped as double with time_series_metric: counter.
• Gauge: All other metric names are mapped as double with time_series_metric: gauge.

This means Prometheus histograms and summaries are supported through their component metrics (_sum, _count, _bucket), with each component automatically receiving the correct metric type.

You can override or extend the default metric type inference by creating a metrics-prometheus@custom component template with additional dynamic templates.

For example, to map metrics ending in _counter as counters:

				PUT /_component_template/metrics-prometheus@custom
					{
  "template": {
    "mappings": {
      "dynamic_templates": [
        {
          "counter": {
            "path_match": ["metrics.*_counter"],
            "mapping": {
              "type": "double",
              "time_series_metric": "counter"
            }
          }
        }
      ]
    }
  }
}
		

Custom dynamic templates are merged with the built-in ones. The built-in counter patterns (_sum, _count, _total, _bucket) and the default gauge fallback continue to apply alongside your custom rules.

Elasticsearch automatically installs a built-in index template matching metrics-*.prometheus-* that configures:

• Time series data stream (TSDS) mode
• Labels as passthrough dimensions
• Metrics as passthrough fields with dynamic template-based type inference
• A total field limit of 10,000
• Failure store enabled

A second built-in index template matching exemplars-*.prometheus-* configures the exemplar data streams with the same label mappings.

Exemplars are individual observations that Prometheus can attach to a time series, such as a histogram bucket observation together with the trace ID of the request that produced it. The remote write endpoint stores the exemplars included in a WriteRequest, so you can navigate from a metric to a trace that contributed to it. Exemplars are only stored when you send remote write requests directly to the Elasticsearch endpoint. The managed Prometheus remote write endpoint doesn't forward exemplars yet.

Elasticsearch stores exemplars in a time series data stream named after the metrics data stream of their time series: exemplars of time series routed to metrics-<dataset>.prometheus-<namespace> are written to exemplars-<dataset>.prometheus-<namespace>. Routing by URL path or by labels applies to exemplars in the same way as to samples.

Each exemplar is stored as its own document:

Exemplars never affect whether a request succeeds. If an exemplar can't be indexed, for example because the API key doesn't grant access to exemplars-*.prometheus-*, the samples are still indexed and the request succeeds. Exemplars with non-finite values, exemplars of time series without a __name__ label, and duplicate exemplars that share the same series labels and timestamp are dropped. The remote write protocol has no way to report partial failures, so Elasticsearch only logs these problems at the DEBUG level.

To customize exemplar data streams, for example to change their lifecycle, use the exemplars@custom component template, which applies to all exemplar data streams, or exemplars-prometheus@custom, which applies to Prometheus exemplar data streams only.

Prometheus doesn't send exemplars by default. To include them in remote write requests, start Prometheus with the --enable-feature=exemplar-storage flag and set send_exemplars: true on the remote write target:

remote_write:
  - url: "https://<es_endpoint>/_prometheus/api/v1/write"
    send_exemplars: true
		

Elasticsearch stores exemplars by default If you don't need exemplars, or want to avoid the additional storage, turn it off in one of the following ways.

• Stop sending exemplars at the source (preferred): In Prometheus, remove send_exemplars: true from the remote write target or set it to false, which is the default. This works on every deployment type and saves bandwidth and processing by avoiding sending data that would be discarded anyway.

• Turn off exemplar ingestion in Elasticsearch: Set the dynamic cluster setting xpack.metrics.exemplars.enabled to false. Elasticsearch then discards the exemplars of incoming OTLP metrics and Prometheus remote write requests and only indexes the metric samples. The setting takes effect immediately and applies to both endpoints, including metrics that reach them through the Elastic Cloud Managed OTLP Endpoint on Elastic Cloud Hosted.

  				PUT /_cluster/settings
  					{
    "persistent": {
      "xpack.metrics.exemplars.enabled": false
    }
  }
  		

  On Elastic Cloud Serverless, cluster settings can't be changed. Stop sending exemplars at the source instead, as described in the previous option, or, if you can't change the sender, drop them with an ingest pipeline.

• Drop exemplars with an ingest pipeline: If you can't change the sender or the cluster setting, create an ingest pipeline with a drop processor and set it as the index.default_pipeline of all exemplar data streams through the exemplars@custom component template. Exemplars are still sent to Elasticsearch, but are discarded before they are indexed. This approach works on all deployment types, including Serverless. Because component templates only apply to backing indices created after the change, also update the index settings of any existing exemplar data streams.

• Only the Prometheus remote write 1.0 protocol is supported. Remote write 2.0 is not yet supported.
• Time series with a missing __name__ label are dropped.
• Samples with non-finite values (NaN, Infinity) are silently dropped.
• Staleness markers are not supported.