Manage metrics storage

Metrics data grows quickly. This page walks you through choosing a storage strategy that keeps costs down while retaining the history you still need: confirm your metrics land in a time series data stream (TSDS), then apply the lifecycle settings that match your requirements. Work through the steps in order.

  1. Store metrics in a TSDS

    A time series data stream (TSDS) is an Elasticsearch data stream with index.mode set to time_series. It identifies each series by its dimension fields (for example, the host a CPU metric belongs to), stores values in metric fields typed as counter, gauge, or histogram, and sorts and compresses samples per series. In Elastic's benchmarks, metrics in a TSDS used 70% less disk space than the same data in a regular data stream. The exact impact varies by data set.

    Naming a data stream metrics-* doesn't make it a TSDS. The index template that matches the data stream must set index.mode: time_series. Elastic's built-in templates do this for the OpenTelemetry Protocol (OTLP) and Prometheus remote write paths, so you don't create templates yourself:

    • OTLP metrics from the Managed OTLP Endpoint, Elastic Agent configured as an OTel Collector, and the Elasticsearch OTLP/HTTP endpoint land in metrics-*.otel-* data streams (metrics-generic.otel-default by default). To extend the mappings, use the metrics-otel@custom component template. Refer to Ingest metrics into a TSDS using the OTLP/HTTP endpoint for how that path stores metrics.
    • Prometheus remote write metrics land in metrics-*.prometheus-* data streams (metrics-generic.prometheus-default by default) through the built-in template for that pattern. To extend the mappings, use the metrics-prometheus@custom component template. Refer to Index template for the template that creates those data streams.
    • Elastic Agent integrations store metrics as a TSDS when the integration enables it.

    Choose a TSDS when you add metrics in near real time and in @timestamp order. For logs or traces, use a logs data stream or a regular data stream instead.

    To learn how TSDS storage, dimensions, and metric types work, refer to Time series data streams. To try a TSDS with sample data, follow the TSDS quickstart.

  2. Check metric temporality

    When counters and histograms live in a TSDS, temporality is how each sample relates to the last one. Cumulative values are running totals since the process started. Delta values are the change since the previous sample. Elasticsearch needs the same temporality your producer uses. If they don't match, rates, aggregations, and downsampling can be wrong.

    You don't have to configure temporality if you ingest with the Elasticsearch OTLP/HTTP endpoint, the Managed OTLP Endpoint, or Prometheus remote write. Some paths accept only delta histograms, such as the elasticsearch exporter and, on Elastic Cloud Hosted before 9.5.3, the Managed OTLP Endpoint. For the affected paths and their workarounds, refer to Histogram and counter temporality.

    For how to set and verify temporality, refer to Metric temporality.

  3. Downsample aging metrics

    Downsampling works with a TSDS only. As metrics age, you usually need less detail: last week's CPU at 10-second resolution matters less than the hourly trend. Downsampling replaces those raw samples with coarser buckets plus min, max, sum, and similar stats, so you keep long-term trends at a fraction of the storage cost.

    You typically turn it on in an index lifecycle management (ILM) policy or a data stream lifecycle, not with a one-off API call.

    For configuration and how to query the coarser buckets, refer to Downsampling a time series data stream.

  4. Keep cardinality under control

    Cardinality is the number of unique dimension combinations on a metric. A request-count metric labeled with host, region, and status_code can produce hosts × regions × status_codes separate time series. Each series uses storage and query cost, so an extra high-cardinality label can grow a data stream far faster than extra samples on an existing series. High cardinality is a common cause of unexpected storage growth and slow queries.

    In a TSDS, every new combination of dimension values is a new series, so review which fields are dimensions before you add labels or attributes. For how Elasticsearch defines dimension fields, refer to Dimensions.

  5. Choose a retention policy

    Retention is how long metrics stay in Elasticsearch, when backing indices roll over, and when old data is deleted. Pick the mechanism that matches your deployment and requirements:

    • Use ILM when you want metrics to move through hot, warm, cold, frozen, and delete phases as they age. ILM automates those actions with policies and is available on the Elastic Stack only. Refer to Index lifecycle management for policy phases and how to apply them.

    • Use a data stream lifecycle when you only need rollover, retention, and downsampling, set on the data stream itself. It's available on the Elastic Stack and is the retention option for Serverless. Refer to Data stream lifecycle for how to set retention on the data stream.