Loading

Searchable snapshots for data streams

A data stream lifecycle (DLM) can automatically convert older backing indices to searchable snapshots on the frozen tier. This frozen transition feature lets you retain data for long periods at low storage cost while keeping it searchable, without manual snapshot or mount operations.

This page explains how frozen transitions work, what you need before they can run, and how to diagnose blocked conversions. To configure frozen_after on a new or existing data stream, refer to Next steps.

Frozen transitions run only when the following conditions are met:

  • An appropriate license is required for this feature.
  • A default snapshot repository is registered. DLM uses the cluster's default snapshot repository to store the frozen snapshot. Refer to Manage snapshot repositories for details.
  • The data stream has data stream lifecycle configured with the frozen_after field. This field is valid only on the main data stream lifecycle and cannot be set on the failure-store lifecycle.

When a backing index's creation date is older than the frozen_after period configured on the data stream lifecycle, DLM converts it to a partially mounted searchable snapshot index allocated on frozen nodes. The conversion happens automatically in the background.

The frozen_after field controls when backing indices become eligible for conversion. You set it on the data stream lifecycle when you create or update a stream or index template. To stop future frozen conversions, update the lifecycle and omit or null the frozen_after field. However, after an index is past its frozen_after time and marked as eligible for conversion, changing frozen_after does not stop that conversion.

The frozen_after value must be a positive time unit value.

For example, if you set frozen_after to 30d and data_retention to 365d, backing indices older than 30 days are converted to frozen searchable snapshots. DLM continues to apply the data_retention period: indices older than 365 days are deleted even after they have been frozen. For more about retention, refer to Setting retention for Elasticsearch data streams.

When a backing index becomes eligible for frozen conversion, DLM performs the following steps on the master node:

  1. Marks the index eligible for transition. The index is marked as needing conversion, ready for a worker to pick up that index and start the conversion process.
  2. Marks the index read-only. A write block is applied and a flush is issued to ensure no in-flight writes are lost.
  3. Clones the index. If the original index has replicas, DLM creates a zero-replica clone (named dlm-clone-<original-index>) to reduce snapshot size. The clone is deleted during cleanup.
  4. Force merges to one segment. The index (or its clone) is force-merged to a single segment per shard, which improves query performance and reduces snapshot size.
  5. Takes a snapshot. DLM snapshots the force-merged index to the default snapshot repository. The snapshot is named dlm-frozen-<original-index>.
  6. Mounts as a searchable snapshot. The snapshot is mounted as a partially mounted index on the frozen tier (named dlm-frozen-<original-index>).
  7. Swaps and deletes the originals. The frozen index replaces the original in the data stream. The original and any clone created during the process are then deleted. This swap is atomic, and queries remain consistent before, during, and after the swap.

A background transition service on the master node scans for indices ready for conversion and runs conversions concurrently. You can configure poll intervals, thread pool size, and queue size using the frozen tier transition settings in elasticsearch.yml.

If a conversion is interrupted, DLM might leave behind orphaned clone indices (dlm-clone-*) or snapshots. A separate cleanup service on the master node periodically removes these artifacts. The cleanup interval is controlled by the dlm.frozen.cleanup.poll_interval setting (default: 1d, minimum: 1h).

Conversion errors appear in the lifecycle explain output for individual backing indices. Use the explain data lifecycle API to inspect status:

				GET .ds-my-stream-*/_lifecycle/explain

For transient failures, DLM records the error and retries the conversion on the next poll cycle. Each conversion step is idempotent, so retries resume from the point at which the error occurred.

Conversions stop retrying when the explain output reports an unrecoverable error, such as a missing snapshot repository or a license compliance failure.

  1. Run the explain API on the affected backing indices and note the reported error.
  2. If the error indicates a missing snapshot repository, re-register the default snapshot repository. Refer to Manage snapshot repositories.
  3. If the error indicates a license compliance failure, renew the license. Refer to Elasticsearch pricing and subscriptions.