Loading

Configure data retention with Streams

Managing data retention across multiple indexes typically requires configuring index lifecycle management (ILM), data stream lifecycle, index templates, and index settings, each in a different place. Streams replaces this with a single UI so you can control storage and meet regulatory or compliance requirements.

The Retention tab provides a single place to manage lifecycle policies for your streams:

  • Set retention periods per stream: Configure how long each stream retains data without touching ILM policies, index templates, or index settings directly.
  • Parent retention cascades to child streams: For wired streams, parent stream retention policies automatically apply to child streams. Override at the child level when a specific child stream needs different retention settings.
  • Monitor storage in one view: See storage size, ingestion averages, and tier distribution so you can align retention periods with storage costs and compliance requirements.
  • Reduce storage with downsampling: For time series data, replace high-resolution metrics with statistical summaries as data ages. This reduces storage costs and lets you retain data longer, with some loss of temporal precision.

To edit data retention in Elastic Stack, you need the following data stream level privileges:

  • manage_data_stream_lifecycle
  • manage_ilm

For more information, refer to Granting privileges for data streams and aliases.

Follow these steps to review your stream's storage footprint, choose a retention method, and apply the policy.

  1. Open the Retention tab

    1. Open Streams from the navigation menu or use the global search field.
    2. Select your stream from the list.
    3. Go to the Data lifecycle tab.

  2. Review storage and ingestion data

    Before setting a retention policy, review the following panels to understand your data's footprint:

    • Storage size: Total data volume and document count for the stream.
    • Ingestion averages: Estimated ingestion per day and per month, based on total stream size divided by stream age.
    • Data lifecycle or ILM policy data tiers: The amount of data in each phase (Hot, Warm, Cold, Frozen) so you can see where data is accumulating.
    • Ingestion over time: A chart of estimated ingestion volume over time to help spot trends or spikes.

    Use this information to decide how long you need to retain data and which retention method best fits your cost and compliance requirements.

    For more information on data retention, refer to Data stream lifecycle.

  3. Choose and configure a retention method

    Select Edit retention method to open the configuration options, then choose one of the following methods:

    • Inherit retention: Use retention settings from the stream's index template (classic streams) or parent stream (wired streams).
      • Classic streams: This preserves existing data streams' behavior while still benefiting from Streams' other features.
      • Wired streams: Child streams automatically inherit lifecycle settings and updates from their parent stream.
    • Set a retention period: Define a minimum number of days before data is deleted. Data stays in the hot phase for best performance. Recommended when retention is specific to a single stream.
    • Follow an ILM policy: Apply an existing ILM policy to automate how data moves through lifecycle phases as it ages. Recommended when you want to share a policy across multiple streams.

    To use the retention settings from the stream's index template (classic streams) or parent stream (wired streams) without setting a custom period or policy:

    1. Select Edit retention method.
    2. Turn on Inherit from index template or parent stream.

    For wired streams, you can override retention for a specific child stream by opening that stream's Retention tab and configuring a different method. The child stream will then use its own policy instead of inheriting from the parent.

    To set a specific retention period:

    1. Select Edit retention method.
    2. Turn off Inherit from index template or parent stream, if enabled.
    3. Select Custom period.
    4. Set the number of days you want to retain data.

    To define a global default retention policy for serverless projects, refer to project settings.

    Select an existing ILM policy to automate how data moves through phases (Hot, Warm, Cold) as it ages. ILM policies let you standardize data retention across Streams and other data streams.

    To follow an existing policy:

    1. Select Edit retention method.
    2. Turn off Inherit from index template or parent stream, if enabled.
    3. Select ILM policy, then choose a pre-defined policy from the list.

    After selecting a policy, you can configure data lifecycle phases directly from the Retention tab.

    If the policy you need doesn't exist, refer to Configure a lifecycle policy to create one.

When a stream follows an ILM policy, the Data lifecycle panel shows the phases defined in that policy as a visual bar. You can edit existing phases or add new ones directly from the Retention tab:

  • To edit an existing phase, select the phase in the Data lifecycle panel and select the edit icon ( ).
  • To add a phase, select Add data phase, then choose a phase.

This opens the Edit data phases window where you can configure or update your phases. The following phases are available:

Hot
The index is actively updated and queried. This is the default phase for all data. Options include enabling read-only access and downsampling.
Warm
The index is updated infrequently but still queried. Set the minimum age for data to move into this phase. Options include enabling read-only access and downsampling.
Cold
The index is rarely updated or queried, and slower query performance is acceptable. Set the minimum age for data to move into this phase. Options include enabling read-only access, downsampling, and searchable snapshots.
Frozen
The index is no longer updated and is queried rarely. Optimized for long-term retention at the lowest possible cost. Set the minimum age for data to move into this phase and configure a snapshot repository. The frozen phase requires a snapshot repository.
Delete
Remove the index after a specified period of time. Set how long data is stored before deletion and optionally delete any associated searchable snapshots.

For more information, refer to index lifecycle management phases and actions.

Searchable snapshots let you search infrequently accessed, read-only data directly from a snapshot repository without needing replica shards, significantly reducing storage costs. A local data cache is kept to allow fast repeat access. They are best suited for historical data that requires infrequent access or archival.

Searchable snapshots are available in the Cold and Frozen phases.

For more information, refer to Searchable snapshots.

Downsampling reduces storage for time series data by replacing original metrics with statistical summaries at a higher sampling interval. For example, metrics sampled every 10 seconds can be consolidated into hourly data points as the data ages, significantly reducing storage while keeping the data queryable.

Downsampling is available in the Hot, Warm, and Cold phases and only applies to time series data streams.

For more information, refer to Downsampling concepts.

When a document fails to ingest because of a processor error or a mapping conflict, it is written to the failure store instead of being dropped. This lets you inspect what went wrong and fix issues using the actual failing documents, rather than losing data silently.

You can enable and configure failure store retention directly from the Retention tab. Select Enable failure store to turn it on and set the retention period for failed documents.

To review and resolve ingestion failures, refer to Manage data quality.