Configure downsampling directly in Elastic Streams, no more JSON editing needed

Starting in Elastic 9.4 (generally available), Streams lets you view and configure downsampling directly in the Retention tab, alongside retention periods, data tiers, and ingestion context. Open a stream, see how it ages, and change it in one place.

Elasticsearch has supported downsampling for a while, through ILM policies and data stream lifecycle. But configuring it meant leaving the stream you were looking at, finding the right policy or lifecycle definition, editing JSON, and hoping the intervals were valid. That round trip is gone.

ILM-backed streams

ILM ties downsampling to phases. Each phase (hot, warm, cold) can carry one downsample action with a fixed_interval. Streams now shows these actions on the data lifecycle timeline. Click a phase to open a flyout, set the interval, and watch the timeline update before you save.

Here's the same policy as JSON. This is what Streams is reading and writing when you use the flyout:

{
  "phases": {
    "hot": {
      "min_age": "0ms",
      "actions": {
        "rollover": { "max_age": "1d" },
        "downsample": {
          "fixed_interval": "5m",
          "wait_timeout": "1d"
        }
      }
    },
    "warm": {
      "min_age": "2d",
      "actions": {}
    },
    "cold": {
      "min_age": "4d",
      "actions": {
        "downsample": {
          "fixed_interval": "10m",
          "wait_timeout": "1d"
        }
      }
    },
    "frozen": {
      "min_age": "8d",
      "actions": {
        "searchable_snapshot": {
          "snapshot_repository": "found-snapshots",
          "force_merge_index": true
        }
      }
    },
    "delete": {
      "min_age": "30d",
      "actions": {
        "delete": {
          "delete_searchable_snapshot": true
        }
      }
    }
  }
}

ILM policies are often shared across multiple streams. If you edit one that's in use elsewhere, Streams warns you and offers Save as new policy so you can fork instead of changing the original.

Data stream lifecycle (DLM) streams

Data stream lifecycle takes a different approach: instead of one downsample per phase, you define a sequence of steps (up to 10), each with an after delay and a fixed_interval. Streams shows this as a visual ladder you can build up step by step.

This is the same configuration, but as an API call. Each step in the UI maps to one object in the downsampling array:

{
  "data_retention": "14d",
  "downsampling": [
    { "after": "1d", "fixed_interval": "5m" },
    { "after": "3d", "fixed_interval": "1h" },
    { "after": "7d", "fixed_interval": "4h" }
  ]
}

Get started

• Open Streams in Kibana and select a stream backed by a time series data stream (TSDS).
• Go to the Retention tab. The data lifecycle timeline shows your current downsampling configuration (if any).
• For ILM streams, click a phase on the timeline to open the flyout and configure the downsample interval.
• For DLM streams, add or edit downsample steps directly in the lifecycle view.

Learn more

• Downsampling concepts
• Configuring downsampling (ILM and data stream lifecycle)
• Manage data retention for Streams

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.