Index lifecycle management overviewedit

You can create and apply Index lifecycle management (ILM) policies to automatically manage your indices according to your performance, resiliency, and retention requirements. Index lifecycle policies can trigger actions such as:

  • Rollover - Redirect an alias to begin writing to a new index when the existing index reaches a certain age, number of docs, or size.
  • Shrink - Reduce the number of primary shards in an index.
  • Force merge - Manually trigger a merge to reduce the number of segments in each shard of an index and free up the space used by deleted documents.
  • Freeze - Make an index read-only and minimize its memory footprint.
  • Delete - Permanently remove an index, including all of its data and metadata.

Typically, you associate a lifecycle policy with an index template so it is automatically applied to new indices. You can also apply a policy manually when you create an index.

ILM simplifies managing indices in hot-warm-cold architectures, which are common when you’re working with time-series data such as logs and metrics.

As an index ages, it moves through four possible phases:

  • Hot—​the index is actively being updated and queried.
  • Warm—​the index is no longer being updated, but is still being queried.
  • Cold—​the index is no longer being updated and is seldom queried. The information still needs to be searchable, but it’s okay if those queries are slower.
  • Delete—​the index is no longer needed and can safely be deleted.

A lifecycle policy controls how an index moves between phases and what actions to perform during each phase. You can specify:

  • The maximum size or age at which you want to roll over to a new index.
  • The point at which the index is no longer being updated and the number of primary shards can be reduced.
  • When to force a merge to permanently delete documents marked for deletion.
  • The point at which the index can be moved to less performant hardware.
  • The point at which the availability is not as critical and the number of replicas can be reduced.
  • When the index can be safely deleted.

For example, if you are indexing metrics data from a fleet of ATMs into Elasticsearch, you might define a policy that says:

  1. When the index reaches 50GB, roll over to a new index.
  2. Move the old index into the warm stage, mark it read only, and shrink it down to a single shard.
  3. After 7 days, move the index into the cold stage and move it to less expensive hardware.
  4. Delete the index once the required 30 day retention period is reached.

To use ILM, all nodes in a cluster must run the same version. Although it might be possible to create and apply policies in a mixed-version cluster, there is no guarantee they will work as intended. Attempting to use a policy that contains actions that aren’t supported on all nodes in a cluster will cause errors.