Loading

Configure logs data retention

Your data retention policies define how long Elasticsearch keeps your log data before automatically removing it. Setting an appropriate data retention period helps manage storage costs and keeps your log data manageable.

Manage log data retention in the following ways:

  • Manage log data retention in Streams: Streams provides a single, centralized UI within Kibana that simplifies common tasks, including setting data retention. The Retention tab lets you manage how your stream retains data and provides insight into data ingestion and storage size.
  • Manage log data retention in data streams: Data streams store append-only time series data across multiple indices and use ILM to automate backing index management, including automatic rollover and tiered storage.

A data stream lets you store append-only time series data across multiple indices while giving you a single named resource for requests. Data streams also provide the following benefits:

  • ILM out of the box to automate the management of the backing indices.
  • Automatic rollover to ensure backing indices stay within optimal size and performance limits.
  • Tiered storage (hot, warm, and cold phases) to optimize storage and performance.

Refer to the data stream lifecycle docs for more information.

The logs@lifecycle ILM policy is preconfigured for common logging use cases. View or duplicate the policy at Stack Management → Index Lifecycle Policies or find Index Lifecycle Policies in the global search field.

The logs ILM policy provides a foundation for your logs data streams, but you might need to tailor it to fit your situation. Common modifications include:

  • Adjust hot, warm, and cold phase transitions.
  • Set retention durations for different phases.
  • Update rollover conditions.

Refer to the Customize built-in policies tutorial) for more on modifying the logs ILM policy.

When continuously indexing timestamped documents, you need to periodically roll over to a new index to ensure that backing indices stay within optimal size and performance limits.

Refer to the Automate rollover tutorial for more information.

You can get information about a data stream lifecycle and modify it using the Kibana Index Management tools or the Elasticsearch lifecycle API. Refer to Update the lifecycle of a data stream for more information.