Apply an ILM policy to an integration namespace

Data retention settings in the integration policy editor lets you choose an index lifecycle management (ILM) policy for a namespace, instead of editing component templates by hand. Fleet applies your choice to every data stream the integration defines in that namespace.

The ILM policy is tied to an (integration, namespace) pair rather than to a single integration policy. Every integration policy that uses the same integration and namespace shares one ILM policy, and different namespaces can use different policies. Use this feature when you want one retention policy for a whole environment, such as production.

• Namespace index templates must be enabled for the namespace. Until they are, Data retention settings stays unavailable.
• The ILM policy must already exist. Create it on the Index Lifecycle Policies page first. You can't create a policy from the integration policy editor.
• You need the manage_ilm cluster privilege. Fleet enforces this privilege in the UI and in the API.
• You can't change Data retention settings on a managed integration policy.

1. Find Integrations in the navigation menu or use the global search field.
2. Select the integration you want, then:
   • To create an integration policy, click Add <integration>.
   • To edit an existing one, open the Integration policies tab and select the policy.
3. In the Integration settings section, expand Advanced options.
4. In the Namespace field, enter the namespace you want a custom ILM policy for.
5. Turn on Use dedicated index templates for this namespace, if it isn't on already.
6. From Data retention settings, select an ILM policy.
7. Save the policy.

You can complete these steps in a single pass. You don't need to save the integration policy before you select an ILM policy.

Data retention settings lists the ILM policies you created. Built-in policies that Elasticsearch manages, such as logs and metrics, and internal policies whose names begin with a dot, aren't listed.

Use the Fleet update package API to apply an ILM policy to a namespace:

				PUT kbn:/api/fleet/epm/packages/system
					{
  "namespace_customization_settings": {
    "production": { "ilm_policy": "my-retention-policy" }
  }
}
		

To clear the policy for a namespace, pass an empty object for it:

				PUT kbn:/api/fleet/epm/packages/system
					{
  "namespace_customization_settings": {
    "production": {}
  }
}
		

Fleet rejects the request in these cases:

• The namespace isn't opted in for namespace index templates.
• The ILM policy doesn't exist.
• You don't have the manage_ilm cluster privilege.
• The namespace doesn't match the allowed namespace prefixes for your Kibana space.

Fleet applies the changes asynchronously after the request succeeds. For the full request schema, refer to the update package API.

For each data stream, Fleet creates a managed component template that holds the index.lifecycle.name setting. These component templates are named using the same pattern as the namespace index templates:

<type>-<dataset>@namespace.<namespace>
		

Each namespace index template references its component template in composed_of, after the <namespace>@custom component template and before the data stream-level @custom template. For example, logs-system.application@namespace.production contains:

[
  "logs@mappings",
  "logs@settings",
  "logs-system.application@package",
  "logs@custom",
  "system@custom",
  "production@custom",
  "logs-system.application@namespace.production",
  "logs-system.application@custom",
  "ecs@mappings",
  ".fleet_globals-1",
  ".fleet_agent_id_verification-1"
]
		

Because the data stream-level @custom template comes later in the list, it takes precedence. If you set index.lifecycle.name in logs-system.application@custom, that setting wins over the ILM policy you apply here.

Fleet creates a component template for every data stream the integration defines, including data streams that no integration policy has enabled yet. A data stream you enable later uses the same ILM policy.

To verify the managed component template and the reference to it:

				GET _component_template/logs-system.application@namespace.production
		

				GET _index_template/logs-system.application@namespace.production
		

To check which ILM policy a data stream uses:

				GET _data_stream/logs-system.application-production
		

Fleet applies your changes with a background task, so the component templates can take a few seconds to appear after you save. The task also triggers a rollover for the affected data streams on a best-effort basis.

ILM policies take effect when new backing indices are created. Existing backing indices keep the policy they were created with. If the automatic rollover didn't happen, you can roll over a data stream yourself with the Elasticsearch rollover API:

				POST logs-system.application-production/_rollover
		

• To use a different ILM policy, select it in Data retention settings on any integration policy that uses that integration and namespace. The change applies to all integration policies for the pair.
• To stop using a custom ILM policy while the namespace keeps its dedicated index templates, select None (use default). Fleet deletes the managed component templates, and the data streams return to the ILM policy defined by the integration's base index templates. The <namespace>@custom component template and any settings or mappings in it stay in place.
• If you turn off Use dedicated index templates for this namespace, or remove the namespace from Namespaces with dedicated index templates on the integration's Settings tab, Fleet also clears the ILM policy for that namespace. You don't need to clear it first. This removes the namespace index templates as well, so use None (use default) if you want to keep your other namespace-level customizations.

As with any other ILM change, existing backing indices keep the policy they were created with until the data stream rolls over.

• You can apply only one ILM policy per (integration, namespace) pair. Two integration policies that use the same integration and namespace can't use different ILM policies.
• The ILM policy applies to every data stream the integration defines. To target a single data stream, use Scenario 2 for all namespaces, or Scenario 3 for one namespace.
• You can select ILM policies only. To use a data stream lifecycle instead, configure it outside of Fleet.
• An index.lifecycle.name setting in a data stream-level @custom component template overrides the policy you apply here.

• Customize data streams with namespace index templates
• Tutorials: Customize data retention policies
• Index lifecycle management (ILM)
• Create an ILM policy
• Manage lifecycle policies for integrations data