Universal Profiling index life cycle management | Elastic Observability [8.13]

› › ›

« Manage data storage Configure probabilistic profiling »

Universal Profiling index life cycle managementedit

Index lifecycle policies allow you to automate the lifecycle of your profiling indices as they grow and age. A default policy is applied, but you can customize it based on your business needs.

Default policyedit

The default Universal Profiling index lifecycle policy includes the following rollover and delete definitions:

Rollover: Rollover prevents a single index from growing too large and optimizes indexing and search performance. After an age or size metric threshold is met, a new index is created and all subsequent updates are written to the new index.
Delete: The delete phase permanently removes the index after a time threshold is met.

The following table lists the default thresholds for rollover and delete:

Rollover	Warm tier	Delete
after 30 days or 50 GB	after 30 days	after 60 days

The rollover condition blocks phase transitions which means that indices are kept 30 days after rollover on the hot tier.

To view the Universal Profiling index lifecycle policy in Kibana, navigate to Stack Management → Index Lifecycle Management, and search for profiling.

Default ILM policies can change between minor versions. This is not considered a breaking change as index management should continually improve and adapt to new features.

Configure a custom index lifecycle policyedit

Complete the following steps to configure a custom index lifecycle policy.

Step 1: Create an index lifecycle policyedit

Navigate to Stack Management → Index Lifecycle Policies.
Click Create policy.
Name your new policy, for example custom-profiling-policy.
Customize the policy to your liking.
Click Save policy.

See Manage the index lifecycle to learn more about ILM policies.

Step 2: Apply the index lifecycle policyedit

To apply your new index lifecycle policy for Universal Profiling, create a new component template named profiling-ilm@custom.

To apply a custom ILM policy, you must name the component template profiling-ilm@custom. Other names are not supported.

From the Index Management page, select the Component Template tab and click Create component template.
Enter profiling-ilm@custom as the name and click Next.
In Index settings, set the ILM policy name created in the previous step:
```
{
  "lifecycle": {
    "name": "custom-profiling-policy"
  }
}
```
Continue to the Review step, and select the Request tab. Your request should look similar to the following image.

If it does, click Create component template.

Step 3: Rollover indicesedit

To confirm that Universal Profiling is now using the new index template and ILM policy, navigate to Dev Tools and run the following:

GET _ilm/policy/custom-profiling-policy

The name of the custom ILM policy chosen in Step 1.

If the custom policy is already applied, the result should include the following:

{
    "in_use_by": {
      "indices": [
        ...
      ],
      "data_streams": [
        ...
        "profiling-events-all",
        ...
      ],
      "composable_templates": [
        "profiling-stackframes",
        "profiling-symbols-global",
        "profiling-metrics",
        "profiling-stacktraces",
        "profiling-executables",
        "profiling-hosts",
        "profiling-events"
      ]
    }
}

If the result is empty, the custom ILM policy is not yet in use. New ILM policies only take effect when new indices are created, so either wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB), or force a rollover using the Elasticsearch rollover API:

POST /profiling-events-5pow01/_rollover/
POST /profiling-events-5pow02/_rollover/
POST /profiling-events-5pow03/_rollover/
POST /profiling-events-5pow04/_rollover/
POST /profiling-events-5pow05/_rollover/
POST /profiling-events-5pow06/_rollover/
POST /profiling-events-5pow07/_rollover/
POST /profiling-events-5pow08/_rollover/
POST /profiling-events-5pow09/_rollover/
POST /profiling-events-5pow10/_rollover/
POST /profiling-events-5pow11/_rollover/
POST /profiling-events-all/_rollover/
POST /profiling-executables/_rollover/
POST /profiling-hosts/_rollover/
POST /profiling-metrics/_rollover/
POST /profiling-stackframes/_rollover/
POST /profiling-stacktraces/_rollover/
POST /profiling-symbols-global/_rollover/

After the rollover, the custom ILM policy will be applied to new indices and data streams.

« Manage data storage Configure probabilistic profiling »