Custom index lifecycle management with APM Serveredit
The default ILM policy can be customized to match the needs of your data.
APM Server will assist you in creating a custom index lifecycle policy,
but you must first ensure that ILM and ILM setup are enabled.
Then create a custom policy and map it to APM event types.
All of this can be done directly from the apm-server.yml file.
If you do not want APM Server to create policies and templates, or if you want to use unmanged indices, see the ILM configuration reference for a full list of configuration settings.
Enable ILMedit
Before enabling ILM, ensure the following conditions are true:
- The Elasticsearch instance supports ILM.
-
output.elasticsearchis enabled. -
Custom
indexorindicessettings are not configured.
Enable ILM by setting ilm.enabled to "auto" (the default):
apm-server:
ilm:
enabled: "auto"
Set up ILM with APM Serveredit
Enable APM Server to create managed indices,
and customized polices and templates that are written to Elasticsearch, by setting apm-server.ilm.setup.enabled
to true.
apm-server:
ilm:
setup:
enabled: true
ILM can now be customized via the remaining apm-server.ilm.setup.* configuration options.
Create a custom ILM policyedit
You can define as many policies as you’d like, but they only need to be created once, and will persist through version upgrades. Any change in existing ILM policies will only take place once the next phase is entered.
APM Server doesn’t do any validation on policies.
Instead, if something is incorrectly defined, Elasticsearch will respond with 400 and APM Server won’t connect.
The default ILM policy can be viewed and edited in two places:
-
In your
apm-server.ymlconfiguration file. - On the Index lifecycle policies page in Kibana.
Here’s an example of a custom ILM policy, named apm-error-span-policy,
that applies all four phases to its index lifecycle, including a cold phase with frozen indices,
and a delete phase after 30 days.
ilm:
setup:
policies:
- name: "apm-error-span-policy"
policy:
phases:
hot:
actions:
rollover:
max_size: "50gb"
max_age: "1d"
set_priority:
priority: 100
warm:
min_age: "7d"
actions:
set_priority:
priority: 50
readonly: {}
cold:
min_age: "30d"
actions:
set_priority:
priority: 0
freeze: {}
delete:
min_age: "60d"
actions:
delete: {}
Here’s an example of different policy, named apm-transaction-metric-policy,
that keeps data in the hot, warm, and cold phases for a longer period of time,
and does not delete any data.
ilm:
setup:
policies:
- name: "apm-transaction-metric-policy"
policy:
phases:
hot:
actions:
rollover:
max_size: "50gb"
max_age: "30d"
set_priority:
priority: 100
warm:
min_age: "60d"
actions:
set_priority:
priority: 50
readonly: {}
cold:
min_age: "90d"
actions:
set_priority:
priority: 0
freeze: {}
Head on over to the Elasticsearch documentation to learn more about all available policy phases and actions.
After starting up APM Server, you can confirm the policy was created by using the GET lifecycle policy API:
GET _ilm/policy
Map ILM policies to an event typeedit
If your policy isn’t mapped to an event type, it will not be sent to Elasticsearch.
Policies are mapped to event types using the ilm.setup.mapping configuration.
Using the example from the previous step, we can map the apm-error-span-policy
to errors and spans, and the apm-transaction-metric-policy to transactions and metrics.
ilm:
enabled: "auto"
setup:
mapping:
- event_type: "error"
policy_name: "apm-error-span-policy"
- event_type: "span"
policy_name: "apm-error-span-policy"
- event_type: "transaction"
policy_name: "apm-transaction-metric-policy"
- event_type: "metric"
policy_name: "apm-transaction-metric-policy"
Example ILM configurationedit
Now that we have all of the puzzle pieces, we can put them together to see what a custom ILM configuration might look like.
As a reminder, the example below creates two different policies, one for errors and spans,
and another for transactions and metrics.
The apm-error-span-policy applies all four phases to its index lifecycle, including a cold phase with frozen indices,
and a delete phase after 30 days.
The apm-transaction-metric-policy keeps data in the hot, warm, and cold phases for a longer period of time,
and does not delete any data.
ilm:
enabled: "auto"
setup:
mapping:
- event_type: "error"
policy_name: "apm-error-span-policy"
- event_type: "span"
policy_name: "apm-error-span-policy"
- event_type: "transaction"
policy_name: "apm-transaction-metric-policy"
- event_type: "metric"
policy_name: "apm-transaction-metric-policy"
enabled: true
policies:
- name: "apm-error-span-policy"
policy:
phases:
hot:
actions:
rollover:
max_size: "50gb"
max_age: "1d"
set_priority:
priority: 100
warm:
min_age: "7d"
actions:
set_priority:
priority: 50
readonly: {}
cold:
min_age: "30d"
actions:
set_priority:
priority: 0
freeze: {}
delete:
min_age: "60d"
actions:
delete: {}
- name: "apm-transaction-metric-policy"
policy:
phases:
hot:
actions:
rollover:
max_size: "50gb"
max_age: "30d"
set_priority:
priority: 100
warm:
min_age: "60d"
actions:
set_priority:
priority: 50
readonly: {}
cold:
min_age: "90d"
actions:
set_priority:
priority: 0
freeze: {}