Reduce storage
editReduce storage
editThe amount of storage for APM data depends on several factors: how many services you are instrumenting, how much traffic the services see, agent and server settings, and for how long you keep monitoring data.
Reduce the sample rate
editThe transaction sample rate directly influences the number of documents (more precisely, spans) to be indexed and therefore is the most obvious way to reduce storage.
The transaction sample rate is a configuration setting of the agents.
Reducing it does not affect the collection of metrics such as Transactions Per Second.
Reduce collected stacktrace information
editElastic APM agents collect stacktrace
information under certain circumstances.
This can be very helpful by identifying issues in your code,
but it also comes with an overhead at collection time and increases the storage usage.
Stacktrace collection settings are managed in the agents.
Delete data
editYou might want to keep data only for a defined time period and delete old documents periodically, or you might also want to delete data collected for specific services or customers, or delete specific indices. Depending on your use case, you can either delete data periodically with Curator, by using the Delete By Query API, or by using the Kibana Index Management UI.
Delete data periodically
editTo delete data periodically you can use Curator and set up a cron job to run it.
By default APM indices have the pattern apm-%{[beat.version]}-{type}-%{+yyyy.MM.dd}
.
With the curator command line interface you can, for instance, see all your existing indices:
curator_cli --host localhost show_indices --filter_list '[\{"filtertype":"pattern","kind":"prefix","value":"apm-"\}]' apm-6.4.3-error-2018.05.10 apm-6.4.3-error-2018.05.11 apm-6.4.3-error-2018.05.12 apm-6.4.3-sourcemap apm-6.4.3-span-2018.05.10 apm-6.4.3-span-2018.05.11 apm-6.4.3-span-2018.05.12 apm-6.4.3-transaction-2018.05.10 apm-6.4.3-transaction-2018.05.11 apm-6.4.3-transaction-2018.05.12
And then delete any span indices older than 1 day:
curator_cli --host localhost delete_indices --filter_list '[\{"filtertype":"pattern","kind":"prefix","value":"apm-6.4.3-span-"\}, \{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":1,"direction":"older"\}]' INFO Deleting selected indices: [apm-6.4.3-span-2018.05.10, apm-6.4.3-span-2018.05.11] INFO ---deleting index apm-6.4.3-span-2018.05.10 INFO ---deleting index apm-6.4.3-span-2018.05.11 INFO "delete_indices" action completed.
Delete data matching a query
editTo delete documents matching a specific query, for example, all documents with a given context.service.name, use the following request:
POST /apm-*/_delete_by_query { "query": { "bool": { "must": [ { "term": { "context.service.name": { "value": "old-service-name" } } } ] } } }
See delete by query for further information on this topic.
Delete data via Kibana Index Management UI
editSelect the indices you want to delete, and click the Manage indices button to see the available actions. Then click delete indices. Follow the Kibana Index Management documentation to get started with the index management UI.