spaceedit

This filtertype will iterate over the actionable list and match indices when their cumulative disk consumption exceeds disk_space gigabytes. They are first ordered by age, or by alphabet, so as to guarantee the oldest indices are deleted first. They will remain in, or be removed from the actionable list based on the value of exclude.

Deleting Indices By Spaceedit

- filtertype: space
  disk_space: 100

This filtertype is for those who want to retain indices based on disk consumption, rather than by a set number of days. There are some important caveats regarding this choice:

  • Elasticsearch cannot calculate the size of closed indices. Elasticsearch does not keep tabs on how much disk-space closed indices consume. If you close indices, your space calculations will be inaccurate.
  • Indices consume resources just by existing. You could run into performance and/or operational snags in Elasticsearch as the count of indices climbs.
  • You need to manually calculate how much space across all nodes. The total you give will be the sum of all space consumed across all nodes in your cluster. If you use shard allocation to put more shards or indices on a single node, it will not affect the total space reported by the cluster, but you may still run out of space on that node.

These are only a few of the caveats. This is still a valid use-case, especially for those running a single-node test box.

For use cases, where "like" indices are being counted, and their name pattern guarantees date sorting is equal to alphabetical sorting, it is unnecessary to set use_age to True, as index names will be sorted in reverse order by default. For this case, this means that disk space calculations will start beginning with the newest indices, and proceeding through to the oldest.

Age-based sortingedit

- filtertype: space
  disk_space: 100
  use_age: True
  source: creation_date

For use cases, where "like" indices are being counted, and their name pattern guarantees date sorting is equal to alphabetical sorting, it is unnecessary to set use_age to True, as index names will be sorted in reverse order by default. For this case, this means that disk space calculations will start beginning with the newest indices, and proceeding through to the oldest.

Where this is not the case, the use_age setting can be used to ensure that index or snapshot ages are properly considered for sorting:

All of the age-related settings from the age filter are supported.

Reversing sortingedit

Important

The reverse setting is ignored when use_age is True. When use_age is True, sorting is always from newest to oldest, ensuring that old indices are always selected first.

Using the default configuration, reverse is True. Given These indices:

index1 10g
index2 10g
index3 10g
index4 10g
index5 10g

And this filter:

- filtertype: space
  disk_space: 21

The indices will be sorted alphabetically and iterated over in the indicated order (the value of reverse) and the total disk space compared after adding the size of each successive index. In this example, that means that index5 will be added first, and the running total of consumed disk space will be 10g. Since 10g is less than the indicated threshold of 21, index5 will be removed from the actionable list.

On the next iteration, the amount of space consumed by index4 will be added, which brings the running total to 20g, which is still less than the 21 threshold, so index4 is also removed from the actionable list.

This process changes when the iteration adds the disk space consumed by index3. Now the running total crosses the 21 threshold established by disk_space (the running total is now 30g). Even though it is only 1g in excess of the total, index3 will remain in the actionable list. The threshold is absolute.

The remaining indices, index2 and index1 will also be in excess of the threshold, so they will also remain in the actionable list.

So in this example index1, index2, and index3 will be acted on by the action for this block.

If you were to run this with loglevel set to DEBUG, you might see messages like these in the output:

...Removed from actionable list: index5, summed disk usage is 10GB and disk limit is 21.0GB.
...Removed from actionable list: index4, summed disk usage is 20GB and disk limit is 21.0GB.
...Remains in actionable list: index3, summed disk usage is 30GB and disk limit is 21.0GB.
...Remains in actionable list: index2, summed disk usage is 40GB and disk limit is 21.0GB.
...Remains in actionable list: index1, summed disk usage is 50GB and disk limit is 21.0GB.

In some cases, you may wish to filter in the reverse order. To accomplish this, you set reverse to False:

- filtertype: space
  disk_space: 21
  reverse: False

This time indices index1 and index2 will be the ones removed from the actionable list, leaving index3, index4, and index5 to be acted on by the given action.

If you were to run this with loglevel set to DEBUG, you might see messages like these in the output:

...Removed from actionable list: index1, summed disk usage is 10GB and disk limit is 21.0GB.
...Removed from actionable list: index2, summed disk usage is 20GB and disk limit is 21.0GB.
...Remains in actionable list: index3, summed disk usage is 30GB and disk limit is 21.0GB.
...Remains in actionable list: index4, summed disk usage is 40GB and disk limit is 21.0GB.
...Remains in actionable list: index5, summed disk usage is 50GB and disk limit is 21.0GB.

Required settingsedit

Optional settingsedit