In some cases, it may be useful to reindex data into ILM-managed indices. This is more complex than simply attaching policies to existing indices as described in the previous section, and requires pausing indexing during the reindexing process. However, this technique may be useful in cases where periodic indices were created with very small amounts of data leading to excessive shard counts, or for indices which grow steadily over time, but have not been broken up into time-series indices leading to shards which are much too large, situations that cause significant performance problems.

Before getting started with reindexing data, the new index structure should be set up. For this section, we’ll be using the same setup described in {ilm-imit} with existing indices.

First, we’ll set up a policy with rollover, and can include any additional phases required. For simplicity, we’ll just use rollover:

PUT _ilm/policy/mylogs_condensed_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d",
            "max_size": "50G"
          }
        }
      }
    }
  }
}

And now we’ll update the index template for our indices to include the relevant ILM settings:

PUT _template/mylogs_template
{
  "index_patterns": [
    "ilm-mylogs-*" 
  ],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index": {
      "lifecycle": {
        "name": "mylogs_condensed_policy", 
        "rollover_alias": "mylogs" 
      }
    }
  },
  "mappings": {
    "properties": {
      "message": {
        "type": "text"
      },
      "@timestamp": {
        "type": "date"
      }
    }
  }
}

And create the first index with the alias specified in the rollover_alias setting in the index template:

PUT ilm-mylogs-000001
{
  "aliases": {
    "mylogs": {
      "is_write_index": true
    }
  }
}

All new documents should be indexed via the mylogs alias at this point. Adding new data to the old indices during the reindexing process can cause data to be added to the old indices, but not be reindexed into the new indices.

By default, ILM only checks rollover conditions every 10 minutes. Under normal indexing load, this usually works well, but during reindexing, indices can grow very, very quickly. We’ll need to set the poll interval to something shorter to ensure that the new indices don’t grow too large while waiting for the rollover check:

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": "1m" 
  }
}

We’re now ready to reindex our data using the reindex API. If you have a timestamp or date field in your documents, as in this example, it may be useful to specify that the documents should be sorted by that field - this will mean that all documents in ilm-mylogs-000001 come before all documents in ilm-mylogs-000002, and so on. However, if this is not a requirement, omitting the sort will allow the data to be reindexed more quickly.

POST _reindex
{
  "source": {
    "index": "mylogs-*", 
    "sort": { "@timestamp": "desc" }
  },
  "dest": {
    "index": "mylogs", 
    "op_type": "create" 
  }
}

Once this completes, indexing new data can be resumed, as long as all new documents are indexed into the alias used above. All data, existing and new, can be queried using that alias as well. We should also be sure to set the ILM poll interval back to its default value, because keeping it set too low can cause unnecessary load on the current master node:

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": null
  }
}

All of the reindexed data should now be accessible via the alias set up above, in this case mylogs. Once you have verified that all the data has been reindexed and is available in the new indices, the existing indices can be safely removed.