Warning: Not enough nodes to allocate all shard replicas

Distributing copies of the data (index shard replicas) on different nodes can parallelize processing requests, speeding up search queries. To achieve this, increase the number of replica shards up to the maximum value (total number of nodes minus one), which also protects against hardware failure. If the index has a preferred tier, Elasticsearch will only place the copies of the data for that index on nodes in the target tier.

If a warning is encountered with not enough nodes to allocate all shard replicas, you can influence this behavior by adding more nodes to the cluster (or tier), or by reducing the index.number_of_replicas index setting.

To accomplish this, complete the following steps:

1. Determine which data tier needs more capacity to identify the tier where shards need to be allocated.
2. Resize your deployment to add capacity and accommodate all shard replicas.
3. Check and adjust the index replicas limit to determine the current value and reduce it if needed.

You can run the following step using either API console or direct Elasticsearch API calls.

To determine which tiers an index's shards can be allocated to, use the get index setting API to retrieve the configured value for the index.routing.allocation.include._tier_preference setting:

				GET /my-index-000001/_settings/index.routing.allocation.include._tier_preference?flat_settings
		

The response looks like this:

{
  "my-index-000001": {
    "settings": {
      "index.routing.allocation.include._tier_preference": "data_warm,data_hot"
    }
  }
}
		

1. Represents a comma-separated list of data tier node roles this index is allowed to be allocated on. The first tier in the list has the highest priority and is the tier the index is targeting. In this example, the tier preference is data_warm,data_hot, so the index is targeting the warm tier. If the warm tier lacks capacity, the index will fall back to the data_hot tier.

After you've identified the tier that needs more capacity, you can resize your deployment to distribute the shard load and allow previously unassigned shards to be allocated.

If it is not possible to increase capacity by resizing your deployment, you can reduce the number of replicas of your index data. You achieve this by inspecting the index.number_of_replicas index setting index setting and decreasing the configured value.

1. Use the get index settings API to retrieve the configured value for the index.number_of_replicas index setting.

   				GET /my-index-000001/_settings/index.number_of_replicas
   		

   The response looks like this:

   {
     "my-index-000001" : {
       "settings" : {
         "index" : {
           "number_of_replicas" : "2"
         }
       }
     }
   }
   		

   1. Represents the currently configured value for the number of replica shards required for the index

2. Use the _cat/nodes API to find the number of nodes in the target tier:

   				GET /_cat/nodes?h=node.role
   		

   The response looks like this, containing one row per node:

   himrst
   mv
   himrst
   		

   You can count the rows containing the letter representing the target tier to know how many nodes you have. See Query parameters for details. The example above has two rows containing h, so there are two nodes in the hot tier.

3. Use the update index settings API to decrease the value for the total number of replica shards required for this index. As replica shards cannot reside on the same node as primary shards for high availability, the new value needs to be less than or equal to the number of nodes found above minus one. Since the example above found 2 nodes in the hot tier, the maximum value for index.number_of_replicas is 1.

   				PUT /my-index-000001/_settings
   					{
     "index" : {
       "number_of_replicas" : 1
     }
   }
   		

   1. The new value for the index.number_of_replicas index configuration is decreased from the previous value of 2 to 1. It can be set as low as 0 but configuring it to 0 for indices other than searchable snapshot indices may lead to temporary availability loss during node restarts or permanent data loss in case of data corruption.

Reduce the index.number_of_replicas index setting.