Red or yellow cluster health status
editRed or yellow cluster health status
editA red or yellow cluster health status indicates one or more shards are not assigned to a node.
- Red health status: The cluster has some unassigned primary shards, which means that some operations such as searches and indexing may fail.
- Yellow health status: The cluster has no unassigned primary shards but some unassigned replica shards. This increases your risk of data loss and can degrade cluster performance.
When your cluster has a red or yellow health status, it will continue to process searches and indexing where possible, but may delay certain management and cleanup activities until the cluster returns to green health status. For instance, some ILM actions require the index on which they operate to have a green health status.
In many cases, your cluster will recover to green health status automatically. If the cluster doesn’t automatically recover, then you must manually address the remaining problems so management and cleanup activities can proceed.
Diagnose your cluster status
editCheck your cluster status
Use the cluster health API.
resp = client.cluster.health( filter_path="status,*_shards", ) print(resp)
response = client.cluster.health( filter_path: 'status,*_shards' ) puts response
const response = await client.cluster.health({ filter_path: "status,*_shards", }); console.log(response);
GET _cluster/health?filter_path=status,*_shards
A healthy cluster has a green status
and zero unassigned_shards
. A yellow
status means only replicas are unassigned. A red status means one or
more primary shards are unassigned.
View unassigned shards
To view unassigned shards, use the cat shards API.
resp = client.cat.shards( v=True, h="index,shard,prirep,state,node,unassigned.reason", s="state", ) print(resp)
response = client.cat.shards( v: true, h: 'index,shard,prirep,state,node,unassigned.reason', s: 'state' ) puts response
const response = await client.cat.shards({ v: "true", h: "index,shard,prirep,state,node,unassigned.reason", s: "state", }); console.log(response);
GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state
Unassigned shards have a state
of UNASSIGNED
. The prirep
value is p
for
primary shards and r
for replicas.
To understand why an unassigned shard is not being assigned and what action you must take to allow Elasticsearch to assign it, use the cluster allocation explanation API.
resp = client.cluster.allocation_explain( filter_path="index,node_allocation_decisions.node_name,node_allocation_decisions.deciders.*", index="my-index", shard=0, primary=False, ) print(resp)
response = client.cluster.allocation_explain( filter_path: 'index,node_allocation_decisions.node_name,node_allocation_decisions.deciders.*', body: { index: 'my-index', shard: 0, primary: false } ) puts response
const response = await client.cluster.allocationExplain({ filter_path: "index,node_allocation_decisions.node_name,node_allocation_decisions.deciders.*", index: "my-index", shard: 0, primary: false, }); console.log(response);
GET _cluster/allocation/explain?filter_path=index,node_allocation_decisions.node_name,node_allocation_decisions.deciders.* { "index": "my-index", "shard": 0, "primary": false }
Fix a red or yellow cluster status
editA shard can become unassigned for several reasons. The following tips outline the most common causes and their solutions.
Re-enable shard allocation
editYou typically disable allocation during a restart or other
cluster maintenance. If you forgot to re-enable allocation afterward, Elasticsearch will
be unable to assign shards. To re-enable allocation, reset the
cluster.routing.allocation.enable
cluster setting.
resp = client.cluster.put_settings( persistent={ "cluster.routing.allocation.enable": None }, ) print(resp)
response = client.cluster.put_settings( body: { persistent: { 'cluster.routing.allocation.enable' => nil } } ) puts response
const response = await client.cluster.putSettings({ persistent: { "cluster.routing.allocation.enable": null, }, }); console.log(response);
PUT _cluster/settings { "persistent" : { "cluster.routing.allocation.enable" : null } }
Recover lost nodes
editShards often become unassigned when a data node leaves the cluster. This can occur for several reasons, ranging from connectivity issues to hardware failure. After you resolve the issue and recover the node, it will rejoin the cluster. Elasticsearch will then automatically allocate any unassigned shards.
To avoid wasting resources on temporary issues, Elasticsearch delays allocation by one minute by default. If you’ve recovered a node and don’t want to wait for the delay period, you can call the cluster reroute API with no arguments to start the allocation process. The process runs asynchronously in the background.
resp = client.cluster.reroute( metric="none", ) print(resp)
response = client.cluster.reroute( metric: 'none' ) puts response
const response = await client.cluster.reroute({ metric: "none", }); console.log(response);
POST _cluster/reroute?metric=none
Fix allocation settings
editMisconfigured allocation settings can result in an unassigned primary shard. These settings include:
- Shard allocation index settings
- Allocation filtering cluster settings
- Allocation awareness cluster settings
To review your allocation settings, use the get index settings and cluster get settings APIs.
resp = client.indices.get_settings( index="my-index", flat_settings=True, include_defaults=True, ) print(resp) resp1 = client.cluster.get_settings( flat_settings=True, include_defaults=True, ) print(resp1)
response = client.indices.get_settings( index: 'my-index', flat_settings: true, include_defaults: true ) puts response response = client.cluster.get_settings( flat_settings: true, include_defaults: true ) puts response
const response = await client.indices.getSettings({ index: "my-index", flat_settings: "true", include_defaults: "true", }); console.log(response); const response1 = await client.cluster.getSettings({ flat_settings: "true", include_defaults: "true", }); console.log(response1);
GET my-index/_settings?flat_settings=true&include_defaults=true GET _cluster/settings?flat_settings=true&include_defaults=true
You can change the settings using the update index settings and cluster update settings APIs.
Allocate or reduce replicas
editTo protect against hardware failure, Elasticsearch will not assign a replica to the same node as its primary shard. If no other data nodes are available to host the replica, it remains unassigned. To fix this, you can:
- Add a data node to the same tier to host the replica.
-
Change the
index.number_of_replicas
index setting to reduce the number of replicas for each primary shard. We recommend keeping at least one replica per primary.
resp = client.indices.put_settings( settings={ "index.number_of_replicas": 1 }, ) print(resp)
response = client.indices.put_settings( body: { 'index.number_of_replicas' => 1 } ) puts response
const response = await client.indices.putSettings({ settings: { "index.number_of_replicas": 1, }, }); console.log(response);
PUT _settings { "index.number_of_replicas": 1 }
Free up or increase disk space
editElasticsearch uses a low disk watermark to ensure data nodes have enough disk space for incoming shards. By default, Elasticsearch does not allocate shards to nodes using more than 85% of disk space.
To check the current disk space of your nodes, use the cat allocation API.
resp = client.cat.allocation( v=True, h="node,shards,disk.*", ) print(resp)
response = client.cat.allocation( v: true, h: 'node,shards,disk.*' ) puts response
const response = await client.cat.allocation({ v: "true", h: "node,shards,disk.*", }); console.log(response);
GET _cat/allocation?v=true&h=node,shards,disk.*
If your nodes are running low on disk space, you have a few options:
- Upgrade your nodes to increase disk space.
- Delete unneeded indices to free up space. If you use ILM, you can update your lifecycle policy to use searchable snapshots or add a delete phase. If you no longer need to search the data, you can use a snapshot to store it off-cluster.
-
If you no longer write to an index, use the force merge API or ILM’s force merge action to merge its segments into larger ones.
resp = client.indices.forcemerge( index="my-index", ) print(resp)
response = client.indices.forcemerge( index: 'my-index' ) puts response
const response = await client.indices.forcemerge({ index: "my-index", }); console.log(response);
POST my-index/_forcemerge
-
If an index is read-only, use the shrink index API or ILM’s shrink action to reduce its primary shard count.
resp = client.indices.shrink( index="my-index", target="my-shrunken-index", ) print(resp)
response = client.indices.shrink( index: 'my-index', target: 'my-shrunken-index' ) puts response
const response = await client.indices.shrink({ index: "my-index", target: "my-shrunken-index", }); console.log(response);
POST my-index/_shrink/my-shrunken-index
-
If your node has a large disk capacity, you can increase the low disk watermark or set it to an explicit byte value.
resp = client.cluster.put_settings( persistent={ "cluster.routing.allocation.disk.watermark.low": "30gb" }, ) print(resp)
const response = await client.cluster.putSettings({ persistent: { "cluster.routing.allocation.disk.watermark.low": "30gb", }, }); console.log(response);
PUT _cluster/settings { "persistent": { "cluster.routing.allocation.disk.watermark.low": "30gb" } }
Reduce JVM memory pressure
editShard allocation requires JVM heap memory. High JVM memory pressure can trigger circuit breakers that stop allocation and leave shards unassigned. See High JVM memory pressure.
Recover data for a lost primary shard
editIf a node containing a primary shard is lost, Elasticsearch can typically replace it
using a replica on another node. If you can’t recover the node and replicas
don’t exist or are irrecoverable, Allocation
Explain will report no_valid_shard_copy
and you’ll need to do one of the following:
- restore the missing data from snapshot
- index the missing data from its original data source
- accept data loss on the index-level by running Delete Index
-
accept data loss on the shard-level by executing Cluster Reroute allocate_stale_primary or allocate_empty_primary command with
accept_data_loss: true
Only use this option if node recovery is no longer possible. This process allocates an empty primary shard. If the node later rejoins the cluster, Elasticsearch will overwrite its primary shard with data from this newer empty shard, resulting in data loss.
resp = client.cluster.reroute( metric="none", commands=[ { "allocate_empty_primary": { "index": "my-index", "shard": 0, "node": "my-node", "accept_data_loss": "true" } } ], ) print(resp)
const response = await client.cluster.reroute({ metric: "none", commands: [ { allocate_empty_primary: { index: "my-index", shard: 0, node: "my-node", accept_data_loss: "true", }, }, ], }); console.log(response);
POST _cluster/reroute?metric=none { "commands": [ { "allocate_empty_primary": { "index": "my-index", "shard": 0, "node": "my-node", "accept_data_loss": "true" } } ] }