Disk watermark flood stage (frozen)
Disk usage on one or more frozen nodes has crossed the frozen flood-stage watermark. Elasticsearch might block writes to indices with shards on those nodes.
For a complete list of insights, refer to AutoOps insights.
| Field | Value |
|---|---|
| Component | Elasticsearch |
| Severity | Critical |
| Scope | Node |
| Domains | cluster-health, disk-storage, shard-allocation, stability |
You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.
The default customization settings are:
| Setting | Type | Default |
|---|---|---|
| Early warning margin before the frozen flood-stage watermark | Percentage | 0 |
The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.
The flood stage disk watermark has been exceeded by: 12 GB
The frozen nodes that have crossed the watermark are: es-data-01 and es-data-02
The remaining available disk space until full capacity is reached is: 48 GB
Largest indices on node: logs-prod-000045
AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.
Increase disk size
Condition: Shown when frozen flood-stage watermark is crossed.
Increase disk size on the affected node (current capacity: 500 GB).
Temporary disk watermark increase
Condition: Shown when available disk space > 100GB.
Temporarily raise the disk watermark so you can free space before the cluster restricts shard allocation. After you reclaim disk, return the watermark to its previous value. Use the action below to update cluster.routing.allocation.disk.watermark.high.
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.disk.watermark.high": "90%"
}
}
Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Move shard to less loaded node
Condition: Shown when another frozen node with available disk space exists.
Move shard 0 of logs-prod-000045 from es-data-01 to es-data-02 with the action below to rebalance disk use.
POST _cluster/reroute
{
"commands": [
{
"move": {
"index": "logs-prod-000045",
"shard": 0,
"from_node": "es-data-01",
"to_node": "es-data-02"
}
}
]
}
Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Elasticsearch tracks several disk watermarks as a node fills up. Frozen nodes use a dedicated flood-stage setting, cluster.routing.allocation.disk.watermark.flood_stage.frozen, because the frozen tier is designed to operate near capacity — searchable snapshots and the shared cache model mean a frozen node at high disk utilization is not automatically the same incident as flood-stage pressure on a hot or warm node.
When the frozen flood-stage threshold is crossed, Elasticsearch can block writes to indices that have a shard on the affected frozen node. Searches continue, but write operations through those shards fail until you free space, add frozen capacity, or move shards to a frozen node with available headroom.