The Elasticsearch cluster is specified using a list of
NodeSet represents a group of Elasticsearch nodes sharing the same specification (both Elasticsearch configuration and Kubernetes Pod configuration).
apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: quickstart spec: version: 8.6.2 nodeSets: - name: master-nodes count: 3 config: master: true data: false volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: standard - name: data-nodes count: 10 config: master: false data: true volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 1000Gi storageClassName: standard
The Elasticsearch resource above defines two
NodeSets: one for master nodes, using 10Gi volumes, and one for data nodes, using 1000Gi volumes. The Elasticsearch cluster is composed of 13 nodes: 3 master nodes and 10 data nodes.
ECK handles smooth upgrades from one cluster specification to another. You can apply a new Elasticsearch specification at any time.
Here are a few examples based on the Elasticsearch specification above:
To add five additional Elasticsearch data nodes: change
count: 15in the
To increase the RAM memory limit of data nodes to 32Gi: set a different resources limits in the existing
To replace dedicated master and dedicated data nodes by nodes having both master and data roles: replace the 2 existing
NodeSetsby a single one with a different name and the corresponding Elasticsearch configuration settings.
To upgrade Elasticsearch from version
7.3.0: change the value in the
NodeSet changes with no downtime and makes sure that:
- Before a node is removed, the relevant data is migrated to other nodes (but see Limitations).
When a cluster topology changes, the Elasticsearch orchestration settings
_cluster/voting_config_exclusionsare adjusted accordingly.
Rolling upgrades are performed safely, reusing the
PersistentVolumesof the upgraded Elasticsearch nodes.
Behind the scenes, ECK translates each
NodeSet specified in the Elasticsearch resource into a StatefulSet in Kubernetes. The StatefulSet specification is based on the
countcorresponds to the number of replicas in the
StatefulSet, each replica leading to the creation of a
Pod, which corresponds to a single Elasticsearch node
podTemplatecan be used to specify custom settings for the Elasticsearch Pod, overriding the default ones set by ECK on the generated
StatefulSetname is built from the Elasticsearch resource name and the
NodeSetname. Each Pod will be assigned the
StatefulSetname suffixed by an ordinal. The corresponding Elasticsearch node has the same name as the
Pod creation is handled by the
StatefulSet controller in Kubernetes. ECK relies on the OnDelete StatefulSet update strategy since it needs full control over when and how Pods get upgraded to a new revision.
Pod is removed and recreated (maybe with a newer revision), the
StatefulSet controller makes sure that the
PersistentVolumes attached to the original
Pod are then attached to the new
Depending on how the
NodeSets are updated, ECK handles the Kubernetes resources reconciliation in various ways.
When a new
NodeSetis added to the Elasticsearch resource, ECK creates the corresponding
StatefulSet. It also sets up Secrets and ConfigMaps to hold the TLS certificates and Elasticsearch configuration files.
When the node count of an existing
NodeSetis increased, ECK increases the replicas of the corresponding
When the node count of an existing
NodeSetis decreased, ECK migrates data away from the corresponding Elasticsearch nodes to remove, then decreases the replicas of the corresponding
StatefulSet, once data migration is over. Corresponding PersistentVolumeClaims are automatically removed.
When an existing
NodeSetis removed, ECK migrates data away from the corresponding Elasticsearch nodes to remove, decreases the
StatefulSetreplicas accordingly, then finally removes the corresponding
When the specification of an existing
NodeSetis updated (for example the Elasticsearch configuration, or the
PodTemplateresources requirements), ECK performs a rolling upgrade of the corresponding Elasticsearch nodes. In order to do so, it follows Elasticsearch rolling upgrade best practices, to slowly upgrade
Podsto the newest revision while preventing unavailability of the Elasticsearch cluster. In most cases, it corresponds to restarting Elasticsearch nodes one by one and reusing the same
PersistentVolumedata. Note that some cluster topologies may cause the cluster to be unavailable during the upgrade.
When an existing
NodeSetis renamed, ECK performs the creation of a new
NodeSetwith the new name, and the removal of the old
NodeSet, according to the
NodeSetcreation and removal patterns described above. Elasticsearch data is migrated away from the deprecated
NodeSetbefore removal. The Elasticsearch resource update strategy controls how many nodes can exist above or below the target node count during the upgrade.
In all these cases, ECK handles
StatefulSet operations according to the Elasticsearch orchestration best practices, by adjusting the orchestration settings
Based on how Kubernetes and
StatefulSets operate, ECK orchestration has the following limitations:
Storage requirements (including volume size) of an existing
NodeSetcannot be updated. StatefulSet volumes expansion is not available in Kubernetes yet. To upgrade the storage size, you can create a new
NodeSet, or rename an existing one. Renaming a
NodeSetautomatically creates a new
StatefulSetwith the specified storage size. The original
StatefulSetis removed once the Elasticsearch data is migrated to the nodes of the new
Cluster availability is not be guaranteed in the following cases:
- During the rolling upgrade of single-node clusters
- For clusters that have indices with no replicas
If an Elasticsearch node holds the only copy of a shard, this shard becomes unavailable while the node is upgraded. Clusters with more than one node and at least one replica per index are considered best practice.
Pendingduring a rolling upgrade if the Kubernetes scheduler cannot re-schedule them back. This is especially important when using local
PersistentVolumes. If the Kubernetes node bound to a local
PersistentVolumedoes not have enough capacity to host an upgraded
Podwhich was temporarily removed, that
Podwill stay Pending.
Rolling upgrades can make progress if the Elasticsearch cluster health is green. ECK will also make progress if the cluster health is yellow under the following conditions:
A cluster version upgrade is in progress and some
Podsare not up to date
- There are no initializing or relocating shards
- A cluster version upgrade is in progress and some
If the above conditions are met, then ECK can delete a
Pod for upgrade even if the cluster health is yellow as long as the
Pod is not holding the last available replica of a shard.
The health of the cluster is deliberately ignored in the following cases:
If all the Elasticsearch nodes of a
NodeSetare unavailable, probably caused by a misconfiguration, the operator ignores the cluster health and upgrades nodes of the
If an Elasticsearch node to upgrade is not healthy, and not part of the Elasticsearch cluster, the operator ignores the cluster health and upgrades the Elasticsearch node.
- Elasticsearch versions cannot be downgraded. For example it is impossible to downgrade an existing cluster from version 7.3.0 to 7.2.0. This is not supported by Elasticsearch.
Advanced users may force an upgrade by manually deleting
Pods themselves. The deleted
Pods will be automatically recreated at the latest revision.
Operations that reduce the number of nodes in the cluster cannot make progress without user intervention if the Elasticsearch index replica settings are incompatible with the intended downscale. Specifically if the Elasticsearch index settings demand a higher number of shard copies than data nodes in the cluster after the downscale operation, ECK cannot migrate the data away from the node about to be removed. Users can address this in the following ways:
- Adjust the Elasticsearch index settings to a number of replicas that allow the desired node removal
auto_expand_replicasto automatically adjust the replicas to the number of data nodes in the cluster
Intro to Kibana
ELK for Logs & Metrics