Requirements for leader indicesedit
This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
Cross-cluster replication works by replaying the history of individual write operations that were performed on the shards of the leader index. This means that the history of these operations needs to be retained on the leader shards so that they can be pulled by the follower shard tasks. The underlying mechanism used to retain these operations is soft deletes. A soft delete occurs whenever an existing document is deleted or updated. By retaining these soft deletes up to configurable limits, the history of operations can be retained on the leader shards and made available to the follower shard tasks as it replays the history of operations.
Soft deletes must be enabled for indices that you want to use as leader indices. Enabling soft deletes requires the addition of some index settings at index creation time. You must add these settings to your create index requests or to the index templates that you use to manage the creation of new indices.
This means that cross-cluster replication can not be used on existing indices. If you have existing data that you want to replicate from another cluster, you must reindex your data into a new index with soft deletes enabled.
Soft delete settingsedit
-
index.soft_deletes.enabled -
Whether or not soft deletes are enabled on the index. Soft deletes can only be
configured at index creation and only on indices created on or after 6.5.0. The
default value is
false. -
index.soft_deletes.retention.operations -
The number of soft deletes to retain. Soft deletes are collected during merges
on the underlying Lucene index yet retained up to the number of operations
configured by this setting. The default value is
0.
For more information about index settings, see Index modules.
Setting soft deletes on indices created by APM Server or Beatsedit
If you want to replicate indices created by APM Server or Beats, and are allowing APM Server or Beats to manage index templates, you need to enable soft deletes on the underlying index templates. To enable soft deletes on the underlying index templates, incorporate the following changes to the relevant APM Server or Beats configuration file.
setup.template.overwrite: true setup.template.settings: index.soft_deletes.enabled: true index.soft_deletes.retention.operations: 1024
For additional information on controlling the index templates managed by APM Server or Beats, see the relevant documentation on loading the Elasticsearch index template.
Setting soft deletes on indices created by Logstashedit
If you want to replicate indices created by Logstash, and are using Logstash to manage index templates, you need to configure soft deletes on a custom Logstash index template. To configure soft deletes on the underlying index template, incorporate the following change to a custom Logstash template.
{
"settings" : {
"index.soft_deletes.enabled" : true,
"index.soft_deletes.retention.operations" : 1024
}
}
Additionally, you will need to configure the Elasticsearch output plugin to use this custom template.
output {
elasticsearch {
template => "/path/to/custom/logstash/template.json"
}
}
For additional information on controlling the index templates managed by Logstash, see the relevant documentation on the Elasticsearch output plugin.