Searchable snapshots let you use snapshots to search infrequently accessed and read-only data in a very cost-effective fashion. The cold and frozen data tiers use searchable snapshots to reduce your storage and operating costs.
Searchable snapshots eliminate the need for replica shards, potentially halving the local storage needed to search your data. Searchable snapshots rely on the same snapshot mechanism you already use for backups and have minimal impact on your snapshot repository storage costs.
Using searchable snapshotsedit
Searching a searchable snapshot index is the same as searching any other index.
By default, searchable snapshot indices have no replicas. The underlying snapshot
provides resilience and the query volume is expected to be low enough that a
single shard copy will be sufficient. However, if you need to support a higher
query volume, you can add replicas by adjusting the
If a node fails and searchable snapshot shards need to be recovered elsewhere, there
is a brief window of time while Elasticsearch allocates the shards to other nodes where
the cluster health will not be
green. Searches that hit these shards may fail
or return partial results until the shards are reallocated to healthy nodes.
You typically manage searchable snapshots through ILM. The
searchable snapshots action automatically converts
a regular index into a searchable snapshot index when it reaches the
frozen phase. You can also make indices in existing snapshots searchable by
manually mounting them using the mount snapshot API.
To mount an index from a snapshot that contains multiple indices, we recommend creating a clone of the snapshot that contains only the index you want to search, and mounting the clone. You should not delete a snapshot if it has any mounted indices, so creating a clone enables you to manage the lifecycle of the backup snapshot independently of any searchable snapshots. If you use ILM to manage your searchable snapshots then it will automatically look after cloning the snapshot as needed.
You can control the allocation of the shards of searchable snapshot indices using the same mechanisms as for regular indices. For example, you could use Index-level shard allocation filtering to restrict searchable snapshot shards to a subset of your nodes.
The speed of recovery of a searchable snapshot index is limited by the repository
max_restore_bytes_per_sec and the node setting
indices.recovery.max_bytes_per_sec just like a normal restore operation. By
max_restore_bytes_per_sec is unlimited, but the default for
indices.recovery.max_bytes_per_sec depends on the configuration of the node.
See Recovery settings.
We recommend that you force-merge indices to a single segment per shard before taking a snapshot that will be mounted as a searchable snapshot index. Each read from a snapshot repository takes time and costs money, and the fewer segments there are the fewer reads are needed to restore the snapshot or to respond to a search.
Searchable snapshots are ideal for managing a large archive of historical data. Historical information is typically searched less frequently than recent data and therefore may not need replicas for their performance benefits.
For more complex or time-consuming searches, you can use Async search with searchable snapshots.
You can also use alternative implementations of these repository types, for instance Minio, as long as they are fully compatible. Use the Repository analysis API to analyze your repository’s suitability for use with searchable snapshots.
How searchable snapshots workedit
When an index is mounted from a snapshot, Elasticsearch allocates its shards to data nodes within the cluster. The data nodes then automatically retrieve the relevant shard data from the repository onto local storage, based on the mount options specified. If possible, searches use data from local storage. If the data is not available locally, Elasticsearch downloads the data that it needs from the snapshot repository.
If a node holding one of these shards fails, Elasticsearch automatically allocates the
affected shards on another node, and that node restores the relevant shard data
from the repository. No replicas are needed, and no complicated monitoring or
orchestration is necessary to restore lost shards. Although searchable snapshot
indices have no replicas by default, you may add replicas to these indices by
index.number_of_replicas. Replicas of searchable snapshot shards are
recovered by copying data from the snapshot repository, just like primaries of
searchable snapshot shards. In contrast, replicas of regular indices are restored by
copying data from the primary.
To search a snapshot, you must first mount it locally as an index. Usually ILM will do this automatically, but you can also call the mount snapshot API yourself. There are two options for mounting a snapshot, each with different performance characteristics and local storage footprints:
- Full copy
Loads a full copy of the snapshotted index’s shards onto node-local storage within the cluster. This is the default mount option. ILM uses this option by default in the
Search performance for a full-copy searchable snapshot index is normally comparable to a regular index, since there is minimal need to access the snapshot repository. While recovery is ongoing, search performance may be slower than with a regular index because a search may need some data that has not yet been retrieved into the local copy. If that happens, Elasticsearch will eagerly retrieve the data needed to complete the search in parallel with the ongoing recovery.
- Shared cache
This functionality is in technical preview and may be changed or removed in a future release. Elastic will apply best effort to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
Uses a local cache containing only recently searched parts of the snapshotted index’s data. ILM uses this option by default in the
frozenphase and corresponding frozen tier.
If a search requires data that is not in the cache, Elasticsearch fetches the missing data from the snapshot repository. Searches that require these fetches are slower, but the fetched data is stored in the cache so that similar searches can be served more quickly in future. Elasticsearch will evict infrequently used data from the cache to free up space.
Although slower than a full local copy or a regular index, a shared-cache searchable snapshot index still returns search results quickly, even for large data sets, because the layout of data in the repository is heavily optimized for search. Many searches will need to retrieve only a small subset of the total shard data before returning results.
To mount a searchable snapshot index with the shared cache mount option, you
must configure the
xpack.searchable.snapshot.shared_cache.size setting to
reserve space for the cache on one or more nodes. Indices mounted with the
shared cache mount option are only allocated to nodes that have this setting
You can configure the setting in
Currently, you can configure
xpack.searchable.snapshot.shared_cache.size on any node. In a future release,
you will only be able to configure this setting on nodes with the
You can set
xpack.searchable.snapshot.shared_cache.size to any size between a
couple of gigabytes up to 90% of available disk space. We only recommend larger
sizes if you use the node exclusively on a frozen tier or for searchable
Back up and restore searchable snapshotsedit
You can use regular snapshots to back up a cluster containing searchable snapshot indices. When you restore a snapshot containing searchable snapshot indices, these indices are restored as searchable snapshot indices again.
Before you restore a snapshot containing a searchable snapshot index, you must first register the repository containing the original index snapshot. When restored, the searchable snapshot index mounts the original index snapshot from its original repository. If wanted, you can use separate repositories for regular snapshots and searchable snapshots.
A snapshot of a searchable snapshot index contains only a small amount of metadata which identifies its original index snapshot. It does not contain any data from the original index. The restore of a backup will fail to restore any searchable snapshot indices whose original index snapshot is unavailable.
Reliability of searchable snapshotsedit
The sole copy of the data in a searchable snapshot index is the underlying snapshot, stored in the repository. If the repository fails or corrupts the contents of the snapshot then the data is lost. Although Elasticsearch may have made copies of the data onto local storage, these copies may be incomplete and cannot be used to recover any data after a repository failure. You must make sure that your repository is reliable and protects against corruption of your data while it is at rest in the repository.
The blob storage offered by all major public cloud providers typically offers very good protection against data loss or corruption. If you manage your own repository storage then you are responsible for its reliability.