Elasticsearch searchable snapshots
Searchable snapshots transform the way you balance storage costs with search performance. Search across data in snapshots stored on AWS S3, Microsoft Azure Storage, and Google Cloud Storage, which can greatly reduce storage costs.
Time series data grows at an exponential rate. And as data becomes more of a differentiator to how your business and operations run, it can become expensive to store and search all your data. Data tiers, an integrated and automated way of managing the lifecycle of your data, can help you balance your storage costs.
Valuable data in Elasticsearch is stored in the hot tier for fast search queries. As data ages it loses importance and is searched less frequently, this data is moved to the warm tier consisting of lower cost, less performant compute and storage nodes. As data becomes less critical and read-only, it’s stored in snapshots on object storage such as S3. To search this data, however, requires a restore which isn’t immediately accessible for search.
Introducing searchable snapshots a capability that brings S3 and other object stores to life: by allowing you to directly search the data stored in your snapshot. The cold tier utilizing searchable snapshots you can further reduce the cost of storage as much as half with a tradeoff on search performance. This is accomplished by separating the data used for resiliency from the data used for search, giving you the ability to balance storage costs and performance to fit your needs.
"I am really excited about new features in Elasticsearch — especially searchable snapshots. It is a common architecture design, storing snapshots to S3 as a cheap data backup. However, having this data searchable expands our options and could allow us to reduce the total cost of service.
Cold data with S3 replicas has the benefits of both, price and performance. In the future, I am looking forward to exploring the frozen tier for its usability and performance."