Elasticsearch searchable snapshots
Searchable snapshots transform the way you balance storage costs with search performance. Search across data in snapshots stored on AWS S3, Microsoft Azure Storage, and Google Cloud Storage, which can reduce storage costs by half.
Time series data grows at an exponential rate. And as data becomes more of a differentiator to how your business and operations run, it can become expensive to store and search all your data. Data tiers, an integrated and automated way of managing the lifecycle of your data, can help you balance your storage costs.
Valuable data in Elasticsearch is stored in the hot tier for fast search queries. As data ages it loses importance and is searched less frequently, this data is moved to the warm tier consisting of lower cost, less performant compute and storage nodes. As data becomes less critical and read-only, it’s stored in snapshots on object storage such as S3. To search this data, however, requires a restore which isn’t immediately accessible for search.
Introducing searchable snapshots a capability that brings S3 and other object stores to life: by allowing you to directly search the data stored in your snapshot. The cold tier utilizing searchable snapshots you can further reduce the cost of storage as much as half with a tradeoff on search performance. This is accomplished by separating the data used for resiliency from the data used for search, giving you the ability to balance storage costs and performance to fit your needs.
"I am really excited about new features in Elasticsearch — especially searchable snapshots. It is a common architecture design, storing snapshots to S3 as a cheap data backup. However, having this data searchable expands our options and could allow us to reduce the total cost of service.
Cold data with S3 replicas has the benefits of both, price and performance. In the future, I am looking forward to exploring the frozen tier for its usability and performance."
Coming soon: the frozen tier. The frozen tier lets you search snapshots stored exclusively within S3 or your choice of object store. Imagine having the ability to directly search nearly an unlimited amount of data on demand. — all while having the access and search performance you need at the lowest storage cost possible. Built with features like async search and background search, the frozen tier will provide the best search experience for this class of storage.