In 7.12, we’re thrilled to be releasing the technical preview of the frozen tier, a feature that allows you to search object stores like S3 directly so you can store and search an unlimited amount of data at the lowest cost possible.

<p>We’re thrilled to announce the technical preview of the frozen tier in 7.12, enabling you to completely decouple compute from storage and directly search data in object stores such as AWS S3, Microsoft Azure Storage, and Google Cloud Storage. The next major milestone in our data tier journey, the frozen tier significantly expands your data reach by storing massive amounts of data for the long haul at much lower cost while keeping it fully active and searchable.&nbsp;
</p><p>We've long supported multiple data tiers for data lifecycle management — hot for high speed and warm for lower cost and performance. Both leverage local hardware for storing your primary data and redundant copies. More recently, we’ve introduced the cold tier, which allows you to store up to twice the data on the same amount of hardware over the warm tier by eliminating the need to store your redundant copies locally. Although the primary data is still local for optimal performance, indices in the cold tier are backed by searchable snapshots stored in your object store for redundancy.&nbsp;
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt19095cb5707a02ca/6055113796151d0ef942a673/Price_Performance.PNG" data-sys-asset-uid="blt19095cb5707a02ca" alt="Price_Performance.PNG" style="display: block; margin: auto;">
</p><p>The frozen tier takes it a big step further by removing the need to store any data locally at all. Instead, it uses searchable snapshots to directly search data stored in the object store without any need to rehydrate it first. A local cache stores recently queried data for optimal performance on repeat searches. Storage costs go down significantly as a result — up to 90% over the hot or warm tiers, and up to 80% over the cold tier. The fully automated lifecycle of your data is now complete: from hot to warm to cold and then to frozen, all while ensuring you have the access and search performance you need at the lowest storage cost possible.
</p><h2>You never outgrow good data</h2><p>Whether it’s for observability, security, or enterprise search, your IT data can keep growing at an exponential rate. It’s commonplace for organizations to ingest and search many terabytes a day. This data is critical not only for day-to-day success but also for historical reference. Unlimited lookback for security investigations, drilling into years of APM data for trend identification, or the occasional discovery for regulatory compliance are all key use cases for keeping your data around and accessible for longer periods of time. Satisfying these use cases, however, can quickly become very expensive if you don’t have the right tools or technology to store the date while ensuring it’s easily searchable.
</p><div class="video embed-container" style="height: 319.725px;">
	<img class="vidyard-player-embed" src="https://videos.elastic.co/watch/pNAEdpYwEYBriYvkAaR6Ci?" data-uuid="pNAEdpYwEYBriYvkAaR6Ci?" data-v="4" data-type="inline" data-autoplay="1" data-loop="1" data-disable_analytics="1" data-hidden_controls="1" data-muted="1" disable_analytics="1" style="width: 100%; margin: auto; display: block;">
<p>That’s where the frozen tier comes in. It opens the door to all of these use cases, now making it cost effective to store years of data at a cost comparable to just archiving data in S3 or another object store. The key difference is that with the frozen tier the data remains fully searchable in Elasticsearch, and all your Kibana dashboards pulling data from the frozen tier will simply just work. Gone are the days of having to manually find and pull data out of the archive, restore it, and then make it available for search. The same is true for balancing trade-offs on which data to retain and which to delete. It all becomes easy and seamless now with the frozen tier.
</p>
<h2>How it works</h2>
<p>The frozen tier leverages searchable snapshots to decouple compute from storage. As data migrates from warm or cold to frozen based on your index lifecycle management (ILM) policy, your indices on local nodes are migrated to S3 or your object store of choice. The cold tier migrates indices to the object store, but it still retains a single, full copy of the data on local nodes to ensure fast and consistent searches. The frozen tier, on the other hand, eliminates the local copy altogether and directly searches the data in the object store. It builds a local cache of recently queried data for faster repeat searches, but the size of the cache only needs to be a fraction of the full size of data stored in the frozen tier.
</p>
<p>With a typical 10% local cache size, this means you only need a handful of local tier nodes to handle many hundreds of terabytes of frozen tier data. Here’s a simple comparison: if your typical warm tier node with 64 GB of RAM manages 10 TB, a cold tier node will be able to handle about twice as much at 20 TB, and a frozen tier node will jump up to 100 TB. This amounts to a 1:1500 RAM-to-storage ratio, and that’s just a conservative estimate.
</p>
<h2>Price vs performance</h2>
<p>So what’s the tradeoff? Well, to no surprise, it’s performance. That’s why we offer these various data tiers, so that you have the flexibility to define the appropriate ILM policies for your organization to determine for how long and how much data should reside in the hot, warm, cold, and frozen tiers. Data stored in the frozen tier is meant to be occasionally searched, and it doesn’t need the faster performance of the other tiers.
</p>
<p>We’ve also made great progress in optimizing for the best possible user experience with slower running searches. We’ve developed asynchronous search in Elasticsearch, which lets us provide a natural experience in Kibana by allowing you to render dashboards in the background and retrieve them later. We’ve also introduced a series of query efficiency improvements to speed up slow-running queries, including skipping indices that don’t match with prefiltering, exiting searches early where possible, using block-max WAND for text search, and so on.&nbsp;
</p>
<p>Because all data is indexed by default in Elasticsearch, searching data in the frozen tier is especially efficient because we can leverage the concise index structures to return results over large datasets very quickly without having to scan the data itself. On top of that, one of the neatest things we’ve done with searchable snapshots is to use our deep expertise in Lucene to only pull down those subsets of the index that are really needed to answer a query.&nbsp;
</p>
<p>Together, all of these optimizations ensure the best and fastest experience possible when searching the frozen tier. Together with a newly redesigned ILM UI to make it far easier to set up and configure your ILM policies, you now have all the tools you need to get going quickly and effectively with the complete set of Elastic’s data tiers.
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5c8881aefd7d6699/605510378a6d130ee5e8289c/ILM_policy_summary.gif" data-sys-asset-uid="blt5c8881aefd7d6699" alt="ILM_policy_summary.gif" style="display: block; margin: auto;">
</p>
<h2>Public or private storage, your choice<br>
</h2>
<p>Our approach at Elastic has always been to offer you the highest possible flexibility with the lowest amount of disruption. In addition to the official support we have for AWS S3, Azure Cloud Storage, Google Cloud Storage and MinIO, we are now releasing a repository test kit for testing and validating any S3-compatible object store to work with searchable snapshots, the cold tier, and the frozen tier.
</p>
<p>Available as an easily consumable API, the test kit allows you to run a series of quick tests against your own S3-compatible object store. If they’re successful, you can use it to store and search your snapshots, and to enable it as the object store for the cold and frozen tiers. Because this is a validation test kit, it’s important to note that this doesn’t mean we’re officially supporting any S3 compatible&nbsp;storage that passes the validation. If an issue is found, the problem will need to be reproducible on supported S3 devices in order for us to fix the issue.
</p>
<h2>Timeline</h2>
<p>The frozen tier is being introduced as a technical preview in Elastic 7.12. The hot, warm, and cold tiers are already generally available, as is the underlying searchable snapshots capability that supports the cold and frozen tiers. The frozen tier is also available in Elastic Cloud as of 7.12, with a simpler, out-of-the-box slider coming soon.&nbsp;
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb103c49e6ead8214/60550fadcc77b20f047a8d4a/TIme_Line.PNG" data-sys-asset-uid="bltb103c49e6ead8214" alt="TIme_Line.PNG" style="display: block; margin: auto;">
</p>
<h2>Get started today</h2>
<p>To get started with the frozen tier, spin up a cluster on <a href="https://info.elastic.co/elasticsearch-service-trial-30d.html" target="_blank">Elastic Cloud</a> or install the latest version of the Elastic Stack. Already have Elasticsearch running? Just upgrade your clusters to 7.12 and give it a try. If you want to try it out on Elastic Cloud, details are available in the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.12/searchable-snapshots.html#searchable-snapshots-frozen-tier-on-cloud" target="_blank">Elastic Cloud documentation</a>. These steps will go away once the frozen slider becomes available. If you’d like to know more about the frozen tier, you can read the <a href="https://www.elastic.co/blog/introducing-elasticsearch-searchable-snapshots" target="_blank">searchable snapshots blog</a>, the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.12/searchable-snapshots.html" target="_blank">searchable snapshots product documentation</a> or the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.10/data-tiers.html" target="_blank">data tiers product documentation</a>.
</p><p><br></p><p><em>The release and timing of any features or functionality described in this document remain at Elastic’s sole discretion. Any features or functionality not currently available may not be delivered on time or at all.</em><br></p></div>

blog-banner-generic-elastic.png

blog-thumb-generic-elastic.png

Directly search S3 with the new frozen tier

In 7.11, we’re excited to announce schema on read in the Elastic Stack. We now offer the best of both worlds on a single platform. Schema on read for flexibility and schema on write for performance. We call our implementation runtime fields.

<p>In 7.11, we’re excited to announce support for schema on read in the Elastic Stack. We now offer the best of both worlds on a single platform — the performance and scale of the existing schema on write mechanism that our users love and depend on, coupled with a new level of flexibility for defining and executing queries with schema on read. We call our implementation of schema on read <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/runtime.html">runtime fields</a>.
</p><p>Runtime fields enable you to create and query fields that are evaluated only at query time. Instead of indexing all the fields in your data as it’s ingested, you can pick and choose which fields are indexed and which ones are calculated only at runtime as you execute your queries. Runtime fields support new use cases and you won’t have to reindex any of your data. Adapt to changing log file formats, query new fields that were not indexed at ingest time, fix errors in index mappings, or iterate on the preparation and perfection of indices over time.
</p> <strong><h2>Schema on write or schema on read — why not both?</h2></strong><p>Schema on write remains the default way that Elasticsearch handles incoming data. All fields in a document are indexed as it’s ingested. This is what makes running searches in Elastic so fast, regardless of the volume of data returned or the number of queries executed. It’s also a big part of what our users love about Elastic.
</p><p>Schema on write works really well if you know your data before it is ingested. That way, the schema can be fully defined in the index mapping. It also requires sticking to that defined schema when queries are run against the index. However, in the real world data can often change. New data sources may arrive, for example. An added layer of flexibility to dynamically extract or query new fields after the data has been indexed adds tremendous value, even if it comes at a slight cost to performance.&nbsp;
</p><p>That’s where schema on read comes in. Data can be ingested in raw form without any indexing, except for certain necessary fields such as timestamp or response code. Other fields can be created on the fly when queries are run against the data. You don’t need to have intimate knowledge of your data ahead of time, nor do you have to predict all the possible ways that the data may eventually be queried. You can change the data structure at any time, even after the documents have been indexed.
</p><p>Here’s what’s unique about how we’ve done it. We’ve built runtime fields on the same Elasticsearch platform — the same architecture, the same tools, and the same interfaces you are already using. There are no new datastores, languages, or components, and there’s no additional procedural overhead. Schema on read and schema on write work together and seamlessly complement each other, so that you can decide which fields to calculate when a query requires them and which fields to index when the document is ingested into Elasticsearch.&nbsp;
</p><p>By offering you the best of both worlds on a single stack, we make it easy for you to decide which combination of schema on write and schema on read works best for your use cases.&nbsp;
</p> <strong><h2>More ways to get value out of your data</h2></strong><p>Define fields at runtime and use them right away. Some of the most useful scenarios are outlined below.
</p> <strong><h3>Add new fields to indexed documents<strong> </strong></h3></strong><p>The ability to add fields to documents is a common requirement for addressing a new need that did not exist when the documents were ingested and indexed. It could be the addition of an entirely new field, or just a field that needs to be calculated (or extracted and transformed) from other fields. For example, the date from the message field may need to be transformed to show the day of the week, or the IP address may need to be extracted from the client.address field so that it can be used in a map visualization. Either way, it’s significantly faster to just add a runtime field to address this than to update the index mapping and reindex the documents.
</p> <strong><h3>Use ephemeral fields</h3></strong><p>Analysts can define ephemeral fields that only exist within the context of a query or visualization, allowing them to work in their own confines without affecting the document index and without overriding the work of others on the same data. Empowering users to autonomously adjust the data for their needs leads to lower costs and higher productivity, cutting out the need to request admin users to make changes to the index or wait for the requests to be approved and completed.
</p> <strong><h3>Fix errors in the index</h3></strong><p>Indexed fields can be shadowed with runtime fields to immediately fix errors that were made when the documents were ingested. This capability makes those indexed documents more usable and eliminates the need for lengthy QA cycles, thus reducing costs and facilitating the rapid loading of documents.
</p> <strong><h3>Iterate during data preparation</h3></strong><p>As new users set up their data sources and begin ingesting data, they may not have a complete picture of how to define or perfect their index mappings. Runtime fields give them the leverage to not have to index all fields ahead of time, and runtime fields can be added gradually based on need. Over time, users can then observe the sizes of their indices, as well as the frequency and performance of different kinds of queries. During peak hours, if the performance of the most frequently run queries is falling behind optimal levels, some of the runtime fields that appear in those queries can be converted to indexed fields. This workflow makes introducing new data sources much faster, reduces the size of indices, increases ingestion speed, and lowers costs.
</p> <strong><h2>7.11: Runtime fields in beta</h2></strong><p>In 7.11, runtime fields are available in beta. You can define a runtime field in an index mapping alongside the indexed fields. The runtime field definition requires nothing more than a datatype and a painless script that specifies how to evaluate the field at query time. Regex can be used in the painless script. You can also define a runtime field without a script, and the system will use the value of the _source field that has the same name as its value. You can then use the Elasticsearch API to query those runtime fields in the same way that you query the indexed fields via Query DSL. You can further define a runtime field on the fly in the query itself to operate within the confines of that query.
</p><p>In Kibana, in 7.11 runtime fields are exposed through the Kibana Query Language (KQL) bar. You can run queries or filters in KQL on runtime fields as you would on indexed fields.
</p> <strong><h2>What’s next for runtime fields</h2></strong><p>While 7.11 provides the fundamentals to get you going with runtime fields, we’re building a number of features to provide you with a comprehensive experience. Going forward, runtime fields will be fully exposed in Kibana for search and exploration in both Discover and Lens. Runtime fields defined in index mappings will be visible alongside the indexed fields to create index patterns, making it easy for you to build your visualizations from any combination of the two.
</p><p>Kibana will also expose a UI editor to define and test runtime fields directly in your query interface, and this editor will be available in the various Kibana apps. This will make it easy for analysts to create runtime fields on the fly by themselves, operating only within the context of the queries and visualizations they are running at any point in time.&nbsp;
</p><p>Beyond that, we’ll add the ability to override indexed fields with runtime fields and convert runtime fields back to indexed fields. This will greatly help the use cases that focus on data preparation and perfecting indices over time. A Grok UI editor will be added, as will the ability to enrich your data at query time from an external index.&nbsp;
</p><p>Balancing performance and flexibility, this simple yet powerful implementation of schema on read opens up new ways of indexing and searching data while complementing the existing schema on write mechanism that Elastic is known for.&nbsp;
</p> <strong><h2>Get started today</h2></strong><p>To get started with runtime fields, spin up a cluster on the <a href="https://info.elastic.co/elasticsearch-service-trial-30d.html">Elasticsearch Service</a> or <a href="https://www.elastic.co/downloads/">install the latest version</a> of the Elastic Stack. Already have Elasticsearch running? Just upgrade your clusters to 7.11 and give it a try. If you’d like to know more about how it works, you can read the <a href="https://elastic.co/blog/getting-started-with-elasticsearch-runtime-fields">runtime fields detailed technical blog</a>&nbsp;or the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/runtime.html">runtime fields documentation</a>.&nbsp;
</p>

blog-banner-search-results-orange.png

blog-thumb-search-results-orange.png

Runtime fields: Schema on read for Elastic

In 7.10, we’re thrilled to be releasing the beta of searchable snapshots, a feature that allows you to reduce storage costs while no significant impact to performance so you can store and search everything.

<p>In 7.10, we're thrilled to be releasing the beta of searchable snapshots, a feature that transforms the way in which you can use your choice of object store (such as AWS S3, Microsoft Azure Storage, Google Cloud Storage, or equivalent) to trade off between dramatically reducing storage costs, ingesting and retaining more data in the Elastic Stack, and the fast search performance you're used to with the Elastic Stack. While we've had support for backing up data to low-cost object stores for a long time, with searchable snapshots you can now use them as an active part of storing and searching your data.
</p><p>We will be using searchable snapshots to power two new first-class data tiers: cold tier, which is also in beta in 7.10, and a future frozen tier. We've long supported multiple data tiers for data lifecycle management - hot for high speed, and warm for lower cost and performance. The new cold tier, powered by searchable snapshots, can reduce your storage costs by up to 50%, increasing the local storage density of your read-only data by offloading the redundant copy of the data to a low-cost object store. The frozen tier, currently in development, will take it a step further and fully store data exclusively in the low-cost object store while keeping it fully searchable, with local cache for fast queries on frequently accessed data. And, like all features we build, there are APIs to directly control how searchable snapshots load, manage, and search data from your object store. These new capabilities will make it both easier and cheaper to manage your growing data volumes in Elastic, empowering you to cost-effectively meet your data retention requirements while opening up new use cases such as giving your team the ability to have unlimited lookback for security investigations or year-over-year performance comparisons on Black Friday.
</p><h2>An evolving journey</h2><p>Time series data is everywhere. It is logs, metrics, traces, security events. It's the backbone of security and observability use cases, and many more. We've been continuously investing in making it easier, faster and more efficient to manage and scale such data over time. This is critical because of how quickly it grows. If you're collecting a terabyte of data per day, for example, that's seven terabytes a week. Kept over several years, this easily grows to petabytes of data. Users need a way to manage this exponential storage growth and still have the ability to search it.
</p><p>Our approach to solving this problem has been to look at the lifecycle of data. When data is first ingested, it's likely to be heavily searched. When you're investigating an incident, for example, you need fast access to all the relevant data to identify and resolve the issue. When an attacker compromises a host or an application, your ability to respond quickly often determines the impact of the breach. But data may also be categorized into different levels of use depending on the source or type. Some data may only be needed to be kept for legal or compliance reasons, or the occasional lookback for comparison purposes. Users therefore need different levels of storage and processing power for these different levels of needs, whether it's based on age, data source, or other criteria.
</p><p>We have been on a mission to give you the ability to balance the cost, performance, and capabilities to meet your needs. This involves investment at all levels of our stack, but a core pillar of our approach is data tiers - managing the lifecycle of data. This concept isn't new, and has been around since the earliest versions of Elasticsearch. Index Lifecycle Management (ILM) provides some conventions to make it easy to manage data across hot (fast machines with SSDs) and warm nodes (lower-cost machines that may have spinning disks), and we've had support for it in Elastic Cloud for years. Snapshot Lifecycle Management (SLM) further makes it easy to use low-cost object stores from AWS, Google, Azure, and on-prem storage vendors to take and store backups. While these snapshots are a key part of many deployments, they haven't been an active part of the data tiering story. Why? Because snapshots haven't been searchable. But that's all changing now with searchable snapshots, which allow us to create new, cheaper data tiers that leverage these low-cost object stores while bringing your backups to life.
</p><h2>Introducing searchable snapshots</h2><p>We're really excited about searchable snapshots, as it allows us to use S3 and other object stores in entirely new ways. While you can continue to use your object store to store your backed-up data as snapshots, you can now bring that object store to life, keeping it always online and available, by making your snapshots directly searchable by Elasticsearch. To build this and give a good experience, we've made changes at all layers of our products - from Kibana, to Elasticsearch, and all the way down to Lucene. In fact, we've used our deep expertise in Lucene to optimize the search mechanism to pull down only those subsets of the snapshot index that are really needed to answer your query or load your dashboard. Searchable snapshots makes the process of recovering or retrieving data from your snapshot-backed indices in S3 or other object store totally seamless and fast, and it has also allowed us to develop new data tiers that offer you more value at a lower cost.
</p><h3>The cold tier</h3><p>The new cold tier, available in beta in the 7.10 release, reduces your cluster storage by up to 50% over the warm tier. It maintains the same level of reliability and redundancy as your hot and warm tiers, with full support for automatic recovery from hardware failures on any of your nodes. This makes it much more cost effective to ask questions of your data like "How does this spike compare to last month?" or "Has this user logged into a restricted system in the last 6 months?"
</p><p>How did we do this? Well, in your hot and warm tiers, half of your disk is used to store replica shards. These redundant copies ensure fast and consistent query performance and provide you with resiliency in case a machine fails. If that happens, a replica seamlessly takes over as the primary, and your indexing and searches continue unabated.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltab265f0cc2b23f59/5fb58cd342256d5ffdf4473b/diagram-primary-shard-disk-replica-shard.jpg" data-sys-asset-uid="bltab265f0cc2b23f59" alt="diagram-primary-shard-disk-replica-shard.jpg">
</p><p>But once your data becomes read-only, redundancy can easily be offloaded. Your snapshot repository is perfect for this, since it's much cheaper to store data in S3 than on either local SSDs or spinning disks. So, in the cold tier, your replica shards are stored in S3 as snapshots. As a result, we've doubled the usable capacity of your cold nodes for the same cost as before - with modest impact to query performance.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf83d69b7e5f12449/5fb58cf609ed4c6f277ee4ae/diagram-higher-performing-disk-es-snapshot-on-s3.jpg" data-sys-asset-uid="bltf83d69b7e5f12449" alt="diagram-higher-performing-disk-es-snapshot-on-s3.jpg">
</p><p>If there's a local node or disk failure in the cold tier, we use searchable snapshots to automatically recover using the replica indices stored as snapshots in S3, making these indexes available to serve search requests in a fraction of the time it takes to do a regular snapshot restore. That's how it all comes together.
</p><h3>The frozen tier</h3><p>Imagine if you had unlimited lookback in your security investigations, or could drill into the raw data from APM to see how your customer behavior has changed over the last two years. That's where the frozen tier comes in, and it opens the door to entirely new use cases with both types and volumes of data that previously just haven't been cost-effective to use Elasticsearch for. Think about how powerful the concept of searchable S3 can be to your business objectives. Actively in development right now, the frozen tier will let you directly search data stored in S3 or your choice of object store. With the frozen tier, there won't be a need to store any of your data locally at all - it can all just be stored as snapshots in S3. And here's the neat thing about the frozen tier - there's no need to pull out your frozen data and rehydrate it in the off chance that you do need to access it for an audit or security investigation. You can just use searchable snapshots to run your queries against it directly.
</p><p>With the frozen tier, what we will be offering is unprecedented: the ability to search nearly an unlimited amount of data, on demand, with costs approaching the cost of storing that data on S3. The fully automated lifecycle of your data becomes complete - from hot, to warm, to cold and then to frozen, all while ensuring you have the access and search performance you need at the lowest storage cost possible.
</p><h3>Optimizing for the best user experience</h3><p>Releasing groundbreaking new capabilities is one thing, and we always strive to do that for you. The other key piece to this is ensuring that everything else works well in unison with these new features to offer you the best possible user experience.
</p><ul>
	<li>Simplified data tier configuration: We've greatly simplified and streamlined the way you set up your data tiers and configure your ILM policies with new roles you assign to your data nodes, which are then used by the Elastic Stack to automatically allocate your data to the appropriate tier when using Index Lifecycle Management. </li>
	<li>Async search: While we've done everything we can to make searching S3 fast, we're not magicians. Queries against S3 will simply just take longer than milliseconds. And when they do, we want to provide the best possible user experience. That's why we've developed an asynchronous search mechanism in Elasticsearch which significantly enhances the experience in Kibana around long running queries. You can now execute a search request asynchronously without having to wait for the results. Instead, you can monitor the progress of the request and retrieve results at a later stage. You can even retrieve partial results as they become available before the search has completed. </li>
	<li>Query efficiency: We've introduced a series of improvements to skip searching indices that don't match or aren't even necessary when you run a search. For example, indices that we know won't have any matches are automatically skipped by pre-filtering based on time or other properties in the data. Searches are also exited early wherever possible, using block-max WAND for text search, sorted queries sorting the shards we search, stopping the search when we have enough matches, and so on.  </li>
</ul><p>Each enhancement offers value just by itself, but together the whole is far greater than the sum of its parts. We're constantly keeping the bigger picture in mind as we develop features, and tying them seamlessly back to all the capabilities already available in the Elastic Stack.
</p><h3>Solving for use cases and our solutions</h3><p>Imagine the value you could unlock if you could easily and cost-effectively search years of logs, metrics, and APM traces with searchable snapshots on object stores like S3. So long, rehydration! With searchable snapshots and Elastic Observability, you'll be able to directly query years of archived data without having to go through the slow and costly process of restoring indices from snapshots before doing a search.
</p><p>What if you could arm threat hunters and analysts with years of high-volume security data sources on object stores like S3, made easily accessible through searchable snapshots? With searchable snapshots and Elastic Security, you can collect high-volume security-related data, such as IDS, NetFlow, DNS, PCAP, or endpoint data, at greater scale and keep it accessible for longer than previously practical on new data tiers that reduce costs while preserving searchability.
</p><p>Finally, consider the ability to search across all of your application content and historical workplace records without breaking the bank by searching object stores with searchable snapshots. Elastic Enterprise Search will also benefit from the new searchable snapshot capabilities launching in the Elastic Stack. Whether you're supporting additional orders of magnitude of application content or searching across historical organizational records that can be safely stored in object stores like S3, you can store all the archived and historical content in a searchable way without breaking the bank.
</p><h2>The journey continues</h2><p>We're excited about the big steps we've taken with the beta releases of searchable snapshots and the cold tier in 7.10. And we're just as excited about the road ahead - a frozen tier to come soon after, as well as both managed cold and frozen tiers with simple sliders in Elastic Cloud to really simplify the signup and subscription flow for users. As always, it's an ongoing journey for us, and what keeps us going is the continuous additional value we keep providing to you with every release.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt932bb37af00919be/5fb58d1465bdd35303e03971/diagram-searchable-snapshots-today-next-future.jpg" data-sys-asset-uid="blt932bb37af00919be" alt="diagram-searchable-snapshots-today-next-future.jpg">
</p><h2>Get started today </h2><p>To get started with searchable snapshots and begin storing data in the cold tier, spin up a cluster on <a href="https://info.elastic.co/elasticsearch-service-trial-30d.html">Elasticsearch Service</a> or <a href="/downloads/">install the latest version</a> of the Elastic Stack. Already have Elasticsearch running? Just upgrade your clusters to 7.10 and give it a try. If you want to know more you can read the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.10/data-tiers.html">data tiers</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.x/searchable-snapshots-apis.html">searchable snapshots</a> documentation.
</p>

Articles by Sajjad Ahmed

Directly search S3 with the new frozen tier

Runtime fields: Schema on read for Elastic

Introducing Elasticsearch searchable snapshots

Products & Solutions

Company

Resources

Solutions

Elastic (ELK) Stack

Register Today

Featured topics

News

Articles by Sajjad Ahmed

Directly search S3 with the new frozen tier

Runtime fields: Schema on read for Elastic

Introducing Elasticsearch searchable snapshots