<h2>First, some context</h2><a href="https://www.elastic.co/blog/hot-warm-architecture-in-elasticsearch-5-x">Hot-Warm</a> architectures are often used when we want to get the most out of our hardware. It is particularly useful when we have time-based data, like <a href="https://www.elastic.co/blog/observability-with-the-elastic-stack">logs, metrics, and APM data</a>. Most of these setups rely on the fact that this data is read-only (after ingest) and that indices can be time(or size)-based. So they can be easily deleted based on our desired retention period. In this architecture, we categorize Elasticsearch nodes into two types: 'hot' and 'warm'.Hot nodes hold the most recent data and thus handle all indexing load. Since recent data is usually the most frequently queried, these nodes will be the most powerful in our cluster: fast storage, high memory and CPU. But that extra power gets expensive, so it doesn’t make sense to store older data that isn’t queried as often on a hot node.On the other hand, warm nodes will be the ones dedicated to long-term storage in a more cost-efficient way. Data on the warm nodes is not as likely to be queried often and data within the cluster will move from hot to warm nodes based on our planned retention (achieved through <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/shard-allocation-filtering.html">shard allocation filtering</a>), while still being available online for queries.Starting with Elastic Stack 6.3, we've been building new features to enhance hot-warm architectures and simplify working with time-based data.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte8ad3502d7fba4c1/5c98d0a5c9ab24634dcd6d86/hot-warm-features-frozen-indices-transp.png" data-sys-asset-uid="blte8ad3502d7fba4c1" alt="hot-warm-features-frozen-indices-transp.png" width="550" style="margin: auto;display: block;"/><a href="https://www.elastic.co/blog/data-rollups-in-elasticsearch-you-know-for-saving-space">Data rollups</a> were first introduced in version 6.3, to save storage. In time-series data, we want fine-grained detail for the most recent data. But it is very unlikely that we will need the same for historical data, where we will typically look at datasets as a whole. And this is where rollups came in, since starting at version 6.5 we can <a href="https://www.elastic.co/blog/how-to-create-manage-and-visualize-elasticsearch-rollup-data-in-kibana">create, manage and visualize rollup data in Kibana</a>.Shortly after, we added <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/modules-snapshots.html#_source_only_repository">source-only snapshots</a>. This minimal snapshots will provide a significant reduction of snapshot's storage, with the tradeoff of having to reindex data if we want to restore and query. This has been available since version 6.5.In version 6.6, we released two powerful features, Index Lifecycle Management (ILM) and Frozen Indices.<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/index-lifecycle-management.html">ILM</a> provides the means to automate your indices management over time. It simplifies moving indices from hot to warm, allows deletion when indices are too old, or automates force merging indices down to one segment.And for the rest of this blog, we’ll talk about frozen indices.<h2>Why freeze an index?</h2>One of the biggest pain points with “old” data is that, regardless of age, indices still have a significant memory footprint. Even if we place them on cold nodes, they still use heap.A possible solution could be to close the index. If we close an index, it won’t require memory, but we will need to re-open it to run a search. Reopening indices will incur an operational cost and also require the heap it was using before being closed.On each node, there is a memory (heap) to storage ratio that will limit the amount of storage available per node. It may vary from as low as 1:8 (memory:data) for memory intensive scenarios, to something close to 1:100 for less demanding memory use cases.This is where frozen indices come in. What if we could have indices that are still open — keeping them searchable — but do not occupy heap? We could add more storage to data nodes that hold frozen indices, and break the 1:100 ratio, understanding the tradeoff that searches might be slower.When we freeze an index, it becomes read-only and its transient data structures are dropped from memory. In turn, when we run a query on frozen indices, we will have to load the data structures to memory. Searching a frozen index doesn’t have to be slow. Lucene heavily depends on the filesystem cache which might have enough capacity to retain significant portions of your index in memory. In such a case searches are comparable in speed per shard. Yet, a frozen index is still throttled such that only one frozen shard is executing per node at the same time. This aspect might slow down searches compared to unfrozen indices.<h2>How freezing works</h2>Frozen indices are searched through a dedicated, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/modules-threadpool.html#search-throttled">searched-throttled</a> threadpool. This by default uses a single thread, to ensure that frozen indices are loaded into memory one at a time. If concurrent searches are happening, they will get in the queue to add additional protections to prevent nodes from running out of memory.So, in a hot-warm architecture, we will now be able to transition indices from hot to warm, and then be able to freeze them before archiving or deleting them, allowing us to reduce our hardware requirements.Before frozen indices, to reduce infrastructure cost we had to snapshot and archive our data, which adds a significant operational cost. We would have to restore data if we needed to search again. Now, we can keep our historical data available for search, without a significant memory overhead. And if we need to write again on an already froze index, we can just <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/unfreeze-index-api.html">unfreeze it</a>.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta19e364f7d3f763a/5c98d09ec9ab24634dcd6d80/index-states-transp.png" data-sys-asset-uid="blta19e364f7d3f763a" alt="index-states-transp.png" width="400" style="margin: auto;display: block;"/><h2>How to freeze an Elasticsearch index</h2>Frozen indices are easy to implement in your cluster, so let’s get started on how to use the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/freeze-index-api.html">Freeze index API</a> and how to <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-index-frozen.html" target="_self">search on frozen indices</a>.First, we'll start by creating some sample data on a test index.<pre class="prettyprint">POST /sampledata/_doc { "name": "Jane", "lastname": "Doe" } POST /sampledata/_doc { "name": "John", "lastname": "Doe" } </pre>And then check that our data has been ingested. This should return two hits:<pre class="prettyprint">GET /sampledata/_search </pre>As a best practice, before you freeze an index it’s recommended to first run a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html">force_merge</a>. This will ensure that each shard has only a single segment on disk. It will also provide much better compression and simplifies the data structures we will need when running an aggregation or a sorted search request on the frozen index. Running searches on a frozen index with multiple segments can have a significant performance overhead up to multiple orders of magnitude.<pre class="prettyprint">POST /sampledata/_forcemerge?max_num_segments=1 </pre>The next step is to just invoke a freeze on our index via de Freeze index API endpoint.<pre class="prettyprint">POST /sampledata/_freeze </pre><h2>Searching frozen indices</h2>Now that it’s frozen, you’ll see that regular searches won’t work. The reason for this is that, to limit memory consumption per node, frozen indices are&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/searching_a_frozen_index.html">throttled</a>. &nbsp;Since we could target a frozen index by mistake, we’ll prevent accidental slowdowns by specifically adding ignore_throttled=false to the request.<pre class="prettyprint">GET /sampledata/_search?ignore_throttled=false { "query": { "match": { "name": "jane" } } } </pre>Now we can check the status of our new index, by running the following request:<pre class="prettyprint">GET _cat/indices/sampledata?v&amp;h=health,status,index,pri,rep,docs.count,store.size </pre>This will return a result similar to the following, with the index status being ‘open’:<pre class="prettyprint">health status index pri rep docs.count store.size green open sampledata 5 1 2 17.8kb </pre>As mentioned above, we must protect the cluster from running out of memory, thus there is a limit in the number of frozen indices we can concurrently load for search on a node. The number of threads in the search-throttled threadpool defaults to 1, with a default queue of 100. This means that if we run more than one request, they will be queued up to a hundred. We can monitor the&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/cat-thread-pool.html#cat-thread-pool">threadpool status</a>, to check queues and rejections, with the following request:<pre class="prettyprint">GET _cat/thread_pool/search_throttled?v&amp;h=node_name,name,active,rejected,queue,completed&amp;s=node_name </pre>Which should return a response similar to:<pre class="prettyprint">node_name name active rejected queue completed instance-0000000000 search_throttled 0 0 0 25 instance-0000000001 search_throttled 0 0 0 22 instance-0000000002 search_throttled 0 0 0 0 </pre>Frozen indices might be slower, but they can be pre-filtered in a very efficient manner. It is also recommended to set the request parameter pre_filter_shard_size to 1.<pre class="prettyprint">GET /sampledata/_search?ignore_throttled=false&amp;pre_filter_shard_size=1 { "query": { "match": { "name": "jane" } } } </pre>This will not add a significant overhead to the query and will allow us to take advantage of the usual scenario. For example, when searching on a date range on time-series indices, not all shards will match.<h2>How to write to a frozen Elasticsearch index</h2>What will happen if we try to write on an already frozen index? Let’s go for it and find out.<pre class="prettyprint">POST /sampledata/_doc { "name": "Janie", "lastname": "Doe" } </pre>What happened? Frozen indices are read-only, so writing is blocked. We can check these in the index settings:<pre class="prettyprint">GET /sampledata/_settings?flat_settings=true </pre>Which will return:<pre class="prettyprint">{ "sampledata" : { "settings" : { "index.blocks.write" : "true", "index.frozen" : "true", .... } } } </pre>We have to use the&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/unfreeze-index-api.html">Unfreeze index API</a>, invoking the unfreeze endpoint on the index.<pre class="prettyprint">POST /sampledata/_unfreeze </pre>And now we'll be able to create a third document and search for it.<pre class="prettyprint">POST /sampledata/_doc { "name": "Janie", "lastname": "Doe" } GET /sampledata/_search { "query": { "match": { "name": "janie" } } } </pre>Unfreezing should be done only under exceptional situations. And remember to always run a `force_merge` before freezing the index again to ensure optimal performance.<h2>Using frozen indices in Kibana</h2>To begin with, we will need to load some sample data, like the <a href="https://www.elastic.co/guide/en/kibana/6.6/tutorial-sample-data.html">sample flight data</a>.Click on the “Add” button for Sample flight data.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt142abc81bf2e4ac0/5c98d104632578d5055bb49f/kibana-load-data.png" alt="kibana-load-data.png" width="700" style="margin: auto;display: block;" data-sys-asset-uid="blt142abc81bf2e4ac0"/>We should now be able to see the loaded data by clicking the “View data” button. The dashboard will be similar to this one.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt70ea061b7b398da9/5c98d0ff026179ca592ee205/flight-dashboard.png" width="100%" data-sys-asset-uid="blt70ea061b7b398da9" alt="flight-dashboard.png" style="margin: auto;display: block;"/>Now we can test freezing the index:<pre class="prettyprint">POST /kibana_sample_data_flights/_forcemerge?max_num_segments=1 POST /kibana_sample_data_flights/_freeze </pre>And if we go back to our dashboard, we’ll notice that the data has apparently “disappeared”.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0a68db6e72e372e7/5c98d0f9acf2956c56cb1953/empty-flight-dashboard.png" data-sys-asset-uid="blt0a68db6e72e372e7" alt="empty-flight-dashboard.png" width="100%" style="margin: auto;display: block;"/>We need to tell Kibana to allow searches on frozen indices, which is disabled by default.Go to Kibana Management, choose Advanced Settings. In the Search section, you will find that “Search in frozen indices” is disabled. Toggle to enable and save the changes.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf5f1d58c21e9ba65/5c98d0f249a201165fca1028/frozen-indices-kibana-settings.png" data-sys-asset-uid="bltf5f1d58c21e9ba65" alt="frozen-indices-kibana-settings.png" width="100%" style="margin: auto;display: block;"/>And the flight's dashboard will show the data again.<h2>Wrapping up</h2>Frozen Indices are a very powerful tool in hot-warm architectures. They enable a more cost-effective solution for increased retention while retaining online search. I recommend that you test your search latency with your hardware and data, to come with the right sizing and search latency for your frozen indices.Check out the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/frozen-indices.html">Elasticsearch documentation</a> to learn more about the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/freeze-index-api.html">Freeze index API</a>. And as always, if you have any questions, reach out on our <a href="https://discuss.elastic.co/">Discuss</a> forums. Happy freezing!

frozen-indices-freeze-index-api.jpg

frozen-indices-freeze-index-api-small.jpg

Creating frozen indices with the Elasticsearch Freeze index API

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Blog Footer

headshot-imma-valls.jpg

Senior Support Engineer

Imma Valls

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Load More

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

Author

Articles by Imma Valls

Senior Support Engineer, Elastic

Creating frozen indices with the Elasticsearch Freeze index API

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS