Cluster management strategies such as adjusting availability zones and snapshot management can help you manage costs and ensure that your deployment configuration suits your use case.

Welcome to another blog in our series on cost management and optimisation in Elasticsearch Service. In previous installments, we looked at <a href="/blog/optimizing-costs-elastic-cloud-hot-warm-index-lifecycle-management">hot-warm architecture and index lifecycle management</a> as ways of managing the costs associated with data retention and at <a href="/blog/optimizing-costs-elastic-cloud-replica-shard-management">managing replicas</a> as a means of optimising the structure of your Elasticsearch Service deployment. Be sure to check out the other blogs in the series for additional tips to help you as you build out your deployment. And if you haven’t already, sign up to a <a href="/cloud/elasticsearch-service/signup">14-day free trial of Elasticsearch Service</a> on Elastic Cloud so you can follow along.In this blog, we’ll look at cluster management strategies including adjusting availability zones and snapshot management.Optimising the size and configuration of your deployment is crucial to ensuring your deployment suits your use case. For snapshot management in particular, being able to increase or decrease the frequency of snapshots depending on your use case can help you manage costs associated with snapshot storage and API requests. Furthermore, if you need to temporarily pause your deployment, knowing how to snapshot to an external repository is crucial.All of these strategies can help you manage costs in Elastic Cloud. Balancing your architectural considerations between cost and resiliency is key to deploying these strategies. It is imperative to consider where and how they should be used to minimise the impact of any potential disruption.Before we jump into snapshots, let’s take a brief look at adjusting availability zones for existing and new deployments.<h2>Adjusting availability zones</h2>Data centers can and do fail, so planning is crucial. In Elastic Cloud, you can always adjust the number of availability zones for your cluster, whether during a deployment or at the start of a deployment. When <a href="https://www.elastic.co/guide/en/cloud/current/ec-planning.html#ec-ha">planning for production</a> it is important to consider the number of availability zones you want. You can cut costs by reducing the number of availability zones, but you’ll also increase your risk of data loss. The best practice is to have at least two data centers for production and three for mission-critical systems. Single-zone architectures are suitable for development and test environments, but they are not recommended for production environments.If you already have a deployment on Elastic Cloud, you can adjust the number of availability zones for all nodes (hot, warm, Kibana, machine learning, APM) by selecting the deployment from the <a href="https://cloud.elastic.co/home">Cloud Console</a> and clicking Edit in the menu on the left.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb77e303c0b135b76/5f2857e47f3b7908db030180/blog-cloud-costs-zones-edit.png" data-sys-asset-uid="bltb77e303c0b135b76" alt="" style="display: block;margin: auto;"/> For brand new deployments, you can configure the number of availability zones at the bottom of the Create deployment screen by clicking Customize deployment at the bottom of the page. This will take you to a slider view similar to the image above where all nodes and availability zones can be adjusted as you wish.<h2>Managing snapshots</h2><a href="https://www.elastic.co/guide/en/cloud/current/ec-snapshot-restore.html">Snapshots</a> are important for backing up data stored in <a href="/cloud/">Elastic Cloud</a>. In addition to recovery from data center failure, snapshots can be used to bring up a new cluster for testing or recreate a deleted index.For Elastic Stack versions 7.6 and later, you can use the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/snapshot-lifecycle-management.html">snapshot lifecycle management (SLM)</a> functionality in <a href="https://www.elastic.co/guide/en/kibana/7.7/snapshot-repositories.html">Kibana</a>. For older versions, you’ll need to use the <a href="https://www.elastic.co/guide/en/cloud/current/ec-restoring-snapshots.html">Elastic Cloud UI</a>.The larger the volume of data stored in your deployment and the higher the frequency of snapshots, the more likely you are to incur costs related to snapshot storage. Regardless of your use case, you might find it beneficial to review our previous blogs on <a href="/blog/optimizing-costs-elastic-cloud-hot-warm-index-lifecycle-management">hot-warm and ILM</a> and <a href="/blog/optimizing-costs-elastic-cloud-replica-shard-management">replica management</a> while considering the following section.&nbsp;If you’d like more information about costs as you look to adopt the best practices in this blog post, also check out our blog on data transfer and snapshot storage pricing.<h2>Best practices for snapshotting</h2>Each deployment of the Elastic Stack is unique and has different requirements, so not all of the following recommendations may apply to you. However, the default settings are usually sufficient for most users. The settings can also be adjusted to fit your individual requirements.The configuration options that can be adjusted are<ul><li>frequency by the schedule</li><li>content by the indices included</li><li>retention by age and count</li></ul>If the data set is relatively static, such as web pages about an organisation, snapshots can be <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/trigger-schedule.html#schedule-cron">scheduled</a> to occur less frequently, such as once a day or every four hours. Even for time series data, such as logs, the frequency can be reduced as well. The logs may be available locally on application servers long enough to allow the logs to be ingested again to fill the gap between the last snapshot and now, so no data would be lost in the event of a failure.Another option is to reduce the number of snapshots kept. With fewer snapshots kept, the storage space used is smaller, especially if the data changes frequently within the Elasticsearch cluster. If the snapshots are aged off more quickly and a number of indices are no longer contained in any snapshot, the disk space used by those indices will be released. If there is adequate monitoring and alerting in place to detect data problems quickly, it may not be necessary to restore more than a day or two back, allowing more of the older snapshots to be deleted.Only a subset of the indices in a cluster may need to be backed up. For example, for observability, it’s common to snapshot very little or no metric data related to system health monitoring and retain the snapshots for a short period of time. The value of this data set significantly decreases as it gets older, so the cost of retaining it is often higher than the value it provides. Another example is to snapshot only the system indices storing Kibana dashboards in a development or non-production deployment but not the underlying business data, since the business data can be restored from another cluster such as the production cluster.Finally, a combination of the three configuration options can be used in multiple <a href="https://www.elastic.co/guide/en/kibana/7.7/snapshot-repositories.html#kib-snapshot-policy">SLM policies</a> to snapshot different sets of indices at their own rate that are kept for a period of time. For example, a deployment used for order analytics may group indices into three different policies. Indices for small data sets that have a source of truth elsewhere and can be reloaded quickly may only be snapshotted once a week and retained for two weeks. A second group containing data that is not stored elsewhere, such as Kibana dashboards, saved queries, and machine learning models, is snapshotted every 30 minutes. And a third group of indices, including data that is also in an official system of record but can’t easily be ingested again due to the size of the data set, is snapshotted a few times a day.The decision of how to best use the three control options requires careful analysis of acceptable risk and costs.<h3>Snapshotting to a custom repository</h3>Another choice that can be made is where the Elasticsearch snapshots are stored. With Elasticsearch Service, there is the option to use a <a href="https://www.elastic.co/guide/en/cloud/current/ec-custom-repository.html">custom repository</a>. Depending on your existing cloud provider agreements, it may be more cost effective to use a repository from an existing account. The custom repository needs to be located in the same cloud provider and region as the Elasticsearch cluster to reduce data transfer costs between the cluster and the repository. A custom repository does not get around the limited set of Elasticsearch <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/snapshot-restore.html#_version_compatibility">major versions</a> that each Elasticsearch version supports. It does, however, support longer-term storage of data that is common for security analytics and compliance use cases.When using a custom repository, the maintenance and monitoring is the responsibility of the user. Although the default Elastic-managed snapshot repository can be used in conjunction with a custom repository, you’ll need to ensure the two do not conflict with each other, as only <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/snapshots-monitor-snapshot-restore.html#_stop_snapshot_and_restore_operations">one snapshot</a> can be run at a time. Again, there are benefits, risks, and costs involved with where the snapshots are stored that will need to be analyzed for each deployment individually.<h3>Review snapshots on Elastic Cloud</h3>Here’s how you can review your snapshots:<ol><li>Open your Elasticsearch Service deployment<ol><li>Open the <a href="https://cloud.elastic.co/home">Elastic Cloud UI</a></li><li>Click on Elasticsearch Service to view your <a href="https://cloud.elastic.co/deployments">deployments</a></li><li>Click on the tile for the deployment</li></ol></li><li>On the left menu, click on Snapshots located under Deployments → {deployment name} → Elasticsearch<a href="https://github.com/elastic/consulting/blob/master/projects/blogs/images/cloud_7.7_deployment_overview.png"></a></li><li>The display will differ slightly depending on the version of the Elasticsearch cluster. Somewhere around the middle of the top half of the page is a section with the status of the snapshots.<a href="https://github.com/elastic/consulting/blob/master/projects/blogs/images/cloud_7.7_snapshots_status.png"></a></li></ol><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltffcdba4d3daf7a1f/5f2858e6d9a24d7c67e4e2a9/blog-cloud-costs-zones-snapshots.png" data-sys-asset-uid="bltffcdba4d3daf7a1f" alt="" style="display: block;margin: auto;"/>The two key pieces of information here are the time of the last successful snapshot and the scheduled time for the next snapshot. The last successful snapshot time tells you whether the snapshots are working and how much data could be lost on failure. The next snapshot time indicates when the process is scheduled to run again to create a new snapshot.An active and successful snapshot process is important for protecting the data stored in an Elasticsearch cluster. You’ve indexed your data into Elasticsearch because it matters, so it’s critical to ensure that snapshots are properly set up and functioning.<h3>Snapshot lifecycle management (SLM)</h3>If your deployment is version 7.6 or later, you’ll manage your snapshots using <a href="https://www.elastic.co/guide/en/kibana/7.7/snapshot-repositories.html">Kibana</a>.To modify the frequency and retention of snapshots, click Snapshot and Restore, which will open Kibana in a new window. Kibana will open to the Snapshot and Restore page in <a href="https://www.elastic.co/guide/en/kibana/7.7/management.html">Management</a>. Click on Policies.Locate the cloud-snapshot-policy row and click on the pencil icon on the right to edit the policy.This will open a new page that will guide you through <a href="https://www.elastic.co/guide/en/kibana/7.7/snapshot-restore-tutorial.html#create-policy-example">a multi-step process</a> for adjusting the frequency at which snapshots are taken and the retention period of the snapshots. Both of these factors have an impact on the costs associated with snapshot storage. It is also possible to apply the settings to different indices, which gives you a finer level of granularity when it comes to managing these settings for an individual index or groups of indices.&nbsp;The other option for modifying the frequency of snapshots from Elastic Cloud is to click Modify frequency, which will directly open the Policies tab of the Snapshot and Restore page.<a href="https://github.com/elastic/consulting/blob/master/projects/blogs/images/cloud_7.7_snapshots_modify-frequency.png"></a>If you need to restore data from a snapshot, you’ll do this on the Snapshots tab. Click the restore icon to the right of the snapshot name.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc2a4522a17a061f0/5f28597f7f3b7908db030186/blog-cloud-costs-zones-restore.png" data-sys-asset-uid="bltc2a4522a17a061f0" alt="" style="display: block;margin: auto;"/>This will open a new page that will guide you through <a href="https://www.elastic.co/guide/en/kibana/7.7/snapshot-restore-tutorial.html#restore-snapshot-example">a multi-step process</a> for restoring from a snapshot. If only a subset of indices needs to be restored, click the slider under the Indices section to select a subset of the indices from a snapshot.<a href="https://github.com/elastic/consulting/blob/master/projects/blogs/images/kibana_7.7_snapshot_restore-indices.png"></a><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc2b93ed288c1d1b2/5f2859c5d9a24d7c67e4e2ad/blog-cloud-costs-zones-subset.png" data-sys-asset-uid="bltc2b93ed288c1d1b2" alt="" style="display: block;margin: auto;"/><h3>Elastic Cloud UI</h3>If your deployment is version 7.5 or older, you’ll manage your snapshots using the <a href="https://www.elastic.co/guide/en/cloud/current/ec-restoring-snapshots.html">Elasticsearch Service UI</a>.To modify the frequency and retention of the snapshots, click Edit settings in the center of the page.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt591e8fefb6927b27/5f285a209af97004e7f61c44/blog-cloud-costs-zones-edit-settings.png" data-sys-asset-uid="blt591e8fefb6927b27" alt="" style="display: block;margin: auto;"/>A pop-up window will appear where you can view and modify the current snapshot interval and snapshot count to retain.<a href="https://github.com/elastic/consulting/blob/master/projects/blogs/images/cloud_6.8_snapshots_settings.png"></a>If you need to restore data from a snapshot, you can do so by clicking Restore to the right of the relevant snapshot name.On the fly-out window that appears on the right, there is an option to restore only a subset of indices from the snapshot.The other option for restoring from a snapshot is to click on the snapshot name, which will open a new page that lists the indices within the snapshot and presents the same form at the bottom of the page.<h2>Next steps</h2>There are a number of ways to manage and optimise your data within Elastic Cloud. An optimal setup will ensure that the costs associated with your deployment remain as low as possible. Be sure to check out the previous parts in this blog series for more on how <a href="/blog/optimizing-costs-elastic-cloud-hot-warm-index-lifecycle-management">hot-warm and ILM</a> and <a href="/blog/optimizing-costs-elastic-cloud-replica-shard-management">replica management</a> can help you optimise costs, as well as our previous blog on cost-saving strategies <a href="/blog/cost-saving-strategies-for-the-elasticsearch-service-data-storage-efficiency">in Elastic Cloud</a>.If you’re interested in more advanced management techniques, we offer a variety of <a href="/training/">Elastic Training</a> courses, including <a href="/training/elasticsearch-engineer-1">Elasticsearch Engineer I</a>, which provides a strong foundation for getting started with Elasticsearch, and <a href="/training/elasticsearch-engineer-2">Elasticsearch Engineer II</a>, which digs into more advanced cluster management techniques and best practices for capacity planning and scaling. We also encourage you to check out our <a href="https://discuss.elastic.co/">discussion forums</a>, where you can connect with Elastic users from around the world.

blog-banner-cloud-midnight.png

blog-thumb-cloud-midnight.png

Optimizing costs in Elastic Cloud: Availability zones and snapshot management

Managing replicas in Elastic Cloud can help you optimise the structure of your data in Elasticsearch and reduce costs.

This is part of our series on <a href="/blog/optimizing-costs-elastic-cloud-hot-warm-index-lifecycle-management">cost management and optimisation</a> in Elasticsearch Service. If you’re new to the cloud, be sure to think about these topics as you build out your deployment. If you are yet to start, you can test out the content here by signing up to a <a href="/cloud/elasticsearch-service/signup">14-day free trial of Elasticsearch Service</a> on Elastic Cloud.
<a href="/blog/optimizing-costs-elastic-cloud-hot-warm-index-lifecycle-management">In a previous blog post</a>, we looked at <a href="/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management">hot-warm architecture</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/index-lifecycle-management.html">index lifecycle management (ILM)</a> as ways of managing the costs associated with data retention. In this blog, we’ll demonstrate how managing replicas in Elastic Cloud can help you optimise the structure of your data in Elasticsearch and reduce costs. We’ll cover how to configure replicas for your use case, such as by reducing replicas in non-production environments to save on storage and reducing replicas in time series data as the need for high availability decreases. All of the features discussed in this series are free and open in the Elastic Stack and available in Elastic Cloud.
<h2>Managing replica shards</h2>In Elasticsearch, data is stored in indices as&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/glossary.html#glossary-document">JSON documents</a>. An <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/glossary.html#glossary-index">index</a> is composed of<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/glossary.html"> shards</a>, which are groups of documents that are allocated to&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/glossary.html#glossary-node">nodes</a>&nbsp;(running instances of the Elasticsearch process)&nbsp;belonging to a&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/glossary.html#glossary-cluster">cluster</a>. Each document is stored in a single primary shard, and each primary shard can have zero or more replicas, or copies of the primary shard. When a document is indexed into Elasticsearch, it is<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/docs-replication.html"> first stored to the primary shard and is then sent to the replica shards</a>. During the indexing operation, the<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/index-modules-translog.html"> transaction log</a> is written to before the write operation is acknowledged back to the client. This ensures that document changes are not lost in the event of a power failure.
Replica shards are important to the<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/high-availability-cluster-design.html"> resiliency</a> of the data stored in Elasticsearch. Part of the shard allocation logic in Elasticsearch includes ensuring that replica shards are not on the same node as the primary shard. To maintain high availability, if a node leaves a cluster — for example, during a network failure — one of the replica shards is promoted to be the primary shard if the prior primary shard is unavailable. The promotion to primary cannot be delayed, but <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/delayed-allocation.html">the re-assignment of replicas can be if necessary</a>. Elasticsearch Service ensures an additional level of resiliency by ensuring replica shards are<a href="https://www.elastic.co/guide/en/cloud/current/ec-configure-deployment-settings.html#ec-high-availability"> spread across the availability zones</a> using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/modules-cluster.html#shard-allocation-awareness">shard allocation awareness</a>.
The number of replica shards is controlled at the index level. <code>index.number_of_replicas</code> is a<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/index-modules.html#dynamic-index-settings"> dynamic setting</a>, meaning it can be set when an index is<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/indices-create-index.html#create-index-settings"> created</a> and later<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/indices-update-settings.html"> updated</a> if needed. Additionally, there is the option to set<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/index-modules.html#dynamic-index-settings"> <code>index.auto_expand_replicas</code></a> to an integer range, which allows Elasticsearch to allocate additional replica shards as qualifying Elasticsearch nodes become available.
For example, this would be the process to set the number of replicas during index creation and then update the number on a <code>testing-v1</code> index using&nbsp;<a href="https://www.elastic.co/guide/en/kibana/7.7/console-kibana.html">Console</a>:
<pre class="prettyprint">PUT testing-v1
{
 "settings": {
 "index": {
 "number_of_shards": 1,
 "number_of_replicas": 1
 }
 }
}
PUT testing-v1/_settings
{
 "index": {
 "number_of_replicas": 2
 }
}
</pre>This setting can also be edited using <a href="https://www.elastic.co/guide/en/kibana/7.7/management.html">Management</a> in Kibana 7.4 and higher. Navigate to Stack Management → Elasticsearch → <a href="https://www.elastic.co/guide/en/kibana/7.7/managing-indices.html">Index Management</a> and click on the index name under the Name column. This will bring up<a href="https://www.elastic.co/guide/en/kibana/7.7/managing-indices.html#_view_and_edit_indices"> a pop-up window</a> on the right side of the page. Then click on the Edit settings tab. Finally, update the <code>index.number_of_replicas</code> value in the settings entry field and save the change.<a href="https://github.com/elastic/consulting/blob/master/projects/blogs/images/kibana_7.7_index-management_indices_settings.png"> </a>
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbb3cb3c9e0375c21/5f1f097a208c6a29342524dc/blog-cloud-costs-shard-settings.png" data-sys-asset-uid="bltbb3cb3c9e0375c21" alt="blog-cloud-costs-shard-settings.png" "="">
<h2>Using replica shards</h2>Now that we have covered the what, why, and how of replica shards, we will get into the details of how to utilise replicas to our advantage. A search request can be handled by any of the copies of a shard, including both primaries and replicas. If you have a high volume of searches against a data set, especially a small data set such as store locations on a company website, more than one replica per shard may be of use. This is because additional replicas typically — but not always — increase<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/tune-for-search-speed.html#_replicas_might_help_with_throughput_but_not_always"> search speed</a>. The cost of additional replica shards is the additional disk space, memory, and CPU cycles used by each copy.
It is worth noting that the default value for replica shards in Elasticsearch is one. This represents a compromise between indexing speed, search speed, and compute resources. A minimum of one replica is required for high availability because at least one replica shard is needed to fail over to if a node goes offline.
If the Elasticsearch cluster is used for non-production purposes, such as a development deployment, the number of replicas can be set to zero to save resources. This configuration does not provide any data resiliency but is cheaper to run. Running a non-production cluster without replicas is a good option if lost data can be either reloaded or restored from another cluster or from a snapshot, as recovery time is ample compared to a production environment.
<h2>Automation</h2>The ability to change the number of replica shards is nice, but it is even better to be able to automate the process. In Kibana there are two functionalities to assist with this: <a href="https://www.elastic.co/guide/en/kibana/7.7/managing-indices.html#manage-index-templates">index templates</a> and<a href="https://www.elastic.co/guide/en/kibana/7.7/example-using-index-lifecycle-policy.html"> ILM policies</a>. Index templates allow for a collection of index settings, mappings, and aliases to be applied automatically when an index is created if the index name matches the index pattern. Two of the index settings that may be included are the number of replicas and auto-expand replicas. An ILM policy defines actions that should be taken based on the phase of a given index. One of the actions that can be taken is to<a href="https://www.elastic.co/guide/en/kibana/7.7/example-using-index-lifecycle-policy.html#_modify_the_policy"> change the number of replicas</a> in the warm phase in Elasticsearch Service.
Under the hood, when data moves into the warm phase, indices can no longer be written to. This gives us another opportunity to save on costs, as we can choose not to store any replica data in our warm nodes. In the event of a warm node failure, we would restore from the latest snapshot rather than from replicas.
For example, if we needed to create indices containing time series data that would be written to on day 1 and heavily searched on the next day, a combination of<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/set-up-lifecycle-policy.html"> a template and ILM policies</a> could be used to satisfy this requirement. <a href="/blog/optimizing-costs-elastic-cloud-hot-warm-index-lifecycle-management">In our previous cost optimisation blog</a>, we covered the basics of creating an ILM policy in Kibana. Here are the steps we’d take for this example:
<ol>
	<li>Open Kibana → Management → Elasticsearch → Index Lifecycle Policies.</li>
	<li>Click Create policy. <a href="https://github.com/elastic/consulting/blob/master/projects/blogs/images/kibana_7.7_ilm_create-policy.png"></a></li>
	<li>Fill in the entry form fields as follows and click Save as new policy.</li>
</ol><table style="width:100%">
<tbody>
<tr style="background-color:#E8F6FE;">
	<td style="text-align: center; font-weight:700;">Phase
	</td>
	<td style="text-align: center; font-weight:700;">Field
	</td>
	<td style="text-align: center; font-weight:700;">Value
	</td>
</tr>
<tr>
	<td>--
	</td>
	<td>Policy name
	</td>
	<td>time-series-rollover
	</td>
</tr>
<tr style="background-color:#F6F7FA;">
	<td>Hot
	</td>
	<td>Maximum age
	</td>
	<td>1 day
	</td>
</tr>
<tr>
	<td>Warm
	</td>
	<td>Number of replicas
	</td>
	<td>2
	</td>
</tr>
</tbody>
</table><ol start="4">
	<li>Open Index Management. <a href="https://github.com/elastic/consulting/blob/master/projects/blogs/images/kibana_7.7_management_index.png"></a></li>
	<li>Open the Index Templates tab.<a href="https://github.com/elastic/consulting/blob/master/projects/blogs/images/kibana_7.7_index-management_index-templates.png"> </a></li>
	<li>Click Create a template.<a href="https://github.com/elastic/consulting/blob/master/projects/blogs/images/kibana_7.7_index-management_create-template.png"></a></li>
	<li>Fill in the multiple-page entry form as follows and click Create template.</li>
</ol><table style="width:100%">
<tbody>
<tr style="background-color:#E8F6FE;">
	<td style="text-align: center; font-weight:700;">Step
	</td>
	<td style="text-align: center; font-weight:700;">Entry
	</td>
	<td style="text-align: center; font-weight:700;">Value
	</td>
</tr>
<tr>
	<td>1
	</td>
	<td>Name
	</td>
	<td>time-series
	</td>
</tr>
<tr style="background-color:#F6F7FA;">
	<td>1
	</td>
	<td>Index patterns
	</td>
	<td>time-series-*
	</td>
</tr>
<tr>
	<td>1
	</td>
	<td>Version
	</td>
	<td>20200616
	</td>
</tr>
<tr>
	<td>2
	</td>
	<td>Index settings
	</td>
	<td>
		<pre class="prettyprint">{
 "number_of_shards":1,
 "number_of_replicas":1,
 "index.lifecycle.name": "time-series-rollover", 
 "index.lifecycle.rollover_alias": "time-series-alias" 
}
 </pre>
	</td>
</tr>
</tbody>
</table><ol start="8">
	<li>Create the<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.7/set-up-lifecycle-policy.html#create-initial-index"> initial index</a> using<a href="https://www.elastic.co/guide/en/kibana/7.7/console-kibana.html"> Console.</a></li>
</ol><pre class="prettyprint">#request
PUT time-series-000001
{
 "aliases": {
 "time-series-alias": {
 "is_write_index": true
 }
 }
} 
#response
{
 "acknowledged" : true,
 "shards_acknowledged" : true,
 "index" : "time-series-000001"
}
</pre><ol start="9">
	<li>Start indexing documents using <code>time-series-alias</code> as the index name.</li>
</ol><h2>Wrapping up (for now)</h2>	As we’ve seen, managing replica shards can deliver cost savings by allowing you to optimise the structure of your data in Elasticsearch. Elastic Cloud makes this easy by allowing you to size your cluster and automate the process of changing the number of replica shards.
 In an upcoming post, we'll take a look at <a href="/blog/optimizing-costs-elastic-cloud-availability-zones-snapshot-management">availability zones and snapshot management</a>, so stay tuned for more tips and tricks on reducing costs while delivering great features to your end users and a simple control experience to your admins. And remember that you can try out everything we’ve covered here with a <a href="/elasticsearch/service">free 14-day trial of Elasticsearch Service on Elastic Cloud</a>.

Optimizing costs in Elastic Cloud: Replica shard management

Managing data using hot-warm architecture and ILM is a cost effective way of retaining data and great way to easily keep your cloud costs down.

Welcome to our series on cost management and optimisation in Elasticsearch Service. With the increased functionality in <a href="https://www.elastic.co/cloud">Elastic Cloud</a>, it is now easier than ever to utilise many of the free and open features of the Elastic Stack to optimise your cloud deployment.&nbsp;
This blog is a great resource for reviewing your existing high availability and data management strategies when it comes to cost management. If you’re new to the cloud be sure to think about these topics as you build out your deployment. If you are yet to start, you can test out the content here by signing up to a <a href="https://www.elastic.co/cloud/elasticsearch-service/signup">14-day free trial of Elasticsearch Service</a> on Elastic Cloud.
This post will focus on assisting you in managing your data in Elastic Cloud through cost-efficient storage tiers using <a href="https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management">hot-warm architecture</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/index-lifecycle-management.html">index lifecycle management</a>.
In upcoming posts, we’ll tackle topics like <a href="/blog/optimizing-costs-elastic-cloud-replica-shard-management">managing data volume and resiliency with replica management</a>, <a href="/blog/optimizing-costs-elastic-cloud-availability-zones-snapshot-management">availability zone adjustments, and snapshot management practices</a>.
For many of the examples in this blog series, we will show you how to manage data using Kibana. Utilising Kibana is a great way to manage your deployments given the increased functionality in the Elastic Stack. If you haven’t had the opportunity to explore them we’d highly recommend you check them out!
Without further ado, let’s get to it.
<h2>Hot-warm architecture and index lifecycle management</h2>In a previous <a href="https://www.elastic.co/blog/cost-saving-strategies-for-the-elasticsearch-service-data-storage-efficiency">data storage efficiency</a> blog post, we looked at the cost savings associated with adopting hot-warm architecture on Elasticsearch Service. The blog also contains guidance on utilising <a href="https://www.elastic.co/blog/data-rollups-in-elasticsearch-you-know-for-saving-space">data rollups</a>, which is another data optimisation strategy you may wish to use. In general, we recommend reading as much as you can about different strategies before making changes in production. If you are a customer, please&nbsp;do not hesitate to get in touch with your account team before making any updates. And as always, anyone can visit our <a href="https://discuss.elastic.co">forums</a> for any questions they have.&nbsp;
For hot-warm architectures, there are points worth covering before we talk about index lifecycle management (ILM).
The hot-warm architecture on Elasticsearch Service helps you manage your data to control costs while still retaining meaningful data for longer periods of time — and providing the same useful capabilities of the Elastic Stack: consolidation of data from various sources, visualisation options, alerting, anomaly detection, and more. Depending on how you configure your hot-warm cluster with ILM policies, you can save considerably on costs. Check out the details in the data storage efficiency blog linked above to see how you can save up to 60%.
<h2>Moving from I/O-optimised to hot-warm</h2>In Elasticsearch Service, if you already have an existing I/O-optimised deployment, you can simply migrate to a <a href="https://www.elastic.co/guide/en/cloud/current/ec-getting-started-templates-hot-warm.html">hot-warm deployment</a> by adding warm nodes to your cluster. If you are creating a new deployment, you can create a hot-warm deployment and optionally restore a snapshot from another cluster.
Once you have created or migrated to a hot-warm deployment on Elasticsearch Service it is now possible to set up your ILM policies and apply them to the relevant indices using the lifecycle policy UI in Kibana. Here’s a step-by-step guide on how to do so.
<h2>Creating an index lifecycle policy from Kibana</h2>An index lifecycle policy enables you to define rules for when to perform certain actions, such as a rollover or force merge on an index. Index lifecycle management automates execution of those actions at a specified time. These are the three different phases of an index lifecycle in Elasticsearch Service:
<ul>
	<li>Hot phase: Your data initially lands in this phase and is stored on high-performance SSDs specifically chosen to give speedy results for high indexing and query rates.</li>
	<li>Warm phase: Data that is infrequently queried, such as older data, gets moved to the warm phase where the data resides on denser, more cost-effective storage to help you save on costs.</li>
	<li>Delete phase: The index is no longer needed and can safely be removed.</li>
</ul>Now let’s take an example of creating an ILM policy with 60 days retention (14 days hot, 46 days warm).
To update or create an ILM policy from Kibana, go to Management → Elasticsearch → Index Lifecycle Policies → Create policy. If you are using a 7.0+ Beat, then one or more default policies will already be there, and can just be edited. Otherwise you'll need to fill in the following:
<h3>Name</h3>Provide your policy name. This policy name will be used in your index template, so any index matching the index pattern configured in the index template will have that ILM policy applied to it.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt705e0b867ce85ad0/5f1205528b0e59084c9f9099/blog-cloud-costs-ilm-name.png" data-sys-asset-uid="blt705e0b867ce85ad0" alt="Configure your ILM policy name" style="display: block; margin: auto;">
<h3>Hot phase</h3>The hot phase is enabled and active by default for new indices.&nbsp;Those indices will stay on the hot phase unless it’s configured to move to one of index lifecycle phases. While the current phase for the new indices is “hot phase”, rollover can still be enabled to avoid having large shard size at specified conditions. These are the three configurable conditions that ILM will check and will rollover indices whichever condition matches first:
<ul>
	<li>Maximum index size (e.g. GB)</li>
	<li>Maximum # of documents</li>
	<li>Maximum age (e.g. 7 days)</li>
</ul>To meet our policy requirement of 14 days hot, we will enable rollover whenever <code>Maximum_index_size</code> exceeds 50GB (following shard size best practices to not exceed 30-50GB) or <code>Maximum_age</code> passes 2 days, whichever comes first. With that configured, indices will stay in the hot phase until the warm phase is activated and configured, which we’ll cover next.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb2b16f604f7cb8c6/5f12056b600d867e81bcbba9/blog-cloud-costs-ilm-hot.png" data-sys-asset-uid="bltb2b16f604f7cb8c6" alt="Configure your ILM hot phase" style="display: block; margin: auto;">
<h3>Warm phase</h3>While activating warm phase, you can either choose to enable “Move to warm phase on rollover” or specify a “timing for warm phase”. In our example, we will configure timing for warm phase instead of enabling “Move to warm phase on rollover” since we want to keep our data in hot phase for 14 days. To meet our policy requirement of 14 days hot and 46 days warm, we will configure timing for warm phase at 12 days from rollover +2 days <code>Maximum_age</code> from hot phase.
You can also select a node attribute to control shard allocation. In Elasticsearch Service, if the Hot-warm deployment template is used while provisioning your cluster, then your data nodes will automatically be tagged with hot-warm attributes
<ul>
	<li>data:hot represents data nodes tagged with hot attribute</li>
	<li>data:warm represents data nodes tagged with warm attribute</li>
</ul>Both Shrink and Force Merge can be enabled in the warm phase for faster searches since indices are no longer being written to.
<ul>
	<li>Shrink index can be enabled which will run the <code>shrink</code> API to shrink an existing index into a new index with fewer primary shards, which will in turn decrease the volume of data and the cost associated with it.</li>
	<li>Force merge can also be enabled which will run the <code>force merge</code> API to reduce the number of segments in each shard by merging some of them together, and also frees up the space used by deleted documents.</li>
</ul><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt72fb1a478665c363/5f1205825bbd1b7f3aca8779/blog-cloud-costs-ilm-warm.png" data-sys-asset-uid="blt72fb1a478665c363" alt="Configure your ILM warm phase" style="display: block; margin: auto;">
<h3>Delete phase</h3>Lastly, delete phase can be activated for indices that are no longer needed. In our example, the ILM policy is to keep data for no more than 60 days, so we will activate the delete phase and configure timing for delete phase to 58 days from rollover, plus the 2 days in hot phase which adds up to 60 days retention time.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd478c80dd9a72767/5f1205930d1d2008538cfb28/blog-cloud-costs-ilm-delete.png" data-sys-asset-uid="bltd478c80dd9a72767" alt="Configure your ILM delete phase" style="display: block; margin: auto;">
<h2>Create an index template and bootstrap first index</h2>We have created a "60_days_retention_policy" ILM policy, but we have not yet assigned it to an index. To assign the ILM policy to an index on creation/rollover, we need to add it to an index template. Templates get applied to new indices if the new index being created matches the index pattern.This template will make sure that any new index that starts with the name "events" will be created with one primary shard and the policy we created above will be attached.To create an index template from Kibana:<ol>
<li>Go to Management &gt; Elasticsearch &gt; Index Management &gt; Index Templates &gt; Create a template.</li>
<li>Define the index template name and index patterns.</li>
<li>Add the number of primary shards and the ILM policy created earlier and define your rollover alias.</li>
<li>For this example, we will leave the default settings in the Mappings and Aliases sections.</li>
</ol>The next thing we need to do is to bootstrap the first index. Run the following command from Dev Tools to create an "events" alias and point it to the events-000001 index. The alias name should match the <code>index.lifecycle.rollover_alias</code> in the template.
<pre class="prettyprint">PUT events-000001
{
 "aliases": {
 "events": {
 "is_write_index": true
 }
 }
}
</pre>After creating the first index, verify that index has the index lifecycle policy and rollover alias attached to it. Go to Management &gt; Elasticsearch &gt; Index Management &gt; Indices.

<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte3afb8b0dbb52ed4/5f4d26370ab106265a18741a/blog-cost-optimization-hot-warm-bootstrap.png" data-sys-asset-uid="blte3afb8b0dbb52ed4" alt="" style="display: block; margin: auto;">You will always write the data to the alias ("events" in this example). The alias will point to the index being written to. When events-000001 gets to 50GB and rolls over to events-000002, the alias will automatically be pointed to events-000002 by ILM.<h2>Additional considerations</h2>This concludes creating an ILM policy from Kibana.&nbsp;However, it is considered&nbsp; a best practice to have an ILM policy that is based on the size of indices where observation and measurement of an index size took place over a period of time in order to choose the right rollover condition. This can be achieved by setting up your rollover condition and fine-tune as you go. If daily indices are being used (created by Logstash or another client) and you want to use the index lifecycle policy to manage aging data, you can disable the rollover action in the hot phase. You can then transition to the warm and delete phases based on the time of index creation.
<h2>Wrapping up (for now)</h2>Managing data using hot-warm architecture and ILM is a great start for optimising the data contained within your deployment. It is a cost-effective way of retaining data in your cluster and can help you realise your data management policy while providing cost savings compared to the default I/O-optimised architecture.
Stay tuned as we release more cost saving tips and tricks (like <a href="/blog/optimizing-costs-elastic-cloud-replica-shard-management">replica management</a>). And if you haven’t tried our managed Elasticsearch Service on Elastic Cloud yet, take it for a spin in a <a href="https://www.elastic.co/elasticsearch/service">14-day free trial</a>. It’s packed full of great features for your end users, simple controls for your admins, and cost saving opportunities for Finance.

Optimizing costs in Elastic Cloud: Hot-warm + index lifecycle management

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

headshot-tom-lawler-300x300.jpg

Customer Success Manager

Tom Lawler

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/elastic-cloud-account-terms" target="_self">Conditions générales d'utilisation</a>, et vous acceptez de recevoir les <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">types de notification sélectionnés</a> à l'adresse que vous avez fournie plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/serverless-preview-terms" target="_self">Conditions générales d'utilisation</a>, et vous permettez à Elastic de <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">vous contacter</a> à propos de produits et services similaires en utilisant les informations que vous avez fournies plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

En envoyant votre demande d'inscription, vous acceptez les <a href="/fr/agreements/global/cloud-services-monthly" rel="nofollow">conditions générales d'utilisation d'Elastic Cloud</a>. Vos données personnelles seront traitées conformément à la <a href="/fr/legal/privacy-statement">déclaration de confidentialité d'Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/terms-of-use" target="_self">Conditions générales d'utilisation</a>, et vous permettez à Elastic de <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">vous contacter</a> à propos de produits et services similaires en utilisant les informations que vous avez fournies plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.

Explore similar demos

Overview

You'll also receive an email with related content.

You'll also receive an email with related content

Nous vous enverrons plus de contenu susceptible de vous intéresser.

Speakers

Close

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Le contenu de cette page n'est pas disponible dans la langue sélectionnée. Chez Elastic, nous mettons tout en œuvre pour vous proposer du contenu dans différentes langues. En attendant, nous vous remercions de votre patience !

See more insights

Articles by

Author

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Voir à quelle heure débute le webinar dans mon fuseau horaire

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Si vous ne pouvez pas vous libérer le jour J, inscrivez-vous et nous vous enverrons l'enregistrement. Vous recevrez aussi un e-mail contenant des ressources connexes.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

Thank you for registering. We will send you a confirmation email soon.

Nous vous remercions de votre inscription. Vous allez recevoir un e-mail de confirmation.

Author

Articles by Tom Lawler

Customer Success Manager, Elastic

Optimizing costs in Elastic Cloud: Availability zones and snapshot management

Optimizing costs in Elastic Cloud: Replica shard management

Optimizing costs in Elastic Cloud: Hot-warm + index lifecycle management