A template that you typically use for time-series analytics and log aggregation workloads that benefit from tiered-storage automatic index curation. Includes features to manage resources efficiently when you need greater capacity, such as:
- A tiered architecture with two different types of data nodes, hot and warm.
- Time-based indices, with automatic index curation to move indices from hot to warm nodes over time by changing their shard allocation.
The two type of data nodes in a hot-warm architecture each have their own characteristics:
- Hot data node
- Handles all indexing of new data in the cluster and holds the most recent daily indices that tend to be queried most frequently. Indexing is an I/O intensive activity and the hardware these nodes run on needs to be more powerful and use SSD storage.
- Warm data node
- Handles a large amount of read-only indices that are not queried frequently. With read-only indices, warm nodes can use very large spindle drives instead of SSD storage.
Index Curation
One of the key features of a hot-warm architecture, time-based index curation automates the task of moving data from hot to warm nodes as it ages. When you deploy a hot-warm architecture, Elastic Cloud performs regular index curation according to these rules:
- Index curation moves indices from one Elasticsearch node to another by changing their shard allocation, always from hot to warm.
- Index curation is always time-based and takes place when an index reaches the age specified, in days, weeks, or months.
- Index curation always targets indexes according to one or more matching patterns. If an index matches a pattern, Elastic Cloud moves it from a hot to a warm node.
You can define which indices get curated and when during the process of creating your deployment.
To learn more about how hot-warm architectures work with Elasticsearch, see “Hot-Warm” Architecture in Elasticsearch 5.x.
Future versions of Elasticsearch will support index lifecycle management, which will offer richer index curation semantics compared to our current cloud-based index curation mechanism. We recommend using these built-in index life cycle management features when they become available.
In This Template
The following features are included with this template:
- Amazon Web Services (AWS)
Elasticsearch:
- Data nodes - hot: Starts at 4 GB memory x 3 availability zones. Hosted on AWS i3 instances.
- Data nodes - warm: Starts at 4 GB memory x 3 availability zones. Data nodes must be at least 4 GB in size. Hosted on AWS d2 instances.
- Master nodes: One master node per availability zone, plus a tiebreaker master node when you use two zones. With fewer than five data nodes, master nodes are co-located with the data nodes; with five or more data nodes, dedicated master-eligible nodes are used. Hosted on AWS r4 instances.
- Kibana: Starts at 1 GB memory x 1 availability zone. Hosted on AWS r4 instances.
Machine learning (ML): Disabled by default. The functionality is pre-wired into the template, but you must explicitly enable it in the UI. Hosted on AWS m5 instances.
- Google Cloud Platform (GCP)
Elasticsearch:
- Data nodes - hot: Starts at 4 GB memory x 3 availability zones. Hosted on custom I/O-optimized GCP instances.
- Data nodes - warm: Starts at 4 GB memory x 3 availability zones. Data nodes must be at least 4 GB in size. Hosted on custom GCP instances with extra persistent storage.
- Master nodes: One master node per availability zone, plus a tiebreaker master node when you use two zones. With fewer than five data nodes, master nodes are co-located with the data nodes; with five or more data nodes, dedicated master-eligible nodes are used. Hosted on custom memory-optimized GCP instances.
- Kibana: Starts at 1 GB memory x 1 availability zone. Hosted on custom memory-optimized GCP instances.
Machine learning (ML): Disabled by default. The functionality is pre-wired into the template, but you must explicitly enable it in the UI. Hosted on custom CPU-optimized GCP instances.