Hot-Warm Architecture Template

A template that you typically use for time-series analytics and log aggregation workloads that benefit from tiered-storage automatic index curation. Includes features to manage resources efficiently when you need greater capacity, such as:

  • A tiered architecture with two different types of data nodes, hot and warm.
  • Time-based indices, with automatic index curation to move indices from hot to warm nodes over time by changing their shard allocation.

The two type of data nodes in a hot-warm architecture each have their own characteristics:

Hot data node
Handles all indexing of new data in the cluster and holds the most recent daily indices that tend to be queried most frequently. Indexing is an I/O intensive activity and the hardware these nodes run on needs to be more powerful and use SSD storage.
Warm data node
Handles a large amount of read-only indices that are not queried frequently. With read-only indices, warm nodes can use very large spindle drives instead of SSD storage.

Index Curation

One of the key features of a hot-warm architecture, time-based index curation automates the task of moving data from hot to warm nodes as it ages. When you deploy a hot-warm architecture, Elastic Cloud performs regular index curation according to these rules:

  • Index curation moves indices from one Elasticsearch node to another by changing their shard allocation, always from hot to warm.
  • Index curation is always time-based and takes place when an index reaches the age specified, in days, weeks, or months.
  • Index curation always targets indexes according to one or more matching patterns. If an index matches a pattern, Elastic Cloud moves it from a hot to a warm node.

You can define which indices get curated and when during the process of creating your deployment.

To learn more about how hot-warm architectures work with Elasticsearch, see “Hot-Warm” Architecture in Elasticsearch 5.x.

In This Template

The following features are included with this template:

Amazon Web Services (AWS)
  • Elasticsearch:

    • Data nodes - hot: Starts at 4 GB memory x 3 availability zones. Hosted on AWS i3 instances.
    • Data nodes - warm: Starts at 4 GB memory x 3 availability zones. Data nodes must be at least 4 GB in size. Hosted on AWS d2 instances.
    • Master nodes: One master node per availability zone, plus a tiebreaker master node when you use two zones. With fewer than five data nodes, master nodes are co-located with the data nodes. With five or more data nodes, dedicated master-eligible nodes are used. Hosted on AWS r4 instances.
  • Kibana: Starts at 1 GB memory x 1 availability zone. Hosted on AWS r4 instances.
  • Machine learning (ML): Disabled by default. The functionality is pre-wired into the template, but you must explicitly enable it in the UI. Hosted on AWS m5 instances.

Hosted on AWS r4 instances.

Google Cloud Platform (GCP)