A template that you typically use for time-series analytics and log aggregation workloads that benefit from tiered-storage automatic index curation. Includes features to manage resources efficiently when you need greater capacity, such as:
- A tiered architecture with two different types of data nodes, hot and warm.
- Time-based indices, with automatic index curation to move indices from hot to warm nodes over time by changing their shard allocation.
The two type of data nodes in a hot-warm architecture each have their own characteristics:
- Hot data node
- Handles all indexing of new data in the cluster and holds the most recent daily indices that tend to be queried most frequently. Indexing is an I/O intensive activity and the hardware these nodes run on needs to be more powerful and use SSD storage.
- Warm data node
- Handles a large amount of read-only indices that are not queried frequently. With read-only indices, warm nodes can use very large spindle drives instead of SSD storage. Reducing the overall cost of retaining data over time yet making it accessible for queries.
One of the key features of a hot-warm architecture, time-based index curation automates the task of moving data from hot to warm nodes as it ages. When you deploy a hot-warm architecture, Elastic Cloud Enterprise performs regular index curation according to these rules:
- Index curation moves indices from one Elasticsearch node to another by changing their shard allocation, always from hot to warm.
- Index curation is always time-based and takes place when an index reaches the age specified, in days, weeks, or months.
- Index curation always targets indexes according to one or more matching patterns. If an index matches a pattern, Elastic Cloud Enterprise moves it from a hot to a warm node.
While you create your deployment, you can define which indices get curated and when. To know more about index curation, see Configure index management
To know more about how hot-warm architectures work with Elasticsearch, see “Hot-Warm” Architecture in Elasticsearch 5.x.
In this templateedit
The following features are included with this template:
Data nodes - hot: Starts at 4 GB memory x 1 availability zone. Uses the
Data nodes - warm: Starts at 4 GB memory x 1 availability zone. Uses the
Additional master-eligible node added when choosing 2 availability zones (to create a quorum of 3).
When 1 AZ or 3 AZ are selected, the data nodes act as master-eligible node and there is no requirement for an additional master-eligible node.
Configurations beyond 5 nodes per AZ can also spin up a dedicated master-eligible set of nodes (in 3 AZs always) to offload the data nodes. Uses the
- Data nodes - hot: Starts at 4 GB memory x 1 availability zone. Uses the
Kibana: Starts at 1 GB memory x 1 availability zone. Uses the
Machine learning (ML): Disabled by default. The functionality is pre-wired into the template, but you must explicitly enable it in the UI. Uses the
APM (application performance monitoring): Disabled by default. The functionality is pre-wired into the template, but you must explicitly enable it in the UI. Uses the
To use this template effectively, you must tag your allocators and edit the default instance configurations, so that ECE knows where to host the Elastic Stack products that are part of your deployment.
Intro to Kibana
ELK for Logs & Metrics