Here are the highlights of what’s new and improved in Elasticsearch 7.12!
Frozen tier and shared snapshot cachesedit
With 7.12, we’re adding a frozen phase to ILM along with a corresponding frozen data tier as experimental features. You can use the new frozen phase and frozen tier to archive rarely searched data as searchable snapshots for long-term storage and maximum cost savings.
The key advantage of the frozen tier is an experimental new option for mounting searchable snapshots: shared snapshot caches. Rather than loading a full copy of a snapshotted index into your cluster, this option uses a small local cache containing only recently searched parts of the index’s data. If a search requires data that’s not in the cache, Elasticsearch fetches the missing data from the snapshot repository. These caches have a fixed size and can significantly reduce your storage costs. ILM uses shared snapshot caches by default in the frozen phase and corresponding frozen tier.
The frozen tier is not yet available on the Elasticsearch Service. However, you can recreate a frozen tier on the Elasticsearch Service using shared snapshot caches. See Configure a frozen tier on the Elasticsearch Service.
To add a node to the frozen tier in an on-premise cluster, assign it the
data_frozen node role and set
elasticsearch.yml. See the Data tiers
Analyze snapshot repositoriesedit
While there are many third-party storage systems, not all of them work well as snapshot repositories. Some systems perform poorly or behave incorrectly as a repository, especially when multiple Elasticsearch nodes access them concurrently.
To detect and avoid these problems, you can now use the repository analysis API to analyze a repository’s storage system before using it in production. The API performs a collection of read and write operations on the repository. These operations are designed to detect incorrect behavior and measure the performance characteristics of a system.
in lookups and functionsedit
not in lookup
operators now support case-insensitive variants:
not in~. You can
also make an EQL function
case-insensitive by adding
~ after the function name, such as
In 7.12, we added the
regex keywords to EQL. You can use the
keyword to match strings to wildcard
patterns, such as
ba?. You can use the
regex keyword to match
strings to regular expressions. You can use both
regex to match a
string against a list of patterns.
Retention policy for transformsedit
Continuous transforms add new data to the destination index as new entities are
encountered. 7.12 introduces a data retention policy for transforms that
enables you to delete old data. This is especially useful for the
transform that has been added in 7.11. For example, if you collect the latest
host information you might want to assume hosts have been decommissioned if they
have not appeared in the logs for more than 30 days. At each checkpoint,
transformed documents whose age is greater than a configured value are deleted.
Data frame analytics training is influenced by settings called hyperparameters. One of the training outputs is now hyperparameter importance. This value shows which hyperparameter had the most impact on the training, and what the optimum values were for each one. You can use this information to speed up the training process as the hyperparameter optimization process can be stopped when you see that gains in accuracy flatten.
Search-time runtime fields support for transformsedit
You can now use
runtime_mappings in transforms to support search-time
runtime fields in searches of the source index. The syntax is the same as for a
standard search. After specifying them, the runtime fields can be used in the
rest of the transform configuration.