Elasticsearch 7.3.0 is here! Learn about data frames, voting-only master nodes, vector similarity functions, and more.

<p>Today we are pleased to announce the release of Elasticsearch 7.3.0, based on Lucene 8.1.0. This is the latest stable release, and is already available for deployment via our <a href="/products/elasticsearch/service">Elasticsearch Service</a>.
</p><p>
	Latest stable release in 7.x:
</p><ul>
	<li><a href="/downloads/elasticsearch">Download Elasticsearch 7.3.0</a></li>
	<li><a href="/guide/en/elasticsearch/reference/7.3/release-notes-7.3.0.html">Elasticsearch 7.3.0 release notes</a></li>
	<li><a href="/guide/en/elasticsearch/reference/7.3/breaking-changes.html">Elasticsearch 7.3 breaking changes</a></li>
</ul><p>You can read about all the changes in the release notes linked above, but let's take a look at a few changes worth highlighting.
</p><h2>Data frames — transform and pivot your streaming data</h2><p>Data frame transforms are a core new feature in Elasticsearch that enable you to transform an existing index to a secondary, summarized index. <a href="/guide/en/elasticsearch/reference/7.3/data-frame-apis.html">Data frame transforms</a> enable you to pivot your data and create entity-centric indices that can summarize the behavior of an entity. This organizes the data into an analysis-friendly format.
</p><p>Data frame transforms were originally available in 7.2. With 7.3 they can now run either as a single batch transform or continuously incorporating new data as it is ingested.
</p><p>Imagine you are streaming audit logs events from the many different hosts you have in your data center, and you want to analyze user behavior to find anything that looks suspicious. With data frames you can group the logs events by user, by host, and by day of the week. So for each user you have a count of interactions per server grouped by type of request. Organizing the data by entities and summarizing many events makes it easier to run different numerical analysis models and spot unusual behaviors.
</p><p>Data frames allow for new possibilities of machine learning analysis (such as outlier detection, which could be a perfect match for the above security example), but they can be useful for other types of visualisations or custom types of analysis. We are curious to see what users are up to using this feature.
</p><p>Data frame transforms is released as beta under a free Basic license.
</p><h2>Giving search a boost</h2><p>Elasticsearch is for search, and here are some exciting new search features we're releasing with 7.3.
</p><h3>Discover the least-frequent values</h3><p>We’ve added an all new <a href="/guide/en/elasticsearch/reference/7.3/search-aggregations-bucket-rare-terms-aggregation.html"><code>rare_terms</code> aggregation</a>, which uses a resource efficient algorithm with predictable results. It is an aggregation designed to identify the long-tail of keywords, such as terms that have low doc counts. From a technical perspective, rare terms aggregation works by maintaining a map of terms with a counter associated with each value. The counter is incremented each time the term is identified. If the counter surpasses a predefined threshold, the term is removed from the map and inserted into a <a href="https://en.wikipedia.org/wiki/Cuckoo_filter">cuckoo filter</a>. If a future term is found in the cuckoo filter, we assume it was previously removed from the map and is "common." This aggregation is designed to be more memory efficient than the alternative, which is setting a terms aggregation to <code>size: MAX_LONG</code>, or ordering a terms aggregation by count ascending, <a href="/guide/en/elasticsearch/reference/7.3/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order">in which errors are unbounded</a>.
</p><p>There are multiple use cases for rare terms aggregation; for example, <a href="/products/siem">SIEM</a> users are frequently interested in rare occurrences, which are sometimes suspected to be a manifestation of a security event.
</p><h3>Built-in vector similarity functions for document script scoring</h3><p>There are many popular algorithms that represent records by vectors (such as <a href="https://en.wikipedia.org/wiki/Word2vec">word2vec</a> and <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network">convolutional neural networks</a>), which allow relating to vector similarity as a measurement for record similarity. In this release we added <a href="/guide/en/elasticsearch/reference/7.3/query-dsl-script-score-query.html#vector-functions">two predefined functions</a> to use for calculating vector similarity between a given query vector and document vectors:
</p><ul>
	<li>Cosine similarity</li>
	<li>Dot product similarity</li>
</ul><p>These are the two types of distance functions most frequently used in vector comparisons. We released these as Painless script functions, so users have full flexibility to use them in relevance ranking in combination with other fields. Users can use these Painless functions for scoring through the <a href="/guide/en/elasticsearch/reference/7.3/query-dsl-script-score-query.html"><code>script_score</code></a> query. We plan to release additional vector similarity functions such as Euclidean distance and Manhattan distance in future releases, because each of these vector similarity functions proved superior for specific scenarios.
</p><p>Some of our users are using Elasticsearch as the data source for their machine learning algorithms, and have been requesting such functionality. We are thrilled to see what novel use the community will now find for Elasticsearch with the introduction of this capability.
</p><p>This experimental functionality is released under a free Basic license.
</p><h3>Improved intervals query</h3><p>In 7.0 we introduced the <a href="/guide/en/elasticsearch/reference/7.3/query-dsl-intervals-query.html">intervals query</a>. This query is used when users want to find records in which words or phrases are within a certain distance from each other. It provides advanced search options in an easy-to-define syntax and yields accurate results.
</p><p>The intervals query is great for use cases such as legal and patent searches. Version 7.3 includes two important additions to intervals queries:
</p><ul>
	<li>The <a href="/guide/en/elasticsearch/reference/7.3/query-dsl-intervals-query.html#intervals-wildcard"><code>wildcard</code> rule</a> within an intervals query allows for relating to a set of terms defined with a wildcard (* and ?), as well as choosing the analyzer that will be used.</li>
	<li>The <a href="/guide/en/elasticsearch/reference/7.3/query-dsl-intervals-query.html#intervals-prefix"><code>prefix</code> rule</a> allows for defining intervals between terms that start with specific letters, with the option to <a href="/guide/en/elasticsearch/reference/7.3/index-prefixes.html">index specifically for prefix </a>or to limit the query expansion to 128 alternative terms.</li>
</ul><p>Now that intervals queries are nearing feature parity with span queries, users can switch to intervals queries for the likes of legal and patent search.
</p><h3>Efficiently handle records with large number of dynamic fields</h3><p>The new <a href="/guide/en/elasticsearch/reference/7.3/flattened.html">flattened object field</a>  will allow an entire flat JSON object to be indexed into a single field. This comes in handy for situations in which documents include a very large number of fields, such as HTTP headers or image metadata. Sub-fields of flattened objects behave almost exactly like keyword fields, so only basic queries and aggregations are allowed (no support for numeric range queries, full-text search, or highlighting). Prior to 7.3, records with a large number of fields had to be indexed into separate fields, which can greatly increase the number of mappings, making mappings more difficult to manage and increasing the size of the cluster state.
</p><p>Support for flattened object types is released under a free Basic license.
</p><h3>Update index synonyms list without index downtime</h3><p>With this feature, synonym filters used by <a href="/guide/en/elasticsearch/reference/7.3/search-analyzer.html">search analyzers</a> can be updated quickly and flexibly. For example, e-commerce businesses can add new synonyms flexibly to verify that new products are associated with users’ queries, and do not return empty result sets. With the new <a href="/guide/en/elasticsearch/reference/7.3/indices-reload-analyzers.html">Reload Search Analyzers</a> API, updating synonym filters only requires placing the file on the node and calling the Reload Search Analyzers API, which loads the search analyzers without restarting the index shards. This enables a user to update the synonyms per index without search and indexing downtime (index close and reopen).
</p><p>This experimental feature is released under a free Basic license.
</p><h2>Hold on, we have more cool stuff</h2><p>
	7.3 isn't just about improved search functionality, it also includes plenty of other new niceties.
</p><h3>A new type of master node just registered to vote</h3><p>Elasticsearch 7.0 introduced a <a href="/blog/a-new-era-for-cluster-coordination-in-elasticsearch">new cluster coordination layer</a>, with a number of improvements, including faster master elections, removal of the minimum_master_nodes setting, and use of <a href="/elasticon/conf/2018/sf/reliable-by-design-applying-formal-methods-to-distributed-systems">formal methods for design validation</a>. There was another important benefit to the new cluster coordination layer in Elasticsearch 7.0: It could be used as a foundation for important improvements in Elasticsearch, such as voting-only master-eligible nodes. A voting-only master-eligible node is a node that can participate in master elections but will not act as a master in the cluster (it only votes in elections). By only voting in elections, a smaller machine can be used, and fewer hardware resources are required for the cluster. Head over to the Elasticsearch docs for <a href="/guide/en/elasticsearch/reference/7.3/modules-node.html#voting-only-node">instructions on setting up a voting-only master-eligible node</a> in Elasticsearch 7.3.
</p><p>Voting-only master-eligible nodes are available under a free Basic license.
</p><h3>Aliases can travel cross cluster</h3><p>Cross-cluster replication (CCR) was released as a <a href="/blog/elasticsearch-6-7-0-released">GA feature in Elasticsearch 6.7</a>. CCR has a variety of use cases, including <a href="/blog/cross-datacenter-replication-with-elasticsearch-cross-cluster-replication">cross-datacenter and cross-region replication</a>, replicating data to get closer to the application server and user, and maintaining a centralized reporting cluster replicated from a large number of smaller clusters. Elasticsearch 7.3 includes additional CCR functionality to ensure aliases that are manipulated on leader indices are replicated to the follower indices. Note: this process ignores <a href="/guide/en/elasticsearch/reference/7.3/indices-aliases.html#aliases-write-index">write aliases</a> because follower indices do not receive direct writes, so write aliases have no use.
</p><h3>SQL query support for frozen indices for API clients and JDBC/ODBC drivers</h3><p>This feature allows for <a href="/guide/en/elasticsearch/reference/7.3/sql-index-frozen.html">querying of frozen</a> indices through dedicated <a href="/webinars/introduction-to-elasticsearch-sql">SQL grammar extension</a>. Frozen indices are a super efficient way to keep older data that is not frequently searched, and doing that at low cost. Because users frequently will refrain from including frozen indices in their “normal” queries, when using SQL you need to explicitly request to include frozen indices. This can be done by using the FROZEN reserved word, such as <code>SELECT * FROM FROZEN myIndex LIMIT 10;</code>.
</p><h3>GUI support for snapshot restore and deletion</h3><p>Elasticsearch management UI (Kibana &gt; Management &gt; Elasticsearch) keeps evolving. In this version we enhanced the previously released <a href="/blog/elasticsearch-7-2-0-released">Snapshot Repositories section</a>, now called Snapshot and Restore, with the ability to restore from an existing snapshot. The snapshot restore wizard walks you through defining a restore task. The progress of currently running restores can be tracked in the Restore Status view. Snapshot deletion is also now available from the UI. For more information about these enhancements, see <a href="/guide/en/kibana/7.3/snapshot-repositories.html">Snapshot and Restore</a>.
</p><p>This UI functionality is released under a free Basic license.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5830fc372cacbead/5d3a1da3ef4fb0543504c330/ES-7-3-release-snapshot--restore-blog.png" data-sys-asset-uid="blt5830fc372cacbead" alt="Snapshot and Restore UI" style="display: block; margin: auto;">
</p><h3>Discover your most unusual data using outlier detection</h3><p>The goal of <a href="/guide/en/elastic-stack-overview/7.3/dfa-outlier-detection.html">outlier detection</a> is to find the most unusual data points in an index. We analyze the numerical fields of each data point (document in an index) and annotate them with how unusual they are.
</p><p>We use unsupervised outlier detection, which means there is no need to provide a training data set to teach outlier detection to recognize outliers. In practice, this is achieved by using an ensemble of distance-based and density-based techniques to identify those data points that are the most different from the bulk of the data in the index. We assign to each analyzed data point an outlier score, which captures how different the entity is from other entities in the index.
</p><p>In addition to new outlier detection functionality, we are introducing the <a href="/guide/en/elasticsearch/reference/7.3/evaluate-dfanalytics.html">Evaluate API</a>, which enables users to compute a range of performance metrics such as confusion matrices, precision, recall, the <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic">receiver-operating characteristics (ROC) curve</a>, and the area under the ROC curve. If you are running outlier detection on a source index that has already been labeled to indicate which points are truly outliers and which are normal, you can use the Evaluate API to assess the performance of the outlier detection analytics on your dataset.
</p><p>This experimental feature is released under a Platinum license.
</p><h2>Conclusion</h2><p>Please <a href="/downloads/elasticsearch">download Elasticsearch 7.3.0</a>, try it out, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/">Discuss forums</a>. You can report any problems on the <a href="https://github.com/elastic/elasticsearch/issues">GitHub issues page</a>.
</p>

blog-banner-release-elasticsearch.png

blog-thumb-release-elasticsearch.png

Elasticsearch 7.3.0 released

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

headshot-yaron-parasol-300x300.png

Yaron Parasol

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/elastic-cloud-account-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">ausgewählte Nachrichten und Informationen</a> an die oben von Ihnen bereitgestellten Kontaktangaben zu senden. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/serverless-preview-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.</p>

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

<p>Beim Absenden akzeptieren Sie die <a href="/de/agreements/global/cloud-services-monthly" rel="nofollow">Nutzungsbedingungen von Elastic Cloud</a>. Die Verarbeitung Ihrer personenbezogenen Daten unterliegt der <a href="/de/legal/privacy-statement">Datenschutzerklärung von Elastic</a>.
</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

<p>Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/terms-of-use" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.</p>

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Der Inhalt dieser Seite ist in der ausgewählten Sprache nicht verfügbar. Wir bei Elastic arbeiten daran, die bereitgestellten Inhalte in verschiedenen Sprachen anzubieten. Bis dahin bitten wir Sie um etwas Geduld und hoffen auf Ihr Verständnis!

Author

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Articles by

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Sie können nicht dabei sein? Registrieren Sie sich, damit wir Ihnen die Aufzeichnung zukommen lassen können. Außerdem erhalten Sie eine E‑Mail mit Begleitmaterial.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Learn More

Read less

You'll also receive an email with related content

Wir schicken Ihnen zudem relevante Informationen, die von Interesse sein könnten.

Thank you for registering. We will send you a confirmation email soon.

Vielen Dank für Ihre Registrierung. Sie erhalten in Kürze eine Bestätigungs-E‑Mail von uns.

Author

Articles by Yaron Parasol

Elasticsearch 7.3.0 released

Sign up for Elastic Cloud free trial