Elasticsearch 7.2.0 is here! Learn about our new proximity ranking (geo and time), enhancements to SQL, and support for OpenID Connect.

<p>Today we are pleased to announce the release of Elasticsearch 7.2.0, based on Lucene 8.0.0. This is the latest stable release, and is already available for deployment via our <a href="/cloud/elasticsearch-service">Elasticsearch Service</a> on <a href="https://www.elastic.co/cloud">Elastic Cloud</a>.
</p><p>Latest stable release in 7.2.0:
</p><ul>
	<li><a href="https://www.elastic.co/downloads/elasticsearch">Download Elasticsearch 7.2.0</a></li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/release-notes-7.2.0.html">Elasticsearch 7.2.0 release notes</a></li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/breaking-changes.html">Elasticsearch 7.2 breaking changes</a></li>
</ul><p>You can read about all the changes in the release notes linked above, but there are a few changes which are worth highlighting:
</p><h2>Geo and Time Proximity as Ranking Consideration</h2><p>In many scenarios, things that are closer are more relevant, and now there is a simple and effective way to include proximity among the relevance ranking considerations. When searching for information, records that are nearer to a point in time (typically more recent) are typically more relevant. When searching for anything with physical presence, the geographical proximity should typically play a part in defining relevancy. In other words, including proximity (time or geo) in relevance ranking will improve ranking in most scenarios. With Elasticsearch 7.2.0 users can normalize the time or geographical distance using saturation normalization function and add them to the relevance ranking score, using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/query-dsl-distance-feature-query.html">distance feature query</a> to achieve better relevance ranking. This feature already leverages the ‘<a href="https://www.elastic.co/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand">top-k faster queries</a>’ performance improvement we introduced in 7.0. Read more about this capability and how to configure it in our blog post <a href="https://www.elastic.co/blog/distance-feature-query-time-and-geo-in-elasticsearch-result-ranking">Distance Feature Query: Time and Geo in Elasticsearch Result Ranking</a>.
</p><h2>Typeahead Search</h2><p>Those of you that have worked with search systems, probably noticed that getting suggested search results while you type the query can save you a lot of time.
</p><p>The needs and benefits are thus clear and simple. The engineering challenge is, however, significant: there are typically many words in the index that start with the few letters that the user keyed in, yet the system has but tens of milliseconds to come up with a few top ranking suggestions before the user types in the next letter, potentially with a spelling mistake… This is a clear case in which there is no need to know the number of results or make some aggregation about them, but rather to bring the few top ranking results, typically the ones that are most frequently found in the index. Because only the top ranking results are of interest, we can leverage the ‘<a href="https://www.elastic.co/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand">top-k faster queries</a>’ mechanism to achieve better performance with lower hardware requirements. The improved performance and reduced consumption of computing resources in turn allows us to do more: in 7.2 we support cases in which the word that the user keyed in is not necessarily the first word in the field, so a user may key in e.g. ‘hem’ and get the suggestion ‘Ernest Hemingway’ from the field.
</p><p>To achieve that we introduced a new field type -- <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/search-as-you-type.html">search_as_you_type</a> -- that is optimized for providing results from the field while the user is typing the query. The number of shingles used is configurable so that any user can balance index size and recall considerations according to their particular needs.
</p><h2>Resiliency Improvements<br></h2><p>Elasticsearch used to be, you know, just for search. Nowadays, it is also rapidly becoming the main data repository and analytical system for many users, or in other words, the source of truth. One of the reasons for this change is the system’s reliability and resilience. We at Elastic take this responsibility very seriously, and we keep improving the mechanisms that allow users to deal with malfunctions if they occur, and to do that in an economic manner. Features like replication, snapshot and backup allow our customers to maintain a more reliable service. In 7.2 we added two new mechanisms for that: replicated closed indices and snapshot UI.
</p><h3>Replicated Closed Indices</h3><p>In 7.2 we have added a layer on top of the closed indices mechanism, that allows replicating closed indices. We made this improvement in response to many users’ requests and as part of our commitment to resiliency and ease of use. We do expect, however, that the majority of our users will be using Frozen Indices instead of Closed Indices. <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/frozen-indices.html">Frozen Indices</a> have nearly no memory footprint, which is the main reason for which users use Closed Indices, but Frozen Indices can be searched almost as if they were open indices, without having to actively manage the logic and workflow of opening and closing the index. &nbsp;
</p><h3>Snapshot Repositories UI</h3><p>Elasticsearch’s snapshot mechanism allows keeping a backup of data at a point in time, using an efficient incremental snapshotting mechanism.
</p><p>With 7.2 we added a new <a href="https://www.elastic.co/guide/en/kibana/7.2/snapshot-repositories.html">Snapshot Repositories</a> app to Kibana. This app allows you to register repositories using all the cloud and on-prem storage plugins. It also allows you to browse the repositories and snapshots that you've created. We plan to enhance this app with functionality for taking, restoring, and deleting snapshots in future releases.
</p><p>Snapshot Repositories &nbsp;is released under Basic license.
</p><p><span></span><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blteeb08831c4456723/5d000b737e77466b173d9250/register_repo.png" data-sys-asset-uid="blteeb08831c4456723" alt="Register your snapshot repository" "="">
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc54a919bb57eb2e1/5d000b83c06daee314d1f4fc/snapshot_details.png" data-sys-asset-uid="bltc54a919bb57eb2e1" alt="Snapshots tab in Snapshot Repositories" "="">
</p><h2>Enhancements to Elasticsearch SQL</h2><p>The ability to query Elasticsearch using SQL statements has just recently been made generally available as part of <a href="https://www.elastic.co/blog/elasticsearch-6-7-0-released">Elasticsearch 6.7</a> and is gaining popularity, in particular with those who are fluent in SQL but new to DSL and in scenarios of integration, including with BI tools, which frequently have built in integration through SQL. In line with its popularity, we keep expanding Elasticsearch’s SQL capabilities. Here are some of the things we added in 7.2:
</p><ul>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/sql-functions-geo.html">Geographical queries through SQL statements</a></li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/search-aggregations-metrics-median-absolute-deviation-aggregation.html">Median Absolute Deviation</a>, which calculates the median of each data point’s deviation from the median of the whole sample, i.e. for &nbsp;random variable x it is median(|median(x)-xi|)</li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/sql-functions-conditional.html">CASE/WHEN/ELSE/END</a> to define a set of if-else statements</li>
</ul><p>The above SQL functionality is released under Basic license.
</p><h2>Data Frames</h2><p>In 7.2 we have added <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/data-frame-apis.html">Data Frames</a>, a new plugin that allows you to pivot and aggregate your data from one or multiple indices to a single, separate index. This enables you to transform existing indexed data for specific use-cases. For example, you can now summarize users’ behaviors and gain new insights using a data frame pivot transform.
</p><p>Data frames are available under Basic license.
</p><h2>HTML Strip Processor</h2><p>The <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/htmlstrip-processor.html">HTML Strip Processor</a> simply replaces all HTML tags with “\n”. This operation means the _source field contains text that is more readable to humans. If you want to display the source data but not as an HTML, this new processor will probably save you some development time. It’s a fairly frequent case, so we decided to include it in the product.
</p><h2>OpenID Connect Realm</h2><p>In today’s software world security is a top priority. Unfortunately, security needs sometimes end up burdening users, e.g. by requesting users to authenticate separately to different systems. To avoid that, modern software systems use authentication layers to allow users to roam freely between various systems while automatically identifying themselves to every new system they access and retaining their permissions.
</p><p>So what is OpenID Connect? OpenID Connect is a commonly used authentication layer on top of oAuth2. oAuth2 is a protocol that allows third-party applications to grant limited access to an HTTP service. OpenID Connect is the authentication backbone used by Okta, Google and others, to allow users to authenticate in the background when moving between connected systems.
</p><p>OpenID Connect (OIDC) support is widely adopted by enterprise systems, so the newly added support in Elasticsearch and Kibana provides many of our customers with new (but commonly asked for) choices of how to secure their stack.
</p><p>OpenID support is provided under Platinum license.
</p><h2>Conclusion</h2><p>Please <a href="https://www.elastic.co/downloads/elasticsearch">download Elasticsearch 7.2.0</a>, try it out, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our forum. You can report any problems on the <a href="https://github.com/elastic/elasticsearch/issues">GitHub issues page</a>.
</p>

elasticsearch-blog-banner.png

elasticsearch-blog-thumb.png

Elasticsearch 7.2.0 released

Coming soon to Elasticsearch, you'll be able to implement the Distance Feature Query to use proximity (geographical and time) as part of your ranking score.

<p>In Elasticsearch, you will soon be able to use proximity (geographical or time) as part of your ranking score. The new way for combining proximity within the ranking score is easy to configure, performs well, and will likely yield superior ranking in many scenarios. This blog explains why and how.
</p><h2>Why use geographical distance and time for ranking?</h2><p>Consider a search for a restaurant. One typically uses a criteria, including type, price range, rating, as well as physical proximity. All other things kept equal, you would prefer the nearest restaurant. This is not unique — it would be the same in countless other relevance ranking scenarios e.g. when searching for a job, a date, a plumber, a dog-walker, a babysitter, a tourist attraction. In fact, relevance ranking for anything in which physical location matters — which is almost anything that exists in the physical world — would benefit from relating to geographical distance.
</p><p>A similar situation occurs with time related search, e.g. when searching for an academic article, a newspaper article, a blogpost or a product review, one will typically prefer recent records over less recent records (assuming all other aspects are equal). As with physical proximity in search for things that have physical location in the real world, recency would be the typical preference in searches for information. In other words, almost any relevance ranking criteria would benefit from the inclusion of proximity, either date proximity or geographic proximity.
</p><p>We have frequently seen date and geographical proximity used in either of the following ways:
</p><ol>
	<li>As a filter, i.e. dichotomic criteria, e.g. retrieve only records nearer than x or within the last y days, and rank these records by other search criteria (e.g. using <a href="https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables">BM25</a> on text)</li>
	<li>As the only ranking criteria, e.g. sort all the records that answer a dichotomic criteria by their geographical or date proximity</li>
</ol><p>In other words, proximity is either used as a retrieval criterion, rather than for relevance ranking, or is used as a single criterion for relevance ranking.
</p><p>The problem is that the user is often willing to walk a longer distance for a better ranked restaurant, wait a little longer for a better ranked babysitter or read the most cited article even if it is not the most recent. In other words our actual search criteria as searchers combines proximity (space and time) with other criteria (such as word frequency related criteria) into a single relevance score.
</p><h2>Can’t you do it with Function Score or Painless script?</h2><p>Picking our other hat, as the people designing, tuning and programming search algorithms, we frequently avoided such a combined proximity and word frequency (or various other numeric) ranking criteria because it was technically challenging. There are two main challenges:
</p><ol>
	<li><strong>Normalization</strong> - The proximity score has to be normalized so that it will have the right impact when summed together with other relevance scores, such as BM25. The default in Lucene is to simply sum the various scores into one number and in most cases it makes sense, as long as the different scores have been normalized first. The normalization itself, however, is not trivial.</li>
	<li><strong>Performance</strong> - Re-ranking with proximity took a significant performance toll. To deal with it, the more sophisticated Elastic users frequently formed a criteria that did not include proximity to select a very small subset of most relevant records, and then reranked only these records after taking proximity into account. This compromise means the algorithm will not consider results that could become the most relevant ones after proximity is taken into account.</li>
</ol><p>We want to assist our users with non-trivial algorithmic challenges, so we sprung into action. In an upcoming version,&nbsp;we will enable users to rank all records with a combination of the ‘normal’ ranking score and a normalized proximity (time or space) score.
</p><p>So can you add proximity to the ranking algorithm using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-function-score-query.html">Function Score</a> or <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-function-score-query.html#function-script-score">Script Score</a>? It depends. It is possible to relate to proximity and to normalize it with function score and Painless script currently, but only on a small subset of the result set, and with a significant performance toll. Sometimes these quantitative differences sum up to a qualitative difference: with distance_feature queries, ranking by proximity in combination with other relevance ranking algorithms, becomes the reasonable principle ranking mechanism in many scenarios.
</p><h2>Saturation Normalization Function</h2><p>For the normalization, we used the concept of the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.x/query-dsl-rank-feature-query.html#_saturation_2">Saturation</a> normalization function. The concept behind the Saturation normalization function is similar to the one used in BM25 to relate to the length of the text in the document in setting the ranking score for that document. A record’s score will be computed using the following formula: <math>
	<mi>Score</mi>
	<mo>=</mo>
	<mi>Boost</mi>
	<mo>*</mo>
	<mfrac>
	<mi>Pivot</mi>
	<mrow>
	<mi>Pivot</mi>
	<mo>+</mo>
	<mi>Distance</mi>
	</mrow>
	</mfrac>
	</math>. Assuming <math>
	<mi>Pivot</mi>
	<mo>&gt;</mo>
	<mi>0</mi>
	</math>, the function <math>
	<mfrac>
	<mi>Pivot</mi>
	<mrow>
	<mi>Pivot</mi>
	<mo>+</mo>
	<mi>Distance</mi>
	</mrow>
	</mfrac>
	</math> is a nonlinear strictly decreasing function starting from <math>
	<mi>1</mi>
	</math> when <math>
	<mi>Distance</mi>
	<mo>=</mo>
	<mi>0</mi>
	</math> (assuming <math>
	<mi>Distance</mi>
	<mo>&gt;=</mo>
	<mi>0</mi>
	</math>) with <math>
	<msub>
	<mi>lim</mi>
	<mrow>
	<mi>Distance</mi>
	<mo>→</mo>
	<mi>∞</mi>
	</mrow>
	</msub>
	<mfrac>
	<mi>Pivot</mi>
	<mrow>
	<mi>Pivot</mi>
	<mo>+</mo>
	<mi>Distance</mi>
	</mrow>
	</mfrac>
	<mo>=</mo>
	<mi>0</mi>
	</math>. One way to think of the <math>
	<mi>Pivot</mi>
	</math> parameter is that it defines the distance at which <math>
	<mfrac>
	<mi>Pivot</mi>
	<mrow>
	<mi>Pivot</mi>
	<mo>+</mo>
	<mi>Distance</mi>
	</mrow>
	</mfrac>
	<mo>=</mo>
	<mfrac>
	<mi>1</mi>
	<mi>2</mi>
	</mfrac>
	</math>but another way to think of it is that it defines how quickly the contribution of distance to the score diminishes. The boost parameter is an optional parameter that by default is set to <math>
	<mi>1</mi>
	</math>, and allows to set the influence of distance on the overall score in a way that is reduced linearly to the distance, i.e. it is a way to indicate how important will distance be in the overall score, regardless of the specific area of the scale a record is in.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9ac76ab76b19c976/5cd0ccf34c5fe89364167936/geo-distance-relevance.png" data-sys-asset-uid="blt9ac76ab76b19c976" alt="Linear relevance reduction" style="display: block; margin: auto;">
</p><p>The above is the function plot for the function&nbsp;<math style="background-color: initial;"><mi>Score</mi>
	<mo>=</mo>
	<mfrac>
	<mi>Pivot</mi>
	<mrow>
	<mi>Pivot</mi>
	<mo>+</mo>
	<mi>Distance</mi>
	</mrow>
	</mfrac>
	</math> with <math style="background-color: initial;">
	<mi>Pivot</mi>
	</math> set to <math style="background-color: initial;">
	<mi>2</mi>
	</math> and <math style="background-color: initial;">
	<mi>Distance</mi>
	</math> plotted on the X axis.
</p><p>The bottom line is that by providing one mandatory parameter (<math style="background-color: initial;"><mi>Pivot</mi>
	</math>) the user gets a normalized score, ranging from <math style="background-color: initial;">
	<mi>1</mi>
	</math> to <math style="background-color: initial;">
	<mi>0</mi>
	</math> (given that our computers are discrete and have a ‘highest number’ that functions as infinity in this case). This score allows distance to impact more when the distance is small and less when the distance is large, which is how people typically want distance to impact.
</p><h2>Better performance enabling better ranking</h2><p>In theory, all of this could be done currently using re-ranking through Painless script. The addition of <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.x/query-dsl-distance-feature-query.html">Distance Feature Query</a> simplifies it considerably and opens the door for those who are less proficient with Painless, which is important, but not the main benefit. The main issue is that in practice the performance toll frequently meant that the proximity impact on the ranking score could only be calculated for a tiny fraction of the records. The implication was that it was quite likely that the most relevant records after the introduction of the proximity score would not be in the set for which this impact would be calculated.
</p><p>The Distance Feature Query performs better than the Painless script, but that’s not the big news. The really big news is that the distance feature query was designed to leverage the performance improvements resulting from the faster top-k queries (see <a href="https://www.elastic.co/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand">Magic WAND: Faster Retrieval of Top Hits in Elasticsearch</a> blog post). Those who can afford to get a lower boundary for number of results instead of an exact count can enjoy a potentially unbounded improvement in performance because the algorithm will disqualify at an early stage all records that cannot be one of the top K candidates.
</p><h2>Examples</h2><p>Let’s plan an application for hotel reviews. A core functionality is to allow searching for reviews about a hotel. The user will want to search for the specific hotel, and to get reviews about aspects that are of interest to the user in the hotel as well as being comprehensive and as recent as possible.
</p><p>Say our record structure is as follows:
</p><pre class="prettyprint">PUT reviews
{
  "mappings": {
    "properties": {
      "hotel": {
        "type": "text"
      },
      "review": {
        "type": "text"
      },
      "date": {
        "type": "date"
      }
    }
  }
}
PUT reviews/_doc/1
{
  "hotel": "Fawlty Towers",
  "review": "The hotel would have been managed in a perfect way, if it wasn't for the guests",
  "date" : "2017-12-27"
}
PUT reviews/_doc/2
{
  "hotel": "Fawlty Towers",
  "review": "The restaurant is, well, interesting",
  "date": "2018-07-07"
}
PUT reviews/_doc/3
{
  "hotel": "Fawlty Towers",
  "review": "We had an interesting vacation there. The service at the restaurant is a great chance to brush the dust off your Spanish. Mr. Fawlty himself had recently moved to The Life of Brian.",
  "date": "2019-03-25"
}
GET reviews/_search
{
    "query": {
        "bool" : {
            "must" : {
                "match" : {"hotel": "Fawlty Towers"}
            },
            "should" : [
                {"distance_feature" : {"field" : "date","pivot" : "30d","origin": "now"}},
                {"match" : {"review": "restaurant"}}    
            ]
        }
    }
}
Results:
{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1.5571662,
    "hits": [
      {
        "_index": "reviews",
        "_type": "_doc",
        "_id": "3",
        "_score": 1.5571662,
        "_source": {
          "hotel": "Fawlty Towers",
          "review": "We had an interesting vacation there. The service at the restaurant is a great chance to brush the dust off your Spanish. Mr. Fawlty himself had recently moved to The Life of Brian.",
          "date": "2019-03-25"
        }
      },
      {
        "_index": "reviews",
        "_type": "_doc",
        "_id": "2",
        "_score": 1.0365787,
        "_source": {
          "hotel": "Fawlty Towers",
          "review": "The restaurant is, well, interesting",
          "date": "2018-07-07"
        }
      },
      {
        "_index": "reviews",
        "_type": "_doc",
        "_id": "1",
        "_score": 0.32892755,
        "_source": {
          "hotel": "Fawlty Towers",
          "review": "The hotel would have been managed in a perfect way, if it wasn't for the guests",
          "date": "2017-12-27"
        }
      }
    ]
  }
}
</pre><p>The query filters to show only records in which the hotel is “Fawlty Towers”. Within these records, the ranking is based on the word restaurant, which two records contain. By BM25 only, record 2 would be ranked higher than record 1, just because both records include the word “restaurant” once and the review in the second record is longer than in the 3rd one (the first record does not include the word restaurant and so would be ranked lowest). However, due to the time distance query, the 3rd record would show up first, and the user will get the latest update about Fawlty Towers.
</p><p>The distance feature query skips non-competitive records and through that achieved better performance, which allows running the combined relevance criteria that includes both time proximity and BM25 on natural language.
</p><h2>Conclusion</h2><p>Proximity in either geographical space or time is an important aspect of almost every relevance ranking use case. Adding distance ranking to the relevance ranking, together with the normalization capability, will for sure prove useful in real life scenarios, and represents our long term strategic investment in relevance ranking.
</p><p>Just as important is the full story of how this capability came about. It’s all started with a user from the community that came up with <a href="https://issues.apache.org/jira/browse/LUCENE-4100">a suggestion to incorporate the MAXSCORE</a> algorithm into Lucene. That original suggestion, although it was not accepted to Lucene in its original form, later led to our contribution of the block-max WAND algorithm to Lucene, and more generally introduced APIs that help compute top hits without evaluating all matches. This in-turn led to the development of Lucene's FeatureField for static features — exposed in Elasticsearch via the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.x/rank-feature.html#rank-feature">rank_feature</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.x/rank-features.html">rank_features</a> fields — and now feature queries for dynamically computed distances. This cycle of contributing and exposing the code we produce so that the community can come up with feedback and ideas, helps steer us towards new, useful and innovative directions.
</p><p>We’re excited about this new feature, and we hope you are too. So for now, read more about the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.x/query-dsl-distance-feature-query.html"><code>distance_feature</code> query</a>, and once it&nbsp;is available, download it and take it for a spin. Once you’ve had a chance to test this new geo-ranking feature out, <a href="https://discuss.elastic.co">give us your feedback and comments</a> about how you want to use distance feature query. Your input will help direct us in our future development.
</p>

Articles by Gilad Gal

Elasticsearch 7.2.0 released

Distance Feature Query: Time and Geo in Elasticsearch Result Ranking

Products & Solutions

Company

Resources

Featured topics

News

Articles by Gilad Gal

Elasticsearch 7.2.0 released

Distance Feature Query: Time and Geo in Elasticsearch Result Ranking