Elasticsearch에서 숫자 필드와 날짜 필드의 정렬 쿼리 속도를 최대 수십 배까지 향상할 수 있는 최근 최적화 작업에 대해 알아보세요. 정렬 쿼리는 검색 도메인(전자 상거래, 보안)에 관계없이 흔하게 사용됩니다.

필드에 대한 문서 <X> 값(왼쪽)과 필드별로 정렬된 인덱스에 있는 필드에 대한<X> 동일한 문서 <X> 값(오른쪽).

elastic-blog-docvalues-1.png

여러 세그먼트가 있는 인덱스에서 search_after로 정렬 최적화(왼쪽) 및 인덱스를 단일 세그먼트로 강제 병합(오른쪽).  최적화(2021년 9월 13일) 전에는 search_after를 사용한 내림차순 정렬에 1,400~1,800밀리초가 걸렸고 최적화 후에는 200~300밀리초로 단축되었습니다.

elastic-blog-sortoptimizations-2.png

blog-thumb-search-charts.png

Optimizing sort queries in Elasticsearch for faster results

Elasticsearch에서 정렬 쿼리를 최적화하여 응답 속도 향상

Elasticsearch 7.0 brings some new tools to make relevance tuning easier. Learn about the new rank_feature and rank_features fields, and Script Score Queries.

Improving relevance is hard, it really is. There are often a variety of parameters and ranking characteristics to potentially change: how much boost should the title field get vs the body? &nbsp;Should you boost a category field, and if so, how much? Each of these are parameters you could manipulate to potentially get better (or worse!) results. Ideally you would employ a number of people to tag result sets for a large number of queries, and then optimize parameters that affect relevance to improve relevance metrics. The most common obstacle to this way of improving relevance is the creation of a training set, because it requires a lot of human time. Smaller organizations often end up guessing a scoring formula based on their knowledge of the domain that they are operating in. In any case, at some point you will need to combine textual relevance with other relevance signals such as popularity or authority. Elasticsearch 7.0 comes with some new tools that make this task easier.
Most text ranking schemes — including BM25 (the default ranking scheme in Elasticsearch) — merge the score contributions of multiple terms via a sum. It makes it natural to incorporate more score contributions via a sum as well. However these linear combinations only work well if features follow comparable distributions. For this reason, users have been asking for ways to get normalized scores with Elasticsearch, which has a number of <a href="https://wiki.apache.org/lucene-java/ScoresAsPercentages">drawbacks</a> and can't be done without collecting all matches and scores.
A 2005 paper, <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/craswell_sigir05.pdf">Relevance Weighting for Query Independent Evidence</a>, takes the opposite approach and engineers features in such a way that they look more like BM25 scores. In particular, one suggested function is called <code>satu</code>, which stands for "saturation", and is the function that is used to incorporate term frequency into BM25 scores. This paper has been influential to Elasticsearch 7.0 and lead to the development of two new fields: <code>rank_feature</code> and <code>rank_features</code>, as well as a new query called <code>rank_feature</code> that operates on these fields.
<h2>New <code>rank_feature</code> and <code>rank_features</code> fields</h2>The <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/rank-feature.html"><code>rank_feature</code></a> field works similarly to a regular <code>float</code> field from the outside, but indexes data in a way that allows Elasticsearch to run queries efficiently when it is used for ranking. <code>rank_feature</code> is typically used for measures of the relevance of a document such as popularity or authority, or irrelevance such as URL length in web search. <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/rank-features.html"><code>rank_features</code></a> works very similarly but is better suited for sparse features, such as a set of weighted tags or categories
As often, an example is worth a thousand words, so here is what you would do when indexing web pages in order for scores to be computed as <code>score = bm25_score + satu(pagerank)</code> thanks to the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-rank-feature-query.html"><code>rank_feature</code> query</a>:
<pre class="prettyprint">PUT test
{
 "mappings": {
 "properties": {
 "pagerank": {
 "type": "rank_feature"
 },
 "content": {
 "type": "text"
 },
 "url": {
 "type": "keyword"
 }
 }
 }
}
PUT test/_doc/1
{
 "url": "http://en.wikipedia.org/wiki/2016_Summer_Olympics",
 "content": "Rio 2016",
 "pagerank": 50
}
PUT test/_doc/2
{
 "url": "http://en.wikipedia.org/wiki/2016_Brazilian_Grand_Prix",
 "content": "Formula One motor race held on 13 November 2016 at the Autódromo José Carlos Pace in São Paulo, Brazil",
 "pagerank": 20
}
PUT test/_doc/3
{ 
 "url": "http://en.wikipedia.org/wiki/Deadpool_(film)",
 "content": "Deadpool is a 2016 American superhero film",
 "pagerank": 35
}
POST test/_refresh
GET test/_search
{
 "query": {
 "bool": {
 "must": [
 {
 "match": {
 "content": "2016"
 }
 }
 ],
 "should": [
 {
 "rank_feature": {
 "field": "pagerank"
 }
 }
 ]
 }
 }
}
</pre>Note that these fields can neither be searched or aggregated. If you want to be able to search or aggregate them as well, you would need to use <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/multi-fields.html">multi-fields</a> to map these fields both as a <code>float</code> and a <code>rank_feature</code> field. For instance the below index creation call maps the <code>pagerank</code> field as a <code>float</code> and <code>pagerank.feature</code> as a <code>rank_feature</code> field:
<pre class="prettyprint">PUT test
{
 "mappings": {
 "properties": {
 "pagerank": {
 "type": "float",
 "fields": {
 "feature": {
 "type": "rank_feature"
 }
 }
 }
 }
 }
}
</pre>The <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-rank-feature-query.html"><code>rank_feature</code> query</a> gives some options to change the function that is used to turn these features into scores: 3 different functions are available, which all have some tuning parameters. Unless you have a training set that you can use to tune these parameters, we recommend that you only use query boosts to tune the weight of your features compared to your main query and otherwise stick to the defaults which are the saturation function with parameters that are computed automatically from index statistics.
These fields provide little flexibility but come with a great benefit: they work with <a href="https://www.elastic.co/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand">top-k retrieval optimizations</a> that we also brought to Elasticsearch 7.0, which often helps retrieve the top matches of a query while only collecting a small subset of matches. As a consequence, you don't need to sacrifice performance entirely in order to apply one or two features on the whole result set. On the other hand if you need more flexibility but sacrificing performance is acceptable, a better way is to go with scripting.
<h2>Full flexibility with <code>script_score</code></h2><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-script-score-query.html">Script Score Query</a>, designed to replace Function Score Query, allows you to define a scoring formula for your documents through Painless scripting. Script Score Query comes with some predefined functions that can be used in combining relevance signals. Two of these functions are <code>saturation</code> and <code>sigmoid</code>. For example, the previous example with rank_feature can be approximated with the following example using a script_score query:
<pre class="prettyprint">GET /_search
{
 "query" : {
 "script_score" : {
 "query" : {
 "match": { "content": "2016"}
 },
 "script" : {
 "source" : "_score + saturation(doc['pagerank'].value, 50))"
 }
 }
 }
}
</pre>Here, we define a document rank as a combination of the document’s textual score with its pagerank value. The pagerank value is transformed beforehand through a saturation function to give as a value from 0 to 1.
The decay functions of the script_score query for <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-script-score-query.html#decay-functions">numeric</a>, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-script-score-query.html#_decay_functions_for_geo_fields">geo</a> or <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-script-score-query.html#_decay_functions_for_date_fields">date</a> fields may be used for the transformation of relevance signals of numeric, geo or date types. If your relevance signals have a form of <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/dense-vector.html">dense</a> or <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/sparse-vector.html">sparse</a> vector fields, soon it will be possible to use these fields in scoring as well as we are adding vector functions to the script_score query.
Using scripts you can also completely define your own scoring formula appropriate for your domain. For example, this <a href="https://www.elastic.co/blog/better-than-average-sort-by-best-rating-with-elasticsearch">blog about better result sorting</a> presents a script for ranking products based on users’ ratings for an online retail industry.
<h2>Conclusion</h2>While most of the work is still on the user side, we hope that these new tools are going to make improving relevance easier. We are especially excited that retrieving top matches doesn't require to collect all matches all the time, including when the score contains non-textual relevance signals, as this can result in significantly faster query results. We hope to bring similar tools in the near future for dynamic features that need to be computed on the fly such as recency or geo-distance.
Download the 7.0.0 beta and give <code>rank_feature</code>(s) and <code>script_score</code> a go, and let us know what you think. Become an <a href="https://www.elastic.co/blog/elastic-pioneer-program-7-0">Elastic Pioneer</a> and you’ll also have a chance to win some cool Elastic swag.

relevance-tuning-is-easier.jpg

relevance-tuning-thumb.jpg

Easier Relevance Tuning in Elasticsearch 7.0

Language Models in Elasticsearch

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Mayya Sharipova

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

제출하면 귀하가 <a href="/kr/legal/elastic-cloud-account-terms" target="_self">Elastic 서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 연락처로 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">선택한 커뮤니케이션</a>을 수신하는 데 동의한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/serverless-preview-terms" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

제출하면 <a href="/kr/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 서비스 약관</a>에 동의하시게 됩니다. 개인 데이터는 <a href="/kr/legal/privacy-statement">Elastic의 개인정보 취급방침</a>에 따라 처리됩니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/terms-of-use" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

이 페이지의 콘텐츠는 선택하신 언어로 제공되지 않습니다. Elastic은 다양한 언어로 콘텐츠를 제공하기 위해 최선을 다하고 있습니다.조금만 더 기다려주세요!

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

내 표준 시간대로 웨비나 시작 시간 확인하기

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

참여가 어려우신가요? 등록하시면 녹화 자료를 보내드리겠습니다. 이메일로 관련 콘텐츠도 받아보실 수 있습니다.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

You'll also receive an email with related content

이메일을 통해 관련 내용을 보내드리겠습니다.

Thank you for registering. We will send you a confirmation email soon.

등록해 주셔서 감사합니다. 곧 확인 이메일을 보내드리겠습니다.

Author

Articles by Mayya Sharipova

Elasticsearch에서 정렬 쿼리를 최적화하여 응답 속도 향상

Easier Relevance Tuning in Elasticsearch 7.0

Language Models in Elasticsearch

Sign up for Elastic Cloud free trial