elasticsearch-weekly-blog-banner.png

elasticsearch-weekly-blog-thumb.png

This Week in Elasticsearch and Apache Lucene - 2020-03-13

This Week in Elasticsearch - 2020-02-28

<html><head></head><body>
	Some time ago, we published a <a href="/blog/using-korean-analyzers">blog post</a> about which Korean analyzer you should use in Elasticsearch. The article presented three third-party plugins that you can install to enhance Korean support within your Elasticsearch cluster. We are excited to announce that as of Elasticsearch 6.4, we&#x2019;ll provide an official, best-in-class plugin called <a href="/guide/en/elasticsearch/plugins/6.4/analysis-nori.html">analysis-nori</a> to handle this popular language. In this blog post, I am going to describe the story behind Nori, the new Lucene module that is used by the plugin to handle Korean text analysis.
<h3>Segmenting Korean text</h3>Spaces are used in Korean to delimit what is called an eojeol, a sequence of morphemes agglutinated together to form a meaning. For instance &#xC120;&#xC0DD;&#xB2D8;&#xAED8;&#xC11C;&#xA0;is an eojeol made of three components, &#xC120;&#xC0DD; a noun that means teacher, &#xB2D8; an honorific suffix and &#xAED8;&#xC11C; the honorific form of the particles &#xC774; and &#xAC00; denoting the subject of a sentence. This eojeol means that an esteemed teacher is the subject of a sentence. Detecting these individual morphemes automatically is a difficult task that requires knowledge of the vocabulary and morphology of words in the language.&#xA0;
<h3>MeCab</h3>
	The number of rules and exceptions in the language makes the development of a rule-based system quite complex so most of the successful implementations of Korean morphological analyzers are done through probabilistic modeling. The 21st Century Sejong Project, started in 1998 by the Korean government, initiated the creation of large-scale Korean corpus. With the publication of this&#xA0;corpus over the years that follow it became easier to build probabilistic modeling of Korean morphology. 
Today almost all the available morphological analyzers for Korean found their origin in the 21st Century Sejong Project. The mecab-ko-dic is one of them, it uses MeCab (pronounced Mekabu): a popular open source morphological analysis engine to train a probabilistic model of the Korean morphology from parts of the corpus created by the 21st Century Sejong Project.

	MeCab was originally written for the segmentation of Japanese. It uses the conditional random fields to learn a bigram model from an annotated corpus of sentences where the cost of moving from one morpheme to another depends only on the previous state. It does not depend on the language which is why it can be used to analyze Japanese and Korean seamlessly, the only difference being the dictionary that is used. 
Lucene has a morphological analyzer for Japanese since version 3.6, it uses the <a href="https://github.com/taku910/mecab/tree/master/mecab-ipadic">IPADIC</a> dictionary also created with MeCab to segment and tag Japanese morphemes with their part of speech. 
Since both dictionaries were created with the same tool, it was tempting to try to reuse the Lucene&#x2019;s Japanese analyzer to handle this Korean dictionary. The advantage of such a method is that we could benefit from a robust analyzer optimized over the years in terms of memory consumption and throughput. This is how Nori, &quot;play&quot; in Korean,&#xA0;was born. It started by borrowing the code of the Japanese analyzer and evolved to a standalone morphological analyzer for Korean.
	The <a href="https://bitbucket.org/eunjeon/mecab-ko-dic/downloads/"> mecab-ko-dic dictionary</a> defines a model to analyze the korean language morphologically.
It is made of different files that expose the informations needed to perform the morphological analysis in MeCab. The total size on disk after decompression of 
	<a href="https://bitbucket.org/eunjeon/mecab-ko-dic/downloads/mecab-ko-dic-2.0.3-20170922.tar.gz"> the latest distribution</a> is 219MB. Considering that it is the textual form of the dictionary, one of the main challenges for the new Korean module was to provide a compressed form of this data that can be quickly loaded in memory for fast lookups. To achieve this, Nori translates the original dictionary into a binary format. The transformation is done offline, as a standalone process, and the resulting binary dictionary is added as a resource inside the module. This transformation is needed to diminish the size of the module: we don&#x2019;t want to distribute 200MB of dictionary in a Lucene&#x2019;s module. This conversion is also done to provide a format that is optimized for lookups. Morphological analysis requires one lookup in the dictionary for each character in the input, so we need efficient data structures. In the next section I&#x2019;ll describe how we use the original dictionary to build a smaller, optimized for lookups, binary version of the same data.
<h4>*.csv file</h4>
	The vocabulary is defined in a set of comma separated values file where an entry looks like this:
<pre class="prettyprint">&#xB3C4;&#xC11C;&#xAD00;,1781,3535,2110,NNG,*,T,&#xB3C4;&#xC11C;&#xAD00;,Compound,*,*,&#xB3C4;&#xC11C;/NNG/*+&#xAD00;/NNG/*
</pre>The first four columns are generic in the MeCab format. The first column is the surface form of the term, &#xB3C4;&#xC11C;&#xAD00; (doseogwan), a library in english. It is followed by the left context ID for this term, the right context ID and finally the cost of the term. MeCab uses a bigram language model so the left and right context IDs are used to compute the cost of moving from one state to another. The costs for each transition (left to right) are available in a file named matrix.def.
The fifth column is the part of speech for the term based on the Sejong corpus classification.

	You can find the full list and their meaning <a href="https://lucene.apache.org/core/7_4_0/analyzers-nori/org/apache/lucene/analysis/ko/POS.Tag.html">in this Enum POS doc</a>, NNG being a general noun. 
The extra columns on the right are features that are specific to Korean and the mecab-ko-dic format. In this example, the term is considered as a &#x201C;Compound&#x201D;, a noun made of other nouns and the decompounding is provided in the last column, &#xB3C4;&#xC11C;&#xAD00; (library) =&gt; &#xB3C4;&#xC11C; (book) + &#xAD00; (house).

	When Nori analyzes an input it needs to perform a lookup in this dictionary on every character in order to build the lattice of all possible segmentations. 
In order to achieve this efficiently we encode the full vocabulary in an 
	<a href="http://blog.mikemccandless.com/2010/12/using-finite-state-transducers-in.html">FST (finite state transducer)</a> where the transition labels are the Korean symbols of the dictionary terms and the output is an ID that links the surface form of the terms to the list of associated entries with their feature array.

	There are 11,172 possible symbols in Korean so the FST is very dense at the root and becomes sparse very quickly. It encodes the 811,757 terms included in the mecab-ko-dic with 171,397 nodes and 826,926 arcs in less than 5.4MB.

	Nori&#x2019;s feature array uses a custom binary format that encodes each entry of the original csv files using 9 bytes in average for a total size of 7MB. This compression was made possible by taking into account the particularity of the Korean language. For instance, nouns are invariable in Korean. So instead of saving the nouns that compose a specific compound, we encode the size in characters of each split. For the example above, the de-compound of &#xB3C4;&#xC11C;&#xAD00;, can be encoded with 2 for &#xB3C4;&#xC11C; and 1 for &#xAD00;. Since compounds are always made of nouns we can also omit the part of speech associated with each split to finally reduce 16 bytes into 2.
<h4>Matrix.def</h4>
	This file contains the connection costs between a right context ID and a left context ID.
These costs are computed by MeCab during the learning phase and they are used at analysis time to compute the cost of a possible segmentation.
The matrix is big. There are 2690 left context IDs and 3815 right context IDs so the full matrix contains 10,262,350 cells. The original file encodes the matrix with one triplet (right context ID, left context ID, cost) per line for a total of 139MB. To add this information in the binary dictionary we compress the matrix using variable length encoding and serialize the result in a file. This compression reduces the size to less than 12MB. However we need fast lookups in this matrix so when the module starts we load a fully decompressed version of the matrix in memory using 2 bytes per transition (costs are encoded using 16 bits) for a total of 20MB. This is quite a big resource to load in the java heap so this structure is loaded in a direct byte buffer outside of the heap.

	The final size of the binary dictionary is 24MB on disk. This is a good compression ratio considering that the binary format is optimized for lookups. Providing a lightweight dictionary is important since it allows the usage of the Lucene&#x2019;s Korean module in small deployment where there is a limited amount of RAM available.
<h3>Morphological analysis</h3>
	Now that we have a well-constructed binary dictionary that can be used to lookup efficiently any terms that appear in the mecab-ko-dic dictionary, we can analyze text using the Viterbi algorithm to find the most likely segmentation (called the Viterbi path) of any input written in Korean. The figure below shows the Viterbi lattice built from the sentence 21&#xC138;&#xAE30; &#xC138;&#xC885;&#xACC4;&#xD68D;. (the 21st century Sejong plan):&#xA0;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0ad9298344ee3463/5c57e077a2b68bd90b9fee82/viterbi_path.png" data-sys-asset-uid="blt0ad9298344ee3463" alt="viterbi_path.png" style="display: block; margin: auto;">

	 
	The Viterbi path in green shows the segmentation that Nori outputs:
<pre class="prettyprint">21 + &#xC138;&#xAE30; (century) + &#xC138;&#xC885; (Sejong) + &#xACC4;&#xD68D; (Plan)
</pre>
	Usually the algorithm to find the best segmentation uses three steps. It finds the next sentence boundary in the input first, then it builds the lattice of all possible path for this sentence and finally it computes the best path (segmentation) by applying the cost of each transition. These steps are applied until all sentences in the input are consumed. Applying these steps sequentially can be slow, so Nori uses a different approach. It uses a single step that is optimized for throughput. It consumes the input forward, character by character, and builds the Viterbi lattice on the fly. Costs are also computed on the fly in order to keep only the least costly path at each character boundary. For instance in the figure above, the path 21 + &#xC138; + &#xC138; is pruned as soon as we discover that 21 + &#xC138;&#xAE30; has a lower cost. The processing stops at each frontier (when a state has only one possible transition) or after 1024 characters (to avoid memory issues if every state has multiple transitions) to output the best segmentation of the current lattice and restart the processing on the next character window.
<h3>Benchmarks</h3>Now that we know how Nori is implemented let&#x2019;s see how it behaves in a real use case.
I used <a href="https://www.elastic.co/fr/blog/rally-1-0-0-released-benchmark-elasticsearch-like-we-do">Rally</a> to create a small benchmark to compare the indexing throughput of Nori against two popular community plugins: <a href="https://bitbucket.org/eunjeon/seunjeon">seunjeon</a> and <a href="https://github.com/HowookJeong/elasticsearch-analysis-arirang"> arirang </a>. The benchmark consists of indexing 413,985 documents extracted from the Korean Wikipedia in a single shard using 8 clients.
The Seunjeon plugin also uses the mecab-ko-dic and has several options to customize the output. I used the latest version (6.1.1.1, as of May 29, 2018) with the default analyzer provided in the plugin and tested with and without the compress option (-Dseunjeon.compress=true) on two Elasticsearch configurations, one with a heap size of 512m and one with 4G.
Arirang uses its own dictionary and does not provide any options so I used the default analyzer provided in the plugin.
The figure below shows the median throughput of the bulk indexing for both configurations:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt09f2221fafb3a46f/5c57e08122d96be10bcca373/indexing-throughput.png" data-sys-asset-uid="blt09f2221fafb3a46f" alt="indexing-throughput.png" style="display: block; margin: auto;">&#xA0;Nori and Arirang performed similarly with an indexing throughput of more than 3000 documents per second. However the Arirang plugin fails to index 10% of the corpus due to negative offsets set by the analyzer. The Seunjeon plugin has a peak at 400 documents per second in its non-compressed version, but it crashed several times with an out of memory error when run on the 512m node, which explains why the result is missing for this configuration. Using the compressed version fixed the issue on the 512m configuration resulting in an indexing throughput of 130 documents per second, almost 30 times slower than Arirang and Nori.
<h3>Conclusion</h3>In this article we presented Nori, a fast, lightweight Korean analyzer added to Lucene 7.4.0.
We&#x2019;re trying our best to enhance languages support in Lucene and Elasticsearch, and Nori is a good example of how we approach things. Natural language processing is a journey not a destination so stay tuned.&#xA0;We have more improvements coming soon and in the meantime we encourage you to learn more about this new plugin through the 
	<a href="/guide/en/elasticsearch/plugins/6.4/analysis-nori.html"> Elasticsearch documentation on Nori</a>, and <a href="/downloads/elasticsearch">download the latest version of Elasticsearch</a> to try it yourself!
</body></html>

nori_banner.jpg

meca-nori-analyzer-thumbnail.jpg

Nori: The Official Elasticsearch Plugin for Korean Language Analysis

Introducing Index Sorting in Elasticsearch 6.0! Users can enjoy improved query time performance by storing documents on disk in a specific order at index time.

In Elasticsearch 6.0 we’re introducing a new feature called Index Sorting. Users can now optimize Elasticsearch indexes to store documents on disk in a specific order. We’re very excited for Index Sorting, as it’s another useful tool in optimizing Elasticsearch performance! Through this article, we’ll dive into a number of areas:<ul><li>Lucene’s Index Sorting functionality</li><li>Examples where Index Sorting will improve query performance</li><li>Caveats to consider in using Index Sorting for time series data</li><li>Performance considerations</li></ul><h2>Index Sorting in Lucene</h2><h3>Lucene’s IndexSorter</h3>Many years ago, Lucene introduced an offline tool known as the <a href="https://issues.apache.org/jira/browse/LUCENE-2482">IndexSorter</a>. The IndexSorter copied a source index to a new destination index, and ordered the documents on disk based on a user specified order. At that time, because it was not possible to update the destination index directly, users of this feature had to re-build a sorted view every time new documents were added to the source index. The IndexSorter was the first attempt to provide a way to sort documents on disk, not at search time, but at index time.With index sorting, a new concept called “early termination” was introduced. Suppose for instance that you want to retrieve N documents sorted by date (date being a field in the index). If the index is sorted on disk by this date field it would be possible to “stop” the request after visiting the first N documents that match the query (since they are already in the order the user specified). This is what we call “early termination”. Early termination of a query can bring significant improvement to search response times, especially for sort-based queries, and led to the increased popularity of the IndexSorter tool among Lucene users. The static nature of the tool prevented its usage for indices with a lot of updates, which is why it was eventually replaced with a solution that allows incremental updates. Instead of doing a one-time sort of a static index, a new solution was proposed to sort documents at merge time.<h3>Lucene improvements</h3>Originally, Lucene indexed documents in the order they were received, and assigned each document an incremental (and internal) document id (assigned on a per segment basis). The first document indexed in a segment had a document id of 0, and so on. At search time, each segment is visited in document id order, to retrieve documents that match a user query. In order to retrieve the best N documents for a query, Lucene needs to visit every document matching the query across all segments. If the query matches millions of documents, retrieving only the best N would still require millions of documents to be visited.A Lucene index creates a new segment whenever a refresh is triggered. This new segment contains all the documents that were added after the last refresh. When the segment is flushed it becomes visible to the searcher and new documents can appear in search results. Because refreshes occur constantly, the number of segments can easily explode in an index. Segment merges happen in the background to limit the number of segments from growing too large. Merges are triggered based on a policy that selects segments eligible for merging, the selected segments are then merged in a new segment that replaces the old segments. By default, the segment merge process copies documents from different segments to a new segment based on their internal document ids. In order to replace the static tool (the IndexSorter mentioned above), a <a href="https://issues.apache.org/jira/browse/LUCENE-4752">new merge policy was introduced</a> to allow index sorting for dynamic indices that reorders documents during the merge process based on a configurable order (the value of a field for instance). This new design was a huge step in the right direction, and allowed an index to be sorted on the fly and to use this information on a per-segment basis. Some segments are sorted (segments created by a merge) and some are not (the newly flushed segments). At merge time, the unsorted segments are first “sorted” and then merged with other sorted segments.This merge policy that lived in a module was then moved to a top-level option on the IndexWriterConfig to make index sorting a <a href="https://issues.apache.org/jira/browse/LUCENE-6766">first class citizen</a> in Lucene.Though some benchmarks showed that the cost of sorting at merge time can divide the total throughput of indexation by a factor of 2:<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta924260b54220c12/5fa9877924612d7400da3968/blog-index-sort-6-1.png" width="624" height="212" alt="Screen Shot 2017-08-10 at 10.16.39 AM.png" style="border-width: initial;border-style: none;transform: rotate(0rad);"/><a href="https://home.apache.org/~mikemccand/lucenebench/sparseResults.html#index_throughput">https://home.apache.org/~mikemccand/lucenebench/sp...</a>The reason for the reduction in indexing performance is simple: re-sorting segments has a cost, causing merge time and memory consumption for these indices to increase by a large factor.Since re-sorting multiple segments at a time is costly, we decided to sort documents earlier in the indexation process. Instead of waiting for merge times to sort multiple segments, we’ve moved the sorting to flush time (when the segments are first created): <a href="https://issues.apache.org/jira/browse/LUCENE-7579">LUCENE-7579.</a> If all segments are already sorted, merging can occur using a simple merge-sort strategy, which is much faster. This new strategy was first introduced in Lucene 6.5 and increased the throughput benchmarks by almost 65% (see annotation V).As you can see in this story index sorting had a lot of history in Lucene but until now it was not available in Elasticsearch. Thanks to all these optimizations, we’ve decided to unlock this feature in Elasticsearch 6.0 and we’re really excited to show how this feature can help you to optimize your use case with this new release!<h2>Index Sorting in Action</h2><h3>Early termination of search queries</h3>It’s very common in applications to query for the top X results, sorted by value Y (top player scores, new users, latest events, etc.). In most cases, Elasticsearch will not have enough information to quickly gather the first X results and sort them until the entire data set has been examined. Doc values make this process more efficient, however, in the cases where the dataset is extremely large, a lot more values will be examined and compared than are needed by the user.With the introduction of index sorting in Elasticsearch 6.0, we can now specify the ordering of documents on disk, allowing Elasticsearch to short circuit and return queries more efficiently. For instance, if we’re creating a leaderboard for a video game company to track the top 3 player scores (and we have a very large number of players!), we can instruct Elasticsearch to store documents in the order of their player score, allowing us to compute the leaderboard much more efficiently.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb218fa68545af041/5fa98780ffb61e50a0faacfb/blog-index-sort-6-2.jpg" width="624" height="187" alt="leaderboard.jpg" style="border-width: initial;border-style: none;transform: rotate(0rad);"/> <pre class="prettyprint">// Get the top 3 player scores (based on the number of points) GET scores/score/_search { "size": 3, "sort": [ { "points": "desc" } ] } </pre>Depending on the version of Elasticsearch, and on usage of index sorting, we can store the documents on disk very efficiently for the query above:<img src="https://api.contentstack.io/v2/uploads/599b0598949407fa18d59c85/download?uid=blt3a1f0c3350489703" data-sys-asset-uid="blt3a1f0c3350489703" alt="first_diag.png"/> The query above will still need to return a count for the number of results (and requires a little extra work). We can remove this requirement with the new option "track_total_hits"&nbsp;set to false:<pre class="prettyprint">// Get the top 3 player scores (based on the number of points) GET scores/score/_search { "size": 3, "track_total_hits" : false, "sort": [ { "points": "desc" } ] } </pre>We now have a very efficient leaderboard query for top player scores, using a sorted index.<h3></h3><h3>Specifying an index sorting order in Elasticsearch 6.0</h3>To continue with our example above (creating a leaderboard of top player scores), we will need to tell Elasticsearch how to order the documents on disk. We can do this by providing a definition in the settings for the index:<pre class="prettyprint">PUT scores { "settings" : { "index" : { "sort.field" : "points", "sort.order" : "desc" } }, "mappings": { "score": { "properties": { "points": { "type": "long" }, "playerid": { "type": "keyword" }, "game" : { "type" : "keyword" } } } } } </pre>The example above will sort documents on disk by the points field (in descending order). This is helpful for the simple query above (for top 3 player scores).&nbsp;<h3>Grouping documents within an index by similar structure</h3>There are many advantages to storing documents sorted by a similar type. For instance, if there is an index named “scores”, some scores may come from the game “Joust”, and include specific fields such as “top-speed” and “farthest-jump”, a score for a different game, such as “Dragon’s Lair” may include fields for “sword-fight-score” and “goblins-killed”:<pre class="prettyprint">// Score for the game "Joust" { "game" : "joust", "playerid" : "1234", "top-speed" : 212, "farthest-jump" : 49 } </pre><pre class="prettyprint">// Score for the game "Dragon’s Lair" { "game" : "dragons-lair", "playerid" : "5678", "sword-fight-score" : 89, "goblins-killed" : 3 } </pre>Storing the documents on disk sorted by game will help place similar documents (with similar field names) together. The advantages to this are query speed (although it’s important to remember this really depends on the query) and compression. Storing similar fields closer together may lead to better compression, and Elasticsearch (and in turn Lucene) is able to store the deltas more efficiently:<pre class="prettyprint">PUT scores { "settings" : { "index" : { "sort.field" : "game", "sort.order" : "desc" } } } </pre><h3>More efficient AND conjunctions</h3>Using index sorting to locate documents on disk in a specific order can also <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/index-modules-index-sorting-conjunctions.html">improve AND conjunctions</a>, complex queries with many conditions.Let’s continue with our video game example, when a player joins a game, they must be paired up with another player in the same region, skill level, and course. A sample query to find similar players for starting a new match may look similar to the following (get 10 players within the "EU"&nbsp;region, playing "Dragon’s Lair", with a skill rating of 9, and at the "Castle" map):<pre class="prettyprint">GET players/player/_search { "size": 3, "track_total_hits" : false, "query" : { "bool" : { "filter" : [ { "term" : { "region" : "eu" } }, { "term" : { "game" : "dragons-lair" } }, { "term" : { "skill-rating" : 9 } }, { "term" : { "map" : "castle" } } ] } } } </pre>Let's look at how the Elasticsearch may gather the results needed for the query:<img src="https://api.contentstack.io/v2/uploads/599616921eff86ee22016a11/download?uid=blt9aa0104633a60b1f" data-sys-asset-uid="blt9aa0104633a60b1f" alt="new_query_without.png"/>Now, let's specify the ordering of the documents on disk to improve our query above:<pre class="prettyprint">PUT players { "settings" : { "index" : { "sort.field" : ["region", "game", "skill-rating", "map"], "sort.order" : ["asc", "asc", "asc", "asc"] } }, "mappings": { "player": { "properties": { "playerid": { "type": "keyword" }, "region": { "type": "keyword" }, "skill-rating" : { "type" : "integer" }, "game" : { "type" : "keyword" }, "map" : { "type" : "keyword" } } } } } </pre>We can now see the documents are placed closer together:<img src="https://api.contentstack.io/v2/uploads/5996174b2d8186e8071aa7e5/download?uid=blt79a4bd69300b6a4c" data-sys-asset-uid="blt79a4bd69300b6a4c" alt="new_query_with2.png"/>By using a sorted index, we can locate the documents with&nbsp;similar field&nbsp;values closer together, making our query to find players for a given match more efficient.<h3>When index sorting isn't a good fit</h3>Storing sorted values on disk requires a lot more work at index time from Elasticsearch than storing unsorted values. In some cases the performance overhead of index sorting can decrease write performance by as much as 40-50%. For this reason it is very important to determine if the application should be optimized for query performance or write performance. Optimizing an application for write performance (and taking the hit on query performance) will most likely mean index sorting is not a good option.You can check the throughput for indexation with and without index sorting. As mentioned above, the performance hit will vary widely and depend on your use case. For example, the geonames Elasticsearch benchmark shows a very small performance hit for Index Sorting (the blue line labeled “Append Sorted”):<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltabbe6b882b3ff86d/5fa9878142256d5ffdf418a3/blog-index-sort-6-3.png" width="624" height="340" alt="Screen Shot 2017-08-10 at 1.11.12 PM.png" style="width: 624;height: 340;"/> <a href="https://elasticsearch-benchmarks.elastic.co/index.html#tracks/geonames/nightly/30d">https://elasticsearch-benchmarks.elastic.co/index....</a>Alternatively, the “NYC Taxis” benchmark shows a large drop in indexing performance with index sorting:<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte19365eb5b04302c/5fa9877a09ed4c6f277eb5f6/blog-index-sort-6-4.png" width="624" height="339" alt="Screen Shot 2017-08-10 at 1.12.42 PM.png" style="width: 624;height: 339;"/> <a href="https://elasticsearch-benchmarks.elastic.co/index.html#tracks/nyc-taxis/nightly/30d">https://elasticsearch-benchmarks.elastic.co/index....</a>In system design, there are tradeoffs at almost every level, with index sorting, the tradeoff we’re considering is less efficient writes (as the document must be sorted) for faster queries (in specific scenarios) vs more efficient writes and slower queries (as the results must be sorted at query time).Similar to any new feature, it is very important to test index sorting with your specific use case and dataset.<h2>We’re not finished</h2>This is only the beginning, we’ll continue to improve index sorting for a larger range of use cases!Hopefully this article gives you a good overview of index sorting as a great new tool to consider in your Elasticsearch 6.0 toolbox. In addition to this blog post, the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/index-modules-index-sorting.html">documentation on Index Sorting</a> can be a great resource to bookmark. If you want to try out the new index sorting functionality, download <a href="https://www.elastic.co/blog/elasticsearch-6-0-0-beta1-released">6.0.0-beta1</a> and <a href="https://www.elastic.co/blog/elastic-pioneer-program-6-0">become a pioneer</a>!

sorting-leaderboard.png

index-sorting-img.png

Introducing Index Sorting in Elasticsearch 6.0

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Software Engineer

Jim Ferenczi

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/elastic-cloud-account-terms" target="_self">服务条款</a>，并且您同意通过您提供的上述联系方式接收<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">所选的通信</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/serverless-preview-terms" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

提交即表示您同意 <a href="/cn/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 服务条款</a>。我们将会按照 <a href="/cn/legal/privacy-statement">Elastic 的隐私声明</a>处理您的个人数据。

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/terms-of-use" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

本页内容尚不支持所选语言。Elastic 正在不断努力，以实现对多种语言内容的支持。感谢您在此期间给予的耐心与陪伴！

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

查看本次网络研讨会在我的时区的开始时间

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

无法实时参加？注册后我们会给您发送录像。您还将收到相关内容的电子邮件。

Author

Articles by Jim Ferenczi

Software Engineer, Elastic

This Week in Elasticsearch and Apache Lucene - 2020-03-13

This Week in Elasticsearch - 2020-02-28

Nori: The Official Elasticsearch Plugin for Korean Language Analysis

Introducing Index Sorting in Elasticsearch 6.0