Classification has been added to machine learning in Elasticsearch 7.6. We'll make Elasticsearch learn the characteristics of the movies we liked and disliked, and then ask it to predict whether we’ll like or dislike a movie.

<a href="https://www.elastic.co/guide/en/machine-learning/7.6/dfa-classification.html">Classification</a> has been added to <a href="/what-is/elasticsearch-machine-learning">machine learning</a> in Elasticsearch 7.6. Unlike <a href="https://www.elastic.co/guide/en/machine-learning/7.6/xpack-ml.html">anomaly detection</a>, classification is supervised machine learning to predict the class or category of a given data point in a dataset.
There are two types of classification: multi-class and binary classification. For example, multi-class classification predicts which category a movie belongs to, such as action, animation, or comedy. Binary classification predicts whether I like this movie or not. We released the binary classification feature in Elastic Stack 7.6.1 so this blog post will focus on building a model using that capability.
We'll make Elasticsearch learn the characteristics of the movies we liked and disliked, and then ask it to predict whether we’ll like or dislike a movie based on its characteristics.
 <h2>Create deployment</h2>To create a deployment, you can start a free 14-day <a href="/elasticsearch/service">free trial of Elasticsearch Service</a>.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3520c1f202109bf6/5e93759207387e0c9bfffcd0/movie-classification-ml-deployment-blog.png" data-sys-asset-uid="blt3520c1f202109bf6" alt="Creating a cloud deployment" style="display: block; margin: auto;">
<figcaption>Fig. 1 Create deployment.</figcaption>Once your trial is started, go to <a href="https://cloud.elastic.co">Elastic Cloud</a> and start a new deployment. We can click the “Customize deployment” button at the bottom of the screen to activate the machine learning node as shown below, then click the “Create deployment” button to create the deployment. You can refer to the <a href="/guide/en/cloud/current/ec-configure-settings.html">Elastic Cloud documentation</a>&nbsp;for the deployment configuration.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt42315dbd5936e043/5e9375da0b43937636a92327/movie-classification-ml-ml-mode-blog.png" data-sys-asset-uid="blt42315dbd5936e043" alt="Enabling ML mode" style="display: block; margin: auto;">
<figcaption>Fig. 2 Enable machine learning node.</figcaption>You can see the password while the deployment is being created.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt694e86d6b5e7e7da/5e9375f99517590d79790792/movie-classification-ml-password-blog.png" data-sys-asset-uid="blt694e86d6b5e7e7da" alt="Cloud password" style="display: block; margin: auto;">
<figcaption>Fig. 3 Password</figcaption>You can see the Cloud ID when the deployment creation completes.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt34618919d890f19c/5e93761657998f3cbe194515/movie-classification-ml-cloud-id-blog.png" data-sys-asset-uid="blt34618919d890f19c" alt="Cloud ID" style="display: block; margin: auto;">
<figcaption>Fig. 4 Cloud ID</figcaption>
<h2>Prepare data</h2>We can download information such as the film title, year of production, country, and genre as an Excel file from the <a href="http://www.kobis.or.kr/kobis/business/main/main.do">Korean Film Council website</a> (in Korean). We can add the “like” field to show whether we liked it or not (1 if you like, 0 if not), and save it as a csv file (movies.csv). Elasticsearch can learn this data and create a model, and then create a system that predicts whether we will like or not based on the title, year of production, country of production, director, and company of a new movie.
<pre class="prettyprint">title,year,country,genre,like,director,company 
Memento,2000,USA,thriller,0,Christopher Nolan,
Harry Potter And The Prisoner Of Azkaban,2004,USA,drama,1,Alfonso Cuarón,Warner Bros. 
…
</pre>If we put the field name in the first line, we can ingest into Elasticsearch using the following logstash.conf file. We can set the “cloud_id” field to the Cloud ID from Fig. 4 to, and the “cloud_auth” field to the password from Fig. 3 in the format of “elastic:&lt;Password&gt;”.
We should add "pipeline.workers: 1" in config/logstash.yml because we have "autodetect_column_names =&gt; true" in the <a href="/guide/en/logstash/7.6/plugins-filters-csv.html#plugins-filters-csv-autodetect_column_names">csv filter</a>.
<pre class="prettyprint">input {
 file {
 path =&gt; "/home/kiju/github/7/logstash-7.6.1/movies.csv"
 sincedb_path =&gt; "/dev/null"
 start_position =&gt; "beginning"
 }
}

filter {
 csv {
 autodetect_column_names =&gt; true
 }
}

output {
 elasticsearch {
 cloud_id =&gt; "movies:dXMtY2VudHJhbDEuZ2NwLmNsb3VkLmVzLmlvJDc4ZWNkNmJmODMyNjRlYTQ5ZDkwNTc4Njk5YjFmZDE5JGE0ODg3ZmM3ODk5MTRhZjlhODIyNTdmMDgyYmNlZWI2"
 cloud_auth =&gt; "elastic:&lt;Password&gt;"
 index =&gt; "movie2"
 }
}
</pre>From Kibana Dev Tools, we can define the mapping in Elasticsearch as below:
<pre class="prettyprint">PUT movie2
{
 "mappings": {
 "properties": {
 "title": {
 "type": "text"
 },
 "year": {
 "type": "integer"
 },
 "country": {
 "type": "keyword"
 },
 "genre": {
 "type": "keyword"
 },
 "like": {
 "type": "integer"
 },
 "director": {
 "type": "keyword"
 },
 "company": {
 "type": "keyword"
 }
 }
 }
}
</pre>And then run Logstash as below:
<pre class="prettyprint">$ bin/logstash -f logstash.conf
</pre>We can see if the data is correctly ingested in Elasticsearch.
<pre class="prettyprint">GET movie2/_search
=&gt;
{
...
 "hits" : {
 "total" : {
 "value" : 211,
 "relation" : "eq"
 },
 "max_score" : 1.0,
 "hits" : [
 {
 "_index" : "movie2",
 "_type" : "_doc",
 "_id" : "BCqX2HABaohLi7nef_QH",
 "_score" : 1.0,
 "_source" : {
 "@timestamp" : "2020-03-14T10:29:31.761Z",
 "genre" : "thriller",
 "title" : "Memento",
 ...
 }
 },
 …
</pre><h2>Classification</h2>Now we can run classification from Kibana to generate a model to predict whether we’ll like a movie or not.
We can select “Data Frame Analysis” from the “Machine Learning” menu and click the “Create analytics job” button as shown below. Select “classification” as the job type, and select “like” as the predicted target for the dependent variable. In the excluded fields, we can list fields that are not related to the characteristics of the movies, then click the “Create” button and then the “Start” button to run supervised learning.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4e2adb13530934ee/5e937655af360229b410cfb0/movie-classification-ml-classification-blog.png" data-sys-asset-uid="blt4e2adb13530934ee" alt="Start supervised learning" style="display: block; margin: auto;">
<figcaption>Fig. 5 Create analytics job</figcaption>We can click the “Close” button to close the sub-window and use inference to predict whether we’ll like or dislike a movie once the job progress reaches 100%.
 <h2>Inference</h2>First, we can find the ID of the model created as a result of mjob3 execution using the “GET _ml/inference/_all” command. model_id consists of the job name and the timestamp.
<pre class="prettyprint">GET _ml/inference/_all
=&gt;
{
...
 {
 "model_id" : "mjob3-1584187672850",
 "created_by" : "_xpack",
 "version" : "7.6.1",
 "description" : "",
 "create_time" : 1584187672850,
 "tags" : [
 "mjob1"
 ],
…
</pre>Next, we can run the inference processor in an ingest pipeline. If you provide information about the movie in question as a document to the simulate pipeline API, you can see the predicted_value. If it is 1, you are likely to like it.
<pre class="prettyprint">POST _ingest/pipeline/_simulate
{
 "pipeline": {
 "processors": [
 {
 "inference": {
 "model_id": "mjob3-1584187672850",
 "inference_config": {
 "classification": {}
 },
 "field_mappings": {}
 }
 }
 ]
 },
 "docs": [
 {
 "_source": {
 "title": "Cloud Bread - The Day The Laundry Flew Off In The Wind",
 "year": 2010,
 "country": "South Korea",
 "genre": "animation",
 "director": "Jay woon Jang"
 }
 },
 {
 "_source": {
 "country": "South Korea",
 "year": "'2008",
 "director": "Han-min Kim",
 "title": "Handphone",
 "genre": "thriller"
 }
 }
 ]
}
=&gt;
{
 "docs" : [
 {
 "doc" : {
 "_index" : "_index",
 "_type" : "_doc",
 "_id" : "_id",
 "_source" : {
 "country" : "South Korea",
 "year" : 2010,
 "director" : "Jay woon Jang",
 "genre" : "animation",
 "title" : "Cloud Bread - The Day The Laundry Flew Off In The Wind",
 "ml" : {
 "inference" : {
 "predicted_value" : "1",
 "model_id" : "mjob3-1584187672850"
 }
...
 },
 {
 "doc" : {
 "_index" : "_index",
 "_type" : "_doc",
 "_id" : "_id",
 "_source" : {
 "country" : "South Korea",
 "year" : "'2008",
 "director" : "Han-min Kim",
 "genre" : "thriller",
 "title" : "Handphone",
 "ml" : {
 "inference" : {
 "predicted_value" : "0",
 "model_id" : "mjob3-1584187672850"
 }
...
 ]
}
</pre><h2>Outcome</h2>After doing all this, the model says that I am likely to enjoy “Cloud Bread - The Day The Laundry Flew Off In The Wind” and not likely to enjoy "Handphone". Since schools are closed due to COVID-19, I watched “Cloud Bread - The Day The Laundry Flew Off In The Wind” with my kids and really enjoyed it.
 <h2>Trying it for yourself</h2>So that's how you can&nbsp;predict&nbsp;whether you’ll like a movie you haven't seen with the help of&nbsp;<a href="/guide/en/machine-learning/7.6/dfa-classification.html">classification</a>. This method can also be used to predict loan risk, detect defects in a car part, or detect cancer in a DNA sequence. I encourage to to try using classification in your own data sets, either <a href="/downloads">locally</a> or in <a href="/elasticsearch/service">Elastic Cloud</a>.For reference, the data file (in Korean) and the configuration files I used for testing are available in <a href="https://github.com/kiju98/ml/raw/master/logstash.zip">my GitHub repo</a>. <a href="https://github.com/kiju98/ml/raw/master/logstash.zip"></a>

blog-banner-bank-of-tvs.jpg

blog-thumb-bank-of-tvs.jpg

Curate your movie queue using classification-based supervised learning in Elasticsearch

Using classification-based supervised learning in Elasticsearch to curate your movie queue

Everything you need to know about Nori, the official Elasticsearch plugin for Korean language analysis added in 6.4.

<html><head></head><body>
	Some time ago, we published a <a href="/blog/using-korean-analyzers">blog post</a> about which Korean analyzer you should use in Elasticsearch. The article presented three third-party plugins that you can install to enhance Korean support within your Elasticsearch cluster. We are excited to announce that as of Elasticsearch 6.4, we&#x2019;ll provide an official, best-in-class plugin called <a href="/guide/en/elasticsearch/plugins/6.4/analysis-nori.html">analysis-nori</a> to handle this popular language. In this blog post, I am going to describe the story behind Nori, the new Lucene module that is used by the plugin to handle Korean text analysis.
<h3>Segmenting Korean text</h3>Spaces are used in Korean to delimit what is called an eojeol, a sequence of morphemes agglutinated together to form a meaning. For instance &#xC120;&#xC0DD;&#xB2D8;&#xAED8;&#xC11C;&#xA0;is an eojeol made of three components, &#xC120;&#xC0DD; a noun that means teacher, &#xB2D8; an honorific suffix and &#xAED8;&#xC11C; the honorific form of the particles &#xC774; and &#xAC00; denoting the subject of a sentence. This eojeol means that an esteemed teacher is the subject of a sentence. Detecting these individual morphemes automatically is a difficult task that requires knowledge of the vocabulary and morphology of words in the language.&#xA0;
<h3>MeCab</h3>
	The number of rules and exceptions in the language makes the development of a rule-based system quite complex so most of the successful implementations of Korean morphological analyzers are done through probabilistic modeling. The 21st Century Sejong Project, started in 1998 by the Korean government, initiated the creation of large-scale Korean corpus. With the publication of this&#xA0;corpus over the years that follow it became easier to build probabilistic modeling of Korean morphology. 
Today almost all the available morphological analyzers for Korean found their origin in the 21st Century Sejong Project. The mecab-ko-dic is one of them, it uses MeCab (pronounced Mekabu): a popular open source morphological analysis engine to train a probabilistic model of the Korean morphology from parts of the corpus created by the 21st Century Sejong Project.

	MeCab was originally written for the segmentation of Japanese. It uses the conditional random fields to learn a bigram model from an annotated corpus of sentences where the cost of moving from one morpheme to another depends only on the previous state. It does not depend on the language which is why it can be used to analyze Japanese and Korean seamlessly, the only difference being the dictionary that is used. 
Lucene has a morphological analyzer for Japanese since version 3.6, it uses the <a href="https://github.com/taku910/mecab/tree/master/mecab-ipadic">IPADIC</a> dictionary also created with MeCab to segment and tag Japanese morphemes with their part of speech. 
Since both dictionaries were created with the same tool, it was tempting to try to reuse the Lucene&#x2019;s Japanese analyzer to handle this Korean dictionary. The advantage of such a method is that we could benefit from a robust analyzer optimized over the years in terms of memory consumption and throughput. This is how Nori, &quot;play&quot; in Korean,&#xA0;was born. It started by borrowing the code of the Japanese analyzer and evolved to a standalone morphological analyzer for Korean.
	The <a href="https://bitbucket.org/eunjeon/mecab-ko-dic/downloads/"> mecab-ko-dic dictionary</a> defines a model to analyze the korean language morphologically.
It is made of different files that expose the informations needed to perform the morphological analysis in MeCab. The total size on disk after decompression of 
	<a href="https://bitbucket.org/eunjeon/mecab-ko-dic/downloads/mecab-ko-dic-2.0.3-20170922.tar.gz"> the latest distribution</a> is 219MB. Considering that it is the textual form of the dictionary, one of the main challenges for the new Korean module was to provide a compressed form of this data that can be quickly loaded in memory for fast lookups. To achieve this, Nori translates the original dictionary into a binary format. The transformation is done offline, as a standalone process, and the resulting binary dictionary is added as a resource inside the module. This transformation is needed to diminish the size of the module: we don&#x2019;t want to distribute 200MB of dictionary in a Lucene&#x2019;s module. This conversion is also done to provide a format that is optimized for lookups. Morphological analysis requires one lookup in the dictionary for each character in the input, so we need efficient data structures. In the next section I&#x2019;ll describe how we use the original dictionary to build a smaller, optimized for lookups, binary version of the same data.
<h4>*.csv file</h4>
	The vocabulary is defined in a set of comma separated values file where an entry looks like this:
<pre class="prettyprint">&#xB3C4;&#xC11C;&#xAD00;,1781,3535,2110,NNG,*,T,&#xB3C4;&#xC11C;&#xAD00;,Compound,*,*,&#xB3C4;&#xC11C;/NNG/*+&#xAD00;/NNG/*
</pre>The first four columns are generic in the MeCab format. The first column is the surface form of the term, &#xB3C4;&#xC11C;&#xAD00; (doseogwan), a library in english. It is followed by the left context ID for this term, the right context ID and finally the cost of the term. MeCab uses a bigram language model so the left and right context IDs are used to compute the cost of moving from one state to another. The costs for each transition (left to right) are available in a file named matrix.def.
The fifth column is the part of speech for the term based on the Sejong corpus classification.

	You can find the full list and their meaning <a href="https://lucene.apache.org/core/7_4_0/analyzers-nori/org/apache/lucene/analysis/ko/POS.Tag.html">in this Enum POS doc</a>, NNG being a general noun. 
The extra columns on the right are features that are specific to Korean and the mecab-ko-dic format. In this example, the term is considered as a &#x201C;Compound&#x201D;, a noun made of other nouns and the decompounding is provided in the last column, &#xB3C4;&#xC11C;&#xAD00; (library) =&gt; &#xB3C4;&#xC11C; (book) + &#xAD00; (house).

	When Nori analyzes an input it needs to perform a lookup in this dictionary on every character in order to build the lattice of all possible segmentations. 
In order to achieve this efficiently we encode the full vocabulary in an 
	<a href="http://blog.mikemccandless.com/2010/12/using-finite-state-transducers-in.html">FST (finite state transducer)</a> where the transition labels are the Korean symbols of the dictionary terms and the output is an ID that links the surface form of the terms to the list of associated entries with their feature array.

	There are 11,172 possible symbols in Korean so the FST is very dense at the root and becomes sparse very quickly. It encodes the 811,757 terms included in the mecab-ko-dic with 171,397 nodes and 826,926 arcs in less than 5.4MB.

	Nori&#x2019;s feature array uses a custom binary format that encodes each entry of the original csv files using 9 bytes in average for a total size of 7MB. This compression was made possible by taking into account the particularity of the Korean language. For instance, nouns are invariable in Korean. So instead of saving the nouns that compose a specific compound, we encode the size in characters of each split. For the example above, the de-compound of &#xB3C4;&#xC11C;&#xAD00;, can be encoded with 2 for &#xB3C4;&#xC11C; and 1 for &#xAD00;. Since compounds are always made of nouns we can also omit the part of speech associated with each split to finally reduce 16 bytes into 2.
<h4>Matrix.def</h4>
	This file contains the connection costs between a right context ID and a left context ID.
These costs are computed by MeCab during the learning phase and they are used at analysis time to compute the cost of a possible segmentation.
The matrix is big. There are 2690 left context IDs and 3815 right context IDs so the full matrix contains 10,262,350 cells. The original file encodes the matrix with one triplet (right context ID, left context ID, cost) per line for a total of 139MB. To add this information in the binary dictionary we compress the matrix using variable length encoding and serialize the result in a file. This compression reduces the size to less than 12MB. However we need fast lookups in this matrix so when the module starts we load a fully decompressed version of the matrix in memory using 2 bytes per transition (costs are encoded using 16 bits) for a total of 20MB. This is quite a big resource to load in the java heap so this structure is loaded in a direct byte buffer outside of the heap.

	The final size of the binary dictionary is 24MB on disk. This is a good compression ratio considering that the binary format is optimized for lookups. Providing a lightweight dictionary is important since it allows the usage of the Lucene&#x2019;s Korean module in small deployment where there is a limited amount of RAM available.
<h3>Morphological analysis</h3>
	Now that we have a well-constructed binary dictionary that can be used to lookup efficiently any terms that appear in the mecab-ko-dic dictionary, we can analyze text using the Viterbi algorithm to find the most likely segmentation (called the Viterbi path) of any input written in Korean. The figure below shows the Viterbi lattice built from the sentence 21&#xC138;&#xAE30; &#xC138;&#xC885;&#xACC4;&#xD68D;. (the 21st century Sejong plan):&#xA0;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0ad9298344ee3463/5c57e077a2b68bd90b9fee82/viterbi_path.png" data-sys-asset-uid="blt0ad9298344ee3463" alt="viterbi_path.png" style="display: block; margin: auto;">

	 
	The Viterbi path in green shows the segmentation that Nori outputs:
<pre class="prettyprint">21 + &#xC138;&#xAE30; (century) + &#xC138;&#xC885; (Sejong) + &#xACC4;&#xD68D; (Plan)
</pre>
	Usually the algorithm to find the best segmentation uses three steps. It finds the next sentence boundary in the input first, then it builds the lattice of all possible path for this sentence and finally it computes the best path (segmentation) by applying the cost of each transition. These steps are applied until all sentences in the input are consumed. Applying these steps sequentially can be slow, so Nori uses a different approach. It uses a single step that is optimized for throughput. It consumes the input forward, character by character, and builds the Viterbi lattice on the fly. Costs are also computed on the fly in order to keep only the least costly path at each character boundary. For instance in the figure above, the path 21 + &#xC138; + &#xC138; is pruned as soon as we discover that 21 + &#xC138;&#xAE30; has a lower cost. The processing stops at each frontier (when a state has only one possible transition) or after 1024 characters (to avoid memory issues if every state has multiple transitions) to output the best segmentation of the current lattice and restart the processing on the next character window.
<h3>Benchmarks</h3>Now that we know how Nori is implemented let&#x2019;s see how it behaves in a real use case.
I used <a href="https://www.elastic.co/fr/blog/rally-1-0-0-released-benchmark-elasticsearch-like-we-do">Rally</a> to create a small benchmark to compare the indexing throughput of Nori against two popular community plugins: <a href="https://bitbucket.org/eunjeon/seunjeon">seunjeon</a> and <a href="https://github.com/HowookJeong/elasticsearch-analysis-arirang"> arirang </a>. The benchmark consists of indexing 413,985 documents extracted from the Korean Wikipedia in a single shard using 8 clients.
The Seunjeon plugin also uses the mecab-ko-dic and has several options to customize the output. I used the latest version (6.1.1.1, as of May 29, 2018) with the default analyzer provided in the plugin and tested with and without the compress option (-Dseunjeon.compress=true) on two Elasticsearch configurations, one with a heap size of 512m and one with 4G.
Arirang uses its own dictionary and does not provide any options so I used the default analyzer provided in the plugin.
The figure below shows the median throughput of the bulk indexing for both configurations:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt09f2221fafb3a46f/5c57e08122d96be10bcca373/indexing-throughput.png" data-sys-asset-uid="blt09f2221fafb3a46f" alt="indexing-throughput.png" style="display: block; margin: auto;">&#xA0;Nori and Arirang performed similarly with an indexing throughput of more than 3000 documents per second. However the Arirang plugin fails to index 10% of the corpus due to negative offsets set by the analyzer. The Seunjeon plugin has a peak at 400 documents per second in its non-compressed version, but it crashed several times with an out of memory error when run on the 512m node, which explains why the result is missing for this configuration. Using the compressed version fixed the issue on the 512m configuration resulting in an indexing throughput of 130 documents per second, almost 30 times slower than Arirang and Nori.
<h3>Conclusion</h3>In this article we presented Nori, a fast, lightweight Korean analyzer added to Lucene 7.4.0.
We&#x2019;re trying our best to enhance languages support in Lucene and Elasticsearch, and Nori is a good example of how we approach things. Natural language processing is a journey not a destination so stay tuned.&#xA0;We have more improvements coming soon and in the meantime we encourage you to learn more about this new plugin through the 
	<a href="/guide/en/elasticsearch/plugins/6.4/analysis-nori.html"> Elasticsearch documentation on Nori</a>, and <a href="/downloads/elasticsearch">download the latest version of Elasticsearch</a> to try it yourself!
</body></html>

nori_banner.jpg

meca-nori-analyzer-thumbnail.jpg

Nori: The Official Elasticsearch Plugin for Korean Language Analysis

We can use multi-fields to index and search documents written in multiple languages. Now we’ll try to optimize using a language detector.

world-flags.jpg

How to Search Chinese, Japanese, and Korean Text with Elasticsearch 6.2 - Part 3: Language Detector

This document introduces custom plugins (language analyzers) for Korean, Japanese, and Chinese and suggests an approach to use multi-fields to index and search

How to Search Chinese, Japanese, and Korean Text with Elasticsearch 6.2 - Part 2: Multi-fields

How to Search Chinese, Japanese, and Korean Text with Elasticsearch 5.4.0 - Part 1: Analyzers

How to Search Chinese, Japanese, and Korean Text with Elasticsearch 6.2 - Part 1: Analyzers

JSH_5362.jpg

Ask Me Anything Insights from Elastic{ON} Tour Seoul

What kind of Text Analyzers are available for Korean and how they are different.

<html><head></head><body>Hangul (Korean alphabet) was created in 1443 by King Sejong the Great. Before that, Korean people used Chinese characters but only Yangban, the ruling class people could actually learn and use it and the ordinary people could hardly use it because Chinese language is so different from Korean language and it was too difficult for the ordinary people to spare time to learn.
Hangul is a phonetic alphabet and consists of 24 characters: 14 consonants (&#x3131;[g], &#x3134;[n], &#x3137;[d], &#x3139;[l/r], &#x3141;[m], &#x3142;[b], &#x3145;[s], &#x3147;[null/ng], &#x3148;[j], &#x314A;[ch], &#x314B;[k], &#x314D;[p], &#x314C;[t], and &#x314E;[h]) and 10 vowels (&#x314F;[a], &#x3151;[ja], &#x3153;[&#x259;], &#x3155;[j&#x259;], &#x3157;[o], &#x315B;[jo], &#x315C;[u], &#x3160;[ju], &#x3161;[&#x26F;], and &#x3163;[i]). We can combine them and make 11172 characters (syllables), e.g. &#x314E;+&#x314F;+&#x3134;=&#xD55C;.
A Korean analyzer is required to search Korean documents effectively. Korean is an agglutinative language whereas English is an inflectional language and Chinese is an isolated language. A predicate changes its form according to its ending (e.g., &#x2018;&#xBA39;&#xB2E4;&#x2019; and &#x2018;&#xBA39;&#xACE0;&#x2019;), and a noun is usually followed by one or more postpositions (e.g., &#xC5D8;&#xB77C;&#xC2A4;&#xD2F1;&#xC11C;&#xCE58;(noun)+&#xB97C;(postposition)). If we query without a Korean analyzer, we can only get a single form of the predicates or nouns. For example, if we query &#x2018;&#xC5D8;&#xB77C;&#xC2A4;&#xD2F1;&#xC11C;&#xCE58;&#x2019;, we don&#x2019;t get the documents including &#x2018;&#xC5D8;&#xB77C;&#xC2A4;&#xD2F1;&#xC11C;&#xCE58;&#xB97C;&#x2019;. A Korean analyzer analyzes &#x2018;&#xC5D8;&#xB77C;&#xC2A4;&#xD2F1;&#xC11C;&#xCE58;&#xB97C; &#xC774;&#xC6A9;&#xD574;&#xC11C; &#xD55C;&#xAD6D;&#xC5B4; &#xBB38;&#xC11C;&#xB4E4;&#xC744; &#xD6A8;&#xACFC;&#xC801;&#xC73C;&#xB85C; &#xAC80;&#xC0C9;&#xD558;&#xB824;&#xBA74; &#xD55C;&#xAD6D;&#xC5B4; &#xBD84;&#xC11D;&#xAE30;&#xAC00; &#xD544;&#xC694;&#xD569;&#xB2C8;&#xB2E4;&#x2019; and extracts tokens such as &#x2018;&#xC5D8;&#xB77C;&#xC2A4;&#xD2F1;&#xC11C;&#xCE58;&#x2019;, &#x2018;&#xB97C;&#x2019;, &#x2018;&#xC774;&#xC6A9;&#x2019;, &#x2018;&#xD574;&#xC11C;&#x2019;, &#x2018;&#xD55C;&#xAD6D;&#xC5B4;&#x2019;, &#x2018;&#xBB38;&#xC11C;&#x2019;, &#x2018;&#xB4E4;&#x2019;, &#x2018;&#xC744;&#x2019;, &#xA0;&#x2018;&#xD6A8;&#xACFC;&#xC801;&#x2019;, &#x2018;&#xC73C;&#xB85C;&#x2019;, &#x2018;&#xAC80;&#xC0C9;&#x2019;, &#x2018;&#xD558;&#xB824;&#xBA74;&#x2019;, &#x2018;&#xD55C;&#xAD6D;&#xC5B4;&#x2019;, &#x2018;&#xBD84;&#xC11D;&#xAE30;&#x2019;, &#x2018;&#xAC00;&#x2019;, &#x2018;&#xD544;&#xC694;&#x2019;, and &#x2018;&#xD569;&#xB2C8;&#xB2E4;&#x2019;. With these tokens, we can query &#x2018;&#xC5D8;&#xB77C;&#xC2A4;&#xD2F1;&#xC11C;&#xCE58;&#x2019; and get the documents including either &#x2018;&#xC5D8;&#xB77C;&#xC2A4;&#xD2F1;&#xC11C;&#xCE58;&#x2019; or &#x2018;&#xC5D8;&#xB77C;&#xC2A4;&#xD2F1;&#xC11C;&#xCE58;&#xB97C;&#x2019;.
Currently Elasticsearch has commercial and open source analyzers and provides APIs to implement analyzers. Among them, seunjeon, arirang, and open-korean-text are the widely used open source Korean analyzers. Open-korean-text supports only Elasticsearch 5.x. I installed these three Korean analyzers on Elasticsearch 5.5.0 and measured time and memory consumption during analysis.
<h4>seunjeon</h4><ul>
	<li>URL: <a href="https://bitbucket.org/eunjeon/seunjeon">https://bitbucket.org/eunjeon/seunjeon</a></li>
	<li><a href="https://bitbucket.org/eunjeon/seunjeon"></a>Description:&#xA0;A mecab-ko-dic based Korean analyzer running on JVM. It provides Java and Scala interfaces. It includes the dictionary and you don&#x2019;t need to install mecab-ko-dic separately.</li>
	<li>License:&#xA0;Apache 2.0</li>
</ul><h4>arirang</h4><ul>
	<li>URL:&#xA0;<a href="https://github.com/HowookJeong/elasticsearch-analysis-arirang" target="_blank">https://github.com/HowookJeong/elasticsearch-analysis-arirang</a></li>
	<li>Description: korean analyzer (lucene analyzer kr arirang)</li>
	<li>License: as-is</li>
</ul><h4>open-korean-text</h4><ul>
	<li>URL:&#xA0;<a href="https://github.com/open-korean-text/open-korean-text" target="_blank">https://github.com/open-korean-text/open-korean-text</a><a href="https://github.com/open-korean-text/open-korean-text"></a></li>
	<li>Description:&#xA0;Open-source Korean Text Processor (Official Fork of twitter-korean-text)</li>
	<li>License:&#xA0;Apache 2.0<a href="http://www.apache.org/licenses/LICENSE-2.0"></a></li>
</ul> 
To see the effect of the JIT compiler, I ran the same test twice when I measure the analysis time. I used &#x2018;time&#x2019; (http://man7.org/linux/man-pages/man1/time.1.html) to measure time to analyze a Korean text (see at.sh, ot.sh, and st.sh in the appendix). I ran it once just after starting Elasticsearch, and then ran it again without restarting Elasticsearch.<img src="https://api.contentstack.io/v2/uploads/59f2c02d6de6514c0c2a6c29/download?uid=blte94f3665843bf77b" data-sys-asset-uid="blte94f3665843bf77b" alt="speed.png"> Fig. 1 Time to analyze a Korean text Arirang is the fastest in the both runs, but the 2nd run of seunjeon is much faster than its 1st run. Open-korean-text is similar or a bit slower than seunjeon.Memory (Java heap) consumption was measured at the three points: just before the analysis, maximum usage during the analysis, and just after the analysis. I used &#x2018;jstat -gc&#x2019; (<a href="https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.html" target="_blank">https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.html</a>) to measure the memory consumption (see am.sh, om.sh, and sm.sh in the appendix). Again, I ran the same test twice. I ran it once just after starting Elasticsearch, and then ran it again without restarting Elasticsearch.Arirang showed almost no difference among them, whereas seunjeon showed big increase during the analysis. Open-korean-text showed moderate increase during the analysis but it is mostly released after analysis.
<img src="https://api.contentstack.io/v2/uploads/59f2c0672bdc30600c3fc4eb/download?uid=blt09fc794b5011a415" data-sys-asset-uid="blt09fc794b5011a415" alt="mem-1st.png"> Fig. 2 Memory consumption during Korean text analysis (1st run)
<img src="https://api.contentstack.io/v2/uploads/59f2c0882bdc30600c3fc4f1/download?uid=blt4303dd2f93211bb7" data-sys-asset-uid="blt4303dd2f93211bb7" alt="mem-2nd.png"> Fig. 3 Memory consumption during Korean text analysis (2st run) Finally let&#x2019;s see the analysis results. The original string is the preface King Sejong wrote when he invented Hangul. I used the same string for the time and memory consumption test.&#xA0;
&#x201C;&#xB098;&#xB77C;&#xC758; &#xB9D0;&#xC774; &#xC911;&#xAD6D;&#xACFC; &#xB2EC;&#xB77C; &#xD55C;&#xC790;&#xC640;&#xB294; &#xC11C;&#xB85C; &#xB9DE;&#xC9C0; &#xC544;&#xB2C8;&#xD560;&#xC0C8; &#xC774;&#xB7F0; &#xAE4C;&#xB2ED;&#xC73C;&#xB85C; &#xC5B4;&#xB9AC;&#xC11D;&#xC740; &#xBC31;&#xC131;&#xC774; &#xC774;&#xB974;&#xACE0;&#xC790; &#xD558;&#xB294; &#xBC14; &#xC788;&#xC5B4;&#xB3C4; &#xB9C8;&#xCE68;&#xB0B4; &#xC81C; &#xB73B;&#xC744; &#xB2A5;&#xD788; &#xD3B4;&#xC9C0; &#xBABB;&#xD560; &#xC0AC;&#xB78C;&#xC774; &#xB9CE;&#xC73C;&#xB2C8;&#xB77C; &#xB0B4; &#xC774;&#xB97C; &#xC704;&#xD558;&#xC5EC;, &#xAC00;&#xC5FE;&#xAC8C; &#xC5EC;&#xACA8; &#xC0C8;&#xB85C; &#xC2A4;&#xBB3C;&#xC5EC;&#xB35F; &#xC790;&#xB97C; &#xB9CC;&#xB4DC;&#xB098;&#xB2C8; &#xC0AC;&#xB78C;&#xB9C8;&#xB2E4; &#xD558;&#xC5EC;&#xAE08; &#xC27D;&#xAC8C; &#xC775;&#xD600; &#xB0A0;&#xB9C8;&#xB2E4; &#xC500;&#xC5D0; &#xD3B8;&#xC548;&#xCF00; &#xD558;&#xACE0;&#xC790; &#xD560; &#xB530;&#xB984;&#xC774;&#xB2C8;&#xB77C;&#x201D; (translation: Korean language is different from Chinese language. It doesn&#x2019;t fit in with Chinese characters. Many Korean people have difficulty expressing themselves with Chinese characters. To help them, I invented new 28 characters. I hope people learn and use them easily in daily life.)
<img src="https://api.contentstack.io/v2/uploads/59f2c0b66de6514c0c2a6c2f/download?uid=blt7e5b7142081471df" data-sys-asset-uid="blt7e5b7142081471df" alt="terms.png"> Table 1.&#xA0;Analysis results Open-korean-text extracts the most tokens and differentiates adjective and verb but mis-analyzed &#x2018;&#xB2EC;&#xB77C;&#x2019;, &#x2018;&#xC0C8;&#x2019;, &#x2018;&#xB9C8;&#xCE68;&#xB0B4;&#x2019;, and &#x2018;&#xCF00;&#x2019; as nouns. Seunjeon also mis-analyzed &#x2018;&#xB0B4;&#x2019; as verb. Arirang doesn&#x2019;t provide the part of speech information.By the way, how much portion of the total indexing time does the Korean analyzers take? When I tested with seunjeon, the total indexing time was 29.036 seconds (1st run) and 20.952 seconds (2nd run). The Korean analyzers took 47.83% and 26.67% of the total indexing time respectively. Memory consumption increase was 802 MB during analysis and 853 MB during indexing, which means Korean analyzer takes much part of the total memory consumption during indexing. Analysis time affects indexing time more than search time because in search time, you&#x2019;ll only analyze short keywords, not the already indexed long text.
Based on this test result, no one Korean analyzer seems to be absolutely superior. Arirang can be a good candidate when speed and memory consumption is important. But you&#x2019;ll have to use seunjeon or open-korean-text if you need the part of speech information. One thing to remember is that Korean analyzers take considerable amount of time and memory during indexing and the choice of Korean analyzers is very important in your Elasticsearch configuration.Appendix (<a href="https://github.com/kiju98/analyzers/blob/master/scripts.tar.bz2" target="_blank">scripts.tar.bz2</a>)
<ul><li>Scripts to measure analysis time of seunjeon, arirang, and open-korean-text (st.sh, at.sh, and ot.sh)</li><li>Scripts to measure memory consumption of seunjeon, arirang, and open-korean-text during analysis (sm.sh, am.sh, and om.sh)</li><li>Scripts to measure indexing time of seunjeon (sit.sh)</li><li>Scripts to measure memory consumption of seunjeon during indexing (sim.sh)</li></ul></body></html>

Articles by Kiju Kim

Using classification-based supervised learning in Elasticsearch to curate your movie queue

Nori: The Official Elasticsearch Plugin for Korean Language Analysis

How to Search Chinese, Japanese, and Korean Text with Elasticsearch 6.2 - Part 3: Language Detector

How to Search Chinese, Japanese, and Korean Text with Elasticsearch 6.2 - Part 2: Multi-fields

How to Search Chinese, Japanese, and Korean Text with Elasticsearch 6.2 - Part 1: Analyzers

Ask Me Anything Insights from Elastic{ON} Tour Seoul

Which Korean analyzer shall I use?

Products & Solutions

Company

Resources

Featured topics

News

Articles by Kiju Kim

Using classification-based supervised learning in Elasticsearch to curate your movie queue

Nori: The Official Elasticsearch Plugin for Korean Language Analysis

How to Search Chinese, Japanese, and Korean Text with Elasticsearch 6.2 - Part 3: Language Detector

How to Search Chinese, Japanese, and Korean Text with Elasticsearch 6.2 - Part 2: Multi-fields

How to Search Chinese, Japanese, and Korean Text with Elasticsearch 6.2 - Part 1: Analyzers

Ask Me Anything Insights from Elastic{ON} Tour Seoul

Which Korean analyzer shall I use?