Elasticsearch Reference [7.2] » Analysis » Token Filters » Compound Word Token Filters
«  Synonym Graph Token Filter     Reverse Token Filter  »

Compound Word Token Filtersedit

The hyphenation_decompounder and dictionary_decompounder token filters can decompose compound words found in many German languages into word parts.

Both token filters require a dictionary of word parts, which can be provided as:

word_list

An array of words, specified inline in the token filter configuration, or

word_list_path

The path (either absolute or relative to the config directory) to a UTF-8 encoded file containing one word per line.

Hyphenation decompounderedit

The hyphenation_decompounder uses hyphenation grammars to find potential subwords that are then checked against the word dictionary. The quality of the output tokens is directly connected to the quality of the grammar file you use. For languages like German they are quite good.

XML based hyphenation grammar files can be found in the Objects For Formatting Objects (OFFO) Sourceforge project. Currently only FOP v1.2 compatible hyphenation files are supported. You can download offo-hyphenation_v1.2.zip directly and look in the offo-hyphenation/hyph/ directory. Credits for the hyphenation code go to the Apache FOP project .

Dictionary decompounderedit

The dictionary_decompounder uses a brute force approach in conjunction with only the word dictionary to find subwords in a compound word. It is much slower than the hyphenation decompounder but can be used as a first start to check the quality of your dictionary.

Compound token filter parametersedit

The following parameters can be used to configure a compound word token filter:

type

Either dictionary_decompounder or hyphenation_decompounder.

word_list

A array containing a list of words to use for the word dictionary.

word_list_path

The path (either absolute or relative to the config directory) to the word dictionary.

hyphenation_patterns_path

The path (either absolute or relative to the config directory) to a FOP XML hyphenation pattern file. (required for hyphenation)

min_word_size

Minimum word size. Defaults to 5.

min_subword_size

Minimum subword size. Defaults to 2.

max_subword_size

Maximum subword size. Defaults to 15.

only_longest_match

Whether to include only the longest matching subword or not. Defaults to false

Here is an example:

PUT /compound_word_example
{
    "settings": {
        "index": {
            "analysis": {
                "analyzer": {
                    "my_analyzer": {
                        "type": "custom",
                        "tokenizer": "standard",
                        "filter": ["dictionary_decompounder", "hyphenation_decompounder"]
                    }
                },
                "filter": {
                    "dictionary_decompounder": {
                        "type": "dictionary_decompounder",
                        "word_list": ["one", "two", "three"]
                    },
                    "hyphenation_decompounder": {
                        "type" : "hyphenation_decompounder",
                        "word_list_path": "analysis/example_word_list.txt",
                        "hyphenation_patterns_path": "analysis/hyphenation_patterns.xml",
                        "max_subword_size": 22
                    }
                }
            }
        }
    }
}
«  Synonym Graph Token Filter     Reverse Token Filter  »