dictionary-based stemming via the
hunspell token filter.
Hunspell hunspell.github.io is the
spell checker used by Open Office, LibreOffice, Chrome, Firefox, Thunderbird, and many
other open and closed source projects.
A Hunspell dictionary consists of two files with the same base name—such as
en_US—but with one of two extensions:
- Contains all the root words, in alphabetical order, plus a code representing all possible suffixes and prefixes (which collectively are known as affixes)
Contains the actual prefix or suffix transformation for each code listed
The Hunspell token
filter looks for dictionaries within a dedicated Hunspell
directory, which defaults to
files should be placed in a subdirectory whose name represents the language
or locale of the dictionaries. For instance, we could create a Hunspell
stemmer for American English with the following layout:
The location of the Hunspell directory can be changed by setting
Per-language settings file, described in the following section.
--- ignore_case: true strict_affix_parsing: true
The meaning of these settings is as follows:
Hunspell dictionaries are case sensitive by default: the surname
Bookeris a different word from the noun
booker, and so should be stemmed differently. It may seem like a good idea to use the
hunspellstemmer in case-sensitive mode, but that can complicate things:
- A word at the beginning of a sentence will be capitalized, and thus appear to be a proper noun.
- The input text may be all uppercase, in which case almost no words will be found.
- The user may search for names in all lowercase, in which case no capitalized words will be found.
As a general rule, it is a good idea to set
The quality of dictionaries varies greatly.
Some dictionaries that are
available online have malformed rules in the
.afffile. By default, Lucene will throw an exception if it can’t parse an affix rule. If you need to deal with a broken affix file, you can set
falseto tell Lucene to ignore the broken rules.
You can test the new analyzer with the
and compare its output to that of the
An interesting property of the
hunspell stemmer, as can be seen in the
preceding example, is that it can remove prefixes as well as as suffixes. Most
algorithmic stemmers remove suffixes only.
Hunspell dictionaries can consume a few megabytes of RAM. Fortunately, Elasticsearch creates only a single instance of a dictionary per node. All shards that use the same Hunspell analyzer share the same instance.
While it is not necessary to understand the
format of a Hunspell dictionary in
order to use the
hunspell tokenizer, understanding the format will help you
write your own custom dictionaries. It is quite simple.
For instance, in the US English dictionary, the
en_US.dic file contains an entry for
analyze, which looks like this:
en_US.aff file contains the prefix or suffix rules for the
S flags. Each flag consists of a number of rules, only one of
which should match. Each rule has the following format:
[type] [flag] [letters to remove] [letters to add] [condition]
For instance, the following is suffix (
D. It says that, when a
word ends in a consonant (anything but
u) followed by
y, it can have the
y removed and
ied added (for example,
SFX D y ied [^aeiou]y
The rules for the
S flags mentioned previously are as follows:
More information about the Hunspell syntax can be found on the Hunspell documentation site.