Symbol Synonymsedit

The final part of this chapter is devoted to symbol synonyms, which are unlike the synonyms we have discussed until now. Symbol synonyms are string aliases used to represent symbols that would otherwise be removed during tokenization.

While most punctuation is seldom important for full-text search, character combinations like emoticons may be very signficant, even changing the meaning of the text. Compare these:

  • I am thrilled to be at work on Sunday.
  • I am thrilled to be at work on Sunday :(

The standard tokenizer would simply strip out the emoticon in the second sentence, conflating two sentences that have quite different intent.

We can use the mapping character filter to replace emoticons with symbol synonyms like emoticon_happy and emoticon_sad before the text is passed to the tokenizer:

PUT /my_index
{
  "settings": {
    "analysis": {
      "char_filter": {
        "emoticons": {
          "type": "mapping",
          "mappings": [ 
            ":)=>emoticon_happy",
            ":(=>emoticon_sad"
          ]
        }
      },
      "analyzer": {
        "my_emoticons": {
          "char_filter": "emoticons",
          "tokenizer":   "standard",
          "filter":    [ "lowercase" ]
          ]
        }
      }
    }
  }
}

GET /my_index/_analyze?analyzer=my_emoticons
I am :) not :( 

The mappings filter replaces the characters to the left of => with those to the right.

Emits tokens i, am, emoticon_happy, not, emoticon_sad.

It is unlikely that anybody would ever search for emoticon_happy, but ensuring that important symbols like emoticons are included in the index can be helpful when doing sentiment analysis. Of course, we could equally have used real words, like happy and sad.

Tip

The mapping character filter is useful for simple replacements of exact character sequences. For more-flexible pattern matching, you can use regular expressions with the pattern_replace character filter.