IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Tokenizer reference Classic tokenizer »
Elastic Docs ›Elasticsearch Guide [8.17] ›Text analysis ›Tokenizer reference

Character group tokenizer

edit
IMPORTANT: This documentation is no longer updated. Refer to Elastic's version policy and the latest documentation.

Character group tokenizer

edit

The char_group tokenizer breaks text into terms whenever it encounters a character which is in a defined set. It is mostly useful for cases where a simple custom tokenization is desired, and the overhead of use of the pattern tokenizer is not acceptable.

Configuration

edit

The char_group tokenizer accepts one parameter:

tokenize_on_chars

A list containing a list of characters to tokenize the string on. Whenever a character from this list is encountered, a new token is started. This accepts either single characters like e.g. -, or character groups: whitespace, letter, digit, punctuation, symbol.

max_token_length

The maximum token length. If a token is seen that exceeds this length then it is split at max_token_length intervals. Defaults to 255.

Example output

edit
resp = client.indices.analyze(
    tokenizer={
        "type": "char_group",
        "tokenize_on_chars": [
            "whitespace",
            "-",
            "\n"
        ]
    },
    text="The QUICK brown-fox",
)
print(resp)
response = client.indices.analyze(
  body: {
    tokenizer: {
      type: 'char_group',
      tokenize_on_chars: [
        'whitespace',
        '-',
        "\n"
      ]
    },
    text: 'The QUICK brown-fox'
  }
)
puts response
const response = await client.indices.analyze({
  tokenizer: {
    type: "char_group",
    tokenize_on_chars: ["whitespace", "-", "\n"],
  },
  text: "The QUICK brown-fox",
});
console.log(response);
POST _analyze
{
  "tokenizer": {
    "type": "char_group",
    "tokenize_on_chars": [
      "whitespace",
      "-",
      "\n"
    ]
  },
  "text": "The QUICK brown-fox"
}

returns

{
  "tokens": [
    {
      "token": "The",
      "start_offset": 0,
      "end_offset": 3,
      "type": "word",
      "position": 0
    },
    {
      "token": "QUICK",
      "start_offset": 4,
      "end_offset": 9,
      "type": "word",
      "position": 1
    },
    {
      "token": "brown",
      "start_offset": 10,
      "end_offset": 15,
      "type": "word",
      "position": 2
    },
    {
      "token": "fox",
      "start_offset": 16,
      "end_offset": 19,
      "type": "word",
      "position": 3
    }
  ]
}
« Tokenizer reference Classic tokenizer »