Keyword type family
editKeyword type family
editThe keyword family includes the following field types:
-
keyword
, which is used for structured content such as IDs, email addresses, hostnames, status codes, zip codes, or tags. -
constant_keyword
for keyword fields that always contain the same value. -
wildcard
for unstructured machine-generated content. Thewildcard
type is optimized for fields with large values or high cardinality.
Keyword fields are often used in sorting,
aggregations, and term-level
queries, such as term
.
Avoid using keyword fields for full-text search. Use the text
field type instead.
Keyword field type
editBelow is an example of a mapping for a basic keyword
field:
resp = client.indices.create( index="my-index-000001", mappings={ "properties": { "tags": { "type": "keyword" } } }, ) print(resp)
response = client.indices.create( index: 'my-index-000001', body: { mappings: { properties: { tags: { type: 'keyword' } } } } ) puts response
res, err := es.Indices.Create( "my-index-000001", es.Indices.Create.WithBody(strings.NewReader(`{ "mappings": { "properties": { "tags": { "type": "keyword" } } } }`)), ) fmt.Println(res, err)
const response = await client.indices.create({ index: "my-index-000001", mappings: { properties: { tags: { type: "keyword", }, }, }, }); console.log(response);
PUT my-index-000001 { "mappings": { "properties": { "tags": { "type": "keyword" } } } }
Mapping numeric identifiers
Not all numeric data should be mapped as a numeric field data type.
Elasticsearch optimizes numeric fields, such as integer
or long
, for
range
queries. However, keyword
fields
are better for term
and other
term-level queries.
Identifiers, such as an ISBN or a product ID, are rarely used in range
queries. However, they are often retrieved using term-level queries.
Consider mapping a numeric identifier as a keyword
if:
-
You don’t plan to search for the identifier data using
range
queries. -
Fast retrieval is important.
term
query searches onkeyword
fields are often faster thanterm
searches on numeric fields.
If you’re unsure which to use, you can use a multi-field to map
the data as both a keyword
and a numeric data type.
Parameters for basic keyword fields
editThe following parameters are accepted by keyword
fields:
-
doc_values
-
Should the field be stored on disk in a column-stride fashion, so that it
can later be used for sorting, aggregations, or scripting? Accepts
true
(default) orfalse
. -
eager_global_ordinals
-
Should global ordinals be loaded eagerly on refresh? Accepts
true
orfalse
(default). Enabling this is a good idea on fields that are frequently used for terms aggregations. -
fields
- Multi-fields allow the same string value to be indexed in multiple ways for different purposes, such as one field for search and a multi-field for sorting and aggregations.
-
ignore_above
-
Do not index any string longer than this value. Defaults to
2147483647
so that all values would be accepted. Please however note that default dynamic mapping rules create a subkeyword
field that overrides this default by settingignore_above: 256
. -
index
-
Should the field be quickly searchable? Accepts
true
(default) andfalse
.keyword
fields that only havedoc_values
enabled can still be queried, albeit slower. -
index_options
-
What information should be stored in the index, for scoring purposes.
Defaults to
docs
but can also be set tofreqs
to take term frequency into account when computing scores. -
meta
- Metadata about the field.
-
norms
-
Whether field-length should be taken into account when scoring queries.
Accepts
true
orfalse
(default). -
null_value
-
Accepts a string value which is substituted for any explicit
null
values. Defaults tonull
, which means the field is treated as missing. Note that this cannot be set if thescript
value is used. -
on_script_error
-
Defines what to do if the script defined by the
script
parameter throws an error at indexing time. Acceptsfail
(default), which will cause the entire document to be rejected, andcontinue
, which will register the field in the document’s_ignored
metadata field and continue indexing. This parameter can only be set if thescript
field is also set. -
script
-
If this parameter is set, then the field will index values generated
by this script, rather than reading the values directly from the
source. If a value is set for this field on the input document, then
the document will be rejected with an error.
Scripts are in the same format as their
runtime equivalent. Values emitted by the
script are normalized as usual, and will be ignored if they are longer
that the value set on
ignore_above
. -
store
-
Whether the field value should be stored and retrievable separately from
the
_source
field. Acceptstrue
orfalse
(default). -
similarity
-
Which scoring algorithm or similarity should be used. Defaults
to
BM25
. -
normalizer
-
How to pre-process the keyword prior to indexing. Defaults to
null
, meaning the keyword is kept as-is. -
split_queries_on_whitespace
-
Whether full text queries should split the input on whitespace
when building a query for this field.
Accepts
true
orfalse
(default). -
time_series_dimension
-
(Optional, Boolean)
Marks the field as a time series dimension. Defaults to
false
.The
index.mapping.dimension_fields.limit
index setting limits the number of dimensions in an index.Dimension fields have the following constraints:
-
The
doc_values
andindex
mapping parameters must betrue
. -
Dimension values are used to identify a document’s time series. If dimension values are altered in any way during indexing, the document will be stored as belonging to different from intended time series. As a result there are additional constraints:
-
The field cannot use a
normalizer
.
-
The field cannot use a
-
The
Synthetic _source
editSynthetic _source
is Generally Available only for TSDB indices
(indices that have index.mode
set to time_series
). For other indices
synthetic _source
is in technical preview. Features in technical preview may
be changed or removed in a future release. Elastic will work to fix
any issues, but features in technical preview are not subject to the support SLA
of official GA features.
keyword
fields support synthetic _source
in their
default configuration. Synthetic _source
cannot be used together with
a normalizer
or copy_to
.
By default, synthetic source sorts keyword
fields and removes duplicates.
For example:
resp = client.indices.create( index="idx", mappings={ "_source": { "mode": "synthetic" }, "properties": { "kwd": { "type": "keyword" } } }, ) print(resp) resp1 = client.index( index="idx", id="1", document={ "kwd": [ "foo", "foo", "bar", "baz" ] }, ) print(resp1)
response = client.indices.create( index: 'idx', body: { mappings: { _source: { mode: 'synthetic' }, properties: { kwd: { type: 'keyword' } } } } ) puts response response = client.index( index: 'idx', id: 1, body: { kwd: [ 'foo', 'foo', 'bar', 'baz' ] } ) puts response
const response = await client.indices.create({ index: "idx", mappings: { _source: { mode: "synthetic", }, properties: { kwd: { type: "keyword", }, }, }, }); console.log(response); const response1 = await client.index({ index: "idx", id: 1, document: { kwd: ["foo", "foo", "bar", "baz"], }, }); console.log(response1);
PUT idx { "mappings": { "_source": { "mode": "synthetic" }, "properties": { "kwd": { "type": "keyword" } } } } PUT idx/_doc/1 { "kwd": ["foo", "foo", "bar", "baz"] }
Will become:
{ "kwd": ["bar", "baz", "foo"] }
If a keyword
field sets store
to true
then order and duplicates
are preserved. For example:
resp = client.indices.create( index="idx", mappings={ "_source": { "mode": "synthetic" }, "properties": { "kwd": { "type": "keyword", "store": True } } }, ) print(resp) resp1 = client.index( index="idx", id="1", document={ "kwd": [ "foo", "foo", "bar", "baz" ] }, ) print(resp1)
response = client.indices.create( index: 'idx', body: { mappings: { _source: { mode: 'synthetic' }, properties: { kwd: { type: 'keyword', store: true } } } } ) puts response response = client.index( index: 'idx', id: 1, body: { kwd: [ 'foo', 'foo', 'bar', 'baz' ] } ) puts response
const response = await client.indices.create({ index: "idx", mappings: { _source: { mode: "synthetic", }, properties: { kwd: { type: "keyword", store: true, }, }, }, }); console.log(response); const response1 = await client.index({ index: "idx", id: 1, document: { kwd: ["foo", "foo", "bar", "baz"], }, }); console.log(response1);
PUT idx { "mappings": { "_source": { "mode": "synthetic" }, "properties": { "kwd": { "type": "keyword", "store": true } } } } PUT idx/_doc/1 { "kwd": ["foo", "foo", "bar", "baz"] }
Will become:
{ "kwd": ["foo", "foo", "bar", "baz"] }
Values longer than ignore_above
are preserved but sorted to the end.
For example:
resp = client.indices.create( index="idx", mappings={ "_source": { "mode": "synthetic" }, "properties": { "kwd": { "type": "keyword", "ignore_above": 3 } } }, ) print(resp) resp1 = client.index( index="idx", id="1", document={ "kwd": [ "foo", "foo", "bang", "bar", "baz" ] }, ) print(resp1)
response = client.indices.create( index: 'idx', body: { mappings: { _source: { mode: 'synthetic' }, properties: { kwd: { type: 'keyword', ignore_above: 3 } } } } ) puts response response = client.index( index: 'idx', id: 1, body: { kwd: [ 'foo', 'foo', 'bang', 'bar', 'baz' ] } ) puts response
const response = await client.indices.create({ index: "idx", mappings: { _source: { mode: "synthetic", }, properties: { kwd: { type: "keyword", ignore_above: 3, }, }, }, }); console.log(response); const response1 = await client.index({ index: "idx", id: 1, document: { kwd: ["foo", "foo", "bang", "bar", "baz"], }, }); console.log(response1);
PUT idx { "mappings": { "_source": { "mode": "synthetic" }, "properties": { "kwd": { "type": "keyword", "ignore_above": 3 } } } } PUT idx/_doc/1 { "kwd": ["foo", "foo", "bang", "bar", "baz"] }
Will become:
{ "kwd": ["bar", "baz", "foo", "bang"] }
Constant keyword field type
editConstant keyword is a specialization of the keyword
field for
the case that all documents in the index have the same value.
resp = client.indices.create( index="logs-debug", mappings={ "properties": { "@timestamp": { "type": "date" }, "message": { "type": "text" }, "level": { "type": "constant_keyword", "value": "debug" } } }, ) print(resp)
response = client.indices.create( index: 'logs-debug', body: { mappings: { properties: { "@timestamp": { type: 'date' }, message: { type: 'text' }, level: { type: 'constant_keyword', value: 'debug' } } } } ) puts response
const response = await client.indices.create({ index: "logs-debug", mappings: { properties: { "@timestamp": { type: "date", }, message: { type: "text", }, level: { type: "constant_keyword", value: "debug", }, }, }, }); console.log(response);
PUT logs-debug { "mappings": { "properties": { "@timestamp": { "type": "date" }, "message": { "type": "text" }, "level": { "type": "constant_keyword", "value": "debug" } } } }
constant_keyword
supports the same queries and aggregations as keyword
fields do, but takes advantage of the fact that all documents have the same
value per index to execute queries more efficiently.
It is both allowed to submit documents that don’t have a value for the field or that have a value equal to the value configured in mappings. The two below indexing requests are equivalent:
resp = client.index( index="logs-debug", document={ "date": "2019-12-12", "message": "Starting up Elasticsearch", "level": "debug" }, ) print(resp) resp1 = client.index( index="logs-debug", document={ "date": "2019-12-12", "message": "Starting up Elasticsearch" }, ) print(resp1)
response = client.index( index: 'logs-debug', body: { date: '2019-12-12', message: 'Starting up Elasticsearch', level: 'debug' } ) puts response response = client.index( index: 'logs-debug', body: { date: '2019-12-12', message: 'Starting up Elasticsearch' } ) puts response
const response = await client.index({ index: "logs-debug", document: { date: "2019-12-12", message: "Starting up Elasticsearch", level: "debug", }, }); console.log(response); const response1 = await client.index({ index: "logs-debug", document: { date: "2019-12-12", message: "Starting up Elasticsearch", }, }); console.log(response1);
POST logs-debug/_doc { "date": "2019-12-12", "message": "Starting up Elasticsearch", "level": "debug" } POST logs-debug/_doc { "date": "2019-12-12", "message": "Starting up Elasticsearch" }
However providing a value that is different from the one configured in the mapping is disallowed.
In case no value
is provided in the mappings, the field will automatically
configure itself based on the value contained in the first indexed document.
While this behavior can be convenient, note that it means that a single
poisonous document can cause all other documents to be rejected if it had a
wrong value.
Before a value has been provided (either through the mappings or from a
document), queries on the field will not match any documents. This includes
exists
queries.
The value
of the field cannot be changed after it has been set.
Parameters for constant keyword fields
editThe following mapping parameters are accepted:
Metadata about the field. |
|
|
The value to associate with all documents in the index. If this parameter is not provided, it is set based on the first document that gets indexed. |
Wildcard field type
editThe wildcard
field type is a specialized keyword field for unstructured
machine-generated content you plan to search using grep-like
wildcard
and regexp
queries. The wildcard
type is optimized for fields with large values or high
cardinality.
Internally the wildcard
field indexes the whole field value using ngrams and stores the full string.
The index is used as a rough filter to cut down the number of values that are then checked by retrieving and checking the full values.
This field is especially well suited to run grep-like queries on log lines. Storage costs are typically lower than those of keyword
fields but search speeds for exact matches on full terms are slower. If the
field values share many prefixes, such as URLs for the same website, storage
costs for a wildcard
field may be higher than an equivalent keyword
field.
You index and search a wildcard field as follows
resp = client.indices.create( index="my-index-000001", mappings={ "properties": { "my_wildcard": { "type": "wildcard" } } }, ) print(resp) resp1 = client.index( index="my-index-000001", id="1", document={ "my_wildcard": "This string can be quite lengthy" }, ) print(resp1) resp2 = client.search( index="my-index-000001", query={ "wildcard": { "my_wildcard": { "value": "*quite*lengthy" } } }, ) print(resp2)
response = client.indices.create( index: 'my-index-000001', body: { mappings: { properties: { my_wildcard: { type: 'wildcard' } } } } ) puts response response = client.index( index: 'my-index-000001', id: 1, body: { my_wildcard: 'This string can be quite lengthy' } ) puts response response = client.search( index: 'my-index-000001', body: { query: { wildcard: { my_wildcard: { value: '*quite*lengthy' } } } } ) puts response
const response = await client.indices.create({ index: "my-index-000001", mappings: { properties: { my_wildcard: { type: "wildcard", }, }, }, }); console.log(response); const response1 = await client.index({ index: "my-index-000001", id: 1, document: { my_wildcard: "This string can be quite lengthy", }, }); console.log(response1); const response2 = await client.search({ index: "my-index-000001", query: { wildcard: { my_wildcard: { value: "*quite*lengthy", }, }, }, }); console.log(response2);
PUT my-index-000001 { "mappings": { "properties": { "my_wildcard": { "type": "wildcard" } } } } PUT my-index-000001/_doc/1 { "my_wildcard" : "This string can be quite lengthy" } GET my-index-000001/_search { "query": { "wildcard": { "my_wildcard": { "value": "*quite*lengthy" } } } }
Parameters for wildcard fields
editThe following parameters are accepted by wildcard
fields:
Accepts a string value which is substituted for any explicit |
|
Do not index any string longer than this value. Defaults to |
Limitations
edit-
wildcard
fields are untokenized like keyword fields, so do not support queries that rely on word positions such as phrase queries. -
When running
wildcard
queries anyrewrite
parameter is ignored. The scoring is always a constant score.
Synthetic _source
editwildcard
fields support synthetic _source
so long as they don’t
declare copy_to
.
Synthetic source always sorts wildcard
fields. For example:
resp = client.indices.create( index="idx", mappings={ "_source": { "mode": "synthetic" }, "properties": { "card": { "type": "wildcard" } } }, ) print(resp) resp1 = client.index( index="idx", id="1", document={ "card": [ "king", "ace", "ace", "jack" ] }, ) print(resp1)
response = client.indices.create( index: 'idx', body: { mappings: { _source: { mode: 'synthetic' }, properties: { card: { type: 'wildcard' } } } } ) puts response response = client.index( index: 'idx', id: 1, body: { card: [ 'king', 'ace', 'ace', 'jack' ] } ) puts response
const response = await client.indices.create({ index: "idx", mappings: { _source: { mode: "synthetic", }, properties: { card: { type: "wildcard", }, }, }, }); console.log(response); const response1 = await client.index({ index: "idx", id: 1, document: { card: ["king", "ace", "ace", "jack"], }, }); console.log(response1);
PUT idx { "mappings": { "_source": { "mode": "synthetic" }, "properties": { "card": { "type": "wildcard" } } } } PUT idx/_doc/1 { "card": ["king", "ace", "ace", "jack"] }
Will become:
{ "card": ["ace", "jack", "king"] }