Mapping changesedit

string fields replaced by text/keyword fieldsedit

The string field datatype has been replaced by the text field for full text analyzed content, and the keyword field for not-analyzed exact string values. For backwards compatibility purposes, during the 5.x series:

  • string fields on pre-5.0 indices will function as before.
  • New string fields can be added to pre-5.0 indices as before.
  • text and keyword fields can also be added to pre-5.0 indices.
  • When adding a string field to a new index, the field mapping will be rewritten as a text or keyword field if possible, otherwise an exception will be thrown. Certain configurations that were possible with string fields are no longer possible with text/keyword fields such as enabling term_vectors on a not-analyzed keyword field.

Default string mappingsedit

String mappings now have the following default mappings:

{
  "type": "text",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above": 256
    }
  }
}

This allows to perform full-text search on the original field name and to sort and run aggregations on the sub keyword field.

Numeric fieldsedit

Numeric fields are now indexed with a completely different data-structure, called BKD tree, that is expected to require less disk space and be faster for range queries than the previous way that numerics were indexed.

Term queries will return constant scores now, while they used to return higher scores for rare terms due to the contribution of the document frequency, which this new BKD structure does not record. If scoring is needed, then it is advised to map the numeric fields as `keyword`s too.

Note that this keyword mapping do not need to replace the numeric mapping. For instance if you need both sorting and scoring on your numeric field, you could map it both as a number and a keyword using fields:

PUT my_index
{
  "mappings": {
    "my_type": {
      "properties": {
        "my_number": {
          "type": "long",
          "fields": {
            "keyword": {
              "type":  "keyword"
            }
          }
        }
      }
    }
  }
}

Also the precision_step parameter is now irrelevant and will be rejected on indices that are created on or after 5.0.

geo_point fieldsedit

Like Numeric fields the Geo point field now uses the new BKD tree structure. Since this structure is fundamentally designed for multi-dimension spatial data, the following field parameters are no longer needed or supported: geohash, geohash_prefix, geohash_precision, lat_lon. Geohashes are still supported from an API perspective, and can still be accessed using the .geohash field extension, but they are no longer used to index geo point data.

_timestamp and _ttledit

The _timestamp and _ttl fields were deprecated and are now removed. As a replacement for _timestamp, you should populate a regular date field with the current timestamp on application side. For _ttl, you should either use time-based indices when applicable, or cron a delete-by-query with a range query on a timestamp field

index propertyedit

On all field datatypes (except for the deprecated string field), the index property now only accepts true/false instead of not_analyzed/no. The string field still accepts analyzed/not_analyzed/no.

Doc values on unindexed fieldsedit

Previously, setting a field to index:no would also disable doc-values. Now, doc-values are enabled by default on all types but text and binary, regardless of the value of the index property.

Floating points use float instead of doubleedit

When dynamically mapping a field containing a floating point number, the field now defaults to using float instead of double. The reasoning is that floats should be more than enough for most cases but would decrease storage requirements significantly.

normsedit

norms now take a boolean instead of an object. This boolean is the replacement for norms.enabled. There is no replacement for norms.loading since eager loading of norms is not useful anymore now that norms are disk-based.

fielddata.formatedit

Setting fielddata.format: doc_values in the mappings used to implicitly enable doc-values on a field. This no longer works: the only way to enable or disable doc-values is by using the doc_values property of mappings.

fielddata.filter.regexedit

Regex filters are not supported anymore and will be dropped on upgrade.

Source-transform removededit

The source transform feature has been removed. Instead, use an ingest pipeline.

Field mapping limitsedit

To prevent mapping explosions, the following limits are applied to indices created in 5.x:

  • The maximum number of fields in an index is limited to 1000.
  • The maximum depth for a field (1 plus the number of object or nested parents) is limited to 20.
  • The maximum number of nested fields in an index is limited to 50.

See Settings to prevent mappings explosion for more.

_parent field no longer indexededit

The join between parent and child documents no longer relies on indexed fields and therefore from 5.0.0 onwards the _parent field is no longer indexed. In order to find documents that refer to a specific parent id, the new parent_id query can be used. The GET response and hits inside the search response still include the parent id under the _parent key.

Source format optionedit

The _source mapping no longer supports the format option. It will still be accepted for indices created before the upgrade to 5.0 for backwards compatibility, but it will have no effect. Indices created on or after 5.0 will reject this option.

Object notationedit

Core types no longer support the object notation, which was used to provide per document boosts as follows:

{
  "value": "field_value",
  "boost": 42
}

Boost accuracy for queries on _alledit

Per-field boosts on the _all are now compressed into a single byte instead of the 4 bytes used previously. While this will make the index much more space-efficient, it also means that index time boosts will be less accurately encoded.

_ttl and _timestamp cannot be creatededit

You can no longer create indexes with _ttl or _timestamp enabled. Indexes with them enabled created before 5.0 will continue to work.

You should replace _timestamp in new indexes by adding a field to your source either in the application producing the data or with an ingest pipline like this one:

PUT _ingest/pipeline/timestamp
{
  "description" : "Adds a timestamp field at the current time",
  "processors" : [ {
    "set" : {
      "field": "timestamp",
      "value": "{{_ingest.timestamp}}"
    }
  } ]
}

PUT newindex/type/1?pipeline=timestamp
{
  "example": "data"
}

GET newindex/type/1

Which produces

{
  "_source": {
    "example": "data",
    "timestamp": "2016-06-21T18:48:55.560+0000"
  },
  ...
}

If you have an old index created with 2.x that has _timestamp enabled then you can migrate it to a new index with the a timestamp field in the source with reindex:

POST _reindex
{
  "source": {
    "index": "oldindex"
  },
  "dest": {
    "index": "newindex"
  },
  "script": {
    "lang": "painless",
    "inline": "ctx._source.timestamp = ctx._timestamp; ctx._timestamp = null"
  }
}

You can replace _ttl with time based index names (preferred) or by adding a cron job which runs a delete-by-query on a timestamp field in the source document. If you had documents like this:

POST index/type/_bulk
{"index":{"_id":1}}
{"example": "data", "timestamp": "2016-06-21T18:48:55.560+0000" }
{"index":{"_id":2}}
{"example": "data", "timestamp": "2016-04-21T18:48:55.560+0000" }

Then you could delete all of the documents from before June 1st with:

POST index/type/_delete_by_query
{
  "query": {
    "range" : {
      "timestamp" : {
        "lt" : "2016-05-01"
      }
    }
  }
}

Keep in mind that deleting documents from an index is very expensive compared to deleting whole indexes. That is why time based indexes are recommended over this sort of thing and why _ttl was deprecated in the first place.

Blank field names is not supportededit

Blank field names in mappings is not allowed after 5.0.