Breaking changes in 7.0edit

This section discusses the changes that you need to be aware of when migrating your application to Elasticsearch 7.0.

See also Release highlights and Release notes.

Indices created before 7.0edit

Elasticsearch 7.0 can read indices created in version 6.0 or above. An Elasticsearch 7.0 node will not start in the presence of indices created in a version of Elasticsearch before 6.0.

Important

Reindex indices from Elasticsearch 5.x or before

Indices created in Elasticsearch 5.x or before will need to be reindexed with Elasticsearch 6.x in order to be readable by Elasticsearch 7.x.

Aggregations changesedit

Deprecated global_ordinals_hash and global_ordinals_low_cardinality execution hints for terms aggregations have been removededit

These execution_hint are removed and should be replaced by global_ordinals.

search.max_buckets in the cluster settingedit

The dynamic cluster setting named search.max_buckets now defaults to 10,000 (instead of unlimited in the previous version). Requests that try to return more than the limit will fail with an exception.

missing option of the composite aggregation has been removededit

The missing option of the composite aggregation, deprecated in 6.x, has been removed. missing_bucket should be used instead.

Replaced params._agg with state context variable in scripted metric aggregationsedit

The object used to share aggregation state between the scripts in a Scripted Metric Aggregation is now a variable called state available in the script context, rather than being provided via the params object as params._agg.

Make metric aggregation script parameters reduce_script and combine_script mandatoryedit

The metric aggregation has been changed to require these two script parameters to ensure users are explicitly defining how their data is processed.

Analysis changesedit

Limiting the number of tokens produced by _analyzeedit

To safeguard against out of memory errors, the number of tokens that can be produced using the _analyze endpoint has been limited to 10000. This default limit can be changed for a particular index with the index setting index.analyze.max_token_count.

Limiting the length of an analyzed text during highlightingedit

Highlighting a text that was indexed without offsets or term vectors, requires analysis of this text in memory real time during the search request. For large texts this analysis may take substantial amount of time and memory. To protect against this, the maximum number of characters that will be analyzed has been limited to 1000000. This default limit can be changed for a particular index with the index setting index.highlight.max_analyzed_offset.

delimited_payload_filter renamingedit

The delimited_payload_filter was deprecated and renamed to delimited_payload in 6.2. Using it in indices created before 7.0 will issue deprecation warnings. Using the old name in new indices created in 7.0 will throw an error. Use the new name delimited_payload instead.

standard filter has been removededit

The standard token filter has been removed because it doesn’t change anything in the stream.

Cluster changesedit

: is no longer allowed in cluster nameedit

Due to cross-cluster search using : to separate a cluster and index name, cluster names may no longer contain :.

New default for wait_for_active_shards parameter of the open index commandedit

The default value for the wait_for_active_shards parameter of the open index API is changed from 0 to 1, which means that the command will now by default wait for all primary shards of the opened index to be allocated.

Shard preferences _primary, _primary_first, _replica, and _replica_first are removededit

These shard preferences are removed in favour of the _prefer_nodes and _only_nodes preferences.

Cluster-wide shard soft limitedit

Clusters now have soft limits on the total number of open shards in the cluster based on the number of nodes and the cluster.max_shards_per_node cluster setting, to prevent accidental operations that would destabilize the cluster. More information can be found in the documentation for that setting.

Indices changesedit

: is no longer allowed in index nameedit

Due to cross-cluster search using : to separate a cluster and index name, index names may no longer contain :.

index.unassigned.node_left.delayed_timeout may no longer be negativeedit

Negative values were interpreted as zero in earlier versions but are no longer accepted.

_flush and _force_merge will no longer refreshedit

In previous versions issuing a _flush or _force_merge (with flush=true) had the undocumented side-effect of refreshing the index which made new documents visible to searches and non-realtime GET operations. From now on these operations don’t have this side-effect anymore. To make documents visible an explicit _refresh call is needed unless the index is refreshed by the internal scheduler.

Limit to the difference between max_size and min_size in NGramTokenFilter and NGramTokenizeredit

To safeguard against creating too many index terms, the difference between max_ngram and min_ngram in NGramTokenFilter and NGramTokenizer has been limited to 1. This default limit can be changed with the index setting index.max_ngram_diff. Note that if the limit is exceeded a error is thrown only for new indices. For existing pre-7.0 indices, a deprecation warning is logged.

Limit to the difference between max_size and min_size in ShingleTokenFilteredit

To safeguard against creating too many tokens, the difference between max_shingle_size and min_shingle_size in ShingleTokenFilter has been limited to 3. This default limit can be changed with the index setting index.max_shingle_diff. Note that if the limit is exceeded a error is thrown only for new indices. For existing pre-7.0 indices, a deprecation warning is logged.

Document distribution changesedit

Indices created with version 7.0.0 onwards will have an automatic index.number_of_routing_shards value set. This might change how documents are distributed across shards depending on how many shards the index has. In order to maintain the exact same distribution as a pre 7.0.0 index, the index.number_of_routing_shards must be set to the index.number_of_shards at index creation time. Note: if the number of routing shards equals the number of shards _split operations are not supported.

Skipped background refresh on search idle shardsedit

Shards belonging to an index that does not have an explicit index.refresh_interval configured will no longer refresh in the background once the shard becomes "search idle", ie the shard hasn’t seen any search traffic for index.search.idle.after seconds (defaults to 30s). Searches that access a search idle shard will be "parked" until the next refresh happens. Indexing requests with wait_for_refresh will also trigger a background refresh.

Remove deprecated url parameters for Clear Indices Cache APIedit

The following previously deprecated url parameter have been removed:

  • filter - use query instead
  • filter_cache - use query instead
  • request_cache - use request instead
  • field_data - use fielddata instead

network.breaker.inflight_requests.overhead increased to 2edit

Previously the in flight requests circuit breaker considered only the raw byte representation. By bumping the value of network.breaker.inflight_requests.overhead from 1 to 2, this circuit breaker considers now also the memory overhead of representing the request as a structured object.

Parent circuit breaker changesedit

The parent circuit breaker defines a new setting indices.breaker.total.use_real_memory which is true by default. This means that the parent circuit breaker will trip based on currently used heap memory instead of only considering the reserved memory by child circuit breakers. When this setting is true, the default parent breaker limit also changes from 70% to 95% of the JVM heap size. The previous behavior can be restored by setting indices.breaker.total.use_real_memory to false.

fix value for index.shard.check_on_startup is removededit

Deprecated option value fix for setting index.shard.check_on_startup is not supported.

elasticsearch-translog is removededit

Use the elasticsearch-shard tool to remove corrupted translog data.

Mapping changesedit

The _all meta field is removededit

The _all field deprecated in 6 have now been removed.

The _uid meta field is removededit

This field used to index a composite key formed of the _type and the _id. Now that indices cannot have multiple types, this has been removed in favour of _id.

The _default_ mapping is no longer allowededit

The _default_ mapping has been deprecated in 6.0 and is now no longer allowed in 7.0. Trying to configure a _default_ mapping on 7.x indices will result in an error.

index_options for numeric fields has been removededit

The index_options field for numeric fields has been deprecated in 6 and has now been removed.

Limiting the number of nested json objectsedit

To safeguard against out of memory errors, the number of nested json objects within a single document across all fields has been limited to 10000. This default limit can be changed with the index setting index.mapping.nested_objects.limit.

The update_all_types option has been removededit

This option is useless now that all indices have at most one type.

The classic similarity has been removededit

The classic similarity relied on coordination factors for scoring to be good in presence of stopwords in the query. This feature has been removed from Lucene, which means that the classic similarity now produces scores of lower quality. It is advised to switch to BM25 instead, which is widely accepted as a better alternative.

Similarities fail when unsupported options are providededit

An error will now be thrown when unknown configuration options are provided to similarities. Such unknown parameters were ignored before.

Search and Query DSL changesedit

Changes to queriesedit

  • The default value for transpositions parameter of fuzzy query has been changed to true.
  • The query_string options use_dismax, split_on_whitespace, all_fields, locale, auto_generate_phrase_query and lowercase_expanded_terms deprecated in 6.x have been removed.
  • Purely negative queries (only MUST_NOT clauses) now return a score of 0 rather than 1.
  • The boundary specified using geohashes in the geo_bounding_box query now include entire geohash cell, instead of just geohash center.
  • Attempts to generate multi-term phrase queries against non-text fields with a custom analyzer will now throw an exception
  • An envelope crossing the dateline in a `geo_shape `query is now processed correctly when specified using REST API instead of having its left and right corners flipped.

Adaptive replica selection enabled by defaultedit

Adaptive replica selection has been enabled by default. If you wish to return to the older round robin of search requests, you can use the cluster.routing.use_adaptive_replica_selection setting:

PUT /_cluster/settings
{
    "transient": {
        "cluster.routing.use_adaptive_replica_selection": false
    }
}

Search API returns 400 for invalid requestsedit

The Search API returns 400 - Bad request while it would previously return 500 - Internal Server Error in the following cases of invalid request:

  • the result window is too large
  • sort is used in combination with rescore
  • the rescore window is too large
  • the number of slices is too large
  • keep alive for scroll is too large
  • number of filters in the adjacency matrix aggregation is too large
  • script compilation errors

Scroll queries cannot use the request_cache anymoreedit

Setting request_cache:true on a query that creates a scroll (scroll=1m) has been deprecated in 6 and will now return a 400 - Bad request. Scroll queries are not meant to be cached.

Scroll queries cannot use rescore anymoreedit

Including a rescore clause on a query that creates a scroll (scroll=1m) has been deprecated in 6.5 and will now return a 400 - Bad request. Allowing rescore on scroll queries would break the scroll sort. In the 6.x line, the rescore clause was silently ignored (for scroll queries), and it was allowed in the 5.x line.

Term Suggesters supported distance algorithmsedit

The following string distance algorithms were given additional names in 6.2 and their existing names were deprecated. The deprecated names have now been removed.

  • levenstein - replaced by levenshtein
  • jarowinkler - replaced by jaro_winkler

popular mode for Suggestersedit

The popular mode for Suggesters (term and phrase) now uses the doc frequency (instead of the sum of the doc frequency) of the input terms to compute the frequency threshold for candidate suggestions.

Limiting the number of terms that can be used in a Terms Query requestedit

Executing a Terms Query with a lot of terms may degrade the cluster performance, as each additional term demands extra processing and memory. To safeguard against this, the maximum number of terms that can be used in a Terms Query request has been limited to 65536. This default maximum can be changed for a particular index with the index setting index.max_terms_count.

Limiting the length of regex that can be used in a Regexp Query requestedit

Executing a Regexp Query with a long regex string may degrade search performance. To safeguard against this, the maximum length of regex that can be used in a Regexp Query request has been limited to 1000. This default maximum can be changed for a particular index with the index setting index.max_regex_length.

Limiting the number of auto-expanded fieldsedit

Executing queries that use automatic expansion of fields (e.g. query_string, simple_query_string or multi_match) can have performance issues for indices with a large numbers of fields. To safeguard against this, a hard limit of 1024 fields has been introduced for queries using the "all fields" mode ("default_field": "") or other fieldname expansions (e.g. "foo").

Invalid _search request bodyedit

Search requests with extra content after the main object will no longer be accepted by the _search endpoint. A parsing exception will be thrown instead.

Context Completion Suggesteredit

The ability to query and index context enabled suggestions without context, deprecated in 6.x, has been removed. Context enabled suggestion queries without contexts have to visit every suggestion, which degrades the search performance considerably.

For geo context the value of the path parameter is now validated against the mapping, and the context is only accepted if path points to a field with geo_point type.

Semantics changed for max_concurrent_shard_requestsedit

max_concurrent_shard_requests used to limit the total number of concurrent shard requests a single high level search request can execute. In 7.0 this changed to be the max number of concurrent shard requests per node. The default is now 5.

max_score set to null when scores are not trackededit

max_score used to be set to 0 whenever scores are not tracked. null is now used instead which is a more appropriate value for a scenario where scores are not available.

Negative boosts are not allowededit

Setting a negative boost in a query, deprecated in 6x, are not allowed in this version. To deboost a specific query you can use a boost comprise between 0 and 1.

Packaging changesedit

systemd service file is no longer configurationedit

The systemd service file /usr/lib/systemd/system/elasticsearch.service was previously marked as a configuration file in rpm and deb packages. Overrides to the systemd elasticsearch service should be made in /etc/systemd/system/elasticsearch.service.d/override.conf.

tar package no longer includes windows specific filesedit

The tar package previously included files in the bin directory meant only for windows. These files have been removed. Use the zip package instead.

Plugins changesedit

Azure Repository pluginedit

  • The legacy azure settings which where starting with cloud.azure.storage. prefix have been removed. This includes account, key, default and timeout. You need to use settings which are starting with azure.client. prefix instead.
  • Global timeout setting cloud.azure.storage.timeout has been removed. You must set it per azure client instead. Like azure.client.default.timeout: 10s for example.

See Azure Repository settings.

Google Cloud Storage Repository pluginedit

  • The repository settings application_name, connect_timeout and read_timeout have been removed and must now be specified in the client settings instead.

See Google Cloud Storage Client Settings.

S3 Repository Pluginedit

  • The plugin now uses the path style access pattern for all requests. In previous versions it was automatically determining whether to use virtual hosted style or path style access.

Analysis Plugin changesedit

  • The misspelled helper method requriesAnalysisSettings(AnalyzerProvider<T> provider) has been renamed to requiresAnalysisSettings

File-based discovery pluginedit

  • This plugin has been removed since its functionality is now part of Elasticsearch and requires no plugin. The location of the hosts file has moved from $ES_PATH_CONF/file-discovery/unicast_hosts.txt to $ES_PATH_CONF/unicast_hosts.txt. See the file-based hosts provider documentation for further information.

Security Extensionsedit

As a consequence of the change to Realm settings, the getRealmSettings method has been removed from the SecurityExtension class, and the settings method on RealmConfig now returns the node’s (global) settings. Custom security extensions should register their settings by implementing the standard Plugin.getSettings method, and can retrieve them from RealmConfig.settings() or using one of the RealmConfig.getSetting methods. Each realm setting should be defined as an AffixSetting as shown in the example below:

Setting.AffixSetting<String> MY_SETTING = Setting.affixKeySetting(
  "xpack.security.authc.realms." + MY_REALM_TYPE + ".", "my_setting",
  key -> Setting.simpleString(key, properties)
);

The RealmSettings.simpleString method can be used as a convenience for the above.

API changesedit

Camel case and underscore parameters deprecated in 6.x have been removededit

A number of duplicate parameters deprecated in 6.x have been removed from Bulk request, Multi Get request, Term Vectors request, and More Like This Query requests.

The following camel case parameters have been removed:

  • opType
  • versionType, _versionType

The following parameters starting with underscore have been removed:

  • _parent
  • _retry_on_conflict
  • _routing
  • _version
  • _version_type

Instead of these removed parameters, use their non camel case equivalents without starting underscore, e.g. use version_type instead of _version_type or versionType.

Thread pool infoedit

In previous versions of Elasticsearch, the thread pool info returned in the nodes info API returned min and max values reflecting the configured minimum and maximum number of threads that could be in each thread pool. The trouble with this representation is that it does not align with the configuration parameters used to configure thread pools. For scaling thread pools, the minimum number of threads is configured by a parameter called core and the maximum number of threads is configured by a parameter called max. For fixed thread pools, there is only one configuration parameter along these lines and that parameter is called size, reflecting the fixed number of threads in the pool. This discrepancy between the API and the configuration parameters has been rectified. Now, the API will report core and max for scaling thread pools, and size for fixed thread pools.

Similarly, in the cat thread pool API the existing size output has been renamed to pool_size which reflects the number of threads currently in the pool; the shortcut for this value has been changed from s to psz. The min output has been renamed to core with a shortcut of cr, the shortcut for max has been changed to mx, and the size output with a shortcut of sz has been reused to report the configured number of threads in the pool. This aligns the output of the API with the configuration values for thread pools. Note that core and max will be populated for scaling thread pools, and size will be populated for fixed thread pools.

The parameter fields deprecated in 6.x has been removed from Bulk requestedit

and Update request. The Update API returns 400 - Bad request if request contains unknown parameters (instead of ignored in the previous version).

Remove support for suggest metric/index metric in indices stats and nodes stats APIsedit

Previously, suggest stats were folded into search stats. Support for the suggest metric on the indices stats and nodes stats APIs remained for backwards compatibility. Backwards support for the suggest metric was deprecated in 6.3.0 and now removed in 7.0.0.

In the past, fields could be provided either as a parameter, or as part of the request body. Specifying fields in the request body as opposed to a parameter was deprecated in 6.4.0, and is now unsupported in 7.0.0.

copy_settings is deprecated on shrink and split APIsedit

Versions of Elasticsearch prior to 6.4.0 did not copy index settings on shrink and split operations. Starting with Elasticsearch 7.0.0, the default behavior will be for such settings to be copied on such operations. To enable users in 6.4.0 to transition in 6.4.0 to the default behavior in 7.0.0, the copy_settings parameter was added on the REST layer. As this behavior will be the only behavior in 8.0.0, this parameter is deprecated in 7.0.0 for removal in 8.0.0.

The deprecated stored script contexts have now been removededit

When putting stored scripts, support for storing them with the deprecated template context or without a context is now removed. Scripts must be stored using the script context as mentioned in the documentation.

Get Aliases API limitations when X-Pack security is enabled removededit

The behavior and response codes of the get aliases API no longer vary depending on whether X-Pack security is enabled. Previously a 404 - NOT FOUND (IndexNotFoundException) could be returned in case the current user was not authorized for any alias. An empty response with status 200 - OK is now returned instead at all times.

Put User API response no longer has user objectedit

The Put User API response was changed in 6.5.0 to add the created field outside of the user object where it previously had been. In 7.0.0 the user object has been removed in favor of the top level created field.

Source filtering url parameters _source_include and _source_exclude have been removededit

The deprecated in 6.x url parameters are now removed. Use _source_includes and _source_excludes instead.

Java API changesedit

isShardsAcked deprecated in 6.2 has been removededit

isShardsAcked has been replaced by isShardsAcknowledged in CreateIndexResponse, RolloverResponse and CreateIndexClusterStateUpdateResponse.

prepareExecute removed from the client apiedit

The prepareExecute method which created a request builder has been removed from the client api. Instead, construct a builder for the appropriate request directly.

Some Aggregation classes have moved packagesedit

  • All classes present in org.elasticsearch.search.aggregations.metrics.* packages were moved to a single org.elasticsearch.search.aggregations.metrics package.
  • All classes present in org.elasticsearch.search.aggregations.pipeline.* packages were moved to a single org.elasticsearch.search.aggregations.pipeline package. In addition, org.elasticsearch.search.aggregations.pipeline.PipelineAggregationBuilders was moved to org.elasticsearch.search.aggregations.PipelineAggregationBuilders

Retry.withBackoff methods with Settings removededit

The variants of Retry.withBackoff that included Settings have been removed because Settings is no longer needed.

Settings changesedit

The default for node.name is now the hostnameedit

node.name now defaults to the hostname at the time when Elasticsearch is started. Previously the default node name was the first eight characters of the node id. It can still be configured explicitly in elasticsearch.yml.

Percolatoredit

  • The deprecated index.percolator.map_unmapped_fields_as_string setting has been removed in favour of the index.percolator.map_unmapped_fields_as_text setting.

Index thread pooledit

  • Internally, single-document index/delete/update requests are executed as bulk requests with a single-document payload. This means that these requests are executed on the bulk thread pool. As such, the indexing thread pool is no longer needed and has been removed. As such, the settings thread_pool.index.size and thread_pool.index.queue_size have been removed.

Write thread pool fallbackedit

  • The bulk thread pool was replaced by the write thread pool in 6.3.0. However, for backwards compatibility reasons the name bulk was still usable as fallback settings thread_pool.bulk.size and thread_pool.bulk.queue_size for thread_pool.write.size and thread_pool.write.queue_size, respectively, and the system property es.thread_pool.write.use_bulk_as_display_name was available to keep the display output in APIs as bulk instead of write. These fallback settings and this system property have been removed.

Http enabled setting removededit

  • The setting http.enabled previously allowed disabling binding to HTTP, only allowing use of the transport client. This setting has been removed, as the transport client will be removed in the future, thus requiring HTTP to always be enabled.

Http pipelining setting removededit

  • The setting http.pipelining previously allowed disabling HTTP pipelining support. This setting has been removed, as disabling http pipelining support on the server provided little value. The setting http.pipelining.max_events can still be used to limit the number of pipelined requests in-flight.

Cross-cluster search settings renamededit

The cross-cluster search remote cluster connection infrastructure is also used in cross-cluster replication. This means that the setting names search.remote.* used for configuring cross-cluster search belie the fact that they also apply to other situations where a connection to a remote cluster as used. Therefore, these settings have been renamed from search.remote.* to cluster.remote.*. For backwards compatibility purposes, we will fallback to search.remote.* if cluster.remote.* is not set. For any such settings stored in the cluster state, or set on dynamic settings updates, we will automatically upgrade the setting from search.remote.* to cluster.remote.*. The fallback settings will be removed in 8.0.0.

Audit logfile local node infoedit

The following settings have been removed:

  • xpack.security.audit.logfile.prefix.emit_node_host_address, instead use xpack.security.audit.logfile.emit_node_host_address
  • xpack.security.audit.logfile.prefix.emit_node_host_name, instead use xpack.security.audit.logfile.emit_node_host_name
  • xpack.security.audit.logfile.prefix.emit_node_name, instead use xpack.security.audit.logfile.emit_node_name

The new settings have the same meaning as the removed ones, but the prefix name component is no longer meaningful as logfile audit entries are structured JSON documents and are not prefixed by anything. Moreover, xpack.security.audit.logfile.emit_node_name has changed its default from true to false. All other settings mentioned before, have kept their default value of false.

Security realms settingsedit

The settings for all security realms must now include the realm type as part of the setting name, and the explicit type setting has been removed.

A realm that was previous configured as:

xpack.security.authc.realms:
  ldap1:
    type: ldap
    order: 1
    url: "ldaps://ldap.example.com/"

Must be migrated to:

xpack.security.authc.realms:
  ldap.ldap1:
    order: 1
    url: "ldaps://ldap.example.com/"

Any realm specific secure settings that have been stored in the elasticsearch keystore (such as ldap bind passwords, or passwords for ssl keys) must be updated in a similar way.

Scripting changesedit

getDate() and getDates() removededit

Fields of type long and date had getDate() and getDates() methods (for multi valued fields) to get an object with date specific helper methods for the current doc value. In 5.3.0, date fields were changed to expose this same date object directly when calling doc["myfield"].value, and the getter methods for date objects were deprecated. These methods have now been removed. Instead, use .value on date fields, or explicitly parse long fields into a date object using Instance.ofEpochMillis(doc["myfield"].value).

Accessing missing document values will throw an erroredit

doc['field'].value will throw an exception if the document is missing a value for the field field.

To check if a document is missing a value, you can use doc['field'].size() == 0.

Script errors will return as 400 error codesedit

Malformed scripts, either in search templates, ingest pipelines or search requests, return 400 - Bad request while they would previously return 500 - Internal Server Error. This also applies for stored scripts.

Snapshot stats changesedit

Snapshot stats details are provided in a new structured way:

  • total section for all the files that are referenced by the snapshot.
  • incremental section for those files that actually needed to be copied over as part of the incremental snapshotting.
  • In case of a snapshot that’s still in progress, there’s also a processed section for files that are in the process of being copied.

Deprecated number_of_files, processed_files, total_size_in_bytes and processed_size_in_bytes snapshot stats properties have been removededit

  • Properties number_of_files and total_size_in_bytes are removed and should be replaced by values of nested object total.
  • Properties processed_files and processed_size_in_bytes are removed and should be replaced by values of nested object processed.

High-level REST client changesedit

API methods accepting Header argument have been removededit

All API methods accepting headers as a Header varargs argument, deprecated since 6.4, have been removed in favour of the newly introduced methods that accept instead a RequestOptions argument. In case you are not specifying any header, e.g. client.index(indexRequest) becomes client.index(indexRequest, RequestOptions.DEFAULT). In case you are specifying headers e.g. client.index(indexRequest, new Header("name" "value")) becomes client.index(indexRequest, RequestOptions.DEFAULT.toBuilder().addHeader("name", "value").build());

Cluster Health API default to cluster leveledit

The Cluster Health API used to default to shards level to ease migration from transport client that doesn’t support the level parameter and always returns information including indices and shards details. The level default value has been aligned with the Elasticsearch default level: cluster.

Low-level REST client changesedit

Deprecated flavors of performRequest have been removededit

We deprecated the flavors of performRequest and performRequestAsync that do not take Request objects in 6.4.0 in favor of the flavors that take Request objects because those methods can be extended without breaking backwards compatibility.

Removed setHostsedit

We deprecated setHosts in 6.4.0 in favor of setNodes because it supports host metadata used by the NodeSelector.