Elasticsearch version 7.10.0edit

Also see Breaking changes in 7.10.

Known issuesedit

  • SQL: If a WHERE clause contains at least two relational operators joined by AND, of which one is a comparison (<=, <, >=, >) and another one is an inequality (!=, <>), both against literals or foldable expressions, the inequality will be ignored. The workaround is to substitute the inequality with a NOT IN operator.

    We have fixed this issue in Elasticsearch 7.10.1 and later versions. For more details, see #65488.

Breaking changesedit

Authentication
  • API key name should always be required for creation #59836 (issue: #59484)
Network
  • Set specific keepalive options by default on supported platforms #59278

Breaking Java changesedit

Machine Learning
  • Deprecate allow_no_jobs and allow_no_datafeeds in favor of allow_no_match #60601 (issue: #60642)
Mapping
  • Pass SearchLookup supplier through to fielddataBuilder #61430 (issue: #59332)

Deprecationsedit

Cluster Coordination
Machine learning
  • Renames */inference* APIs to */trained_models* #63097

New featuresedit

Aggregations
Features/Features
  • Add data tiers (hot, warm, cold, frozen) as custom node roles #60994 (issue: #60848)
  • Allocate newly created indices on data_hot tier nodes #61342 (issue: #60848)
Features/ILM+SLM
Machine Learning
  • Implement AucRoc metric for classification #60502 (issue: #62160)
Mapping
  • Introduce 64-bit unsigned long field type #60050 (issue: #32434)
Search
  • Add search fields option to support high-level field retrieval #60100 (issues: #49028, #55363)

Enhancementsedit

Aggregations
  • Adds hard_bounds to histogram aggregations #59175 (issue: #50109)
  • Allocate slightly less per bucket #59740
  • Improve reduction of terms aggregations #61779 (issue: #51857)
  • Speed up date_histogram by precomputing ranges #61467
Analysis
  • Support ignore_keywords flag for word delimiter graph token filter #59563 (issue: #59491)
Authentication
  • Cache API key doc to reduce traffic to the security index #59376 (issues: #53940, #55836)
  • Include authentication type for the authenticate response #61247 (issue: #61130)
  • Oidc additional client auth types #58708
  • Warn about unlicensed realms if no auth token can be extracted #61402 (issue: #61090)
Authorization
  • Add DEBUG logging for undefined role mapping field #61246 (issue: #48562)
  • Add more context to index access denied errors #60357 (issue: #42166)
CRUD
  • Add configured indexing memory limit to node stats #60342
  • Dedicated threadpool for system index writes #61655
Cluster Coordination
  • Add more useful toString on cluster state observers #60277
  • Fail invalid incremental cluster state writes #61030
  • Provide option to allow writes when master is down #60605
Distributed
  • Detect noop of update index settings #61348
  • Thread safe clean up of LocalNodeModeListeners #59932 (issue: #59801)
Features/CAT APIs
  • Adding Hit counts and Miss counts for QueryCache exposed through REST API #60114 (issue: #48645)
Features/Features
  • Add aggregation list to node info #60074 (issue: #52057)
  • Adding new require_alias option to indexing requests #58917 (issue: #55267)
Features/ILM+SLM
  • Move internal index templates to composable templates #61457
Features/Ingest
  • Add network from MaxMind Geo ASN database #61676
  • Allow_duplicates option for append processor #61916 (issue: #57543)
  • Configurable output format for date processor #61324 (issue: #42523)
  • Enhance the ingest node simulate verbose output #60433 (issue: #56004)
  • Per processor description for verbose simulate #58207 (issue: #57906)
  • Preserve grok pattern ordering and add sort option #61671 (issue: #40819)
Features/Java High Level REST Client
  • HLRC: UpdateByQuery API with wait_for_completion being false #58552 (issues: #35202, #46350)
Infra/Core
  • Add logstash system index APIs #53350
  • Deprecate REST access to System Indices #60945
  • Speed up Compression Logic by Pooling Resources #61358
  • System index reads in separate threadpool #57936 (issues: #37867, #50251)
Infra/Logging
Infra/Packaging
  • Add UBI docker builds #60742
  • Upgrade Centos version in Dockerfile to 8 #59019
Infra/Resiliency
Infra/Scripting
Machine Learning
  • Add a "verbose" option to the data frame analytics stats endpoint #59589 (issue: #59125)
  • Add new include flag to get trained models API for model training metadata #61922
  • Add new feature_processors field for data frame analytics #60528 (issue: #59327)
  • Add new n_gram_encoding custom processor #61578
  • During nightly maintenance delete jobs whose original deletion tasks were lost #60121 (issue: #42840)
  • Suspend persistence of trained model stats when ML upgrade mode is enabled #61143
  • Calculate total feature importance to store with model metadata #1387
  • Change outlier detection feature_influence format to array with nested objects #1475, #62068
  • Add timeouts to named pipe connections #1514, #62993 (issue: #1504)
Mapping
Network
  • Improve deserialization failure logging #60577 (issue: #38939)
  • Log and track open/close of transport connections #60297
Performance
  • Speed up empty highlighting many fields #61860
SQL
  • Add option to provide the delimiter for the CSV format #59907 (issue: #41634)
  • Implement DATE_PARSE function for parsing strings into DATE values #57391 (issue: #54962)
  • Implement FORMAT function #55454 (issue: #54965)
Search
  • Avoid reloading _source for every inner hit #60494 (issue: #32818)
  • Cancel multisearch when http connection closed #61399
  • Enable cancellation for msearch requests #61337
  • Executes incremental reduce in the search thread pool #58461 (issues: #51857, #53411)
  • Introduce point in time APIs in x-pack basic #61062 (issues: #26472, #46523)
  • ParametrizedFieldMapper to run validators against default value #60042 (issue: #59332)
  • Add case insensitive flag for "term" family of queries #61596 (issue: #61546)
  • Add case insensitive support for regex queries #59441
  • Tweak toXContent implementation of ParametrizedFieldMapper #59968
  • Implement fields value fetching for the text, search_as_you_type and token_count field types #63515
  • Make term/prefix/wildcard/regex query parsing more lenient, with respect to the case_insensitive flag #63926 (issue: #63893)
Snapshot/Restore
  • Add repositories metering API #60371
  • Clone Snapshot API #61839
  • Determine shard size before allocating shards recovering from snapshots #61906
  • Introduce index based snapshot blob cache for Searchable Snapshots #60522
  • Validate snapshot UUID during restore #59601 (issue: #50999)
Store
  • Report more details of unobtainable ShardLock #61255 (issue: #38807)
Transform

Bug fixesedit

Aggregations
  • Fix AOOBE when setting min_doc_count to 0 in significant_terms #60823 (issues: #60683, #60824)
  • Make sure non-collecting aggs include sub-aggs #64214 (issue: #64142)
  • Composite aggregation must check live docs when the index is sorted #63864
  • Fix broken parent and child aggregator #63811
Allocation
  • Fix scheduling of ClusterInfoService#refresh #59880
Authorization
  • Fix doc-update interceptor for indices with DLS and FLS #61516
  • Report anonymous roles in authenticate response #61355 (issues: #47195, #53453, #57711, #57853)
  • Add view_index_metadata privilege over metricbeat-* for monitoring agent #63750 (issue: #63750)
CRUD
  • Propagate forceExecution when acquiring permit #60634 (issue: #60359)
Cluster Coordination
  • Reduce allocations when persisting cluster state #61159
Distributed
  • Fix cluster health rest api wait_for_no_initializing_shards bug #58379
  • Fix cluster health when closing #61709
Engine
  • Fix estimate size of translog operations #59206
Features/ILM+SLM
Features/Java Low Level REST Client
  • Handle non-default port in Cloud-Id #61581
Features/Stats
  • Remove sporadic min/max usage estimates from stats #59755
Features/Watcher
  • Correct the query dsl for watching elasticsearch version #58321 (issue: #58261)
  • Fix passing params to template or script failed in watcher #58559 (issue: #57625)
Geo
Infra/Core
  • Throws IndexNotFoundException in TransportGetAction for unknown System indices #61785 (issue: #57936)
  • Handle missing logstash index exceptions #63698
  • XPack Usage API should run on MANAGEMENT threads #64160
Infra/Packaging
  • Allow running the Docker image with a non-default group #61194 (issue: #60864)
  • Set the systemd initial timeout to 75 seconds #60345 (issue: #60140)
Machine Learning
  • Adjusting inference processor to support foreach usage #60915 (issue: #60867)
  • Get data frame analytics jobs stats API can return multiple responses if more than one error #60900 (issue: #60876)
  • Do not mark the data frame analytics job as FAILED when a failure occurs after the node is shutdown #61331 (issue: #60596)
  • Improve handling of exception while starting data frame analytics process #61838 (issue: #61704)
  • Fix progress on resume after final training has completed for classification and regression. Previously, progress was shown stuck at zero for final training. #1443
  • Avoid potential "Failed to compute quantile" and "No values added to quantile sketch" log errors training regression and classification models if there are features with mostly missing values #1500
  • Correct the anomaly detection job model state min_version #1546
Mapping
  • Improve ignore_malformed handling for dates #60211 (issue: #52634)
Network
  • Let isInetAddress utility understand the scope ID on ipv6 #60172 (issue: #60115)
  • Suppress noisy SSL exceptions #61359
Search
  • Allows nanosecond resolution in search_after #60328 (issue: #52424)
  • Consolidate validation for docvalue_fields #59473
  • Correct how field retrieval handles multifields and copy_to #61309 (issue: #61033)
  • Apply boost only once for distance_feature query #63767
  • Fixed NullPointerException in significant_text aggregation when field does not exist #64144 (issue: #64045)
  • Fix async search to retry updates on version conflict #63652 (issue: #63213)
  • Fix sorted query when date_nanos is used as the numeric_type #64183 (issue: #63719)
Snapshot/Restore
  • Avoid listener call under SparseFileTracker#mutex #61626 (issue: #61520)
  • Ensure repo not in use for wildcard repo deletes #60947
  • Fix Test Failure in testCorrectCountsForDoneShards #60254 (issue: #60247)
  • Minimize cache file locking during prewarming #61837 (issue: #58658)
  • Prevent snapshots to be mounted as system indices #61517 (issue: #60522)
  • Make Searchable Snapshot’s CacheFile Lock less #63911 (issue: #63586)
  • Don’t Generate an Index Setting History UUID unless it’s supported #64213 (issue: #64152)
SQL
  • Allow unescaped wildcard (*) in LIKE pattern #63428 (issue: #55108)
  • Validate integer paramete in string functions #63728 (issue: #58923)
  • Remove filter from field_caps requests #63840 (issue: #63832)

Upgradesedit

Infra/Packaging
  • Upgrade bundled JDK to 15.0.1 and switch to AdoptOpenJDK #64253
Store
  • Upgrade to Lucene-8.7.0 #64532