Elasticsearch version 7.15.0edit

Also see Breaking changes in 7.15.

Known issuesedit

  • Aggregations: In Elasticsearch 7.14.0–7.15.0, when a cross-cluster search (CCS) request is proxied, the memory for the aggregations on the proxy node will not be freed. The trigger is cross-cluster search using aggregations where minimize roundtrips is not effective (for example, when minimize roundtrips is explicitly disabled, or implicitly disabled when using scroll, async and point-in-time searches).

    This affects Kibana CCS aggregations because Kibana uses async search by default. This issue can also happen in all modes of remote connections configured for cross-cluster search (sniff and proxy). In sniff mode, we only connect to a subset of the remote nodes (by default 3). So if the remote node we want to send a request to is not one of those 3, we must send the request as a proxy request. The workaround is to periodically restart nodes with heap pressure.

    We have fixed this issue in Elasticsearch 7.15.1 and later versions. For more details, see #78404.

Breaking changesedit

Infra/Core
Search

Deprecationsedit

Store
  • Deprecate SimpleFS and replace it with NIOFS #75156

New featuresedit

EQL
  • Add multi-valued fields support. Part 1 #74567
  • Backwards compatibility support for multi value fields update #76605
  • Multi-value fields support #76610
Geo
  • Remove vector tile feature flag #76068
Infra/Node Lifecycle
  • Remove Node Shutdown API feature flag #76588 (issue: #70338)
Machine Learning
  • Add model_prune_window field to AD job config #75741 (issue: #1962)
Mapping
  • Unsigned longs should be compatible with index sorting #75599

Enhancementsedit

Allocation
  • Clarify allocation explain if random shard chosen #75670
  • Prefer primaries in cluster allocation explain #76220
Authentication
  • Add kibana-system service account #76449
  • Support 2b and 2y prefixes in bcrypt #76083 (issues: #51132, #56086)
  • Support password_hash for Change Password API #75500
Authorization
  • Add threshold time logging for Authorized indices #75439
  • Fix privileges for GetRollupIndexCapabilities API #75614 (issue: #74779)
  • Granting kibana_system reserved role access to "all" privileges to .internal.alerts* index #76624
Cluster Coordination
  • Distinguish timeouts/failures in follower checker #75519
  • Indicate quorumness of discovered nodes earlier #74823
  • Warn on possible master service starvation #74820
Data streams
  • Add filter support to data stream aliases #74784 (issue: #66163)
EQL
  • Minimise CCS roundtrips #76076
Engine
Geo
  • Add mvt field type format to geo fields #75367 (issue: #73872)
ILM+SLM
  • Add phase configs to ILM telemetry #74575 (issue: #69152)
  • Add validation of the number_of_shards parameter in Shrink Action of ILM #74219 (issue: #72724)
  • Allow ILM move-to-step without action or name #75435 (issue: #58128)
  • Update ILM message for out of order phases error #75099 (issue: #70336)
Infra/CLI
  • In MultiCommand, show command list when required #75993
Infra/Logging
  • Adds minimal traceparent header support to Elasticsearch #74210
Infra/Plugins
  • Add support in plugins for a separate spi classloader #76288 (issue: #74448)
Infra/Resiliency
  • Print suggestion to view log on fatal error #75418 (issue: #61767)
Infra/Scripting
  • Add Fields API to aggregation scripts and field scripts #76325
  • Add ability to allow list instance methods on the script class #76045
  • Add fields api to runtime fields contexts #76265
  • Script: Def encoding parser #74840
  • Script: Fields API for Filter context #76119
  • Script: Fields API for Sort and Score scripts #75863 (issue: #61388)
  • Script: Mangle user function names #72892 (issue: #69742)
  • Script: ulong via fields API #76519 (issue: #64361)
Ingest
  • Add support for _meta field to ingest pipelines #75905
  • Configurable media_type for mustache template encoding on append processor #76210
  • Json processor: add_to_root_conflict_strategy option #74967
  • Json processor: allow duplicate keys #74956
  • Map iteration support for ForEach processor #64062 (issue: #55215)
  • Support for wildcards and override option for dot_expander processor #74601
License
  • Add persistent licensed feature tracking #76476
Machine Learning
  • Add datafeed_config field to anomaly detection job configs #74265
  • Adding datafeed_config to job in high level rest client #75338
  • Adding new p_value scoring heuristic to significant terms aggregation #75313
  • Delete expired annotations #75617 (issue: #75572)
  • Deleting a job now deletes the datafeed if necessary #76010
  • Integrating ML with the node shutdown API #75188
  • Throttle job audit msgs if delayed data occurs for consecutive buckets #75815 (issue: #73679)
  • Speed up training of regression and classification models on very large data sets #1941
  • Improve regression and classification training accuracy for small data sets #1960
  • Prune models for split fields (by, partition) that haven’t seen data updates for a given period of time #1962
Mapping
  • Handle runtime subfields when shadowing dynamic mappings #75595 (issue: #75454)
Network
  • Drop ReceiveTimeoutTransportException stack trace #75671
  • Ensure indexing_data is compressed appropriately #76321 (issue: #73497)
  • Respond with same compression scheme received #76372 (issue: #73497)
Other
  • Add resolve_during_rolling_upgrade field to deprecation issue #74226 (issue: #73091)
Packaging
  • Accept settings in snake case in Docker image #74327 (issue: #74036)
  • Change env var prefix in Docker settings support #76192 (issues: #74327, #76148)
  • Reduce default Docker image size in 7.x #75079
  • Tighten up write permissions in Docker image #70635
Query Languages
  • Introduce ParserUtils to consolidate code #76399
Recovery
  • Add peer recoveries using snapshot files when possible #76237 (issue: #73496)
  • Add peer recovery planners that take into account available snapshots #75840
  • Compute latest snapshot directly in TransportGetShardSnapshotAction #76254 (issue: #73496)
  • Keep track of data recovered from snapshots in RecoveryState #76499 (issue: #73496)
Reindex
  • Log reindex bulk request completion #75935
Search
  • Add index disk usage API #74051 (issue: #74051)
  • Allow specifying index in pinned queries #74873 (issue: #67855)
  • Compress async search responses before storing #74766 (issue: #67594)
  • Emit multiple fields from a runtime field script #75108 (issue: #68203)
  • Replaced field_masking_span occurrences with respective ParseField #74718 (issue: #63527)
  • Search performance - better caching logic for queries on wildcard field #76035 (issue: #75848)
  • Support search slicing with point-in-time #74457 (issue: #65740)
  • Track Lucene field usage #74227 (issue: #73944)
  • Unsigned long should accept value with zero fraction #75646 (issue: #67565)
Security
  • A new search API for API keys - core search function #75335 (issues: #71023, #73705)
  • Add sort and pagination support for QueryApiKey API #76144 (issue: #71023)
  • HLRC support for query API key API #76520 (issues: #71023, #75335, #76144)
  • Query API key - Rest spec and yaml tests #76238 (issue: #71023)
  • Return file-backed service tokens from all nodes #75200 (issue: #74530)
  • OnChange listener now only runs when there is actual change #75207 (issue: #44701)
Snapshot/Restore
  • Add the ability to fetch the latest successful shard snapshot #75080 (issue: #73496)
  • Fix SnapshotInfo#toXContent delegation #75899
  • Implement Numeric Offset Parameter in Get Snapshots API #76233 (issue: #74350)
  • Include extra snapshot details in logs/APIs #75917
  • Prevent searchable snapshots indices to be shrunk/split #75227 (issue: #56595)
  • Return Total Result Count and Remaining Count in Get Snapshots Response #76150
Stats
  • Adding shard count to node stats api #75433
  • Aggregate data tier index stats separately from node stats #76322
Store
  • Replace internal usages of SimpleFS with NIOFS #74996
TLS
  • Add KeyUsage, ExtendedKeyUsage, CipherSuite & Protocol to SSL diagnos… #65634 (issue: #63784)
  • Solved issue "certutil: large passwords not set" #30944 #36689
Task Management
  • Adding the option to abort persistent tasks locally #74115
  • Include reason in cancellation exceptions #75332 (issue: #74825)
Transform
  • Align transform checkpoint range with date_histogram interval for better performance #74004 (issue: #62746)
  • Changing hard coded 10k page size limit to 65k #74651 (issue: #57719)
  • Improve performance by using point in time API for search #74984 (issue: #73481)
  • Optimize composite agg execution using ordered groupings #75424

Bug fixesedit

Aggregations
  • Fix docCountError calculation for multiple reduces #76391 (issues: #43874, #75667)
  • Fix wrong error upper bound when performing incremental reductions #43874 (issue: #40005)
Allocation
  • Active masternode crash caused by datanode Input/output error #76480 (issues: #1, #436)
Distributed
  • Wrap exception with SendRequestTransportException instead of TransportException #73551
Geo
  • Alias field does not work with geo_shape query #74895
  • Use contains relation for geo_point intersection over geo_point field #75272 (issue: #75103)
ILM+SLM
  • Changing wait_for_snapshot to check start time rather than finish time #75644
Indices APIs
  • Fix pending deletes timeout log message #75776
Infra/Core
  • Add system data streams to feature state snapshots #75902 (issue: #75860)
  • Fix for JarHell Bootstrap Check can yield false positives #76217 (issue: #75701)
Infra/Scripting
  • Fix the error message for instance methods that don’t exist #76512
Ingest
  • Execute EnrichPolicyRunner on a non dedicated master node #76881 (issue: #70436)
Machine Learning
  • Allow for larger models in the inference step for data frame analytics #76116 (issue: #76093)
  • Ignore exceptions while opening job after SIGTERM to JVM #75850
  • Prevent accidentally asking for more resources when scaling down and improve scaling size estimations #74691 (issue: #74709)
  • When setting upgrade mode ensure that internal actions don’t throw unnecessary permissions errors #75880
  • Fix potential "process stopped unexpectedly: Fatal error" for training regression and classification models #1997 (issue: #1956)
Network
  • Ensure replica requests are marked as index_data (#75008) #75014 (issue: #73497)
  • Fix memory/breaker leaks for outbound responses (#76474) #76536 (issues: #65921, #76474)
Search
  • Remove invalid path from Open PIT rest spec #77609
  • Skip noop normalizing #74656
Security
  • Don’t remove warning headers on all failure #76434 (issue: #75739)
Snapshot/Restore
  • Fix Get Snapshots Request Cancellation with ignore_unavailable=true #78004 (issue: #77980)
  • Fix Spurious Warnings During Snapshot Delete #75911
  • Include full blob key in Azure read operations exceptions #75920 (issue: #75916)
  • Not throwing FileNotFoundException if BlobContainer has been deleted #75991
Transform
  • Fix potential deadlock when using stop_at_checkpoint #76034 (issue: #75846)

Upgradesedit

Ingest
  • Upgrade Tika to 1.27 for ingest #75191
Network
  • Upgrade to Netty 4.1.66 #76135
Query Languages
Snapshot/Restore
  • Upgrade GCS SDK to 1.117.1 #74938