Elasticsearch version 7.6.0edit

Coming in 7.6.0.

Also see Breaking changes in 7.6.

Breaking changesedit

Mapping
  • Add a cluster setting to disallow loading fielddata on _id field #49166 (issues: #26472, #43599)

Breaking Java changesedit

Security
  • Support Client and RoleMapping in custom Realms #50534 (issue: #48369)

Deprecationsedit

Analysis
  • Deprecate and remove camel-case nGram and edgeNGram tokenizers #50862 (issue: #50561)
Authorization
  • Deprecating kibana_user and kibana_dashboard_only_user roles #46456
Distributed
  • Deprecate synced flush #50835 (issue: #50776)
  • Deprecate indices without soft-deletes #50502
Features/Indices APIs
  • Emit warnings when index templates have multiple mappings #50982
  • Ensure we emit a warning when using the deprecated template field. #50831 (issue: #49460)
Infra/Core
  • Deprecate the local parameter of /_cat/nodes #50499 (issue: #50088)
Reindex
Search
  • Update the signature of vector script functions. #48604
  • Deprecate the sparse_vector field type. #48315
  • Add a deprecation warning regarding allocation awareness in search request #48351 (issue: #43453)

New featuresedit

Aggregations
  • New Histogram field mapper that supports percentiles aggregations. #48580 (issue: #48578)
  • Implement stats aggregation for string terms #47468
Analysis
  • Implement Lucene EstonianAnalyzer, Stemmer #49149 (issue: #48895)
Authentication
  • Password Protected Keystore (Feature Branch) #49210
Features/ILM+SLM
Features/Ingest
Machine Learning
  • Implement precision and recall metrics for classification evaluation #49671 (issue: #48759)
  • Explain data frame analytics API #49455
  • Machine learning model inference ingest processor #49052
  • Implement accuracy metric for multi-class classification #47772 (issue: #48759)
  • Add feature importance values to classification and regression results (using tree SHapley Additive exPlanation, or SHAP) #857
Mapping
Search
  • Add fuzzy intervals source #49762 (issue: #49595)
  • Add a listener to track the progress of a search request locally #49471 (issue: #49091)

Enhancementsedit

Aggregations
  • Add reusable HistogramValue object #49799 (issue: #49683)
  • Optimize composite aggregation based on index sorting #48399 (issue: #48130)
Allocation
  • Auto-expand indices according to allocation filtering rules #48974
  • Do not cancel ongoing recovery for noop copy on broken node #48265 (issue: #47974)
  • Quieter logging from the DiskThresholdMonitor #48115 (issue: #48038)
  • Faster access to INITIALIZING/RELOCATING shards #47817 (issues: #46941, #48579)
Analysis
  • Check for deprecations when analyzers are built #50908 (issue: #42349)
  • Make Multiplexer inherit filter chains analysis mode #50662 (issue: #50554)
  • Allow custom characters in token_chars of ngram tokenizers #49250 (issue: #25894)
Authentication
  • Add Debug/Trace logging for authentication #49575 (issue: #49473)
Authorization
  • Increase Size and lower TTL on DLS BitSet Cache #50535 (issues: #43669, #49260)
  • Add monitor_snapshot cluster privilege #50489 (issue: #50210)
  • Remove reserved roles for code search #50068 (issue: #49842)
  • [Code] Remove code_admin/code_user roles #48164
  • Resolve the role query and the number of docs lazily #48036
CCR
  • Improve error message when pausing index #48915
  • Use MultiFileTransfer in CCR remote recovery #44514 (issue: #44468)
CRUD
  • print id detail when id is too long. #49433
  • Add preflight check to dynamic mapping updates #48817 (issue: #35564)
Cluster Coordination
Distributed
  • Use retention lease in peer recovery of closed indices #48430 (issue: #45136)
Engine
  • Do not force refresh when write indexing buffer #50769
  • Deleted docs disregarded for if_seq_no check #50526
  • Allow realtime get to read from translog #48843
  • Do not warm up searcher in engine constructor #48605 (issue: #47186)
  • Add a new merge policy that interleaves old and new segments on force merge #48533 (issue: #37043)
  • Refresh should not acquire readLock #48414 (issue: #47186)
Features/ILM+SLM
  • Refresh cached phase policy definition if possible on new poli… #50820 (issue: #48431)
  • Make the UpdateRolloverLifecycleDateStep retryable #50702 (issue: #48183)
  • Make InitializePolicyContextStep retryable #50685 (issue: #48183)
  • ILM retryable async action steps #50522 (issues: #44135, #48183)
  • Make the TransportRolloverAction execute in one cluster state update #50388
  • ILM open/close steps are noop if idx is open/close #48614
  • ILM Make the check-rollover-ready step retryable #48256 (issue: #44135)
Features/Ingest
  • Foreach processor - fork recursive call #50514
  • Sync grok patterns with logstash patterns #50381
  • Replace required pipeline with final pipeline #49470 (issue: #49247)
  • Add templating support to enrich processor #49093
  • Introduce on_failure_pipeline ingest metadata inside on_failure block #49076 (issue: #44920)
  • Add templating support to pipeline processor. #49030 (issue: #39955)
  • Add option to split processor for preserving trailing empty fields #48664 (issue: #48498)
  • Change grok watch dog to be Matcher based instead of thread based. #48346 (issues: #43673, #47374)
  • update ingest-user-agent regexes.yml #47807
Features/Java High Level REST Client
Features/Monitoring
  • Significantly Lower Monitoring HttpExport Memory Footprint #48854
  • Validate proxy base path at parse time #47912 (issue: #47711)
  • Validate index name time format setting at parse time #47911 (issue: #47711)
  • Validate monitoring header overrides at parse time #47848 (issue: #47711)
  • Validate monitoring username at parse time #47821 (issue: #47711)
  • Validate monitoring password at parse time #47740 (issue: #47711)
Features/Stats
Features/Watcher
  • Log attachment generation failures #50080
  • Don’t dump a stacktrace for invalid patterns when executing elasticse… #49744 (issue: #49642)
Geo
  • "CONTAINS" support for BKD-backed geo_shape and shape fields #50141 (issue: #41204)
  • Adds support for geo-bounds filtering in geogrid aggregations #50002
  • Introduce faster approximate sinh/atan math functions #49009 (issue: #41166)
  • Add IndexOrDocValuesQuery to GeoPolygonQueryBuilder #48449
Infra/Core
  • Add "did you mean" to ObjectParser #50938
  • Consistent case in CLI option descriptions #49635
  • Improve resiliency to formatting JSON in server #48553 (issue: #48450)
  • Don’t close stderr under --quiet #47208 (issue: #46900)
Infra/Packaging
  • Respect ES_PATH_CONF on package install #50158
  • Restrict support for CMS to pre-JDK 14 #49123 (issue: #46973)
  • Remove parsed JVM settings from general settings in Windows service daemon manager #49061 (issue: #48796)
  • Package the JDK into jdk.app on macOS #48765
  • Add UBI-based Docker images #48710 (issue: #48429)
Infra/Plugins
  • Report progress of multiple plugin installs #51001 (issue: #50924)
  • Allow installing multiple plugins as a transaction #50924 (issue: #50443)
Infra/Scripting
  • Scripting: ScriptFactory not required by compile #50344 (issue: #49466)
  • Scripting: Cache script results if deterministic #50106 (issue: #49466)
  • Scripting: Groundwork for caching script results #49895 (issue: #49466)
  • Scripting: add available languages & contexts API #49652 (issue: #49463)
  • Scripting: fill in get contexts REST API #48319 (issue: #47411)
  • Scripting: get context names REST API #48026 (issue: #47411)
Infra/Settings
  • Add parameter to make sure that log of updating IndexSetting be more detailed #49969 (issue: #49818)
  • Enable dependent settings values to be validated #49942
  • Do not reference values for filtered settings #48066
License
  • Add max_resource_units to enterprise license #50735
  • Add setting to restrict license types #49418 (issue: #48508)
  • Support "enterprise" license types #49223 (issue: #48510)
Machine Learning
  • Add audit warning for 1000 categories found early in job #51146 (issue: #50749)
  • Add num_top_feature_importance_values param to regression and classification #50914
  • Implement force deleting a data frame analytics job #50553 (issue: #48124)
  • Delete unused data frame analytics state #50243
  • Make each analysis report desired field mappings to be copied #50219 (issue: #50119)
  • Retry bulk indexing of state docs #50149 (issue: #50143)
  • Persist/restore state for data frame analytics classification #50040
  • Introduce randomize_seed setting for regression and classification #49990
  • Pass prediction_field_type to C++ analytics process #49861 (issue: #49796)
  • Add optional source filtering during data frame reindexing #49690 (issue: #49531)
  • Add default categorization analyzer definition to ML info #49545
  • Add graceful retry for anomaly detector result indexing failures #49508 (issue: #45711)
  • Lower minimum model memory limit value for data frame analytics jobs from 1MB to 1kB #49227 (issue: #49168)
  • Improve model_memory_limit user experience for data frame analytics jobs #44699
  • Improve performance of boosted tree training for both classification and regression #775
  • Reduce the peak memory used by boosted tree training and fix an overcounting bug estimating maximum memory usage #781
  • Stratified fractional cross validation for regression #784
  • Added geo_point supported output for lat_long function records #809, #47050
  • Use a random bag of the data to compute the loss function derivatives for each new tree which is trained for both regression and classification #811
  • Emit prediction_probability field alongside prediction field in ml results #818
  • Reduce memory usage of machine learning native processes on Windows #844
  • Reduce runtime of classification and regression #863
  • Stop early training a classification and regression forest when the validation error is no longer decreasing #875
  • Emit prediction_field_name in data frame analytics results using the type provided as prediction_field_type parameter #877
  • Improve performance updating quantile estimates #881
  • Migrate to use Bayesian optimisation for initial hyperparameter value line searches and stop early if the expected improvement is too small #903
  • Stop cross-validation early if the predicted test loss has a small chance of being smaller than for the best parameter values found so far #915
  • Optimize decision threshold for classification to maximize minimum class recall #926
  • Include categorization memory usage in the model_bytes field in model_size_stats, so that it is taken into account in node assignment decisions #927 (issue:https://github.com/elastic/ml-cpp/issues/724[#724])
Mapping
  • Add telemetry for flattened fields. #48972
Network
  • Add certutil http command #49827
  • Do not load SSLService in plugin contructor #49667 (issue: #44536)
  • Netty4: switch to composite cumulator #49478
  • Add the simple strategy to cluster settings #49414 (issue: #49067)
  • Deprecate misconfigured SSL server config #49280 (issue: #45892)
  • Improved diagnostics for TLS trust failures #48911
Percolator
  • Refactor percolator’s QueryAnalyzer to use QueryVisitors #49238 (issue: #45639)
Ranking
  • Support search_type in Rank Evaluation API #48542 (issue: #48503)
Recovery
  • Use peer recovery retention leases for indices without soft-deletes #50351 (issues: #45136, #46959)
  • Recovery buffer size 16B smaller #50100
Reindex
  • Reindex sort deprecation warning take 2 #49855 (issue: #49458)
SQL
  • SQL: Handle uberjar scenario where the ES jdbc driver file is bundled in another jar #51856 (issue: #50201)
  • SQL: add trace logging for search responses coming from server #50530
  • SQL: Add TRUNC alias for TRUNCATE #49571 (issue: #41195)
  • SQL: binary communication implementation for drivers and the CLI #48261 (issue: #47785)
  • SQL: Verify Full-Text Search functions not allowed in SELECT #51568 (issue: #47446)
Search
  • Add Validation for maxQueryTerms to be greater than 0 for MoreLikeThisQuery #49966 (issue: #49927)
  • Optimize numeric sort on match_all queries #49717 (issue: #48804)
  • Pre-sort shards based on the max/min value of the primary sort field #49092 (issue: #49091)
  • Optimize sort on long field #48804
  • Search optimisation - add canMatch early aborts for queries on "_index" field #48681 (issue: #48473)
  • #48475 Pure disjunctions should rewrite to a MatchNoneQueryBuilder #48557
  • Disable caching when queries are profiled #48195 (issue: #33298)
  • BlendedTermQuery’s equals method should consider boosts #48193 (issue: #48184)
  • Increase the number of vector dims to 2048 #46895
Security
  • Make .async-search-* a restricted namespace #50294
  • Security should not reload files that haven’t changed #50207 (issue: #50063)
Snapshot/Restore
Store
Transform
  • Improve force stop robustness in case of an error #51072
  • Add actual timeout in message #50140
  • Automatic deletion of old checkpoints #49496
  • Improve error handling of script errors #48887 (issue: #48467)
  • Add wait_for_checkpoint flag to stop #47935 (issue: #45293)

Bug fixesedit

Aggregations
  • Use #name() instead of #simpleName() when generating doc values #51920 (issues: #50307, #51847)
  • Fix a sneaky bug in rare_terms #51868 (issue: #51020)
  • Support time_zone on composite’s date_histogram #51172 (issues: #45199, #45200)
  • Fix format problem in composite of unmapped #50869 (issue: #50600)
  • SingleBucket aggs need to reduce their bucket’s pipelines first #50103 (issue: #50054)
  • Avoid precision loss in DocValueFormat.RAW#parseLong #49063 (issue: #38692)
  • Fix ignoring missing values in min/max aggregations #48970 (issue: #48905)
Allocation
  • Collect shard sizes for closed indices #50645 (issue: #33888)
  • Auto-expand replicated closed indices #48973
  • Ignore dangling indices created in newer versions #48652 (issue: #34264)
  • Handle negative free disk space in deciders #48392 (issue: #48380)
Analysis
  • Fix caching for PreConfiguredTokenFilter #50912 (issue: #50734)
  • Throw Error on deprecated nGram and edgeNGram custom filters #50376 (issue: #50360)
  • _analyze api does not correctly use normalizers when specified #48866 (issue: #48650)
Audit
Authentication
  • Preserve ApiKey credentials for async verification #51244
  • Don’t fallback to anonymous for tokens/apikeys #51042 (issue: #50171)
  • Populate User metadata with OpenIDConnect collections #50521 (issue: #50250)
  • Always return 401 for not valid tokens #49736 (issue: #38866)
  • Fix iterate-from-1 bug in smart realm order #49473
  • Remove unnecessary details logged for OIDC #48746
  • Add owner flag parameter to the rest spec #48500 (issue: #48499)
Authorization
  • Fix memory leak in DLS bitset cache #50635 (issue: #49261)
  • Validate field permissions when creating a role #50212 (issues: #46275, #48108)
  • Validate field permissions when creating a role #48108 (issue: #46275)
CCR
  • CCR should auto-retry rejected execution exceptions #49213
CRUD
  • Block too many concurrent mapping updates #51038 (issue: #50670)
  • Ensure meta and document field maps are never null in GetResult #50112 (issue: #48215)
  • Replicate write actions before fsyncing them #49746
  • Do not mutate request on scripted upsert #49578 (issue: #48670)
  • Fix Transport Stopped Exception #48930 (issue: #42612)
  • Return consistent source in updates #48707
  • Close query cache on index service creation failure #48230 (issue: #48186)
Cluster Coordination
  • Import replicated closed dangling indices #50649
  • Ignore metadata of deleted indices at start #48918
  • Make elasticsearch-node tools custom metadata-aware #48390
Discovery-Plugins
  • Make EC2 Discovery Cache Empty Seed Hosts List #50607 (issue: #50550)
  • Make EC2 Discovery Plugin Retry Requests #50550 (issue: #50462)
Distributed
  • Exclude nested documents in LuceneChangesSnapshot #51279
  • Closed shard should never open new engine #47186 (issues: #45263, #47060)
  • Fix meta version of task index mapping #50363 (issue: #48393)
Engine
  • Do not wrap soft-deletes reader for segment stats #51331 (issues: #51192, #51303)
  • Account soft-deletes in FrozenEngine #51192 (issue: #50775)
  • Account trimAboveSeqNo in committed translog generation #50205 (issue: #49970)
  • Greedily advance safe commit on new global checkpoint #48559 (issue: #48532)
  • Do not ignore exception when trim unreferenced readers #48470
Features/Features
  • Fix X-Pack SchedulerEngine Shutdown #48951
Features/ILM+SLM
  • Fix SLM check for restore in progress #50868
  • Handle failure to retrieve ILM policy step better #49193 (issue: #49128)
  • Don’t halt policy execution on policy trigger exception #49128
  • Re-read policy phase JSON when using ILM’s move-to-step API #48827
  • Don’t schedule SLM jobs when services have been stopped #48658 (issue: #47749)
  • Ensure SLM stats does not block an in-place upgrade from 7.4 #48367
  • Ensure SLM stats does not block an in-place upgrade from 7.4 #48361
  • Add SLM support to xpack usage and info APIs #48096 (issue: #43663)
  • Change policy_id to list type in slm.get_lifecycle #47766 (issue: #47765)
Features/Ingest
  • Fix ignore_missing in CsvProcessor #51600
  • Don’t overwrite target field with SetSecurityUserProcessor #51454 (issue: #51428)
  • Fix ingest simulate response document order if processor executes async #50244
  • Allow list of IPs in geoip ingest processor #49573 (issue: #46193)
  • Do not wrap ingest processor exception with IAE #48816 (issue: #48810)
  • Introduce dedicated ingest processor exception #48810 (issue: #48803)
Features/Java High Level REST Client
  • Support es7 node http publish_address format #49279 (issue: #48950)
  • Add slices to delete and update by query in HLRC #48420
  • fix incorrect comparison #48208
  • Fix HLRC parsing of CancelTasks response #47017
  • Prevent deadlock by using separate schedulers #48697 (issues: #41451, #47599)
Features/Java Low Level REST Client
  • Improve warning value extraction performance in Response #50208 (issue: #24114)
Features/Monitoring
Geo
  • Guard against null geoBoundingBox #50506 (issue: #50505)
  • Geo: Switch generated GeoJson type names to camel case (#50285) #50400 (issue: #49568)
  • Geo: Switch generated WKT to upper case #50285 (issue: #49568)
  • Fix typo when assigning null_value in GeoPointFieldMapper #49645
  • Fix handling of circles in legacy geo_shape queries #49410 (issue: #49296)
  • GEO: intersects search for geo_shape return wrong result #49017
  • Geo: improve handling of out of bounds points in linestrings #47939 (issue: #43916)
Highlighting
  • Fix invalid break iterator highlighting on keyword field #49566
Infra/Core
  • Ignore virtual ethernet devices that disappear #51581 (issue: #49914)
  • Guess root cause support unwrap #50525 (issue: #50417)
  • Allow parsing timezone without fully provided time #50178 (issue: #49351)
  • [Java.time] Retain prefixed date pattern in formatter #48703 (issue: #48698)
  • Don’t drop user’s MaxDirectMemorySize flag on jdk8/windows #48657 (issues: #44174, #48365)
  • Warn when MaxDirectMemorySize may be incorrect (Windows/JDK8 only issue) #48365 (issue: #47384)
  • [Java.time] Calculate week of a year with ISO rules #48209 (issues: #41670, #42588, #43275, #43652)
Infra/Logging
  • Slow log must use separate underlying logger for each index #47234 (issue: #42432)
Infra/Packaging
  • Extend systemd timeout during startup #49784 (issue: #49593)
Infra/REST API
  • Return 400 when handling invalid JSON #49552 (issue: #49428)
  • Slash missed in indices.put_mapping url #49468
Machine Learning
  • Fix 2 digit year regex in find_file_structure #51469
  • Validate classification dependent_variable cardinality is at least two #51232
  • Do not copy mapping from dependent variable to prediction field in regression analysis #51227
  • Handle nested and aliased fields correctly when copying mapping #50918 (issue: #50787)
  • Fix off-by-one error in ml_classic tokenizer end offset #50655
  • Improve uniqueness of result document IDs #50644 (issue: #50613)
  • Fix accuracy metric in multi-class confusion matrix #50310 (issue: #48759)
  • Fix race condition when stopping a data frame analytics jobs immediately after starting it #50276 (issues: #49680, #50177)
  • Apply source query on data frame analytics memory estimation #49517 (issue: #49454)
  • Fix r_squared eval when variance is 0 #49439
  • Blacklist a number of prediction field names #49371 (issue: #48808)
  • Make data frame analytics more robust for very short-lived analyses #49282 (issue: #49095)
  • Fixes potential memory corruption when determining seasonality #852
  • Prevent prediction_field_name clashing with other fields in machine learning results #861
  • Include out-of-order as well as in-order terms in categorization reverse searches #950 (issue:https://github.com/elastic/ml-cpp/issues/949[#949])
Mapping
  • Ensure that field collapsing works with field aliases. #50722 (issues: #32648, #50121)
  • Improve DateFieldMapper ignore_malformed handling #50090 (issues: #46675, #50081)
  • Annotated text type should extend TextFieldType #49555 (issue: #49289)
  • Ensure parameters are updated when merging flattened mappings. #48971 (issue: #48907)
Network
  • Fix TransportMasterNodeAction not Retrying NodeClosedException #51325
Percolator
  • Correctly handle MSM for nested disjunctions #50669 (issue: #50305)
  • Fix query analyzer logic for mixed conjunctions of terms and ranges #49803 (issue: #49684)
Recovery
  • Check allocation id when failing shard on recovery #50656 (issue: #50508)
  • Migrate peer recovery from translog to retention lease #49448 (issue: #45136)
  • Ignore Lucene index in peer recovery if translog corrupted #49114
Reindex
SQL
  • SQL: Fix milliseconds handling in intervals #51675 (issue: #41635)
  • SQL: Fix ORDER BY YEAR() function #51562 (issue: #51224)
  • SQL: change the way unsupported data types fields are handled #50823
  • SQL: Optimisation fixes for conjunction merges #50703 (issue: #49637)
  • SQL: Fix issue with CAST and NULL checking. #50371 (issue: #50191)
  • SQL: fix NPE for JdbcResultSet.getDate(param, Calendar) calls #50184 (issue: #50174)
  • SQL: COUNT DISTINCT returns 0 instead of NULL for no matching docs #50037 (issue: #50013)
  • Fix LOCATE function optional parameter handling #49666 (issue: #49557)
  • Fix NULL handling for FLOOR and CEIL functions #49644 (issue: #49556)
  • Handle NULL arithmetic operations with INTERVALs #49633 (issue: #49297)
  • Fix issue with GROUP BY YEAR() #49559 (issue: #49386)
  • Fix issue with CASE/IIF pre-calculating results #49553 (issue: #49388)
  • Fix issue with folding of CASE/IIF #49449 (issue: #49387)
  • Fix issues with WEEK/ISO_WEEK/DATEDIFF #49405 (issues: #48209, #49376)
  • SQL: Fix issue with mins & hours for DATEDIFF #49252
  • SQL: Failing Group By queries due to different ExpressionIds #43072 (issues: #33361, #34543, #36074, #37044, #40001, #40240, #41159, #42041, #46316)
Search
  • Fix upgrade of custom similarity #50851 (issue: #50763)
  • Fix NPE bug inner_hits #50709 (issue: #50539)
  • Collect results in a local list when notifying partial results #49828 (issue: #49778)
  • Fixes a bug in interval filter serialization #49793 (issue: #49519)
  • Correctly handle duplicates in unordered interval matching #49775
  • Correct rewritting of script_score query #48425 (issue: #48081)
  • Do not throw errors on unknown types in SearchAfterBuilder #48147 (issue: #48074)
Security
  • Always consume the body in has privileges #50298 (issue: #50288)
Snapshot/Restore
  • Fix Overly Aggressive Request DeDuplication #51270 (issue: #51253)
  • Guard Repository#getRepositoryData for exception throw #50970
  • Fix Index Deletion During Partial Snapshot Create #50234 (issues: #50200, #50202)
  • Fix Index Deletion during Snapshot Finalization #50202 (issues: #45689, #50200)
  • Fix RepoCleanup not Removed on Master-Failover #49217
  • Make FsBlobContainer Listing Resilient to Concurrent Modifications #49142 (issue: #37581)
  • Fix SnapshotShardStatus Reporting for Failed Shard #48556 (issue: #48526)
  • Cleanup Concurrent RepositoryData Loading #48329 (issue: #48122)
Transform
  • Fix mapping deduction for scaled_float #51990 (issue: #51780)
  • Fix stats can return old state information if security is enabled #51732 (issue: #51728)
  • Fail to start/put on missing pipeline #50701 (issue: #50135)
  • Fix possible audit logging disappearance after rolling upgrade #49731 (issue: #49730)
  • Do not fail checkpoint creation due to global checkpoint mismatch #48423 (issue: #48379)

Upgradesedit

Engine
  • Upgrade to Lucene 8.4.0. #50518
Infra/Packaging
  • Upgrade the bundled JDK to JDK 13.0.2 #51511