Elasticsearch version 6.6.0edit

Also see Breaking changes in 6.6..

Breaking changesedit

Cross-cluster replication
  • Change get autofollow patterns API response format #36203 (issue: #36049)
Ranking
  • Forbid negative scores in function_score query #35709 (issue: #33309)
Search
  • Remove the distinction between query and filter context in QueryBuilders #35354 (issue: #35293)
  • Throw a parsing exception when boost is set in span_or query (#28390) #34112 (issue: #28390)
Watcher

Breaking Java changesedit

Infra/Core
  • Drop settings member from AbstractComponent #35083
Infra/Logging
Java High Level REST Client
  • HLRC XPack Protocol clean up: Migration; Graph; Watcher #34639 (issue: #34451)
ZenDiscovery
  • Make node field in JoinRequest private #36405

Deprecationsedit

Aggregations
Core
  • Deprecate use of scientific notation in epoch time parsing #36691
  • Add backcompat for joda time formats #36531
Features
  • Deprecation info API: fix value for index.shard.check_on_startup #36458 (issues: #33194, #36024)
  • Deprecation info API: negative index.unassigned.node_left.delayed_timeout #36454 (issues: #26828, #36024)
Mapping
  • Add warning about upcoming expanded fields limit #34906 (issue: #26541)
Scripting
  • Adds deprecation logging to ScriptDocValues#getValues. #34279 (issue: #22919)
Search
  • Warn in multi-search on unknown keys in meatdata #36104 (issue: #35938)
  • Deprecate the _termvector endpoint. #36098 (issue: #8484)
  • Deprecate _source_include and _source_exclude url parameters #33475 (issue: #22792)
Security
Settings
  • Deprecate setting index.optimize_auto_generated_id in 6.x #28862 (issue: #27600)
Watcher
  • Watcher deprecate notification service settings #36403
  • Undeprecate /_watcher endpoints #36269

New featuresedit

Aggregations
Analysis
  • Add support for inlined user dictionary in Nori #36123 (issue: #35842)
  • Add a prebuilt ICU Analyzer #34958 (issue: #34285)
Index Lifecycle Management
  • Adds Index lifecycle feature #35193
Java High Level REST Client
Java Low Level REST Client
  • Make warning behavior pluggable per request #36345
  • Add PreferHasAttributeNodeSelector #36005
Geo
  • Fork Lucene’s LatLonShape Classes to local lucene package #36794
  • Integrate Lucene’s LatLonShape (BKD Backed GeoShapes) as default geo_shape indexing approach #35320 (issue: #32039)
Machine learning
  • Store job configuration information in the new .ml-config index #36698 (issue: #32905)
  • Determine when data is missing from a bucket due to ingest latency #35387 (issue: #35131)
Search
  • Added soft limit to open scroll contexts #25244 #36009 (issue: #25244)
  • Make lucene’s IntervalQuery available via the Query DSL #32406 (issue: #29636)
SQL

Enhancementsedit

Aggregations
  • Enforce max_buckets limit only in the final reduction phase #36152 (issues: #32125, #35921)
  • Histogram aggs: add empty buckets only in the final reduce step #35921
  • Handles exists query in composite aggs #35758
  • Added parent validation for auto date histogram #35670
  • Allow unmapped fields in composite aggregations #35331 (issue: #35317)
  • Refactor children aggregator into a generic ParentJoinAggregator #34845 (issue: #34210)
  • Add parent-aggregation to parent-join module #34210 (issue: #9705)
Analysis
  • Ensure TokenFilters only produce single tokens when parsing synonyms #34331 (issue: #34298)
Audit
  • Add "request.id" to file audit logs #35536
Authentication
  • Invalidate Token API enhancements - HLRC #36362
  • Add DEBUG/TRACE logs for LDAP bind #36028
  • Add Tests for findSamlRealm #35905
  • Add realm information for Authenticate API #35648
  • Formal support for "password_hash" in Put User #35242 (issue: #34729)
  • Enhance Invalidate Token API #35388 (issues: #34556, #35115)
Authorization
  • Add origin_address to authentication_success #36409
  • Improve exact index matching performance #36017
  • manage_token privilege for kibana_system #35751
  • Grant .tasks access to kibana_system role #35573
  • Native roles store uses mget to retrieve roles #33531 (issue: #33205)
Build
  • Sounds like typo in exception message #35458
  • Improve validation of yaml suites #34957 (issue: #34735)
  • Enforce skip headers when needed #34735 (issue: #34650)
  • Improve validation of do sections #34734 (issue: #34651)
Core
  • Override the JVM DNS cache policy #36570
  • Added wait_for_metadata_version parameter to cluster state api. #35535
  • Extract RunOnce into a dedicated class #35489
Cross-cluster replication
  • Add time since last auto follow fetch to auto follow stats #36542 (issues: #33007, #35895)
  • Clean followed leader index UUIDs in auto follow metadata #36408 (issue: #33007)
  • Change AutofollowCoordinator to use wait_for_metadata_version #36264 (issues: #33007, #35895)
  • Refactor AutoFollowCoordinator to track leader indices per remote cluster #36031 (issues: #33007, #35895)
  • Refactor auto follow coordinator #35895 (issue: #33007)
CRUD
  • Document Seq No powered optimistic concurrency control #37284 (issues: #10708, #36148)
  • Rename seq# powered optimistic concurrency control parameters to ifSeqNo/ifPrimaryTerm #36757 (issues: #10708, #36148)
  • Expose Sequence Number based Optimistic Concurrency Control in the rest layer #36721 (issues: #10708, #36148)
  • Add doc’s sequence number + primary term to GetResult and use it for updates #36680 (issues: #10708, #36148)
  • Add seq no powered optimistic locking support to the index and delete transport actions #36619 (issues: #10708, #36148)
Distributed
  • Allow asynchronous block operations to be delayed in IndexShardOperationPermits #35999 (issues: #35540, #35850)
  • TransportResyncReplicationAction should not honour blocks #35795 (issues: #35332, #35597)
  • Add global and index level blocks to IndexSettings #35695 (issues: #35332, #35597)
  • Expose all permits acquisition in IndexShard and TransportReplicationAction #35540 (issue: #33888)
  • Add a java level freeze/unfreeze API #35353 (issues: #34352, #34357)
  • Check blocks while having index shard permit in TransportReplicationAction #35332 (issue: #33888)
  • Apply masterNodeTimeout to MasterNodeRequest transmission #35235
Engine
  • Add sequence numbers based optimistic concurrency control support to Engine #36467 (issues: #10708, #36148)
  • Require soft-deletes when access changes snapshot #36446
  • Use delCount of SegmentInfos to calculate numDocs #36323
  • Always configure soft-deletes field of IndexWriterConfig #36196 (issue: #36141)
  • Always return false from refreshNeeded on ReadOnlyEngine #35837 (issue: #35785)
  • Add a _freeze / _unfreeze API #35592 (issue: #34352)
  • Engine.newChangesSnapshot may cause unneeded refreshes if called concurrently #35169
  • Do not alloc full buffer for small change requests #35158
  • Add IndexShardOperationPermits.asyncBlockOperations(ActionListener<Releasable>) #34902 (issue: #33888)
  • Add a frozen engine implementation #34357 (issue: #34352)
Features
  • Deprecation check for discovery configuration #36666 (issue: #36024)
  • Simplify deprecation issue levels #36326
  • Deprecation check for tribe node #36240 (issue: #36024)
  • Deprecation check for : in Cluster/Index name #36185 (issue: #36024)
  • Deprecation check for renamed bulk threadpool settings #36662 (issue: #36024)
  • Deprecation check for audit log prefix settings #36661 (issue: #36024)
  • Deprecation check for classic similarity #36577 (issue: #36024)
  • Deprecation check for HTTP pipelining #36521 (issue: #36024)
  • Deprecation check for index threadpool #36520 (issue: #36024)
  • Deprecation check for percolator.map_unmapped_fields_as_string #36460 (issue: #36024)
  • Deprecation check for http.enabled setting #36394 (issues: #29601, #36024)
  • Deprecation check for File Discovery plugin #36190 (issue: #36024)
Geo
  • Adds a name of the field to geopoint parsing errors #36529 (issue: #15965)
  • Add support to ShapeBuilders for building Lucene geometry #35707 (issue: #35320)
Ingest
  • Make the ingest-geoip databases even lazier to load #36679
  • ingest: grok fix duplicate patterns JAVACLASS and JAVAFILE #35886
Java High Level REST Client
Java Low Level REST Client
  • On retry timeout add root exception #25576
Machine Learning
  • Create the machine learning annotations index #36731 (issue: #33376)
  • Add cluster setting to enable/disable config migration #36700 (issue: #32905)
  • Enable the use of endpoints starting with _ml instead of _xpack/ml #36373 (issue: #36315)
  • Add audits when deprecation warnings occur while datafeeds start #36233
  • Add lazy parsing for DatafeedConfig:Aggs,Query #36117
  • Add support for rollup indexes in datafeeds #34654
Monitoring
Network
  • Unify transport settings naming #36623
  • Move compression config to ConnectionProfile #35357 (issue: #34483)
  • Simplify Transport Compression Setting #34959 (issue: #33844)
  • Allow to enable pings for specific remote clusters #34753 (issues: #30247, #34405)
Packaging
  • Introduce Docker images build #36246
  • Move creation of temporary directory to Java #36002 (issue: #31003)
  • Update procrun executables to version 1.1.0 #35147
Plugins
  • Plugin install: don’t print download progress in batch mode #36361
Recovery
  • Exposed engine must include all operations below global checkpoint during rollback #36159 (issue: #32867)
  • Use soft-deleted docs to resolve strategy for engine operation #35230 (issues: #0, #1, #33656, #34474)
  • Put a fake allocation id on allocate stale primary command #34140 (issue: #33432)
Rollup
  • Add non-X-Pack centric rollup endpoints #36383 (issues: #35958, #35962)
  • Add more diagnostic stats to job #35471
  • Add default fields to job configs #34831
  • Add wait_for_completion option to StopRollupJob API #34811 (issue: #34574)
Scripting
  • Update joda compat methods to use compat class #36654
  • [Painless] Add boxed type to boxed type casts for method/return #36571
  • [Painless] Add def to boxed type casts #36506
  • [Scripting] Make Max Script Length Setting Dynamic #35184 (issue: #23209)
  • [Painless] Add instance bindings #34410
Search
  • Use SearchRequest copy constructor in ExpandSearchPhase #36772 (issue: #36641)
  • Add copy constructor to SearchRequest #36641 (issue: #32125)
  • Add raw sort values to SearchSortValues transport serialization #36617 (issue: #32125)
  • Warn when using use_dis_max in multi_match #36614 (issue: #36488)
  • Add sort and collapse info to SearchHits transport serialization #36555 (issue: #32125)
  • Add default methods to DocValueFormat #36480
  • Extend field caps API to mark meta fields #36309
  • Respect indices options on _msearch #35887
  • Allow efficient can_match phases on frozen indices #35431 (issues: #34352, #34357)
  • Apply ignore_throttled also to concrete indices #35335 (issue: #34354)
  • Adapt field limit deprecation warning #35302 (issue: #35284)
  • Upgrade 6.x to lucene-7.6.0-snapshot-f9598f335b #35225
  • Prevent throttled indices to be searched through wildcards by default #34354 (issues: #33732, #34352)
  • check for null argument is already done in splitStringByCommaToArray #34268
  • has_parent builder: exception message/param fix #31182
Security
  • Option to use endpoints starting with _security #36379 (issue: #36293)
  • Make credentials mandatory when launching xpack/migrate #36197 (issues: #29847, #33972)
  • Make credentials mandatory when launching x-pack/migrate #33972 (issue: #29847)
Snapshot/Restore
  • Allow Parallel Restore Operations #36397
  • Repo creation out of ClusterStateTask #36157 (issue: #9488)
  • Add read-only repository verification #35731 (issue: #35703)
SQL
  • Make FULL non-reserved keyword in the grammar #37377 (issue: #37376)
  • Extend the ODBC metric by differentiating between 32 and 64bit platforms #36753 (issue: #36740)
  • Fix wrong appliance of StackOverflow limit for IN #36724 (issue: #36592)
  • Introduce NOW/CURRENT_TIMESTAMP function #36562 (issue: #36534)
  • Move requests' parameters to requests JSON body #36149 (issue: #35992)
  • Make INTERVAL millis optional #36043 (issue: #36032)
  • Implement data type verification for conditionals #35916 (issue: #35907)
  • Implement GREATEST and LEAST functions #35879 (issue: #35878)
  • Implement null safe equality operator <=> #35873 (issue: #35871)
  • SYS COLUMNS returns ODBC specific schema #35870 (issue: #35376)
  • Polish grammar for intervals #35853
  • Add filtering to SYS TYPES #35852 (issue: #35342)
  • Implement NULLIF(expr1, expr2) function #35826 (issue: #35818)
  • Lock down JDBC driver #35798 (issue: #35437)
  • Implement NVL(expr1, expr2) #35794 (issue: #35782)
  • Implement ISNULL(expr1, expr2) #35793 (issue: #35781)
  • Implement IFNULL variant of COALESCE #35762 (issue: #35749)
  • XPack FeatureSet functionality #35725 (issue: #34821)
  • Perform lazy evaluation of mismatched mappings #35676 (issues: #35659, #35675)
  • Improve validation of unsupported fields #35675 (issue: #35673)
  • Move internals from Joda to java.time #35649 (issue: #35633)
  • Improve CircuitBreaker logic for SqlParser #35300 (issue: #35299)
  • Upgrade jline to version 3.8.2 #35288
  • New SQL CLI logo #35261
  • Introduce Coalesce function #35253 (issue: #35060)
  • Optimizer rule for folding nullable expressions #35080 (issue: #34826)
  • Improve painless script generated from IN #35055 (issue: #34750)
  • Implement CAST between STRING and IP #34949 (issue: #34799)
  • Fix function args verification and error msgs #34926 (issues: #33469, #34752)
  • Handle X-Pack or X-Pack SQL not being available in a more graceful way #34736 (issue: #30009)
Stats
  • Handle OS pretty name on old OS without OS release #35453 (issue: #35440)
  • Add more detailed OS name on Linux #35352
Task Management
  • Periodically try to reassign unassigned persistent tasks #36069 (issue: #35792)
  • Only require task permissions #35667 (issue: #35573)
  • Retry if task can’t be written #35054 (issue: #33764)

Bug fixesedit

Aggregations
  • Fix MultiValuesSourceFieldConfig toXContent #36525 (issue: #36474)
  • Cache the score of the parent document in the nested agg #36019 (issues: #34555, #35985)
  • Correct implemented interface of ParsedReverseNested #35455 (issue: #35449)
  • Handle IndexOrDocValuesQuery in composite aggregation #35392
  • Preserve format when aggregation contains unmapped date fields #35254 (issue: #31760)
  • Check self references in metric agg after last doc collection (#33593) #34001
Audit
  • Fix origin.type for connection_* events #36410
  • Fix deprecation of audit log settings #36175 (issue: #36162)
  • Fix IndexAuditTrail rolling restart on rollover edge #35988 (issue: #33867)
Authentication
Build
  • Use explicit deps on test tasks for check #36325
  • Fix jdbc jar pom to not include deps #36036 (issue: #32014)
  • Fix official plugins list #35661 (issue: #35623)
Circuit Breakers
  • Modify BigArrays to take name of circuit breaker #36461 (issue: #31435)
Core
  • Revert back to joda’s multi date formatters #36814 (issues: #36447, #36602)
  • Fix CompositeBytesReference#slice to not throw AIOOBE with legal offsets. #35955 (issue: #35950)
  • Suppress CachedTimeThread in hot threads output #35558 (issue: #23175)
  • Upgrade to Joda 2.10.1 #35410 (issue: #33749)
  • XContent: Check for bad parsers #34561 (issue: #34351)
Cross-cluster replication
  • Add fatal_exception field for ccr stats in monitoring mapping #37563
  • When removing an AutoFollower also mark it as removed. #37402 (issue: #36761)
  • Make shard follow tasks more resilient for restarts #37239 (issue: #37231)
  • Resume follow Api should not require a request body #37217 (issue: #37022)
  • Report error if auto follower tries auto follow a leader index with soft deletes disabled #36886 (issue: #33007)
  • Remote cluster license checker and no license info. #36837 (issue: #36815)
  • Make CCR resilient against missing remote cluster connections #36682 (issues: #36255, #36667)
  • Fix follow stats API’s follower index filtering feature #36647
  • AutoFollowCoordinator and follower index already created #36540 (issue: #33007)
  • AutoFollowCoordinator should tolerate that auto follow patterns may be removed #35945 (issue: #35937)
  • Only auto follow indices when all primary shards have started #35814 (issue: #35480)
  • Avoid NPE in follower stats when no tasks metadata #35802
  • Fix the names of CCR stats endpoints in usage API #35438
CRUD
  • Synchronize WriteReplicaResult callbacks #36770
  • Fix DeleteRequest validation for nullable or empty id/type #35314 (issue: #35297)
  • Fix UpdateRequest.fromXContent #35257 (issues: #29293, #34069)
Distributed
  • Combine the execution of an exclusive replica operation with primary term update #36116 (issue: #35850)
  • ActiveShardCount should not fail when closing the index #35936
Engine
  • Wrap can_match reader with ElasticsearchDirectoryReader #35857
  • Copy checkpoint atomically when rolling generation #35407
Features
  • Handle Null in FetchSourceContext#fetchSource #36839 (issue: #29293)
Geo
  • More robust handling of ignore_malformed in geoshape parsing #35603 (issues: #34047, #34498)
  • Better handling of malformed geo_points #35554 (issue: #35419)
  • Enables coerce support in WKT polygon parser #35414 (issue: #35059)
  • Further improve robustness of geo shape parser for malformed shapes #34498 (issues: #31449, #34047)
Index lifecycle management
  • Remove indexing_complete when removing policy #36620
Index APIs
  • Fix duplicate phrase in shrink/split error message #36734 (issue: #36729)
  • Make XContentBuilder in AliasActions build is_write_index field #35071
  • Raise a 404 exception when document source is not found (#33384) #34083 (issue: #33384)
Ingest
  • Fix on_failure with Drop processor #36686 (issue: #36151)
  • Support default pipelines + bulk upserts #36618 (issue: #36219)
  • Support default pipeline through an alias #36231 (issue: #35817)
  • Dot_expander_processor prevent null add/append to source document #35106
Machine Learning
  • Fix cause of "Sample out of bounds" error message. #335
  • Fix hang when closing a job or creating a forecast. This problem occurs if you created a forecast for a large job and temporary storage was not cleaned up. #352 (issue: #350)
  • Wait for autodetect to be ready in the datafeed #37349 (issues: #36810, #37227)
  • Stop datafeeds when their jobs are stale #37227 (issue: #36810)
  • Order get job stats API response by job id #36841 (issue: #36683)
Mapping
  • Make sure to accept empty unnested mappings in create index requests. #37089
Monitoring
  • Add missing error type mapping for apm-server #36178 (issue: #1614)
Network
  • Do not resolve addresses in remote connection info #36671 (issue: #35658)
  • Always compress based on the settings #36522 (issue: #36399)
  • http.publish_host should contain CNAME #32806 (issue: #22029)
Packaging
  • Fix error message when package install fails due to missing Java #36077 (issue: #31845)
  • Add missing entries to conffiles #35810 (issue: #35691)
Ranking
  • QueryRescorer should keep the window size when rewriting #36836
  • Fix a bug in function_score queries where we use the wrong boost_mode. #35148 (issue: #35123)
Recovery
  • Register ResyncTask.Status as a NamedWriteable #36610
Rollup
  • Fix Rollup’s metadata parser #36791 (issue: #36726)
  • Fix rollup search statistics #36674
  • Improve handling of failures on first search #35269
  • Proactively resolve index patterns in RollupSearch endoint #34930 (issue: #34828)
Scripting
  • Properly support no-offset date formatting #36316 (issue: #36306)
  • [Painless] Generate Bridge Methods #36097
  • Fix serialization bug in painless execute api request #36075 (issue: #36050)
  • Actually add joda time back to whitelist #35965 (issue: #35915)
  • Add back joda to whitelist #35915 (issue: #35913)
  • [Painless] Partially fixes def boxed types casting #35563 (issue: #35351)
  • Add back lookup vars in score script #34833
Search
  • Inner hits fail to propagate doc-value format. (#36310) #36355 (issue: #36310)
  • Fix custom AUTO issue with Fuzziness#toXContent #35807 (issue: #33462)
  • Fix analyzed prefix query in query_string #35756 (issue: #31702)
  • Fix problem with MatchNoDocsQuery in disjunction queries #35726 (issue: #34708)
  • Fix phrase_slop in query_string query #35533 (issue: #35125)
  • Add a More Like This query routing requirement check (#29678) #33974
Security
Settings
  • Fix setting by time unit #37192
  • Fix handling of fractional byte size value settings #37172
  • Fix handling of fractional time value settings #37171
  • Correctly Identify Noop Updates #36560 (issue: #36496)
Snapshot/Restore
  • Improve Resilience SnapshotShardService #36113 (issue: #32265)
  • Register Azure max_retries setting #35286
  • Restore Should Check Min. Version #34676 (issue: #34264)
SQL
  • Fix issue with field names containing "." #37364 (issue: #37128)
  • Proper handling of COUNT(field_name) and COUNT(DISTINCT field_name) #37254 (issue: #30285)
  • Fix COUNT DISTINCT filtering #37176 (issue: #37086)
  • Fix issue with wrong NULL optimization #37124 (issue: #35872)
  • Count distinct doesn’t recognize that a string field has a keyword version #37176 (issue: #37087)
  • Handle the bwc Joda ZonedDateTime scripting class in Painless #37024 (issue: #37023)
  • Fix bug regarding histograms usage in scripting #36866
  • Fix issue with always false filter involving functions #36830 (issue: #35980)
  • Protocol returns ISO 8601 String formatted dates instead of Long for JDBC/ODBC requests #36800 (issue: #36756)
  • Fix translation of LIKE/RLIKE keywords #36672 (issues: #36039, #36584)
  • Scripting support for casting functions CAST and CONVERT #36640 (issue: #36061)
  • Fix translation to painless for conditionals #36636 (issue: #36631)
  • Concat should be always not nullable #36601 (issue: #36169)
  • Fix MOD() for long and integer arguments #36599 (issue: #36364)
  • Fix issue with complex HAVING and GROUP BY ordinal #36594 (issue: #36059)
  • Be lenient for tests involving comparison to H2 but strict for csv spec tests #36498 (issue: #36483)
  • Non ISO 8601 versions of DAY_OF_WEEK and WEEK_OF_YEAR functions #36358 (issue: #36263)
  • Do not ignore all fields whose names start with underscore #36214 (issue: #36206)
  • SUM() and LIKE condition doesn’t work anymore #36672 (issue: #36161)
  • Fix issue with wrong data type for scripted Grouping keys #35969 (issue: #35662)
  • Fix translation of math functions to painless #35910 (issue: #35654)
  • Build: Fix jdbc jar to include deps #35602
  • Fix query translation for scripted queries #35408 (issue: #35232)
  • Clear the cursor if nested inner hits are enough to fulfill the query required limits #35398 (issue: #35176)
  • Fix null handling for AND and OR in SELECT #35277 (issue: #35240)
  • Handle null literal for AND and OR in WHERE #35236 (issue: #35088)
  • Introduce NotEquals node to simplify expressions #35234 (issues: #35210, #35233)
  • Introduce IsNull node to simplify expressions #35206 (issues: #34876, #35171)
  • Handle wildcard expansion on incorrect fields #35134 (issue: #35092)
  • Fix null handling for IN ⇒ painless script #35124 (issues: #35108, #35122)
  • Register missing processors #35121 (issue: #35119)
  • Fix NPE thrown if HAVING filter evals to null #35108 (issue: #35107)
  • Proper handling of nested fields at the beginning of the columns list #35068 (issue: #32951)
  • Fix incorrect AVG data type #34948 (issue: #33773)
  • Add CAST and CONVERT to SHOW FUNCTIONS #34940 (issue: #34939)
  • Handle aggregation for null group #34916 (issue: #34896)
  • Provide null-safe scripts for Not and Neg #34877 (issue: #34848)
  • Return error with ORDER BY on non-grouped. #34855 (issue: #34590)
  • Fix negation of equals comparison. #34680 (issue: #34558)
  • CAST doesn’t work in ORDER BY. #36640 (issue: #34557)
  • COUNT(column) takes into account NULLs #37254 (issue: #34549)
Watcher

Regressionsedit

Scripting
  • Use Number as a return value for BucketAggregationScript #35653 (issue: #35351)

Upgradesedit

Network