This Week in Elasticsearch and Apache Lucene - 2018-08-24

Por
Paul Sanwald
•
Adrien Grand
•
Jay Modi
•
Yannick Welsch
•
Colin Goodheart-Smithe

Elasticsearch

Tokens, Kerberos, Kibana

In order for Kibana to add Kerberos support, we are adding support for the client credentials grant type to the token API, removing the need to pass username and password when using the get token API. Kibana (or another client) can proxy the Authorization header from the client to elasticsearch in a request to get a token and a token will be created that allows access as the authenticated user that made the request.

Generalising Rollups to allow similar new features

We have been working on a PR which refactors the indexing side of rollups to allow code reuse when building similar features. In particular, the ML team are looking to implement "data frames" which are very close to the idea of entity centric indexing and allow a user to "rollup" data based on something other than a time series. Data frames will introduce some new challenges on the indexing side, since incremental updates will involve updating the existing document representing the entity; whereas the current rollup indexer does not merge new data with existing data.

Cross Cluster Replication

Auto-follow patterns allow a cluster to automatically follow new indices that are created by a leader cluster, which is particularly useful for time-based indices. We have worked on a plan to implement auto-follow patterns, and we opened a first PR for this work.

Zen2

A fundamental building block of the Zen2 algorithm is the master term, which is a number that identifies the period of leadership of a particular master-eligible node, and is similar to primary terms, which track the reign of a primary shard. No two nodes may ever become master in the same term, and each node has a current term which can only ever increase. The current term is stored in the cluster state and included in transport messages, which allows each node to identify delayed messages coming from a stale master. Every node in the cluster should be in the same master term, so when a node discovers a newer term it must increase its own term to match. This is unavoidably a little disruptive: when a node leaves a term it stops following that term's elected master, but the newly-chosen term may not yet have an elected master at all, so for a period the cluster will be masterless and unable to process any cluster-state updates. The main reason for entering a new term is that the master of the current term seems to have failed, but nodes cannot individually tell the difference between an unhealthy master and an unhealthy network connecting them to a healthy master. The solution to this is a so-called pre-voting phase in which each node first asks its peers whether they would vote for it if it were to trigger an election. If pre-voting fails to obtain enough votes then most other nodes are either unreachable or following an existing, healthy master, and there is no point in starting an election. If the pre-voting phase succeeds then this means that there are enough reachable nodes that believe the master to have failed, which is a reliable indicator that it is wise to begin a new term and start a proper election. We have implemented this pre-voting phase and prepared the leader-side logic for the follow-up election.

Changes

Changes in 6.3:

  • Trim unreferenced translog when the safe commit advanced #32967

Changes in 6.4:

  • Watcher: Properly find next valid date in cron expressions #32734
  • For filters aggregations, make sure that rewrites preserve other_bucket. #32921
  • Ensure that _exists queries on keyword fields use norms when they’re available. #33006
  • Logging: Use settings when building daemon threads #32751
  • Fix quoted exists query #33019
  • Fix multi fields empty query #33017
  • Fix north pole overflow error in GeoHashUtils.bbox() #32891
  • All Translog inner closes should happen after tragedy exception is set #32674
  • Protect scheduler engine against throwing listeners #32998
  • Enable FIPS140LicenseBootstrapCheck #32903

Changes in 6.5:

  • Do NOT allow termvectors on nested fields #32728
  • [Rollup] Return empty response when aggs are missing #32796
  • Use a dedicated ConnectionManger for RemoteClusterConnection #32988
  • TESTS: Fix Random Fail in MockTcpTransportTests #33061
  • [Watcher] Improved error messages for CronEvalTool #32800
  • Search: Support of wildcard on docvalue_fields #32980
  • Change query field expansion #33020
  • SQL: skip uppercasing/lowercasing function tests for AZ locales as well #32910
  • HLRC: Create server agnostic request and response #32912
  • GraphClient for the high level REST client and associated tests. #33025
  • Add settings updater for 2 affix settings #33050
  • Allow engine to recover from translog upto a seqno #33032
  • Expose max_concurrent_shard_requests in _msearch #33016
  • Enable avoiding mmap bootstrap check #32421
  • Generalize remote license checker #32971
  • Make Geo Context Mapping Parsing More Strict (6.x version) #32862
  • Add mzn and dz to unsupported locales #32957
  • Scripted metric aggregations: add deprecation warning and system property to control legacy params #31597
  • Remove client connections from TcpTransport (#31886) #32954
  • NETWORKING: Make RemoteClusterConn. Lazy Resolve DNS #32764

Changes in 7.0:

  • Set maxScore for empty TopDocs to Nan rather than 0 #32938
  • GraphClient for the high level REST client and associated tests #32366
  • Handle 6.4.0+ BWC for Application Privileges #32929
  • BREAKING: Make Geo Context Mapping Parsing More Strict #32821
  • Fix InternalAutoDateHistogram reproducible failure #32723

Lucene

Lucene 8

We added a new "Lucene80" codec to reflect the fact that codec changes have been made in order to record impacts into skip lists.

On the Elasticsearch side, Jim removed the Elasticsearch 5.x version constants, which is required to upgrade to Lucene 8.0 since the Elasticsearch 5.x version constants referenced now unsupported Lucene 6.x version constants. Progress has also been made on the upgrade/lucene-8 branch, which is now almost free of compile errors. We will look into test failures next week.

Other