This Week in Elasticsearch and Apache Lucene - 2018-08-24
Tokens, Kerberos, Kibana
In order for Kibana to add Kerberos support, we are adding support
for the client credentials grant type to the token
API, removing the need to pass username and password when using the get
token API. Kibana (or another client) can proxy the
Authorization header from the client to elasticsearch in a request to get a token and a token will be created that allows access as the authenticated user that made the request.
Generalising Rollups to allow similar new features
We have been working on a PR which refactors the indexing side of rollups to allow code reuse when building similar features. In particular, the ML team are looking to implement "data frames" which are very close to the idea of entity centric indexing and allow a user to "rollup" data based on something other than a time series. Data frames will introduce some new challenges on the indexing side, since incremental updates will involve updating the existing document representing the entity; whereas the current rollup indexer does not merge new data with existing data.
Cross Cluster Replication
Auto-follow patterns allow a cluster to automatically follow new indices that are created by a leader cluster, which is particularly useful for time-based indices. We have worked on a plan to implement auto-follow patterns, and we opened a first PR for this work.
A fundamental building block of the Zen2 algorithm is the master term, which is a number that identifies the period of leadership of a particular master-eligible node, and is similar to primary terms, which track the reign of a primary shard. No two nodes may ever become master in the same term, and each node has a current term which can only ever increase. The current term is stored in the cluster state and included in transport messages, which allows each node to identify delayed messages coming from a stale master. Every node in the cluster should be in the same master term, so when a node discovers a newer term it must increase its own term to match. This is unavoidably a little disruptive: when a node leaves a term it stops following that term's elected master, but the newly-chosen term may not yet have an elected master at all, so for a period the cluster will be masterless and unable to process any cluster-state updates. The main reason for entering a new term is that the master of the current term seems to have failed, but nodes cannot individually tell the difference between an unhealthy master and an unhealthy network connecting them to a healthy master. The solution to this is a so-called pre-voting phase in which each node first asks its peers whether they would vote for it if it were to trigger an election. If pre-voting fails to obtain enough votes then most other nodes are either unreachable or following an existing, healthy master, and there is no point in starting an election. If the pre-voting phase succeeds then this means that there are enough reachable nodes that believe the master to have failed, which is a reliable indicator that it is wise to begin a new term and start a proper election. We have implemented this pre-voting phase and prepared the leader-side logic for the follow-up election.
Changes in 6.3:
- Trim unreferenced translog when the safe commit advanced #32967
Changes in 6.4:
- Watcher: Properly find next valid date in cron expressions #32734
- For filters aggregations, make sure that rewrites preserve other_bucket. #32921
- Ensure that _exists queries on keyword fields use norms when they’re available. #33006
- Logging: Use settings when building daemon threads #32751
- Fix quoted exists query #33019
- Fix multi fields empty query #33017
- Fix north pole overflow error in GeoHashUtils.bbox() #32891
- All Translog inner closes should happen after tragedy exception is set #32674
- Protect scheduler engine against throwing listeners #32998
- Enable FIPS140LicenseBootstrapCheck #32903
Changes in 6.5:
- Do NOT allow termvectors on nested fields #32728
- [Rollup] Return empty response when aggs are missing #32796
- Use a dedicated ConnectionManger for RemoteClusterConnection #32988
- TESTS: Fix Random Fail in MockTcpTransportTests #33061
- [Watcher] Improved error messages for CronEvalTool #32800
- Search: Support of wildcard on docvalue_fields #32980
- Change query field expansion #33020
- SQL: skip uppercasing/lowercasing function tests for AZ locales as well #32910
- HLRC: Create server agnostic request and response #32912
- GraphClient for the high level REST client and associated tests. #33025
- Add settings updater for 2 affix settings #33050
- Allow engine to recover from translog upto a seqno #33032
- Expose max_concurrent_shard_requests in _msearch #33016
- Enable avoiding mmap bootstrap check #32421
- Generalize remote license checker #32971
- Make Geo Context Mapping Parsing More Strict (6.x version) #32862
- Add mzn and dz to unsupported locales #32957
- Scripted metric aggregations: add deprecation warning and system property to control legacy params #31597
- Remove client connections from TcpTransport (#31886) #32954
- NETWORKING: Make RemoteClusterConn. Lazy Resolve DNS #32764
Changes in 7.0:
- Set maxScore for empty TopDocs to Nan rather than 0 #32938
- GraphClient for the high level REST client and associated tests #32366
- Handle 6.4.0+ BWC for Application Privileges #32929
- BREAKING: Make Geo Context Mapping Parsing More Strict #32821
- Fix InternalAutoDateHistogram reproducible failure #32723
We added a new "Lucene80" codec to reflect the fact that codec changes have been made in order to record impacts into skip lists.
On the Elasticsearch side, Jim removed the Elasticsearch 5.x version constants, which is required to upgrade to Lucene 8.0 since the Elasticsearch 5.x version constants referenced now unsupported Lucene 6.x version constants. Progress has also been made on the upgrade/lucene-8 branch, which is now almost free of compile errors. We will look into test failures next week.
- There are ongoing discussions to integrate a new Arabic snowball stemmer.
- Javadocs of the org.apache.lucene.index package need love.
- We fixed a bug where Lucene would miss to update soft deletes when carrying over hard deletes that occurred while a merge was running.
- The StoredField class has missing null checks.
- We are discussing how best to expose matches on IntervalQuery.
- The change to propagate min competitive scores to sub clauses triggered an impressive speedup on AndHighOrMedMed and recovered part of the slowdown on AndMedOrHighHigh.
- interesting performance numbers regarding optimizing reads on sparse doc values.