This Week in Elasticsearch and Apache Lucene - 2018-10-19


User Privileges API

We merged the new get user privileges api. This api will be used by Kibana to retrieve a list of resources patterns from ES security, and use that to determine whether a user has any Kibana space access.

API Keys

In order to support hadoop jobs and the Kibana task manager, we need a way to represent a user that has a much longer expiration than the current tokens that we provide. The plan to support this is to add support for api keys, which can have an expiration and be "scoped" using a role definition. We raised a PR to add support for creating these keys, which will be incorporated into a feature branch as we continue development.


To provide high availability in a cluster where master-eligible nodes are failing, being replaced or retired, a Zen2 master auto-adapts (a.k.a. reconfigures) the set of master-eligible nodes that have a voting right in future leader elections. We have implemented the reconfiguration rules, integrated the auto-reconfiguration logic, and added tests to show that the system always converges in a reasonable time to the optimal configuration.


In a situation where a primary shard in a following index fails on the following cluster, the new primary will end up retrying some recent operations, which may cause duplicate operations to be processed. We added support for dealing with these duplicates, which requires special handling in the follower replication logic to ensure that all shard copies have the same term for this retried operation.

We made some smaller API changes and added support for keeping persistent tasks around after they have failed to keep track of the fatal error. This helps users figure out what error caused a shard follow task to fail having to find a log entry on the specific node that task ran on. We also added a new stat to the CCR stats api that reports the total fetch time from the perspective of the leader cluster. Currently the stats api already reports the total fetch time from the perspective of the follower cluster. This new stat in combination with the existing total fetch time stat turned out to be valuable in order to determine whether the time is spent on collecting the operations to replicate in leader cluster or sending these operations over the network to the follower cluster.

Frozen indices

We laid out a plan for implementing frozen indices and opened PRs for the first steps, a search_throttled threadpool, additional logic to prevent search_throttled indices to be searched though wildcards by default and a frozen engine implementation that allows us to release resources after searches complete.


Changes in 5.6:

  • Add a limit for graph phrase query expansion #34061
  • X-Pack:
    • [ML] Add missing return after calling listener #4470

Changes in 6.5:

  • Fill LocalCheckpointTracker with Lucene commit #34474
  • HLRC: Delete role API #34620
  • BREAKING: Geo: Don’t flip longitude of envelopes crossing dateline #34535
  • SQL: Introduce support for NULL values (#34573) #34640
  • Fix completion suggester’s score tie-break #34508
  • BREAKING: HLRC XPack Protocol clean up: Licence, Misc #34469
  • [HLRC] Add Start/Stop Watch Service APIs. #34317
  • Rollup add default metrics to histo groups #34534
  • Don’t omit default values when updating routing exclusions (#32721) #33638
  • Check stemmer language setting early #34601
  • Core: Parse floats in epoch millis parser #34504
  • Add GetRollupCaps API to high level rest client #32880
  • Handle missing user in user privilege APIs #34575
  • [HLRC] Add support for Delete role mapping API #34531
  • [Rollup] Add support for date histo format when searching #34537
  • SQL: return constants for all matching records in constants-containing SELECTs #34576
  • Add get-user-privileges API #33928
  • Backport Structured Audit Logging #33894
  • Ensure map keys cannot be self referencing #34569
  • Pass the host name on as server_name if proxy mode is on #34559
  • add start trial API to HLRC #33406
  • Add Debug/Trace logging to token service #34022
  • HLRC: Add support for XPack Post Start Basic Licence API #33606
  • JDBC: Fix artifactId in pom #34478
  • Create/Update role mapping API #34171
  • Deprecate negative query boost (#34486) #34512
  • HLRC: Get SSL Certificates API #34135
  • Rollup adding support for date field metrics (#34185) #34200
  • [Monitoring] Add additional necessary mappings for apm-server #34392
  • Lock down Engine.Searcher #34363
  • Introduce index settings version #34429
  • HLRC: Fix the tags to be in sync with docs. #34501
  • Do not update number of replicas on no indices #34481
  • BREAKING: Use RoleRetrievalResult for better caching #34197
  • Fix handling of empty keyword in terms aggregation #34457

Changes in 7.0:

  • SQL: Introduce support for NULL values #34573
  • BREAKING: Fix threshold frequency computation in Suggesters #34312
  • Remove hand-coded XContent duplicate checks #34588
  • Watcher: Validate email adresses when storing a watch #34042
  • Add delete rollup job support to HL REST Client #34066
  • BREAKING: Disallow negative query boost #34486


No update this week, see you next time!

  • We're hiring

    Work for a global, distributed team where finding someone like you is just a Zoom meeting away. Flexible work with impact? Development opportunities from the start?