This Week in Elasticsearch and Apache Lucene - 2019-01-19

Elasticsearch Highlights

Leniency in request handling

We are trying to remove leniency throughout Elasticsearch. It makes for a more predictable experience for our users, where Elasticsearch does what the user thinks they are trying to do, or otherwise tells them when something seems amiss. An example we have recently decided to address is HTTP requests with a body, where the endpoint doesn’t actually consume the body. We have recently merged a change to reject HTTP requests with a body when the endpoint doesn’t use the body. This is a breaking change that will ship in 7.0.


We added a whitelist to the HttpClient that is used by Watcher. This allows admins to allow only specific URL's to be called from Watcher. For passivity this defaults to all, but in future versions will require this to be explicitly set. (#36817)

Index management

We worked on a couple of PRs adding support for various index states to the index management UI. One added marking indices with a "Rollup" badge when they are created by a rollup job. We also added support for frozen indexes to index management, including freeze/unfreeze actions in the context menu and a "Frozen" badge in the index list.

Index Lifecycle Management

We submitted PR for the first system level index, .watch-history, to be managed by ILM. This lays the groundwork for a future system level index to be managed by ILM. The general strategy is to ship a default, but allow pre-existing policies, or changes to the cluster's policy such that we only apply the default policy if one does not already exist. The pre-existing watcher history cleaner service will be deprecated and removed. (#37443)


We are happy to see that our benchmarks have served as a good safety net performance regressions. This week we found a significant performance regression in the interaction between types removal in the bulk API and the deprecation logger. There is an intermediate fix for this and we have opened a follow-up issue to optimize deduplication in the deprecation logger.

Speaking of deprecation, We have raised 2 PRs to address some performance issues uncovered in this week’s FixIt meeting whilst discussing this issue.

Permissions to access restricted indices

The .security index has long had restricted access; in 2.x no external access was allowed (originally) and in 5.0+ access was restricted to superusers. This was done to avoid having users with permission to read all indices suddenly have permissions to read security information such as password hashes when the native realm was introduced. While thinking about backing up the .security index, it became clear that superuser was too powerful of a role to be using for backups. Albert has embarked on lightening this restriction and we've gone through a few iterations, but have landed on a new field with the indices section of a role called allow_restricted_indices that defaults to false. As of now, the only restricted indices are the security indices but we've kept this generic so that we have freedom to address this in the future.


We continue to work on the custom TLS configuration for reindex by making use of the new library he added last week. Additionally, Tim will be removing TLSv1.0 from the default list of protocols for 7.0as this protocol has known security weaknesses. We have worked on a new TLS tutorial that covers encrypting internode communications. We will be working to update this tutorial to cover the new Zen2 discovery and expand the tutorial to also include setting up HTTPS.

Total hits accuracy in Elasticsearch 7.0

Following the response format change for search requests that allowed us to return the total hits that match a query as a lower bound, we opened a PR that will change the default for all search requests to count the total hits accurately up to 10,000. This means that any search request that matches more than 10,000 documents will now return:

{ "hits": { "total": "10000", "relation": "gte" } }

This will be the behavior by default in 7 and beyond, "track_total_hits" must be set to true explicitly in the request to get the old behavior back.

Alternative CCS execution

We opened a PR to handle the merging of search responses coming from different clusters. In the alternative cross cluster search the reduction happens per shards inside each cluster first and then the responses of each cluster (the local one + the remotes) are merged to form the final response. We are also already working on the logical follow up which is the final integration of the alternative mode in the search layer.


We merged the minimal geo objects library into master, and are working on integrating it with the JDBC driver. Igor is also continuing to diagnose the performance problems on the new geoshapes rally test. The current theory is that the slowdown is caused by increasingly slow merging of ever-larger segments.

Cross Cluster Replication

We continue work on shard history retention leases, having added persistence to Lucene on commits, and opening a PR to sync retention leases to replicas This has triggered additional discussions about the durability of these leases and the guarantees that we want from the storage and replication system.

We added a new “CCR follow info" API that reports information about all the follower indices in the cluster together with their parameters and settings from the time where the following got initiated or resumed. We have also adapted the follow stats API to return a 404 instead of 200 with empty body if there are no follow shard tasks for the requested follower indices.

Following this work, we continued work on the Advanced Settings UI for adding a follower index. After review, the UI was revised to be a simple hide/show toggle for all settings. We also worked on PR that introduces the ability to edit a follower index. Editing refers to changing the follower index's advanced settings only. Since settings can only be changed upon resuming a follower, a warning is surfaced to the user that we will be pausing and resuming the follower in order to update it. We also added remote cluster validation to the create auto-follow pattern form and add/edit follower index forms. This presents a nice UX when

  1. There are no remote clusters
  2. When a selected remote cluster is not connected
  3. When a previously set remote cluster has been removed.

We are adding timeout handling and rate limiting to the CCR recover from remote functionality. We have also extended the security model for the remote cluster proxy so that the recovery from remote can fetch the Lucene file chunks under the system security context.


Changes in 6.5:

  • SQL: Fix issue with field names containing "." #37364

Changes in 6.6:

  • Document Seq No powered optimistic concurrency control #37284
  • Add fatal_exception field for ccr stats in monitoring mapping #37563
  • SQL: Describe aliases as views #37496
  • When removing an AutoFollower also mark it as removed. #37402
  • Use executor SAME to handle search related handlers #37427
  • SQL: Make FULL non-reserved keyword in the grammar #37377
  • Fix artifactId in plugin poms #37315
  • Geo: Do not normalize the longitude with value -180 for Lucene shapes #37299

Changes in 6.7:

  • Create specific exception for when snapshots are in progress #37550
  • Add set_priority action to ILM #37397
  • Propagate Errors in executors to uncaught exception handler #36137
  • Update Put Watch to allow unknown fields #37494
  • Make recovery source send operations non-blocking #37503
  • Allow field types to optimize phrase prefix queries #37436
  • Change file descriptor limit to 65535 #37537
  • SQL: Add protocol tests and remove jdbc_type from drivers response #37516
  • Deprecate requests that have an unconsumed body #37534
  • SQL: Remove slightly used meta commands #37506
  • Packaging: Remove permission editing in postinst #37242
  • Prepare to make send translog of recovery non-blocking #37458
  • Update Delete Watch to allow unknown fields #37435
  • New mapping signature and mapping string source fixed. #37401
  • Restore lost @Inject annotation #37452
  • Make finalize step of recovery source non-blocking #37458
  • Update Delete Watch to allow unknown fields #37435
  • New mapping signature and mapping string source fixed. #37401
  • Restore lost @Itic/elasticsearch/pull/37440"> #37440
  • Add simple method to write collection of writeables #37448
  • Introduce retention lease serialization #37447
  • Add deprecation warnings for ssl config fallback #36847
  • Security Audit includes HTTP method for requests #37322
  • Make recovery source partially non-blocking #37291
  • SQL: Use declared source for error messages #37161
  • SQL: [Tests] Fix and enable internalClusterTests #37300
  • Use Sequence number powered OCC for processing updates #37308
  • Support include_type_name in RestGetIndicesAction #37267
  • MAPPING: Improve Precision for scaled_float #37169
  • Add validation for retention lease construction #37312

Changes in 7.0:

  • Deprecate _type from LeafDocLookup #37491
  • Reject all requests that have an unconsumed body #37504
  • Fix duplicate removal when merging completion suggestions #36996
  • Report terms and version if cluster does not form #37473
  • BREAKING: Update the default for include_type_name to false. #37285
  • Geo: Adds a set of no dependency geo classes for JDBC driver #36477
  • Make sure to use the resolved type in DocumentMapperService#extractMappings. #37451
  • BREAKING: Security: remove SSL settings fallback #36846
  • Adding mapping for hostname field #37288
  • Performance fix. Reduce deprecation calls for the same bulk request #37415
  • Remove unused empty constructors from suggestions classes #37295
  • Watcher: Add whitelist to HttpClient #36817
  • Date/Time parsing: Use java time API instead of exception handling #37222

Apache Lucene

Off-heap terms index

A long-term Elasticsearch community member contributed a patch that allows to store the terms index off-heap. This is appealing due to the fact that the terms index is often the most memory-intensive part of a Lucene index. We want to be careful due to the fact that the access pattern is very random, but discussion on the issue suggests that we might be able to do changes to Finite State Transducer that would make it more disk-friendly.

Less disk-intensive and faster merging of points

Points (aka BKD trees) require lots of transient disk space when building trees for more than 1 dimension (1D fields like numerics don't need any transient disk space at all) in order to materialize sorted views of the data on every dimension. We tuned the logic a bit in order to build these materialized views more lazily, which helps build them on smaller subsets of the data. This change ended up lowering transient disk usage significantly and speeding up merging of points a bit. Our initial benchmarks reported 24% less transient peak disk usage when writing geo points (2 dimensions) and 60% less transient peak disk usage when writing shapes (4 indexed dimensions). Nightly benchmarks then reported 4% faster indexing of geo points and 15% faster indexing of shapes.

Lucene has moved to gitbox

Lucene has moved from to If your git clone points to, you will need to update it. If you had cloned Lucene from Github, you don't need to do anything.

One significant benefit from this move is that we are now able to merge pull requests directly from Github, which is going to ease contributing and merging contributions.

CONTAINS support for shape search

One missing feature from BKD-backed shapes in order to reach feature parity with current shapes is support for CONTAINS, ie. finding all indexed shapes that contain the query shape. In order to support this, we had to record which edges of the indexed triangles are also edges of the original polygon so that query shapes that cross an edge of the original polygon would not match.


  • We're hiring

    Work for a global, distributed team where finding someone like you is just a Zoom meeting away. Flexible work with impact? Development opportunities from the start?