This Week in Elasticsearch and Apache Lucene - 2019-03-08


Ingest node lookup processor

We are beginning to think about what an ingest node lookup processor may look like.

Snapshot support in Index Lifecycle Management

We are starting efforts to bring snapshot support to ILM.

Replicated closed indices

Last week, we merged the core replicated closed indices feature into master. We have since added the capability for replicated closed indices to be snapshotted. This also changes snapshot requests in v8.0.0 to automatically expand wildcards to closed indices and hence start snapshotting closed indices by default. For v7.1.0 and above, wildcards are by default only expanded to open indices, which can be changed by explicitly setting the expand_wildcards option either to all or closed. Note that indices are always restored as open indices, even if they have been snapshotted as closed replicated indices.

Cross Cluster Replication

We are adding a forget follower API so that follower shard history retention leases can be manually removed from the primary in case that unfollowing was unable to do so (e.g., if the remote cluster was offline when the unfollow API was executed); while the shard history retention leases would eventually expire, 12 hours is a long time to wait, which would cause history to be kept around much longer than necessary, and would also prevent ILM from performing lifecycle actions such as shrink and delete.

Rolling upgrades from non-secured to secured clusters

A rolling upgrade of a non-secured cluster to a secured cluster is currently not possible due to the need to enable TLS. We have opened a meta-issue to track work on this and, as a first step, opened a PR which adds a setting, which when enabled, will allow the netty-transport to accept incoming plaintext connections even if TLS is enabled.

Resiliency improvements

We backported a long list of recent snapshot resiliency fixes to 6.7, which fixes issues with snapshots hanging.

We fixed a data replication bug where an acknowledged write can be lost. If a write request fails to execute on a replica after completing on the primary, the primary will reach out to the master to remove that failing shard from the in-sync set. If the node with primary is shutting down while reaching out to the master to fail the replica, this can trigger an exception that might unfortunately be ignored and leaves a very short window for the write request to mistakenly be acknowledged to the client. The acknowledged write will be lost if the failing shard is subsequently promoted to primary.

We adapted the cluster state recovery logic in 7.0+ not to close bad indices as this turned out to be problematic for both Zen2 as well as replicated closed indices.

Security token service

We merged the work to support concurrent refresh of tokens, which will avoid some issues in Kibana when a token needs to be refreshed and concurrent requests try to do so.

We also opened a PR to refactor the token service in preparation for splitting the tokens into a separate index since this is data that users probably do not want to backup.


Lucene 8.0.0

The release process is underway, but a couple of Solrbugs were discovered that will require a re-spin of the release candidates.


We have proposed that we use a new TermDeletedAttribute rather than removing tokens to address issues with stop words that occur at articulation points.

We are iterating on improving detection of how a line crosses a polygon, which is especially important when crossing the dateline.

We are iterating on fixing a bug with forced merges.

We are looking into only decoding blocks of term frequencies when at least on frequency from the block is requested.


Changes in Elasticsearch

Changes in 8.0:

  • Bug fix for AnnotatedTextHighlighter - port of 39525 #39747
  • BREAKING: Add lower bound on poll_interval #39593
  • BREAKING: Remove deprecated _time and _term sort orders #39450
  • Handle total hits equal to tracktotalhits #37907

Changes in 7.1:

  • Combine overriddenOps and skippedOps in translog #39771
  • Simulate BWC tests that start with the current version #39102
  • Chunk + Throttle Netty Writes #39286
  • Bug fix for AnnotatedTextHighlighter - port of 39525 #39749
  • Allow snapshotting replicated closed indices #39644
  • Optimize Bulk Message Parsing and Message Length Parsing #39634
  • Tie-break completion suggestions with same score and surface form #39564
  • Allow inclusion of unloaded segments in stats #39512
  • Add stopword support to IntervalBuilder #39637
  • Shortcut counts on exists queries #39570
  • Clearer error message - installing windows service #33804
  • Fix Token Service retry mechanism #39639
  • Support concurrent refresh of refresh tokens #39631
  • Add support for S3 intelligent tiering (#38836) #39376
  • Enhancements to IndicesQueryCache. #39099
  • Support concurrent refresh of refresh tokens #38382

Changes in 7.0:

  • Update logstash-management.json to use typeless template #38653
  • Change licence expiration date pattern #39681
  • Bug fix for AnnotatedTextHighlighter - port of 39525 #39750
  • Add option to force load term dict into memory #39741
  • Drop node if asymmetrically partitioned from master #39598
  • Avoid serialising state if it was already serialised #39179
  • Types removal security index template #39705
  • Bundle java in distributions #38013
  • Index on rollup.rollup_search.json is a list #39097
  • ilm.explain_lifecycle documents human again #39113
  • metric on watcher stats is a list not an enum #39114
  • Update spec files that erroneously documented parts as optional #39122
  • SQL: Enable accurate hit tracking on demand #39527

Changes in 6.7:

  • Ingest ingest then create index #39607
  • Introduce forget follower API #39718
  • Bug fix for AnnotatedTextHighlighter #39525
  • Add Docker build type #39378
  • Use any index specified by .watches for Watcher #39541
  • Fixing the custom object serialization bug in diffable utils. #39544
  • Fix security index auto-create and state recovery race #39582
  • Check for .watches that wasn't upgraded properly #39609
  • SQL: Don't allow inexact fields for MIN/MAX #39563
  • SQL: fix COUNT DISTINCT column name #39537
  • Make _doc work as an alias of the actual type of an index. #39505

Changes in 6.6:

  • Don’t ack if unable to remove failing replica #39584
  • Use cause to determine if node with primary is closing #39723
  • Fix Fuzziness#asDistance(String) #39643
  • SQL: Fix merging of incompatible multi-fields #39560
  • SQL: ignore UNSUPPORTED fields for JDBC and ODBC modes in 'SYS COLUMNS' #39518

Changes in Elasticsearch Management UI

Changes in 7.1:

  • Change common router errors to be thrown instead of returned #32670
  • Fix common router request handler issue #32275
  • Fix API endpoint typo indices/clearcaches -> indices/clearcache. #32582
  • Add user_action app for gathering telemetry on user actions #31543
  • fixing issue with request body not being sent for console DELETE requ… #32407

Changes in 6.7:

  • [CCR] Fix form bug + add missing i18n #32544

Changes in Elasticsearch SQL ODBC Driver

Changes in 7.1:

  • Test: add Kibana's sample data to the tests #123
  • Support for local timezone #118

Changes in Rally

Changes in 1.1.0:

  • Add user tags when comparing races #660
  • Add distribution flavor to metrics metadata #662
  • Sort the track parameters / user tags when listing races #659