This Week in Elasticsearch and Apache Lucene - 2019-03-01

Elasticsearch

Proximity boost

We are working to implement proximity boosting in Elasticsearch. It uses the Lucene's DistanceFeatureQuery to support efficient boosting by recency and geo distance. The functionality is similar to the function score query except that the new query uses the BKD-tree and the max score optimization to efficiently skip non-competitive documents when the total number of hits is not tracked accurately so it should be much faster without any loss in terms of precision.

SQL

A "columnar" results option for json, yaml, smile and cbor formats has been added to SQL, which makes every result's row contain all the values of a specific column from the current page of results. This helps some applications like Canvas parse the SQL results in a more efficient way. For a more detailed explanation and some examples, see the docs in the PR

Performance

We are working to add benchmarks for the default distribution and we are getting closer to deployment. This is a pretty complex change overall and required changes to Rally itself. To ensure a smooth transition and to understand whether performance differences are due to changes in Elasticsearch or the environment, we will run our benchmarks on both the current and the new environment for a certain period of time and will then decide on a day to cutover to the new environment.

Watcher UI

We have made great strides converting the Watcher UI to React/EUI. Along the way we've also had a chance to use the new React Hooks feature instead of Redux, resulting in a terser and simpler architecture. This work is inspiring some UI/UX improvements which we can incorporate along the way.

Create threshold alert UI

Watcher

Hipchat has been shut down so we are removing our support for it. We are deprecating Hipchat support in 6.7 (#39160) and removing it (#39199) from Watcher for 7.0.

Replicated closed indices

We have merged the closed replicated indices feature into the master branch. Closing an index now causes its shards to be reinitialized and then be replicated as regular shards. Closed indices and closed shards should also pop up in the cluster state as well as in other APIs like the Cluster Health APIs or the Recovery APIs.

Cross Cluster Replication

We have wrapped up the integration of shard history retention leases with CCR. This included removing a retention lease when we unfollow and renewing retention leases while following.

Lucene

Lucene 7.7.1

Lucene 7.7.1 is out, so we can start again the release process for 8.0 soon.

Other

Changes

Changes in Elasticsearch

Changes in 8.0:

  • Unify blob store compress setting #39346
  • Re-enable backwards compatibility #39459
  • BREAKING: Remove Hipchat support from Watcher #39199

Changes in 7.1:

  • Add support for replicating closed indices #39499
  • Never block on scheduled refresh if a refresh is running #39462
  • Re-enable BWC #39460
  • Upgrade to lucene 8.0.0-snapshot-ff9509a8df #39444
  • SQL: introduce the columnar option for REST requests #39287
  • Rename SearchRequest#withLocalReduction #39108
  • Completion suggestions to be reduced once instead of twice #39255
  • Reduce garbage from allocations in DeprecationLogger #38780
  • Fix anaylze NullPointerException when AnalyzeTokenList tokens is null #39332

Changes in 7.0:

  • Add details about what acquired the shard lock last #38807
  • Obsolete pre 7.0 noarch package in rpm #39472
  • Use https to obtain Lucene snapshots #39458
  • Upgrade to lucene 8.0.0-snapshot-ff9509a8df #39350
  • Upgrade to lucene 8.0.0-snapshot-ff9509a8df #39445
  • Use non-ILM template setting up watch history template & ILM disabled #39325
  • Enable soft-deletes by default for 7.0+ indices #38929
  • Clean GatewayAllocator when stepping down as master #38885

Changes in 6.7:

  • Correct name of basicdatetimenomillis #39367
  • Do not wait for advancement of checkpoint in recovery #39006
  • Fixed required fields and paths list #39358
  • Provide a clearer error message on keystore add #39327
  • Conditionally build BWC projects in parallel #39396
  • Switch internal security index to ".security-7" #39337
  • Renew retention leases while following #39335
  • Handle failure to release retention leases in ILM #39281
  • SQL: change the default precision for CURRENT_TIMESTAMP function #39391
  • Protect against the leader index being removed #39351
  • Ignore waitForActiveShards when syncing leases #39224

Changes in 6.6:

  • SQL: Enhance checks for inexact fields #39427
  • SQL: Use underlying exact field for LIKE/RLIKE #39443
  • BREAKING: Disable BWC mode in TokenService by default #38881
  • Bubble up exception when processing NoOp #39338

Changes in Elasticsearch Hadoop Plugin

Changes in 7.0:

  • Fix scala variant plugin in Gradle 5.0 #1255
  • Update Gradle to v5.0 #1253

Changes in Elasticsearch Management UI

Changes in 7.1:

  • fix client validation for rollup index name #32067
  • Fix common license checker issues #31339
  • [Rollup] Unit test for detail panel #31690
  • remove unused testbed directory in watcher #31917
  • Fix Rollup Job List test, which broke due to change from jobs prop to hasJobs #31830
  • Fix spacing error in license header in CCR, Remote Clusters, and Rollups. #31869
  • [Rollup] Add unit tests for Job listing page #31689

Changes in 6.7:

  • fixing show system indices toggle turning off others #32258
  • disabling ccr and remote clusters when index mgmt is disabled #32203
  • fixing issues with maximum documents and maximum size being set #31998
  • fix Platinum features link in license management #31945
  • fix clearing number of replicas from existing policy #31905

Changes in 6.6:

  • Fix bug where rollup job search would display an empty prompt if no jobs matched the search. #31642

Changes in Elasticsearch SQL ODBC Driver

Changes in 6.6:

  • Fix: string and floats conversion fixes #117

Changes in Rally

Changes in 1.1.0:

  • BREAKING: Pass request-params as is in supported operations #649
  • Add compatibility layer for existing metrics store #655
  • Allow to use ES 7.x as metrics store #654
  • Properly authenticate at proxy server #652
  • Updates to support 7.x APIs #648

Changes in Rally Tracks

  • Adds a geoshapes track #61
  • Remove document types #66