This Week in Elasticsearch and Apache Lucene - 2019-03-01
Elasticsearch
Proximity boost
We are working to implement proximity boosting in Elasticsearch. It uses the Lucene's DistanceFeatureQuery
to support efficient boosting by recency and geo distance. The functionality is similar to the function score query except that the new query uses the BKD-tree and the max score optimization to efficiently skip non-competitive documents when the total number of hits is not tracked accurately so it should be much faster without any loss in terms of precision.
SQL
A "columnar" results option for json, yaml, smile and cbor formats has been added to SQL, which makes every result's row contain all the values of a specific column from the current page of results. This helps some applications like Canvas parse the SQL results in a more efficient way. For a more detailed explanation and some examples, see the docs in the PR
Performance
We are working to add benchmarks for the default distribution and we are getting closer to deployment. This is a pretty complex change overall and required changes to Rally itself. To ensure a smooth transition and to understand whether performance differences are due to changes in Elasticsearch or the environment, we will run our benchmarks on both the current and the new environment for a certain period of time and will then decide on a day to cutover to the new environment.
Watcher UI
We have made great strides converting the Watcher UI to React/EUI. Along the way we've also had a chance to use the new React Hooks feature instead of Redux, resulting in a terser and simpler architecture. This work is inspiring some UI/UX improvements which we can incorporate along the way.
Watcher
Hipchat has been shut down so we are removing our support for it. We are deprecating Hipchat support in 6.7 (#39160) and removing it (#39199) from Watcher for 7.0.
Replicated closed indices
We have merged the closed replicated indices feature into the master branch. Closing an index now causes its shards to be reinitialized and then be replicated as regular shards. Closed indices and closed shards should also pop up in the cluster state as well as in other APIs like the Cluster Health APIs or the Recovery APIs.
Cross Cluster Replication
We have wrapped up the integration of shard history retention leases with CCR. This included removing a retention lease when we unfollow and renewing retention leases while following.
Lucene
Lucene 7.7.1
Lucene 7.7.1 is out, so we can start again the release process for 8.0 soon.
Other
- It would be useful to be able to merge range queries on the same field into a single range query for efficiency. This might be helpful for Kibana's KQL.
- We are looking into a bug with forced merges.
- There is increasing interest for vector search.
- We noticed unnecessary memory usage due to the construction of BKD tree writers that are never used, such as the 1D case that has optimized logic.
- Distance filtering support for geo shapes is being added.
- Polygon2D's crossing logic looks buggy.
- We are iterating on a new API to recursively visit query trees.
- There are changes being made to the IndexWriter to make it more resilient resilient to corruptions.
Changes
Changes in Elasticsearch
Changes in 8.0:
- Unify blob store compress setting #39346
- Re-enable backwards compatibility #39459
- BREAKING: Remove Hipchat support from Watcher #39199
Changes in 7.1:
- Add support for replicating closed indices #39499
- Never block on scheduled refresh if a refresh is running #39462
- Re-enable BWC #39460
- Upgrade to lucene 8.0.0-snapshot-ff9509a8df #39444
- SQL: introduce the columnar option for REST requests #39287
- Rename SearchRequest#withLocalReduction #39108
- Completion suggestions to be reduced once instead of twice #39255
- Reduce garbage from allocations in DeprecationLogger #38780
- Fix anaylze NullPointerException when AnalyzeTokenList tokens is null #39332
Changes in 7.0:
- Add details about what acquired the shard lock last #38807
- Obsolete pre 7.0 noarch package in rpm #39472
- Use https to obtain Lucene snapshots #39458
- Upgrade to lucene 8.0.0-snapshot-ff9509a8df #39350
- Upgrade to lucene 8.0.0-snapshot-ff9509a8df #39445
- Use non-ILM template setting up watch history template & ILM disabled #39325
- Enable soft-deletes by default for 7.0+ indices #38929
- Clean GatewayAllocator when stepping down as master #38885
Changes in 6.7:
- Correct name of basicdatetimenomillis #39367
- Do not wait for advancement of checkpoint in recovery #39006
- Fixed required fields and paths list #39358
- Provide a clearer error message on keystore add #39327
- Conditionally build BWC projects in parallel #39396
- Switch internal security index to ".security-7" #39337
- Renew retention leases while following #39335
- Handle failure to release retention leases in ILM #39281
- SQL: change the default precision for CURRENT_TIMESTAMP function #39391
- Protect against the leader index being removed #39351
- Ignore waitForActiveShards when syncing leases #39224
Changes in 6.6:
- SQL: Enhance checks for inexact fields #39427
- SQL: Use underlying exact field for LIKE/RLIKE #39443
- BREAKING: Disable BWC mode in TokenService by default #38881
- Bubble up exception when processing NoOp #39338
Changes in Elasticsearch Hadoop Plugin
Changes in 7.0:
Changes in Elasticsearch Management UI
Changes in 7.1:
- fix client validation for rollup index name #32067
- Fix common license checker issues #31339
- [Rollup] Unit test for detail panel #31690
- remove unused testbed directory in watcher #31917
- Fix Rollup Job List test, which broke due to change from jobs prop to hasJobs #31830
- Fix spacing error in license header in CCR, Remote Clusters, and Rollups. #31869
- [Rollup] Add unit tests for Job listing page #31689
Changes in 6.7:
- fixing show system indices toggle turning off others #32258
- disabling ccr and remote clusters when index mgmt is disabled #32203
- fixing issues with maximum documents and maximum size being set #31998
- fix Platinum features link in license management #31945
- fix clearing number of replicas from existing policy #31905
Changes in 6.6:
- Fix bug where rollup job search would display an empty prompt if no jobs matched the search. #31642
Changes in Elasticsearch SQL ODBC Driver
Changes in 6.6:
- Fix: string and floats conversion fixes #117
Changes in Rally
Changes in 1.1.0:
- BREAKING: Pass request-params as is in supported operations #649
- Add compatibility layer for existing metrics store #655
- Allow to use ES 7.x as metrics store #654
- Properly authenticate at proxy server #652
- Updates to support 7.x APIs #648