This Week in Elasticsearch and Apache Lucene - 2019-04-05
Elasticsearch
Highlights
6.7.0 Docker upgrade bug fixed
We fixed a bug introduced in 6.7.0 that was preventing users of the official Docker image from upgrading. This was included in the 6.7.1 release.
Snapshot lifecycle management
We now persist the last successful and failed snapshots into the cluster state to be visible for admins.
The put snapshot lifecycle API now validates as much as possible when a policy is created.
Snapshot/Restore UI
We added support for creating and editing repositories.
Single-node discovery now uses Zen2
We have switched the single-node discovery type (discovery.type: single-node) from using a dedicated implementation to using the existing default (Zen2) discovery implementation. With this, only a single implementation is now used across the codebase, which fixes a few problems with the previous dedicated implementation that was ignoring some of the node settings (e.g. became master even though node.master was set to false).
Geo
We added optimization logic to convert ST_Distance function calls into a geo_distance query when it is called in WHERE clauses. We are also improving handling of nulls in geo functions/types.
We are refactoringGeoHashUtils, both as a general cleanup of the utility functions and to facilitate whitelisting geo functions for use in Painless.
Rollups and timezones
We removed timezone validation on range queries in rollup searches. Rollup is pretty aggressive about validating search requests to make sure they match the job, but in this case the validation was unnecessary and could block valid requests.
Also related to timezones, we fixed a bug in how Rollup rewrites date_histo aggregations. We had been validating the timezone of the incoming date_histo, but had forgotten to set the valid timezone on the newly rewritten rollup date_histo.
SQL
We introduced the TIME data type. This type conforms to ANSI SQL and is compatible with all date-related functionalities like date-time functions and maths, grouping, and histograms.
Lucene
Luke to become a Lucene module?
After years of discussion, it looks like Luke may become a Lucene module. If you are not familiar with Luke, it is a tool that allows to introspect the content of Lucene indices.
Towards a more compact terms dictionary?
A new postings format was proposed whose terms dictionary seems to have a smaller disk footprint and perform faster when looking up keys that exist. However, it is not able to skip going to disk for keys that do not exist in the dictionary.
Other
- We made
nestedqueries faster when hit counts are not needed. - We are making
Weight#matchesless costly by leveraging thescorerSupplier API. - We made
WordDelimiterGrapFilteremit the original token first. - We are refactoring
EdgeTreeto make is simpler and more general.
Changes
Changes in Elasticsearch
Changes in 8.0:
- BREAKING: Make remote cluster resolution stricter #40419
- BREAKING: Remove obsolete security settings #40496
Changes in 7.1:
- Rest-High-Level-Client:fix uri encode bug when url path start with '/' #34436
- HLRC: Convert xpack methods to client side objects #40705
- Add an OpenID Connect authentication realm #40674
- Replace javax activation with jakarta activation #40247
- SQL: Implement CURRENT_TIME/CURTIME functions #40662
- Deprecate versions of Java prior to Java 11 #40756
- Add Bulk Delete Api to BlobStore #40322
- Don't mark shard as refreshPending on stats fetching #40458
- BREAKING: Drop support for Ubuntu 14.04 #40709
- Show SSL usage when security is not disabled #40672
- Support roles with application privileges against wildcard applications #40398
- SQL: Introduce SQL TIME data type #39802
- Reindex from Remote allow date math #40303
- Fix merging of text field mappers #40627
- SQL: Fix precedence of
::psql like CAST operator #40665
Changes in 7.0:
- Rollup ignores time_zone on date histogram #40844
- Don't replace forbidden pattern failures when found #40710
- Help Eclipse infering lambda parameter types #40747
- Do not perform cleanup if Manifest write fails with dirty exception #40519
Changes in 6.7:
- Allow ILM to stop if indices have nonexistent policies #40820
- Resync should not send operations without sequence number #40433
- Avoid background sync on relocated primary #40800
- Remove timezone validation on rollup range queries #40647
- Fix alias resolution runtime complexity. #40263
- Bat scripts to work with JAVA_HOME with parantheses #39712
- Use TAR instead of DOCKER build type before 6.7.0 #40723
- Allow single digit milliseconds in strict date parsing #40676
- SQL: have LIKE/RLIKE use wildcard and regexp queries #40628
- Correct bug in ScriptDocValues #40488
- Remove dynamic objects from security index #40499
Changes in 6.6:
- SQL: Fix display size for DATE/DATETIME #40669
Changes in Elasticsearch Management UI
Changes in 7.1:
- Internationalize Rollup Jobs, Remote Clusters, and Watcher app titles. #34470
- Add collector to ui_metric app so that it reports its own telemetry. #34210
- Fix and reenable Rollup JobList client integration tests. #34215
Changes in 7.0:
- filter only on name in index management #34372
- [Search Profiler] Quick style fix up including dark theme #33445
Changes in 6.7:
- [ILM] Remove "Beta" badge #34454
- fixing crash when zero replicas configured in a phase for policy #34480
Changes in Rally
Changes in 1.1.0: