This Week in Elasticsearch and Apache Lucene - 2019-04-05

Elasticsearch

Highlights

6.7.0 Docker upgrade bug fixed

We fixed a bug introduced in 6.7.0 that was preventing users of the official Docker image from upgrading. This was included in the 6.7.1 release.

Snapshot lifecycle management

We now persist the last successful and failed snapshots into the cluster state to be visible for admins.

The put snapshot lifecycle API now validates as much as possible when a policy is created.

Snapshot/Restore UI

We added support for creating and editing repositories.

Single-node discovery now uses Zen2

We have switched the single-node discovery type (discovery.type: single-node) from using a dedicated implementation to using the existing default (Zen2) discovery implementation. With this, only a single implementation is now used across the codebase, which fixes a few problems with the previous dedicated implementation that was ignoring some of the node settings (e.g. became master even though node.master was set to false).

Geo

We added optimization logic to convert ST_Distance function calls into a geo_distance query when it is called in WHERE clauses. We are also improving handling of nulls in geo functions/types.

We are refactoringGeoHashUtils, both as a general cleanup of the utility functions and to facilitate whitelisting geo functions for use in Painless.

Rollups and timezones

We removed timezone validation on range queries in rollup searches. Rollup is pretty aggressive about validating search requests to make sure they match the job, but in this case the validation was unnecessary and could block valid requests.

Also related to timezones, we fixed a bug in how Rollup rewrites date_histo aggregations. We had been validating the timezone of the incoming date_histo, but had forgotten to set the valid timezone on the newly rewritten rollup date_histo.

SQL

We introduced the TIME data type. This type conforms to ANSI SQL and is compatible with all date-related functionalities like date-time functions and maths, grouping, and histograms.

Lucene

Luke to become a Lucene module?

After years of discussion, it looks like Luke may become a Lucene module. If you are not familiar with Luke, it is a tool that allows to introspect the content of Lucene indices.

Towards a more compact terms dictionary?

A new postings format was proposed whose terms dictionary seems to have a smaller disk footprint and perform faster when looking up keys that exist. However, it is not able to skip going to disk for keys that do not exist in the dictionary.

Other

Changes

Changes in Elasticsearch

Changes in 8.0:

  • BREAKING: Make remote cluster resolution stricter #40419
  • BREAKING: Remove obsolete security settings #40496

Changes in 7.1:

  • Rest-High-Level-Client:fix uri encode bug when url path start with '/' #34436
  • HLRC: Convert xpack methods to client side objects #40705
  • Add an OpenID Connect authentication realm #40674
  • Replace javax activation with jakarta activation #40247
  • SQL: Implement CURRENT_TIME/CURTIME functions #40662
  • Deprecate versions of Java prior to Java 11 #40756
  • Add Bulk Delete Api to BlobStore #40322
  • Don't mark shard as refreshPending on stats fetching #40458
  • BREAKING: Drop support for Ubuntu 14.04 #40709
  • Show SSL usage when security is not disabled #40672
  • Support roles with application privileges against wildcard applications #40398
  • SQL: Introduce SQL TIME data type #39802
  • Reindex from Remote allow date math #40303
  • Fix merging of text field mappers #40627
  • SQL: Fix precedence of :: psql like CAST operator #40665

Changes in 7.0:

  • Rollup ignores time_zone on date histogram #40844
  • Don't replace forbidden pattern failures when found #40710
  • Help Eclipse infering lambda parameter types #40747
  • Do not perform cleanup if Manifest write fails with dirty exception #40519

Changes in 6.7:

  • Allow ILM to stop if indices have nonexistent policies #40820
  • Resync should not send operations without sequence number #40433
  • Avoid background sync on relocated primary #40800
  • Remove timezone validation on rollup range queries #40647
  • Fix alias resolution runtime complexity. #40263
  • Bat scripts to work with JAVA_HOME with parantheses #39712
  • Use TAR instead of DOCKER build type before 6.7.0 #40723
  • Allow single digit milliseconds in strict date parsing #40676
  • SQL: have LIKE/RLIKE use wildcard and regexp queries #40628
  • Correct bug in ScriptDocValues #40488
  • Remove dynamic objects from security index #40499

Changes in 6.6:

  • SQL: Fix display size for DATE/DATETIME #40669

Changes in Elasticsearch Management UI

Changes in 7.1:

  • Internationalize Rollup Jobs, Remote Clusters, and Watcher app titles. #34470
  • Add collector to ui_metric app so that it reports its own telemetry. #34210
  • Fix and reenable Rollup JobList client integration tests. #34215

Changes in 7.0:

  • filter only on name in index management #34372
  • [Search Profiler] Quick style fix up including dark theme #33445

Changes in 6.7:

  • [ILM] Remove "Beta" badge #34454
  • fixing crash when zero replicas configured in a phase for policy #34480

Changes in Rally

Changes in 1.1.0:

  • [Experimental] Capture peak usage of JVM mem pools #674
  • Chart generator segment memory and new track combinations #668
  • Honor runtime JDK in provisioner #673
  • Add ability to download from S3 buckets #671