This Week in Elasticsearch and Apache Lucene - 2018-04-13

There's been a lot happening on the Elasticsearch team over the past couple weeks, here's an update!

Security

Ioannis is in the final stages of work to use FIPS compliant algorithms, and, as of this morning, has ES with X-Pack running locally on a FIPS JVM. Yogesh continues work on kerberos support. Tim continues to work towards ways to improve TLS setup. Jay is thinking about ways GDPR will drive new features into our security offering, and Tim has already prototyped a public key encryption ingest processor that will encrypt a value using a public key and an “outside ES” private key. Another area where GDPR requirements may drive features under discussion are auditing and permissions, such as the ability to support write once indices (no edit/overwrite).

Distributed

Lee is pushing ahead with separation of XContent. Nhat, Martijn, Simon and Jason are hard at work transitioning Simon's soft deletes proof of concept into a working thing that can be used to back CCR.

Core-Infra

Daniel and Dimitrios have been simplifying rally to make it even better and easier to use, especially for benchmarking elasticsearch in low memory environments. Luca continues to move mountains on the high level REST client, and Colin and Tal are making a core-infra guest appearance through their work on Index Lifecycle Management.

Search + Aggregations

Martijn has been fixing percolator bugs, Jim F is working towards replacing the current missing option with “missing_buckets”, Adrien continues to make great progress on types removal.

Lucene

Alan is making changes to help all the future highlighters of the world, Simon is continuing work on Soft Deletes integration, Jim continues to do great work on the Korean Analyzer.

Lucene 7.3.0 was released!

Changes in 6.2:

  • Fixed bug when non percolator docs end up in the search hits #29447
  • Fixed a msm accounting error that can occur during analyzing a percolator query #29415
  • Fix more query extraction bugs. #29388
  • Don’t allow referencing the pattern bank name in the pattern bank #29295
  • X-Pack:
    • Watcher: Reenable email property settings
    • Watcher: Refactor/Fix TransportWatcherServiceAction

Changes in 6.3:

  • Add primary term to translog header #29227
  • Add periodic flush count to flush stats #29360
  • ElasticsearchMergePolicy should extend from MergePolicyWrapper #29476
  • Restrict Document list access in ParseContext #29463
  • BREAKING: Fail _search request with trailing tokens #29428
  • Grammar matters.. #29462
  • Add useful message when no input from terminal #29369
  • Deprecate slicing on _uid. #29353
  • Fix date and ip sources in the composite aggregation #29370
  • Close translog writer if exception on write channel #29401
  • Add indices options support to _rank_eval #29386
  • Fixes query_string query equals timezone check #29406
  • Allow using distance measure in the geo context precision #29273
  • Enhance error for out of bounds byte size settings #29338
  • Fixed quote_field_suffix in query_string #29332
  • Fix bwc in GeoDistanceQuery serialization #29325
  • BREAKING: Remove silent batch mode from install plugin #29359
  • Check presence of multi-types before validating new mapping #29316
  • X-Pack:
    • SQL: GROUP BY use composite instead of terms aggregation
    • Cache number of live documents with document-level security.
    • [Rollup] Rename size to page_size
    • [Rollup] Add value_count metric
    • Watcher: Refactor/Fix TransportWatcherServiceAction
    • SAML: Make alias for signing key optional
    • [Rollup] Validate field mapping before creating Rollup Job
    • Security: minimize automatons as they are combined
    • Upgrade to lucene-7.3.0
    • Optimize Logging Audit volatile conf flags
    • Watcher: Prevent watch acknowledgement while watch is executing
    • Watcher: Execute Watch API returns better error message

Changes in 7.0:

  • Add an include_type_name option. #29453
  • BREAKING: Remove legacy mapping code. #29224
  • BREAKING: Version conflict exception message enhancement #29432
  • BREAKING: Make purely negative queries return scores of 0. #26015
  • Remove 5x bwc in LocaleUtils#parse #29417
  • Remove delimited_payload_filter #27705
  • BREAKING: Remove undocumented action.master.force_local setting #29351
  • X-Pack:
    • SQL: GROUP BY use composite instead of terms aggregation
    • Add migration check for deprecated <span >delimited_payload_filter</span>