20 April 2018

This Week in Elasticsearch and Apache Lucene - 2018-04-20

By Tom CallahanPaul Sanwald

Highlights

Elasticsearch Java rest client - the Elasticsearch team is considering removing support for JDK-7 on the elasticsearch java rest client, as JDK-7 has been End of Life for four years now.  We would like to hear from you if you would be adversely affected.

The new korean analyzer, called Nori, has been merged to master and backported to the 7.x branch. It will be available in Lucene 7.4!

We fixed a bug which caused field values for binary fields from one document to be returned in the search for a different document.

Added an endpoint that will enable execution of basic Painless scripts directly via a new rest API. This enables requests such as:

POST /_scripts/painless/_execute { "script": { "source": "params.var1 / params.var2", "params": { "var1": 1071, "var2": 63 } } }

And Elasticsearch will respond with 

{ "result": "17" }

Right now such scripts only have access to the script parameters, but the power of this API will be expanded over time.

Added the "include_type_name" flag to the search and document API’s.  Using this flag will enable behavior that will become mandatory in elasticsearch 8.0 - which is the release in which elasticsearch will remove support for types.

Released version 0.10.0 of our Rally benchmarking framework.  Version 0.10.0 removes the requirement to have a local gradle installation, as well as adding several new features.  Please be sure to read up on the breaking changes in this major version if you are affected.

Performed some reorganization of a few important thread pools and the operations than run on them.  Prior to Elasticsearch version 5.3, single-document writes (including inserts, updates, and deletes) ran on the index thread pool, while multi-document writes ran on the bulk thread pool.  In addition, analyze requests shared the index thread pool.  As of version 5.3, all document writes run inside the bulk thread pool.  Due to this change, with version 6.3, the bulk thread pool is being renamed to the write thread pool, the index thread pool, will be removed, and analyze operations will take place in a new thread pool with a single thread.

We continue to work on leveraging formal modeling to find bugs in the distributed layer of Elasticsearch. 

We are doing some under-the-hood work to move to the recently-added soft-delete feature of Lucene.  In the future, this work will allow Elasticsearch to keep historical versions of indexed documents.

Changes in 6.2:

  • Plugins: Fix native controller confirmation for non-meta plugin #29434

Changes in 6.3:

  • Avoid side-effect in VersionMap when assertion enabled #29585
  • Add support to match_phrase query for zero_terms_query. #29598
  • Deprecate use of htmlStrip as name for HtmlStripCharFilter #27429
  • BREAKING: Rename the bulk thread pool to write thread pool #29593
  • Fix missing node id prefix in startup logs #29534
  • Painless execute api #29164
  • Networking: Deprecate http.enabled setting #29591
  • Modify Painless grammar to support right brackets as statement delimiters #29566
  • Enforce access to translog via engine #29542
  • Fix binary doc values fetching in _search #29567
  • Remove flatSettings support from request classes #29560
  • Add null_value support to geo_point type #29451
  • Introduce analyze thread pool #29541
  • Deprecate the index thread pool #29540
  • Add setting to control warning headers #29516
  • REST high-level client: add support for Indices Update Settings API [take 2] #29327
  • Fix overflow error in parsing of long geohashes #29418
  • Avoid self-deadlock in the translog #29520
  • X-Pack:
    • SQL: Mimick Elasticsearch defaults for sql action
    • [Rollup] Add scaled_float to allowed metric mapper types
    • Allow license installation in dev mode without TLS
    • Reflect renaming of bulk thread pool
    • [Rollup] Document type should be _doc
    • BREAKING: Tie zip output to command line options in certutil
    • Trigger log4j loading prior to context classloader
    • Remove legacy help files
    • Remove copy-pasted code
    • [Rollup] Document type should be _doc
    • SQL: improve conversion of Date types
    • SQL: Expand SYS TABLES behavior to support ODBC spec
    • SQL: Fix bug in transforming Expression properties
    • SQL: add conversion to same and NULL data type
    • SQL: Add folding to Math functions
    • SQL: Enable some date agg tests
    • SAML: Support multiple decryption keys for SP

Changes in 7.0:

  • BREAKING: Remove bulk fallback for write thread pool #29609
  • Add a CHANGELOG file for release notes. #29450
  • BREAKING: Remove the index thread pool #29556
  • Add the include_type_name option to the search and document APIs. #29506
  • Using ObjectParser in UpdateRequest #29293
  • Fix incorrect references to zero_terms_docs in query parsing error messages. #29599
  • X-Pack:
    • Remove index thread pool from monitoring filters