This Week in Elasticsearch and Apache Lucene - 2018-10-05

Elasticsearch

Here's the latest update from the elasticsearch team.

Highlights

FIPS

PKCS11-NSS in FIPS mode support has been fixed, this will be available for 6.4.3. This concludes our FIPS 140 support story as Elasticsearch can run with a BouncyCastle FIPS provider, since 6.4.0, and a SunPKCS11-NSS provider in FIPS mode. These two are by far the most popular providers that have a FIPS mode for the JVMs we support. There is a very restrictive limitation with the use of SunPKCS11-NSS, and that is only a single Keystore can be used for Elasticsearch and this means, a single key,certificate pair, and TLS1.2 is also not currently supported. Details and the analysis behind these limitations can be found here.

Ingest

In 6.5 an ingest pipeline can call other pipelines. We have added support to properly instrument statistics for pipelines in those cases as well.

In 6.5 processors can be executed based on conditions. We have an open PR that adds better support for those processors, and also a PR to add instrumentation per processor (current, count, time, failed) to the existing ingest node stats pipeline instrumentation.

ES Hadoop

We now accept and validate aliases for writing that have an index set as the write index. This is the major component needed for supporting Index Lifecycle Management features going forward for Hadoop

Sorting results using the first matching nested document

We made a change that preserves the order of nested documents when writing them to the Lucene segment. The root document still comes last but the nested documents are written in the order specified in the request. This change allow us to progress with a community PR [3] which will add the ability to sort results based on the first matching nested document.

Zen2

We have been focused on the Zen2 testing infrastructure, which now allows simulating a run of a Zen2 cluster in the presence of network disruptions, unusual timings, and failing components. Test runs with this infrastructure are fully randomized, yet deterministic and reproducible. The purpose of this infrastructure is two-fold: 1) Explore Zen2's state space to find safety violations and 2) get the cluster into a weird state and then check that it recovers appropriately quickly. This test infrastructure has already proven to be super useful, bringing to light a number of smaller bugs in exceptional situations.

We integrated the cluster state applier into Zen2, which is the component that exposes cluster states to the rest of the system, for example to allocate or activate shards on a node. He has also used the above test infrastructure to check the cluster state acking functionality.

Changes

Changes in 5.6:
  • Fix logging of cluster state update descriptions #34243
Changes in 6.4:
  • Support PKCS#11 tokens as keystores and truststores #34063
Changes in 6.5:
  • Tasks: Document that status is not semvered #34270
  • Replace version with reader cache key in IndicesRequestCache #34189
  • [CI] Fix bogus ScheduleWithFixedDelayTests.testRunnableRunsAtMostOnceAfterCancellation #34296
  • HLRC: Add activate watch action #33988
  • [Security] Get Alias API wildcard exclusion with Security #34144
  • [Authz] Allow update settings action for system user #34030
  • [TESTS] Set SO_LINGER and SO_REUSEADDR on the mock socket #34211
  • Security: upgrade unboundid ldapsdk to 4.0.8 #34247
  • Add early termination support for min/max aggregations #33375
  • Security: reduce memory usage of DnRoleMapper #34250
  • BREAKING: Require combine and reduce scripts in scripted metrics aggregation #33452
  • Preserve the order of nested documents in the Lucene index #34225
  • [GCE Discovery] Automatically set project-id and zone #33721
  • Add support for ack watch to the HLRC. #33962
  • HLRC: Add throttling for update & delete-by-query #33951
  • Fix logging of cluster state update descriptions #34182
  • BREAKING: Disallow "enabled" attribute change for types in mapping update (#33566) #33933
  • HLRC: Add get rollup job #33921
  • Fix use of hostname in Windows service #34193
  • SQL: Remove more ANTLR4 grammar ambiguities #34074
  • Fix cross fields mode of the query_string query #34216
  • Completion types with multi-fields support #34ef="https://github.com/elastic/elasticsearch/pull/34193"> #34193
  • SQL: Remove more ANTLR4 grammar ambiguities #34074
  • Fix cross fields mode of the query_string query #34216
  • Completion types with multi-fields support #34ef="https://github.com/elastic/elasticsearch/pull/34193"> #34193
  • #34072
  • SQL: Fix function resolution #34137

Apache Lucene

Spans are hard

Getting span_near queries right is hard. The fact that there might be multiple spans at the same start position and that they might have different lengths causes the current span_near implementation to sometimes omit some combinations. This issue has been open for a long time but a (large - 10k new lines) patch was recently submitted. This might trigger interesting discussions regarding spans vs. intervals as intervals only expose minimum intervals, which helps avoid a number of these problems.

Other
Adoptme

We believe that sorting an index by norm might help the new block-max-WAND optimizations perform better. It would be interesting to introduce the ability to sort by norm at index-time and see whether it improves query performance compared to an index in a random order.