This Week in Elasticsearch and Apache Lucene - 2018-11-02

By
Alan Woodward
•
Tom Callahan
•
David Turner
•
Jason Tedor
•
Jay Modi
•
Colin Goodheart-Smithe

Elasticsearch

Highlights

Dual-stack Plaintext/TLS

We’ve started work on a dual-stack plaintext/TLS transport layer, with an objective of supporting unsecured to secured cluster transitions through a rolling upgrade. We opened an issue to discuss what rules to put around this functionality, as we do not want to introduce vulnerabilities permitting downgrades to plain text. We hope that this work will reduce issues moving from a trial license to a real license.

Hadoop

ES-Hadoop now resolves mapping conflicts when reading from multiple indices by ‘up-casting’ fields to more generic types during the read. This eases changes made back 6.0.0 to be more assertive about mapping conflicts.

Index Lifecycle Management

The feature branch for Index Lifecycle Management has been merged to the master and 6.x branches, where development will continue. While this work is not yet complete, merging into our mainline development branches is a significant milestone and shows the level of testing and effort that has gone into this feature.

Cross-Cluster Replication

We’ve expanded Rally’s support for benchmarking CCR, as well as performed some more important benchmarks of CCR’s behavior, including security on/off and an append-only workload vs. a workload with conflicts.

We’re finishing a pull request to replicate settings changes from the source index to the follower index. This work is critical to ensuring successful replication in the face of a settings change on the leader index affecting analysis for example.

SAML Troubleshooting Docs

We’ve added some stack docs to identify common problems and solutions when configuring SAML.

Changelog

Changes in 6.4:

  • Make XContentBuilder in AliasActions build is_write_index field #35071

Changes in 6.5:

  • SQL: Improve painless script generated from IN #35055
  • Fix a bug in function_score queries where we use the wrong boost_mode. #35148
  • SQL: handle wildcard expansion on incorrect fields #35134
  • SQL: Register missing processors #35121
  • SQL: Fix NPE thrown if HAVING filter evals to null #35108
  • SQL: Optimizer rule for folding nullable expressions #35080
  • [Rollup] Proactively resolve index patterns in RollupSearch endoint #34930
  • SQL: Proper handling of nested fields at the beginning of the columns list #35068
  • SQL: Fix failing test due to random timezone #35067
  • SQL: Add CAST and CONVERT to SHOW FUNCTIONS #34940
  • SQL: Implement CAST between STRING and IP #34949
  • SQL: Fix function args verification and error msgs #34926
  • SQL: Fix incorrect AVG data type #34948

Changes in 6.6:

  • BREAKING: Watcher: fix metric stats names #34951
  • Allow to enable pings for specific remote clusters #34753
  • BREAKING: Loggers: Drop last deprecated logger function #35082
  • Packaging: Update procrun executables to version 1.1.0 #35147
  • XPack/HLRC request/response compatibility tests #35129
  • BREAKING: Core: Drop settings member from AbstractComponent #35083
  • HLRest: add security authenticate API #33552
  • BREAKING: HLRC XPack Protocol clean up: Migration; Graph; Watcher #34639
  • median absolute deviation agg #34482
  • Native roles store uses mget to retrieve roles #33531
  • BREAKING: HLRC XPack Protocol clean up: Migration; Graph; Watcher #34964
  • Deprecate _source_include and _source_exclude url parameters #33475
  • Add warning about upcoming expanded fields limit #34906
  • Add start rollup job support to HL REST Client #34623
  • HLRC: Add security Create Token API #34791
  • [HLRC] Add support for get role mappings API #34637

Changes in 7.0:

  • Bulk Api support for global parameters #34528
  • Replace parameter unicodeSetFilter with unicode_set_filter #29215
  • Remove deprecated url parameters _source_include and _source_exclude #35097
  • XPack/HLRC request/response compatibility tests #34547
  • HLRC: Drop extra level from user parser #34932


Lucene

The 7.6 branch will be cut towards the end of next week, with the aim of releasing in the last week of November.