Elasticsearch version 6.3.1edit

Known issue with Elasticsearch 6.3.x on JDK 10

There appears to be a bug in JDK 10 impacting Elasticsearch 6.3.x (note that the official Docker images are based on JDK 10). The issue appears to impact the C2 compiler on hardware that supports AVX-512 instructions (e.g., Skylake-X CPUs). The issue manifests as a segmentation fault in Java code executing org.apache.lucene.util.MergedIterator#pullTop:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fc7ebc050eb, pid=1, tid=151
#
# JRE version: OpenJDK Runtime Environment (10.0.1+10) (build 10.0.1+10)
# Java VM: OpenJDK 64-Bit Server VM (10.0.1+10, mixed mode, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# J 14088 c2 org.apache.lucene.util.MergedIterator.pullTop()V (135 bytes) @ 0x00007fc7ebc050eb [0x00007fc7ebc041e0+0x0000000000000f0b]

or executing org.apache.lucene.util.MergedIterator#pushTop:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f8c43de5371, pid=1, tid=140
#
# JRE version: OpenJDK Runtime Environment (10.0.1+10) (build 10.0.1+10)
# Java VM: OpenJDK 64-Bit Server VM (10.0.1+10, mixed mode, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# J 13039 c2 org.apache.lucene.util.MergedIterator.pushTop()V (92 bytes) @ 0x00007f8c43de5371 [0x00007f8c43de4c40+0x0000000000000731]

A workaround is to disable the C2 compiler from using AVX-512 instructions via the JVM option -XX:UseAVX=2. This issue has been taken upstream to the JDK maintainers and the workaround will be necessary until a version of the JDK that is not prone to this issue is released. For additional background, see #31425.

Also see 6.3.

Bug fixesedit

Authentication
  • Security: fix joining cluster with production license #31341 (issue: #31332)
  • Security: fix token bwc with pre 6.0.0-beta2 #31254 (issues: #30743, #31195)
  • Compliant SAML Response destination check #31175
Ingest
  • Ingest Attachment: Upgrade Tika to 1.18 #31252
Java High Level REST Client
  • Change bulk’s retry condition to be based on RestStatus #29329 (issues: #28885, #29254)
Java Low Level REST Client
  • Avoid setting connection request timeout #30384 (issue: #24069)
Machine Learning
  • Fixes a bug introduced in 6.3.0 which may cause the a node to hang and drop out of the cluster if Machine Learning is enabled and has been actively used. The issue can be redressed in 6.3.0 by disabling Machine Learning on all nodes #31691 (issue: #31683)
Network
  • Ensure we don’t use a remote profile if cluster name matches #31331 (issue: #29321)
Packaging
  • Add package pre-install check for java binary #31343 (issue: #29665)
Recovery
  • Fix missing historyUUID in peer recovery when rolling upgrade 5.x to 6.3 #31506 (issue: #31482)
Rollup
  • [Rollup] Metric config parser must use builder so validation runs #31159
SQL
  • JDBC: Fix stackoverflow on getObject and timestamp conversion #31735 (issue: #31734)
  • SQL: Preserve scoring in bool queries #30730 (issue: #29685)
Scripting
  • Painless: Fix bug for static method calls on interfaces #31348
Search
  • Fix race in clear scroll #31259
  • Cross-cluster search: preserve remote status code #30976 (issue: #27461)
Security
  • Preserve thread context when connecting to remote cluster #31574 (issues: #31241, #31462)
Watcher
  • Watcher: Fix put watch action #31524