elasticsearch-weekly-blog-banner.png

elasticsearch-weekly-blog-thumb.png

This Week in Elasticsearch - 2020-02-28

This Week in Elasticsearch and Apache Lucene - 2020-02-18

This Week in Elasticsearch and Apache Lucene - 2020-01-31

This Week in Elasticsearch and Apache Lucene - 2019-12-02

This Week in Elasticsearch and Apache Lucene - 2019-10-21

This Week In Elasticsearch, September 2nd

This Week in Elasticsearch and Apache Lucene - 2019-05-24

This Week in Elasticsearch and Apache Lucene - 2019-04-26

This Week in Elasticsearch and Apache Lucene - 2019-03-29

This Week in Elasticsearch and Apache Lucene - 2019-03-01

This Week in Elasticsearch and Apache Lucene - 2019-02-08

This Week in Elasticsearch and Apache Lucene - 2019-01-11

<html><head></head><body><h3>Elasticsearch</h3><h4>Remote clusters UI
	</h4><p>Work continues on the remote clusters UI. We <a href="https://github.com/elastic/kibana/pull/25695">have</a> <a href="https://github.com/elastic/kibana/pull/25773">finished</a> <a href="https://github.com/elastic/kibana/pull/25833">the UI</a> <a href="https://github.com/elastic/kibana/pull/25837">layer</a>, and have been working on the <a href="https://github.com/elastic/kibana/pull/25632">app layer</a> <a href="https://github.com/elastic/kibana/pull/25688">to integrate</a> <a href="https://github.com/elastic/kibana/pull/25782">the UI</a> <a href="https://github.com/elastic/kibana/pull/25835">with endpoints</a>. So far the work has been done based on the assumption that add/edit/delete are all persistent settings. Next, we will work on handling transient settings as well as settings that come from elasticsearch.yml.
</p><h4>Index lifecycle management UI
</h4><p>Work on this UI is progressing nicely.  The flows from the index management app to add a policy to and remove a policy from an index are completed. We also added the ability to see <a href="https://github.com/elastic/kibana/pull/25553">the error stack trace for lifecycle errors and to see the phase definition</a>. Remaining tasks include adding the ability to disable the UI, adding a button to add a policy to an index template from the policy CRUD UI, adding filter buttons in index management to filter on managed/unmanaged and lifecycle phase, and <a href="https://github.com/elastic/kibana/pull/22768">updating autocomplete rules for console to include new ILM APIs</a>.
</p><h4>Query Script Query
</h4><p>There is a new query called <a href="https://github.com/elastic/elasticsearch/pull/34533">script_score</a> that allows users to manipulate the score of documents through a Painless script. The query also includes a script context which exposes a number of useful methods including the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#function-decay">Decay functions</a> of the function_score query.  This new query is the first part of the work to <a href="https://github.com/elastic/elasticsearch/issues/27588">deprecate and remove the function_score query</a> in favour of easier to use alternatives.
</p><h4>Rare terms aggregation
</h4><p>We have a PR for a new <a href="https://github.com/elastic/elasticsearch/pull/35718">aggregation</a> that is designed to surface rare terms. This is designed to be used instead of ordering a terms agg by count ascending which has an unbounded error. The rare terms aggregation uses a different strategy to merge the results from different shards to ensure a better precision.
</p><h4>Build
</h4><p>As every PR against our repo is built on our CI infrastructure as part of the review process, it is very important to have fast feedback cycles. By <a href="https://github.com/elastic/elasticsearch/pull/35473">splitting the build</a> and running it on multiple worker nodes we were able to reduce the total PR build time by 50% down to 90 minutes. We have also enabled <a href="https://gradle.com/build-scans/">Gradle build scans</a> so we can find more room for improvements.
</p><h4>Backup of Security
</h4><p>It&#x2019;s currently possible to backup the .security index, but it&#x2019;s not an ideal experience. We have <a href="https://github.com/elastic/elasticsearch/issues/34454">a plan of tasks</a> we want to complete to improve this experience and we have started working through this list.
</p><p>We have opened <a href="https://github.com/elastic/elasticsearch/pull/35820">a PR for a &#x201C;snapshot&#x201D; privilege</a> and have been thinking about how to allow some low-privileged users the ability to list the .security index, and potentially see some metadata about it, but not necessarily be able to search it, or get documents from it.
</p><h4>Zen 2 
</h4><p>We now have a storage layer that can <a href="https://github.com/elastic/elasticsearch/pull/35049">atomically write cluster states</a>, and the <a href="https://github.com/elastic/elasticsearch/issues/33958">remaining follow-up work</a> will focus on integrating the new storage layer with the existing Zen2 components.
</p><p>We also fixed a long-standing issue where nodes miss applying a cluster state update, indefinitely lagging behind the master in terms of applied cluster states. This can potentially impact other operations in the cluster, for example block indexing operations to wait on mapping updates. The solution is for the master to <a href="https://github.com/elastic/elasticsearch/pull/35685">track applied cluster state versions</a> of each node and to kick a node out of the cluster if it doesn&apos;t apply a published cluster state or any of its follow-up published states within some timeout, giving us a strong bound on how stale the cluster state applied on every node in the cluster can be.
</p><h4>Cross Cluster Replication
</h4><p>We are also continuing to <a href="https://github.com/elastic/elasticsearch/pull/35594">investigate</a> and benchmark update- and refresh-heavy use cases, to see if the number of refreshes (and merges) can be improved without ruining the indexing throughput. This work has led to two Lucene changes to improve lock contention when indexing happens concurrently to refresh / open reader calls: 1) <a href="https://issues.apache.org/jira/browse/LUCENE-8569">Improve merge performance</a> in the common case where soft- and hard-deletes are not mixed and 2) <a href="https://issues.apache.org/jira/browse/LUCENE-8571">prevent blocking on resolving deletes</a>.
</p><p>The work to implement &quot;recovery from remote&quot; by <a href="https://github.com/elastic/elasticsearch/pull/35719">bootstrapping</a> the creation of the follower index through the Snapshot/Restore functionality has started by prototyping a CCR Snapshot/Restore repository that gets <a href="https://github.com/elastic/elasticsearch/pull/35801">registered based on available remote clusters</a>.
</p><h4>Changes
</h4><p>Changes in 6.4:
</p><ul>
	<li>[CI] Reactivate 3rd party tests on CI [branch 6.4] <a href="https://github.com/elastic/elasticsearch/pull/35733">#35733</a></li>
</ul><p>Changes in 6.5:
</p><ul>
	<li>Avoid NPE in follower stats when no tasks metadata <a href="https://github.com/elastic/elasticsearch/pull/35802">#35802</a></li>
	<li>[Scripting] Use Number as a return value for BucketAggregationScript <a href="https://github.com/elastic/elasticsearch/pull/35653">#35653</a></li>
	<li>SQL: Improve validation of unsupported fields <a href="https://github.com/elastic/elasticsearch/pull/35675">#35675</a></li>
</ul><p>Changes in 6.6:
</p><ul>
	<li>BREAKING: Forbid negative scores in functon_score query <a href="https://github.com/elastic/elasticsearch/pull/35709">#35709</a></li>
	<li>SQL: Implement NVL(expr1, expr2) <a href="https://github.com/elastic/elasticsearch/pull/35794">#35794</a></li>
	<li>TransportResyncReplicationAction should not honour blocks <a href="https://github.com/elastic/elasticsearch/pull/35795">#35795</a></li>
	<li>GEO: More robust handling of ignore_malformed in geoshape parsing <a href="https://github.com/elastic/elasticsearch/pull/35603">#35603</a></li>
	<li>SQL: Implement ISNULL(expr1, expr2) <a href="https://github.com/elastic/elasticsearch/pull/35793">#35793</a></li>
	<li>HLRC: ML Delete event from Calendar <a href="https://github.com/elastic/elasticsearch/pull/35760">#35760</a></li>
	<li>[HLRC] Fix issue in equals impl for GlobalOperationPrivileges <a href="https://github.com/elastic/elasticsearch/pull/35721">#35721</a></li>
	<li>[GEO] Add support to ShapeBuilders for building Lucene geometry <a href="https://github.com/elastic/elasticsearch/pull/35707">#35707</a></li>
	<li>SQL: Implement IFNULL variant of COALESCE <a href="https://github.com/elastic/elasticsearch/pull/35762">#35762</a></li>
	<li>SQL: Perform lazy evaluation of mismatched mappings <a href="https://github.com/elastic/elasticsearch/pull/35676">#35676</a></li>
	<li>[HLRC] Add support for get application privileges API <a href="https://github.com/elastic/elasticsearch/pull/35556">#35556</a></li>
	<li>[Tests] Fix SimpleQueryStringBuilderTests <a href="https://github.com/elastic/elasticsearch/pull/35784">#35784</a></li>
	<li>manage_token privilege for kibana_system <a href="https://github.com/elastic/elasticsearch/pull/35751">#35751</a></li>
	<li>SQL: Introduce INTERVAL support <a href="https://github.com/elastic/elasticsearch/pull/35521">#35521</a></li>
	<li>[HLRC] Added support for CCR Resume Follow API <a href="https://github.com/elastic/elasticsearch/pull/35638">#35638</a></li>
	<li>Clean up PutLicenseResponse <a href="https://github.com/elastic/elasticsearch/pull/35689">#35689</a></li>
	<li>Clean up StartBasicResponse <a href="https://github.com/elastic/elasticsearch/pull/35688">#35688</a></li>
	<li>Add a prebuilt ICU Analyzer <a href="https://github.com/elastic/elasticsearch/pull/34958">#34958</a></li>
	<li>[HLRC] Added support for CCR Unfollow API <a href="https://github.com/elastic/elasticsearch/pull/35693">#35693</a></li>
	<li>Fix problem with MatchNoDocsQuery in disjunction queries <a href="https://github.com/elastic/elasticsearch/pull/35726">#35726</a></li>
	<li>HLRC: ML Get Calendar Events <a href="https://github.com/elastic/elasticsearch/pull/35747">#35747</a></li>
	<li>HLRC ML Add Event To Calendar API <a href="https://github.com/elastic/elasticsearch/pull/35704">#35704</a></li>
	<li>HLRC: ML Delete job from calendar <a href="https://github.com/elastic/elasticsearch/pull/35713">#35713</a></li>
	<li>Add a _freeze / _unfreeze API <a href="https://github.com/elastic/elasticsearch/pull/35592">#35592</a></li>
	<li>HLRC: Add &quot;_has_privileges&quot; API to Security Client <a href="https://github.com/elastic/elasticsearch/pull/35479">#35479</a></li>
	<li>HLRC for _mtermvectors <a href="https://github.com/elastic/elasticsearch/pull/35266">#35266</a></li>
	<li>HLRC: ML Add Job to Calendar API <a href="https://github.com/elastic/elasticsearch/pull/35666">#35666</a></li>
	<li>Build: Fix official plugins list <a href="https://github.com/elastic/elasticsearch/pull/35661">#35661</a></li>
	<li>HLRC: Add ML delete filter action <a href="https://github.com/elastic/elasticsearch/pull/35382">#35382</a></li>
	<li>SQL: Move internals from Joda to java.time <a href="https://github.com/elastic/elasticsearch/pull/35649">#35649</a></li>
	<li>Fix phrase_slop in query_string query <a href="https://github.com/elastic/elasticsearch/pull/35533">#35533</a></li>
</ul><p>Changes in 7.0:
</p><ul>
	<li>Add a new query type - ScriptScoreQuery <a href="https://github.com/elastic/elasticsearch/pull/34533">#34533</a></li>
	<li>Deprecate types in validate query requests. <a href="https://github.com/elastic/elasticsearch/pull/35575">#35575</a></li>
	<li>Deprecate types in count and msearch. <a href="https://github.com/elastic/elasticsearch/pull/35421">#35421</a></li>
</ul><h3>Lucene
</h3><h4>BM25F
</h4><p>A <a href="https://issues.apache.org/jira/browse/LUCENE-8216">first implementation of BM25F</a> has been pushed to sandbox. BM25F is an extension to BM25 that supports scoring across multiple fields with different degrees of importance. The way it works is quite simple, for instance say that you want to give <code>title</code> a weight of 5 and <code>body</code> a weight of 1 then BM25F will score across these two fields as if there was a single field that consisted of the concatenation of the body field and 5 times the title field. While this first implementation is already helpful, this is currently implemented via a custom query that ignores the similarity that is configured on IndexSearcher. There is still work needed in order to make it more flexible, eg. by accepting different values of b (the BM25 parameter that controls length normalization) per field and making it easier to use. We hope to eventually leverage this functionality to improve scoring in Elasticsearch when using the <code>cross_fields</code> way of scoring across multiple fields.
</p><h4>Benchmarks for LatLonShape
</h4><p>While we would eventually want to benchmark LatLonShape - our new BKD-based support for shapes - with realistic datasets of shapes (polygons, lines, etc), we started simple by running our current benchmark for points with LatLonShape. This already gives us a good indication of how much overhead LatLonShape adds compared with LatLonPoint when only indexing points. See the purple line at <a href="http://people.apache.org/~mikemccand/geobench.html">http://people.apache.org/~mikemccand/geobench.html</a> if you are curious. LatLonShape is still at an early stage of its development and we hope to bridge some of the gap with LatLonPoint in the near future.
</p><h4>Faster merging when indexing geo shapes
</h4><p>We recently enhanced BKD trees so that only a subset of dimensions is indexed, which in-turn helps use BKD trees as a R-tree. We noticed that we are still doing work on non-indexed dimensions which is <a href="https://issues.apache.org/jira/browse/LUCENE-8562">not necessary</a>. Fixing it is expecting to make merging faster for fields that index shapes. We will hopefully soon see a bump in the indexing speed charts (see above paragraph).
</p><h4>Less contention when applying deletes
</h4><p>Deletes may be applied either while indexing or refreshing. Today this is a blocking call, which might block some indexing threads. However blocking is only required when refreshing, in order to make sure that all deletes are reflected in the new point-in-time view of the index. <a href="https://issues.apache.org/jira/browse/LUCENE-8571">an optimization that only makes it blocking for refreshes</a> has been implemented to solve this.
</p><h4>Normalization as a first-class citizen
</h4><p>The ability for analysis chains to only normalize content and skip tokenization, stemming and all other components that don&apos;t work with partial terms (think of wildcard queries) always felt a bit like a second-class citizen as you had to check for whether a component implemented a special marker interface to know whether it should be applied for normalization. This was fixed by <a href="https://issues.apache.org/jira/browse/LUCENE-8497">moving normalization directly to the CharFilter and TokenFilter factory classes</a>.
</p><h4>Resources around Lucene
</h4><p>As highlighted in a <a href="https://twitter.com/jpountz/status/1065534715120226304">recent Twitter conversation</a>, the Lucene wiki has a useful <a href="https://wiki.apache.org/lucene-java/LucenePapers">page</a> that collects various resources that either describe Lucene&apos;s internals or have been influential.
</p><h4>Other
</h4><p> - The Lucene 7.6 release process is on pause during Thanksgiving.
</p><p> - Nori, the korean tokenizer, could hit a <a href="https://issues.apache.org/jira/browse/LUCENE-8570">NullPointerException if the part-of-speech attribute is not set</a>.
</p><p> - Nori doesn&apos;t do a good job with <a href="https://issues.apache.org/jira/browse/LUCENE-8548">non-CJK tokens</a>.
</p><p> - The &quot;flexible&quot; query parser can fail with <a href="https://issues.apache.org/jira/browse/LUCENE-8572">index-out-of-bounds exceptions</a> with non-default locales.
</p><p> - In the general case, counting soft deletes upon merge requires a linear scan. Simon <a href="https://issues.apache.org/jira/browse/LUCENE-8569">optimized the common special case that a segment consists only of soft deletes</a>, as in that case the number of soft deletes is already known via IndexReader#numDocs.
</p></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-11-23

<html><head></head><body><p><span style="font-size: 24px;">Elasticsearch
</span></p><p dir="ltr"><strong>Frozen/Closed Indices
</strong></p><p dir="ltr">We&#x2019;ve added an <a href="https://github.com/elastic/elasticsearch/pull/35431">optimization for frozen shards</a> that allows executing &#x201C;can_match&quot; phases efficiently without opening the underlying index reader. This is particularly useful for time-based indices where older indices will typically be moved to frozen indices. Filtering frozen indices with date filters can then efficiently skip large portions of frozen shards that can&apos;t match the query in the pre-filter phase. We also added <a href="https://github.com/elastic/elasticsearch/pull/35592">documentation for using frozen indices</a> as well as the relevant _freeze and _unfreeze APIs. If you want to learn more about using frozen indices, give this one a read.
</p><p dir="ltr">We&#x2019;re continuing working on the close index API, making <a href="https://github.com/elastic/elasticsearch/pull/35540">core changes to our replication and shard-level permit system</a> to allow a clean transition from open to closed index.
</p><p dir="ltr"><strong>Geo in SQL
</strong></p><p dir="ltr">We&#x2019;ve been working on <a href="https://github.com/elastic/elasticsearch/pull/35554">changes</a> <a href="https://github.com/elastic/elasticsearch/pull/35414">to make</a> geo in Elasticsearch more robust and have now laid enough groundwork so we can expose the first GeoSQL functions in the SQL plugin. The <a href="https://github.com/elastic/elasticsearch/pull/35416">first function</a> is <code>ST_WKTToSQL</code> and can be used to convert geo-shapes represented in WKT (Well Known Text) to GeoShape objects that Elasticsearch understands.
</p><p dir="ltr"><strong>Pre-hashed Password in PUT Users API
</strong></p><p dir="ltr">We <a href="https://github.com/elastic/elasticsearch/pull/35536">added formal support</a> for the password_hash field in the put users api. This came about because of a <a href="https://github.com/elastic/elasticsearch/issues/33845">request</a> to have a bulk API for creating users; we discussed the notion of a bulk API for users and did not feel this was the right direction. However, the request identified a valid concern with the throughput of the users API.  We performed a cursory performance test with a naive, single-threaded python script to create users.  With default options, we were able to create about 8.5 users per second, whereas with client-side hashing and refresh=false, we were able to create about 37 users per second.  If an administrator had 5000 users to create, this would take the overall time from about 10 minutes to about 2.5 minutes.
</p><p dir="ltr"><strong>Cross-Cluster Replication
</strong></p><p dir="ltr">We published the <a href="https://www.elastic.co/guide/en/elastic-stack-overview/current/xpack-ccr.html">CCR</a> <a href="https://www.elastic.co/guide/en/elastic-stack-overview/current/ccr-getting-started.html">getting started guide</a> and <a href="https://www.elastic.co/guide/en/elastic-stack-overview/current/ccr-overview.html">overview</a> docs for CCR beta. With 6.5 out of the door, our eyes are now all set on driving to a successful GA.  We are moving the auto-follow logic (based on the auto-follow patterns) to long-polling. The <a href="https://github.com/elastic/elasticsearch/pull/35535">first step</a> of this effort is to add a new option to the cluster state API to wait, up to a timeout, for a given metadata version.  
</p><p dir="ltr">We&#x2019;ve implemented a fix to <a href="https://github.com/elastic/elasticsearch/pull/35594">address indexing slowdowns</a> we observed in update- or refresh-heavy use cases and will be benchmarking the effects of this fix.
</p><p dir="ltr"><strong>Tasks API + Security
</strong></p><p dir="ltr">A bug was found shortly after the 6.5.0 release in that the permissions supplied for the Kibana system role prevent full use of the tasks api. Ultimately, this boils down to the fact that we leak the implementation detail that tasks are stored in an index and the kibana system role doesn&apos;t grant access to this index. For 6.5.1, we <a href="https://github.com/elastic/elasticsearch/pull/35573">added these permissions</a> as a short term fix.  As a temporary workaround, users may run Kibana as a user with the <code>kibana_system</code> role and a role that grants <code>&quot;create_index&quot;, &quot;read&quot;,  and &quot;create&quot;</code> privileges on the <code>.tasks</code> index.
</p><p dir="ltr"><strong>Audit Request ID
</strong></p><p dir="ltr">We <a href="https://github.com/elastic/elasticsearch/pull/35536">opened a PR</a> that will add a synthesized ID based on a user&#x2019;s request to each event in the file audit log.  This will allow users to correlate events such as &#x201C;authentication success&#x201D; and &#x201C;access granted&#x201D; for the same request.
</p><p dir="ltr"><strong>Zen2
</strong></p><p dir="ltr">We are working towards running all existing integration tests and REST test suites using Zen2 and are very close to putting the last missing pieces into place for this.
</p><p dir="ltr">We added the capability for <a href="https://github.com/elastic/elasticsearch/pull/35446">safely down-scaling</a> the number of master-eligible nodes in a cluster. With this, we can switch a substantial portion of our integration tests over to Zen2. We also introduced a <a href="https://github.com/elastic/elasticsearch/pull/35488">bootstrapping process</a> that will allow Elasticsearch to determine its own initial cluster configuration. At the moment this is controlled by a temporary node setting which works well for our REST tests (where we are sure to start the right number of nodes or die trying).  Initial tests with this PR show that we can soon switch our REST tests over to Zen2.
</p><p dir="ltr">We&#x2019;ve also completed the most <a href="https://github.com/elastic/elasticsearch/pull/35049">critical aspect</a> of the cluster state persistence layer that will give us much stronger atomicity guarantees.
</p><p dir="ltr">Finally, we implemented <a href="https://github.com/elastic/elasticsearch/pull/35443">serialization compatibility</a> between Zen1 and Zen2 transport actions, allowing a Zen2 node to join a fully-formed cluster with a Zen1 master and vice-versa. Follow-up work will focus on failure conditions and an automated transition from a Zen1 master to a Zen2 master, ultimately enabling a smooth rolling upgrade experience from 6.x to 7.0.
</p><p dir="ltr"><strong>Changes
</strong></p><p dir="ltr"><strong>Changes in 6.5:
</strong></p><ul>
	<li dir="ltr">Grant .tasks access to kibana_system role <a href="https://github.com/elastic/elasticsearch/pull/35573">#35573</a></li>
	<li dir="ltr">Handle IndexOrDocValuesQuery in composite aggregation <a href="https://github.com/elastic/elasticsearch/pull/35392">#35392</a></li>
	<li dir="ltr">SQL: clear the cursor if nested inner hits are enough to fulfill the query required limits <a href="https://github.com/elastic/elasticsearch/pull/35398">#35398</a></li>
	<li dir="ltr">Correct implemented interface of ParsedReverseNested <a href="https://github.com/elastic/elasticsearch/pull/35455">#35455</a></li>
	<li dir="ltr">SQL: Fix query translation for scripted queries <a href="https://github.com/elastic/elasticsearch/pull/35408">#35408</a></li>
	<li dir="ltr">Upgrade to Joda 2.10.1 <a href="https://github.com/elastic/elasticsearch/pull/35410">#35410</a></li>
</ul><p dir="ltr"><strong>Changes in 6.6:
</strong></p><ul>
	<li dir="ltr">HLRC: migration api - upgrade <a href="https://github.com/elastic/elasticsearch/pull/34898">#34898</a></li>
	<li dir="ltr">[RCI] Check blocks while having index shard permit in TransportReplicationAction <a href="https://github.com/elastic/elasticsearch/pull/35332">#35332</a></li>
	<li dir="ltr">HLRC: Add parameters to stopRollupJob API <a href="https://github.com/elastic/elasticsearch/pull/35545">#35545</a></li>
	<li dir="ltr">Clean up XPackInfoResponse class and related tests <a href="https://github.com/elastic/elasticsearch/pull/35547">#35547</a></li>
	<li dir="ltr">Extract RunOnce into a dedicated class <a href="https://github.com/elastic/elasticsearch/pull/35489">#35489</a></li>
	<li dir="ltr">Suppress CachedTimeThread in hot threads output <a href="https://github.com/elastic/elasticsearch/pull/35558">#35558</a></li>
	<li dir="ltr">HLRC: Adding ML Update Filter API <a href="https://github.com/elastic/elasticsearch/pull/35522">#35522</a></li>
	<li dir="ltr">Add Delete Privileges API to HLRC <a href="https://github.com/elastic/elasticsearch/pull/35454">#35454</a></li>
	<li dir="ltr">HLRC: add support for get license basic/trial status API <a href="https://github.com/elastic/elasticsearch/pull/33176">#33176</a></li>
	<li dir="ltr">Formal support for &quot;password_hash&quot; in Put User <a href="https://github.com/elastic/elasticsearch/pull/35242">#35242</a></li>
	<li dir="ltr">Add stop rollup job support to HL REST Client <a href="https://github.com/elastic/elasticsearch/pull/34702">#34702</a></li>
	<li dir="ltr">[Rollup] Add wait_for_completion option to StopRollupJob API <a href="https://github.com/elastic/elasticsearch/pull/34811">#34811</a></li>
	<li dir="ltr">HLRC: Adding ml get filters api <a href="https://github.com/elastic/elasticsearch/pull/35502">#35502</a></li>
	<li dir="ltr">Rest HL client: Add watcher stats API <a href="https://github.com/elastic/elasticsearch/pull/35185">#35185</a></li>
	<li dir="ltr">HLRC support for getTask <a href="https://github.com/elastic/elasticsearch/pull/35166">#35166</a></li>
	<li dir="ltr">Allow efficient can_match phases on frozen indices <a href="https://github.com/elastic/elasticsearch/pull/35431">#35431</a></li>
	<li dir="ltr">[HLRC] Added support for CCR Put Follow API <a href="https://github.com/elastic/elasticsearch/pull/35409">#35409</a></li>
	<li dir="ltr">Handle OS pretty name on old OS without OS release <a href="https://github.com/elastic/elasticsearch/pull/35453">#35453</a></li>
	<li dir="ltr">Geo: enables coerce support in WKT polygon parser <a href="https://github.com/elastic/elasticsearch/pull/35414">#35414</a></li>
	<li dir="ltr">[HLRC] Add GetRollupIndexCaps API <a href="https://github.com/elastic/elasticsearch/pull/35102">#35102</a></li>
	<li dir="ltr">Fix the names of CCR stats endpoints in usage API <a href="https://github.com/elastic/elasticsearch/pull/35438">#35438</a></li>
	<li dir="ltr">Add a java level freeze/unfreeze API <a href="https://github.com/elastic/elasticsearch/pull/35353">#35353</a></li>
</ul><p dir="ltr"><strong>Changes in 7.0:
</strong></p><ul>
	<li dir="ltr">Replace usages of AtomicBoolean based block of code by the RunOnce class <a href="https://github.com/elastic/elasticsearch/pull/35553">#35553</a></li>
</ul><p dir="ltr"><span style="font-size: 24px;">Apache Lucene
</span></p><p dir="ltr"><strong>Lucene 7.6
</strong></p><p dir="ltr">Most blockers for <a href="https://markmail.org/message/iwxgvfyntizfionf">Lucene 7.6</a> are resolved and we will cut the release branch soon.
</p><p dir="ltr"><strong>Searching for geo points in a polygon got faster
</strong></p><p dir="ltr">A costly operation when running point-in-polygon queries is to check whether a line segment crosses a rectangle. We added a simple optimization that first <a href="https://github.com/apache/lucene-solr/blob/95d01c6583b825b6b87591e4f27002c285ea25fb/lucene/core/src/java/org/apache/lucene/geo/EdgeTree.java#L330">checks whether either end of the segment belongs to the rectangle</a> before running <a href="https://www.cs.cmu.edu/~quake/robust.html">more costly computations</a>. This triggered a <a href="http://people.apache.org/~mikemccand/geobench.html#search-polyMedium">20% throughput improvement</a> for one query that we use for benchmarking which searches for points in London.
</p><p dir="ltr"><strong>Other
</strong></p><ul>
	<li dir="ltr">Should we <a href="https://issues.apache.org/jira/browse/LUCENE-8563">stop multiplying every BM25 score by (k1+1)</a>?</li>
	<li dir="ltr">Can we <a href="https://issues.apache.org/jira/browse/LUCENE-8551">garbage-collect unused fields</a>?</li>
	<li dir="ltr">Toke is proposing interesting ideas to <a href="https://issues.apache.org/jira/browse/LUCENE-8374">speed up reads on sparse doc values</a>, but there is disagreement whether this should be added to existing codecs via a cache, or only folded into a new codec and computed at index time.</li>
	<li dir="ltr">Itamar Syn-Hershko would like the simple query parser to <a href="https://issues.apache.org/jira/browse/LUCENE-8565">support a field operator</a>.</li>
	<li dir="ltr">Perhaps we should <a href="https://issues.apache.org/jira/browse/LUCENE-8564">make it easier to walk token graphs</a> in order to more easily add support for graphs to a greater number of token filters.</li>
	<li dir="ltr">Chistophe Bismuth has worked on making <a href="https://issues.apache.org/jira/browse/LUCENE-8026">ExitableDirectoryReader capable of interrupting range and geo queries</a>.</li>
	<li dir="ltr">Christophe is also working with us on <a href="https://issues.apache.org/jira/browse/LUCENE-8463">early-terminating queries sorted by _doc</a> and <a href="https://issues.apache.org/jira/browse/LUCENE-8464">speeding up collection of top-k hits with constant-scoring queries</a>.</li>
	<li dir="ltr">We found a bug that <a href="https://issues.apache.org/jira/browse/LUCENE-8556">identifying ears of a polygon based on the encoded coordinates is unsafe</a> and needs decoded coordinates.</li>
	<li dir="ltr">We identified room for improvement when <a href="https://issues.apache.org/jira/browse/LUCENE-8562">merging point fields that have data-only dimensions</a> as we are still doing operations on these dimensions that are only necessary for indexed dimensions.</li>
</ul></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-11-16

<html><head></head><body><h3 dir="ltr" id="docs-internal-guid-d750e132-7fff-0a1d-4a64-8f6a98a5dde8">Elasticsearch
</h3><h4 dir="ltr">OpenID Connect Realm
</h4><p dir="ltr">We are beginning work on OpenID Connect, an authentication layer that is built on top of OAuth2, which is an authorization protocol. We have <a href="https://github.com/elastic/elasticsearch/issues/35339">summarized implementation plan in a meta issue</a>.
</p><h4 dir="ltr">Disabling hit counts by default
</h4><p dir="ltr">We will be <a href="https://github.com/elastic/elasticsearch/issues/33028">disabling hit counts on search request by default in 7.0</a>. This means that we will not track the total hits on searches by default. Tracking total hits creates quite a lot of overhead to search requests and if we do not track total hits we can take advantage of some big improvements in Lucene where entire blocks of documents can be skipped if their results are not competitive. In some cases the search can be returned 1000s of times faster with these optimisations.
</p><p dir="ltr">There are a few important points to note here. Users can still opt to track the total hits and will have two options for this:
</p><ol>
	<li dir="ltr">Accurately track total hits - This option will provide an accurate count for the total hits as we have today but with the trade off that we will not be able to use any of the above mentioned optimisations so the user will not benefit from the search performance boosts.</li>
	<li dir="ltr">Track total hits up to a user defined value - With this option the user will specify the lower bound on the number of hits Elasticsearch should track in the search request. This means that if the user sets this value to 1000, Elasticsearch will accurately track the total hits (and therefore not use any of the performance optimisations) until it has seen 1000 hits (or until the search is complete if there are less than 1000 documents that match the query). When Elasticsearch has seen 1000 hits it will switch to using the performance optimisations and the total hits returned in the response will be a lower bound on the true total hits.</li>
</ol><p dir="ltr">The exact request and response changes we intend to make are still being worked out but we have <a href="https://github.com/elastic/elasticsearch/pull/35291">raised a PR</a> that shows one option of how this might work.
</p><h4 dir="ltr">Closed &amp; frozen indices
</h4><p dir="ltr">We <a href="https://github.com/elastic/elasticsearch/pull/34902">started</a> to <a href="https://github.com/elastic/elasticsearch/pull/35332">implement</a> our plan for transitioning an open index to closed. The goals of this transition are to ensure that new indexing requests are rejected while the existing data is properly flushed to disk, so that reopening the shards of this replicated closed index will not require a recovery from translog. This API will also be crucial for frozen indices, as frozen indices will rely on proper closing of the index first. Moving an index to frozen will internally consist of three steps
</p><ol>
	<li>Moving it to closed using the above functionality
	</li>
	<li>marking the index as frozen
	</li>
	<li>reopening it again, this time as frozen.
	</li>
</ol><p dir="ltr">Going through the close/open cycle allows us to switch out the &#x201C;engine&quot; implementation, which is the part of the shard that is responsible for executing the actual searches. We have implemented steps 2 and 3 with an API to <a href="https://github.com/elastic/elasticsearch/pull/35353">mark a closed index as frozen</a> and by adding a <a href="https://github.com/elastic/elasticsearch/pull/34357">frozen engine implementation</a> which allows lazily opening and releasing search resources by wrapping searchers in a LazyDirectoryReader that also allows to release and reset the underlying index readers after any and before secondary search phases.
</p><h4 dir="ltr">CCR and Networking
</h4><p dir="ltr">Cross-cluster replication will potentially ship large volumes of data all across the globe. To save on network bandwidth, we would like to <a href="https://github.com/elastic/elasticsearch/issues/34483">allow compressing remote-cluster traffic independently of within-cluster traffic</a>. We have introduced a <a href="https://github.com/elastic/elasticsearch/pull/35357">namespaced setting for compression</a> that allows users to configure compression on a per remote cluster basis. Tim has also <a href="https://github.com/elastic/elasticsearch/pull/35181">changed the number of threads</a> we use for network IO. The netty-based transport was using 2 * number of core threads both per client and per server profile. With this change, this modifies the netty transport to use 2 * number of core threads for the entire transport, aligning it with how the nio transport already works.
</p><h4 dir="ltr">Changes
</h4><p dir="ltr"><strong>Changes in 5.6:
	</strong>
</p><ul>
	<li dir="ltr">Fix DeleteRequest validation for nullable or empty id/type<a href="https://github.com/elastic/elasticsearch/pull/35314"> #35314</a></li>
</ul><p dir="ltr"><strong>Changes in 6.5:
	</strong>
</p><ul>
	<li dir="ltr">SQL: Handle null literal for AND and OR in WHERE<a href="https://github.com/elastic/elasticsearch/pull/35236"> #35236</a></li>
	<li dir="ltr">Engine.newChangesSnapshot may cause unneeded refreshes if called concurrently<a href="https://github.com/elastic/elasticsearch/pull/35169"> #35169</a></li>
	<li dir="ltr">watcher: Fix integration tests to ensure correct start/stop of Watcher<a href="https://github.com/elastic/elasticsearch/pull/35271"> #35271</a></li>
	<li dir="ltr">Scripting: Add back lookup vars in score script<a href="https://github.com/elastic/elasticsearch/pull/34833"> #34833</a></li>
	<li dir="ltr">Use soft-deleted docs to resolve strategy for engine operation<a href="https://github.com/elastic/elasticsearch/pull/35230"> #35230</a></li>
	<li dir="ltr">Ignore date ranges containing now when pre-processing a percolator query<a href="https://github.com/elastic/elasticsearch/pull/35160"> #35160</a></li>
	<li dir="ltr">SQL: Improve CircuitBreaker logic for SqlParser<a href="https://github.com/elastic/elasticsearch/pull/35300"> #35300</a></li>
	<li dir="ltr">Register Azure max_retries setting<a href="https://github.com/elastic/elasticsearch/pull/35286"> #35286</a></li>
	<li dir="ltr">SQL: Fix null handling for AND and OR in SELECT<a href="https://github.com/elastic/elasticsearch/pull/35277"> #35277</a></li>
	<li dir="ltr">SQL: Introduce NotEquals node to simplify expressions<a href="https://github.com/elastic/elasticsearch/pull/35234"> #35234</a></li>
	<li dir="ltr">Do not alloc full buffer for small change requests<a href="https://github.com/elastic/elasticsearch/pull/35158"> #35158</a></li>
	<li dir="ltr">SQL: Fix null handling for IN painless script<a href="https://github.com/elastic/elasticsearch/pull/35124"> #35124</a></li>
</ul><p dir="ltr"><strong>Changes in 6.6:
	
</strong></p><ul>
	<li dir="ltr">HLRC: Add InvalidateToken security API<a href="https://github.com/elastic/elasticsearch/pull/35114"> #35114</a></li>
	<li dir="ltr">Preserve format when aggregation contains unmapped date fields<a href="https://github.com/elastic/elasticsearch/pull/35254"> #35254</a></li>
	<li dir="ltr">Allow unmapped fields in composite aggregations<a href="https://github.com/elastic/elasticsearch/pull/35331"> #35331</a></li>
	<li dir="ltr">Remove ALL shard check in CheckShrinkReadyStep<a href="https://github.com/elastic/elasticsearch/pull/35346"> #35346</a></li>
	<li dir="ltr">[ILM] Check shard and relocation status in AllocationRoutedStep<a href="https://github.com/elastic/elasticsearch/pull/35316"> #35316</a></li>
	<li dir="ltr">Add a frozen engine implementation<a href="https://github.com/elastic/elasticsearch/pull/34357"> #34357</a></li>
	<li dir="ltr">Put a fake allocation id on allocate stale primary command<a href="https://github.com/elastic/elasticsearch/pull/34140"> #34140</a></li>
	<li dir="ltr">Apply ignore_throttled also to concrete indices<a href="https://github.com/elastic/elasticsearch/pull/35335"> #35335</a></li>
	<li dir="ltr">[CCR] Added HLRC support for pause follow API<a href="https://github.com/elastic/elasticsearch/pull/35216"> #35216</a></li>
	<li dir="ltr">HLRC: add support for the clear realm cache API<a href="https://github.com/elastic/elasticsearch/pull/35163"> #35163</a></li>
	<li dir="ltr">Fix UpdateRequest.fromXContent<a href="https://github.com/elastic/elasticsearch/pull/35257"> #35257</a></li>
	<li dir="ltr">SQL: Upgrade jline to version 3.8.2<a href="https://github.com/elastic/elasticsearch/pull/35288"> #35288</a></li>
	<li dir="ltr">SQL: new SQL CLI logo<a href="https://github.com/elastic/elasticsearch/pull/35261"> #35261</a></li>
	<li dir="ltr">ingest: dot_expander_processor prevent null add/append to source document<a href="https://github.com/elastic/elasticsearch/pull/35106"> #35106</a></li>
	<li dir="ltr">Prevent throttled indices to be searched through wildcards by default<a href="https://github.com/elastic/elasticsearch/pull/34354"> #34354</a></li>
	<li dir="ltr">SQL: Introduce Coalesce function<a href="https://github.com/elastic/elasticsearch/pull/35253"> #35253</a></li>
	<li dir="ltr">[Scripting] Make Max Script Length Setting Dynamic<a href="https://github.com/elastic/elasticsearch/pull/35184"> #35184</a></li>
	<li dir="ltr">Add dedicated step for checking shrink allocation status<a href="https://github.com/elastic/elasticsearch/pull/35161"> #35161</a></li>
	<li dir="ltr">[Monitoring] Add cluster metadata to cluster_stats docs (#33860)<a href="https://github.com/elastic/elasticsearch/pull/34023"> #34023</a></li>
	<li dir="ltr">Upgrade 6.x to lucene-7.6.0-snapshot-f9598f335b<a href="https://github.com/elastic/elasticsearch/pull/35225"> #35225</a></li>
	<li dir="ltr">Remove Joda usage from ILM<a href="https://github.com/elastic/elasticsearch/pull/35220"> #35220</a></li>
	<li dir="ltr">HLRC: Add ML API PUT filter<a href="https://github.com/elastic/elasticsearch/pull/35175"> #35175</a></li>
	<li dir="ltr">Small corrections to HLRC doc for _termvectors<a href="https://github.com/elastic/elasticsearch/pull/35221"> #35221</a></li>
	<li dir="ltr">Add document _count API support to Rest High Level Client.<a href="https://github.com/elastic/elasticsearch/pull/34267"> #34267</a></li>
	<li dir="ltr">Adds Index lifecycle feature<a href="https://github.com/elastic/elasticsearch/pull/35193"> #35193</a></li>
</ul><p dir="ltr"><strong>Changes in 7.0:
</strong></p><ul>
	<li dir="ltr">Make limit on number of expanded fields configurable<a href="https://github.com/elastic/elasticsearch/pull/35284"> #35284</a></li>
	<li dir="ltr">BREAKING: Logfile auditing settings remove after deprecation<a href="https://github.com/elastic/elasticsearch/pull/35205"> #35205</a></li>
	<li dir="ltr">Watcher: Ignore system locale/timezone in croneval CLI tool<a href="https://github.com/elastic/elasticsearch/pull/33215"> #33215</a></li>
	<li dir="ltr">Upgrade to lucene-8.0.0-snapshot-31d7dfe6b1<a href="https://github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/github.com/elastic/gitith wait_for_completion false&lt;a href=" https:="" github.com="" elastic="" elasticsearch="" pull="" 35202"=""> #35202</a></li>
</ul><h3 dir="ltr"><br></h3><h3 dir="ltr">Apache Lucene</h3><h4 dir="ltr">Lucene 7.6
</h4><p dir="ltr">We are chasing<a href="https://markmail.org/message/txhrhlwffppbbpop"> the last blockers</a> for the release.
</p><h4 dir="ltr">Reduce reads on sparse doc values
</h4><p dir="ltr">In Lucene 7 sparse doc values uses a block encoding that requires to read block headers when advancing to a random document. This <a href="https://issues.apache.org/jira/browse/LUCENE-8374">issue</a> tries to reduce these reads with an additional data structure that indexes the start of each block and allows to jump forward more efficiently. The patch is targeting 8.0 at the moment but there are also discussions to<a href="https://markmail.org/message/s3dvoweg3gt7kq5w"> backport</a> it to 7x.
</p><h4 dir="ltr">Other:
</h4><p dir="ltr">We added a support for a<a href="https://issues.apache.org/jira/browse/LUCENE-8554"> LatLonShapeQuery</a> that allows to query LatLonShape fields by    arbitrary lines.</p><p dir="ltr">We merged a user patch that fixes a<a href="https://issues.apache.org/jira/browse/LUCENE-8558"> performance bug</a> in field infos merging.
</p></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-11-09

<html><head></head><body><p dir="ltr"><span style="font-size: 24px;">Elasticsearch
</span></p><p dir="ltr"><strong>Highlights</strong></p><p dir="ltr"><strong>Dual-stack Plaintext/TLS
</strong></p><p dir="ltr">We&#x2019;ve started work on a dual-stack plaintext/TLS transport layer, with an objective of supporting unsecured to secured cluster transitions through a rolling upgrade.  We opened an <a href="https://github.com/elastic/elasticsearch/issues/35095">issue</a> to discuss what rules to put around this functionality, as we do not want to introduce vulnerabilities permitting downgrades to plain text.  We hope that this work will reduce issues moving from a trial license to a real license.
</p><p dir="ltr"><strong>Hadoop
</strong></p><p dir="ltr">ES-Hadoop now <a href="https://github.com/elastic/elasticsearch-hadoop/issues/1074">resolves mapping conflicts</a> when reading from multiple indices by &#x2018;up-casting&#x2019; fields to more generic types during the read. This eases changes made back 6.0.0 to be more assertive about mapping conflicts.
</p><p dir="ltr"><strong>Index Lifecycle Management
</strong></p><p dir="ltr">The feature branch for Index Lifecycle Management has been merged to the master and 6.x branches, where development will continue.  While this work is not yet complete, merging into our mainline development branches is a significant milestone and shows the level of testing and effort that has gone into this feature.
</p><p dir="ltr"><strong>Cross-Cluster Replication
</strong></p><p dir="ltr">We&#x2019;ve <a href="https://github.com/elastic/rally/pull/590">expanded</a> <a href="https://github.com/elastic/rally/pull/589">Rally</a>&#x2019;s support for benchmarking CCR, as well as performed some more important benchmarks of CCR&#x2019;s behavior, including security on/off and an append-only workload vs. a workload with conflicts.
</p><p dir="ltr">We&#x2019;re finishing a <a href="https://github.com/elastic/elasticsearch/pull/35089">pull request</a> to replicate settings changes from the source index to the follower index.  This work is critical to ensuring successful replication in the face of a settings change on the leader index affecting analysis for example.
</p><p dir="ltr"><strong>SAML Troubleshooting Docs
</strong></p><p dir="ltr">We&#x2019;ve added some stack <a href="https://github.com/elastic/stack-docs/pull/135">docs</a> to identify common problems and solutions when configuring SAML.
</p><p dir="ltr"><strong>Changelog
</strong></p><p dir="ltr"><strong>Changes in 6.4:
</strong></p><ul>
	<li dir="ltr">Make XContentBuilder in AliasActions build is_write_index field <a href="https://github.com/elastic/elasticsearch/pull/35071">#35071</a></li>
</ul><p dir="ltr"><strong>Changes in 6.5:
</strong></p><ul>
	<li dir="ltr">SQL: Improve painless script generated from IN <a href="https://github.com/elastic/elasticsearch/pull/35055">#35055</a></li>
	<li dir="ltr">Fix a bug in function_score queries where we use the wrong boost_mode. <a href="https://github.com/elastic/elasticsearch/pull/35148">#35148</a></li>
	<li dir="ltr">SQL: handle wildcard expansion on incorrect fields <a href="https://github.com/elastic/elasticsearch/pull/35134">#35134</a></li>
	<li dir="ltr">SQL: Register missing processors <a href="https://github.com/elastic/elasticsearch/pull/35121">#35121</a></li>
	<li dir="ltr">SQL: Fix NPE thrown if HAVING filter evals to null <a href="https://github.com/elastic/elasticsearch/pull/35108">#35108</a></li>
	<li dir="ltr">SQL: Optimizer rule for folding nullable expressions <a href="https://github.com/elastic/elasticsearch/pull/35080">#35080</a></li>
	<li dir="ltr">[Rollup] Proactively resolve index patterns in RollupSearch endoint <a href="https://github.com/elastic/elasticsearch/pull/34930">#34930</a></li>
	<li dir="ltr">SQL: Proper handling of nested fields at the beginning of the columns list <a href="https://github.com/elastic/elasticsearch/pull/35068">#35068</a></li>
	<li dir="ltr">SQL: Fix failing test due to random timezone <a href="https://github.com/elastic/elasticsearch/pull/35067">#35067</a></li>
	<li dir="ltr">SQL: Add CAST and CONVERT to SHOW FUNCTIONS <a href="https://github.com/elastic/elasticsearch/pull/34940">#34940</a></li>
	<li dir="ltr">SQL: Implement CAST between STRING and IP <a href="https://github.com/elastic/elasticsearch/pull/34949">#34949</a></li>
	<li dir="ltr">SQL: Fix function args verification and error msgs <a href="https://github.com/elastic/elasticsearch/pull/34926">#34926</a></li>
	<li dir="ltr">SQL: Fix incorrect AVG data type <a href="https://github.com/elastic/elasticsearch/pull/34948">#34948</a></li>
</ul><p dir="ltr"><strong>Changes in 6.6:
</strong></p><ul>
	<li dir="ltr">BREAKING: Watcher: fix metric stats names <a href="https://github.com/elastic/elasticsearch/pull/34951">#34951</a></li>
	<li dir="ltr">Allow to enable pings for specific remote clusters <a href="https://github.com/elastic/elasticsearch/pull/34753">#34753</a></li>
	<li dir="ltr">BREAKING: Loggers: Drop last deprecated logger function <a href="https://github.com/elastic/elasticsearch/pull/35082">#35082</a></li>
	<li dir="ltr">Packaging: Update procrun executables to version 1.1.0 <a href="https://github.com/elastic/elasticsearch/pull/35147">#35147</a></li>
	<li dir="ltr">XPack/HLRC request/response compatibility tests <a href="https://github.com/elastic/elasticsearch/pull/35129">#35129</a></li>
	<li dir="ltr">BREAKING: Core: Drop settings member from AbstractComponent <a href="https://github.com/elastic/elasticsearch/pull/35083">#35083</a></li>
	<li dir="ltr">HLRest: add security authenticate API <a href="https://github.com/elastic/elasticsearch/pull/33552">#33552</a></li>
	<li dir="ltr">BREAKING: HLRC XPack Protocol clean up: Migration; Graph; Watcher <a href="https://github.com/elastic/elasticsearch/pull/34639">#34639</a></li>
	<li dir="ltr">median absolute deviation agg <a href="https://github.com/elastic/elasticsearch/pull/34482">#34482</a></li>
	<li dir="ltr">Native roles store uses mget to retrieve roles <a href="https://github.com/elastic/elasticsearch/pull/33531">#33531</a></li>
	<li dir="ltr">BREAKING: HLRC XPack Protocol clean up: Migration; Graph; Watcher <a href="https://github.com/elastic/elasticsearch/pull/34964">#34964</a></li>
	<li dir="ltr">Deprecate _source_include and _source_exclude url parameters <a href="https://github.com/elastic/elasticsearch/pull/33475">#33475</a></li>
	<li dir="ltr">Add warning about upcoming expanded fields limit <a href="https://github.com/elastic/elasticsearch/pull/34906">#34906</a></li>
	<li dir="ltr">Add start rollup job support to HL REST Client <a href="https://github.com/elastic/elasticsearch/pull/34623">#34623</a></li>
	<li dir="ltr">HLRC: Add security Create Token API <a href="https://github.com/elastic/elasticsearch/pull/34791">#34791</a></li>
	<li dir="ltr">[HLRC] Add support for get role mappings API <a href="https://github.com/elastic/elasticsearch/pull/34637">#34637</a></li>
</ul><p dir="ltr"><strong>Changes in 7.0:
</strong></p><ul>
	<li dir="ltr">Bulk Api support for global parameters <a href="https://github.com/elastic/elasticsearch/pull/34528">#34528</a></li>
	<li dir="ltr">Replace parameter unicodeSetFilter with unicode_set_filter <a href="https://github.com/elastic/elasticsearch/pull/29215">#29215</a></li>
	<li dir="ltr">Remove deprecated url parameters _source_include and _source_exclude <a href="https://github.com/elastic/elasticsearch/pull/35097">#35097</a></li>
	<li dir="ltr">XPack/HLRC request/response compatibility tests <a href="https://github.com/elastic/elasticsearch/pull/34547">#34547</a></li>
	<li dir="ltr">HLRC: Drop extra level from user parser <a href="https://github.com/elastic/elasticsearch/pull/34932">#34932</a></li>
</ul><p dir="ltr"><br></p><p dir="ltr"><span style="font-size: 24px;">Lucene
</span></p><p dir="ltr">The 7.6 branch will be cut towards the end of next week, with the aim of releasing in the last week of November.
</p><ul>
	<li dir="ltr">We added a new LatLonShapeLineQuery that allows you to query for documents that either intersect or are disjoint from a line string (<a href="https://issues.apache.org/jira/browse/LUCENE-8554">https://issues.apache.org/jira/browse/LUCENE-8554</a>)</li>
	<li dir="ltr">Should we add some specialised filters to deal with how Hangul characters are entered on mobile devices? (<a href="https://issues.apache.org/jira/browse/LUCENE-8553">https://issues.apache.org/jira/browse/LUCENE-8553</a>)</li>
	<li dir="ltr">We made some improvements to the Tessellator classes that break shapes up into triangles for indexing and querying (<a href="https://issues.apache.org/jira/browse/LUCENE-8549">https://issues.apache.org/jira/browse/LUCENE-8549</a>)</li>
	<li dir="ltr">Discussions continue on what we should do with WordDelimiterGraphFilter (<a href="https://issues.apache.org/jira/browse/LUCENE-8509">https://issues.apache.org/jira/browse/LUCENE-8509</a>)</li>
</ul></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-11-02

<h2>Elasticsearch<br></h2><h4>Cross Cluster Replication (CCR)</h4><p dir="ltr">We added <a href="https://github.com/elastic/elasticsearch/pull/34726">docs for the CCR APIs</a>,  <a href="https://github.com/elastic/elasticsearch/pull/34638">simplified</a> some of the CCR APIs, and <a href="https://github.com/elastic/elasticsearch/pull/34776">aligned the terminology</a> used in the CCR APIs to be closer to what we have for Cross Cluster Search. We also <a href="https://github.com/elastic/elasticsearch/pull/34797">added a new byte-based limit for the write buffer</a> that’s used on the follower to temporarily hold the documents that need to be pushed to the follower shards. The current limit was based on the count of documents in the buffer, making it difficult to come up with a good default as the memory usage then depended on the size of the documents. When the new size limit is hit (512MB by default), fetching changes from the leader cluster will be paused to give the follower shards a chance to catch up with the current documents in the buffer.
</p><p dir="ltr">We continued investigating the effect of concurrent connections on cross-region traffic. We have also measured the read load created by CCR on the leader cluster vs. write load created on follower cluster, and studied the overhead that CCR has on the indexing rate of the leader cluster, findings are summarized <a href="https://github.com/elastic/elasticsearch/issues/31717#issuecomment-432219348">here</a>. The coming weeks will focus on <a href="https://gist.github.com/dliappis/67615df7622071400846ef890430f57a">further benchmarking and testing</a>, so that we can ship CCR with optimal defaults.
</p><h4>Improving indexing performance with selective flushing</h4><p dir="ltr">Currently when we perform a flush on a shard (when we write the segments in memory to disk) we flush all segments in memory at the same time. This can lead to situations where some of the segments we write to disk are large (which is good) and others are small (which is not so good as these will end up needing to be merged). We have <a href="https://github.com/elastic/elasticsearch/pull/34553">raised a PR</a> which will change this behaviour so on a flush we will only write the largest segment in memory to disk. This should mean that the segments flushed to disk are generally larger, require fewer merges and ultimately increase indexing throughput. We will benchmark this change to ensure it has the effect that we are expecting.
</p><h4>More functionality for the SQL plugin</h4><p dir="ltr">We have been working hard to implement new functionality in SQL.
We have 
	<a href="https://github.com/elastic/elasticsearch/pull/34581">added
    support
	</a> for <code>value IN (v1, v2, v3...)</code> expressions, <a href="https://github.com/elastic/elasticsearch/pull/34660">implemented CONVERT</a>, an alternative to CAST with a different syntax (required for ODBC), and is currently working on handling the NOT operator properly which has some complex behaviour when mixed with certain other operators.
</p><p dir="ltr">We have also been working on adding support for <a href="https://github.com/elastic/elasticsearch/pull/34573">NULL values</a> and <a href="https://github.com/elastic/elasticsearch/pull/34758">IP fields</a>, and <a href="https://github.com/elastic/elasticsearch/pull/34718">merged a change</a> that allows queries over multiple indexes where the mapping are not identical between the indexes (although they still need to be compatible).
</p><h4>Types Removal for 7.0</h4><p dir="ltr">We have merged <a href="https://github.com/elastic/elasticsearch/pull/34625">some</a> <a href="https://github.com/elastic/elasticsearch/pull/34662">PRs</a>  to make
    sure that our docs only use a 
	<code>_doc</code> type and also to remove the now obsolete "<a href="https://github.com/elastic/elasticsearch/pull/34663">type exist</a>" query. There is ongoing discussion around how we migrate user from the typed APIs from 6.x to the typeless APIs in 7.0 in a way that is friendly for both existing users upgrading their clusters and users deploying new clusters.
</p><h4>Changes in 5.6:</h4><ul>
	<li dir="ltr">Dependencies: Upgrade to joda time 2.10<a href="https://github.com/elastic/elasticsearch/pull/34760"> #34760</a></li>
	<li dir="ltr">Add a limit for graph phrase query expansion<a href="https://github.com/elastic/elasticsearch/pull/34061"> #34061</a></li>
	<li dir="ltr">[ML] Add missing return after calling listener<a href="https://github.com/elastic/x-pack-elasticsearch/pull/4470"> #4470</a></li>
</ul><h4>Changes in 6.2:</h4><ul>
	<li dir="ltr">Make accounting circuit breaker settings dynamic<a href="https://github.com/elastic/elasticsearch/pull/34372"> #34372</a></li>
</ul><h4>Changes in 6.4:</h4><ul>
	<li dir="ltr">Check self references in metric agg after last doc collection (#33593)<a href="https://github.com/elastic/elasticsearch/pull/34001"> #34001</a></li>
</ul><h4>Changes in 6.5:</h4><ul>
	<li dir="ltr">Adding stack_monitoring_agent role<a href="https://github.com/elastic/elasticsearch/pull/34369"> #34369</a></li>
	<li dir="ltr">SQL: Return error with ORDER BY on non-grouped.<a href="https://github.com/elastic/elasticsearch/pull/34855"> #34855</a></li>
	<li dir="ltr">Don’t omit default values when updating routing exclusions (#32721)<a href="https://github.com/elastic/elasticsearch/pull/33638"> #33638</a></li>
	<li dir="ltr">SQL: Allow min/max aggregates on date fields<a href="https://github.com/elastic/elasticsearch/pull/34699"> #34699</a></li>
	<li dir="ltr">[HLRC] Add support for Delete role mapping API<a href="https://github.com/elastic/elasticsearch/pull/34531"> #34531</a></li>
	<li dir="ltr">ingest: better support for conditionals with simulate?verbose<a href="https://github.com/elastic/elasticsearch/pull/34155"> #34155</a></li>
	<li dir="ltr">ingest: processor stats<a href="https://github.com/elastic/elasticsearch/pull/34724"> #34724</a></li>
	<li dir="ltr">SQL: Fix edge case: &lt;field&gt; IN (null)<a href="https://github.com/elastic/elasticsearch/pull/34802"> #34802</a></li>
	<li dir="ltr">SQL: handle X-Pack or X-Pack SQL not being available in a more graceful way<a href="https://github.com/elastic/elasticsearch/pull/34736"> #34736</a></li>
	<li dir="ltr">SQL: Introduce ODBC mode, similar to JDBC<a href="https://github.com/elastic/elasticsearch/pull/34825"> #34825</a></li>
	<li dir="ltr">Improved IndexNotFoundException’s default error message<a href="https://github.com/elastic/elasticsearch/pull/34649"> #34649</a></li>
	<li dir="ltr">NETWORKING: Add SSL Handler before other Handlers<a href="https://github.com/elastic/elasticsearch/pull/34636"> #34636</a></li>
	<li dir="ltr">HLRC: Deactivate Watch API<a href="https://github.com/elastic/elasticsearch/pull/34192"> #34192</a></li>
	<li dir="ltr">SQL: Introduce support for IP fields<a href="https://github.com/elastic/elasticsearch/pull/34758"> #34758</a></li>
	<li dir="ltr">SQL: Fix queries with filter resulting in NO_MATCH<a href="https://github.com/elastic/elasticsearch/pull/34812"> #34812</a></li>
	<li dir="ltr">SQL: Verifier allows aliases aggregates for sorting<a href="https://github.com/elastic/elasticsearch/pull/34773"> #34773</a></li>
	<li dir="ltr">Fix inner_hits retrieval when stored fields are disabled<a href="https://github.com/elastic/elasticsearch/pull/34652"> #34652</a></li>
	<li dir="ltr">SQL: Implement null handling for IN(v1, v2, ...)<a href="https://github.com/elastic/elasticsearch/pull/34750"> #34750</a></li>
	<li dir="ltr">Add cluster-wide shard limit warnings<a href="https://github.com/elastic/elasticsearch/pull/34021"> #34021</a></li>
	<li dir="ltr">Scripting: Add back params._source access in scripted metric aggs<a href="https://github.com/elastic/elasticsearch/pull/34777"> #34777</a></li>
	<li dir="ltr">Test: Fix last reference to SearchScript<a href="https://github.com/elastic/elasticsearch/pull/34731"> #34731</a></li>
	<li dir="ltr">SQL: Support pattern against compatible indices<a href="https://github.com/elastic/elasticsearch/pull/34718"> #34718</a></li>
	<li dir="ltr">SQL: Implement IN(value1, value2, …) expression.<a href="https://github.com/elastic/elasticsearch/pull/34581"> #34581</a></li>
	<li dir="ltr">SQL: Implement CONVERT, an alternative to CAST<a href="https://github.com/elastic/elasticsearch/pull/34660"> #34660</a></li>
	<li dir="ltr">SQL: the SSL default configuration shouldn’t override the https protocol if used<a href="https://github.com/elastic/elasticsearch/pull/34635"> #34635</a></li>
	<li dir="ltr">Handle missing user in user privilege APIs<a href="https://github.com/elastic/elasticsearch/pull/34575"> #34575</a></li>
	<li dir="ltr">Security: don’t call prepare index for reads<a href="https://github.com/elastic/elasticsearch/pull/34568"> #34568</a></li>
	<li dir="ltr">Fill LocalCheckpointTracker with Lucene commit<a href="https://github.com/elastic/elasticsearch/pull/34474"> #34474</a></li>
	<li dir="ltr">HLRC: Delete role API<a href="https://github.com/elastic/elasticsearch/pull/34620"> #34620</a></li>
	<li dir="ltr">BREAKING: Geo: Don’t flip longitude of envelopes crossing dateline<a href="https://github.com/elastic/elasticsearch/pull/34535"> #34535</a></li>
	<li dir="ltr">SQL: Introduce support for NULL values (#34573)<a href="https://github.com/elastic/elasticsearch/pull/34640"> #34640</a></li>
	<li dir="ltr">Fix completion suggester’s score tie-break<a href="https://github.com/elastic/elasticsearch/pull/34508"> #34508</a></li>
	<li dir="ltr">BREAKING: HLRC XPack Protocol clean up: Licence, Misc<a href="https://github.com/elastic/elasticsearch/pull/34469"> #34469</a></li>
	<li dir="ltr">[HLRC] Add Start/Stop Watch Service APIs.<a href="https://github.com/elastic/elasticsearch/pull/34317"> #34317</a></li>
	<li dir="ltr">Check stemmer language setting early<a href="https://github.com/elastic/elasticsearch/pull/34601"> #34601</a></li>
</ul><h4>Changes in 6.6:</h4><ul>
	<li dir="ltr">[Painless] Add instance bindings<a href="https://github.com/elastic/elasticsearch/pull/34410"> #34410</a></li>
	<li dir="ltr">XContent: Check for bad parsers<a href="https://github.com/elastic/elasticsearch/pull/34561"> #34561</a></li>
	<li dir="ltr">check for null argument is already done in splitStringByCommaToArray<a href="https://github.com/elastic/elasticsearch/pull/34268"> #34268</a></li>
</ul><h4>Changes in 7.0:</h4><ul>
	<li dir="ltr">HLRC API for _termvectors<a href="https://github.com/elastic/elasticsearch/pull/33447"> #33447</a></li>
	<li dir="ltr">Empty GetAliases authorization fix<a href="https://github.com/elastic/elasticsearch/pull/34444"> #34444</a></li>
	<li dir="ltr">Core: Move IndexNameExpressionResolver to java time<a href="https://github.com/elastic/elasticsearch/pull/34507"> #34507</a></li>
	<li dir="ltr">Deprecate type exists requests.<a href="https://github.com/elastic/elasticsearch/pull/34663"> #34663</a></li>
	<li dir="ltr">SQL: Introduce support for NULL values<a href="https://github.com/elastic/elasticsearch/pull/34573"> #34573</a></li>
	<li dir="ltr">BREAKING: Fix threshold frequency computation in Suggesters<a href="https://github.com/elastic/elasticsearch/pull/34312"> #34312</a></li>
	<li dir="ltr">Remove hand-coded XContent duplicate checks<a href="https://github.com/elastic/elasticsearch/pull/34588"> #34588</a></li>
	<li dir="ltr">Generate non-encrypted license public key<a href="https://github.com/elastic/elasticsearch/pull/34626"> #34626</a></li>
</ul><h2>Apache Lucene</h2><h4>Geo </h4><ul>
	<li>We have merged all the building blocks required to handle BKD shapes in Lucene, and are now working on exposing this work to Elasticsearch.</li>
	<li><a href="https://issues.apache.org/jira/browse/LUCENE-8496">improved the LatLonShape</a> encoding to speed up search at the cost of a bigger index.</li>
	<li><a href="https://issues.apache.org/jira/browse/LUCENE-8534">Chasing a complex bug</a> in the new bkd based shape field</li>
</ul><h4>Snowball arabic stemmer</h4><p dir="ltr">We have <a href="https://issues.apache.org/jira/browse/LUCENE-8462">contributed a snowball arabic stemmer</a> that will also be available in Lucene 7.6.
</p><p dir="ltr">Arabic language needs good stemming for effective information retrieval. Most Arabic dictionaries arrange their entries alphabetically by root. Meaning that to find the definition of a word, one must find its root first. This contribution extends the set of snowball based stemmers currently supported by Lucene by adding Arabic.
</p><p dir="ltr">The Arabic snowball algorithm was generated from <a href="https://github.com/snowballstem/snowball/blob/master/algorithms/arabic.sbl">https://github.com/snowballstem/snowball/blob/master/algorithms/arabic.sbl</a>. The algorithm is inspired by LarkeyBC02 (<a href="https://dl.acm.org/citation.cfm?doid=564376.564425">https://dl.acm.org/citation.cfm?doid=564376.564425</a>) with an extended support for special cases as well as a normalization step before stemming.
</p><h4>Korean tokenization</h4><ul>
	<li>The Korean dictionary used by Nori's tokenizer had some invalid rules with empty tokens and trailing spaces. These entries <a href="https://issues.apache.org/jira/browse/LUCENE-8462">have been normalized</a> (or removed).</li>
	<li>The Korean tokenizer <a href="https://issues.apache.org/jira/browse/LUCENE-8524">now considers</a> the Hangul Letter Araea (interpunct) as a separator</li>
	<li>The Korean tokenizer should not split on compatible scripts boundaries.</li>
</ul><h4>Other</h4><ul>
	<li> <a href="https://issues.apache.org/jira/browse/LUCENE-8531">We switched</a> back to phrase query when building a graph phrase query with a slop greater than 0 in order to ensure consistency with other phrase queries.</li>
	<li><a href="https://issues.apache.org/jira/browse/LUCENE-8543">Should we</a> have a way to build unordered span queries in the QueryBuilder?</li>
	<li>&nbsp;<a href="https://issues.apache.org/jira/browse/LUCENE-8497">We are working</a> on a type safe implementation of the normalization of terms applied in token filters.</li>
	<li> The word delimiter filter <a href="https://issues.apache.org/jira/browse/LUCENE-8509">can produce backward offsets</a> when used in combination with other filters.</li>
</ul>

This Week in Elasticsearch and Apache Lucene - 2018-10-26

<html><head></head><body><p dir="ltr"><span style="font-size: 24px;">Elasticsearch
</span></p><p dir="ltr"><strong>User Privileges API
</strong></p><p dir="ltr">We merged the new <a href="https://github.com/elastic/elasticsearch/pull/33928">get user privileges api</a>. This api will be used by Kibana to retrieve a list of resources patterns from ES security, and use that to determine whether a user has any Kibana space access.
</p><p dir="ltr"><strong>API Keys 
</strong></p><p dir="ltr">In order to support hadoop jobs and the Kibana task manager, we need a way to represent a user that has a much longer expiration than the current tokens that we provide. The plan to support this is to add support for <a href="https://github.com/elastic/elasticsearch/issues/34383">api keys</a>, which can have an expiration and be &quot;scoped&quot; using a role definition.  We raised a PR to <a href="https://github.com/elastic/elasticsearch/pull/34572">add support for creating these keys</a>, which will be incorporated into a feature branch as we continue development.
</p><p dir="ltr"><strong>Zen2
</strong></p><p dir="ltr">To provide high availability in a cluster where master-eligible nodes are failing, being replaced or retired, a Zen2 master auto-adapts (a.k.a. reconfigures) the set of master-eligible nodes that have a voting right in future leader elections. We have implemented the <a href="https://github.com/elastic/elasticsearch/pull/33924">reconfiguration rules</a>, integrated the <a href="https://github.com/elastic/elasticsearch/pull/34592">auto-reconfiguration</a> logic, and added tests to show that the system always converges in a reasonable time to the optimal configuration.
</p><p dir="ltr"><strong>CCR
</strong></p><p dir="ltr">In a situation where a primary shard in a following index fails on the following cluster, the new primary will end up retrying some recent operations, which may cause duplicate operations to be processed.  We added support for dealing with these duplicates, which requires <a href="https://github.com/elastic/elasticsearch/pull/34412">special handling in the follower replication logic</a> to ensure that all shard copies have the same term for this retried operation.
</p><p dir="ltr">We made some <a href="https://github.com/elastic/elasticsearch/pull/34366">smaller API</a> <a href="https://github.com/elastic/elasticsearch/pull/34518">changes</a> and added support for keeping persistent tasks around after they have failed to <a href="https://github.com/elastic/elasticsearch/pull/34404">keep track of the fatal error</a>.  This helps users figure out what error caused a shard follow task to fail having to find a log entry on the specific node that task ran on. We also <a href="https://github.com/elastic/elasticsearch/pull/34577">added a new stat</a> to the CCR stats api that reports the total fetch time from the perspective of the leader cluster. Currently the stats api already reports the total fetch time from the perspective of the follower cluster. This new stat in combination with the existing total fetch time stat turned out to be valuable in order to determine whether the time is spent on collecting the operations to replicate in leader cluster or sending these operations over the network to the follower cluster.
</p><p dir="ltr"><strong>Frozen indices
</strong></p><p dir="ltr">We laid out a <a href="https://github.com/elastic/elasticsearch/issues/34352">plan for implementing frozen indices</a> and opened PRs for the first steps, a <a href="https://github.com/elastic/elasticsearch/pull/33732">search_throttled threadpool</a>, additional logic to <a href="https://github.com/elastic/elasticsearch/pull/34354">prevent search_throttled indices to be searched though wildcards</a> by default and a <a href="https://github.com/elastic/elasticsearch/pull/34357">frozen engine implementation</a> that allows us to release resources after searches complete.
</p><p dir="ltr"><strong>Changes
</strong></p><p dir="ltr"><strong>Changes in 5.6:
</strong></p><ul>
	<li dir="ltr">Add a limit for graph phrase query expansion <a href="https://github.com/elastic/elasticsearch/pull/34061">#34061</a></li>
	<li dir="ltr">X-Pack:</li>
	<ul>
		<li dir="ltr">[ML] Add missing return after calling listener <a href="https://github.com/elastic/x-pack-elasticsearch/pull/4470">#4470</a></li>
	</ul>
</ul><p dir="ltr"><strong>Changes in 6.5:
</strong></p><ul>
	<li dir="ltr">Fill LocalCheckpointTracker with Lucene commit <a href="https://github.com/elastic/elasticsearch/pull/34474">#34474</a></li>
	<li dir="ltr">HLRC: Delete role API <a href="https://github.com/elastic/elasticsearch/pull/34620">#34620</a></li>
	<li dir="ltr">BREAKING: Geo: Don&#x2019;t flip longitude of envelopes crossing dateline <a href="https://github.com/elastic/elasticsearch/pull/34535">#34535</a></li>
	<li dir="ltr">SQL: Introduce support for NULL values (#34573) <a href="https://github.com/elastic/elasticsearch/pull/34640">#34640</a></li>
	<li dir="ltr">Fix completion suggester&#x2019;s score tie-break <a href="https://github.com/elastic/elasticsearch/pull/34508">#34508</a></li>
	<li dir="ltr">BREAKING: HLRC XPack Protocol clean up: Licence, Misc <a href="https://github.com/elastic/elasticsearch/pull/34469">#34469</a></li>
	<li dir="ltr">[HLRC] Add Start/Stop Watch Service APIs. <a href="https://github.com/elastic/elasticsearch/pull/34317">#34317</a></li>
	<li dir="ltr">Rollup add default metrics to histo groups <a href="https://github.com/elastic/elasticsearch/pull/34534">#34534</a></li>
	<li dir="ltr">Don&#x2019;t omit default values when updating routing exclusions (#32721) <a href="https://github.com/elastic/elasticsearch/pull/33638">#33638</a></li>
	<li dir="ltr">Check stemmer language setting early <a href="https://github.com/elastic/elasticsearch/pull/34601">#34601</a></li>
	<li dir="ltr">Core: Parse floats in epoch millis parser <a href="https://github.com/elastic/elasticsearch/pull/34504">#34504</a></li>
	<li dir="ltr">Add GetRollupCaps API to high level rest client <a href="https://github.com/elastic/elasticsearch/pull/32880">#32880</a></li>
	<li dir="ltr">Handle missing user in user privilege APIs <a href="https://github.com/elastic/elasticsearch/pull/34575">#34575</a></li>
	<li dir="ltr">[HLRC] Add support for Delete role mapping API <a href="https://github.com/elastic/elasticsearch/pull/34531">#34531</a></li>
	<li dir="ltr">[Rollup] Add support for date histo format when searching <a href="https://github.com/elastic/elasticsearch/pull/34537">#34537</a></li>
	<li dir="ltr">SQL: return constants for all matching records in constants-containing SELECTs <a href="https://github.com/elastic/elasticsearch/pull/34576">#34576</a></li>
	<li dir="ltr">Add get-user-privileges API <a href="https://github.com/elastic/elasticsearch/pull/33928">#33928</a></li>
	<li dir="ltr">Backport Structured Audit Logging <a href="https://github.com/elastic/elasticsearch/pull/33894">#33894</a></li>
	<li dir="ltr">Ensure map keys cannot be self referencing <a href="https://github.com/elastic/elasticsearch/pull/34569">#34569</a></li>
	<li dir="ltr">Pass the host name on as server_name if proxy mode is on <a href="https://github.com/elastic/elasticsearch/pull/34559">#34559</a></li>
	<li dir="ltr">add start trial API to HLRC <a href="https://github.com/elastic/elasticsearch/pull/33406">#33406</a></li>
	<li dir="ltr">Add Debug/Trace logging to token service <a href="https://github.com/elastic/elasticsearch/pull/34022">#34022</a></li>
	<li dir="ltr">HLRC: Add support for XPack Post Start Basic Licence API <a href="https://github.com/elastic/elasticsearch/pull/33606">#33606</a></li>
	<li dir="ltr">JDBC: Fix artifactId in pom <a href="https://github.com/elastic/elasticsearch/pull/34478">#34478</a></li>
	<li dir="ltr">Create/Update role mapping API <a href="https://github.com/elastic/elasticsearch/pull/34171">#34171</a></li>
	<li dir="ltr">Deprecate negative query boost (#34486) <a href="https://github.com/elastic/elasticsearch/pull/34512">#34512</a></li>
	<li dir="ltr">HLRC: Get SSL Certificates API <a href="https://github.com/elastic/elasticsearch/pull/34135">#34135</a></li>
	<li dir="ltr">Rollup adding support for date field metrics (#34185) <a href="https://github.com/elastic/elasticsearch/pull/34200">#34200</a></li>
	<li dir="ltr">[Monitoring] Add additional necessary mappings for apm-server <a href="https://github.com/elastic/elasticsearch/pull/34392">#34392</a></li>
	<li dir="ltr">Lock down Engine.Searcher <a href="https://github.com/elastic/elasticsearch/pull/34363">#34363</a></li>
	<li dir="ltr">Introduce index settings version <a href="https://github.com/elastic/elasticsearch/pull/34429">#34429</a></li>
	<li dir="ltr">HLRC: Fix the tags to be in sync with docs. <a href="https://github.com/elastic/elasticsearch/pull/34501">#34501</a></li>
	<li dir="ltr">Do not update number of replicas on no indices <a href="https://github.com/elastic/elasticsearch/pull/34481">#34481</a></li>
	<li dir="ltr">BREAKING: Use RoleRetrievalResult for better caching <a href="https://github.com/elastic/elasticsearch/pull/34197">#34197</a></li>
	<li dir="ltr">Fix handling of empty keyword in terms aggregation <a href="https://github.com/elastic/elasticsearch/pull/34457">#34457</a></li>
</ul><p dir="ltr"><strong>Changes in 7.0:
</strong></p><ul>
	<li dir="ltr">SQL: Introduce support for NULL values <a href="https://github.com/elastic/elasticsearch/pull/34573">#34573</a></li>
	<li dir="ltr">BREAKING: Fix threshold frequency computation in Suggesters <a href="https://github.com/elastic/elasticsearch/pull/34312">#34312</a></li>
	<li dir="ltr">Remove hand-coded XContent duplicate checks <a href="https://github.com/elastic/elasticsearch/pull/34588">#34588</a></li>
	<li dir="ltr">Watcher: Validate email adresses when storing a watch <a href="https://github.com/elastic/elasticsearch/pull/34042">#34042</a></li>
	<li dir="ltr">Add delete rollup job support to HL REST Client <a href="https://github.com/elastic/elasticsearch/pull/34066">#34066</a></li>
	<li dir="ltr">BREAKING: Disallow negative query boost <a href="https://github.com/elastic/elasticsearch/pull/34486">#34486</a></li>
</ul><p><span style="font-size: 24px;"><br></span></p><p><span style="font-size: 24px;">Lucene</span></p><p>No update this week, see you next time!</p></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-10-19

<html><head></head><body><h2 dir="ltr" id="docs-internal-guid-457e07b9-7fff-2495-9af7-41a55a4788ff">Elasticsearch</h2><p dir="ltr">Here&apos;s the latest update from the elasticsearch team.<br></p><h4 dir="ltr">Highlights
</h4><h5 dir="ltr">FIPS
</h5><p dir="ltr">PKCS11-NSS in FIPS mode support <a href="https://github.com/elastic/elasticsearch/pull/34063">has been fixed</a>, this will be available for 6.4.3. This concludes our FIPS 140 support story as Elasticsearch can run with a BouncyCastle FIPS provider, since 6.4.0, and a SunPKCS11-NSS provider in FIPS mode. These two are by far the most popular providers that have a FIPS mode for the JVMs we support. There is a very restrictive limitation with the use of SunPKCS11-NSS, and that is only a single Keystore can be used for Elasticsearch and this means, a single key,certificate pair, and TLS1.2 is also not currently supported. Details and the analysis behind these limitations can be found <a href="https://github.com/elastic/elasticsearch/issues/33459">here</a>.
</p><h5 dir="ltr">Ingest
</h5><p dir="ltr">In 6.5 an ingest pipeline can call other pipelines. We have <a href="https://github.com/elastic/elasticsearch/pull/33912">added support</a> to properly instrument statistics for pipelines in those cases as well.
</p><p dir="ltr">In 6.5 processors can be executed based on conditions. We have an <a href="https://github.com/elastic/elasticsearch/pull/34155">open PR</a> that adds better support for those processors, and also <a href="https://github.com/elastic/elasticsearch/pull/34202">a PR</a> to add instrumentation per processor (current, count, time, failed) to the existing ingest node stats pipeline instrumentation.
</p><h5 dir="ltr">ES Hadoop
</h5><p dir="ltr">We now <a href="https://github.com/elastic/elasticsearch-hadoop/issues/1197">accept and validate</a> aliases for writing that have an index set as the write index. This is the major component needed for supporting Index Lifecycle Management features going forward for Hadoop
</p><p dir="ltr">Sorting results using the first matching nested document
</p><p dir="ltr">We <a href="https://github.com/elastic/elasticsearch/pull/34225">made a change</a> that preserves the order of nested documents when writing them to the Lucene segment. The root document still comes last but the nested documents are written in the order specified in the request. This change allow us to progress with a community PR [3] which will add the ability to sort results based on the first matching nested document.
</p><h5 dir="ltr">Zen2
</h5><p dir="ltr">We have been focused on the<a href="https://github.com/elastic/elasticsearch/pull/34241"> Zen2 testing infrastructure</a>, which now allows simulating a run of a Zen2 cluster in the presence of network disruptions, unusual timings, and failing components. Test runs with this infrastructure are fully randomized, yet deterministic and reproducible. The purpose of this infrastructure is two-fold: 1) Explore Zen2&apos;s state space to find safety violations and 2) get the cluster into a weird state and then check that it recovers appropriately quickly. This test infrastructure has already proven to be super useful, bringing to light a number of smaller bugs in exceptional situations.
</p><p dir="ltr">We<a href="https://github.com/elastic/elasticsearch/pull/34257"> integrated the cluster state applier</a> into Zen2, which is the component that exposes cluster states to the rest of the system, for example to allocate or activate shards on a node. He has also used the above test infrastructure to check the cluster state acking functionality.
</p><h4 dir="ltr">Changes
</h4><h5 dir="ltr">Changes in 5.6:
</h5><ul>
	<li dir="ltr">Fix logging of cluster state update descriptions<a href="https://github.com/elastic/elasticsearch/pull/34243"> #34243</a></li>
</ul><h5 dir="ltr">Changes in 6.4:
</h5><ul>
	<li dir="ltr">Support PKCS#11 tokens as keystores and truststores<a href="https://github.com/elastic/elasticsearch/pull/34063"> #34063</a></li>
</ul><h5 dir="ltr">Changes in 6.5:
</h5><ul>
	<li dir="ltr">Tasks: Document that status is not semvered<a href="https://github.com/elastic/elasticsearch/pull/34270"> #34270</a></li>
	<li dir="ltr">Replace version with reader cache key in IndicesRequestCache<a href="https://github.com/elastic/elasticsearch/pull/34189"> #34189</a></li>
	<li dir="ltr">[CI] Fix bogus ScheduleWithFixedDelayTests.testRunnableRunsAtMostOnceAfterCancellation<a href="https://github.com/elastic/elasticsearch/pull/34296"> #34296</a></li>
	<li dir="ltr">HLRC: Add activate watch action<a href="https://github.com/elastic/elasticsearch/pull/33988"> #33988</a></li>
	<li dir="ltr">[Security] Get Alias API wildcard exclusion with Security<a href="https://github.com/elastic/elasticsearch/pull/34144"> #34144</a></li>
	<li dir="ltr">[Authz] Allow update settings action for system user<a href="https://github.com/elastic/elasticsearch/pull/34030"> #34030</a></li>
	<li dir="ltr">[TESTS] Set SO_LINGER and SO_REUSEADDR on the mock socket<a href="https://github.com/elastic/elasticsearch/pull/34211"> #34211</a></li>
	<li dir="ltr">Security: upgrade unboundid ldapsdk to 4.0.8<a href="https://github.com/elastic/elasticsearch/pull/34247"> #34247</a></li>
	<li dir="ltr">Add early termination support for min/max aggregations<a href="https://github.com/elastic/elasticsearch/pull/33375"> #33375</a></li>
	<li dir="ltr">Security: reduce memory usage of DnRoleMapper<a href="https://github.com/elastic/elasticsearch/pull/34250"> #34250</a></li>
	<li dir="ltr">BREAKING: Require combine and reduce scripts in scripted metrics aggregation<a href="https://github.com/elastic/elasticsearch/pull/33452"> #33452</a></li>
	<li dir="ltr">Preserve the order of nested documents in the Lucene index<a href="https://github.com/elastic/elasticsearch/pull/34225"> #34225</a></li>
	<li dir="ltr">[GCE Discovery] Automatically set project-id and zone<a href="https://github.com/elastic/elasticsearch/pull/33721"> #33721</a></li>
	<li dir="ltr">Add support for ack watch to the HLRC.<a href="https://github.com/elastic/elasticsearch/pull/33962"> #33962</a></li>
	<li dir="ltr">HLRC: Add throttling for update &amp; delete-by-query<a href="https://github.com/elastic/elasticsearch/pull/33951"> #33951</a></li>
	<li dir="ltr">Fix logging of cluster state update descriptions<a href="https://github.com/elastic/elasticsearch/pull/34182"> #34182</a></li>
	<li dir="ltr">BREAKING: Disallow &quot;enabled&quot; attribute change for types in mapping update (#33566)<a href="https://github.com/elastic/elasticsearch/pull/33933"> #33933</a></li>
	<li dir="ltr">HLRC: Add get rollup job<a href="https://github.com/elastic/elasticsearch/pull/33921"> #33921</a></li>
	<li dir="ltr">Fix use of hostname in Windows service<a href="https://github.com/elastic/elasticsearch/pull/34193"> #34193</a></li>
	<li dir="ltr">SQL: Remove more ANTLR4 grammar ambiguities<a href="https://github.com/elastic/elasticsearch/pull/34074"> #34074</a></li>
	<li dir="ltr">Fix cross fields mode of the query_string query<a href="https://github.com/elastic/elasticsearch/pull/34216"> #34216</a></li>
	<li dir="ltr">Completion types with multi-fields support<a href="https://github.com/elastic/elasticsearch/pull/34081"> #34ef=&quot;https://github.com/elastic/elasticsearch/pull/34193&quot;&gt; #34193</a></li>
	<li dir="ltr">SQL: Remove more ANTLR4 grammar ambiguities<a href="https://github.com/elastic/elasticsearch/pull/34074"> #34074</a></li>
	<li dir="ltr">Fix cross fields mode of the query_string query<a href="https://github.com/elastic/elasticsearch/pull/34216"> #34216</a></li>
	<li dir="ltr">Completion types with multi-fields support<a href="https://github.com/elastic/elasticsearch/pull/34081"> #34ef=&quot;https://github.com/elastic/elasticsearch/pull/34193&quot;&gt; #34193</a></li> #34072
	<li dir="ltr">SQL: Fix function resolution<a href="https://github.com/elastic/elasticsearch/pull/34137"> #34137</a></li>
</ul><h2 dir="ltr">Apache Lucene
</h2><h5 dir="ltr">Spans are hard
</h5><p dir="ltr">Getting span_near queries right is hard. The fact that there might be multiple spans at the same start position and that they might have different lengths causes the current span_near implementation to <a href="https://issues.apache.org/jira/browse/LUCENE-7398">sometimes omit some combinations</a>. This issue has been open for a long time but a (large - 10k new lines)<a href="https://issues.apache.org/jira/browse/LUCENE-7398?focusedCommentId=16630529&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16630529"> patch</a> was recently submitted. This might trigger interesting discussions regarding<a href="https://issues.apache.org/jira/browse/LUCENE-7398?focusedCommentId=15673323&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15673323"> spans vs. intervals</a> as intervals only expose minimum intervals, which helps<a href="https://issues.apache.org/jira/browse/LUCENE-8477"> avoid a number of these problems</a>.
</p><h5 dir="ltr">Other
</h5><ul><li> -WordDelimiterFilter wants to modify offsets when joining/splitting tokens yet modifying offsets is illegal from token filters as it is an inevitable source of bugs. As a consequence we are investigating<a href="https://issues.apache.org/jira/browse/LUCENE-8516"> making it a tokenizer</a>.</li><li>Analysis of graph queries should have some<a href="https://issues.apache.org/jira/browse/LUCENE-8479"> protection against too complex graphs</a>, similarly to the maximum clause count on boolean queries. We are tackling it.</li><li>How best to<a href="https://issues.apache.org/jira/browse/LUCENE-8353"> change the default behavior of FrenchLightStemmer</a> without breaking backward compatibility?</li><li>We removed<a href="https://issues.apache.org/jira/browse/LUCENE-8505"> leniency in IndexWriter#addIndexes</a>.
</li></ul><h5 dir="ltr">Adoptme
</h5><p dir="ltr">We believe that<a href="https://issues.apache.org/jira/browse/LUCENE-8510"> sorting an index by norm might help the new block-max-WAND optimizations perform better</a>. It would be interesting to introduce the ability to sort by norm at index-time and see whether it improves query performance compared to an index in a random order.
</p></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-10-05

<html><head></head><body><p><span style="font-size: 24px;">Elasticsearch
</span></p><p dir="ltr"><strong>Highlights
	</strong>
</p><p dir="ltr"><strong>Annotated Text Plugin
	</strong>
</p><p dir="ltr">We have merged the <a href="https://github.com/elastic/elasticsearch/pull/30364">annotated_text plugin</a> which allows users to index text that is a combination of free-text and special markup that is typically used to identify items of interest such as people or organizations (see NER or Named Entity Recognition tools). This new plugin will be available from 6.5.
</p><p dir="ltr"><strong>Watcher
	</strong>
</p><p dir="ltr">With the release of 6.4.1 we fixed an important distributed watch execution bug, that could lead to watches being distributed twice or not at all. To ensure support is aware we have written and marketed a knowledge base article.
</p><p dir="ltr">Some customers push Watcher to its limits and one area where this shows up is ineffective caching. We have implemented an <a href="https://github.com/elastic/elasticsearch/pull/33978">optimization</a> so we detect and avoid unnecessary compilation of templates that clearly don&apos;t require compilation and thus can be returned immediately as is instead of being put into a cache.
</p><p dir="ltr"><strong>Questions around removal of types
	</strong>
</p><p dir="ltr">We have been working on the changes required in 7.0 to support the removal of types from the index APIs. This has raised a number of new discussions:
</p><ul>
	<li dir="ltr">Should we add <a href="https://github.com/elastic/elasticsearch/pull/33953">include_type_name=false</a> to most API calls in the docs?</li>
	<li dir="ltr">How should we <a href="https://github.com/elastic/elasticsearch/pull/34041">make the high-level REST client evolve</a> with the removal of types?</li>
	<li dir="ltr">How can we both <a href="https://github.com/elastic/elasticsearch/pull/33934">keep testing mixed-version clusters</a> and skip types in rest tests?</li>
	<li dir="ltr">Should our <a href="https://github.com/elastic/elasticsearch/issues/15613#issuecomment-424630560">docs always set</a> include_type_name=false?</li>
</ul><p dir="ltr"><strong>Replicated Closed Indices
	</strong>
</p><p dir="ltr">Last week we revived the <a href="https://github.com/elastic/elasticsearch/pull/33903">NoOpEngine</a> (the engine that will be used by closed indices and has 0 memory overhead). We have also opened a <a href="https://github.com/elastic/elasticsearch/pull/34108">PR</a> to add entries in the routing table for shards of closed indices. The PR also adds logic to detect the transition between close and open indices on the data nodes and use to replace existing open shards (i.e., normal engines) with closed ones and vice versa.  Moving forward, closing an index will require all ongoing indexing to the that index to finish, that the global checkpoint has been fully caught up (something that happens quickly once indexing is stopped) and that all operations in the translogs are committed to Lucene.
</p><p dir="ltr"><strong>Zen2
	</strong>
</p><p dir="ltr">Achieving strong consistency properties for Zen2 requires a reliable storage mechanism for cluster states. The current storage mechanism for cluster states does not atomically write out a full cluster state, providing atomicity guarantees only at the per-index level.   We&#x2019;ve decided to move forward with a simple model that extends the current cluster state storage approach, adding the capability for atomically writing out a full cluster state. Our step-by-step plan is recorded on a <a href="https://github.com/elastic/elasticsearch/issues/33958">Github meta issue</a> and we have already started to actively work on the first steps.
</p><p dir="ltr"><strong>Changes
	</strong>
</p><p dir="ltr"><strong>Changes in 5.6:
	</strong>
</p><ul>
	<li dir="ltr">Security: use default scroll keepalive <a href="https://github.com/elastic/elasticsearch/pull/33639">#33639</a></li>
</ul><p dir="ltr"><strong>Changes in 6.4:
	</strong>
</p><ul>
	<li dir="ltr">Add a limit for graph phrase query expansion <a href="https://github.com/elastic/elasticsearch/pull/34031">#34031</a></li>
	<li dir="ltr">Fix AutoQueueAdjustingExecutorBuilder settings validation <a href="https://github.com/elastic/elasticsearch/pull/33922">#33922</a></li>
</ul><p dir="ltr"><strong>Changes in 6.5:</strong>
</p><ul>
	<li dir="ltr">SQL: Internal refactoring of operators as functions <a href="https://github.com/elastic/elasticsearch/pull/34097">#34097</a></li>
	<li dir="ltr">ingest: correctly measure chained pipeline stats <a href="https://github.com/elastic/elasticsearch/pull/33912">#33912</a></li>
	<li dir="ltr">XContentBuilder to handle BigInteger and BigDecimal <a href="https://github.com/elastic/elasticsearch/pull/32888">#32888</a></li>
	<li dir="ltr">[Monitoring] Update beats version <a href="https://github.com/elastic/elasticsearch/pull/34060">#34060</a></li>
	<li dir="ltr">SQL: Prevent StackOverflowError when parsing large statements <a href="https://github.com/elastic/elasticsearch/pull/33902">#33902</a></li>
	<li dir="ltr">Bad regex in CORS settings should throw a nicer error <a href="https://github.com/elastic/elasticsearch/pull/34035">#34035</a></li>
	<li dir="ltr">[HLRC] Support for role mapper expression dsl <a href="https://github.com/elastic/elasticsearch/pull/33745">#33745</a></li>
	<li dir="ltr">Watcher: Reduce script cache churn by checking for mustache tags <a href="https://github.com/elastic/elasticsearch/pull/33978">#33978</a></li>
	<li dir="ltr">Fold EngineSearcher into Engine.Searcher <a href="https://github.com/elastic/elasticsearch/pull/34082">#34082</a></li>
	<li dir="ltr">Build DocStats from SegmentInfos in ReadOnlyEngine <a href="https://github.com/elastic/elasticsearch/pull/34079">#34079</a></li>
	<li dir="ltr">Calculate changed roles on roles.yml reload <a href="https://github.com/elastic/elasticsearch/pull/33525">#33525</a></li>
	<li dir="ltr">Unmapped aggs should not run pipelines if they delegate reduction <a href="https://github.com/elastic/elasticsearch/pull/33528">#33528</a></li>
	<li dir="ltr">Add nested and object fields to field capabilities response <a href="https://github.com/elastic/elasticsearch/pull/33803">#33803</a></li>
	<li dir="ltr">SQL: Fix query translation of GroupBy with Having <a href="https://github.com/elastic/elasticsearch/pull/34010">#34010</a></li>
	<li dir="ltr">Add minimal sanity checks to custom/scripted similarities. (backport) <a href="https://github.com/elastic/elasticsearch/pull/33893">#33893</a></li>
	<li dir="ltr">Clarify RemoteClusterService#groupIndices behaviour <a href="https://github.com/elastic/elasticsearch/pull/33899">#33899</a></li>
	<li dir="ltr">NETWORKING: Upgrade to Netty 4.1.29 <a href="https://github.com/elastic/elasticsearch/pull/33984">#33984</a></li>
	<li dir="ltr">[Monitoring] Add cluster metadata to cluster_stats docs <a href="https://github.com/elastic/elasticsearch/pull/33860">#33860</a></li>
	<li dir="ltr">SQL: functions docs update <a href="https://github.com/elastic/elasticsearch/pull/34000">#34000</a></li>
	<li dir="ltr">SQL: Move away internally from JDBCType to SQLType <a href="https://github.com/elastic/elasticsearch/pull/33913">#33913</a></li>
	<li dir="ltr">Propagate auto_id_timestamp in primary-replica resync <a href="https://github.com/elastic/elasticsearch/pull/33964">#33964</a></li>
	<li dir="ltr">add elasticsearch-shard tool to 6.x <a href="https://github.com/elastic/elasticsearch/pull/33848">#33848</a></li>
</ul><p dir="ltr"><strong>Changes in 7.0:
	</strong>
</p><ul>
	<li dir="ltr">Scripting: Reflect factory signatures in painless classloader <a href="https://github.com/elastic/elasticsearch/pull/34088">#34088</a></li>
	<li dir="ltr">Search: Simply SingleFieldsVisitor <a href="https://github.com/elastic/elasticsearch/pull/34052">#34052</a></li>
</ul><p><span style="font-size: 24px;">Apache Lucene
</span></p><p dir="ltr"><strong>Spans are hard
</strong></p><p dir="ltr">Getting span_near queries right is hard. The fact that there might be multiple spans at the same start position and that they might have different lengths causes the current span_near implementation to <a href="https://issues.apache.org/jira/browse/LUCENE-7398">sometimes omit some combinations</a>. This issue has been open for a long time but a (large - 10k new lines) <a href="https://issues.apache.org/jira/browse/LUCENE-7398?focusedCommentId=16630529&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16630529">patch</a> was recently submitted. This might trigger interesting discussions regarding <a href="https://issues.apache.org/jira/browse/LUCENE-7398?focusedCommentId=15673323&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15673323">spans vs. intervals</a> as intervals only expose minimum intervals, which helps <a href="https://issues.apache.org/jira/browse/LUCENE-8477">avoid a number of these problems</a>.
</p><p dir="ltr"><strong>Other
</strong></p><ul>
	<li dir="ltr">WordDelimiterFilter wants to modify offsets when joining/splitting tokens yet modifying offsets is illegal from token filters as it is an inevitable source of bugs. As a consequence we are investigating <a href="https://issues.apache.org/jira/browse/LUCENE-8516">making it a tokenizer</a>.</li>
	<li dir="ltr">Analysis of graph queries should have some <a href="https://issues.apache.org/jira/browse/LUCENE-8479">protection against too complex graphs</a>, similarly to the maximum clause count on boolean queries.</li>
	<li dir="ltr">How best to <a href="https://issues.apache.org/jira/browse/LUCENE-8353">change the default behavior of FrenchLightStemmer</a> without breaking backward compatibility?</li>
	<li dir="ltr">We removed <a href="https://issues.apache.org/jira/browse/LUCENE-8505">leniency in IndexWriter#addIndexes</a>.</li>
</ul><p dir="ltr"><strong>Adoptme
</strong></p><p>We believe that <a href="https://issues.apache.org/jira/browse/LUCENE-8510">sorting an index by norm might help the new block-max-WAND optimizations perform better</a>. It would be interesting to introduce the ability to sort by norm at index-time and see whether it improves query performance compared to an index in a random order.
</p></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-09-28

<html><head></head><body><p dir="ltr" rel="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 24px;">Elasticsearch
</span></p><p dir="ltr"><strong>Google Summer of Code</strong></p><p dir="ltr"><span class="redactor-invisible-space">We merged the last of the work from <a href="https://github.com/sohaibiftikhar">Sohaib</a>, our Google Summer of Code student. This was an incredibly impressive showing, with the Sohaib completing eleven High-level Rest Client APIs over the summer!&#xA0; Sohaib&apos;s mentors spoke very highly of him throughout this process.&#xA0; Thank you for you contributions, Sohaib!</span></p><p dir="ltr"><strong>Highlights
</strong></p><p dir="ltr"><strong>Kerberos 
</strong></p><p dir="ltr">We&#x2019;re continuing improvements to our Kerberos support. Firefox has a long standing bug where the browser may not use the most secure authentication challenge it receives, instead using the first <code>WWW-Authenticate</code> challenge it receives when there is more than one such header present. In order to workaround this issue, we have added <a href="https://github.com/elastic/elasticsearch/pull/33156">auth scheme sorting</a>. The initial Kerberos support relies on role mapping, which can utilize metadata to map roles so we have added support for adding the realm and full user principal name as <a href="https://github.com/elastic/elasticsearch/pull/33338">metadata fields</a>.
</p><p dir="ltr"><strong>Cross-cluster replication
</strong></p><p dir="ltr">We have also completed the first integral part of <a href="https://github.com/elastic/elasticsearch/pull/33118">auto-follow patterns</a>, which allows users to define index patterns which will automatically trigger the creation of a replication relationship.  Whenever an index is created on a remote cluster matching the wildcard pattern defined by the user, a corresponding index will be created on the following cluster and the replication relationship will be established.  This feature will allow anyone with a time-based use cases (e.g., logging) to ensure that rollover/creation of new indices plays well with CCR.
</p><p dir="ltr"><strong>Lucene 8
</strong></p><p dir="ltr">We <a href="https://github.com/elastic/elasticsearch/pull/33310">upgraded the master branch of Elasticsearch to a snapshot of Lucene 8</a>. This is important to Elasticsearch as it allows us to start tracking any Lucene-induced bugs or performance regressions. More work remains to be done in order to <a href="https://github.com/elastic/elasticsearch/issues/33028">leverage new features of Lucene 8</a>.
</p><p dir="ltr"><strong>Early termination for aggregations
</strong></p><p dir="ltr">We have been working on adding <a href="https://github.com/elastic/elasticsearch/pull/33279">early termination</a> to aggregations. This allows aggregations to add shortcuts in the case where they do not require all documents in the current segment to be collected so the collection for the segment can terminate early and not waste time collecting documents that will not change the aggregation results. The first aggregations to take advantage of this are the min and max aggregations, <a href="https://github.com/elastic/elasticsearch/pull/33375">which will skip</a> collecting individual documents for a segment if the query matches all documents and instead will consult the BKD tree to extract the min/max value for that segment.
</p><p dir="ltr"><strong>Ingest Node
</strong></p><p dir="ltr">We <a href="https://github.com/elastic/elasticsearch/pull/32278">added</a> the ability to drop documents in a processor. This feature, inspired by the <a href="https://www.elastic.co/guide/en/logstash/6.4/plugins-filters-drop.html">drop filter</a> in Logstash, has been <a href="https://github.com/elastic/elasticsearch/issues/23726">requested</a> for some time. This functionality enables users to add a processor to a pipeline that can drop documents for any reason of their choosing. The typical use-case would be based on a conditional.
</p><p dir="ltr">We made an approximately 40% <a href="https://github.com/elastic/elasticsearch/pull/33029">performance improvement</a> to the geoip processor. This was a typical memory/performance tradeoff; we went from approximately 215 KB to 6 MB in a full cache to avoid de-serializing IP -&gt; geo cache hits.
</p><p dir="ltr"><strong>SQL now supports multi-index patterns
</strong></p><p dir="ltr">We added a support for <a href="https://github.com/elastic/elasticsearch/pull/33278">querying multiple indexes</a> in SQL using the multi-index patterns supported in the ES APIs. this means SQL users will now be able to perform queries like SELECT * FROM &quot;tes*,-test*&quot;.
</p><p dir="ltr"><strong>Breaking change in networking settings
</strong></p><p dir="ltr">Because of the role that remote cluster connections are taking wider than cross-cluster search (e.g., cross-cluster replication will use the same infrastructure), we have <a href="https://github.com/elastic/elasticsearch/pull/33413">merged</a> a change to rename the search.remote.*settings to cluster.remote.*. This change will appear in 6.5.0 with the search.remote.* settings being deprecated. We will fallback from cluster.remote.* settings to search.remote.* settings at runtime for the life of 7.x so that users existing configurations will continue to work. The fallback will be removed in 8.0.0. The long support for old settings is needed because a user could perform a full cluster restart from &lt; 6.5.0 to &gt;= 7.0.0. Additionally, we will <a href="https://github.com/elastic/elasticsearch/pull/33537">automatically upgrade</a> to cluster.remote.* any search.remote.*settings that are in the cluster state or set on dynamic cluster settings updates.
</p><p dir="ltr"><strong>Changes
</strong></p><p dir="ltr"><strong>Changes in 6.4:
</strong></p><ul>
	<li dir="ltr">Fix IndexMetaData loads after rollover <a href="https://github.com/elastic/elasticsearch/pull/33394">#33394</a></li>
	<li dir="ltr">Core: Fix IndicesSegmentResponse.toXcontent() serialization <a href="https://github.com/elastic/elasticsearch/pull/33414">#33414</a></li>
	<li dir="ltr">Allow query caching by default again <a href="https://github.com/elastic/elasticsearch/pull/33328">#33328</a></li>
	<li dir="ltr">Null completion field should not throw IAE <a href="https://github.com/elastic/elasticsearch/pull/33268">#33268</a></li>
	<li dir="ltr">[Rollup] Log deprecation if config and request intervals are mixed <a href="https://github.com/elastic/elasticsearch/pull/33284">#33284</a></li>
	<li dir="ltr">[Rollup] Fix Caps Comparator to handle calendar/fixed time <a href="https://github.com/elastic/elasticsearch/pull/33336">#33336</a></li>
</ul><p dir="ltr"><strong>Changes in 6.5:
</strong></p><ul>
	<li dir="ltr">[CCR] Added auto follow patterns feature <a href="https://github.com/elastic/elasticsearch/pull/33118">#33118</a></li>
	<li dir="ltr">HLRest: add xpack put user API <a href="https://github.com/elastic/elasticsearch/pull/32332">#32332</a></li>
	<li dir="ltr">[SECURITY] Set Auth-scheme preference <a href="https://github.com/elastic/elasticsearch/pull/33156">#33156</a></li>
	<li dir="ltr">MockTcpTransport to connect asynchronously (#28203) <a href="https://github.com/elastic/elasticsearch/pull/33476">#33476</a></li>
	<li dir="ltr">[ingest] geo-ip performance improvements <a href="https://github.com/elastic/elasticsearch/pull/33029">#33029</a></li>
	<li dir="ltr">6.x - HLRC: Add ILM Status to HLRC (#33283) <a href="https://github.com/elastic/elasticsearch/pull/33448">#33448</a></li>
	<li dir="ltr">[ingest] 6.x backport - geo-ip performance improvements <a href="https://github.com/elastic/elasticsearch/pull/33446">#33446</a></li>
	<li dir="ltr">Acquire seacher on closing engine should throw AlreadyClosedException <a href="https://github.com/elastic/elasticsearch/pull/33331">#33331</a></li>
	<li dir="ltr">BREAKING: Fix generics in ScriptPlugin#getContexts() <a href="https://github.com/elastic/elasticsearch/pull/33426">#33426</a></li>
	<li dir="ltr">Add interval response parameter to AutoDateInterval histogram <a href="https://github.com/elastic/elasticsearch/pull/33254">#33254</a></li>
	<li dir="ltr">Add user-defined cluster metadata <a href="https://github.com/elastic/elasticsearch/pull/33325">#33325</a></li>
	<li dir="ltr">Add sni name to SSLEngine in netty transport <a href="https://github.com/elastic/elasticsearch/pull/33144">#33144</a></li>
	<li dir="ltr">Add an index setting to control TieredMergePolicy#deletesPctAllowed <a href="https://github.com/elastic/elasticsearch/pull/32907">#32907</a></li>
	<li dir="ltr">Don&#x2019;t count metadata fields towards index.mapping.total_fields.limit <a href="https://github.com/elastic/elasticsearch/pull/33386">#33386</a></li>
	<li dir="ltr">INGEST: Implement Drop Processor <a href="https://github.com/elastic/elasticsearch/pull/32278">#32278</a></li>
	<li dir="ltr">Fix deprecated setting specializations <a href="https://github.com/elastic/elasticsearch/pull/33412">#33412</a></li>
	<li dir="ltr">Add conditional token filter to elasticsearch <a href="https://github.com/elastic/elasticsearch/pull/31958">#31958</a></li>
	<li dir="ltr">REST high-level client: add update by query API <a href="https://github.com/elastic/elasticsearch/pull/32760">#32760</a></li>
	<li dir="ltr">REST high-level client: add delete by query API <a href="https://github.com/elastic/elasticsearch/pull/32782">#32782</a></li>
	<li dir="ltr">SQL: Align SYS TABLE for ODBC SQL_ALL_* args <a href="https://github.com/elastic/elasticsearch/pull/33364">#33364</a></li>
	<li dir="ltr">Introduce private settings <a href="https://github.com/elastic/elasticsearch/pull/33327">#33327</a></li>
	<li dir="ltr">SQL: Show/desc commands now support table ids <a href="https://github.com/elastic/elasticsearch/pull/33363">#33363</a></li>
	<li dir="ltr">Add early termination support to BucketCollector <a href="https://github.com/elastic/elasticsearch/pull/33279">#33279</a></li>
</ul><p dir="ltr">Changes in 7.0:
</p><ul>
	<li dir="ltr">Generalize search.remote settings to cluster.remote <a href="https://github.com/elastic/elasticsearch/pull/33413">#33413</a></li>
	<li dir="ltr">Upgrade to a Lucene 8 snapshot <a href="https://github.com/elastic/elasticsearch/pull/33310">#33310</a></li>
	<li dir="ltr">Fix inner hits retrieval when stored fields are disabled (none) <a href="https://github.com/elastic/elasticsearch/pull/33018">#33018</a></li>
	<li dir="ltr">BREAKING: LLREST: Drop deprecated methods <a href="https://github.com/elastic/elasticsearch/pull/33223">#33223</a></li>
</ul><p dir="ltr"><span style="font-size: 24px;">Lucene
</span></p><p dir="ltr"><strong>Dynamic scoring features
</strong></p><p dir="ltr">After static scoring features like pagerank or url length, we are adding new tools in order to boost by dynamic features such as <a href="https://issues.apache.org/jira/browse/LUCENE-8340">recency</a> or <a href="https://issues.apache.org/jira/browse/LUCENE-8482">geo distance</a>. In particular, using these new queries on their own rather than for boosting is an efficient way to find nearest neighbors to a point.  It will be interesting to discuss how best to <a href="https://github.com/elastic/elasticsearch/issues/33382">expose these new capabilities in Elasticsearch</a>.
</p><p dir="ltr"><strong>Other
</strong></p><ul>
	<li dir="ltr">The fact that Collector is passed a Scorer is problematic since only a small subset of the Scorer functionality may be used by Collectors. We fixed this issue by passing a new abstract class called Scorable to collectors with only methods that may be used, and making Scorer extend Scorable. This in-turn helped <a href="https://issues.apache.org/jira/browse/LUCENE-8483">require that Scorers are constructed with a non-null Weight object</a>.</li>
	<li dir="ltr">Should we use <a href="https://issues.apache.org/jira/browse/LUCENE-8490">different abstractions</a> for queries and scoring contribution from features?</li>
	<li dir="ltr">Can we <a href="https://issues.apache.org/jira/browse/LUCENE-8484">add a way to remove leaves that only contain deleted documents</a>?</li>
	<li dir="ltr">While the fact that intervals only return minimum intervals helps consistency, it also makes <a href="https://issues.apache.org/jira/browse/LUCENE-8477">nested disjunctions sometimes perform in a way that is not intuitive</a>.</li>
	<li dir="ltr">The fact that the regex completion query uses UTF32 but that the completion FST internally uses FST means that <a href="https://issues.apache.org/jira/browse/LUCENE-8480">any non-ASCII characters may never match</a>.</li>
	<li dir="ltr">We <a href="https://issues.apache.org/jira/browse/LUCENE-8422">implemented the matches API on interval queries</a>.</li>
</ul></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-09-07

<html><head></head><body><h2>Elasticsearch
</h2><h4 dir="ltr" id="docs-internal-guid-6a4bb3ce-7fff-69f7-8eb5-48e66833efd2">Exposing IntervalQuery in the query DSL
</h4><p dir="ltr">Lucene 7.4 adds a new IntervalQuery, intended as a longer-term replacement for the Spans family. The problem with span queries is that they are quite tricky to use:
</p><ul>
	<li dir="ltr">Span queries don&apos;t go through analysis so its up to the user to create queries that match the exact terms in the index</li>
	<li dir="ltr">The rules on how different span queries can combine and which queries types can be used with different span query are complex</li>
	<li dir="ltr">Span queries can have complex hierarchies with queries nested at multiple levels</li>
</ul><p dir="ltr">The interval query aims to provide similar capabilities in an easier to use API. The interval query takes a single query and then defines &quot;interval sources&quot; to express the criteria for matching text on a single field defined on the query. There are multiple types of sources including:
</p><ul>
	<li dir="ltr">match - takes some text to match, analyses it and produces a term or phrase</li>
	<li dir="ltr">combine - combines other sources to allow matching sources near each other (either in order or not)</li>
	<li dir="ltr">relate - similar to &quot;combine&quot; in that it takes multiple sources but matches sources which based on the relation to each other (containing, not containing, contained by, etc.)</li>
</ul><p dir="ltr">We have been <a href="https://github.com/elastic/elasticsearch/pull/32406">working on a PR</a> which exposes these queries in Elasticsearch. This is the first of a few PRs for this which will aim to extend the functionality further going forward.
</p><h4 dir="ltr">Ingest Node
</h4><p dir="ltr">We <a href="https://github.com/elastic/elasticsearch/pull/32884">merged support</a>
for <code>dissect</code> into ingest node! This processor breaks strings into
parts as a simpler alternative to <code>grok</code>. Testing has shown that it
is 8% faster than <code>grok</code> for the <code>http_logs</code> Rally track. This will be available in 6.5.0. Check out the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/dissect-processor.html">docs</a> and the <a href="https://github.com/elastic/dissect-specification">specification</a>.
</p><p dir="ltr">We <a href="https://github.com/elastic/elasticsearch/pull/32398/">merged</a> support for conditionals per processor. This enables users to specify conditionals that must be fulfilled for the processor to execute for a given document without having to resort to a script processor to do so. <a href="https://github.com/elastic/elasticsearch/issues/33188">Preview of the documentation</a>.
</p><p dir="ltr">We also <a href="https://github.com/elastic/elasticsearch/pull/32473">added</a> the ability to call another pipeline from within a pipeline, enabling pipeline reuse. This is a feature that was requested by the ingest team.
</p><br><h4 dir="ltr">Watcher
</h4><p dir="ltr">We found a critical bug in distributed watch execution that can lead to some watches not being executed, as well as some watches being executed twice. The underlying problem is that when Watcher watches the cluster state for shard changes to decide which watches it owns, it was only taking into account local shard changes but it should be watching for remote shard changes too as the distribution of watches would change if a replica is added or removed. There is <a href="https://github.com/elastic/elasticsearch/pull/33167">a fix</a> that will be included in 6.4.1.
</p><h4 dir="ltr">Changes
</h4><p dir="ltr">Changes in 5.6:
</p><ul>
	<li dir="ltr">Apply settings filter to get cluster settings API<a href="https://github.com/elastic/elasticsearch/pull/33247"> #33247</a></li>
</ul><p dir="ltr">Changes in 6.3:
</p><ul>
	<li dir="ltr">Painless: Fix Semicolon Regression<a href="https://github.com/elastic/elasticsearch/pull/33212"> #33212</a></li>
</ul><p dir="ltr">Changes in 6.4:
</p><ul>
	<li dir="ltr">SQL: prevent duplicate generation for repeated aggs<a href="https://github.com/elastic/elasticsearch/pull/33252"> #33252</a></li>
	<li dir="ltr">Fix serialization of empty field capabilities response<a href="https://github.com/elastic/elasticsearch/pull/33263"> #33263</a></li>
	<li dir="ltr">Fix nested _source retrieval with includes/excludes<a href="https://github.com/elastic/elasticsearch/pull/33180"> #33180</a></li>
	<li dir="ltr">Watcher: Reload properly on remote shard change<a href="https://github.com/elastic/elasticsearch/pull/33167"> #33167</a></li>
	<li dir="ltr">Parse PEM Key files leniantly<a href="https://github.com/elastic/elasticsearch/pull/33173"> #33173</a></li>
	<li dir="ltr">Core: Add java time xcontent serializers<a href="https://github.com/elastic/elasticsearch/pull/33120"> #33120</a></li>
	<li dir="ltr">Ensure to generate identical NoOp for the same failure<a href="https://github.com/elastic/elasticsearch/pull/33141"> #33141</a></li>
</ul><p dir="ltr">Changes in 6.5:
</p><ul>
	<li dir="ltr">Painless: Fix Bindings Bug<a href="https://github.com/elastic/elasticsearch/pull/33274"> #33274</a></li>
	<li dir="ltr">Watcher: Fix race condition when reloading watches<a href="https://github.com/elastic/elasticsearch/pull/33157"> #33157</a></li>
	<li dir="ltr">Ignore module-info in jar hell checks<a href="https://github.com/elastic/elasticsearch/pull/33011"> #33011</a></li>
	<li dir="ltr">HLRC: add client side RefreshPolicy<a href="https://github.com/elastic/elasticsearch/pull/33209"> #33209</a></li>
	<li dir="ltr">Ingest: Add conditional per processor<a href="https://github.com/elastic/elasticsearch/pull/32398"> #32398</a></li>
	<li dir="ltr">[Rollup] Only allow aggregating on multiples of configured interval<a href="https://github.com/elastic/elasticsearch/pull/32052"> #32052</a></li>
	<li dir="ltr">ingest: Introduce the dissect processor<a href="https://github.com/elastic/elasticsearch/pull/32884"> #32884</a></li>
	<li dir="ltr">Painless: Add Bindings<a href="https://github.com/elastic/elasticsearch/pull/33042"> #33042</a></li>
	<li dir="ltr">HLRC: Use Optional in validation logic<a href="https://github.com/elastic/elasticsearch/pull/33104"> #33104</a></li>
	<li dir="ltr">HLRC: create base timed request class<a href="https://github.com/elastic/elasticsearch/pull/33216"> #33216</a></li>
	<li dir="ltr">Remove unsupported group_shard_failures parameter<a href="https://github.com/elastic/elasticsearch/pull/33208"> #33208</a></li>
	<li dir="ltr">INGEST: Add Pipeline Processor<a href="https://github.com/elastic/elasticsearch/pull/32473"> #32473</a></li>
	<li dir="ltr">REST high-level client: add reindex API<a href="https://github.com/elastic/elasticsearch/pull/32679"> #32679</a></li>
	<li dir="ltr">[Rollup] Better error message when trying to set non-rollup index<a href="https://github.com/elastic/elasticsearch/pull/32965"> #32965</a></li>
	<li dir="ltr">Introduce mapping version to index metadata<a href="https://github.com/elastic/elasticsearch/pull/33147"> #33147</a></li>
	<li dir="ltr">Token API supports the client_credentials grant<a href="https://github.com/elastic/elasticsearch/pull/33106"> #33106</a></li>
	<li dir="ltr">SQL: Enable aggregations to create a separate bucket for missing values<a href="https://github.com/elastic/elasticsearch/pull/32832"> #32832</a></li>
	<li dir="ltr">APM server monitoring<a href="https://github.com/elastic/elasticsearch/pull/32515"> #32515</a></li>
	<li dir="ltr">Add proxy support to RemoteClusterConnection<a href="https://github.com/elastic/elasticsearch/pull/33062"> #33062</a></li>
	<li dir="ltr">Security index expands to a single replica<a href="https://github.com/elastic/elasticsearch/pull/33131"> #33131</a></li>
</ul><p dir="ltr">Changes in 7.0:
</p><ul>
	<li dir="ltr">TESTS: Fix overly long lines<a href="https://github.com/elastic/elasticsearch/pull/33240"> #33240</a></li>
	<li dir="ltr">BREAKING: Remove support for deprecated params._agg/_aggs for scripted metric aggregations<a href="https://github.com/elastic/elasticsearch/pull/32979"> #32979</a></li>
	<li dir="ltr">BREAKING: Scroll queries asking for rescore are considered invalid<a href="https://github.com/elastic/elasticsearch/pull/32918"> #32918</a></li><li dir="ltr">Have circuit breaker succeed on unknown mem usage<a href="https://github.com/elastic/elasticsearch/pull/33125"> #33125</a></li></ul><h2>Apache Lucene<br></h2><h4>Lucene 8</h4><p> We are <a href="https://markmail.org/message/eebgri4k3657tr25" target="_blank">checking out what remains to be done to release Lucene 8.0</a>. 
	</p><h4>	Delete-by-query bug with index sorting</h4><p>There
 is a  bug with deletes-by-query on sorted indices, but
 doesn&apos;t affect Elasticsearch, which always deletes by term. The 
delete-by-query endpoint runs deletes by term under the hood. If you have index sorting enabled and documents that are not flushed yet, then <a href="https://issues.apache.org/jira/browse/LUCENE-8466" target="_blank">delete-by-query might delete the wrong documents</a>. This is because Lucene makes sure that delete-by-query only deletes documents that are already in the indexing buffers (as opposed to other documents that will get indexed until the next flush) by recording the number of documents in the buffer at delete time and only deleting documents whose doc id is less than this number at flush time. The fact that sorted indices reorder documents on flush breaks this functionality. We fixed this bug by resolving the doc ID prior to sorting in order to know whether a document is eligible for deletion.</p><h4>	Other</h4><ul><li>	 We made sure to <a href="https://issues.apache.org/jira/browse/LUCENE-8472" target="_blank">rewrite the query</a> that is used to identify soft deletes before running it.</li><li><a href="https://issues.apache.org/jira/browse/LUCENE-8474" target="_blank">replacing RAMDirectory</a> with a new <a href="https://issues.apache.org/jira/browse/LUCENE-8468" target="_blank">BytesBuffersDirectory</a>.</li><li>TopFieldCollector should <a href="https://issues.apache.org/jira/browse/LUCENE-8432" target="_blank">stop calling the comparator</a> when only counting hits.</li><li>We are having discussions regarding how to <a href="https://issues.apache.org/jira/browse/LUCENE-8196" target="_blank">keep existing functionality of span queries while also fixing their problems</a>.</li><li>We exposed the <a href="https://issues.apache.org/jira/browse/LUCENE-8471" target="_blank">current number of bytes that are being flushed in IndexWriter</a>, which will help know <a href="https://github.com/elastic/elasticsearch/issues/29368" target="_blank">how many more bytes need to be reclaimed via flushNextBuffer</a>.</li></ul><p><a href="https://github.com/elastic/elasticsearch/pull/33125"></a></p></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-08-31

<html><head></head><body><h2 dir="ltr">Elasticsearch
</h2><h4 dir="ltr">Tokens, Kerberos, Kibana
</h4><p dir="ltr">In order for Kibana to add Kerberos support, we are adding support
for the client credentials grant type <a href="https://github.com/elastic/elasticsearch/pull/33106">to the token
    API</a>, removing the need to pass username and password when using the get
      token API. Kibana (or another client) can proxy the <code>Authorization</code> header from the client to elasticsearch in a request to get a token and a token will be created that allows access as the authenticated user that made the request.
</p><h4 dir="ltr">Generalising Rollups to allow similar new features
</h4><p dir="ltr">We have been <a href="https://github.com/elastic/elasticsearch/pull/32743">working on a PR</a> which refactors the indexing side of rollups to allow code reuse when building similar features. In particular, the ML team are looking to implement &quot;data frames&quot; which are very close to the idea of entity centric indexing and allow a user to &quot;rollup&quot; data based on something other than a time series. Data frames will introduce some new challenges on the indexing side, since incremental updates will involve updating the existing document representing the entity; whereas the current rollup indexer does not merge new data with existing data.
</p><h4 dir="ltr">Cross Cluster Replication
</h4><p dir="ltr">Auto-follow patterns allow a cluster to automatically follow new indices that are created by a leader cluster, which is particularly useful for time-based indices. We have worked on a <a href="https://github.com/elastic/elasticsearch/issues/33007">plan</a> to implement auto-follow patterns, and we opened a <a href="https://github.com/elastic/elasticsearch/pull/33118/">first PR</a> for this work.
</p><h4 dir="ltr">Zen2
</h4><p dir="ltr">A fundamental building block of the <a href="https://github.com/elastic/elasticsearch-formal-models/blob/master/ZenWithTerms/tla/ZenWithTerms.tla">Zen2 algorithm</a> is the master term, which is a number that identifies the period of leadership of a particular master-eligible node, and is similar to <a href="https://www.elastic.co/blog/elasticsearch-sequence-ids-6-0">primary terms</a>, which track the reign of a primary shard. No two nodes may ever become master in the same term, and each node has a current term which can only ever increase. The current term is stored in the cluster state and included in transport messages, which allows each node to identify delayed messages coming from a stale master. Every node in the cluster should be in the same master term, so when a node discovers a newer term it must increase its own term to match. This is unavoidably a little disruptive: when a node leaves a term it stops following that term&apos;s elected master, but the newly-chosen term may not yet have an elected master at all, so for a period the cluster will be masterless and unable to process any cluster-state updates. The main reason for entering a new term is that the master of the current term seems to have failed, but nodes cannot individually tell the difference between an unhealthy master and an unhealthy network connecting them to a healthy master. The solution to this is a so-called pre-voting phase in which each node first asks its peers whether they would vote for it if it were to trigger an election. If pre-voting fails to obtain enough votes then most other nodes are either unreachable or following an existing, healthy master, and there is no point in starting an election. If the pre-voting phase succeeds then this means that there are enough reachable nodes that believe the master to have failed, which is a reliable indicator that it is wise to begin a new term and start a proper election. We have <a href="https://github.com/elastic/elasticsearch/pull/32847">implemented this pre-voting phase</a> and prepared the <a href="https://github.com/elastic/elasticsearch/pull/33013">leader-side logic</a> for the follow-up election.
</p><h3 dir="ltr">Changes
</h3><h4 dir="ltr">Changes in 6.3:
</h4><ul>
	<li dir="ltr">Trim unreferenced translog when the safe commit advanced<a href="https://github.com/elastic/elasticsearch/pull/32967"> #32967</a></li>
</ul><h4 dir="ltr">Changes in 6.4:
</h4><ul>
	<li dir="ltr">Watcher: Properly find next valid date in cron expressions<a href="https://github.com/elastic/elasticsearch/pull/32734"> #32734</a></li>
	<li dir="ltr">For filters aggregations, make sure that rewrites preserve other_bucket.<a href="https://github.com/elastic/elasticsearch/pull/32921"> #32921</a></li>
	<li dir="ltr">Ensure that _exists queries on keyword fields use norms when they&#x2019;re available.<a href="https://github.com/elastic/elasticsearch/pull/33006"> #33006</a></li>
	<li dir="ltr">Logging: Use settings when building daemon threads<a href="https://github.com/elastic/elasticsearch/pull/32751"> #32751</a></li>
	<li dir="ltr">Fix quoted exists query<a href="https://github.com/elastic/elasticsearch/pull/33019"> #33019</a></li>
	<li dir="ltr">Fix multi fields empty query<a href="https://github.com/elastic/elasticsearch/pull/33017"> #33017</a></li>
	<li dir="ltr">Fix north pole overflow error in GeoHashUtils.bbox()<a href="https://github.com/elastic/elasticsearch/pull/32891"> #32891</a></li>
	<li dir="ltr">All Translog inner closes should happen after tragedy exception is set<a href="https://github.com/elastic/elasticsearch/pull/32674"> #32674</a></li>
	<li dir="ltr">Protect scheduler engine against throwing listeners<a href="https://github.com/elastic/elasticsearch/pull/32998"> #32998</a></li>
	<li dir="ltr">Enable FIPS140LicenseBootstrapCheck<a href="https://github.com/elastic/elasticsearch/pull/32903"> #32903</a></li>
</ul><h4 dir="ltr">Changes in 6.5:
</h4><ul>
	<li dir="ltr">Do NOT allow termvectors on nested fields<a href="https://github.com/elastic/elasticsearch/pull/32728"> #32728</a></li>
	<li dir="ltr">[Rollup] Return empty response when aggs are missing<a href="https://github.com/elastic/elasticsearch/pull/32796"> #32796</a></li>
	<li dir="ltr">Use a dedicated ConnectionManger for RemoteClusterConnection<a href="https://github.com/elastic/elasticsearch/pull/32988"> #32988</a></li>
	<li dir="ltr">TESTS: Fix Random Fail in MockTcpTransportTests<a href="https://github.com/elastic/elasticsearch/pull/33061"> #33061</a></li>
	<li dir="ltr">[Watcher] Improved error messages for CronEvalTool<a href="https://github.com/elastic/elasticsearch/pull/32800"> #32800</a></li>
	<li dir="ltr">Search: Support of wildcard on docvalue_fields<a href="https://github.com/elastic/elasticsearch/pull/32980"> #32980</a></li>
	<li dir="ltr">Change query field expansion<a href="https://github.com/elastic/elasticsearch/pull/33020"> #33020</a></li>
	<li dir="ltr">SQL: skip uppercasing/lowercasing function tests for AZ locales as well<a href="https://github.com/elastic/elasticsearch/pull/32910"> #32910</a></li>
	<li dir="ltr">HLRC: Create server agnostic request and response<a href="https://github.com/elastic/elasticsearch/pull/32912"> #32912</a></li>
	<li dir="ltr">GraphClient for the high level REST client and associated tests.<a href="https://github.com/elastic/elasticsearch/pull/33025"> #33025</a></li>
	<li dir="ltr">Add settings updater for 2 affix settings<a href="https://github.com/elastic/elasticsearch/pull/33050"> #33050</a></li>
	<li dir="ltr">Allow engine to recover from translog upto a seqno<a href="https://github.com/elastic/elasticsearch/pull/33032"> #33032</a></li>
	<li dir="ltr">Expose max_concurrent_shard_requests in _msearch<a href="https://github.com/elastic/elasticsearch/pull/33016"> #33016</a></li>
	<li dir="ltr">Enable avoiding mmap bootstrap check<a href="https://github.com/elastic/elasticsearch/pull/32421"> #32421</a></li>
	<li dir="ltr">Generalize remote license checker<a href="https://github.com/elastic/elasticsearch/pull/32971"> #32971</a></li>
	<li dir="ltr">Make Geo Context Mapping Parsing More Strict (6.x version)<a href="https://github.com/elastic/elasticsearch/pull/32862"> #32862</a></li>
	<li dir="ltr">Add mzn and dz to unsupported locales<a href="https://github.com/elastic/elasticsearch/pull/32957"> #32957</a></li>
	<li dir="ltr">Scripted metric aggregations: add deprecation warning and system property to control legacy params<a href="https://github.com/elastic/elasticsearch/pull/31597"> #31597</a></li>
	<li dir="ltr">Remove client connections from TcpTransport (#31886)<a href="https://github.com/elastic/elasticsearch/pull/32954"> #32954</a></li>
	<li dir="ltr">NETWORKING: Make RemoteClusterConn. Lazy Resolve DNS<a href="https://github.com/elastic/elasticsearch/pull/32764"> #32764</a></li>
</ul><h4 dir="ltr">Changes in 7.0:
</h4><ul>
	<li dir="ltr">Set maxScore for empty TopDocs to Nan rather than 0<a href="https://github.com/elastic/elasticsearch/pull/32938"> #32938</a></li>
	<li dir="ltr">GraphClient for the high level REST client and associated tests<a href="https://github.com/elastic/elasticsearch/pull/32366"> #32366</a></li>
	<li dir="ltr">Handle 6.4.0+ BWC for Application Privileges<a href="https://github.com/elastic/elasticsearch/pull/32929"> #32929</a></li>
	<li dir="ltr">BREAKING: Make Geo Context Mapping Parsing More Strict<a href="https://github.com/elastic/elasticsearch/pull/32821"> #32821</a></li>
	<li dir="ltr">Fix InternalAutoDateHistogram reproducible failure<a href="https://github.com/elastic/elasticsearch/pull/32723"> #32723</a></li>
</ul><h2 dir="ltr">Lucene
</h2><h4 dir="ltr">Lucene 8
</h4><p dir="ltr">We added a new<a href="https://issues.apache.org/jira/browse/LUCENE-8461"> &quot;Lucene80&quot; codec</a> to reflect the fact that codec changes have been made in order to record impacts into skip lists.
</p><p dir="ltr">On the Elasticsearch side, Jim<a href="https://github.com/elastic/elasticsearch/pull/32937"> removed the Elasticsearch 5.x version constants</a>, which is required to upgrade to Lucene 8.0 since the Elasticsearch 5.x version constants referenced now unsupported Lucene 6.x version constants. Progress has also been made on the<a href="https://github.com/elastic/elasticsearch/tree/upgrade/lucene-8"> upgrade/lucene-8</a> branch, which is now almost free of compile errors. We will look into test failures next week.
</p><h4 dir="ltr">Other
</h4><ul><li> There are<a href="https://issues.apache.org/jira/browse/LUCENE-8462"> ongoing discussions to integrate a new Arabic snowball stemmer</a>.</li><li><a href="https://issues.apache.org/jira/browse/LUCENE-765">Javadocs of the org.apache.lucene.index package need love</a>.
</li><li>We fixed a bug where Lucene would<a href="https://issues.apache.org/jira/browse/LUCENE-8458"> miss to update soft deletes</a> when carrying over hard deletes that occurred while a merge was running.
</li><li>The StoredField class has<a href="https://issues.apache.org/jira/browse/LUCENE-8460"> missing null checks</a>.</li><li>We are discussing how best to<a href="https://issues.apache.org/jira/browse/LUCENE-8422"> expose matches on IntervalQuery</a>.
</li><li>The change to propagate min competitive scores to sub clauses triggered an impressive speedup on<a href="http://people.apache.org/~mikemccand/lucenebench/AndHighOrMedMed.html"> AndHighOrMedMed</a> and recovered part of the slowdown on<a href="http://people.apache.org/~mikemccand/lucenebench/AndMedOrHighHigh.html"> AndMedOrHighHigh</a>.
</li><li>interesting performance numbers regarding<a href="https://issues.apache.org/jira/browse/LUCENE-8374"> optimizing reads on sparse doc values</a>.
</li></ul></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-08-24

<html><head></head><body><p dir="ltr"><span style="font-size: 24px;"><strong>Elasticsearch
</strong></span></p><p dir="ltr"><strong>Cross-cluster replication
</strong></p><p dir="ltr">We have continued working on adding <a href="https://github.com/elastic/elasticsearch/issues/32651">long polling for global checkpoints</a>, so that a follower cluster does not need to periodically poll the leader to ask if there are new changes that the follower does not have. To this end, we have <a href="https://github.com/elastic/elasticsearch/pull/32696">extended the global checkpoint component</a> to allow registering listeners that will be be notified when the global checkpoint is updated. This will be used in CCR for a follower to make a remote request to the leader that attaches a listener to the global checkpoint, and is then called back when the global checkpoint advances.
</p><p dir="ltr"><strong>New dense vector field and cosine similarity in the works
</strong></p><p dir="ltr">We have started working on a <a href="https://github.com/mayya-sharipova/elasticsearch/commits/vector-field">new field type</a> that will allow users to store dense vectors in Elasticsearch. Storing dense vectors can be useful for users wanting to augment relevance with data for models generated outside of Elasticsearch. This would be paired with a new query which would apply&#xA0;<a href="https://en.wikipedia.org/wiki/Cosine_similarity">cosine similarity</a> scoring on top of this field. It&apos;s very early days in this project and we are mostly exploring what&#x2019;s possible and what role Elasticsearch should play in this so there are not yet any PRs or concrete plans for exactly what and when.
</p><p dir="ltr"><strong>Structured Audit Logging</strong></p><p dir="ltr">Our work <a href="https://github.com/elastic/elasticsearch/pull/31931">structured audit logging</a> is now ready for review after a tough fight with tests.
</p><p dir="ltr"><strong>Zen2
</strong></p><p dir="ltr">Master elections in Zen2 will be significantly faster than in current Elasticsearch releases. Today when a cluster is without a master, each node undertakes a master-eligible node discovery and election process.  This is done in a series of rounds, each taking three seconds.
</p><p dir="ltr">In Zen2 the &quot;pinging&quot; or <a href="https://github.com/elastic/elasticsearch/pull/32246">node discovery phase</a> is decoupled from the election process. Rather than exposing the set of discovered nodes at the end of a three-second round, we make the current set of discovered nodes available continuously, allowing us to start an election as soon as we discover enough master-eligible nodes. We are also more enthusiastically reciprocal in our discovery of other nodes: if an unknown node pings us then we ping it straight back, possibly learn about new nodes, start pinging those, and so on, which means that nodes typically converge on a complete picture of the cluster very quickly. The <a href="https://github.com/elastic/elasticsearch/pull/32846">election process in Zen2</a> is triggered by a single node reaching out and requesting votes from the other master-eligible nodes, rather than waiting for each node to independently decide to join the best master it can find. Doing this correctly is a little tricky because if multiple nodes decide to request votes at a similar point in time, then it&apos;s unlikely for any of them to gather enough votes to become elected. Similar as in other <a href="https://raft.github.io/">well-known algorithms</a>, we probabilistically protect against split votes by introducing a random wait on each node before starting an election. We optimistically start with a short wait and adapt by backing off in case of failed elections. We expect this to result in master elections that will take a couple of 100 milliseconds at most, significantly faster than the 3 seconds that we have today. Nodes joining a cluster with an already well-established elected master will also take advantage of the new approach. Today a node performs a full three-second pinging round before joining an existing master. In Zen 2, the node will join the master immediately upon discovery.
</p><p dir="ltr"><strong>Watcher scalability</strong></p><p dir="ltr">We have some users that push hard on the scalability of Watcher. One case that arose recently depends on the fact that when a watch is executed, Watcher will perform single-document operations to manage the triggered watches, watch status, and watch history. If you are executing a lot of watches, these single-document operations can pile up. We opened a pull request to <a href="https://github.com/elastic/elasticsearch/pull/32490">switch to using bulk requests</a> to execute these operations. The defaults will still execute single-document operations. For users not pushing on the limits of Watcher, these defaults make sense so that a node outage does not lead to lost watch history, or watches executing again because a triggered watch was not pruned. Yet, the knob is now there for users that need it.
</p><p dir="ltr"><strong>Fixing role queries that match nested documents
</strong></p><p dir="ltr">We <a href="https://github.com/elastic/elasticsearch/pull/32705">fixed a bug</a> in the way role queries are applied when nested fields are used. We use the role query to restrict documents for a specific user. In 6.3 we added the support for nested documents but it introduced a bug when the role query is defined in a way that can match nested documents as well as root documents (only excluding terms for instance). To fix this bug we now restrict the role query to root documents only and adds the nested documents in a different clause.
</p><p dir="ltr"><strong>Changes
</strong></p><p dir="ltr"><strong>Changes in 6.3:
</strong></p><ul>
	<li dir="ltr">Update WordPress plugins links <a href="https://github.com/elastic/elasticsearch/pull/32194">#32194</a></li>
</ul><p dir="ltr"><strong>Changes in 6.4:
</strong></p><ul>
	<li dir="ltr">Guard against null in email admin watches <a href="https://github.com/elastic/elasticsearch/pull/32923">#32923</a></li>
	<li dir="ltr">Remove client connections from TcpTransport <a href="https://github.com/elastic/elasticsearch/pull/31886">#31886</a></li>
	<li dir="ltr">Mute security-cli tests in FIPS JVM <a href="https://github.com/elastic/elasticsearch/pull/32812">#32812</a></li>
	<li dir="ltr">Unmute WildFly tests in FIPS JVM <a href="https://github.com/elastic/elasticsearch/pull/32814">#32814</a></li>
	<li dir="ltr">[Kerberos] Add debug log statement for exceptions <a href="https://github.com/elastic/elasticsearch/pull/32663">#32663</a></li>
</ul><p dir="ltr"><strong>Changes in 6.5:
</strong></p><ul>
	<li dir="ltr">Watcher: migrate PagerDuty v1 events API to v2 API <a href="https://github.com/elastic/elasticsearch/pull/32285">#32285</a></li>
	<li dir="ltr">Introduce global checkpoint listeners <a href="https://github.com/elastic/elasticsearch/pull/32696">#32696</a></li>
	<li dir="ltr">BREAKING: Use generic AcknowledgedResponse instead of extended classes <a href="https://github.com/elastic/elasticsearch/pull/32859">#32859</a></li>
	<li dir="ltr">Aggregations/HL Rest client fix: missing scores <a href="https://github.com/elastic/elasticsearch/pull/32774">#32774</a></li>
	<li dir="ltr">High Level REST Client: migration get assistance API <a href="https://github.com/elastic/elasticsearch/pull/32744">#32744</a></li>
	<li dir="ltr">High Level REST Client: Add Delete License API <a href="https://github.com/elastic/elasticsearch/pull/32586">#32586</a></li>
	<li dir="ltr">INGEST: Create Index Before Pipeline Execute <a href="https://github.com/elastic/elasticsearch/pull/32786">#32786</a></li>
	<li dir="ltr">Fix NOOP bulk updates <a href="https://github.com/elastic/elasticsearch/pull/32819">#32819</a></li>
</ul><p dir="ltr"><strong>Changes in 7.0:
</strong></p><ul>
	<li dir="ltr">BREAKING: INGEST: Add Configuration Except. Data to Metdata <a href="https://github.com/elastic/elasticsearch/pull/32322">#32322</a></li>
	<li dir="ltr">BREAKING: Introduce the dissect library <a href="https://github.com/elastic/elasticsearch/pull/32297">#32297</a></li>
	<li dir="ltr">Core: Add java time version of rounding classes <a href="https://github.com/elastic/elasticsearch/pull/32641">#32641</a></li></ul><p dir="ltr"><span style="font-size: 24px;"><strong><br></strong></span></p><p dir="ltr"><span style="font-size: 24px;"><strong>Lucene<br></strong></span></p><ul>
	<li dir="ltr">We <a href="https://github.com/elastic/elasticsearch/pull/32730">upgraded the master and 6.x branches of Elasticsearch to a Lucene 7.5 snapshot</a>, which includes the new BKD-based shape support.</li>
	<li dir="ltr">We are undertaking <a href="https://github.com/elastic/elasticsearch/tree/upgrade/lucene-8">an effort to upgrade Elasticsearch to a Lucene 8.0 snapshot</a>, but we first need to <a href="https://github.com/elastic/elasticsearch/pull/32937">get rid of the Elasticsearch 5.x version constants</a>, which refer to unsupported Lucene versions.</li>
	<li dir="ltr">An <a href="https://issues.apache.org/jira/browse/LUCENE-8455">upgraded dependency on ECJ</a> (Java compiler of the Eclipse project) -- to a newer version that does a better job at inferring generic types.</li>
	<li dir="ltr">We pushed a change that <a href="https://issues.apache.org/jira/browse/LUCENE-8448">propagates minimum competitive scores</a> to sub clauses, which is expected to address the <a href="http://people.apache.org/~mikemccand/lucenebench/AndHighOrMedMed.html">slowdowns</a> that we noticed on nested boolean queries. It turns out that this change will help support <a href="https://issues.apache.org/jira/browse/LUCENE-8340">boosting by recency</a> and geo-distance as well.</li>
	<li dir="ltr">Some ideas to <a href="https://issues.apache.org/jira/browse/LUCENE-8374">optimize reads on sparse doc-values fields</a>.</li>
	<li dir="ltr">We are <a href="https://issues.apache.org/jira/browse/LUCENE-8452">discussing performance characteristics of the new BKD-based shape support</a>.</li>
	<li dir="ltr">We found out that the tesselator that we implemented for polygons is <a href="https://issues.apache.org/jira/browse/LUCENE-8454">prone to infinite loops</a>.</li>
	<li dir="ltr">It would be nice to<a href="https://issues.apache.org/jira/browse/LUCENE-8450"> allow token filter to modify offsets</a>, but there is an argument that this introduces too much complexity</li>
</ul></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-08-17

<html><head></head><body><h3 dir="ltr" id="docs-internal-guid-fc2d6170-7fff-b981-83ca-d2080ef0d73c">Elasticsearch</h3><h4 dir="ltr">Highlights</h4><p dir="ltr"><strong>Stability</strong>: We fixed several leaks that have been caused by third-party code. We&#x2019;ve <a href="https://github.com/elastic/elasticsearch/pull/32511">upgraded to Netty 4.1.28</a> which fixes a leak in their SSL implementation, and <a href="https://github.com/elastic/elasticsearch/pull/32616">upgraded Log4J</a> to fix a memory leak. We will ship the Log4J fix with 5.6.11 and 6.4.0.
</p><p dir="ltr"><strong>Watcher</strong>: <a href="https://github.com/elastic/elasticsearch/pull/32285">migrated Watcher to the PagerDuty v2 API</a>; we want to upgrade early, although their v1 events API can still be used. Also, <a href="https://github.com/elastic/elasticsearch/pull/32490">integrated BulkProcessor with Watcher</a> to batch Watcher&apos;s document operations when users run lots of watches.
</p><p dir="ltr"><strong>Improvements to Exception information for search failures</strong>: We have fixed a few different bugs that were uncovered whilst working on Cross Cluster Search. The <a href="https://github.com/elastic/elasticsearch/pull/32608">first</a> makes sure that we preserve the cluster alias when we get a failure in a cross cluster search. This means that if failures happen on different remote clusters with the same index name, the user can see the cluster that the failure happened on. The <a href="https://github.com/elastic/elasticsearch/pull/32677">second</a> fixes the QueryShardException so we report the index_uuid for the shard that failed as well as the other shard information.
</p><p dir="ltr"><strong>Cross Cluster Replication</strong>: We&#x2019;ve<a href="https://github.com/elastic/stack-terraform/commit/022c4d1323e4d2585de71f1e6bac82f09a1994cb"> added support</a> to enable metricbeat (system metrics) collection on all launched nodes. We have updated Rally&apos;s <a href="https://github.com/elastic/rally/pull/546">ccr-stats telemetry device</a>  to comply with the changes in the CCR stats API and set it as default for our benchmarks.
</p><p dir="ltr"><strong>Mapping, Bulk and retries</strong>: We&#x2019;ve had users reporting replica shards with huge translog files, caused by missing sequence numbers that were failed to replicate to the replicas, causing the local checkpoint on these replicas to fall behind. Since the translog is supposed to retain all operations above the local checkpoint, it became very big. The source of those missing sequence numbers turned out to be a mistake in how we handled timeouts while requesting the master to update the index mapping. In short, the entire bulk request was re-executed, causing the sequence numbers issued up to the point where the dynamic mapping were required to be lost. This was<a href="https://github.com/elastic/elasticsearch/pull/30244"> fixed</a> in the 6.3.0 release of Elasticsearch with a targeted solution, but we wanted to have a more structural way to avoid retries all together. That refactoring was<a href="https://github.com/elastic/elasticsearch/pull/31821"> completed</a> this week. The gist of the refactoring is that we now can pause and continue execution from where we were.
</p><p dir="ltr">This may sound like a very technical internal thing (and it is!) but it is exciting as it opens the door to tackle another issue with mapping updates. When an indexing operation requires a mapping update, it currently has to wait for *all* nodes in the cluster to process that change. This is an overkill as only the nodes with the primary and its replicas actually need that mapping to complete the operation. It&apos;s a shame if an overloaded cold node slows a hot node from processing an operation. With the above refactoring, we now have the underpinnings to untangle this dependency.
</p><h3 dir="ltr">Changes</h3><h4 dir="ltr">Changes in 5.6:
</h4><ul>
	<li dir="ltr">LOGGING: Upgrade to Log4J 2.11.1<a href="https://github.com/elastic/elasticsearch/pull/32675"> #32675</a></li>
	<li dir="ltr">Fix content type detection with leading whitespace<a href="https://github.com/elastic/elasticsearch/pull/32632"> #32632</a></li>
	<li dir="ltr">LOGGING: Upgrade to Log4J 2.11.1<a href="https://github.com/elastic/elasticsearch/pull/32616"> #32616</a></li>
</ul><h4 dir="ltr">Changes in 6.4:
</h4><ul>
	<li dir="ltr">LOGGING: Upgrade to Log4J 2.11.1 (#32616)<a href="https://github.com/elastic/elasticsearch/pull/32668"> #32668</a></li>
	<li dir="ltr">Preserve index_uuid when creating QueryShardException<a href="https://github.com/elastic/elasticsearch/pull/32677"> #32677</a></li>
	<li dir="ltr">Make sure that field collapsing supports field aliases.<a href="https://github.com/elastic/elasticsearch/pull/32648"> #32648</a></li>
	<li dir="ltr">Add temporary directory cleanup workarounds<a href="https://github.com/elastic/elasticsearch/pull/32615"> #32615</a></li>
	<li dir="ltr">Networking+Testing: Fix Netty ByteBuf Leaks in Test Code<a href="https://github.com/elastic/elasticsearch/pull/32638"> #32638</a></li>
	<li dir="ltr">Cross-cluster search: preserve cluster alias in shard failures<a href="https://github.com/elastic/elasticsearch/pull/32608"> #32608</a></li>
	<li dir="ltr">[Rollup] Improve ID scheme for rollup documents<a href="https://github.com/elastic/elasticsearch/pull/32558"> #32558</a></li>
	<li dir="ltr">HLRC: Move commercial clients from XPackClient<a href="https://github.com/elastic/elasticsearch/pull/32596"> #32596</a></li>
	<li dir="ltr">Fix race between replica reset and primary promotion<a href="https://github.com/elastic/elasticsearch/pull/32442"> #32442</a></li>
</ul><h4 dir="ltr">Changes in 6.5:
</h4><ul>
	<li dir="ltr">Fix role query that can match nested documents<a href="https://github.com/elastic/elasticsearch/pull/32705"> #32705</a></li>
	<li dir="ltr">Add expected mapping type to MapperException<a href="https://github.com/elastic/elasticsearch/pull/31564"> #31564</a></li>
	<li dir="ltr">SQL: Bug fix for the optional &quot;start&quot; parameter usage inside LOCATE function<a href="https://github.com/elastic/elasticsearch/pull/32576"> #32576</a></li>
	<li dir="ltr">SQL: Ignore H2 comparative tests for uppercasing/lowercasing string functions<a href="https://github.com/elastic/elasticsearch/pull/32604"> #32604</a></li>
	<li dir="ltr">Upgrade to Lucene-7.5.0-snapshot-13b9e28f9d<a href="https://github.com/elastic/elasticsearch/pull/32730"> #32730</a></li>
	<li dir="ltr">Whitelisting / from Circuit Breaker Exception (#32325)<a href="https://github.com/elastic/elasticsearch/pull/32666"> #32666</a></li>
	<li dir="ltr">LOGGING: Upgrade to Log4J 2.11.1 (#32616)<a href="https://github.com/elastic/elasticsearch/pull/32656"> #32656</a></li>
	<li dir="ltr">Core: Fix Java Time DateFormatter printers<a href="https://github.com/elastic/elasticsearch/pull/32592"> #32592</a></li>
	<li dir="ltr">BREAKING: Switch WritePipelineResponse to AcknowledgedResponse<a href="https://github.com/elastic/elasticsearch/pull/32722"> #32722</a></li>
	<li dir="ltr">TESTS: Explicitly Fail Http Client Timeouts<a href="https://github.com/elastic/elasticsearch/pull/32708"> #32708</a></li>
	<li dir="ltr">Prevent cause from being null in ShardOperationFailedException<a href="https://github.com/elastic/elasticsearch/pull/32640"> #32640</a></li>
	<li dir="ltr">CORE: Upgrade to Jackson 2.8.11<a href="https://github.com/elastic/elasticsearch/pull/32670"> #32670</a></li>
	<li dir="ltr">Expose whether or not the global checkpoint updated<a href="https://github.com/elastic/elasticsearch/pull/32659"> #32659</a></li>
	<li dir="ltr">Include translog path in error message when translog is corrupted<a href="https://github.com/elastic/elasticsearch/pull/32251"> #32251</a></li>
	<li dir="ltr">Verify primary mode usage with assertions<a href="https://github.com/elastic/elasticsearch/pull/32667"> #32667</a></li>
	<li dir="ltr">Ignore script fields when size is 0<a href="https://github.com/elastic/elasticsearch/pull/31917"> #31917</a></li>
	<li dir="ltr">Tests: Fix Typo Causing Flaky Settings Test<a href="https://github.com/elastic/elasticsearch/pull/32665"> #32665</a></li>
	<li dir="ltr">Docs: Allow snippets to have line continuation<a href="https://github.com/elastic/elasticsearch/pull/32649"> #32649</a></li>
	<li dir="ltr">INGEST: Fix ThreadWatchDog Throwing on Shutdown<a href="https://github.com/elastic/elasticsearch/pull/32578"> #32578</a></li>
	<li dir="ltr">Rest HL client: Add get license action<a href="https://github.com/elastic/elasticsearch/pull/32438"> #32438</a></li>
	<li dir="ltr">Adds ckb to the list of unsupported languages<a href="https://github.com/elastic/elasticsearch/pull/32611"> #32611</a></li>
	<li dir="ltr">Suppress LicensingDocumentationIT.testPutLicense in release builds<a href="https://github.com/elastic/elasticsearch/pull/32613"> #32613</a></li>
	<li dir="ltr">Add cluster UUID to Cluster Stats API response<a href="https://github.com/elastic/elasticsearch/pull/32206"> #32206</a></li>
</ul><h4 dir="ltr">Changes in 7.0:
</h4><ul>
	<li dir="ltr">serialize suggestion responses as named writeables<a href="https://github.com/elastic/elasticsearch/pull/30284"> #30284</a></li>
</ul><h3 dir="ltr">Apache Lucene</h3><h4 dir="ltr">Faster queries when not counting hits
</h4><p dir="ltr">Nightly<a href="http://people.apache.org/~mikemccand/lucenebench/"> benchmarks</a> were fixed so you can now see how much it helps not to count total hits when computing top hits. This especially helps<a href="http://people.apache.org/~mikemccand/lucenebench/Term.html"> term</a> queries and<a href="http://people.apache.org/~mikemccand/lucenebench/OrHighHigh.html"> disjunctions</a>, even though<a href="http://people.apache.org/~mikemccand/lucenebench/AndHighHigh.html"> conjunctions</a> and<a href="http://people.apache.org/~mikemccand/lucenebench/Phrase.html"> phrase</a> queries got an interesting speedup as well. On the other hand, disjunctions within conjunctions<a href="http://people.apache.org/~mikemccand/lucenebench/AndHighOrMedMed.html"> slowed</a><a href="http://people.apache.org/~mikemccand/lucenebench/AndMedOrHighHigh.html"> down</a>, which we are<a href="https://issues.apache.org/jira/browse/LUCENE-8448"> investigating</a>.
</p><p dir="ltr">We improved collection of top hits with<a href="https://issues.apache.org/jira/browse/LUCENE-8439"> dis-max queries</a> and boolean queries that contain a mix of SHOULD and MUST clauses.
</p><h4 dir="ltr">Lucene 8.0
</h4><p dir="ltr">In the coming weeks, we will start an effort to upgrade the master branch of Elasticsearch to a Lucene 8.0 snapshot so that we can validate that changes in Lucene 8 play well with the way that Elasticsearch leverages Lucene.
</p><h4 dir="ltr">Other
</h4><ul><li> We fixed a<a href="https://issues.apache.org/jira/browse/LUCENE-8442"> test bug</a> with IndexWriter.
</li><li>BlendedInfixSuggester&apos;s way of computing scores is<a href="https://issues.apache.org/jira/browse/LUCENE-8343"> trappy</a> with small weights.
</li><li>The unified highlighter should<a href="https://issues.apache.org/jira/browse/LUCENE-8446"> merge overlapping matches</a>.
</li><li>We are adding support for<a href="https://issues.apache.org/jira/browse/LUCENE-8447"> DISJOINT and WITHIN relations</a> to the new BKD-based shape support.
</li><li>We propose to<a href="https://github.com/mikemccand/luceneutil/pull/20"> benchmark the new BKD-based shape support</a> by reusing the existing benchmark that we use for points.
</li></ul></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-08-10

<html><head></head><body><p><strong></strong></p><h3 dir="ltr" id="docs-internal-guid-22bc5d18-0234-e45b-5ed5-b1b05db8ab07">Elasticsearch</h3><p dir="ltr"><strong>Rollups fix for doc ID bug
</strong></p><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/32558">raised a PR</a> to fix a doc id bug in rollups. Rollup doc Ids will now be a 128-bit hash prepended by the job ID to avoid hash collisions. The change will not cause a backwards compatibility break as jobs will &quot;upgrade&quot; themselves to the new hash, but we will be recommending that users rebuild their rollup jobs from scratch to protect themselves from any hash collisions that occurred prior to upgrade.
</p><p dir="ltr"><strong>Zen2</strong>:
</p><p dir="ltr">We worked on<a href="https://github.com/elastic/elasticsearch/pull/32493"> increasing the testability and reproducibility</a> of MasterService, the class that receives Cluster State Update Tasks from the rest of the code base. We put up a<a href="https://github.com/elastic/elasticsearch/pull/32246"> PR</a> thats adds gossiping to discover other nodes and exchange information required for discovering an existing master or electing a new one. This is the equivalent of the current ZenDiscovery pinging.
</p><p dir="ltr"><strong>Cross Cluster Replication:
</strong></p><p dir="ltr">We reached a first milestone in the<a href="https://github.com/dliappis/stack-terraform/compare/ccr...dliappis:ccr-with-aws?expand=1"> AWS support</a> for CCR benchmarking. We are now able to benchmark between regions eu-central-1 and us-east-2 and terraform is doing all the legwork of setting things up like instances etc. apart from the creation of the VPCs + Peering, which will be next thing to automate.
</p><p dir="ltr">We got<a href="https://gist.github.com/dliappis/674891d6eb620ac37f179d71e3fcf8c4"> results</a> from the http_logs track using the AWS c5.2xlarge instance type vs the n1-highcpu-16 in GCP we&apos;ve been using so far. Both scenarios have identical #nodes, OS, memory, disk, ES build, ES settings and each node per region resides in a different AZ, as is the typical Cloud H/A.
</p><p dir="ltr">Results are quite similar as can be seen in the table below, with the exception that in AWS following indices seem to take a bit longer to fully catch up after bulk has ended; this corroborates my earlier observation that raw throughput per connection across the chosen regions is faster in GCP, but tinkering with some batch and buffer size settings might be worth it on AWS.
</p><h3 dir="ltr">Changes</h3><p dir="ltr"><strong>Changes in 6.3:
</strong></p><ul>
	<li dir="ltr">High-level client: fix clusterAlias parsing in SearchHit<a href="https://github.com/elastic/elasticsearch/pull/32465"> #32465</a></li>
	<li dir="ltr">IndicesClusterStateService should replace an init. replica with an init. primary with the same allocation id<a href="https://github.com/elastic/elasticsearch/pull/32374"> #32374</a></li>
</ul><p dir="ltr"><strong>Changes in 6.4:
</strong></p><ul>
	<li dir="ltr">Release requests in cors handle (#32410)<a href="https://github.com/elastic/elasticsearch/pull/32505"> #32505</a></li>
	<li dir="ltr">REST high-level client: parse back _ignored meta field<a href="https://github.com/elastic/elasticsearch/pull/32362"> #32362</a></li>
	<li dir="ltr">Add licensing enforcement for FIPS mode<a href="https://github.com/elastic/elasticsearch/pull/32437"> #32437</a></li>
	<li dir="ltr">Improve the error message when an index is incompatible with field aliases.<a href="https://github.com/elastic/elasticsearch/pull/32482"> #32482</a></li>
	<li dir="ltr">[Kerberos] Remove Kerberos bootstrap checks<a href="https://github.com/elastic/elasticsearch/pull/32451"> #32451</a></li>
	<li dir="ltr">Make get all app privs requires &quot;*&quot; permission<a href="https://github.com/elastic/elasticsearch/pull/32460"> #32460</a></li>
</ul><p dir="ltr"><strong>Changes in 6.5:
</strong></p><ul>
	<li dir="ltr">SQL: Minor fix for javadoc<a href="https://github.com/elastic/elasticsearch/pull/32573"> #32573</a></li>
	<li dir="ltr">Core: Minor size reduction for AbstractComponent<a href="https://github.com/elastic/elasticsearch/pull/32509"> #32509</a></li>
	<li dir="ltr">INGEST: Enable default pipelines<a href="https://github.com/elastic/elasticsearch/pull/32286"> #32286</a></li>
	<li dir="ltr">Scripting: Conditionally use java time api in scripting<a href="https://github.com/elastic/elasticsearch/pull/31441"> #31441</a></li>
	<li dir="ltr">SQL: Added support for string manipulating functions with more than one parameter<a href="https://github.com/elastic/elasticsearch/pull/32356"> #32356</a></li>
	<li dir="ltr">Logging: Make node name consistent in logger<a href="https://github.com/elastic/elasticsearch/pull/31588"> #31588</a></li>
	<li dir="ltr">Scripting: Fix painless compiler loader to know about context classes<a href="https://github.com/elastic/elasticsearch/pull/32385"> #32385</a></li>
	<li dir="ltr">Increase max chunk size to 256Mb for repo-azure<a href="https://github.com/elastic/elasticsearch/pull/32101"> #32101</a></li>
	<li dir="ltr">HLRC: Add delete watch action<a href="https://github.com/elastic/elasticsearch/pull/32337"> #32337</a></li>
	<li dir="ltr">Use the determinant formula for calculating the orientation of a polygon<a href="https://github.com/elastic/elasticsearch/pull/27967"> #27967</a></li>
	<li dir="ltr">update rollover to leverage write-alias semantics<a href="https://github.com/elastic/elasticsearch/pull/32216"> #32216</a></li>
</ul><p dir="ltr"><strong>Changes in 7.0:
</strong></p><ul>
	<li dir="ltr">NETWORKING: Fix Netty Leaks by upgrading to 4.1.28<a href="https://github.com/elastic/elasticsearch/pull/32511"> #32511</a></li>
	<li dir="ltr">Fix AutoIntervalDateHistogram.testReduce random failures<a href="https://github.com/elastic/elasticsearch/pull/32301"> #32301</a></li>
	<li dir="ltr">[Kerberos] Add missing javadocs<a href="https://github.com/elastic/elasticsearch/pull/32469"> #32469</a></li>
</ul><h3 dir="ltr">Apache Lucene</h3><p dir="ltr"><strong>Lucene 8.0
</strong></p><p dir="ltr">When we initially started the discussion about releasing Lucene 8.0, it was argued that we should first make the new top-hits optimizations easier to use and enabled by default,<a href="https://issues.apache.org/jira/browse/LUCENE-8060"> which we did</a>. As a consequence, we<a href="https://markmail.org/message/mz456wtoavxi27cu"> revived the email thread</a> to discuss a target date, the initial proposal being October 2018.
</p><p dir="ltr"><strong>Faster queries when not counting hits
</strong></p><p dir="ltr"> - We have a working patch that shows<a href="https://issues.apache.org/jira/browse/LUCENE-8204"> great speedups for boolean queries that mix MUST and SHOULD clauses</a> when not counting total hits.
</p><p dir="ltr"> - We are proposing a simple<a href="https://issues.apache.org/jira/browse/LUCENE-8439"> optimization</a> to dis-max queries, which is yet very effective in the common case that one clause dominates the scores, such as when searching across a short and a long field (eg. title/body).
</p><p dir="ltr"> - Sorted indices should<a href="https://issues.apache.org/jira/browse/LUCENE-8432"> stop calling the comparator when they are only counting hits</a>.
</p><p dir="ltr"><strong>BKD-based geo shapes
</strong></p><p dir="ltr"> - We are adding support for<a href="https://issues.apache.org/jira/browse/LUCENE-8440"> indexing lines and points as triangles</a>, so that they could be indexed into the same field as polygons.
</p><p dir="ltr"> - We are adding support for<a href="https://issues.apache.org/jira/browse/LUCENE-8435"> finding all polygons that intersect a given polygon</a> (the query). We currently only support finding polygons that intersect a box.
</p><p dir="ltr"><strong>Other</strong>
</p><p dir="ltr"> - Sorted indices<a href="https://issues.apache.org/jira/browse/LUCENE-8441"> failed to validate that fields used for sorting had the expected doc-value type</a>, which caused a very confusing null-pointer-exception upon abort. Jim fixed the NPE, and added index-time validation so that Lucene fails a single document rather than an entire segment.
</p><p dir="ltr"> - CheckIndex&apos;s duplication of SegmentInfos serialization was completely<a href="https://issues.apache.org/jira/browse/LUCENE-8437"> out-of-date</a>.
</p><p dir="ltr"> - Even though RAMDirectory should only be used for testing, it deserves some<a href="https://issues.apache.org/jira/browse/LUCENE-8438"> cleanups</a>.
</p><p dir="ltr"> - We shouldn&apos;t try to optimize exception handling by using pre-built exceptions in order to<a href="https://issues.apache.org/jira/browse/LUCENE-8434"> avoid filling stack traces</a>. Not only does it make debugging impossible but it also shouldn&apos;t be necessary as the JVM can skip creating stack traces if they are not used.
</p><p dir="ltr"> - Should we<a href="https://issues.apache.org/jira/browse/LUCENE-8433"> add support for comparing Object[] arrays to FutureArrays</a>?
</p><p dir="ltr"> -<a href="https://issues.apache.org/jira/browse/LUCENE-8429"> DaciukMihovAutomatonBuilder now has protection against stack overflows</a>.
</p></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-08-03

<html><head></head><body><p dir="ltr"><span style="font-size: 24px;">Elasticsearch
</span></p><p dir="ltr"><strong>Ingest Processors
</strong></p><p dir="ltr">We have recently undertaken an effort to add significant functionality to the ingest node feature of Elasticsearch. This week&#x2019;s work included PRs to add a <a href="https://github.com/elastic/elasticsearch/pull/32398">conditional</a> to any processor (so that you can only execute a processor if some per-document condition holds), a <a href="https://github.com/elastic/elasticsearch/pull/32278">drop processor</a> (to drop documents in the ingest pipeline), enabling <a href="https://github.com/elastic/elasticsearch/pull/32286">default pipelines</a>, <a href="https://github.com/elastic/elasticsearch/pull/32297">dissect</a> functionality (less powerful than Grok but faster and simpler for <a href="https://www.elastic.co/blog/logstash-dude-wheres-my-chainsaw-i-need-to-dissect-my-logs">certain use-cases</a>), and the ability within the convert processor to parse strings representing <a href="https://github.com/elastic/elasticsearch/pull/32213">hex integers</a>.
</p><p dir="ltr"><strong>Weighted Average Aggregation
</strong></p><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/31037">added</a> a new weighted average metric aggregation, which is similar to the standard average aggregation but uses a weight value from the document together with the values to average. This allows users to produce averages where the denominator (the count) is not necessarily 1 and is instead determined by another field in the document.
</p><p dir="ltr"><strong>Document ID bug in rollups
</strong></p><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/issues/32372">found a bug</a> in our experimental rollups feature where our use of a 32-bit hash to generate document IDs has meant there is a reasonable chance of collisions when the number of rollup documents reaches the order of 100,000s. We have a plan to fix this and migrate current rollup jobs over to the fix.
</p><p dir="ltr"><strong>Security Features
</strong></p><p dir="ltr">A variety of new security features have been merged and will be available in an upcoming release.  These features include Kerberos, FIPS-compliance, and application privileges, the latter of which will clear the way for Kibana to store detailed authorization information in Elasticsearch for more granular security.
</p><p dir="ltr"><strong>Working towards enabling nanosecond timestamps
</strong></p><p dir="ltr">Nanosecond timestamp resolution is a heavily <a href="https://github.com/elastic/elasticsearch/issues/10005">requested</a> <a href="https://github.com/elastic/beats/issues/7559">feature</a> and important for logging use cases where nanosecond timestamps are important for correctly ordering high-speed events (e.g., 10GbE network events; even a gigabit networks can render millisecond resolution insufficient). Today we rely on the Joda library for interacting with dates and times. However, this library only supports milliseconds. Joda was for many years a highly-respected library but is now deprecated in favor of the Java date/time API added in JDK 8 and so will never see support for such resolutions. The problem is Joda time is everywhere in the codebase from aggregations to the clients to ingest to the mapping layer to scripting so it is a massive effort to cutover. We are currently executing on a <a href="https://github.com/elastic/elasticsearch/issues/27330">plan</a> to migrate to the new Java date/time API.
</p><p dir="ltr"><strong>Zen2 Node Discovery
</strong></p><p dir="ltr">We <a href="https://github.com/elastic/elasticsearch/pull/32246">opened a PR</a> which will add to Zen2 the ability to discover master-eligible nodes via a gossip-like mechanism.  This work is an important piece of the usability story for our new cluster coordination layer.
</p><p dir="ltr">More String functions for SQL
</p><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/32356">added support</a> for a number of string manipulation functions for SQL. This is part of a larger effort to support SQL scalar functions so we will be adding more scalar function support for manipulating other data types including dates and numerics.
</p><p dir="ltr"><strong>Changes
</strong></p><p dir="ltr"><strong>Changes in 6.4:
</strong></p><ul>
	<li dir="ltr">Copy missing segment attributes in getSegmentInfo <a href="https://github.com/elastic/elasticsearch/pull/32396">#32396</a></li>
	<li dir="ltr">Introduce fips_mode setting and associated checks <a href="https://github.com/elastic/elasticsearch/pull/32326">#32326</a></li>
	<li dir="ltr">[Kerberos] Add Kerberos authentication support <a href="https://github.com/elastic/elasticsearch/pull/32263">#32263</a></li>
	<li dir="ltr">Security: revert to old way of merging automata <a href="https://github.com/elastic/elasticsearch/pull/32254">#32254</a></li>
	<li dir="ltr">Tribe: Add error with secure settings copied to tribe <a href="https://github.com/elastic/elasticsearch/pull/32298">#32298</a></li>
	<li dir="ltr">Add ERR to ranking evaluation documentation <a href="https://github.com/elastic/elasticsearch/pull/32314">#32314</a></li>
	<li dir="ltr">Introduce Application Privileges with support for Kibana RBAC <a href="https://github.com/elastic/elasticsearch/pull/32309">#32309</a></li>
	<li dir="ltr">Backport to 6.x - Add Snapshots Status API to High Level Rest Client <a href="https://github.com/elastic/elasticsearch/pull/32295">#32295</a></li>
	<li dir="ltr">Register ERR metric with NamedXContentRegistry <a href="https://github.com/elastic/elasticsearch/pull/32320">#32320</a></li>
	<li dir="ltr">[CI] Reactivate 3rd party tests on CI <a href="https://github.com/elastic/elasticsearch/pull/32315">#32315</a></li>
	<li dir="ltr">Rest HL client: Add put watch action (#32026) <a href="https://github.com/elastic/elasticsearch/pull/32191">#32191</a></li>
	<li dir="ltr">Consistent encoder names <a href="https://github.com/elastic/elasticsearch/pull/29492">#29492</a></li>
	<li dir="ltr">Add WeightedAvg metric aggregation <a href="https://github.com/elastic/elasticsearch/pull/31037">#31037</a></li>
	<li dir="ltr">Rename ranking evaluation quality_level to metric_score <a href="https://github.com/elastic/elasticsearch/pull/32168">#32168</a></li>
	<li dir="ltr">Fail shard if IndexShard#storeStats runs into an IOException <a href="https://github.com/elastic/elasticsearch/pull/32241">#32241</a></li>
	<li dir="ltr">Fix range queries on _type field for singe type indices (#31756) <a href="https://github.com/elastic/elasticsearch/pull/32161">#32161</a></li>
	<li dir="ltr">CCE when re-throwing &quot;shard not available&quot; exception in TransportShardMultiGetAction <a href="https://github.com/elastic/elasticsearch/pull/32185">#32185</a></li>
</ul><p dir="ltr"><strong>Changes in 6.5:
</strong></p><ul>
	<li dir="ltr">Release requests in cors handle <a href="https://github.com/elastic/elasticsearch/pull/32410">#32410</a></li>
	<li dir="ltr">Rest HL client: Add put license action <a href="https://github.com/elastic/elasticsearch/pull/32214">#32214</a></li>
	<li dir="ltr">Release requests in cors handler <a href="https://github.com/elastic/elasticsearch/pull/32364">#32364</a></li>
	<li dir="ltr">Add Restore Snapshot High Level REST API <a href="https://github.com/elastic/elasticsearch/pull/32155">#32155</a></li>
	<li dir="ltr">BREAKING: Introduce index store plugins <a href="https://github.com/elastic/elasticsearch/pull/32375">#32375</a></li>
	<li dir="ltr">Add opaque_id to index audit logging <a href="https://github.com/elastic/elasticsearch/pull/32260">#32260</a></li>
	<li dir="ltr">Ingest: Support integer and long hex values in convert <a href="https://github.com/elastic/elasticsearch/pull/32213">#32213</a></li>
</ul><p dir="ltr"><strong>Changes in 7.0:
</strong></p><ul>
	<li dir="ltr">INGEST: Fix Deprecation Warning in Script Proc. <a href="https://github.com/elastic/elasticsearch/pull/32407">#32407</a></li>
	<li dir="ltr">Networking: Fix test leaking buffer <a href="https://github.com/elastic/elasticsearch/pull/32296">#32296</a></li>
</ul><p dir="ltr"><span style="font-size: 24px;">Lucene
</span></p><p dir="ltr"><strong>Disabling hit counts by default
</strong></p><p dir="ltr">The major release highlight of Lucene 8 is going to be a set of optimizations that allow to compute top matches sorted by score more efficiently by not having to visit all matches. Unfortunately, these improvements are not going to be noticed by our users if we keep computing total hit counts by default, which requires to visit all matches.
</p><p dir="ltr">As a consequence, we have been discussing what it would take to <a href="https://issues.apache.org/jira/browse/LUCENE-8060">disable hit counts by default</a>. While some UIs don&apos;t need hit counts at all, for instance a lot of mobile search UIs implement infinite scrolling and don&apos;t give information about the hit count, traditional search UIs still give information about the hit count (often approximated) and provide pagination support. For instance if you want to allow users to paginate up to page 10 and display 20 hits per page, it is useful to count hits accurately up to 200 in order to know how many pages you need to make available to your users. In order to keep Lucene practical, we have been discussing changing our top collectors so that the computation of the hit count is not a yes/no choice, but rather <a href="https://issues.apache.org/jira/browse/LUCENE-8431">a number of hits to count accurately</a>. If the number of matches of a query is below this number then the hit count will be accurate, and otherwise it will be a <a href="https://issues.apache.org/jira/browse/LUCENE-8430">lower bound of the actual hit count</a>.
</p><p dir="ltr"><strong>ReqOptSumScorer to optimize query processing based on impacts
</strong></p><p dir="ltr">One way that queries now optimize collection of top documents is by adding information about the produced scores directly into the skip lists of the inverted index so that documents that don&apos;t yield competitive scores can be skipped. This is currently leveraged by term queries, disjunctions and conjunctions. We are&#xA0;now optimizing <a href="https://issues.apache.org/jira/browse/LUCENE-8204">ReqOptSumScorer</a>, which is used for queries that mix MUST/FILTER and SHOULD clauses. These queries are typically used in order to boost the score of some documents by the value of another field such as &quot;pagerank&quot; or &quot;popularity&quot; by putting the regular query in a MUST clause and the boosting query in a SHOULD clause: this will return the same matches as the regular query, but scores will be the sum of the regular query score and the boosting query score. Having those queries optimized is exciting as it means that one could incorporate features into the score and still benefit from optimized computation of the top hits.
</p><p dir="ltr"><strong>LatLonPoint has moved to core
</strong></p><p dir="ltr">After having lived in sandbox for years, we promoted LatLonPoint to lucene-core. This is the best available option to index geo-points with Lucene, and the one that is used by Elasticsearch&apos;s geo_point field.
</p><p dir="ltr"><strong>SegmentReader now exposes both soft and hard deletes
</strong></p><p dir="ltr">Until now SegmentReader would either expose hard deletes, or soft deletes if you configured a soft-deletes field with your IndexWriter. This proved problematic since you sometimes need to ignore soft deletes, yet not all deleted documents are documents that have been soft-deleted in that case: Lucene also creates born-deleted documents when it hits an exception during the indexing process, for instance if the analyzer throws an exception. In order to make it possible to distinguish legitimate documents that have been soft-deleted from documents that failed indexing, we introduced the ability to fetch hard deletes on a SegmentReader.
</p><p dir="ltr"><strong>Match iterators
</strong></p><p dir="ltr">We introduced a new getSubMatches() method in the matches API, which allows to iterate <a href="https://issues.apache.org/jira/browse/LUCENE-8306">sub-matches</a> of a top-level match. For instance if you think of a phrase query &quot;quick fox&quot; with a slop of 2 and text &quot;the quick yellow fox&quot;, then the query would return &quot;quick yellow fox&quot; as a top-level match, and &quot;quick&quot; and &quot;fox&quot; as sub matches.
</p><p dir="ltr">We are now adding support for matches to <a href="https://issues.apache.org/jira/browse/LUCENE-8422">interval queries</a>.
</p><p dir="ltr"><strong>Other
</strong></p><ul>
	<li dir="ltr">DaciukMihovAutomatonBuilder, a helper class that implements an algorithm that can build an automaton in constant-time from a sorted set of strings, now has <a href="https://issues.apache.org/jira/browse/LUCENE-8429">protection against stack overflows</a>.</li>
	<li dir="ltr">A soft deletes optimization to copy live docs efficiently got <a href="https://issues.apache.org/jira/browse/LUCENE-8410">broken because of an unrelated change</a>.</li>
	<li dir="ltr">The way that Lucene allows to pre-fill PriorityQueue objects with sentinels made could be <a href="https://issues.apache.org/jira/browse/LUCENE-8428">hard to use</a>.</li>
	<li dir="ltr">We found some <a href="https://issues.apache.org/jira/browse/LUCENE-8423">leniency</a> in the way that Lucene carries over previous generations when overriding an existing index.</li>
	<li dir="ltr">Could we <a href="https://issues.apache.org/jira/browse/LUCENE-8432">optimize TopFieldCollector when the index is sorted but hit counts need to be computed</a>?</li>
	<li dir="ltr"><a href="https://issues.apache.org/jira/browse/LUCENE-8424">Null payloads</a> shouldn&apos;t count as non-null payloads for scoring.</li>
	<li dir="ltr">Clarified the <a href="https://issues.apache.org/jira/browse/LUCENE-8415">contract</a> that Directory implementations must obey.</li>
	<li dir="ltr"><a href="https://issues.apache.org/jira/browse/LUCENE-8412">TopFieldCollector no longer computes scores at collection time</a>, this is now done after collection, only on top hits.</li>
	<li dir="ltr">The <a href="https://issues.apache.org/jira/browse/LUCENE-8419">stempel stemmer is sometimes way too aggressive</a>.</li>
</ul></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-07-27

<html><head></head><body><p dir="ltr"><span style="font-size: 24px;">Elasticsearch
	
</span></p><p dir="ltr"><strong>Field aliases
	</strong>
</p><p dir="ltr">We have merged a new field <a href="https://github.com/elastic/elasticsearch/issues/31372">aliases</a> <a href="https://github.com/elastic/elasticsearch/pull/32172">feature</a> which allows users to specify a new <code>alias</code> field type. This new field type points has a property, &#x201C;path&#x201D;, which points to a different field and will resolve to that field whenever it is used in search and aggregations. This helps users migrate from old mappings to new ones when only field names are changing, allowing them to set field aliases on the old indices which use the new field names, so the old indices are still searchable to applications using the new schema.
</p><p dir="ltr"><strong>Kerberos
	</strong>
</p><p dir="ltr">The kerberos work is ready to be merged. The initial implementation will rely on native role mapping from the kerberos principal name. Lookups for users from LDAP/AD will come as a follow up based on lookup realms.
</p><p dir="ltr"><strong>Zen2
	</strong>
</p><p dir="ltr">This week in the internal Zen2 PoC we merged the node retirement API that allows for safely reducing a 2-node cluster to a 1-node one. This is an important API for Elastic Cloud, where many users use a 1-node cluster which can grow to 2 and back to 1 during upgrades. This is the last missing feature in our PoC.
</p><p dir="ltr">We are making good progress on the zen2 branch of the public ES repo, which has seen PRs for the addition of <a href="https://github.com/elastic/elasticsearch/pull/32100">terms to cluster state</a> , <a href="https://github.com/elastic/elasticsearch/pull/32171">the heart of the coordination layer </a>, <a href="https://github.com/elastic/elasticsearch/pull/32242">the beginning of testing framework</a>, <a href="https://github.com/elastic/elasticsearch/pull/32246">and a gossip discovery protocol</a>.
</p><p dir="ltr"><strong>JDK Bug
</strong></p><p dir="ltr"><strong>
</strong></p><p dir="ltr">Several of our users recently encountered a <a href="https://bugs.openjdk.java.net/browse/JDK-8207746">JVM Bug</a> that manifests on machines supporting AVX-512 (e.g., Skylake X) when running Elasticsearch on JDK 10. As our 6.3.0 and 6.3.1 docker images contain JDK 10, this issue affects potentially many users. A <a href="https://github.com/elastic/elasticsearch/issues/31425#issuecomment-405429166">workaround</a> is described on the Elasticsearch Github repo.
</p><p dir="ltr"><strong>Low Memory Resiliency
	</strong>
</p><p dir="ltr">We made an important step forward this week in our efforts to run on smaller heaps.  To date, when evaluating whether or not to service a user&#x2019;s request, Elasticsearch examined memory explicitly reserved by other queries as compared to a total limit (70% of heap).  However, these explicit reservations did not accurately capture memory usage.  With a <a href="https://github.com/elastic/elasticsearch/pull/31767">rebuilt circuit breaker</a>, Elasticsearch now measures real heap memory usage before servicing the request.  The result?  Elasticsearch is now more resilient to OutOfMemoryErrors, while at the same time being able to use more of the heap (95% now).
</p><p dir="ltr"><strong>Fetch tasks are no longer rejected
	</strong>
</p><p dir="ltr">The search thread pool needs to run two main kinds of tasks: query tasks and fetch tasks. We have pushed a <a href="https://github.com/elastic/elasticsearch/pull/31974">change</a> that force-adds fetch tasks to the queue even in the event that the queue is already full. The reasoning behind this is that fetch tasks may only be follow-up to query tasks, so the number of additional fetch tasks that may enter the thread pool is expected to be reasonable.
</p><p dir="ltr"><strong>Rollup capabilities now available on index-level
	</strong>
</p><p dir="ltr">We have added the ability to request the rollup capabilities for a <a href="https://github.com/elastic/elasticsearch/pull/32111">rollup index</a> in addition to the existing API which lists the rollup capabilities for all jobs targeting a pattern of source indexes (i.e. indexes where the original non-rolled up data lives). This will help the rollup UI determine the fields and aggregations available to Kibana index patterns which include rollup indexes.
</p><p dir="ltr"><strong>Changes in 6.0:</strong>
</p><ul>
	<li dir="ltr">Fix put mappings java API documentation <a href="https://github.com/elastic/elasticsearch/pull/31955">#31955</a></li>
</ul><p dir="ltr"><strong>Changes in 6.3:
	</strong>
</p><ul>
	<li dir="ltr">A replica can be promoted and started in one cluster state update <a href="https://github.com/elastic/elasticsearch/pull/32042">#32042</a></li>
	<li dir="ltr">Adjust translog after versionType is removed in 7.0 <a href="https://github.com/elastic/elasticsearch/pull/32020">#32020</a></li>
	<li dir="ltr">Disable C2 from using AVX-512 on JDK 10 <a href="https://github.com/elastic/elasticsearch/pull/32138">#32138</a></li>
</ul><p dir="ltr"><strong>Changes in 6.4:
	</strong>
</p><ul>
	<li dir="ltr">Rest HL client: Add put watch action <a href="https://github.com/elastic/elasticsearch/pull/32026">#32026</a></li>
	<li dir="ltr">Add support for field aliases. <a href="https://github.com/elastic/elasticsearch/pull/32172">#32172</a></li>
	<li dir="ltr">add support for write index resolution when creating/updating documents <a href="https://github.com/elastic/elasticsearch/pull/31520">#31520</a></li>
	<li dir="ltr">ECS Task IAM profile credentials ignored in repository-s3 plugin <a href="https://github.com/elastic/elasticsearch/pull/31864">#31864</a></li>
	<li dir="ltr">Fix rollup on date fields that don&#x2019;t support epoch_millis <a href="https://github.com/elastic/elasticsearch/pull/31890">#31890</a></li>
	<li dir="ltr">Revert &quot;Introduce a Hashing Processor (#31087)&quot; <a href="https://github.com/elastic/elasticsearch/pull/32179">#32179</a></li>
	<li dir="ltr">Call setReferences() on custom referring tokenfilters in _analyze <a href="https://github.com/elastic/elasticsearch/pull/32157">#32157</a></li>
	<li dir="ltr">Add more contexts to painless execute api <a href="https://github.com/elastic/elasticsearch/pull/30511">#30511</a></li>
	<li dir="ltr">Fix range queries on _type field for singe type indices <a href="https://github.com/elastic/elasticsearch/pull/31756">#31756</a></li>
	<li dir="ltr">BREAKING: Configurable password hashing algorithm/cost(#31234) <a href="https://github.com/elastic/elasticsearch/pull/32092">#32092</a></li>
	<li dir="ltr">Handle missing values in painless (#30975) <a href="https://github.com/elastic/elasticsearch/pull/31903">#31903</a></li>
	<li dir="ltr">Painless: Fix Bug with Duplicate PainlessClasses <a href="https://github.com/elastic/elasticsearch/pull/32110">#32110</a></li>
	<li dir="ltr">Ensure to release translog snapshot in primary-replica resync <a href="https://github.com/elastic/elasticsearch/pull/32045">#32045</a></li>
	<li dir="ltr">Check that client methods match API defined in the REST spec <a href="https://github.com/elastic/elasticsearch/pull/31825">#31825</a></li>
	<li dir="ltr">Update monitoring template version to 6040099 <a href="https://github.com/elastic/elasticsearch/pull/32088">#32088</a></li>
	<li dir="ltr">Add exclusion option to keep_types token filter <a href="https://github.com/elastic/elasticsearch/pull/32012">#32012</a></li>
	<li dir="ltr">Add Index UUID to /_stats Response <a href="https://github.com/elastic/elasticsearch/pull/31871">#31871</a></li>
	<li dir="ltr">Bypass highlight query terms extraction on empty fields <a href="https://github.com/elastic/elasticsearch/pull/32090">#32090</a></li>
	<li dir="ltr">Core: Backport java time date formatters <a href="https://github.com/elastic/elasticsearch/pull/31997">#31997</a></li>
	<li dir="ltr">[Rollup] Add new capabilities endpoint based on concrete rollup indices <a href="https://github.com/elastic/elasticsearch/pull/30401">#30401</a></li>
	<li dir="ltr">SQL: allow LEFT and RIGHT as function names <a href="https://github.com/elastic/elasticsearch/pull/32066">#32066</a></li>
	<li dir="ltr">Watcher: Store username on watch execution <a href="https://github.com/elastic/elasticsearch/pull/31873">#31873</a></li>
	<li dir="ltr">Scripting: Remove Dead Code from Painless Module <a href="https://github.com/elastic/elasticsearch/pull/32064">#32064</a></li>
</ul><p dir="ltr"><strong>Changes in 7.0:
	</strong>
</p><ul>
	<li dir="ltr">Handle missing values in painless <a href="https://github.com/elastic/elasticsearch/pull/32207">#32207</a></li>
	<li dir="ltr">Revert &quot;Introduce a Hashing Processor (#31087)&quot; <a href="https://github.com/elastic/elasticsearch/pull/32178">#32178</a></li>
	<li dir="ltr">Adjust SSLDriver behavior for JDK11 changes <a href="https://github.com/elastic/elasticsearch/pull/32145">#32145</a></li>
	<li dir="ltr">Remove versionType from translog <a href="https://github.com/elastic/elasticsearch/pull/31945">#31945</a></li>
	<li dir="ltr">Replace TokenizerFactory with Supplier <a href="https://github.com/elastic/elasticsearch/pull/32063">#32063</a></li>
	<li dir="ltr">Relax TermVectors API to work with textual fields other than TextFieldType <a href="https://github.com/elastic/elasticsearch/pull/31915">#31915</a></li>
</ul><p dir="ltr"><span style="font-size: 24px;">Lucene
	
</span></p><p dir="ltr"><strong>Low-level highlighting components
	</strong>
</p><p dir="ltr">One reason (out of many!) why building a highlighter is challenging is that many things need to be configured: what is a good snippet, how much context should be kept around matches, should snippets that occur next to each other be merged, etc. We are exploring building highlighter components that can be used together rather than a full-fledged highlighter. As a start, we are using the matches API to break text up into passages that contain hits.
</p><p dir="ltr"><strong>Other
	</strong>
</p><ul>
	<li dir="ltr">When sorting by a field, it would be <a href="https://issues.apache.org/jira/browse/LUCENE-8412">more efficient to run a second search to compute scores of the top hits</a> compared to doing it at collection time.</li>
	<li dir="ltr">Since computing sort values is cheap, <a href="https://issues.apache.org/jira/browse/LUCENE-8411">we don&apos;t need a boolean to enable it</a>.</li>
	<li dir="ltr">Soft deletes support had an <a href="https://issues.apache.org/jira/browse/LUCENE-8410">optimization</a> that never kicked in because it relied on outdated assumptions.</li>
	<li dir="ltr">The matches API is getting support for <a href="https://issues.apache.org/jira/browse/LUCENE-8306">extracting sub matches</a>. This is especially useful for phrase queries with slops in order to know where terms of the query occurred.</li>
	<li dir="ltr">Should we allow term vectors to <a href="https://issues.apache.org/jira/browse/LUCENE-8403">only store a subset of the terms that exist in the inverted index</a>?</li>
	<li dir="ltr">A <a href="https://issues.apache.org/jira/browse/LUCENE-8398">missing double cast</a> made TieredMergePolicy&apos;s getter for the max segment size return a wrong value.</li>
</ul></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-07-20

<html><head></head><body><h3 dir="ltr" id="docs-internal-guid-30f7d4ba-9528-70eb-61e9-941a5afb6b76">Elasticsearch
</h3><h4 dir="ltr">Kerberos
</h4><p dir="ltr">The kerberos realm has been <a href="https://github.com/elastic/elasticsearch/pull/31761">merged into the feature branch</a>. We are currently working on a QA test that uses an actual KDC. Once the lookup realms feature has been completed, we will need to integrate the kerberos realm with lookup realms but this will not block the completion of the kerberos realm.
</p><h4 dir="ltr">Structured Audit Logging
</h4><p dir="ltr">We hafe raised an in progress PR for <a href="https://github.com/elastic/elasticsearch/pull/31931">structured audit logging</a>. This PR makes use of the log4j StringMapMessage for audit events. They allow the greatest flexibility in terms of final log line format. Specifically, log4j defines layouts  which format the logline printf style utilizing the values from the map message event. This should allow us to experiment with multiple formats by only changing the log4j config.
</p><h4 dir="ltr">Cross Cluster Replication
</h4><p dir="ltr">We have merged the rewrite for the ShardFollowingTask. The rewrite was meant to simplify the logic and allow us to get better insights into the various internals of the tasks for better monitoring and control (for example, how many operations are already fetched from the leader buffered up for writes on the follower).
</p><h4 dir="ltr">Zen2
</h4><p dir="ltr">The Zen2 project is now managed on our<a href="https://github.com/elastic/elasticsearch/tree/zen2"> public repo</a> and is now being tracked on a<a href="https://github.com/elastic/elasticsearch/issues/32006"> meta issue</a>. Due to the complexity of the project, the POC phase was quite elaborate and the code is relatively mature for a POC. Work will focus on porting the POC code to a production level, simplifying things and adding tests. This is a major milestone. Congratulations to David and Yannick for reaching it!
</p><h4 dir="ltr">Auto-Interval Date Histogram
</h4><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/28993">merged a PR</a> for a new aggregation, called auto_date_histogram. It works like a date_histogram, but instead of specifying the time interval, you specify the max number of buckets you&#x2019;d like, and the aggregation chooses the interval that will be closest to the maximum without going over.
</p><h4 dir="ltr">SQL Drivers
</h4><p dir="ltr">Work on ODBC and JDBC drivers continues, with a goal of getting to parity. We have been working on adding <a href="https://github.com/elastic/elasticsearch-sql-odbc/pull/11">parameterised execution to the ODBC driver</a>, allowing a statement to be prepared once and then executed multiple times with different data parameters. For JDBC, adding <a href="https://github.com/elastic/elasticsearch/pull/31874">single parameter text manipulating functions to SQL</a> which allows users to transform text using function such as LENGTH, UCASE, LCASE and LTRIM
</p><h4 dir="ltr">Changes in 6.3:
</h4><ul>
	<li dir="ltr">SQL: HAVING clause should accept only aggregates<a href="https://github.com/elastic/elasticsearch/pull/31872"> #31872</a></li>
	<li dir="ltr">Fix building AD URL from domain name<a href="https://github.com/elastic/elasticsearch/pull/31849"> #31849</a></li>
	<li dir="ltr">Watcher: Add ssl.trust email account setting<a href="https://github.com/elastic/elasticsearch/pull/31684"> #31684</a></li>
	<li dir="ltr">Watcher: Increase HttpClient parallel sent requests<a href="https://github.com/elastic/elasticsearch/pull/31859"> #31859</a></li>
	<li dir="ltr">Inconsistency between description and example<a href="https://github.com/elastic/elasticsearch/pull/31858"> #31858</a></li>
	<li dir="ltr">SQL: Fix incorrect HAVING equality<a href="https://github.com/elastic/elasticsearch/pull/31820"> #31820</a></li>
</ul><h4 dir="ltr">Changes in 6.4:
</h4><ul>
	<li dir="ltr">Slack message empty text<a href="https://github.com/elastic/elasticsearch/pull/31596"> #31596</a></li>
	<li dir="ltr">Date: Add DateFormatters class that uses java.time<a href="https://github.com/elastic/elasticsearch/pull/31856"> #31856</a></li>
	<li dir="ltr">Tests: Remove use of joda time in some tests<a href="https://github.com/elastic/elasticsearch/pull/31922"> #31922</a></li>
	<li dir="ltr">Add Get Snapshots High Level REST API<a href="https://github.com/elastic/elasticsearch/pull/31980"> #31980</a></li>
	<li dir="ltr">Force execution of fetch tasks<a href="https://github.com/elastic/elasticsearch/pull/31974"> #31974</a></li>
	<li dir="ltr">Add Expected Reciprocal Rank metric<a href="https://github.com/elastic/elasticsearch/pull/31891"> #31891</a></li>
	<li dir="ltr">SQL: Add support for single parameter text manipulating functions<a href="https://github.com/elastic/elasticsearch/pull/31874"> #31874</a></li>
	<li dir="ltr">SQL: Support for escape sequences<a href="https://github.com/elastic/elasticsearch/pull/31884"> #31884</a></li>
	<li dir="ltr">Add Snapshots Status API to High Level Rest Client<a href="https://github.com/elastic/elasticsearch/pull/31515"> #31515</a></li>
	<li dir="ltr">ingest: date_index_name processor template resolution<a href="https://github.com/elastic/elasticsearch/pull/31841"> #31841</a></li>
	<li dir="ltr">Test: fix null failure in watcher test<a href="https://github.com/elastic/elasticsearch/pull/31968"> #31968</a></li>
	<li dir="ltr">Added lenient flag for synonym token filter<a href="https://github.com/elastic/elasticsearch/pull/31484"> #31484</a></li>
	<li dir="ltr">Fix wrong NaN check in MovingFunctions#stdDev()<a href="https://github.com/elastic/elasticsearch/pull/31888"> #31888</a></li>
	<li dir="ltr">Add opaque_id to audit logging<a href="https://github.com/elastic/elasticsearch/pull/31878"> #31878</a></li>
	<li dir="ltr">add support for is_write_index in put-alias body parsing<a href="https://github.com/elastic/elasticsearch/pull/31674"> #31674</a></li>
	<li dir="ltr">Handle missing values in painless<a href="https://github.com/elastic/elasticsearch/pull/30975"> #30975</a></li>
	<li dir="ltr">BREAKING: High Leven REST Client: Add x-pack-info API<a href="https://github.com/elastic/elasticsearch/pull/31870"> #31870</a></li>
	<li dir="ltr">Do not return all indices if a specific alias is requested via get aliases api.<a href="https://github.com/elastic/elasticsearch/pull/29538"> #29538</a></li>
	<li dir="ltr">Ingest: Enable Templated Fieldnames in Rename<a href="https://github.com/elastic/elasticsearch/pull/31690"> #31690</a></li>
	<li dir="ltr">Ingest: Add ignore_missing option to RemoveProc<a href="https://github.com/elastic/elasticsearch/pull/31693"> #31693</a></li>
	<li dir="ltr">Add template config for Beat state to X-Pack Monitoring<a href="https://github.com/elastic/elasticsearch/pull/31809"> #31809</a></li>
	<li dir="ltr">SQL: Remove restriction for single column grouping<a href="https://github.com/elastic/elasticsearch/pull/31818"> #31818</a></li>
	<li dir="ltr">Check timeZone argument in AbstractSqlQueryRequest<a href="https://github.com/elastic/elasticsearch/pull/31822"> #31822</a></li>
	<li dir="ltr">Fix profiling of ordered terms aggs<a href="https://github.com/elastic/elasticsearch/pull/31814"> #31814</a></li>
</ul><h4 dir="ltr">Changes in 6.5:
</h4><ul>
	<li dir="ltr">[X-Pack] Beats centralized management: security role + licensing<a href="https://github.com/elastic/elasticsearch/pull/30520"> #30520</a></li>
</ul><h4 dir="ltr">Changes in 7.0:
</h4><ul>
	<li dir="ltr">Tests: Fix SearchFieldsIT.testDocValueFields<a href="https://github.com/elastic/elasticsearch/pull/31995"> #31995</a></li>
	<li dir="ltr">BREAKING: Remove the ability to index or query context suggestions without context<a href="https://github.com/elastic/elasticsearch/pull/31007"> #31007</a></li>
</ul><h2 dir="ltr">Apache Lucene
</h2><h4 dir="ltr">Points-based geo shapes
</h4><p dir="ltr">Lucene 6 introduced a new API called<a href="https://lucene.apache.org/core/7_4_0/core/org/apache/lucene/index/PointValues.html"> points</a>, which is implemented under the hood with a<a href="https://en.wikipedia.org/wiki/K-d_tree"> BKD-tree</a>. Since then, points been the way to go in order to index and search numerics (1 dimension), geo points and numeric ranges (2 dimensions) as BKD trees provide faster searching and better compression than the previous inverted-index-based support.
</p><p dir="ltr">The next logical step would be to implement shape support on top of the points API, but this isn&apos;t without raising questions: shapes are more commonly indexed using<a href="https://en.wikipedia.org/wiki/R-tree"> R-trees</a> rather than BKD trees, which are designed for points. However adding support for a new type of data-structure to Lucene isn&apos;t a small task, and the proposal to<a href="https://issues.apache.org/jira/browse/LUCENE-7110"> add support for R-trees</a> didn&apos;t get much traction.
</p><p dir="ltr">We are back with a<a href="https://issues.apache.org/jira/browse/LUCENE-8396"> new proposal</a> that uses the BKD tree as a R-tree by tessellating shapes into triangles that are indexed as 6-dimensional points. The minimum and maximum values for each coordinate that the BKD tree provides on inner levels help compute the minimum bounding rectangle that contains all triangles on the leaves, just like a R-tree would provide. We are also confident that the current split mechanism will perform well. Initial results are very promising.
</p><h4 dir="ltr">Other
</h4><ul><li> We fixed a bug that could make the<a href="https://issues.apache.org/jira/browse/LUCENE-8395"> word-delimiter filter add unnecessary gaps</a>.</li><li>Iterating on<a href="https://issues.apache.org/jira/browse/LUCENE-8286"> making the unified highlighter leverage the new matches API</a>.
</li><li>We submitted a first patch to<a href="https://issues.apache.org/jira/browse/LUCENE-8263"> reclaim deletes more aggressively through natural merges</a>.
</li><li>The new setMinCompetitiveScore API that was introduced for top-k queries<a href="https://issues.apache.org/jira/browse/LUCENE-8382"> needs to be used carefully</a> with MultiCollector.
</li></ul></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-07-13

<html><head></head><body><h3 dir="ltr" id="docs-internal-guid-5d5e0316-4e66-0ec2-a96e-e835523c2096">Elasticsearch</h3><h4 dir="ltr">Highlights
</h4><h5 dir="ltr">Upgrades to 6.3.0 from 5.x recovery issue
</h5><p dir="ltr">After releasing 6.3.0 we received reports that upgrades from 5.x to 6.3.0 fail, as replicas would not recover. The problem was caused by a<a href="https://github.com/elastic/elasticsearch/pull/28245"> refactoring</a> which simplified our Engine. One affected item was a BWC bridge that upgrades the Lucene index to include 6.x markers, specifically, the history uuid that is required for ops based recovery. The BWC bridge was moved to the part of recovery that copies over files from another shard. As part of index preparation, we check and place the markers in the newly built Lucene index. However, shards that have been prepared for the upgrade using a Synced Flush command do not copy any files, but use the existing files as is. This bypassed our BWC bridge causing recovery to fail. This uncovered a gap in our testing as well, as using synced flush is the recommended way to do upgrades. We<a href="https://github.com/elastic/elasticsearch/pull/28245">  fixed it</a> for the the coming 6.3.1 release.
</p><h5 dir="ltr">JDK 11
</h5><ul>
</ul><p>With JDK 11 GA around the corner (mid-late September), we are actively working on <a href="https://github.com/elastic/elasticsearch/issues/31230">getting JDK 11 into CI</a>. This has been an onerous effort for a few reasons:
</p><ul>
</ul><ul>
	<li>         no version of Gradle prior to Gradle 4.8 can run on JDK 11
	</li>
	<li>Gradle 4.8 introduced several breaking changes in a minor release</li>
	<li>dealing with JDK 11 breaking changes and test failures
	</li>
</ul><p dir="ltr">Our target is to get this into CI for the 6.x branch after the 6.4 branch is cut in preparation for supporting 6.5 on JDK 11. The Gradle 4.8 <a href="https://github.com/elastic/elasticsearch/pull/31525">work is complete</a>, we have JDK 11 in CI on a feature branch, and the <a href="https://github.com/elastic/elasticsearch/pull/31644">PR to enable JDK 11</a> is almost ready.
</p><h5 dir="ltr">Hadoop
</h5><p dir="ltr">We finalized the <a href="https://github.com/elastic/elasticsearch-hadoop/pull/1149">error handling API</a>, a heavily-demanded feature for enabling handling in the connector for errors that occur trying to serialize records and enables handling situations like fields being of the correct type or missing when trying to extract document metadata. Real-world data is messy and this enables applications that are built on top of the connector to be more selective in their handling of such situations
</p><p dir="ltr">x-opaque-id header now appears in the query slowlog
</p><p dir="ltr">We have changed the query slowlog so it <a href="https://github.com/elastic/elasticsearch/pull/31539">includes the value of the x-opaque-id header</a> if its present. This means that users can add the header to allow them to identify queries in the slow log by an id that means something to them and their application. This should help users and support more easily identify where slow queries are originating from.
</p><h5 dir="ltr">FIPS
</h5><p dir="ltr">We have merged <a href="https://github.com/elastic/elasticsearch/pull/31234">configurable password hashing</a>.
</p><h4 dir="ltr">Pull Requests<br></h4><h5 dir="ltr">Changes in 5.6:
</h5><ul>
	<li dir="ltr">Ingest Attachment: Upgrade Tika to 1.18<a href="https://github.com/elastic/elasticsearch/pull/31252"> #31252</a></li>
</ul><h5 dir="ltr">Changes in 6.3:
</h5><ul>
	<li dir="ltr">Remove item listed in 6.3 notes that&#x2019;s not in 6.3<a href="https://github.com/elastic/elasticsearch/pull/31623"> #31623</a></li>
	<li dir="ltr">Preserve thread context when connecting to remote cluster<a href="https://github.com/elastic/elasticsearch/pull/31574"> #31574</a></li>
	<li dir="ltr">Add package pre-install check for java binary<a href="https://github.com/elastic/elasticsearch/pull/31343"> #31343</a></li>
	<li dir="ltr">Watcher: Fix put watch action<a href="https://github.com/elastic/elasticsearch/pull/31524"> #31524</a></li>
	<li dir="ltr">Fix missing historyUUID in peer recovery when rolling upgrade 5.x to 6.3<a href="https://github.com/elastic/elasticsearch/pull/31506"> #31506</a></li>
</ul><h5 dir="ltr">Changes in 6.4:
</h5><ul>
	<li dir="ltr">Add Get Snapshots High Level REST API<a href="https://github.com/elastic/elasticsearch/pull/31537"> #31537</a></li>
	<li dir="ltr">Add Create Snapshot to High-Level Rest Client<a href="https://github.com/elastic/elasticsearch/pull/31215"> #31215</a></li>
	<li dir="ltr">Fix CreateSnapshotRequestTests Failure<a href="https://github.com/elastic/elasticsearch/pull/31630"> #31630</a></li>
	<li dir="ltr">BREAKING Java: Configurable password hashing algorithm/cost<a href="https://github.com/elastic/elasticsearch/pull/31234"> #31234</a></li>
	<li dir="ltr">Add rest highlevel explain API<a href="https://github.com/elastic/elasticsearch/pull/31387"> #31387</a></li>
	<li dir="ltr">Add MultiSearchTemplate support to High Level Rest client<a href="https://github.com/elastic/elasticsearch/pull/30836"> #30836</a></li>
	<li dir="ltr">Do not check for Azure container existence everytime an Azure object is accessed or modified<a href="https://github.com/elastic/elasticsearch/pull/31617"> #31617</a></li>
	<li dir="ltr">Node selector per client rather than per request<a href="https://github.com/elastic/elasticsearch/pull/31471"> #31471</a></li>
	<li dir="ltr">JDBC driver prepared statement set* methods<a href="https://github.com/elastic/elasticsearch/pull/31494"> #31494</a></li>
	<li dir="ltr">Improve robustness of geo shape parser for malformed shapes<a href="https://github.com/elastic/elasticsearch/pull/31449"> #31449</a></li>
	<li dir="ltr">Get Mapping API to honour allow_no_indices and ignore_unavailable<a href="https://github.com/elastic/elasticsearch/pull/31507"> #31507</a></li>
	<li dir="ltr">REST high-level client: add simulate pipeline API<a href="https://github.com/elastic/elasticsearch/pull/31158"> #31158</a></li>
	<li dir="ltr">Fix Mockito trying to mock IOException that isn&#x2019;t thrown by method (#31433)<a href="https://github.com/elastic/elasticsearch/pull/31527"> #31527</a></li>
	<li dir="ltr">fix writeIndex evaluation for aliases<a href="https://github.com/elastic/elasticsearch/pull/31562"> #31562</a></li>
	<li dir="ltr">Add x-opaque-id to search slow logs<a href="https://github.com/elastic/elasticsearch/pull/31539"> #31539</a></li>
	<li dir="ltr">turn GetFieldMappingsResponse to ToXContentObject<a href="https://github.com/elastic/elasticsearch/pull/31544"> #31544</a></li>
	<li dir="ltr">Add get field mappings to High Level REST API Client<a href="https://github.com/elastic/elasticsearch/pull/31423"> #31423</a></li>
	<li dir="ltr">IndexShard should not return null stats<a href="https://github.com/elastic/elasticsearch/pull/31528"> #31528</a></li>
	<li dir="ltr">fix repository update with the same settings but different type<a href="https://github.com/elastic/elasticsearch/pull/31458"> #31458</a></li>
	<li dir="ltr">Upgrade to Lucene 7.4.0.<a href="https://github.com/elastic/elasticsearch/pull/31529"> #31529</a></li>
	<li dir="ltr">BREAKING Java: Allow multiple unicast host providers<a href="https://github.com/elastic/elasticsearch/pull/31509"> #31509</a></li>
	<li dir="ltr">[PkiRealm] Invalidate cache on role mappings change<a href="https://github.com/elastic/elasticsearch/pull/31510"> #31510</a></li>
	<li dir="ltr">[Security] Check auth scheme case insensitively<a href="https://github.com/elastic/elasticsearch/pull/31490"> #31490</a></li>
</ul><h5 dir="ltr">Changes in 7.0:
</h5><ul>
	<li dir="ltr">ingest: Add ignore_missing property to foreach filter (#22147)<a href="https://github.com/elastic/elasticsearch/pull/31578"> #31578</a></li>
</ul><h2 dir="ltr">Lucene</h2><h4 dir="ltr">Lucene 7.4
</h4><p dir="ltr">Lucene 7.4 is now<a href="https://markmail.org/message/e4mlf6dl4iqskatt"> out</a> and features a korean analyzer, a new matches API, support for soft deletes and detection of emoji sequences in ICUTokenizer. This is the Lucene version that Elasticsearch 7.4 will be based on.
</p><h5 dir="ltr">Forcemerge now honors the maximum segment size
</h5><p dir="ltr">Force-merge has caused a lot of trouble in the past. Say you have a 50GB index and merge it down to one 50GB segment thanks to a force-merge. If your index receives updates, you are in a terrible situation: because the merge policy avoids to create segments that are larger than the maximum segment size (5GB by default) through natural merges, this segment won&apos;t be considered for merging until at least 90% of the documents that it contains are marked as deleted. This is a lot of disk overhead, and there is no easy way to recover from it.
</p><p dir="ltr">In order to avoid this issue, TieredMergePolicy, the default merge policy, has been refactored so that<a href="https://issues.apache.org/jira/browse/LUCENE-7976"> force-merges may not create segments that are larger than the maximum segment size</a>, just like natural merges. Even though it will still be discouraged to force-merge read-write indices, consequences of doing so are going to be much less annoying.
</p><h5 dir="ltr">Other
</h5><p dir="ltr"> - Can Lucene benefit from<a href="https://issues.apache.org/jira/browse/LUCENE-7745"> GPU acceleration</a>? Given the call overhead of JNI and the fact that GPUs don&apos;t like branches, our best guess is BooleanScorer (sometimes referred to as &quot;BS1&quot;), a specialized scorer for top-level disjunctions that scores windows of 2048 documents at once. Most other hot code paths in Lucene involve lots of branches or are called in tight loops, which makes them bad candidates to run on a GPU.
</p><p dir="ltr"> -<a href="https://issues.apache.org/jira/browse/LUCENE-7314"> LatLonPoint should move to core</a>. LatLonPoint is our best implementation for geo points (used by Elasticsearch since 5.x) but has remained in sandbox due to disagreement about whether it should go to lucene-spatial or lucene-core. We hope to fix this soon. This made David start a discussion about whether we should<a href="https://issues.apache.org/jira/browse/LUCENE-8369"> remove the spatial module</a>, which is nearly empty.
</p><p dir="ltr"> -<a href="https://issues.apache.org/jira/browse/LUCENE-8373"> The list of english stopwords is moving from StandardAnalyzer to EnglishAnalyzer</a>, making english a language like any other to Lucene.
</p><p dir="ltr"> -<a href="https://issues.apache.org/jira/browse/LUCENE-8374"> Sparse doc values would benefit from better indexing</a>
</p></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-06-29

<html><head></head><body><p dir="ltr"><strong><span style="font-size: 28px;">Elasticsearch<br></span></strong></p><p dir="ltr"><strong>Highlights
</strong></p><p dir="ltr"><strong>Using dots in aggregation names will be deprecated</strong></p><p dir="ltr">We will begin <a href="https://github.com/elastic/elasticsearch/pull/31468">deprecating using dots</a> in the names of aggregations. Dots are used as a separator in buckets_path, and ordering syntax aggregations with dots in their names has thrown up many bugs.  Therefore, we intend to require that all aggregation names are free of dots.  As this may affect a large number of users, we plan deprecate this ability in 6.4 but not prohibit dots in aggregation names until 8.0 to give users a enough time to modify their applications.
</p><p dir="ltr"><strong>Rollups will use the &#x201C;missing_bucket&#x201D; option from the composite aggregation</strong></p><p dir="ltr">We <a href="https://github.com/elastic/elasticsearch/pull/31402">opened a PR</a> to make rollups use the new &#x201C;missing_bucket&#x201D; on aggregations.  This closes an important limitation with rollups returning bad doc counts when you have multiple non-overlapping schemas.  One situation where this can occur is with different beats each having a different schema but residing in the same index.  With this enhancement, the user should be able to configured a single &quot;combined&quot; job which is more convenient and returns correct doc counts.
</p><p dir="ltr"><strong>Reloadable Secure Settings</strong></p><p dir="ltr">We merged and backported the work for <a href="https://github.com/elastic/elasticsearch/pull/31383">reloadable secure settings</a>. This work allows for re-reading the secure settings stored in the keystore on each node. During this reload, if a component has registered to be notified about update to these settings, it will be notified. This initial work allows for the discovery-ec2, repository-s3, repository-azure, and repository-gcs plugins to update the clients they use that depend on secure settings.
</p><p dir="ltr"><strong>Build Improvements
</strong></p><p dir="ltr">We are making good progress <a href="https://github.com/elastic/elasticsearch/issues/31230">pushing our build forward</a>.  In particular, we&#x2019;ve worked through a variety of issues in order to get our build working with Gradle 4.8 and JDK11 in a <a href="https://github.com/elastic/elasticsearch/tree/feature/31230_gradle48_jdk11">feature branch</a>.
</p><p dir="ltr"><span style="font-size: 20px;"><strong>Changelog
</strong></span></p><p dir="ltr"><strong>Changes in 5.6:
</strong></p><ul>
	<li dir="ltr">Ensure we don&#x2019;t use a remote profile if cluster name matches <a href="https://github.com/elastic/elasticsearch/pull/31331">#31331</a></li>
</ul><p dir="ltr"><strong>Changes in 6.3:
</strong></p><ul>
	<li dir="ltr">[DOCS] Omit shard failures assertion for incompatible responses <a href="https://github.com/elastic/elasticsearch/pull/31430">#31430</a></li>
	<li dir="ltr">Security: fix joining cluster with production license <a href="https://github.com/elastic/elasticsearch/pull/31341">#31341</a></li>
</ul><p dir="ltr"><strong>Changes in 6.4:
</strong></p><ul>
	<li dir="ltr">In NumberFieldType equals and hashCode, make sure that NumberType is taken into account. <a href="https://github.com/elastic/elasticsearch/pull/31514">#31514</a></li>
	<li dir="ltr">Remove QueryCachingPolicy#ALWAYS_CACHE <a href="https://github.com/elastic/elasticsearch/pull/31451">#31451</a></li>
	<li dir="ltr">Add Delete Snapshot High Level REST API <a href="https://github.com/elastic/elasticsearch/pull/31393">#31393</a></li>
	<li dir="ltr">Preserve response headers on cluster update task <a href="https://github.com/elastic/elasticsearch/pull/31421">#31421</a></li>
	<li dir="ltr">Multiplexing token filter <a href="https://github.com/elastic/elasticsearch/pull/31208">#31208</a></li>
	<li dir="ltr">Add get stored script and delete stored script to high level REST API <a href="https://github.com/elastic/elasticsearch/pull/31355">#31355</a></li>
	<li dir="ltr">Skip get_alias tests for 5.x <a href="https://github.com/elastic/elasticsearch/pull/31397">#31397</a></li>
	<li dir="ltr">Avoid sending duplicate remote failed shard requests <a href="https://github.com/elastic/elasticsearch/pull/31313">#31313</a></li>
	<li dir="ltr">Fix defaults in GeoShapeFieldMapper output <a href="https://github.com/elastic/elasticsearch/pull/31302">#31302</a></li>
	<li dir="ltr">RestAPI: Reject forcemerge requests with a body <a href="https://github.com/elastic/elasticsearch/pull/30792">#30792</a></li>
	<li dir="ltr">Use system context for cluster state update tasks <a href="https://github.com/elastic/elasticsearch/pull/31241">#31241</a></li>
	<li dir="ltr">REST high-level client: add validate query API <a href="https://github.com/elastic/elasticsearch/pull/31077">#31077</a></li>
	<li dir="ltr">Expose lucene&#x2019;s RemoveDuplicatesTokenFilter <a href="https://github.com/elastic/elasticsearch/pull/31275">#31275</a></li>
	<li dir="ltr">Add ingest-attachment support for per document indexed_chars limit <a href="https://github.com/elastic/elasticsearch/pull/31352">#31352</a></li>
</ul><p dir="ltr"><strong>Changes in 7.0:
</strong></p><ul>
	<li dir="ltr">Reload secure settings for plugins <a href="https://github.com/elastic/elasticsearch/pull/31383">#31383</a></li>
	<li dir="ltr">lower rollover-info version bound to 6.4 <a href="https://github.com/elastic/elasticsearch/pull/31414">#31414</a></li>
	<li dir="ltr">extend is-write-index serialization support to 6.4 <a href="https://github.com/elastic/elasticsearch/pull/31415">#31415</a></li>
	<li dir="ltr">Choose JVM options ergonomically <a href="https://github.com/elastic/elasticsearch/pull/30684">#30684</a></li>
	<li dir="ltr">BREAKING: Packaging: Remove windows bin files from the tar distribution <a href="https://github.com/elastic/elasticsearch/pull/30596">#30596</a></li>
	<li dir="ltr">BREAKING: Percentile/Ranks should return null instead of NaN when empty <a href="https://github.com/elastic/elasticsearch/pull/30460">#30460</a></li>
</ul><p dir="ltr"><strong><span style="font-size: 28px;">Lucene
</span></strong></p><p dir="ltr"><strong>Lucene 7.4.0
</strong></p><p dir="ltr">The vote has passed, the release will be announced shortly. Elasticsearch was already <a href="https://github.com/elastic/elasticsearch/pull/31529">upgraded</a> to this final release.
</p><p dir="ltr"><strong>Spatial code organization
</strong></p><p dir="ltr">There is an ongoing discussion regarding <a href="https://markmail.org/message/3r4hcixelkfzikmh">whether spatial code should entirely be in the lucene/spatial module, or whether the most commonly used bits should live in lucene/core</a>. The current situation is not good since what we consider the best way to index geo-points is not even in lucene/core or lucene/spatial but <a href="https://issues.apache.org/jira/browse/LUCENE-7314">in lucene/sandbox</a>.
</p><p dir="ltr"><strong>Other
</strong></p><ul>
	<li dir="ltr">ICU4J was <a href="https://issues.apache.org/jira/browse/LUCENE-8366">upgraded to version 62.1</a></li>
	<li dir="ltr">Should we make <a href="https://issues.apache.org/jira/browse/LUCENE-8367">per-dimension drill-down of facets optional</a>?</li>
	<li dir="ltr">We are exploring building <a href="https://issues.apache.org/jira/browse/LUCENE-8349">a highlighter based on the matches API</a>.</li>
	<li dir="ltr">Can we <a href="https://issues.apache.org/jira/browse/LUCENE-8362">support doc values on range fields</a>?</li>
	<li dir="ltr">An <a href="https://issues.apache.org/jira/browse/LUCENE-8365">AIOOBE in the UnifiedHighlighter</a> was due to an off-by-1 error.</li>
	<li dir="ltr">We fixed the (edge) ngram filters to <a href="https://issues.apache.org/jira/browse/LUCENE-8360">correctly set the position increment on end()</a>.</li>
	<li dir="ltr">We&#x2019;re collaborating on a <a href="https://issues.apache.org/jira/browse/LUCENE-8364">refactoring of the geo API</a>.</li>
</ul></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-06-22

<html><head></head><body><h2>Elasticsearch weekly highlights</h2><h4 dir="ltr">Rolling Upgrade to 6.3 Issue
</h4><p dir="ltr">After the release of 6.3.0, we received
  reports of nodes not re-joining their cluster after a rolling upgrade for
    clients that both have a gold/platinum license installed, and security
    enabled. Workaround is to explicitly set <code>xpack.security.enabled:
        true</code> in the elasticsearch.yml file, we are <a href="https://github.com/elastic/elasticsearch/pull/31334">adding to docs</a> as a known issue. Root cause is a change in 6.3.0 where security is disabled for trial licenses unless it is explicitly enabled; for existing gold and platinum licenses it should be enabled by default. The problem occurs when the new node starts, it does not know the current license type and defaults to a trial license, so the node does not send authentication with the ping requests, leading the nodes in the existing cluster reject the requests. <a href="https://github.com/elastic/elasticsearch/pull/31341">A fix is being worked on</a>.
</p><h4 dir="ltr">SAML - RequestedAuthnContext
</h4><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/31238">added support for RequestedAuthnContext</a>, a feature that has been requested by a number of users. This attribute allows Service Providers, in our case the Elastic Stack, to request that the user be authenticated by the Identity Provider in a certain way such as Multi Factor Authentication.
</p><h4 dir="ltr">Zen2
</h4><p dir="ltr">Basic components being in place, we have started to work on one of
the main user facing improvements in Zen2 compared to ZenDiscovery - removing
    the need for a <code>minimum_master_nodes</code> setting. Zen2 has extended the notion of cluster membership with a weight. Each node in the cluster is assigned a weight (currently 0 or 1) and only the votes of nodes with a weight count for election. For a node to become a master, it needs to win the votes for a majority of nodes with weights. When a master eligible node joins an existing cluster, it will receive a weight so it can participate in elections and support the master if other nodes fail. However, when a node leaves the cluster, the master can remove its weight if it still has a majority of the nodes to support it. Once the weight of that node is removed, it can not participate in future elections (until it rejoins). Similarly, if the cluster is partitioned, the majority side (if exists) will be able to remove the weights of all the nodes in the minority side. Once those weights are removed no node from the majority side will vote for a node in the minority side. In the same time the minority side can&apos;t form a cluster because it doesn&apos;t have enough weights based on the last know cluster configuration. This prevents a split brain from happening. The only way for the minority side to form a cluster is to rejoin the majority side and getting weights re-assigned to it.
</p><h4 dir="ltr">The scripted_metric aggregation:experimental no more!
</h4><p dir="ltr">The scripted metric aggregation was added back in 1.4.0 as a way to use scripts to provide aggregation functions that are not available in the product without having to write a custom aggregation. We marked it experimental as we weren&apos;t sure how useful it was.
</p><p dir="ltr">Since then the aggregation has remained largely unchanged but has been used by the community. Because its been in this state for so long, we don&apos;t feel we would be able to defend making large changes or removing it without providing a clear deprecation/migration path, making the experimental status lose its value.
</p><p dir="ltr">So we <a href="https://github.com/elastic/elasticsearch/pull/31298">made a change</a> in 6.4.0 that the scripted_metric aggregation will no longer be experimental. It should still be used as a last resort, but we recognize there is a long-term place for this aggregation, at least until adding a custom aggregation is a much more friendly experience and part of a supported public API. It should also be noted that we still have plans to move it to a plugin.
</p><h4 dir="ltr">NIO
</h4><p dir="ltr">We <a href="https://github.com/elastic/elasticsearch/pull/30827">added CORS support</a> to the transport-nio networking implementation. With this addition it means that transport-nio has now reached feature parity with the transport-netty4 networking implementation. Additionally, there is now benchmarking in place using transport-nio. This will enable us to closely monitor how future changes impact performance. While there is still a lot of work remaining on transport-nio, this progress marks a milestone for the transport-nio effort.
</p><h4 dir="ltr">Changes<br></h4><p dir="ltr">Changes in 5.6:
</p><ul>
	<li dir="ltr">Fix race in clear scroll<a href="https://github.com/elastic/elasticsearch/pull/31259"> #31259</a></li>
</ul><p dir="ltr">Changes in 6.3:
</p><ul>
	<li dir="ltr">Compliant SAML Response destination check<a href="https://github.com/elastic/elasticsearch/pull/31175"> #31175</a></li>
	<li dir="ltr">[Rollup] Metric config parser must use builder so validation runs<a href="https://github.com/elastic/elasticsearch/pull/31159"> #31159</a></li>
	<li dir="ltr">Security: fix token bwc with pre 6.0.0-beta2<a href="https://github.com/elastic/elasticsearch/pull/31254"> #31254</a></li>
	<li dir="ltr">Don&#x2019;t swallow exceptions on replication<a href="https://github.com/elastic/elasticsearch/pull/31179"> #31179</a></li>
</ul><p dir="ltr">Changes in 6.4:
</p><ul>
	<li dir="ltr">BREAKING (java): LLClient: Support host selection<a href="https://github.com/elastic/elasticsearch/pull/30523"> #30523</a></li>
	<li dir="ltr">CCS: don&#x2019;t proxy requests for already connected node<a href="https://github.com/elastic/elasticsearch/pull/31273"> #31273</a></li>
	<li dir="ltr">Remove RestGetAllAliasesAction<a href="https://github.com/elastic/elasticsearch/pull/31308"> #31308</a></li>
	<li dir="ltr">Add Get Aliases API to the high-level REST client<a href="https://github.com/elastic/elasticsearch/pull/28799"> #28799</a></li>
	<li dir="ltr">Treat ack timeout more like a publish timeout<a href="https://github.com/elastic/elasticsearch/pull/31303"> #31303</a></li>
	<li dir="ltr">SQL: Whitelist SQL utility class for better scripting<a href="https://github.com/elastic/elasticsearch/pull/30681"> #30681</a></li>
	<li dir="ltr">Add notion of internal index settings<a href="https://github.com/elastic/elasticsearch/pull/31286"> #31286</a></li>
	<li dir="ltr">REST high-level client: add Cluster Health API<a href="https://github.com/elastic/elasticsearch/pull/29331"> #29331</a></li>
	<li dir="ltr">HLRest: Add get index templates API<a href="https://github.com/elastic/elasticsearch/pull/31161"> #31161</a></li>
	<li dir="ltr">Ignore numeric shard count if waiting for ALL<a href="https://github.com/elastic/elasticsearch/pull/31265"> #31265</a></li>
	<li dir="ltr">Revert upgrade to Netty 4.1.25.Final<a href="https://github.com/elastic/elasticsearch/pull/31282"> #31282</a></li>
	<li dir="ltr">Support RequestedAuthnContext<a href="https://github.com/elastic/elasticsearch/pull/31238"> #31238</a></li>
	<li dir="ltr">Validate xContentType in PutWatchRequest.<a href="https://github.com/elastic/elasticsearch/pull/31088"> #31088</a></li>
	<li dir="ltr">[INGEST] Interrupt the current thread if evaluation grok expressions take too long<a href="https://github.com/elastic/elasticsearch/pull/31024"> #31024</a></li>
	<li dir="ltr">Upgrade to Netty 4.1.25.Final<a href="https://github.com/elastic/elasticsearch/pull/31232"> #31232</a></li>
	<li dir="ltr">Use stronger write-once semantics for Azure repository<a href="https://github.com/elastic/elasticsearch/pull/30437"> #30437</a></li>
	<li dir="ltr">SQL: Make a single JDBC driver jar<a href="https://github.com/elastic/elasticsearch/pull/31012"> #31012</a></li>
	<li dir="ltr">Allow to trim all ops above a certain seq# with a term lower than X<a href="https://github.com/elastic/elasticsearch/pull/30176"> #30176</a></li>
</ul><p dir="ltr">Changes in 7.0:
</p><ul>
	<li dir="ltr">BREAKING: REST hl client: cluster health to default to cluster level<a href="https://github.com/elastic/elasticsearch/pull/31268"> #31268</a></li>
	<li dir="ltr">BREAKING: REST high-level Client: remove deprecated API methods<a href="https://github.com/elastic/elasticsearch/pull/31200"> #31200</a></li>
	<li dir="ltr">Limit the number of concurrent requests per node<a href="https://github.com/elastic/elasticsearch/pull/31206"> #31206</a></li>
	<li dir="ltr">Allow to trim all ops above a certain seq# with a term lower than X, &#x2026;<a href="https://github.com/elastic/elasticsearch/pull/31211"> #31211</a></li>
	<li dir="ltr">Default max concurrent search req. numNodes * 5<a href="https://github.com/elastic/elasticsearch/pull/31171"> #31171</a></li>
</ul><h2 dir="ltr">Apache Lucene<br></h2><h4 dir="ltr">Lucene 7.4
</h4><p dir="ltr">Blockers are resolved, we expect to build a first release candidate on June 18th.
</p><h4 dir="ltr">Other
</h4><p dir="ltr"> - We had to<a href="https://issues.apache.org/jira/browse/LUCENE-8358"> relax an assertion</a> since it might not always hold due to concurrency.
</p><p dir="ltr"> - We want to<a href="https://issues.apache.org/jira/browse/LUCENE-8356"> remove StandardFilter</a> since it doesn&apos;t do anything.
</p><p dir="ltr"> - We fixed<a href="https://issues.apache.org/jira/browse/LUCENE-8357"> explanations of some function-score queries</a>.
</p><p dir="ltr"> - We fixed<a href="https://issues.apache.org/jira/browse/LUCENE-7444"> StandardAnalyzer to no longer remove english stopwords</a>, making it a reasonable default regardless of the language.
</p><p dir="ltr"> - We would like to<a href="https://issues.apache.org/jira/browse/LUCENE-7444"> remove the ability to iterate indexed fields</a>, but this can&apos;t be done at the moment due to the fact that term vectors allow some documents to enable term vectors and other documents to disable them in the same index, so this would have to be fixed first.
</p><p dir="ltr"> - We are proposing to<a href="https://issues.apache.org/jira/browse/LUCENE-7444"> improve field lookup</a> so that it runs in constant time.
</p><p dir="ltr"> -&#xA0; We noticed that AnalyzerWrapper fails to propagate calls to TokenStreamComponents.setReader, which is problematic for some analyzers.
</p><p dir="ltr"> - FrenchLightStemmer removes<a href="https://issues.apache.org/jira/browse/LUCENE-8353"> most diatritical marks</a> but not <a href="https://en.wikipedia.org/wiki/Diaeresis_(diacritic)">diaereses</a>.
</p></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-06-15

<html><head></head><body><h2 dir="ltr" id="docs-internal-guid-3c882a1b-be08-eb2c-e957-b2c34de978f8">Elasticsearch</h2><h4 dir="ltr">Cross Cluster Replication
</h4><p dir="ltr">The primary focus last week was on benchmarking, specifically getting to a point where we index full speed to an index in one region and the follower index in another region keeps up. If you remember, in last week update we reported indexing a 30GB data set and that was fully replicated but it took ~11 hours to complete. After increasing the number of workers and the batch size, we can now do a full run in 1h10m, which is roughly what it takes to index the data. From here we proceed to formalize our benchmarking infra. The plan is to introduce multiple data types, multiple platforms, nightly runs and more.
</p><p dir="ltr">On the API side, we have merged a PR to convert an existing index to a follower, and also added support for syncing the mappings between the leader and following index. The latter is important for use cases like logging where fields can be introduced during indexing. Without syncing those fields, we can&apos;t index the new data on the leader.
</p><h4 dir="ltr">Improvements in Cross Cluster Search
</h4><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/30926">merged a PR</a> that ensures we do not use dedicated master nodes as the gateway node for the remote cluster. This is an important enhancement as it ensure we do not stress remote master nodes by making them coordinate the remote part of a cross cluster search.
</p><h4 dir="ltr">Search Templates now throw 400 (bad request) on a ScriptException
</h4><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/30861">merged a PR</a> which changes the status code returned when a ScriptException is thrown from a 500 (Internal Server Error) to a 400 (Bad Request). This means that if the user has a bug in their script, we will report that request is invalid and the user should correct the request, rather than returning a 500, which would incorrectly indicate a server error.
</p><h4 dir="ltr">SQL
</h4><p dir="ltr">Work continues on the ODBC driver, which allows connectivity between BI tools like Tableau to Elasticsearch. This week, we have been working on data conversion; the date, time and timestamp conversion have been implemented, interval and GUID types are handled as well (proper rejection codes provided).<a href="https://github.com/elastic/elasticsearch-sql-odbc/issues/2"></a>
</p><h4 dir="ltr">Changes in 5.6:
</h4><ul>
	<li dir="ltr">Fsync state file before exposing it<a href="https://github.com/elastic/elasticsearch/pull/30929"> #30929</a></li>
</ul><h4 dir="ltr">Changes in 6.3:
</h4><ul>
	<li dir="ltr">[DOCS] Reset edit links<a href="https://github.com/elastic/elasticsearch/pull/30909"> #30909</a></li>
	<li dir="ltr">Do not serialize basic license exp in x-pack info<a href="https://github.com/elastic/elasticsearch/pull/30848"> #30848</a></li>
</ul><h4 dir="ltr">Changes in 6.4:
</h4><ul>
	<li dir="ltr">Harmonize include_defaults tests<a href="https://github.com/elastic/elasticsearch/pull/30700"> #30700</a></li>
	<li dir="ltr">Refactor Sniffer and make it testable<a href="https://github.com/elastic/elasticsearch/pull/29638"> #29638</a></li>
	<li dir="ltr">Deprecate accepting malformed requests in stored script API<a href="https://github.com/elastic/elasticsearch/pull/28939"> #28939</a></li>
	<li dir="ltr">REST high-level client: add synced flush API (2)<a href="https://github.com/elastic/elasticsearch/pull/30650"> #30650</a></li>
	<li dir="ltr">Reuse expiration date of trial licenses<a href="https://github.com/elastic/elasticsearch/pull/30950"> #30950</a></li>
	<li dir="ltr">Transport client: Don&#x2019;t validate node in handshake<a href="https://github.com/elastic/elasticsearch/pull/30737"> #30737</a></li>
	<li dir="ltr">HLRest: Allow caller to set per request options<a href="https://github.com/elastic/elasticsearch/pull/30490"> #30490</a></li>
	<li dir="ltr">Deprecates indexing and querying a context completion field without context<a href="https://github.com/elastic/elasticsearch/pull/30712"> #30712</a></li>
	<li dir="ltr">Cross Cluster Search: do not use dedicated masters as gateways<a href="https://github.com/elastic/elasticsearch/pull/30926"> #30926</a></li>
	<li dir="ltr">Fix AliasMetaData#fromXContent parsing<a href="https://github.com/elastic/elasticsearch/pull/30866"> #30866</a></li>
	<li dir="ltr">Add Verify Repository High Level REST API<a href="https://github.com/elastic/elasticsearch/pull/30934"> #30934</a></li>
	<li dir="ltr">BREAKING: Include size of snapshot in snapshot metadata<a href="https://github.com/elastic/elasticsearch/pull/29602"> #29602</a></li>
	<li dir="ltr">Add missing_bucket option in the composite agg<a href="https://github.com/elastic/elasticsearch/pull/29465"> #29465</a></li>
	<li dir="ltr">stable filemode for zip distributions<a href="https://github.com/elastic/elasticsearch/pull/30854"> #30854</a></li>
	<li dir="ltr">Fix IndexTemplateMetaData parsing from xContent<a href="https://github.com/elastic/elasticsearch/pull/30917"> #30917</a></li>
	<li dir="ltr">Limit the scope of BouncyCastle dependency<a href="https://github.com/elastic/elasticsearch/pull/30358"> #30358</a></li>
	<li dir="ltr">Move list tasks API under tasks namespace<a href="https://github.com/elastic/elasticsearch/pull/30906"> #30906</a></li>
	<li dir="ltr">SQL: Remove the last remaining server dependencies from jdbc<a href="https://github.com/elastic/elasticsearch/pull/30771"> #30771</a></li>
	<li dir="ltr">Verify signatures on official plugins<a href="https://github.com/elastic/elasticsearch/pull/30800"> #30800</a></li>
	<li dir="ltr">Fix bad version check writing Repository nodes<a href="https://github.com/elastic/elasticsearch/pull/30846"> #30846</a></li>
</ul><h4 dir="ltr">Changes in 7.0:
</h4><ul>
	<li dir="ltr">Remove version read/write logic in Verify Response<a href="https://github.com/elastic/elasticsearch/pull/30879"> #30879</a></li>
	<li dir="ltr">BREAKING: Core: Remove RequestBuilder from Action<a href="https://github.com/elastic/elasticsearch/pull/30966"> #30966</a></li>
	<li dir="ltr">Add &#x201C;took&#x201D; timing info to response for _msearch/template API<a href="https://github.com/elastic/elasticsearch/pull/30961"> #30961</a></li>
	<li dir="ltr">Change ScriptException status to 400 (bad request)<a href="https://github.com/elastic/elasticsearch/pull/30861"> #30861</a></li>
	<li dir="ltr">BREAKING: Include size of snapshot in snapshot metadata<a href="https://github.com/elastic/elasticsearch/pull/30890"> #30890</a></li>
	<li dir="ltr">Change BWC version for VerifyRepositoryResponse<a href="https://github.com/elastic/elasticsearch/pull/30796"> #30796</a></li>
</ul><h2 dir="ltr">Lucene</h2><h4 dir="ltr">Soft deletes
</h4><p dir="ltr">Soft deletes were initially implemented on top of the Lucene index, but recent changes have tightened the integration with IndexWriter. We&apos;ve started discussions about<a href="https://issues.apache.org/jira/browse/LUCENE-8335"> preventing from changing the field that is used to record soft deletes</a> and<a href="https://issues.apache.org/jira/browse/LUCENE-8341"> recording the number of soft deletes into segment infos</a>.
</p><h4 dir="ltr">Other
</h4><ul>
	<li dir="ltr">Fixing UAX29URLEmailTokenizer, which was<a href="https://issues.apache.org/jira/browse/LUCENE-8278"> not detecting urls</a> for top-level domains for which the one-letter-shorter prefix was also a TLD.</li>
	<li dir="ltr">Proposal of a<a href="https://issues.apache.org/jira/browse/LUCENE-8331"> merge simulator</a>, whose goal is to help assess the efficiency of a merge policy and possibly help tune parameters.</li>
	<li dir="ltr">Could Lucene <a href="https://issues.apache.org/jira/browse/LUCENE-8340">efficiently boost based on dynamic features</a> such as recency or geo distance?</li>
	<li dir="ltr">We&apos;ve added an &quot;<a href="https://issues.apache.org/jira/browse/LUCENE-8300">unordered distinct</a>&quot; intervals source that allows to force matches to occur at distinct positions, so that if you search for &quot;a NEAR a&quot; then a single occurrence of &quot;a&quot; won&apos;t be considered a match.</li>
	<li dir="ltr">We made<a href="https://issues.apache.org/jira/browse/LUCENE-8334"> SegmentReader.getSegmentInfo return a snapshot</a>.</li>
	<li dir="ltr">Work on&#xA0;<a href="https://issues.apache.org/jira/browse/LUCENE-8332"> exposing the CompletionTokenStream as a more general-purpose ConcatenateGraphTokenStream</a> that concatenates terms for all paths in the token graph.</li>
	<li dir="ltr"> Explored using<a href="https://issues.apache.org/jira/browse/LUCENE-4892"> a more compact representation of live docs</a> but it also made query processing slower.</li>
	<li dir="ltr"><a href="https://issues.apache.org/jira/browse/LUCENE-8144"> cleaned up the ALWAYS_CACHE caching policy</a>, that should only be used for testing.</li>
</ul></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-06-01

In 1.4.0 Elasticsearch gained a shard level &lsquo;Request Cache&rsquo; which caches the result of the query phase on each shard keyed on the search request its

<html><head></head><body><p><em>This is the final post in a three-part series about Instant Aggregations. Read how it all started in </em><a href="https://www.elastic.co/blog/the-tale-of-caching-and-why-it-matters">The Tale of Caching and Why It Matters</a><em> from Simon Willnauer and Shay Banon and the meaty middle detailed in <a href="https://www.elastic.co/blog/the-great-query-refactoring-thou-shalt-only-parse-once"></a></em><a href="https://www.elastic.co/blog/the-great-query-refactoring-thou-shalt-only-parse-once">The Great Query Refactoring: Thou Shalt Only Parse Once</a><em>. Enjoy the trilogy!</em><br>
</p><p>In 1.4.0 Elasticsearch gained a shard level &#x2018;Request Cache&#x2019; which caches the result of the query phase on each shard keyed on the search request itself. Until 5.0 this feature was disabled by default since it wasn&#x2019;t as useful as it could be for most of the computationally heavy use cases. The cache was intended to make searches faster, especially for aggregations. One problem was that ordering in JSON is not deterministic so although two requests may be logically the same, when rendered to JSON strings they may not be equal. The other main problem is that these computationally heavy use cases tend to use time windows relative to the current time so subsequent requests will tend to have slightly different time ranges. Enabling it would likely be a waste memory for most of the users &#xA0;since they would rarely ever get a cache hit. So why would we add such a feature?
</p><p>Here at Elastics, we try to follow the rule of &#x201C;progress over perfection&#x201D;. Even if we can&#x2019;t utilize it&#x2019;s full potential we get a step closer to our goals. Now after 2 years of engineering effort and <a href="/blog/the-great-query-refactoring-thou-shalt-only-parse-once">refactoring the entire search request</a> representation, we can finally take full advantage of it.
</p><p>From 5.0 the request cache will be enabled by default for all requests with `size:0`. The request cache is most useful for analytics use cases and generally analytics use cases use search requests with `size: 0`.
</p><p>So how did the changes made since 1.4.0 enable us to better use this feature?
</p><p>The search request cache gives massive improvements in search performance when running the same search multiple times on a static index. However, search requests are rarely exactly the same, especially in the time series use case.
</p><p>Lets look at some of the characteristics of typical search for time-series use cases:
</p><ul>
	<li>
	Queries have a &#xA0;time range component often relative to current time (e.g. from now-7d to now)
	</li>
	<li>
	Search requests span &#xA0;across multiple indices
	</li>
	<li>
	Indexing happens on the latest index, all other indices are static
	</li>
	<li>
	Same search requests are run multiple times with different time ranges
	</li>
</ul><p>The above are common characteristics of search requests for a lot of time series use cases including common patterns of usage for Kibana. A lot of Kibana dashboards contain quite a few visualisations a lot of which are complex and can take a number of seconds for the dashboard request to be executed by Elasticsearch. Because search requests are rarely the same, the search request cache can&#x2019;t be used effectively to improve search performance for these use cases.
</p><p>Or can it?
</p><p>How can we make the search request cache work when the time range is always moving? Let&#x2019;s solve this by considering an example. Imagine we have the details for every residential house sale in the UK since 1995 (luckily this information is conveniently <a href="https://data.gov.uk/dataset/land-registry-monthly-price-paid-data">made available</a>). We can index this into 21 yearly indices (e.g. house-prices-1995, house-prices-1996, &#x2026;, house-prices-2016). For the purposes of this explanation let&#x2019;s make each index only have 1 shard (though the same idea can easily be extended to multiple shard indices without modification) Now we can run queries against the indices by using requests like this:
</p><pre>```
GET house-prices-*/_search
{
  &quot;query&quot;: {
    &quot;range&quot;: {
      &quot;date&quot;: {
        &quot;gte&quot;: &quot;2013-09-01&quot;,
        &quot;lt&quot;: &quot;2016-03-01&quot;
      }
    }
  }
}
```
</pre><p>The figure below shows three of these such queries overlaid on the yearly indices. These queries are typical for dashboard style use cases in that they differ only by small amounts (imagine a refreshing dashboard)
</p><p><img src="https://api.contentstack.io/v2/assets/57d6e0bf39cce73c4a091058/download?uid=blt1a2b92ef4097f1ff" height="325" width="850">
</p><p>A couple of things stand out from this diagram:
</p><ul>
	<li>
	All three queries match no documents in house-prices-2012 (highlighted red on the diagram)
	</li>
	<li>
	All three queries will match all documents in house-prices-2014 and house-prices-2015 (highlighted green on the diagram)
	</li>
</ul><p>So the only indices which affect the matching documents between the three queries are house-prices-2013 and house-prices-2016.
</p><p>If we could rewrite the queries on each shard based on the range of values present in that index we could make the queries able to actually utilize the request cache. Below is the diagram with the same three queries rewritten to make them more cachable in the search request cache:
</p><p><img src="https://api.contentstack.io/v2/assets/57d6e1d039cce73c4a0910ad/download?uid=blte0f8b865a8d465c8" height="325" width="850">
</p><p>You can see that in the diagram above, the first query is rewritten and run as a `match_none` query on the `house-prices-2012` indexes shard and a `match_all` query on the `house-prices-2014` and `house-prices-2015` indices shards. This means that when the second and third query is run it can use the cached results of the search on the `house-prices-2012`, `house-prices-2014` and `house-prices-2015` indices shards instead of actually running a search on those shards. So for the second and third queries we only need to actually search the shards on the `house-prices-2013` and `house-prices-2016` indices.
</p><p>Also, even for the first query, running a `match_none` query instead of a range query will be faster since it will not need to try to lookup the date range in the index.
</p><p>So how does this work in practice?
</p><p>This feature not only relies on the search request cache mentioned at the beginning of this post, but it is also heavily based on the query refactoring that we described in a recent <a href="/blog/the-great-query-refactoring-thou-shalt-only-parse-once">blog post</a>. The reason for this is that in order to add the rewrite logic described above we need objects that represent the query and the search request that we can actually rewrite.
</p><p>When the search request is received by the shard, we rewrite the entire request including `query` and `post_filter` QueryBuilder objects. Each type of query has it&#x2019;s own implementation of QueryBuilder which will rewrite according to it&#x2019;s own rules.
</p><p>Compound queries, queries that themselves contain queries (such as `bool` and `constant_score`) will call rewrite on the queries they wrap. Most leaf queries, queries that do not contain other queries, currently do not contain any rewrite logic so just return themselves (to indicate they did not change).
</p><p>The range query however, will check the minimum and maximum value of the field (we will call this the field range) and compare this to the range it contains. There are three cases that we care about when evaluating the rewrite for the range:
</p><ul>
	<li>
	The query range and the field range do not overlap at all - In this case we can rewrite this range as a `match_none` query so we return a `MatchNoneQueryBuilder`
	</li>
	<li>
	The query range contains the entire field range - In this case all documents which contain a value for this field will match the range so we can rewrite the range query as an unbounded range query (not a match_all query) and return it
	</li>
	<li>
	The query range partially overlaps the field range. There&#x2019;s not much we can do to help in this case so we don&#x2019;t rewrite and return the original range query.
	</li>
</ul><p>Now that the search request has been rewritten we can check the cache with the rewritten version and either retrieve the cached result or execute the search and add the result to the cache, keyed on the rewritten search request. This has a nice side effect since the rewritten queries are &#x201C;normalized&#x201D; search requests that are semantically equivalent but differ in the order of keys in the json will now also produce cache hits.
</p><p>Why not use `match_all` ?
</p><p>In the case that the query range contains the entire field range you might think we could rewrite the `range` query as a `match_all` query. We can&#x2019;t do this because we can&#x2019;t assume that all documents have a value for the field. If we rewrite to a `match_all` query we would incorrectly match documents that have no value for the field. For this reason we instead rewrite the range to an unbounded range query (effectively [* TO *]) which still means the query is much more useful to the search request cache.
</p><p>Is this all worth the trouble?
</p><p>To answer this question we&#x2019;ll go back to our yearly house price indices. This dataset contains 21 yearly indices containing 21,304,688 residential UK house sales from 1995 to 2016. I ran this fairly unscientific test on my Macbook Pro. Each index has a single shard. I created a Kibana dashboard showing 14 different visualisations relevant to the data mixing simple and complex visualisations. The top of the dashboard looks like the following:
</p><p><img src="https://api.contentstack.io/v2/assets/57e02db9acc34b3254b8af0b/download?uid=bltf740e6e1aec8ca2b" height="469" width="850">
</p><p>With the request cache disabled, requests to elasticsearch for the dashboard to refresh takes 12s-14s. When the request cache is enabled this drops to ~100ms. The same request is now 100x faster!
</p><p>But this is on completely static data so what happens when we are indexing data?
</p><p>To test this I deleted the data from 2010 to 2016 and loaded it in date order whilst refreshing the dashboard:
</p><p><img src="https://api.contentstack.io/v2/assets/57d6e20f39cce73c4a0910cb/download?uid=blt7d49d71a4510bee5" height="502" width="850">
</p><p>Now the request to Elasticsearch for the dashboard refresh takes 60-200ms. So the query is still at least 50x faster than our original un-cached query! The refresh time varies depending on how much data is present in the changing index since the search on the latest index does not hit the cache. This is because as new data is indexed, the cache on the shards of the latest index is constantly being invalidated.
</p><p>The Instant Aggregations feature enables much better caching of search requests by rewriting the query based on the data present on the shard. As of 5.0.0 the query is only rewritten for date range queries, but with this infrastructure in place it opens the door to the potential of rewriting other types of query to improve the request cache utilization. We even have the potential to rewrite aggregations to make them more cacheable! We have shown in this post that the potential performance improvements when we use this technique can be huge, so watch this space for more improvements in the future.
</p></body></html>

instant-aggregations-elasticsearch-colin-goodheart-smithe-request-cache-full-bleed.jpg

instant-aggregations-elasticsearch-rewriting-queries-thumbnail.jpeg

Instant Aggregations: Rewriting Queries for Fun and Profit

<html><head></head><body><p>
	One of the most visible features coming in 2.0 are the Pipeline Aggregations. This is an extension of the current Aggregations framework, to allow additional computations to be performed on top of the results of aggregations. This allows users to ask questions such as &#x201C;What is the maximum average monthly price?&#x201D; from a date histogram and &#x201C;How many new users are signing up each day?&#x201D; from a date histogram showing total user count each day.
</p><p>
	Internally these aggregation are run after the other aggregations have completed on the coordinating node. This means Pipeline Aggregators are able to use the final results of their sibling aggregations but are not able to go back to the shards to query the index. The first of these new types are:
</p><ul>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline-derivative-aggregation.html">Derivative Aggregation</a></li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline-max-bucket-aggregation.html">Max Bucket Aggregation</a></li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline-min-bucket-aggregation.html">Min Bucket Aggregation</a></li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline-avg-bucket-aggregation.html">Average Bucket Aggregation</a></li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline-sum-bucket-aggregation.html">Sum Bucket Aggregation</a></li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline-bucket-script-aggregation.html">Bucket Script Aggregation</a></li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline-cumulative-sum-aggregation.html">Cumulative Sum Aggregation</a></li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline-movavg-aggregation.html">Moving Average Aggregation</a></li>
</ul><p>
	In this post I will show some of these new aggregations in action. To help demonstrate, I have downloaded spacecraft trajectory data from <a href="http://omniweb.gsfc.nasa.gov/coho/helios/heli.html" target="_blank">NASA&#x2019;s Helioweb site</a> for the Voyager 1 and Voyager 2 spacecraft. Each document in my index represents the <a href="https://en.wikipedia.org/wiki/Ecliptic_coordinate_system" target="_blank">Solar Ecliptic</a> position of one of the spacecraft on a particular day. The data spans from the launch of the Voyager missions in September 1977 to the present day and also includes projected trajectories up to 2020. Below is an example of a document in Elasticsearch:
</p><pre class="prettyprint">{
    &quot;_index&quot;: &quot;helioweb&quot;,
    &quot;_type&quot;: &quot;voyager1&quot;,
    &quot;_id&quot;: &quot;1977-257&quot;,
    &quot;_score&quot;: 1,
    &quot;_source&quot;: {
        &quot;seLon&quot;: 353.8,
        &quot;objectName&quot;: &quot;voyager1&quot;,
        &quot;seLat&quot;: 0.2,
        &quot;year&quot;: 1977,
        &quot;date&quot;: &quot;1977-09-14T10:28:25.178Z&quot;,
        &quot;radAU&quot;: 1.02,
        &quot;dayOfYear&quot;: 257
    }
}
</pre><p>
	<code>seLat</code> and
	<code>seLon</code> are the latitude and longitude of the object in the 
	<a href="https://en.wikipedia.org/wiki/Ecliptic_coordinate_system" target="_blank">Solar Ecliptic Coordinate System</a> respectively, and
	<code>radAU</code>
	is the radial distance of the object from the Sun measured in <a href="https://en.wikipedia.org/wiki/Astronomical_unit" target="_blank">Astronomical Units</a>. In this post we will mainly be concentrating on the <code>date</code>,
	<code>objectName</code>, and 
	<code>radAU</code> fields.
</p><p>
	So first let&#x2019;s plot the average radial distance of Voyager 1 over time. This can be done with the existing aggregations by defining the following aggregations:
</p><pre class="prettyprint">{
  ...
  &quot;aggs&quot;: {
    &quot;histo&quot;: {
      &quot;date_histogram&quot;: {
        &quot;field&quot;: &quot;date&quot;,
        &quot;interval&quot;: &quot;month&quot;
      },
      &quot;aggs&quot;: {
        &quot;avg_distance&quot;: {
          &quot;avg&quot;: {
            &quot;field&quot;: &quot;radAU&quot;
          }
        }
      }
    }
  }
}
</pre><p>
	The graph we get from the results looks pretty uninteresting:
</p><p><img class="img-responsive" src="https://api.contentstack.io/v2/assets/575e4c83d8edd48f76935bf6/download?uid=blt5913aeb0da25eab5" data-sys-asset-uid="blt5913aeb0da25eab5" alt="voyager-1-radial-distance.png">
</p><p>
	Basically all we can find out from this is that since launch Voyager 1 has been travelling further and further away from the Sun. Not exactly surprising for a long distance space probe!
</p><p>
	But what does Voyager 1&#x2019;s radial speed over time look like? This isn&#x2019;t something you can currently do in Elasticsearch. The indexed documents only contain the radial distance of Voyager 1 so its speed cannot be plotted directly. With the new Derivative Aggregation we can take the derivative of that distance to calculate the radial speed each month:
</p><pre class="prettyprint">{
  ...
  &quot;aggs&quot;: {
    &quot;histo&quot;: {
      &quot;date_histogram&quot;: {
        &quot;field&quot;: &quot;date&quot;,
        &quot;interval&quot;: &quot;month&quot;
      },
      &quot;aggs&quot;: {
        &quot;avg_distance&quot;: {
          &quot;avg&quot;: {
            &quot;field&quot;: &quot;radAU&quot;
          }
        },
        &quot;speed&quot;: {
          &quot;derivative&quot;: {
            &quot;buckets_path&quot;: &quot;avg_distance&quot;
          }
        }
      }
    }
  }
}
</pre><p>
	Pipeline aggregations look very similar to the other aggregations in the request, but instead of having a <code>field</code> parameter to reference which 
	<code>field</code> in the document they will work on, they have a 
	<a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline.html#bucket-path-syntax">
	<code>bucket_path</code>
	parameter</a> which references which aggregation to use in their calculation. The resulting aggregation tree for the above request looks like this:
</p><pre class="prettyprint">{
   ...
   &quot;aggregations&quot;: {
      &quot;histo&quot;: {
         &quot;buckets&quot;: [
            {
               &quot;key_as_string&quot;: &quot;1977-09-01T00:00:00.000Z&quot;,
               &quot;key&quot;: 241920000000,
               &quot;doc_count&quot;: 25,
               &quot;avg_distance&quot;: {
                  &quot;value&quot;: 1.0328
               }
            },
            {
               &quot;key_as_string&quot;: &quot;1977-10-01T00:00:00.000Z&quot;,
               &quot;key&quot;: 244512000000,
               &quot;doc_count&quot;: 31,
               &quot;avg_distance&quot;: {
                  &quot;value&quot;: 1.1867741935483873
               },
               &quot;speed&quot;: {
                  &quot;value&quot;: 0.15397419354838737
               }
            },
            &#x2026;
         ]
      }
   }
}
</pre><p>
	You can see that the Derivative Aggregation has added a new value <code>speed</code> to the all but the first bucket. The first bucket is left empty as we require two values to compute the derivative, the current value and the previous value. Since there is no previous value for the first bucket, we can&#x2019;t compute a derivative for it.
</p><p>
	Plotting the speed against time gives us a much more interesting graph:
</p><p><img class="img-responsive" src="https://api.contentstack.io/v2/assets/575e4c840342dfd738262f75/download?uid=blted382364b4175616" data-sys-asset-uid="blted382364b4175616" alt="voyager-1-radial-speed.png">
</p><p>
	Here we can see that Voyager 1&#x2019;s radial speed dropped significantly twice, once around 1979 and once around 1981. Looking at the <a href="http://voyager.jpl.nasa.gov/mission/timeline.html" target="_blank">timeline for the Voyager mission</a> we can see that these points coincide with the closest approach of Voyager 1 with Jupiter and Saturn.
</p><p>
	We can actually use the Min Bucket Aggregation to determine which month Voyager 1 was travelling at its slowest radial speed using the following:
</p><pre class="prettyprint">{
  ...
  &quot;aggs&quot;: {
    &quot;histo&quot;: {
      &quot;date_histogram&quot;: {
        &quot;field&quot;: &quot;date&quot;,
        &quot;interval&quot;: &quot;month&quot;
      },
      &quot;aggs&quot;: {
        &quot;avg_distance&quot;: {
          &quot;avg&quot;: {
            &quot;field&quot;: &quot;radAU&quot;
          }
        },
        &quot;speed&quot;: {
          &quot;derivative&quot;: {
            &quot;buckets_path&quot;: &quot;avg_distance&quot;
          }
        }
      }
    },
    &quot;min-speed&quot;: {
      &quot;min_bucket&quot;: {
        &quot;buckets_path&quot;: &quot;histo &gt; speed&quot;
      }
    }
  }
}
</pre><p>
	Which returns a new aggregation alongside the histogram showing the minimum value for the slowest radial speed and the bucket keys of the bucket which have this minimum value:
</p><pre class="prettyprint">{
   ...
   &quot;aggregations&quot;: {
      &quot;histo&quot;: {
         &quot;buckets&quot;: [
            ...
         ]
      },
      &quot;min-speed&quot;: {
         &quot;value&quot;: 0.10212903225806436,
         &quot;keys&quot;: [
            &quot;1979-04-01T00:00:00.000Z&quot;
         ]
      }
   }
}
</pre><p>
	So the slowest radial speed was in April 1979 which matches the Voyager timeline as the closest approach to Jupiter was during March 1979. Previously this would have required the client application to parse all the buckets, keeping track of the minimum value.
</p><p>
	We can show the radial speed of both Voyager 1 and Voyager 2 in the same response by using the filter aggregation to show the average radial distance of Voyager 1 and Voyager 2 and then use two derivative aggregations, one referencing the Voyager 1 distance and the other referencing the Voyager 2 distance:
</p><pre class="prettyprint">{
  &quot;aggs&quot;: {
    &quot;histo&quot;: {
      &quot;date_histogram&quot;: {
        &quot;field&quot;: &quot;date&quot;,
        &quot;interval&quot;: &quot;month&quot;
      },
      &quot;aggs&quot;: {
        &quot;voyager_1&quot;: {
          &quot;filter&quot;: {
            &quot;term&quot;: {
              &quot;objectName&quot;: &quot;voyager1&quot;
            }
          },
          &quot;aggs&quot;: {
            &quot;avg_distance&quot;: {
              &quot;avg&quot;: {
                &quot;field&quot;: &quot;radAU&quot;
              }
            }
          }
        },
        &quot;voyager_2&quot;: {
          &quot;filter&quot;: {
            &quot;term&quot;: {
              &quot;objectName&quot;: &quot;voyager2&quot;
            }
          },
          &quot;aggs&quot;: {
            &quot;avg_distance&quot;: {
              &quot;avg&quot;: {
                &quot;field&quot;: &quot;radAU&quot;
              }
            }
          }
        },
        &quot;voyager_1_speed&quot;: {
          &quot;derivative&quot;: {
            &quot;buckets_path&quot;: &quot;voyager_1&gt;avg_distance&quot;
          }
        },
        &quot;voyager_2_speed&quot;: {
          &quot;derivative&quot;: {
            &quot;buckets_path&quot;: &quot;voyager_2&gt;avg_distance&quot;
          }
        }
      }
    }
  }
}
</pre><p>
	Now, the <code>buckets_path</code> references the 
	<code>avg_distance</code>
	metric inside each of the filter aggregations using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline.html#bucket-path-syntax">the 
	<code>buckets_path</code>
	syntax</a>. We can now plot the radial speed of Voyager 1 (blue line) and Voyager 2 (red line):
</p><p><img class="img-responsive" src="https://api.contentstack.io/v2/assets/575e4c840342dfd738262f6d/download?uid=bltd0cbe9bc1e7ab546" data-sys-asset-uid="bltd0cbe9bc1e7ab546" alt="voyager-1-and-2-radial-speed.png">
</p><p>
	Again from looking at the <a href="http://voyager.jpl.nasa.gov/mission/timeline.html" target="_blank">timeline for the Voyager mission</a> we can see that the drops in radial speed of Voyager 2 coincide with its closest encounters with Jupiter, Saturn, Uranus and Neptune.
</p><p>
	Since pipeline aggregations happen right at the end of the aggregation phase, they can be chained together (hence the name pipeline). So to calculate the radial acceleration of the Voyager 1 spacecraft we can just calculate the derivative of the speed (which we calculated in the previous pipeline aggregation) by adding the following aggregation to the request:
</p><pre class="prettyprint">{
    ...
       &quot;voyager_1_acceleration&quot;: {
          &quot;derivative&quot;: {
            &quot;buckets_path&quot;: &quot;voyager_1_speed&quot;
          }
        }
    &#x2026;
}
</pre><p>
	Here, instead of referencing a metric aggregation we are referencing the derivative aggregation <code>voyager_1_speed</code>. 
This will add an aggregation called 
	<code>voyager_1_acceleration</code> to the response for all but the first 2 buckets (since we need two values for
	<code>voyager_1_speed</code> to calculate the acceleration).
</p><p>
	These are just the first pipeline aggregations. We are currently working to add more pipeline aggregations to <a href="https://github.com/elastic/elasticsearch/issues/9876" target="_blank">this list</a> and there is a growing list of other ideas for possible pipeline aggregators including one that can be used to filter out buckets which do not meet a specified criteria (using this aggregation you could, for example, ask for only histogram buckets where the average speed is greater than a threshold value). We would also appreciate any feedback you have on the current pipeline aggregations and your suggestions for other pipeline aggregations that we could add in the future.
</p><p>
	Happy exploring!
</p></body></html>

Articles by Colin Goodheart-Smithe