<h2 dir="ltr">Elasticsearch Highlights</h2><h4 dir="ltr">Cloud Identity Provider
</h4><p dir="ltr">We are continuing to work on the Identity Provider (IdP) to enable Single-Sign-On between our Cloud UI (console) and deployed applications, e.g. Kibana. We’re getting close to have something that we can merge into master and 7.x, we have recently added:
</p><ul>
	<li dir="ltr">Storing Service Providers as <a href="https://github.com/elastic/elasticsearch/pull/52401">documents in an ES index</a> in the security cluster (with automatic index creation & <a href="https://github.com/elastic/elasticsearch/pull/52706">template setup</a>)</li>
	<li dir="ltr">An <a href="https://github.com/elastic/elasticsearch/pull/52936">API to add new Service Providers</a> to that index</li>
	<li dir="ltr"><a href="https://github.com/elastic/elasticsearch/pull/52985">Looking up a Service Provider</a> from that index during SSO</li>
	<li dir="ltr">Parsing and validating <a href="https://github.com/elastic/elasticsearch/pull/52018">SP initiated authentication</a> requests</li>
	<li dir="ltr"><a href="https://github.com/elastic/elasticsearch/pull/52415">Creating IdP metadata</a></li>
	<li dir="ltr">Processing <a href="https://github.com/elastic/elasticsearch/pull/52805">secondary authentication (end user) credentials</a> during SSO</li>
</ul><h4 dir="ltr">Searchable snapshots
</h4><p dir="ltr">We continue to work on searchable snapshots, discussing use cases, the technical building blocks, the features to be built out of this, and how it fits into the broader Elastic ecosystem. The initial deliverables are focused on providing a new cheaper warm tier on our hosted Cloud service, while future phases of the project are geared towards a new cold tier with near unlimited scaling capabilities.
</p><p dir="ltr">On the development front, We have <a href="https://github.com/elastic/elasticsearch/pull/51866">added timing information</a> to the stats we collect regarding retrieving data from the blob store when searching a snapshot, and has also added support for <a href="https://github.com/elastic/elasticsearch/pull/52653">frozen searchable snapshots</a>. By limiting the number of searchable snapshot shards that can be searched concurrently we expect to deal more gracefully with datasets whose total size exceeds the size of the cache.
</p><p dir="ltr">We are adding a dedicated API to <a href="https://github.com/elastic/elasticsearch/pull/53084">mount an index from a snapshot into a cluster</a>. This API, which takes repository, snapshot, and index name as input allows to effortlessly make an index in a snapshot available for search, and will serve as a basis to implement ILM actions that transition regular indices to snapshot-powered indices.
</p><p dir="ltr">We have been benchmarking our current searchable snapshots implementation using different datasets (pmc, nyc_taxis, geonames) over different challenges and with various cache size settings. With this, we were able to confirm that the execution of search queries on snapshot-backed indices has similar performance to regular indices, as long as the cache is large enough to fully cache the shard data. We also investigated how the feature performs when only a part of the data can fit into the cache. Here, the results vary more based on a number of factors, and we're looking at ways to improve our current implementation. We also measured raw S3 download performance by running full-restore benchmarks, and identified potential optimizations to make searchable snapshot restores faster (more parallelization; smaller downloads on checksum verifications; async fetch of larger files) as well as fixed a <a href="https://github.com/elastic/elasticsearch/pull/52781">number</a> <a href="https://github.com/elastic/elasticsearch/pull/52770">of</a> <a href="https://github.com/elastic/elasticsearch/pull/52760">bugs</a> in the process.
</p><h4 dir="ltr">Create API Key on behalf of another user
</h4><p dir="ltr">We have an open PR to that would allow one user (typically a system user) to <a href="https://github.com/elastic/elasticsearch/pull/52886">create an API key on behalf of another user</a> (typically an end-user).
</p><p dir="ltr">The API requires credentials for the end-user (a username+password, or an access token), but means that the authorization check is performed against the system-user rather than the end-user.
</p><p dir="ltr">We intentionally do not allow users to create API keys by default. For example, we want to support situations where a cluster administrator requires that users must authenticate via SAML using a second factor device. If those users were allowed to create API keys, that would give them an alternative means of authentication to the cluster that does not use the SAML identity provider and by-passes the intended security controls.
</p><p dir="ltr">However, API Keys have uses within Elastic products for storing a credential that allows a background process to act on behalf of a user. The primary example for this is the Kibana Alerting project. An alert that interacts with Elasticsearch needs to be able to authenticate to Elasticsearch as the user that created the alert, but should not (and may not be able to) store the user’s password. Because alerting uses that API internally and never exposes it to the user, the risk profile is different; it does not give the user a way to by-pass the main authentication process.
</p><p dir="ltr">So it’s OK to let Kibana create API keys for users that are not permitted to create API keys of their own. But, we do not want Kibana to be able to create API keys for any user it chooses - if it could do that, then it would be able to create an API key for the elastic user, and gain superuser privileges. So, this new API allows Kibana to create API keys for users who would not necessarily be able to do that on their own, but requires that they first authenticate to Kibana (so Kibana is effectively restricted to only create API keys for users who are logged in at the time).
</p><h4 dir="ltr">Console bugfixes
</h4><p dir="ltr">We fixed a <a href="https://github.com/elastic/kibana/pull/59143">regression in Console’s proxy</a> that would always overwrite the “host” header forwarded from Kibana. This regression was introduced in 7.5.0 and the fix ships with 7.6.2.
</p><p dir="ltr">We also squashed a few other bugs in Console like <a href="https://github.com/elastic/kibana/pull/58587">respecting loading of autocomplete items</a> selected in settings and restoring <a href="https://github.com/elastic/kibana/pull/59043">autocomplete for the index settings endpoint</a>.
</p><h4 dir="ltr">geo_line aggregation
</h4><p dir="ltr">We have recently been revisiting his <a href="https://github.com/elastic/elasticsearch/issues/41649">geo_line aggregation prototype</a> (<a href="https://github.com/talevy/elasticsearch/commit/82c8f2300c1f2e8403e0a85794b48f8ef8e903ca">WIP commit here</a>), looking at how to best utilize our existing allocation-efficient data structures. GeoLine is an agg that can stitch together "consecutive" points to create a single line shape.  Think of a cargo vessel with a beacon that reports GPS coordinates once per hour.  If you stitch those individual datapoints together, you get the spatio-temporal path the ship has taken.
</p><p dir="ltr">We migrated a prototype to use primitive arrays within Object BigArrays that grow with the help of Lucene's array utils. Sorting of these arrays is done with the help of Lucene's IntroSorter. Need to sync with the Lucene folks to understand if that is the best sorter for the job, it is being chosen for now because it is the simplest.
</p><h4 dir="ltr">Ingest: Performance and Retries
</h4><p dir="ltr">We continue work on the <a href="https://github.com/elastic/elasticsearch/pull/52669">POC</a> to batch shard operations together. Performance testing exhibited how poorly this interacts with the existing <a href="https://github.com/elastic/elasticsearch/issues/51336">request-count-limited queues</a>, for which we are investigating solutions. Benchmarking also shows the effectiveness of batching fsyncs together when there are very high levels of concurrency.
</p><p dir="ltr">We worked with a community contributor on <a href="https://github.com/elastic/elasticsearch/pull/50166">changing the response code when indexing is blocked due to disk space usage exceeding the flood stage</a>, now indicating that this condition is retryable, as the index block will automatically be released once enough disk space is available again. This fixes an <a href="https://github.com/elastic/elasticsearch/issues/49393">issue where our ingest clients were not retrying requests in this situation</a> based on the erroneous response status code.
</p><h2 dir="ltr">Apache Lucene Highlights</h2><h4 dir="ltr">Lucene 8.5</h4><p>We have created Lucene 8.5 branch in preparation for the next release. We have <a href="https://github.com/elastic/elasticsearch/pull/53150" target="_blank">updated Elasticsearch</a> repository with a new snapshot from this branch but unfortunately we had to revert this change as there has introduced some concurrency issue in the IndexWriter. We are working with the community to address the issue.</p><h4>Merge on commit</h4><p>						When using concurrent indexing, IndexWriter will accumulate many small segments which when writing to disk will add search-time cost as searching must visit all those tiny segments. The<a href="https://issues.apache.org/jira/browse/LUCENE-8962" target="_blank"> idea of this change</a> is to collect those small changes and merge them during commit, creating a more efficient index layout.</p><h4>Geo</h4><p>							The LatLonPoint implementation used by Elasticsearch projects geo data into the plane. What it means is that when you define a polygon, the edges of the polygon follows <a href="https://en.wikipedia.org/wiki/Cartesian_coordinate_system" target="_blank">cartesian geometry</a>. </p><p>Geo3DPoint is another Lucene's implementation that projects geo data into an ellipsoid. In this case, edges of a polygon are defined using <a href="https://en.wikipedia.org/wiki/Spherical_geometry" target="_blank">spherical geometry,</a> and those edges are in fact <a href="https://en.wikipedia.org/wiki/Geodesics_on_an_ellipsoid" target="_blank">geodesics of the ellipsoid</a>.  Until now, this implementation could only be used with the WGS84 ellipsoid. We <a href="https://issues.apache.org/jira/browse/LUCENE-9150" target="_blank">have changed</a> it so it can take any ellipsoid.
							</p><h2 dir="ltr">Breaking Changes</h2><h4 dir="ltr"> Elasticsearch
</h4><p dir="ltr">Breaking Changes in 8.0:
</p><ul>
	<li dir="ltr">Percentiles aggregation: disallow specifying same percentile values twice <a href="https://github.com/elastic/elasticsearch/pull/52257">#52257</a></li>
	<li dir="ltr">Remove fixed_auto_queue_size threadpool type <a href="https://github.com/elastic/elasticsearch/pull/52280">#52280</a></li>
	<li dir="ltr">Remove translog retention settings <a href="https://github.com/elastic/elasticsearch/pull/51697">#51697</a></li>
</ul><p dir="ltr">Breaking Changes in 7.7:
</p><ul>
	<li dir="ltr">Add validation for dynamic templates <a href="https://github.com/elastic/elasticsearch/pull/51233">#51233</a></li>
	<li dir="ltr">Make range query rounding consistent <a href="https://github.com/elastic/elasticsearch/pull/50237">#50237</a></li>
</ul><h4 dir="ltr">Breaking Changes in Elasticsearch Hadoop Plugin</h4><p dir="ltr">Breaking Changes in 8.0:
</p><ul>
	<li dir="ltr">Boost default scroll size to 1000 <a href="https://github.com/elastic/elasticsearch-hadoop/pull/1429">#1429</a></li>
</ul><h4 dir="ltr">Breaking Changes in Rally</h4><p dir="ltr">Breaking Changes in 1.5.0:
</p><ul>
	<li dir="ltr">Require at least Python 3.6 <a href="https://github.com/elastic/rally/pull/905">#905</a></li>
</ul><p dir="ltr">Breaking Changes in 1.4.1:
</p><ul>
	<li dir="ltr">Use zeros instead of whitespaces as padding bytes <a href="https://github.com/elastic/rally/pull/899">#899</a></li>
</ul>

This Week in Elasticsearch and Apache Lucene - 2020-03-06

Version 2

<h2>Elasticsearch
</h2><h4 dir="ltr">Slow (completion) stats
</h4><p dir="ltr">We looked into a couple of cases of struggling clusters and saw that the hot threads dominated by computing Engine#completionStats . There's a bit of a pattern here, looking back over past issues: there are some overenthusiastic third-party monitoring systems that hit GET _stats or GET _cluster/stats far too frequently, and these stats can be expensive to compute. The unbounded queue in the management threadpool doesn't help matters either. We opened a PR to suggest <a href="https://github.com/elastic/elasticsearch/pull/51991">streamlining these stats</a> in particular, and an issue to discuss whether we should <a href="https://github.com/elastic/elasticsearch/issues/51992">introduce backpressure for stats requests</a>.
</p><h4 dir="ltr">Security User Ingest Processor
</h4><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/51305">updated the Set Security User Ingest Processor to support additional authentication information</a>.
</p><p dir="ltr">This processor adds a new field to a document to record information about the user that ingested it. It has existed for a long time, but it was of limited use because it was not possible to enforce the use of a specific pipeline or processor, so a malicious user could just write their own fake “user” object into a document and then skip the set security user processor to make it look like the document was ingested by someone else.
</p><p dir="ltr">Elasticsearch 7.5.0 added support for enforced pipelines, which makes the Set Security User Processor much more useful as a true record regarding the origin of documents.
</p><p dir="ltr">We’ve taken the opportunity to give that processor a little bit more love, and it now supports adding fields related to:
</p><ul>
	<li dir="ltr">The realm from which the user was resolved/authenticated</li>
	<li dir="ltr">The type of authentication that was used (realm, bearer token, api key, anonymous)</li>
	<li dir="ltr">The name & ID of the API key that was used, if any.</li>
</ul><p dir="ltr">The last point was the key driver for scheduling this work now. There is work underway to make heavier use of API keys within beats, and recording API key information can tell us exactly which beat, running on which host ingested a particular document, even if all of those API keys are owned by the same logical user.
</p><h4 dir="ltr">Analytics
</h4><p dir="ltr">We opened a PR to add a <a href="https://github.com/elastic/elasticsearch/pull/51948">new boxplot aggregation</a>. A <a href="https://en.wikipedia.org/wiki/Box_plot">boxplot</a> is a fairly common descriptive statistic used to visualize numerical quantities.  It consists of a vertical box, where the upper bound of the box is the 75th percentile, the lower bound is the 25th percentile and a line somewhere inside for the median (50th percentile).  Two "whiskers" extend from the box to touch the min/max.  This visualization gives a huge amount of information in a compact form, and is often used in financial analysis for that reason.  The boxplot agg will live in our licensed Analytics plugin.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4704c68e18fa5cca/5fa9517e4a4abb73ff79c4ab/blog-this-week-2020-02-07.png" data-sys-asset-uid="blt4704c68e18fa5cca" alt="blog-this-week-2020-02-07.png"></p><h4 dir="ltr">Sorted Queries
</h4><p dir="ltr"><a href="https://www.elastic.co/fr/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand?blade=tw&hulk=social">In Elasticsearch 7.0</a> we improved queries sorted by relevance (score), by a large margin, <a href="https://www.elastic.co/fr/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand?blade=tw&hulk=social">for users that don't need to know the exact number of documents that match the query</a>. This new infrastructure built in Lucene specifically for queries sorted by score has been used since then to also improve queries sorted by a field. We pushed a <a href="https://www.elastic.co/fr/blog/this-week-in-elasticsearch-and-apache-lucene-2019-12-02">change</a> in Elasticsearch to optimize queries sorted by a date or long field that proved very efficient on time-based indices. This change takes advantage of the almost-sorted property of a shard built with time-based events to efficiently filter blocks of documents or even entire segments when looking for the oldest or newest documents (sorted by timestamp).
</p><p dir="ltr">However, this optimization only works at the shard level so if you have an index pattern that spans years of data, Elasticsearch still needs to search every shard to return the oldest/newest documents. This issue can be mitigated by adding a range filter that restricts the search to the last N days then weeks, months but that's also the kind of expert usage that we'd like to make obsolete. Search should be fast “out of the box” if you're looking after the oldest/newest documents that match a query in an index pattern.
</p><p dir="ltr">Following, we opened a <a href="https://github.com/elastic/elasticsearch/pull/51852">PR</a> to filter entire shards on sorted queries over time-based indices, which should make timestamp-based sorted queries on time-based indices cheap and fast even for a very large number of shards.
</p><p dir="ltr">This change won't affect queries that run aggregations (users won't see any speed up). While the majority of our users run aggregation, we're also building a lot of new features under the assumption that sorted queries are fast. The current work on EQL for instance relies on sorted queries to implement efficient joins.
</p><p dir="ltr">We are now looking at improving queries that use <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.5/search-request-body.html#request-body-search-search-after">search_after</a>  in order to allow users to iterate quickly over a stream of sorted documents (beyond the first page).
</p><h4 dir="ltr">Benchmarking Highly Concurrent Ingest Performance
</h4><p dir="ltr">Security has been looking into ingest performance with high concurrency and small doc size and as we are investigating possible Elasticsearch improvements, we turned to the performance team for standardized reproducible benchmarks. This turned into a parallelized effort with an initial solution using the<a href="https://locust.io/"> Locust</a> highly concurrent load testing framework and then starting to work on a technical spike of<a href="https://github.com/elastic/rally/issues/852"> a new Rally load generator component based on async I/O</a> to support vastly higher client counts, while we improved the initial Locust script making it a more stable and flexible solution.
</p><h4 dir="ltr">Breaking changes:
</h4><p dir="ltr">No breaking changes to report this week!
</p><h2 dir="ltr">Apache Lucene
</h2><h4 dir="ltr">Off-heap stored fields index
</h4><p dir="ltr">Lucene's stored fields are a document store. Elasticsearch uses it in order to store the _id, _source and _routing values when specified, as well as any user-configured fields that have <code>store: true</code>. This document store has two components: a data file that stores fields for all documents sequentially, and an index file that enables Lucene to know where stored fields for a given document start in the data file. Until now the content of this file was loaded in memory. It's not much memory, if you look at the <a href="https://elasticsearch-benchmarks.elastic.co/index.html#tracks/geonames/nightly/default/30d">nightly benchmarks</a>, geonames only spends 3.8MB on this stored fields index for 11M documents. But it's also a pity that Lucene spends more memory on stored fields than it does on BKD trees or doc values, which are typically more performance-sensitive. The next version of Lucene will have the index read directly from disk.
</p><h4 dir="ltr">Simplification of Geo API
</h4><p dir="ltr">Graduating LatLonShape and XYShape  <a href="https://github.com/apache/lucene-solr/pull/1174">from lucene sandbox to core</a> has opened the possibility to simplify the current Lucene geo API. This <a href="https://github.com/apache/lucene-solr/pull/1170">refactoring effort</a> provides just one query implementation for all queryable geometries. As a side effect, it allows the implementation of queries with multiple geometries more efficiently. In many cases, those queries need to be rewritten as a boolean query, each query representing one of the geometries. With this new API, it would be possible to generate it as a single query.
</p>

This Week in Elasticsearch and Apache Lucene - 2020-02-07

This Week in Elasticsearch and Apache Lucene - 2020-01-31

<h2>Elasticsearch Highlights</h2><h4 dir="ltr">Index Lifecycle Management
</h4><p dir="ltr">The ILM Rollover Action is now fully retryable. This is one of our top issues, and a big win for <a href="https://github.com/elastic/elasticsearch/issues/44135">resiliency</a>.
</p><p dir="ltr">We have extracted a new step as part of the ILM RolloverAction that will wait for the active shards of the newly created index to become available, as opposed to possibly failing the entire rollover step if this wait would timeout or <a href="https://github.com/elastic/elasticsearch/pull/50718">unable to be satisfied</a>. We also made the update settings ILM step <a href="https://github.com/elastic/elasticsearch/pull/51235">retryable</a> and with this, we closed the issue of<a href="https://github.com/elastic/elasticsearch/issues/44135"> adding automatic retries</a> to the <code>RolloverAction</code>, as now, all the steps are retry-able.
</p><h4 dir="ltr">Snapshot resilience and BWC
</h4><p dir="ltr">In 7.6+ we’re changing the way information is stored in a snapshot repository, so that we are resilient when snapshotting to eventually consistent blob stores such as S3. In previous versions, there was a risk of repositories becoming corrupted if a cluster was doing multiple snapshot operations in quick succession, as an eventually consistent repository would not allow the cluster to see the correct base state to update (and would have it incorrectly override stuff, corrupting snapshots in the worst case).
</p><p dir="ltr">We still need to support a scenario where a cluster is, for example, upgraded from 7.5 to 7.6, and if something is wrong with the cluster in 7.6, allow a snapshot that was taken of the cluster in 7.5 to be restored to a new 7.5 cluster (i.e. reverting to an older state of the cluster). However, the new 7.6+ repository format is fundamentally different, not allowing older Elasticsearch versions to read from the repository. As a solution for this use case, we made it so that Elasticsearch 7.6+ will only write in the new 7.6+ repository format if all snapshots in the repository are on 7.6+. This means that as long as 7.5 snapshots are in the repository, ES 7.6+ will still write to the repository in the old way, allowing a 7.5 snapshot to be restored to a 7.5+ cluster.
</p><p dir="ltr">This, in turn, means again that 7.6+ clusters are still at the risk of corrupting snapshots in their repository if they have older snapshots in the repository as the new resilient format can’t be used to write to the repository during that time. So, just as we thought that we had things under control, a new challenge came up. 7.6+ clusters can be converted to make use of SLM, which does not have a cool-down period, and can do back-to-back snapshots. A 7.6+ cluster could now corrupt a repository using SLM if it still had older snapshots in the repo. After evaluating various solutions, we opted to <a href="https://github.com/elastic/elasticsearch/pull/51074">introduce a cool-down period</a> into ES proper. This cool-down period only applies to S3-based repositories (other blob stores provide much higher consistency guarantees), and only in the case where there are older snapshots in the repository (i.e. where we have to write to the repository in the pre-7.6 format). This means that S3 snapshots take a little longer because of the cool-down period, but repository corruption should be avoided by ES 7.6+ clusters, independently of the format in which they write.
</p><p dir="ltr">This work concludes the efforts on making snapshots resilient and safe on all blobstores we support and currently there are no known snapshot resiliency bugs left that we know of. During the snapshot resiliency sync we confirmed this fact with Cloud as well. The next step in this project is to enhance snapshot repositories to allow for concurrent operations. The first step in this is <a href="https://github.com/elastic/elasticsearch/pull/51256">simplifying the state machine</a> for snapshot creation and deletion in a way compatible with concurrent snapshot operations.
</p><h4 dir="ltr">TLS and Authentication Examples for the High Level Rest Client
</h4><p dir="ltr">In the days of Transport client, we provided a special version of the client that included the client-side plugin for X-Pack security. This meant that same sorts of TLS configuration that worked on X-Pack security on the server were also available in the client. With the move to the High Level Rest Client, we have intentionally not copied all of that same support into the new client, but sometimes users struggled to work out how to configure the Rest Clients to connect to secured clusters. SSL in Java is versatile and robust, but it can be hard to get your head around the APIs if you’re not familiar with them.
</p><p dir="ltr">We had some discussions about this last year, and decided that the best step we could take right now was to expand the rest client docs with clearer examples of configuring the High and Low Level Rest Clients to connect to a variety of different TLS configurations, and also added examples for authenticating with Tokens and API Keys.
</p><p dir="ltr"><a href="https://github.com/elastic/elasticsearch/pull/51355">Improved docs</a> will be coming to a website near you.
</p><h2 dir="ltr">Apache Lucene
</h2><h4 dir="ltr">Geo improvements
</h4><p dir="ltr">We opened a PR to <a href="https://github.com/apache/lucene-solr/pull/1182">increase the data dimension limit for BKD</a> in order to support indexing 3D triangles.  There has been a hard limit of 8 dimensions for indexing points since the BKD tree was first introduced, but work since then has allowed this information to be split into ‘indexing’ dimensions, used to built the tree, and ‘data’ dimensions, which are stored in the leaves.  This allows us some freedom to increase the number of dimensions stored while preserving index size and performance.  This improvement will enable us to encode higher dimension data within a lower dimension index (eg 3D tessellated triangles as a 10 dimension point using only the first 6 dimensions for index construction).
</p><p dir="ltr">We also merged some long-standing open issues around the organisation of spatial code within Lucene, moving LatLongShape and XYShape <a href="https://github.com/apache/lucene-solr/pull/1174">from lucene sandbox to core</a> and removing the long obsolete (and mostly empty) <a href="https://github.com/apache/lucene-solr/pull/1173">spatial module</a>.
</p><h4 dir="ltr">Testing
</h4><p dir="ltr">Now that the parallel gradle build is on master, there have been a  number of changes to improve the performance of the lucene test suite.  In particular, adding java options <a href="https://issues.apache.org/jira/projects/LUCENE/issues/LUCENE-9160">to turn off background hotspot compilation</a> has cut execution time massively (on my MBP running the full lucene test suite has gone from 18 minutes to 6 minutes)
</p><h4 dir="ltr">ANN search
</h4><p dir="ltr">The community is continuing to iterate on some implementations of approximate-nearest-neighbour search, in particular <a href="https://issues.apache.org/jira/projects/LUCENE/issues/LUCENE-9004">Hierarchical Navigable Small-Worlds</a> and <a href="https://issues.apache.org/jira/projects/LUCENE/issues/LUCENE-9136">IVFFlat</a>   We’re doing our own investigations into building ANN search on top of existing lucene index structures, and it’s fascinating to follow these various different strands of research all feeding back on each other.  Open development in action!
</p>

This Week in Elasticsearch and Apache Lucene - 2020-01-25

This Week in Elasticsearch and Apache Lucene - 2020-01-17

<h2>Elasticsearch Highlights</h2><h4 dir="ltr">Rally
</h4><p dir="ltr">This week we merged <a href="https://github.com/elastic/rally/pull/830">elastic/rally#830</a>, which adds a completely new approach to how Elasticsearch clusters are managed in Rally. Up to this point, Rally was only able to spin up uniform clusters, i.e. each cluster node is configured identically but this hinders automating more complex benchmark setups. This PR is a complete game changer and allows users to spin up arbitrarily complex cluster architectures (hot-warm, hot-warm-cold, with or without dedicated master nodes, etc.) for any Elasticsearch version between 2.0 and today’s latest commit on master. This new functionality is also achieved without relying on Rally’s internal actor system that will eventually be replaced to reduce complexity and improve stability. It is already available on the development version of Rally and will be the highlight of the upcoming 1.4.0 release. More <a href="https://esrally.readthedocs.io/en/latest/cluster_management.html">examples</a> are available in the documentation.
</p><h4 dir="ltr">Snapshots
</h4><p dir="ltr">We merged a <a href="https://github.com/elastic/elasticsearch/pull/49729">significant change to how we handle repository metadata</a>. It uses the cluster state for keeping track of the blob in which the latest valid repository metadata can be found, allowing us to work around S3's eventually consistent nature. With this change, the eventual consistency will no longer make us accidentally overwrite, and thus corrupt in some cases, the repository metadata.
</p><p dir="ltr">There is one follow-up change left in this effort that will make use of the new information in the cluster state to make a number of repository operations more efficient. The efficiency gains come from not having to list out the repository contents to find the latest version of its metadata and being able to remove a number of fallbacks that we used to work around issues that were resolved automatically by keeping a pointer to the latest metadata in the cluster state.
</p><h4 dir="ltr">Analytics
</h4><p dir="ltr">A new histogram field type <a href="https://github.com/elastic/elasticsearch/pull/48580">was merged</a> on master and 7.x at the end of last week! This field type will allow us to do things like <a href="https://github.com/elastic/beats/issues/14843">store Promethus histograms</a>.
</p><h4 dir="ltr">Geo
</h4><p dir="ltr">After a couple of weeks of getting the EdgeTree into a state that we can properly evaluate the differences in performance of <a href="https://github.com/elastic/elasticsearch/pull/47903">TriangleTree</a> and EdgeTree, it seems most benchmarks suggest it would benefit our aggregations to be backed by the TriangleTree instead of the GeometryTree. This goes against our consensus hypothesis when first starting deciding which tree to implement. Thanks to our determinism to demonstrate that it is likely our consensus is misguided, we had the luxury of actually testing both strategies against one another.
</p><p dir="ltr">To help demonstrate which types of queries (in this case, tile queries) favor each tree, we<a href="https://gist.github.com/talevy/f06ef43be1e97afb1ee53f25b980a4a0"> visualized a precision=9 tiling of Switzerland and mapped the cost of each query</a>. One major failure of the EdgeTree is for queries of tiles that lie just outside (and east of) the shape, but within the shapes bounding box. This results in the EdgeTree traversing O(n) edges. The EdgeTree tends to be slightly faster overall for most queries, but when it is slower than TriangleTree it is much slower which might outweigh its other advantages.
</p><p dir="ltr"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbb317a17ba68c3d8/5f8a0d981252ff30caf6288e/blog-this-week-elastic-2019-12-06-1.jpeg" width="624" height="352">
</p><h4 dir="ltr">Monitoring
</h4><p dir="ltr">We are discussing the migration strategy for 7.x to 8.0 with respect to internal collectors and metricbeat collectors. We decided to decouple the shape changes of the monitoring documents from the upgrade to reduce the number of moving parts and to allow monitoring to continue to work across versions. We also decided that metricbeat collection should not be a requirement prior to upgrade and will need some adjustments to the internal collectors to ensure a smoother upgrade path.
</p><h4 dir="ltr">Snapshot Lifecycle Management in Cloud
</h4><p dir="ltr">We merged a <a href="https://github.com/elastic/kibana/pull/51000">PR</a> that adds Cloud support to SLM. These changes include adequate protections for Cloud-managed policies.
</p><p dir="ltr"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt10b18dee716d9e5b/5f8a0d96dcf0e74b7cbf6f21/blog-this-week-elastic-2019-12-06-2.png" width="556" height="726" style="width: 556px; height: 726px;">
</p><h4 dir="ltr">TLS
</h4><p dir="ltr">We have opened a <a href="https://github.com/elastic/elasticsearch/pull/49827">PR for a new “elasticsearch-certutil http" utility.</a>
</p><p dir="ltr">This sub-command provides a guided process for creating SSL certs for the Elasticsearch HTTP (REST) interface.
</p><p dir="ltr">As always, It Depends, but most customers are best served by using different certificates for transport and http, because the needs & usage of those interfaces are different.
</p><p dir="ltr">For transport you want to use certificates to lock down your cluster. Typically that means running a custom Certificate Authority for the cluster so that the nodes trust one another, and no one else.
</p><p dir="ltr">However, for HTTP you generally want to support access from a variety of clients, in a variety of languages that have their own builtin trusted CAs (or they use the Operating System’s CA list). Ideally you want to use a corporate CA, or if you don’t have one, a single CA for all of your ES clusters so that your clients can be configured once and then be able to connect to all of your clusters. You also need a copy of that CA in formats that are suitable for each client (For most clients that’s PEM, but JVM based clients will typically find PKCS#12 more helpful), and instructions for how to configure those clients.
</p><p dir="ltr">Over time, we will evolve our instructions to recommend using "certutil cert” for transport level certificates, and “certutil http” for http level certificates.
</p><h2></h2><h2>Apache Lucene Highlights</h2><p><strong>Lucene 8.4:&nbsp;</strong>A branch should be cut&nbsp;<a href="https://lists.apache.org/thread.html/d3d63475fc529897000c0374adefef34a0007dbb01a304acfde1d98d%40%3Cdev.lucene.apache.org%3E" target="_blank" data-saferedirecturl="https://www.google.com/url?q=https://lists.apache.org/thread.html/d3d63475fc529897000c0374adefef34a0007dbb01a304acfde1d98d%2540%253Cdev.lucene.apache.org%253E&source=gmail&ust=1575983152871000&usg=AFQjCNF9-J2g6L_51wMkujNoFI-Q-r45yw">later this week</a>, and the first RC will be built shortly afterwards.
</p><ul>
	<li>Efforts have been restarted to&nbsp;<a href="https://issues.apache.org/jira/browse/LUCENE-9077" target="_blank" data-saferedirecturl="https://www.google.com/url?q=https://issues.apache.org/jira/browse/LUCENE-9077&source=gmail&ust=1575983152871000&usg=AFQjCNHEdroobEEXxCth9fwgelmaQmeJgA">move the build to Gradle</a>.</li>
	<li>There is&nbsp;<a href="https://issues.apache.org/jira/browse/LUCENE-9086" target="_blank" data-saferedirecturl="https://www.google.com/url?q=https://issues.apache.org/jira/browse/LUCENE-9086&source=gmail&ust=1575983152871000&usg=AFQjCNH776fPkH3EfUILrnBhJQ5r4D1TPw">benchmarking the new AWS Graviton instances</a>&nbsp;against existing comparable instances for indexing and search loads.Discussion around&nbsp;<a href="https://issues.apache.org/jira/browse/LUCENE-9078" target="_blank" data-saferedirecturl="https://www.google.com/url?q=https://issues.apache.org/jira/browse/LUCENE-9078&source=gmail&ust=1575983152871000&usg=AFQjCNE3OWlz9o0NDIMOfqvjAfxx9PQVCg">disallow configuring term vectors on a per-document basis</a>. This is something that is already enforced by Elasticsearch on top of Lucene.&nbsp;</li>
	<li>A few weeks ago, we&nbsp;<a href="https://issues.apache.org/jira/browse/LUCENE-9049" target="_blank" data-saferedirecturl="https://www.google.com/url?q=https://issues.apache.org/jira/browse/LUCENE-9049&source=gmail&ust=1575983152871000&usg=AFQjCNGA6eXDEQQqsujr0o-NIxAegDlfxg">removed the root cache for the FST</a>&nbsp;that serves as an index of the terms dictionary. However it looks like&nbsp;<a href="https://issues.apache.org/jira/browse/LUCENE-9064" target="_blank" data-saferedirecturl="https://www.google.com/url?q=https://issues.apache.org/jira/browse/LUCENE-9064&source=gmail&ust=1575983152871000&usg=AFQjCNGZLtgcFYN4Un3EEGqEiFHLyJAH2g">removing similar caches for the Kuromoji and Nori analyzers hurts</a>, so we will keep those.</li>
	<li>There is exploration around&nbsp;<a href="https://issues.apache.org/jira/browse/LUCENE-9074" target="_blank" data-saferedirecturl="https://www.google.com/url?q=https://issues.apache.org/jira/browse/LUCENE-9074&source=gmail&ust=1575983152871000&usg=AFQjCNHiesj8_2zqQCm1L5ZPykiqZWkvcw">configuring parallelism of search requests based on the current state of the threadpool</a>: the larger the queue, the fewer threads will be used to search concurrently.</li>
</ul><h4>Changes in Elasticsearch</h4><p>Changes in 8.0:
</p><ul>
	<li>Silence lint warnings in server project - part 2&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49728">#49728</a></li>
	<li>Migrate some of the Docker tests from old repository&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49079">#49079</a></li>
	<li>Add healthchecks to distro docker-compose.yml&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49710">#49710</a></li>
</ul><p>Changes in 7.6:
</p><ul>
	<li>Consistent case in CLI option descriptions&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49635">#49635</a></li>
	<li>Fix task input for docker build&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49814">#49814</a></li>
	<li>Use Cluster State to Track Repository Generation&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49729">#49729</a></li>
	<li>Add reusable HistogramValue object&nbsp;&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49799">#49799</a></li>
	<li>Fix invalid break iterator highlighting on keyword field&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49566">#49566</a></li>
	<li>Fixes a bug in interval filter serialization&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49793">#49793</a></li>
	<li>[Transform] automatic deletion of old checkpoints&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49496">#49496</a></li>
	<li>Scripting: add available languages & contexts API&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49652">#49652</a></li>
	<li>Replicate write actions before fsyncing them&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49746">#49746</a></li>
</ul><p>Changes in 7.5:
</p><ul>
	<li>Fix external integ test zip dep to expect a zip&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49813">#49813</a></li>
	<li>Extend systemd timeout during startup&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49784">#49784</a></li>
	<li>[Transform] Fix possible audit logging disappearance after rolling upgrade&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49731">#49731</a></li>
	<li>SQL: fix LOCATE function optional parameter handling&nbsp;&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49666">#49666</a></li>
	<li>SQL: fix NULL handling for FLOOR and CEIL functions&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49644">#49644</a></li>
	<li>SQL: handle NULL arithmetic operations with INTERVALs&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49633">#49633</a></li>
</ul><p>Changes in 6.8:
</p><ul>
	<li>Support es7 node http publish_address format&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/49279">#49279</a></li>
</ul><h4>Changes in Rally</h4><p>Changes in 1.4.0:
</p><ul>
	<li>Expose API for cluster settings&nbsp;<a href="https://github.com/elastic/rally/pull/831">#831</a></li>
	<li>Manage Elasticsearch nodes with dedicated subcommands&nbsp;<a href="https://github.com/elastic/rally/pull/830">#830</a></li>
	<li>Only keep the most recent build log&nbsp;<a href="https://github.com/elastic/rally/pull/832">#832</a></li>
</ul>

This Week in Elasticsearch and Apache Lucene - 2019-12-06

elasticsearch-weekly-blog-banner.png

elasticsearch-weekly-blog-thumb.png

This Week in Elasticsearch and Apache Lucene - 2019-12-02

<h2>Elasticsearch Highlights</h2><h4>Forging ahead with the Mappings Editor
</h4><p dir="ltr">We have given the core of the mappings editor some great UI/UX improvements. The core is responsible for adding, editing and removing fields to the document. For each field a "name" and a  "type" can be specified. The core is also responsible for managing "nested" fields and "multi-fields". These changes are living in the Mappings Editor feature branch at the moment <a href="https://github.com/elastic/kibana/pull/47562">#47562</a>
</p><p dir="ltr">Next up we are working on the parameters form for each type, starting with "text", "keyword" and “numeric".
</p><p dir="ltr">Here’s the UI</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltcf0c32915a41c2eb/5db6f95bdc674266df3d73f8/pasted_image_0.png" data-sys-asset-uid="bltcf0c32915a41c2eb" alt="pasted_image_0.png"></p><h4 dir="ltr">Snapshot resilience
</h4><p dir="ltr">We merged a <a href="https://github.com/elastic/elasticsearch/pull/46250">big change to the way shard-level snapshot metadata is handled</a>. The master now keeps track of the state of each shard's metadata, which eliminates an issue where a data node could corrupt the repository or part of it as data nodes now only write new blobs (with unique ids) but never update or delete any blobs from the repository. Another effect of this change is that data nodes don't have to list out each shard-level directory to find the latest metadata generation as they used to do. This improves resiliency on S3 (<a href="https://github.com/elastic/elasticsearch/issues/38941">due to it's eventually consistent nature</a>) and reduces the cost of snapshotting an unchanged shard directory by 33% (<a href="https://github.com/elastic/elasticsearch/issues/45736">we want to bring the cost for that down to zero in a follow-up</a>). Since snapshotting unchanged shards can be a significant contributor to snapshot costs, for users that use rolling indices this change should visibly reduce their snapshot cost.
</p><h4 dir="ltr">Vector Fields: path to GA
</h4><p dir="ltr">Today we have two types for vectors, dense_vector, which has a value for all dimensions, and sparse_vector, that keeps only the non-null dimensions. While sparse_vector could be useful in the future, we don't have a good enough sense of use cases/requirements around the current implementation to take it to GA, and we also don’t want to keep it as experimental, It would be a bit confusing if dense_vector were GA, but not sparse_vector. We haven’t seen any usage of <code>sparse_vector</code>, so we opened a <a href="https://github.com/elastic/elasticsearch/pull/48315">PR</a> to deprecate them in 7.6 and removing it in 8.0.
</p><h4 dir="ltr">Diagnostic Trust Manager
</h4><p dir="ltr">We are always working on making security configuration issues easier to diagnose and triage. Tim V has been working on a Diagnostic Trust Manager; the idea here is that this manager wraps another Trust Manager, <code>catch</code>es any failures, and provides logging around what went wrong. This work is early days, but here’s an example log message:
</p><pre dir="ltr">[2019-10-24T21:13:21,223][WARN ][o.e.c.s.DiagnosticTrustManager] [testDiagnosticTrustManagerForHostnameVerificationFailure] failed to establish trust with [SERVER] at [127.0.0.1]; the server provided a certificate with subject name [CN=not-this-host] and fingerprint [b46447f4e16b15a4770a8423884cb5afd31ad81f]; the certificate has subject alternative names [DNS:not.this.host]; the certificate is issued by [CN=Certificate Authority 1,OU=ssl-error-message-test,DC=elastic,DC=co] but the server did not provide a copy of the issuing certificate; the issuing certificate is trusted in this context with fingerprint [2d5980e0c909cfc9ab4a77ad11e1f7b951ee7e13]
</pre><h4 dir="ltr">Geo
</h4><p dir="ltr">We are proposing a n<a href="https://github.com/elastic/elasticsearch/pull/47903">ew data structure for geo_shape doc values</a>. Current approach is based on how we represent query data shapes by representing polygons and lines as an edge tree. The new approach uses the same tessellation strategy used for indexing shapes on the BKD tree where shapes are broken in triangles, lines and points. This approach might be more suitable in order to perform complex spatial operations with shapes but it might have a penalty on the size of those doc values. For example, one of the performance optimizations we can do with doc values is to use the query IndexOrDocValuesQuery that Lucene provides. In order to do that we should be able to compute the spatial relationship (disjoint, within, intersects) between the query shape and the doc value. This computation might be easier to perform with the approach based on geometry tessellation.
</p><h2 dir="ltr">Apache Lucene
</h2><h4 dir="ltr">Off-heap BKD index
</h4><p dir="ltr">Our <a href="https://issues.apache.org/jira/browse/LUCENE-8932">change</a> to move the BKD index off-heap has been merged. In practice, the index will only be loaded off-heap when the index input is an instance of ByteBufferIndexInput (used by MmapDirectory) since other index inputs like the ones created by SimpleFSDirectory on NIOFSDirectory don't support efficient random-access. In order to leverage this change in Elasticsearch, we would need to <a href="https://github.com/elastic/elasticsearch/issues/48509">change our default <code>hybrid</code> directory to open BKD trees with mmap</a>.
</p><h4 dir="ltr">SIMD-based decoding of postings
</h4><p dir="ltr">For space-efficiency reasons, postings don't record doc IDs as 32-bit integers. Instead, doc IDs are <a href="https://www.elastic.co/blog/frame-of-reference-and-roaring-bitmaps">delta-encoded to make them smaller, grouped into blocks of 128 deltas and then encoded on just as many bits are required</a>. Our current decoding logic is a bit naive and only processes one doc ID at a time. Yet it is possible to use <a href="https://en.wikipedia.org/wiki/SIMD">SIMD</a> instructions to decode multiple values at once and make decoding more efficient. Java doesn't expose the ability to use SIMD instructions directly, but its C2 compiler is able to recognize some access patterns that can be vectorized. It doesn't give the same flexibility as one would have when writing C code, but it is still possible to exploit these patterns that the JVM recognizes to make decoding faster. We have a <a href="https://issues.apache.org/jira/browse/LUCENE-9027">prototype</a> that decodes 128 packed integers from a ByteBuffer about 2x faster than the current implementation, which translates to a 0-10% speedup on the queries that we run in our nightly benchmarks. This is unfortunately proving challenging to integrate API-wise as it requires the addition of new APIs to DataInput to more efficiently read longs.
</p><h4 dir="ltr">Other
</h4><ul><li>We should be able to make doc-value terms queries more efficient by <a href="https://issues.apache.org/jira/browse/LUCENE-9025">optimizing the lookup logic</a> for the case that it is called over a sorted sequence of terms.</li><li>We have an open <a href="https://github.com/apache/lucene-solr/pull/966">pull request</a> to implement selection via the <a href="https://en.wikipedia.org/wiki/Median_of_medians">median of medians</a> algorithm.</li><li>We <a href="https://issues.apache.org/jira/browse/LUCENE-9022">disabled caching on GlobalOrdinalsWithScoreQuery</a>, since it is a memory-intensive query that doesn't make sense to cache.</li></ul> <h2 dir="ltr">Changes in Elasticsearch</h2><h4 dir="ltr">Changes in 8.0:
</h4><ul>
	<li dir="ltr">Update docker-compose.yml to fix bootstrap check error <a href="https://github.com/elastic/elasticsearch/pull/47650">#47650</a></li>
	<li dir="ltr">Remove type filter from GetMappings API <a href="https://github.com/elastic/elasticsearch/pull/47364">#47364</a></li>
	<li dir="ltr">Update docker.asciidoc <a href="https://github.com/elastic/elasticsearch/pull/47651">#47651</a></li>
</ul><h4 dir="ltr">Changes in 7.6:
</h4><ul>
	<li dir="ltr">Change grok watch dog to be Matcher based instead of thread based. <a href="https://github.com/elastic/elasticsearch/pull/48346">#48346</a></li>
	<li dir="ltr">Add a deprecation warning regarding allocation awareness in search request <a href="https://github.com/elastic/elasticsearch/pull/48351">#48351</a></li>
	<li dir="ltr">Correct rewritting of script_score query <a href="https://github.com/elastic/elasticsearch/pull/48425">#48425</a></li>
	<li dir="ltr">Do not throw errors on unknown types in SearchAfterBuilder <a href="https://github.com/elastic/elasticsearch/pull/48147">#48147</a></li>
	<li dir="ltr">Geo: improve handling of out of bounds points in linestrings <a href="https://github.com/elastic/elasticsearch/pull/47939">#47939</a></li>
	<li dir="ltr">Deprecate the sparse_vector field type. <a href="https://github.com/elastic/elasticsearch/pull/48315">#48315</a></li>
	<li dir="ltr">Track Shard-Snapshot Index Generation at Repository Root  <a href="https://github.com/elastic/elasticsearch/pull/46250">#46250</a></li>
	<li dir="ltr">[Java.time] Calculate week of a year with ISO rules <a href="https://github.com/elastic/elasticsearch/pull/48209">#48209</a></li>
	<li dir="ltr">Use MultiFileTransfer in CCR remote recovery <a href="https://github.com/elastic/elasticsearch/pull/44514">#44514</a></li>
	<li dir="ltr">Don't build packages on non Linux <a href="https://github.com/elastic/elasticsearch/pull/48246">#48246</a></li>
	<li dir="ltr">update ingest-user-agent regexes.yml <a href="https://github.com/elastic/elasticsearch/pull/47807">#47807</a></li>
	<li dir="ltr">Quieter logging from the DiskThresholdMonitor <a href="https://github.com/elastic/elasticsearch/pull/48115">#48115</a></li>
</ul><h4 dir="ltr">Changes in 7.5:
</h4><ul>
	<li dir="ltr">Add a packagingTask for every os project <a href="https://github.com/elastic/elasticsearch/pull/48400">#48400</a></li>
	<li dir="ltr">[Transform] do not fail checkpoint creation due to global checkpoint mismatch <a href="https://github.com/elastic/elasticsearch/pull/48423">#48423</a></li>
	<li dir="ltr">Add populate_user_metadata in OIDC realm <a href="https://github.com/elastic/elasticsearch/pull/48357">#48357</a></li>
	<li dir="ltr">Ensure SLM stats does not block an in-place upgrade from 7.4 <a href="https://github.com/elastic/elasticsearch/pull/48367">#48367</a></li>
	<li dir="ltr">Read build and runtime java from properties file <a href="https://github.com/elastic/elasticsearch/pull/48355">#48355</a></li>
	<li dir="ltr">SearchSlowLog uses a non thread-safe object to escape json <a href="https://github.com/elastic/elasticsearch/pull/48363">#48363</a></li>
	<li dir="ltr">Add 'javadoc' task to CI lifecycle check tasks <a href="https://github.com/elastic/elasticsearch/pull/48214">#48214</a></li>
	<li dir="ltr">[Docs] Fix testing docs regarding --debug-jvm <a href="https://github.com/elastic/elasticsearch/pull/48293">#48293</a></li>
	<li dir="ltr">Switch to debug with server=n <a href="https://github.com/elastic/elasticsearch/pull/48188">#48188</a></li>
	<li dir="ltr">Fix security origin for TokenService#findActiveTokensFor... <a href="https://github.com/elastic/elasticsearch/pull/47418">#47418</a></li>
	<li dir="ltr">Increase timeout for yml tests <a href="https://github.com/elastic/elasticsearch/pull/48237">#48237</a></li>
	<li dir="ltr">Fix download of 6.x rpm and deb packages <a href="https://github.com/elastic/elasticsearch/pull/48228">#48228</a></li>
</ul><h4 dir="ltr">Changes in 7.4:
</h4><ul>
	<li dir="ltr">Change policy_id to list type in slm.get_lifecycle <a href="https://github.com/elastic/elasticsearch/pull/47766">#47766</a></li>
	<li dir="ltr">Lucene#asSequentialBits gets the leadCost backwards. <a href="https://github.com/elastic/elasticsearch/pull/48335">#48335</a></li>
	<li dir="ltr">SQL: add "format" for "full" date range queries <a href="https://github.com/elastic/elasticsearch/pull/48073">#48073</a></li>
	<li dir="ltr">fix incorrect comparison <a href="https://github.com/elastic/elasticsearch/pull/48208">#48208</a></li>
</ul><h4 dir="ltr">Changes in 7.3:
</h4><ul>
	<li dir="ltr">[7.4] [DOCS] Extends analyzed.fields description in DFA resources <a href="https://github.com/elastic/elasticsearch/pull/48306">#48306</a></li>
</ul><h4 dir="ltr">Changes in 6.8:
</h4><ul>
	<li dir="ltr">Download jdk to a temp file to make sure retries work <a href="https://github.com/elastic/elasticsearch/pull/48249">#48249</a></li>
	<li dir="ltr">Do not reference values for filtered settings <a href="https://github.com/elastic/elasticsearch/pull/48066">#48066</a></li>
	<li dir="ltr">Relative paths, jornalctl in additional logs <a href="https://github.com/elastic/elasticsearch/pull/48276">#48276</a></li>
	<li dir="ltr">Handle negative free disk space in deciders <a href="https://github.com/elastic/elasticsearch/pull/48392">#48392</a></li>
	<li dir="ltr">[DOCS] 6.8.4 release notes <a href="https://github.com/elastic/elasticsearch/pull/48360">#48360</a></li>
	<li dir="ltr">Cleanup Concurrent RepositoryData Loading <a href="https://github.com/elastic/elasticsearch/pull/48329">#48329</a></li>
	<li dir="ltr">Always publish a build scan in CI <a href="https://github.com/elastic/elasticsearch/pull/48348">#48348</a></li>
	<li dir="ltr">Use an env var for the classpath of jar hell task <a href="https://github.com/elastic/elasticsearch/pull/48240">#48240</a></li>
	<li dir="ltr">Fix link to GCP upload in build scans <a href="https://github.com/elastic/elasticsearch/pull/48248">#48248</a></li>
	<li dir="ltr">Close query cache on index service creation failure <a href="https://github.com/elastic/elasticsearch/pull/48230">#48230</a></li>
	<li dir="ltr">Create an upload report once the build completes <a href="https://github.com/elastic/elasticsearch/pull/47642">#47642</a></li>
</ul><h4 dir="ltr">Changes in Rally</h4><h4 dir="ltr">Changes in 1.4.0:
</h4><ul>
	<li dir="ltr">Upgrade Elasticsearch client to latest (7.0.5). <a href="https://github.com/elastic/rally/pull/795">#795</a></li>
	<li dir="ltr">Properly wait for recovery to finish <a href="https://github.com/elastic/rally/pull/800">#800</a></li>
	<li dir="ltr">Don't set retries for restore snapshot <a href="https://github.com/elastic/rally/pull/799">#799</a></li>
	<li dir="ltr">Initialize schedule lazily <a href="https://github.com/elastic/rally/pull/801">#80</a></li>
</ul>

This Week in Elasticsearch and Apache Lucene - 2019-10-28

This Week in Elasticsearch and Apache Lucene - 2019-10-21

This Week in Elasticsearch and Apache Lucene - 2019-10-14

<h2>Elasticsearch</h2><h4>Highlighted Discuss Post</h4><p dir="ltr">We&nbsp;<a href="https://discuss.elastic.co/t/dramatic-performance-degradation-on-range-queries-after-upgrade-es-from-2-4-to-7-1/201132/11">helped a user</a>&nbsp;debug a performance degradation for queries with range clauses: Elasticsearch uses Lucene’s&nbsp;<a href="https://lucene.apache.org/core/7_0_0/core/org/apache/lucene/search/IndexOrDocValuesQuery.html">IndexOrDocValuesQuery</a>&nbsp;to decide whether to use either index structure (points or terms), or doc values in executing a query. A characteristic of the user’s data, large arrays on a field, was causing the planner to pick a slower execution strategy. Ignacio provided a workaround of not using doc values for the field that&nbsp;<a href="https://discuss.elastic.co/t/dramatic-performance-degradation-on-range-queries-after-upgrade-es-from-2-4-to-7-1/201132/13">worked for the user</a>, and&nbsp;<a href="https://issues.apache.org/jira/browse/LUCENE-8990">opened a lucene issue</a>&nbsp;to address the underlying problem.</p><h4>Too many translog files</h4><p dir="ltr">We looked into an issue where a node in a cluster had run out of file descriptors because of a very large number of many small translog files. The default retention policy keeps translog generations up to 12 hours and 512MB to power sequence-number based recoveries, regardless of the number of translog files. We introduced a&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47414">new limit</a>&nbsp;for the number of retained files so that we can protect the cluster better in these situations. Note that this patch is mainly for 6.x (and 7.x clusters that have soft-deletes disabled), as since version 7.4, the age- and size-based translog retention policies are&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/45473">no longer active</a>&nbsp;due to the use of peer recovery retention leases.</p><h4>API Keys</h4><p dir="ltr">We started on a UI for managing API Keys. That highlighted a gap in the GET API (to list existing API keys) which didn’t support listing all keys (even if called by a superuser). We has raised a PR to&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47274">support retrieving all API keys</a>.</p><p dir="ltr">We originally had a requirement that API key names had to be globally unique. This had a few issues</p><ol><li dir="ltr">Individual users could not see each other’s keys, yet we expected them to be able to pick a unique name each time they created one (which would encourage schemes like including your userid and the date in the name).</li><li dir="ltr">The unique check introduced a noticeable performance penalty, particularly when creating API keys in bulk.</li><li dir="ltr">We could never truly enforce uniqueness because the check-for-existing & create-new key could not be done within a single atomic transaction.&nbsp;</li></ol><p dir="ltr">We agreed to&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47549">drop this uniqueness requirement from the API</a>&nbsp;and We have opened a PR for that. We started a discussion about&nbsp;<a href="https://github.com/elastic/elasticsearch/issues/47502">whether the upcoming UI should seek to have unique names</a>&nbsp;within a single user’s set of API keys.</p><h4>Removing types</h4><p dir="ltr">We continue the work on the removal of types. He already handled several APIs in master (<a href="https://github.com/elastic/elasticsearch/pull/46587">Get and MultiGet</a>,&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/46926">Explain</a>,&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/46927">ValidateQuery</a>, ...) and will continue to pick up work from the&nbsp;<a href="https://github.com/elastic/elasticsearch/issues/41059">meta issue</a>. These are all reasonably small changes, but types are deeply woven into a lot of code in Elasticsearch so the cleanup is not over.&nbsp;</p><h4>Point in time searcher</h4><p dir="ltr">We continued the cleanup of the search context in order to be able to rebuild it on every search phase. Today the search context is built once (on the first search phase) and then saved in a map in memory. This is fragile because the context is entirely mutable and exposes a lot of informations that are usable in some phases but not all. For this reason, we opened a&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47198">pr</a>&nbsp;to make the context immutable where possible and to allow building a new one easily. This cleanup will allow to rebuild the context on each phase using the original shard request and passing the needed information directly from the coordinating node. We will now focus on splitting the search context into a reader context that should survive the search phase and a request context that will be re-created on each phase. This is required to implement the ability to reuse the same&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/46543">point in time searcher</a>&nbsp;on multiple queries.</p><h4>Circuit Breaker improvements</h4><p dir="ltr">We opened a PR to wrap&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47515">various usages of Lucene's PriorityQueue</a>&nbsp;with a circuit breaker. This PQ allocates its internal array immediately on instantiation, and if instantiated with a large size (like when people use INT.MAX_VALUE for agg sizes) it could easily OOM a node because we don't track it in a circuit breaker.</p><h4 dir="ltr">Enrich</h4><p dir="ltr">We added monitoring support for Enrich. There is now a collector for Enrich that will add Enrich specific items to the .monitoring-es index. The information here can be used to help troubleshoot and possibly correlate with other metrics recorded. The information recorded mirrors the existing stats and can be&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/46708">found here</a>.</p><p dir="ltr">The get policies api should be able to return more information than just configuration. Things like last policy execution and policy status. This will make it easier for the UI to build a policy overview. In order to facilitate this a config namespace&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47162">was added the the get policies api</a>.</p><p dir="ltr">Executing an enrich policy (i.e. copying data from source index to the .enrich index) can take some time. The policy execute api should support an async model via a wait_for_completion parameter, which will allow the ui execute a policy in the background without waiting for a response.</p><h4>API reference reformatting</h4><p dir="ltr">Work to standardize our API reference material is&nbsp;<a href="https://github.com/elastic/elasticsearch/issues/45195">nearing completion</a>. We finished the Ingest API docs (#<a href="https://github.com/elastic/elasticsearch/issues/47049">47409</a>), Deb is wrapping up the Document APIs (#<a href="https://github.com/elastic/elasticsearch/issues/45365">45365</a>), and finished the Profile API (#<a href="https://github.com/elastic/elasticsearch/pull/47168">47211</a>). Going forward, we should use the&nbsp;<a href="https://github.com/elastic/docs/blob/master/shared/api-ref-ex.asciidoc">API Ref template</a>&nbsp;when we're adding new API docs.&nbsp;</p><h4>Elasticsearch reference reorg</h4><p dir="ltr">You might have noticed that things are moving around within the Elasticsearch reference and that more information is being pulled in from the Stack Overview. For example, We recently moved the following pieces.</p><ul><li dir="ltr">Documentation for transforms is now alongside rollups:&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/transforms.html">https://www.elastic.co/guide/en/elasticsearch/reference/master/transforms.html</a></li><li dir="ltr">Documentation for Watcher is also now in the ES ref:&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/xpack-alerting.html">https://www.elastic.co/guide/en/elasticsearch/reference/master/xpack-alerting.html</a></li></ul><h4>Running SLM retention on-demand</h4><p dir="ltr">The ES team<a href="https://github.com/elastic/elasticsearch/pull/47405">&nbsp;added a new API endpoint</a>&nbsp;this week to run Snapshot Lifecycle Management (SLM) retention immediately, without scheduling it. We submitted a PR that<a href="https://github.com/elastic/kibana/pull/47150">&nbsp;allows users to perform this action through the UI</a>. This may be useful to perform a manual invocation or to perform a one-off cleanup. This is a continuation of the work she started a few weeks ago that added the ability to<a href="https://github.com/elastic/kibana/pull/45193">&nbsp;configure retention</a>&nbsp;when creating a SLM policy.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2e70c438f70d9cfc/5f9720682425cd7a8af684d6/blog-this-week-2019-10-04.png" data-sys-asset-uid="blt2e70c438f70d9cfc" alt="" style="display: block; margin: auto;"></p><h4>Console improvements</h4><p dir="ltr">We fixed a<a href="https://github.com/elastic/kibana/issues/11125">&nbsp;2-year-old bug</a>&nbsp;in Console in which GET requests with bodies were forwarded to Elasticsearch as POST requests. This resulted in requests creating documents instead of returning search results. The<a href="https://github.com/elastic/kibana/pull/46200">&nbsp;solution</a>&nbsp;involved migrating the logic which mediates these requests off the Hapi library.</p><p dir="ltr">We also fixed some important UI issues in Console. This included a<a href="https://github.com/elastic/kibana/pull/47100">&nbsp;bug in Safari where Console did not render properly</a>, and a customer bug reported around the<a href="https://github.com/elastic/kibana/pull/47227">&nbsp;misalignment of the Console menu controls</a>. Additionally, we continue to work on the Console refactor:&nbsp;<a href="https://github.com/elastic/kibana/pull/46793">we have the first iteration of Console autocomplete without directly consuming the Ace editor</a>&nbsp;working this week.</p><h4 dir="ltr"></h4><h2>Apache Lucene</h2><h4>Better heuristic for range queries</h4><p dir="ltr">The most important optimization we have for range queries consists of deciding whether it's more efficient to run the range query on points or doc values. Points work better when the range is the most selective filter of the query, as it can quickly identify all matching documents, but doc values work better when there is another filter that is more selective. This logic relies on figuring out an approximation of the number of matched documents for a range query.</p><p dir="ltr">Until now we were approximating the number of matched documents as an approximation of the number of matched values. However this can trigger bad decisions if you have multi-valued fields, as the number of matches might be over-estimated. Ignacio improved the heuristic by deriving an expected number of matching documents based on the number of matching values as well as the total number of documents and values.</p><h4>Asynchronous caching</h4><p dir="ltr">The query cache is getting the ability to cache filters on a different thread. While this doesn't help with query throughput since the same amount of work needs to be done in total, it might help with latency in case not all search threads are already busy.</p><h4>Searching a single index in parallel&nbsp;</h4><p dir="ltr">The way that Lucene can distribute searching a single index across multiple threads in order to improve latency is receiving a lot of attention these days. One thing that makes it challenging is that recent optimizations like block-max WAND rely on information about previously collected documents in order to skip more documents. So if different subsets of the index get collected independently, each thread needs to do the work of learning what is the minimum score that a document must have to be competitive independently. Jim and Atri are looking into sharing information between threads in order to make things more efficient, which is already proving helpful.</p><h4>Other</h4><ul><li>We are fixing NRTSuggester so that it wouldn't miss suggestions. This can happen today because the suggestion collector sometimes needs to ignore suggestions for deduplication purposes, but it doesn't let the suggester know about it so the suggester might stop collecting while not enough suggestions have been collected yet.</li><li>A simple short circuit helped improve exact seeks in the terms dictionary significantly.</li><li>We wonder whether we should encode the type of triangle when indexing shapes in order to speed up query evaluation.</li><li>We are looking at improving the splitting decisions of our KD tree by taking into account the fact that splitting on one dimension may affect the range of the values of other dimensions.</li></ul><h2 dir="ltr">Changes in Elasticsearch</h2><h4>Changes in 7.5:</h4><ul><li dir="ltr">Allow setting validation against arbitrary types&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47264">#47264</a></li><li dir="ltr">Remove groovy test code from buildSrc&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47416">#47416</a></li><li dir="ltr">Add API to execute SLM retention on-demand&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47405">#47405</a></li><li dir="ltr">Fix highlighting of overlapping terms in the unified highlighter&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47227">#47227</a></li><li dir="ltr">Use optype CREATE for single auto-id index requests&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47353">#47353</a></li><li dir="ltr">Clarify missing java error message&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/46160">#46160</a></li><li dir="ltr">Remove empty buildSrc subproject&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47415">#47415</a></li><li dir="ltr">SQL: Implement DATE_PART function&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47206">#47206</a></li><li dir="ltr">Cancel recoveries even if all shards assigned&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/46520">#46520</a></li><li dir="ltr">Provide better error when updating geo_shape field mapper settings&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47281">#47281</a></li><li dir="ltr">Testclusters fix bwc&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/46740">#46740</a></li><li dir="ltr">Allow optype CREATE for append-only indexing operations&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47169">#47169</a></li><li dir="ltr">Fail allocation of new primaries in empty cluster&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/43284">#43284</a></li></ul><h4 dir="ltr">Changes in 7.4:</h4><ul><li dir="ltr">Fix dependency resolution conflict in SQL security integration tests&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47531">#47531</a></li><li dir="ltr">Fix Rollover error when alias has closed indices&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47148">#47148</a></li><li dir="ltr">Fix es.http.cname_in_publish_address Deprecation Logging&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47451">#47451</a></li><li dir="ltr">Add client jar for mapper-extras&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47430">#47430</a></li><li dir="ltr">Reset Token position on reuse in predicate_token_filter&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47424">#47424</a></li><li dir="ltr">Fix alias field resolution in match query&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47369">#47369</a></li><li dir="ltr">Fix AD realm additional metadata&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47179">#47179</a></li><li dir="ltr">Omit writing index metadata for non-replicated closed indices on data-only node&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47285">#47285</a></li></ul><h4 dir="ltr">Changes in 6.8:</h4><ul><li dir="ltr">Limit number of retaining translog files for peer recovery&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47414">#47414</a></li><li dir="ltr">Use VAULT_TOKEN environment variable if it exists&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/45525">#45525</a></li><li dir="ltr">SQL: wrong number of values for columns&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/42122">#42122</a></li><li dir="ltr">Add workaround for building docker on debian 8&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/47106">#47106</a></li></ul><h4 dir="ltr">Changes in Elasticsearch Hadoop Plugin</h4><p dir="ltr">Changes in 7.4:</p><ul><li dir="ltr">[DOCS] Add 7.4.0 release notes&nbsp;<a href="https://github.com/elastic/elasticsearch-hadoop/pull/1356">#1356</a></li></ul><h4 dir="ltr">Changes in Elasticsearch SQL ODBC Driver</h4><p dir="ltr">Changes in 7.5:</p><ul><li dir="ltr">Fix: don't check logging path if logging disabled&nbsp;<a href="https://github.com/elastic/elasticsearch-sql-odbc/pull/186">#186</a></li><li dir="ltr">Support the new SHAPE data type&nbsp;<a href="https://github.com/elastic/elasticsearch-sql-odbc/pull/185">#185</a></li><li dir="ltr">Provision and test against remote ES instance&nbsp;<a href="https://github.com/elastic/elasticsearch-sql-odbc/pull/184">#184</a></li></ul><h4 dir="ltr">Changes in Rally</h4><p dir="ltr">Changes in 1.4.0:</p><ul><li dir="ltr">Add support for OSNAME and ARCH variables in dist repo URLs.&nbsp;<a href="https://github.com/elastic/rally/pull/781">#781</a></li><li dir="ltr">BREAKING: Gather cluster-level metrics in driver&nbsp;<a href="https://github.com/elastic/rally/pull/779">#779</a></li><li dir="ltr">Expose race-id as command line parameter&nbsp;<a href="https://github.com/elastic/rally/pull/778">#778</a></li></ul><h4 dir="ltr">Changes in Rally Tracks</h4><ul><li dir="ltr">Replace interval parameter with fixed_interval and calendar_interval&nbsp;<a href="https://github.com/elastic/rally-tracks/pull/85">#85</a></li></ul>

This Week in Elasticsearch and Apache Lucene - 2019-10-04

This Week in Elasticsearch and Apache Lucene - 2019-09-09

This Week In Elasticsearch, September 2nd

This Week In Elasticsearch, August 26

<h2>Highlights<br></h2><h4>Automatic cancellation of search requests
</h4><p dir="ltr">We worked on adding <a href="https://github.com/elastic/elasticsearch/pull/45379">support</a> for cancelling requests to the low-level REST client, which unblocks the security manager testing issue we encountered on the search task cancellation PR. We suspect that it's a bug in the HTTP client that is uncovered by the way we reuse HTTP request instance through multiple retries. We're waiting for feedback from the apache async HTTP client dev, either they will fix the bug or we will need to adapt the way we make requests in the low-level client (which would complicate our own code a bit).
</p><h4 dir="ltr">SQL
</h4><p dir="ltr">We fixed an issue regarding the deprecation of the generic “interval” parameter for date histograms usage. Also, we fixed another bug that made any date comparison (&gt;, &lt;, &gt;=, !=....) with constants generated by CURRENT_* functions to fail if the date field involved had a different format than the default one.
</p><p dir="ltr">We have continued the <a href="https://github.com/elastic/elasticsearch-sql-odbc/issues/152">work on CBOR support</a>, bringing in the functionality to unpack the result sets and “forward” them to the application. Along this work the data access pattern for some applications that query data cell-by-cell (like Excel; i.e not “in-bulk”, with bounded arrays) has been optimized. With CBOR, Elasticsearch uses indefinite-length arrays (intended initially for streaming, though not exclusively), which required some further changes in order to avoid double-parsing where possible.
</p><p dir="ltr">We looked into <a href="https://github.com/elastic/elasticsearch/pull/45516">improving</a> the way SQL does pagination across both scroll and agg requests and raised a PR for it.
</p><h4>Cross Cluster Search (CCS) Docs
</h4><p dir="ltr">We have updated the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-cross-cluster-search.html">cross-cluster search docs</a>, featuring new diagrams for <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-cross-cluster-search.html">“How CCS Works”</a>, re-organized headings, and migrated to top-level nav.
</p><h4>TLS Configuration Changes
</h4><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/45514">raised changes that will detect and warn</a> when a server SSL context (like the REST HTTP server) is created with a meaningless combination of <code>verification_mode: none</code> and <code>client_authentication</code> enabled. In this case the server would request a client certificate, but never validate it. That doesn't violate any specs, but is a sign of misconfiguration that could cause administrators to think they have a level of security that they don't. We expect to make this an error in 8.0.
</p><p dir="ltr">We have been working on detecting other potential misconfigurations for server SSL contexts. We plan to make it an error to configure SSL on the HTTP or transport services without providing a certificate and key (this is already an error for HTTP, but not for transport).
</p><p dir="ltr">We also intend to make it an error to configure SSL for HTTP or transport, but not explicitly enable or disable it. We have seen cases where an administrator will set up a keystore and truststore, but forget to turn on xpack.security.http.ssl.enabled (or transport), and it wasn't immediately obvious why TLS wasn't working. It will still be possible to enable or disable SSL at will, but if you configure any SSL settings, users should explicitly enable or disable.
</p><h4>Refresh Interval for security
</h4><p dir="ltr">Occasionally we run into cases where a cluster has a template that sets a high refresh interval and is applied to "*", and would end up being applied to the security indices.
</p><p dir="ltr">This could cause issues as some security APIs do an internal wait-for refresh, which could cause the API to take over 30 seconds to complete, at which point Kibana would timeout.
</p><p dir="ltr">Understandably, the user would not realise that their Kibana login was failing because their template had an index pattern of "*". We had a discussion about what we could do to detect the problem and either fix it or at least make it obvious to the cluster admin. After a bit of back and forth we realised that the solution was very simple - we just need to <a href="https://github.com/elastic/elasticsearch/pull/45434">set an explicit refresh interval</a> when we create the security indices. 
</p><h4 dir="ltr">Hadoop
</h4><p dir="ltr">We fixed an integration issue with Cloudera's fork of Spark 2.2+. Spark is elasticsearch-hadoop's most popular integration, and Cloudera is the most popular commercial vendor of Hadoop/Spark. Cloudera's fork introduced a non-passive method signature that is different from the upstream version of Spark. Technically, this fork is branded as<a href="https://www.cloudera.com/documentation/spark2/latest/topics/spark2.html"> Cloudera's CDS</a> (an enhanced Cloudera Spark). For Cloudera consumers on CDH5, CDS is Cloudera blessed way to use a modern version of Spark.
</p><h4>Snapshot Lifecycle Management (SLM)
</h4><p dir="ltr">We added some additional information when retrieving SLM policies.  SLM currently takes snapshots in an asynchronous manner; when SLM kicks off a snapshot, we don’t wait to see whether the snapshot is currently running, or if it immediately failed, etc. With<a href="https://github.com/elastic/elasticsearch/pull/45245"> #45245</a> when retrieving any SLM policies, the response will also contain information about currently running snapshots, this info includes the name, state, UUID, and start time as well as any failure information. Since this information is read from the cluster state, there is no external call to retrieve this info from a snapshot repo that could potentially cause users to incur transfer costs.
</p><p dir="ltr">We also merged<a href="https://github.com/elastic/elasticsearch/pull/45362"> #45362</a> which added a new API (_slm/stats) to retrieve statistics about the SLM’s snapshot successes and failures, as well as information about retention runs, and time spent deleting snapshots. This information is presented globally as well as on a per-policy basis.
</p><h4 dir="ltr">Geo
</h4><p dir="ltr">We refactored all of the Geometry classes <a href="https://github.com/elastic/elasticsearch/pull/45332">from lat,lon to x,y</a> in master. We decided to make the change for a few reasons:
</p><ul>
	<li dir="ltr">The current order and naming differs from most of the other libraries and standard that we are using in elasticsearch including JTS, WKT, GeoJSON and naming convention used in geosql functions.</li>
	<li dir="ltr">We are in process of extending use of Geometry classes in the context of XYShapes and different projections, which makes generic x and y more appropriate than lat and lon.</li>
	<li dir="ltr">The hierarchy is based on Geometry (not Geography) in the first place, so member naming based on lat and lon doesn't seem to be consistent with the class naming.</li>
</ul><p dir="ltr">It is a breaking java change, but mainly for JDBC geo-sql users. Given that it is a beta feature, and JDBC requires a recompilation anyway, the impact should be relatively minimal.
</p><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/45413">merged support for geo_bounds</a> on shapes in the feature branch, and is now working on geo_grid aggregation for shapes. The circle ingest processor is also close to completion.
</p><h3 dir="ltr">Changes in Elasticsearch</h3><h4></h4><h5>Changes in 7.4:</h5><ul><li dir="ltr">Catch AllocatedTask registration failures <a href="https://github.com/elastic/elasticsearch/pull/45300">#45300</a></li>
	<li dir="ltr">Update the schema for the REST API specification <a href="https://github.com/elastic/elasticsearch/pull/42346">#42346</a></li>
	<li dir="ltr">Add build operating system as build scan tag <a href="https://github.com/elastic/elasticsearch/pull/45558">#45558</a></li>
	<li dir="ltr">Remove leftover debugging logic <a href="https://github.com/elastic/elasticsearch/pull/45556">#45556</a></li>
	<li dir="ltr">Fix remote cache misses for test tasks <a href="https://github.com/elastic/elasticsearch/pull/45521">#45521</a></li>
	<li dir="ltr">Watcher reporting: add email warning if CSV attachment contains values that may be interperted as formulas <a href="https://github.com/elastic/elasticsearch/pull/44460">#44460</a></li>
	<li dir="ltr">Add mapper-extras and the RankFeatureQuery in the hlrc <a href="https://github.com/elastic/elasticsearch/pull/43713">#43713</a></li>
	<li dir="ltr">Fix bats distro test to build distributions <a href="https://github.com/elastic/elasticsearch/pull/45529">#45529</a></li>
	<li dir="ltr">Restore check part 1 and 2 order mistakenly reversed in [#45098] <a href="https://github.com/elastic/elasticsearch/pull/45522">#45522</a></li>
	<li dir="ltr">Fix remote cache misses for checkstyle tasks <a href="https://github.com/elastic/elasticsearch/pull/45512">#45512</a></li>
	<li dir="ltr">Fix issues with serializing BulkByScrollResponse <a href="https://github.com/elastic/elasticsearch/pull/45357">#45357</a></li>
	<li dir="ltr">Add support for cancelling async requests in low-level REST client <a href="https://github.com/elastic/elasticsearch/pull/45379">#45379</a></li>
	<li dir="ltr">Add SSL/TLS settings for watcher email <a href="https://github.com/elastic/elasticsearch/pull/45272">#45272</a></li>
	<li dir="ltr">Fix bug in copying bytes for socket write <a href="https://github.com/elastic/elasticsearch/pull/45463">#45463</a></li>
	<li dir="ltr">Fix location of reaper jar on Windows <a href="https://github.com/elastic/elasticsearch/pull/45464">#45464</a></li>
	<li dir="ltr">Retrieve processors instead of checking existence <a href="https://github.com/elastic/elasticsearch/pull/45354">#45354</a></li>
	<li dir="ltr">Create index with typeless mapping <a href="https://github.com/elastic/elasticsearch/pull/45120">#45120</a></li>
	<li dir="ltr">Geo: Change order of parameter in Geometries to lon, lat <a href="https://github.com/elastic/elasticsearch/pull/45332">#45332</a></li>
	<li dir="ltr">Set start of the week to Monday for root locale <a href="https://github.com/elastic/elasticsearch/pull/43652">#43652</a></li>
</ul><h4></h4><h5>Changes in 7.3:</h5><ul><li dir="ltr">Use bundled JDK in Sys V init <a href="https://github.com/elastic/elasticsearch/pull/45593">#45593</a></li>
	<li dir="ltr">Fix watcher HttpClient URL creation <a href="https://github.com/elastic/elasticsearch/pull/45207">#45207</a></li>
	<li dir="ltr">Fix a bug where mappings are dropped from rollover requests. <a href="https://github.com/elastic/elasticsearch/pull/45411">#45411</a></li>
	<li dir="ltr">Make sure to validate the type before attempting to merge a new mapping. <a href="https://github.com/elastic/elasticsearch/pull/45157">#45157</a></li>
	<li dir="ltr">[ML-DataFrame] fix starting a batch data frame after stopping at runtime  <a href="https://github.com/elastic/elasticsearch/pull/45340">#45340</a></li>
</ul><h4></h4><h5>Changes in 7.2:</h5><ul><li dir="ltr">SQL: adds format parameter to range queries for constant date comparisons <a href="https://github.com/elastic/elasticsearch/pull/45326">#45326</a></li>
</ul><h4></h4><h5>Changes in 6.8:</h5><ul><li dir="ltr">Read git revision directly from repository <a href="https://github.com/elastic/elasticsearch/pull/45547">#45547</a></li>
	<li dir="ltr">Prevent Leaking Search Tasks on Exceptions in FetchSearchPhase and DfsQueryPhase <a href="https://github.com/elastic/elasticsearch/pull/45500">#45500</a></li>
	<li dir="ltr">Allow _update on write alias <a href="https://github.com/elastic/elasticsearch/pull/45318">#45318</a></li>
</ul><h4></h4><h5>Changes in 8.0:</h5><ul><li dir="ltr">Fix build cache misses caused by embedded reaper jar <a href="https://github.com/elastic/elasticsearch/pull/45404">#45404</a></li>
</ul><h4 dir="ltr">Changes in Elasticsearch Hadoop Plugin</h4><h5>Changes in 7.4:</h5><ul><li dir="ltr">Do not retry scroll requests in NetworkClient. <a href="https://github.com/elastic/elasticsearch-hadoop/pull/1330">#1330</a></li>
</ul><h5>Changes in 6.8:</h5><ul><li dir="ltr">Fix for Cloudera CDS 2.2.0+ Spark credentialsRequired method <a href="https://github.com/elastic/elasticsearch-hadoop/pull/1332">#1332</a></li>
</ul><h4 dir="ltr">Changes in Elasticsearch SQL ODBC Driver</h4><h5>Changes in 7.4:</h5><ul><li dir="ltr">Introduce CBOR format support for REST payloads <a href="https://github.com/elastic/elasticsearch-sql-odbc/pull/169">#169</a></li>
</ul><h4 dir="ltr">Changes in Rally</h4><h5>Changes in 1.3.0:</h5><ul><li dir="ltr">BREAKING: Remove lap feature and all references. <a href="https://github.com/elastic/rally/pull/739">#739</a></li>
</ul>

This Week In Elasticsearch, August 16

This Week in Elasticsearch and Apache Lucene - 2019-08-09

This Week in Elasticsearch and Apache Lucene - 2019-08-02

This Week in Elasticsearch and Apache Lucene - 2019-07-0526

This Week in Elasticsearch and Apache Lucene - 2019-07-26

This Week in Elasticsearch and Apache Lucene - 2019-07-12

<h2 dir="ltr" id="docs-internal-guid-d39c7305-7fff-ac75-295a-d245f06a250a">Elasticsearch Dev Highlights<br></h2><h4 dir="ltr">Bitset Level Cache for Document Security
</h4><p dir="ltr">Document Level Security currently uses the shared BitSet cache. However that cache (by design) never expires its entries. The problem is that document level security queries support templates that can make use of user metadata (like usernames and roles). So, it’s possible that every user has their own concrete query which will cause us to construct and cache a separate bitset for each user, for every lucene segment. In some cases this has caused nodes to hold more than 10 Gb worth of bit sets all due to DLS. We opened a PR to<a href="https://github.com/elastic/elasticsearch/pull/43669"> switch DLS to use its own cache, with a configurable memory limit</a> so that we can avoid such problems.
</p><h4 dir="ltr">Snapshot resiliency
</h4><p dir="ltr">We merged the<a href="https://github.com/elastic/elasticsearch/pull/42189"> automatic cleanup logic for orphaned indices</a> that will automatically run on snapshot deletes in ES versions 7.3 and above. We are also looking at adding a dedicated ES endpoint to trigger this clean-up.
</p><h4 dir="ltr">Reindex resiliency
</h4><p dir="ltr">We are making reindex resilient against coordinator crashes by moving parts of it to the persistent tasks framework. We are currently looking at ways to make the API backward compatible. The reindex API, when run with wait_for_completion=false, currently returns a task id which can be used with the<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex-task-api"> Tasks APIs</a> to cancel or get the status of the task. The task IDs returned by the Tasks APIs are local to a node, however. By making reindex resilient to node failures, tasks will be respawned on another node where they will have a fresh task ID. These task IDs provided by the tasks API can therefore not be used for cancelling resilient reindex jobs in the future. We will be looking at creating a proper API for tracking status of and cancelling reindex jobs. We have also added support to<a href="https://github.com/elastic/elasticsearch/pull/43457"> serialize reindex requests to non-JSON content-types</a>, which will be relevant for failovers as well.
</p><p dir="ltr">We are making reindex resilient against data node crashes. While iterating on the prototype, we are factoring out code that does not need to stay on a feature branch for now but can go directly to master to<a href="https://github.com/elastic/elasticsearch/pull/43576"> simplifies reindex testing</a> there as well.
</p><h4 dir="ltr">Updatable Synonyms
</h4><p dir="ltr">We merged the new<a href="https://github.com/elastic/elasticsearch/pull/43313"> feature</a> in x-pack under a Basic license. Starting in 7.3 users will be able to mark their synonyms filters as updatable (with a simple flag). The flag will ensure that these filters are only used on search analyzers for query time expansions.  Updating the synonyms will not require to close/open an index anymore, instead users can send the new synonym file to all nodes and a call to POST /my_index/_reload_search_analyzers will automatically update all search analyzers with the latest version of synonyms. We are now working on adding the new API in the high level rest client.
</p><h4 dir="ltr">Prefix and Wildcard intervals
</h4><p dir="ltr">We<a href="https://github.com/elastic/elasticsearch/pull/43635"> exposed</a> prefix and wildcard sources in the Intervals query.  Both source will throw an error if the prefix/wildcard matches more than 128 terms but the prefix one can take advantage of the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/index-prefixes.html">index_prefixes</a> option of the text field to limit the number of inverted lists that must be visited.
</p><h4 dir="ltr">Ingest
</h4><p dir="ltr">Ever wish you could write an Ingest node processor that did some expensive work and didn't block ingestion? We have been working on giving developers the ability to fork inside of an ingest node processor and callback on success/failure. This frees up the write thread to do other work while the processor is doing something expensive. This will be used by the enrich project, but has other potential uses. (<a href="https://github.com/elastic/elasticsearch/pull/43361">#43361</a>).
</p><h4 dir="ltr">Snapshot and Restore UI
</h4><p dir="ltr">We added the <a href="https://github.com/elastic/kibana/pull/39193">recovery status feature</a>. In master/8.0, she also adjusted Snapshot and Restore for changes to the <a href="https://github.com/elastic/kibana/pull/39533">ES snapshot API response</a>.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt721f2bcf99e931d3/5d165cc56e70b863da03da7e/60147908-f5b06980-9783-11e9-8cd7-1a4b89c4efc1.png" data-sys-asset-uid="blt721f2bcf99e931d3" alt="60147908-f5b06980-9783-11e9-8cd7-1a4b89c4efc1.png"></p><h4 dir="ltr">Snapshot Lifecycle Management (SLM)
</h4><p dir="ltr">We started working on SLM retention. Retention for SLM is the next phase for snapshot lifecycle management that allows a user to specify how many snapshots to keep, and for how long. This work will also eventually allow users to replace any periodic snapshotting with something built into ES. It also will allow the same functionality for the k8s operator. There’s now an SLM retention meta issue (<a href="https://github.com/elastic/elasticsearch/issues/43663">#43663</a>) as well as a feature branch (slm-retention), the<a href="https://github.com/elastic/elasticsearch/pull/43605"> first PR</a> sets up the framework for future work.
</p><h4 dir="ltr">Password Protected Keystore
</h4><p dir="ltr">We worked on PRs for <a href="https://github.com/elastic/elasticsearch/pull/43197">reloading secure settings</a> and<a href="https://github.com/elastic/elasticsearch/pull/38498"> adding password support in our keystore CLIs</a>. We discussed our approach for reading the password on startup, and have an <a href="https://github.com/elastic/elasticsearch/issues/32691#issuecomment-506728906">action plan for this</a>.
</p><h4 dir="ltr">Geo
</h4><p dir="ltr">We<a href="https://github.com/elastic/elasticsearch/pull/43657"> extracted coordinate validation</a> from libs/geo classes to make them compatible with XYShape.  Work has also begun working on ShapeBuilder refactoring, which we think is a prerequisite to the XYShape.  E.g. we want to get the ShapeBuilder story sorted out before we introduce more complexity on top of it with XYShape.
</p><p dir="ltr">We created a <a href="https://github.com/elastic/elasticsearch/issues/43644">meta issue</a> to track development on how the XYShape feature will be integrated into Elasticsearch.  It will introduce a new <code>geometry</code> field type, used for indexing and searching cartesian geometries. Work is progressing quickly on the GeometryFieldMapper and it's looking like we'll have the initial feature (general cartesian geometry support) ready for a 7.x release. That will move us to working on the next big Geo feature: adding support for multiple projections
</p><h4 dir="ltr">Analytics
</h4><p dir="ltr">Inspired by a<a href="https://github.com/elastic/elasticsearch/issues/43550"> feature request,</a> we opened a PR to add a<a href="https://github.com/elastic/elasticsearch/pull/43661"> cumulative cardinality pipeline agg.</a>  While a regular cardinality agg can tell you the number of distinct items each day, a cumulative cardinality agg can tell you how many "new" distinct items you've seen each day (as opposed to "repeat" items you might have seen before).  This is useful for example with tracking entirely new visitors to your site.  It leverages the HyperLogLog sketch that we already serialize to the coordinating node to iteratively merge into a cumulative sketch.  The agg is housed inside a new "data science" xpack plugin and will be Basic+
</p><p dir="ltr">We spent some time adding<a href="https://github.com/elastic/elasticsearch/pull/43494"> support for mocking out scripts</a>, to support the refactoring of integration tests, and better support for<a href="https://github.com/elastic/elasticsearch/pull/43405"> unmapped field types</a>. We also opened a PR to remove the<a href="https://github.com/elastic/elasticsearch/pull/43664"> recursive generic</a> on AggregatorFactory to help simplify type signatures.
</p><h2 dir="ltr">Apache Lucene</h2><h4 dir="ltr">Maximum clause count check
</h4><p dir="ltr">You might be familiar with the fact that Lucene fails boolean queries that have more than 1024 clauses. As of Lucene 9, this limit won't be applied on a per-BooleanQuery basis but<a href="https://issues.apache.org/jira/browse/LUCENE-8811"> on the entire query tree</a>. Before running a query, IndexSearcher now uses the query visitor API in order to visit all clauses of the query and count them.
</p><h4 dir="ltr">Better integration of Block-Max WAND
</h4><p dir="ltr">After improving Lucene's block-max AND to<a href="https://issues.apache.org/jira/browse/LUCENE-8770"> more efficiently run sub-clauses that have two-phase iterators</a> such as phrase queries, We are now<a href="https://issues.apache.org/jira/browse/LUCENE-8806"> tackling a similar issue with Lucene's block-max WAND scorer</a>, the scorer that is used for disjunctions. A first patch was proposed that made sure to run the verification phase of the two-phase iterator lazily. However, contrary to our intuition, benchmarks reported that this made disjunctions that include phrase queries slower. This is due to the fact that the patch assumes that it is cheaper to advance a term query than it is to verify a phrase match, which is not always true. The dataset that the benchmark has run on makes this problem very obvious since it contains documents that are artificially truncated, making phrase queries always fast to check. We are now iterating on a 2nd version of the patch that will hopefully address this issue.
</p><p dir="ltr">Interestingly, the cost of advancing term queries is an issue we had already encountered with conjunctions, so we started looking into what could be done to make it cheaper. The main reason why it is so expensive is that advancing a clause usually involves moving to a new block, and therefore requires decoding doc IDs and term frequencies for this new block - 128 entries. While doc IDs are always needed, there is a chance that none of the term frequencies will be needed. We are working on a change that<a href="https://github.com/apache/lucene-solr/pull/595"> decodes frequencies lazily</a>, which could decrease the cost of advancing a clause by up to 2x in some cases. We already had evidence that it would help with conjunctions and will now do testing with disjunctions as well.
</p><h4 dir="ltr">Fast range queries on doc values
</h4><p dir="ltr">We contributed a<a href="https://issues.apache.org/jira/browse/LUCENE-7714"> special range query which uses binary search on doc values in order to find matches</a>. This only works when the index is sorted by the field that is being queried, but this opens interesting doors as it performs very well yet doesn't require the field to be indexed, which could be interesting for our space-savvy users.
</p><h4 dir="ltr">Better handling of duplicates in BKD trees
</h4><p dir="ltr">We noticed some inefficiencies when storing duplicate values in BKD trees. Storage efficiency is not good since we don't even try to leverage these duplicates for compression unless they fill entire blocks. And query efficiency is not good either since we check whether a given value matches again and again, while we could test it only once. As a consequence Ignacio is contributing 3 changes that will make this better:
</p><p dir="ltr"> <a href="https://issues.apache.org/jira/browse/LUCENE-8888"> Using data dimensions as a tie-breaker when partitioning</a> to make sure that documents that have the same data-only dimensions will be next to each other in the tree as well.
</p><p dir="ltr"> <a href="https://issues.apache.org/jira/browse/LUCENE-8867"> Using run-length compression to store duplicate values</a> on leaves of the BKD tree.
</p><p dir="ltr"> <a href="https://issues.apache.org/jira/browse/LUCENE-8885"> Changing our Points API to allow queries to leverage the fact that there are duplicates on a leaf</a> in order not to do the same computation over and over again.
</p><h4 dir="ltr">Other
</h4><ul>
	<li> Lucene now has<a href="https://issues.apache.org/jira/browse/LUCENE-8855"> utility methods to get a rough estimate of the memory usage of a query</a>. This can be useful when using queries as keys of a cache which is memory-bound.</li>
	<li> A community member is contributing a<a href="https://issues.apache.org/jira/browse/LUCENE-8891"> stemmer for Estonian</a>.</li>
	<li> We are <a href="https://issues.apache.org/jira/browse/LUCENE-8857">refactoring TopDocs#merge</a> to make it easier to control tie-breaking of documents that have the same sort values.</li>
	<li> We are adding a<a href="https://issues.apache.org/jira/browse/LUCENE-8862"> framework to track memory usage of collectors</a>.</li>
	<li> We could add ways to track<a href="https://issues.apache.org/jira/browse/LUCENE-8884"> per-query I/O usage</a> via a directory wrapper.</li>
	<li> We merged a change that<a href="https://issues.apache.org/jira/browse/LUCENE-8859"> allows completion suggesters to be loaded offheap</a>.</li>
	<li> Can we<a href="https://issues.apache.org/jira/browse/LUCENE-8878"> simplify the FieldComparator API</a>?</li>
	<li> We noticed that there is<a href="https://issues.apache.org/jira/browse/LUCENE-8876"> ambiguity in the paper that describes the minimal english stemmer</a>.</li>
	<li> Luke should show<a href="https://issues.apache.org/jira/browse/LUCENE-8874"> SPI names instead of class names</a>, since this is how users look up analysis factories.</li>
	<li> There is a contribution to see<a href="https://issues.apache.org/jira/browse/LUCENE-8793"> intermediate analysis results with Luke</a>.</li>
</ul><h2 dir="ltr">Changes in Elasticsearch</h2><h4 dir="ltr">Changes in 8.0:
</h4><ul>
	<li dir="ltr">BREAKING: Remove deprecated sort options: nested_path and nested_filter<a href="https://github.com/elastic/elasticsearch/pull/42809"> #42809</a></li>
	<li dir="ltr">BREAKING: Remove preconfigured delimited_payload_filter<a href="https://github.com/elastic/elasticsearch/pull/43686"> #43686</a></li>
	<li dir="ltr">Remove compile-time dependency on test fixtures<a href="https://github.com/elastic/elasticsearch/pull/43651"> #43651</a></li>
	<li dir="ltr">Reindex remove outer level size<a href="https://github.com/elastic/elasticsearch/pull/43373"> #43373</a></li>
	<li dir="ltr">Fix GET /_snapshot/_all/_all if there are no repos<a href="https://github.com/elastic/elasticsearch/pull/43558"> #43558</a></li>
</ul><h4 dir="ltr">Changes in 7.3:
</h4><ul>
	<li dir="ltr">Remove nodeId from BaseNodeRequest<a href="https://github.com/elastic/elasticsearch/pull/43658"> #43658</a></li>
	<li dir="ltr">Add version and create_time to data frame analytics config<a href="https://github.com/elastic/elasticsearch/pull/43683"> #43683</a></li>
	<li dir="ltr">Allow reloading of search time analyzers<a href="https://github.com/elastic/elasticsearch/pull/43313"> #43313</a></li>
	<li dir="ltr">Use preconfigured filters correctly in Analyze API<a href="https://github.com/elastic/elasticsearch/pull/43568"> #43568</a></li>
	<li dir="ltr">Require [articles] setting in elision filter<a href="https://github.com/elastic/elasticsearch/pull/43083"> #43083</a></li>
	<li dir="ltr">[ML] Tag destination index with data frame metadata<a href="https://github.com/elastic/elasticsearch/pull/43567"> #43567</a></li>
	<li dir="ltr">Add painless method getByPath, get value from nested collections with dotted path (#43170)<a href="https://github.com/elastic/elasticsearch/pull/43606"> #43606</a></li>
	<li dir="ltr">Add prefix intervals source<a href="https://github.com/elastic/elasticsearch/pull/43635"> #43635</a></li>
	<li dir="ltr">[ML][Data Frame] Add support for allow_no_match for endpoints<a href="https://github.com/elastic/elasticsearch/pull/43490"> #43490</a></li>
	<li dir="ltr">[ML][Data Frame] improve pivot nested field validations<a href="https://github.com/elastic/elasticsearch/pull/43548"> #43548</a></li>
	<li dir="ltr">Ensure relocation target still tracked when start handoff<a href="https://github.com/elastic/elasticsearch/pull/42201"> #42201</a></li>
	<li dir="ltr">Disable testing conventions on Windows<a href="https://github.com/elastic/elasticsearch/pull/43532"> #43532</a></li>
	<li dir="ltr">Fix failing LicensingDocumentationIT test<a href="https://github.com/elastic/elasticsearch/pull/43533"> #43533</a></li>
	<li dir="ltr">Add voting-only master node<a href="https://github.com/elastic/elasticsearch/pull/43410"> #43410</a></li>
	<li dir="ltr">Revert "Test clusters: convert x-pack qa tests (#43283)"<a href="https://github.com/elastic/elasticsearch/pull/43549"> #43549</a></li>
	<li dir="ltr">Properly serialize remote query in ReindexRequest<a href="https://github.com/elastic/elasticsearch/pull/43457"> #43457</a></li>
	<li dir="ltr">Fix CreateRepository Requeset in HLRC<a href="https://github.com/elastic/elasticsearch/pull/43522"> #43522</a></li>
	<li dir="ltr">Fix score mode of the MinimumScoreCollector<a href="https://github.com/elastic/elasticsearch/pull/43527"> #43527</a></li>
	<li dir="ltr">Set document on script when using Bytes.WithScript<a href="https://github.com/elastic/elasticsearch/pull/43390"> #43390</a></li>
	<li dir="ltr">Add annotations to Painless whitelist<a href="https://github.com/elastic/elasticsearch/pull/43239"> #43239</a></li>
	<li dir="ltr">Do not hang on unsupported HTTP methods<a href="https://github.com/elastic/elasticsearch/pull/43362"> #43362</a></li>
</ul><h4 dir="ltr">Changes in 7.2:
</h4><ul>
	<li dir="ltr">Fix propagation of enablePositionIncrements in QueryStringQueryBuilder<a href="https://github.com/elastic/elasticsearch/pull/43578"> #43578</a></li>
	<li dir="ltr">Fix UOE on search requests that match a sparse role query<a href="https://github.com/elastic/elasticsearch/pull/43668"> #43668</a></li>
	<li dir="ltr">Issue deprecation warnings for preconfigured delimited_payload_filter<a href="https://github.com/elastic/elasticsearch/pull/43684"> #43684</a></li>
	<li dir="ltr">[Ml Data Frame] Size the GET stats search by number of Ids requested<a href="https://github.com/elastic/elasticsearch/pull/43206"> #43206</a></li>
	<li dir="ltr">[ML][Data Frame] re-ordering format priorities in dest mapping<a href="https://github.com/elastic/elasticsearch/pull/43602"> #43602</a></li>
	<li dir="ltr">Fix dockerfile for non-local builds<a href="https://github.com/elastic/elasticsearch/pull/43591"> #43591</a></li>
	<li dir="ltr">Fix indices shown in _cat/indices<a href="https://github.com/elastic/elasticsearch/pull/43286"> #43286</a></li>
	<li dir="ltr">[ML][Data Frame] Adjusting error message<a href="https://github.com/elastic/elasticsearch/pull/43455"> #43455</a></li>
</ul><h4 dir="ltr">Changes in 7.1:
</h4><ul>
	<li dir="ltr">Fix the bundled jdk flag to be passed through windows startup<a href="https://github.com/elastic/elasticsearch/pull/43502"> #43502</a></li>
</ul>

This Week in Elasticsearch and Apache Lucene - 2019-06-28

This Week in Elasticsearch and Apache Lucene - 2019-06-21

<h2 dir="ltr" id="docs-internal-guid-ea6a85cb-7fff-a13f-44a8-711e69b9173e">Elasticsearch Highlights
</h2><h4 dir="ltr">Transport Client
</h4><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/42538">removed of the transport client from the codebase</a>, meaning elasticsearch 8.0.0 will not include the java TransportClient. The<a href="https://github.com/elastic/elasticsearch/pull/42640/files"> client jar support has been removed from the build</a>; client jars were jar files that needed to be published so that a component (e.g. reindex, percolator, mustache) could be used with the transport client. We also has <a href="https://github.com/elastic/elasticsearch/pull/42483">removed the transport client from our documentation</a>.   
</p><h4 dir="ltr">Snapshot Restore UI
</h4><p dir="ltr">We started work on the <a href="https://github.com/jen-huang/kibana/tree/feature/snapshot-restore">delete snapshots functionality</a> and opened a PR that fixes some<a href="https://github.com/elastic/kibana/pull/38169"> i18n issues</a>. 
</p><h4 dir="ltr">Better storage of _source
</h4><p dir="ltr">We revived an old<a href="https://github.com/elastic/elasticsearch/issues/9034"> issue</a> about alternative ways to store our <code>_source</code> field in Lucene indices and wrote a quick proof of concept that stores each top-level field of the <code>_source</code> document in a separate stored field. The main benefit of this approach is that Lucene gives an identifier to each field and that we can identify fields by these numbers rather than having to repeat field names over and over again. This gave a 10-20% reduction of disk usage for stored fields on geonames depending on the codec. This is one of several ideas that are being explored these days and could help reduce disk usage, like dropping the _id field on documents that are below the global checkpoint for append-only use-cases, and possibly not index the @timestamp field and use index sorting + binary search on doc values to search on it.
</p><h4 dir="ltr">Faster range query on sorted index
</h4><p dir="ltr">We are looking into optimizing range queries when the index is sorted (<a href="https://issues.apache.org/jira/browse/LUCENE-7714">https://issues.apache.org/jira/browse/LUCENE-7714</a>). We wrote a first prototype that shows an improvement over standard range queries. Another interesting aspect of this optimization is the fact that it enables searching on a numeric field without using the KD tree. So our users who take disk usage seriously could be interested in this feature regardless of whether this performs faster or slower than range queries on KD trees.
</p><h4 dir="ltr">Encrypted Snapshots
</h4><p dir="ltr">As part of our ongoing plan to make it easier and safer to snapshot the security index we want to have a well supported option for storing encrypted snapshots. We currently support server-side encryption at rest (when the cloud provider can do so transparently) but we don’t offer anything client-side, and we don’t have a solution for file system based repositories.
</p><p dir="ltr">We’re a good way through our<a href="https://github.com/elastic/elasticsearch/issues/41910"> thinking about how this would look</a>, but we need to make sure it will be a good fit for our users needs, and that we can build something that will last.
</p><h4 dir="ltr">Rollup GA
</h4><p dir="ltr">We opened an issue detailing what we need for<a href="https://github.com/elastic/elasticsearch/issues/42720"> Rollup GA</a>.  GA work has commenced with on one of the more straightforward items, allowing the DeleteJob API to also delete the rolled-up data.  Today this requires manual user involvement and it is not user-friendly: a delete-by-query followed by updating the <code>_meta</code> mapping field.  This will instead be handled by the DeleteJob API with an optional flag to also delete the data when the job is deleted. The rest of the Rollup GA items are of a similar nature, improvements to management ergonomics.
</p><h4 dir="ltr">Snapshot Resiliency - it's all about hygiene
</h4><p dir="ltr">The snapshot resiliency effort currently focuses on the problem of left-over snapshot files not being cleaned up, resulting in large amounts of left-over files for repositories that have actively been used over longer periods of time. Left-over files can be caused by both snapshot deletions that failed mid-way through as well as snapshots that did not successfully complete. There can be plenty of reasons for these operations to fail, from those that are under Elasticsearch's control, such as coordinating repository access among the nodes in the cluster, and those that are outside of ES’s control, e.g. machines crashing or connectivity issues with the service where snapshots are stored. 
</p><p dir="ltr">Many of the resiliency-related issues that were under Elasticsearch's control have been fixed in recent versions (6.4+). ES will also have to do clean-ups for cases where things went wrong outside of ES's control, as well as do clean-ups for historical ES versions that might have accumulated a lot of left-over files. We are focusing on adding clean-up functionality of left-over files that will be done automatically by future ES versions during regular snapshot operations, and working on a tool that can be periodically run by our Cloud team to clean up repositories that are actively being snapshotted to by older ES versions without the automatic clean-up support.
</p><p dir="ltr">Last week's snapshot resiliency sync focused on the performance characteristics of the auto-clean-up functionality that will be coming in newer ES versions, given that existing repositories might have accumulated a large number of left-over files. We did some preliminary benchmarking of the functionality and it appeared that the main issue would be with Azure since the Azure blob store does not allow for bulk deleting multiple blobs in a single request. Given that Azure does not provide a native API for bulk deletes (unlike S3 and GCS), we decided to approach the problem by adding a thread-pool to the Azure plugin to parallelize delete operations on Azure, which we implemented<a href="https://github.com/elastic/elasticsearch/pull/42783"> here</a>.
</p><p dir="ltr">We also added support for<a href="https://github.com/elastic/elasticsearch/pull/42653"> listing subdirectories on all our repository implementations</a>, which is a prerequisite for efficiently finding left-over index folders. While blob stores, in contrast to file systems, generally do not provide a hierarchical view of files, most implementations (S3, GCP, Azure) provide virtual directory support by allowing certain operations to treat files that share common prefixes with configurable delimiters as belonging to the same virtual directory. We have also beefed up the testing for this, which we need to do against the actual Cloud service in order to be confident that these operations are implemented correctly.
</p><p dir="ltr">We prepared a<a href="https://docs.google.com/document/d/1Jd0CCmW3VYEUG2PC6yCUW_kgDhwUFIJ8dt0GikZt5mY/edit#"> document</a> that outlines the options for implementing a tool for removing left-over snapshot files on Cloud. In contrast to the automated clean-up that is being implemented in Elasticsearch and that will be running as part of the regular snapshot lifecycle, the Cloud clean-up tool can make no assumption about concurrently running snapshots, as that would interfere with existing snapshot operations on Cloud and require complex coordination between the snapshot process triggered by clusters on Cloud and the clean-up tool that will be run on different infrastructure. This means however that this tool needs to take a different algorithmic approach for detecting left-over files, as it might otherwise not correctly distinguish between files that are left-over and those that are in the process of being snapshotted by a concurrently running snapshot process. We've explored two options. The first one relies on file timestamps to be somewhat accurate, which is an option when limiting ourselves to the three Cloud services (S3, GCS, Azure). A second option requires a multi-pass approach, where the tool will require a second run after a certain period of time has passed, having the tool account for the time passed between the runs. We are looking at the operational impact of both approaches, the testing aspect as well as how to package the clean-up tool so that it can be easily run on Cloud infrastructure.
</p><h2 dir="ltr">Apache Lucene
</h2><h4 dir="ltr">Change of score for fuzzy queries
</h4><p dir="ltr">Fuzzy queries are currently rewritten as a disjunction, which has the downside that documents that contain multiple terms from the rewritten query will score better than documents that contain multiple occurrences of one term. For instance <code>foobar~1</code> could be rewritten as <code>fobar OR foobaz</code>, and a document that has one occurrence of <code>fobar</code> and another one of <code>foobaz</code> will likely get a better score than a document that has 2 occurrences of <code>fobar</code>.<a href="https://issues.apache.org/jira/browse/LUCENE-8840">We argue that a better way to score fuzzy queries would be to use a SynonymQuery</a>, which would sum up term frequency across all matched terms in order to compute the score.
</p><h4 dir="ltr">Other
</h4><ul><li>We iterated on two-phase support for disjunctions. This change would help disjunctions run faster when they include phrase queries for instance.
</li><li>We are discussing how<a href="https://issues.apache.org/jira/browse/LUCENE-8811"> query visitors can be used in order to count the number of clauses of a query</a> in order to prevent running queries that have too many clauses.
</li><li>We are discussing a<a href="https://issues.apache.org/jira/browse/LUCENE-8833"> better way to enable preloading on MMapDirectory</a>. Currently enabling preloading on a subset of the index files requires using a FileSwitchDirectory. Maybe we could make this easier to use, and also automate based on the I/O context so that users would not have to be aware of the extensions that are used by codec formats.
</li><li>We are<a href="https://issues.apache.org/jira/browse/LUCENE-8835"> fixing FileSwitchDirectory</a> so that wrapping two directories that have the same content (but behave differently, eg. mmap vs. nio) doesn't cause issues with pending deletes.
</li><li>Should the new<a href="https://issues.apache.org/jira/browse/LUCENE-8791"> collector-based rescorer</a> expose a way to parallelize rescoring across multiple threads?
</li><li>We found a<a href="https://issues.apache.org/jira/browse/LUCENE-8838"> bug in the tessellator</a>, which is due to support of<a href="https://en.wikipedia.org/wiki/Steiner_point_(triangle)"> Steiner points</a>, a feature that we don't use.
</li><li>We are looking into a<a href="https://issues.apache.org/jira/browse/LUCENE-8828"> different way to expose the ability to find intervals that don't overlap</a> since the current implementation proved buggy.
</li><li>There is potential to<a href="https://issues.apache.org/jira/browse/LUCENE-8836"> seek the terms dictionary of doc values more efficiently if two consecutive seeks would go to the same block</a>, would it be useful to some of our workloads?
</li><li>We<a href="https://issues.apache.org/jira/browse/LUCENE-8775"> made tessellation more robust</a>.
</li><li>We fixed<a href="https://issues.apache.org/jira/browse/LUCENE-8831"> LatLonShapeBoundingBoxQuery#hashCode</a> is computed. It would include the system hashcode, which would in-turn make almost every object have a different hashcode.
</li><li>A change allows<a href="https://issues.apache.org/jira/browse/LUCENE-8803"> sorting by a FeatureField</a> was merged, so he followed up with a new change that allows creating a<a href="https://issues.apache.org/jira/browse/LUCENE-8815"> DoubleValuesSource over feature fields</a>. A benefit to Elasticsearch users is that they would be able to sort and aggregate by <code>rank_feature(s)</code> fields without having to introduce a <code>float</code> sub field.
</li></ul><h4 dir="ltr">Changes in Elasticsearch</h4><p dir="ltr">Changes in 8.0:
</p><ul>
	<li dir="ltr">Remove the transport client<a href="https://github.com/elastic/elasticsearch/pull/42538"> #42538</a></li>
	<li dir="ltr">Skip shadow jar logic for javadoc and sources jars<a href="https://github.com/elastic/elasticsearch/pull/42904"> #42904</a></li>
	<li dir="ltr">BREAKING: Removes type from TermVectors APIs<a href="https://github.com/elastic/elasticsearch/pull/42198"> #42198</a></li>
	<li dir="ltr">Make high level rest client a fat jar<a href="https://github.com/elastic/elasticsearch/pull/42771"> #42771</a></li>
	<li dir="ltr">BREAKING: RollupStart endpoint should return OK if job already started<a href="https://github.com/elastic/elasticsearch/pull/41502"> #41502</a></li>
</ul><p dir="ltr">Changes in 7.3:
</p><ul>
	<li dir="ltr">Reindex max_docs parameter name<a href="https://github.com/elastic/elasticsearch/pull/41894"> #41894</a></li>
	<li dir="ltr">Deprecation info for joda-java migration on 7.x<a href="https://github.com/elastic/elasticsearch/pull/42659"> #42659</a></li>
	<li dir="ltr">Add a merge policy that prunes ID postings for soft-deleted but retained documents<a href="https://github.com/elastic/elasticsearch/pull/40741"> #40741</a></li>
	<li dir="ltr">Omit JDK sources archive from bundled JDK<a href="https://github.com/elastic/elasticsearch/pull/42821"> #42821</a></li>
	<li dir="ltr">Add custom metadata to snapshots<a href="https://github.com/elastic/elasticsearch/pull/41281"> #41281</a></li>
	<li dir="ltr">Fix Infinite Loops in ExceptionsHelper#unwrap<a href="https://github.com/elastic/elasticsearch/pull/42716"> #42716</a></li>
	<li dir="ltr">Add Ability to List Child Containers to BlobContainer<a href="https://github.com/elastic/elasticsearch/pull/42653"> #42653</a></li>
	<li dir="ltr">Enable console audit logs for docker<a href="https://github.com/elastic/elasticsearch/pull/42671"> #42671</a></li>
	<li dir="ltr">Enable Parallel Deletes in Azure Repository<a href="https://github.com/elastic/elasticsearch/pull/42783"> #42783</a></li>
	<li dir="ltr">Deduplicate alias and concrete fields in query field expansion<a href="https://github.com/elastic/elasticsearch/pull/42328"> #42328</a></li>
	<li dir="ltr">Permit API Keys on Basic License<a href="https://github.com/elastic/elasticsearch/pull/42787"> #42787</a></li>
	<li dir="ltr">Replicate aliases in cross-cluster replication<a href="https://github.com/elastic/elasticsearch/pull/41815"> #41815</a></li>
	<li dir="ltr">Eclipse libs projects setup fix<a href="https://github.com/elastic/elasticsearch/pull/42852"> #42852</a></li>
	<li dir="ltr">Remove unnecessary usage of Gradle dependency substitution rules<a href="https://github.com/elastic/elasticsearch/pull/42773"> #42773</a></li>
	<li dir="ltr">Fix error with test conventions on tasks that require Docker<a href="https://github.com/elastic/elasticsearch/pull/42719"> #42719</a></li>
	<li dir="ltr">Remove "template" field in IndexTemplateMetaData<a href="https://github.com/elastic/elasticsearch/pull/42099"> #42099</a></li>
	<li dir="ltr">Read the default pipeline for bulk upsert through an alias<a href="https://github.com/elastic/elasticsearch/pull/41963"> #41963</a></li>
</ul><p dir="ltr">Changes in 7.2:
</p><ul>
	<li dir="ltr">Skip installation of pre-bundled integ-test modules<a href="https://github.com/elastic/elasticsearch/pull/42900"> #42900</a></li>
	<li dir="ltr">Fix NPE when rejecting bulk updates<a href="https://github.com/elastic/elasticsearch/pull/42923"> #42923</a></li>
	<li dir="ltr">Use reader attributes to control term dict memory useage<a href="https://github.com/elastic/elasticsearch/pull/42838"> #42838</a></li>
	<li dir="ltr">Avoid clobbering shared testcluster JAR files when installing modules<a href="https://github.com/elastic/elasticsearch/pull/42879"> #42879</a></li>
	<li dir="ltr">NullPointerException when creating a watch with Jira action (#41922)<a href="https://github.com/elastic/elasticsearch/pull/42081"> #42081</a></li>
	<li dir="ltr">Don't require TLS for single node clusters<a href="https://github.com/elastic/elasticsearch/pull/42826"> #42826</a></li>
</ul><p dir="ltr">Changes in 6.8:
</p><ul>
	<li dir="ltr">Fix concurrent search and index delete<a href="https://github.com/elastic/elasticsearch/pull/42621"> #42621</a></li>
	<li dir="ltr">Wire query cache into sorting nested-filter computation<a href="https://github.com/elastic/elasticsearch/pull/42906"> #42906</a></li>
	<li dir="ltr">Enable testing against JDK 13 EA builds<a href="https://github.com/elastic/elasticsearch/pull/40829"> #40829</a></li>
	<li dir="ltr">Fixes a bug in AnalyzeRequest.toXContent()<a href="https://github.com/elastic/elasticsearch/pull/42795"> #42795</a></li>
</ul><h4 dir="ltr">Changes in Elasticsearch Management UI</h4><p dir="ltr">Changes in 7.3:
</p><ul>
	<li dir="ltr">Add repository-azure autocompletion settings<a href="https://github.com/elastic/kibana/pull/37935"> #37935</a></li>
</ul><h4 dir="ltr">Changes in Rally</h4><p dir="ltr">Changes in 1.2.0:
</p><ul>
	<li dir="ltr">Add Rally Docker image to release process<a href="https://github.com/elastic/rally/pull/702"> #702</a></li>
	<li dir="ltr">Add download subcommand<a href="https://github.com/elastic/rally/pull/704"> #704</a></li>
	<li dir="ltr">Provide default for datastore.secure in all cases<a href="https://github.com/elastic/rally/pull/705"> #705</a></li>
</ul>

This Week in Elasticsearch and Apache Lucene - 2019-06-07

<h2 dir="ltr" id="docs-internal-guid-d420c956-7fff-f3d8-a739-a94934c17e0c">Elasticsearch Highlights
</h2><h4 dir="ltr">Deprecation Info for Joda to Java Time Migration
</h4><p dir="ltr">We have been<a href="https://github.com/elastic/elasticsearch/issues/42010"> implementing a deprecation info api for joda-java date</a> changes that are not compatible from 7.0 on. This notifications will inform a user that they need to update their mappings and/or pipelines if they contain patterns that are not compatible.
</p><h4 dir="ltr">Enrich
</h4><p dir="ltr">We are working on adding an enrich cluster privilege for managing enrich policies and executing enrich policies. (<a href="https://github.com/elastic/elasticsearch/pull/42677">#42677</a>)
</p><p dir="ltr">We worked on an improvement for the enrich processor by storing the content used for enrichment in a doc values field via a special meta field mapper instead of _source stored field. This tradeoff is beneficial for enrich, because retrieval speed is more important than optimal storage. (<a href="https://github.com/elastic/elasticsearch/pull/42423">#42423</a>)
</p><p dir="ltr">We added a PR for performing validation on nested mapping fields (<a href="https://github.com/elastic/elasticsearch/pull/42452">#42452</a>) and one for limiting the number of concurrent policy executions (<a href="https://github.com/elastic/elasticsearch/pull/42535">#42535</a>)
</p><h4 dir="ltr">Optimize sort on numeric fields
</h4><p dir="ltr">We have a<a href="https://github.com/elastic/elasticsearch/pull/39770"> PR</a> to apply <a href="https://www.elastic.co/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand">max-score optimizations</a> on queries that sort documents by a numeric field.  Currently all documents that match the query must be evaluated when sorting by a field other than score (relevancy) even if the total hits is not tracked. We have optimizations in place if the sort order is congruent with the index sort, but this implies that the index is sorted, which has a cost at indexing time, and the speedup only applies to a single case.
</p><p dir="ltr">The new approach works for all date and long fields for both ascending and descending order. It’s available by default as long as the field is indexed and doc-values are activated. It uses the LongDistanceFeatureQuery, which is used when sorting by relevance to boost by proximity or by recency, under the hood to skip documents that are not competitive with the same mechanism that is used when sorting by relevance.
</p><p dir="ltr">The initial performance tests showed great improvements, 6-7x, in some cases, and big regressions in others. This was expected since the optimization depends on the cardinality of the field and the repartition of these values within the segments/shard. For these reasons we are working on a heuristic to detect if this optimization should be applied or not depending on the values indexed for the field. We also discussed some ways to randomize the repartition of documents within a shard in order to ensure documents with the greatest/smallest values are not all on the same block. This is the case currently for time-based indices where recent documents appear at the end of each segment/shard so sorting by most recent is slower when this new optimization is applied.
</p><p dir="ltr">We also created a rally-tracks<a href="https://github.com/elastic/rally-tracks/pull/77"> PR</a> to benchmark numeric sort for some of our datasets, this will help to track the progress of this work since we're planning to have multiple iterations on this feature.
</p><h4 dir="ltr">Long running queries
</h4><p dir="ltr">We continued to work on improving the support for long running queries, opening a<a href="https://github.com/elastic/elasticsearch/pull/42510"> PR</a> to separate frozen and non-frozen indices during search in order to minimize the cost of waiting for the response on slow indices (frozen). This sparked an idea on how we could segregate searches based on the type of the indices/shards; a shard that didn't receive any updates for some time could be considered as read-only during the search.
</p><h4 dir="ltr">TLS Error Message Improvements
</h4><p dir="ltr">The default JRE SSL/TLS provider often doesn’t provide error the error messages we’d like; messages like “PKIX path building failed”. Even when the message is somewhat descriptive like “no cipher suites in common”, there isn’t enough information to actually solve the problem; listing the configured cipher suites would be a start. Some time ago, we started the<a href="http://github.com/elastic/tealess"> Tealess</a> project to try and provide useful diagnostics when TLS connections go bad. We’ve<a href="https://github.com/elastic/elasticsearch/pull/42692"> started exploring</a> how we can use some of the ideas from Tealess in Elasticsearch so that the error messages we provide can be more meaningful to end users and guide them towards solutions. It’s early days and we’re not sure whether the current prototype is the direction we want to take, but it’s our current focus for trying to make security simpler.
</p><h4 dir="ltr">ReindexV2
</h4><p dir="ltr">We’re discussing how to make reindexing resilient to both data node and coordinator node failures. For data node resiliency, we can query our source data sorted by sequence numbers to allow reindex to resume after a failure. There will be no inter-object consistency guarantees in case of concurrent updates to the source index during reindex, the assumption being that the vast majority of reindex operations are done from indices that are effectively read-only during the operation.
</p><p dir="ltr">We created a<a href="https://github.com/elastic/elasticsearch/issues/42612"> meta-issue</a> to track the work on the reindex resiliency-related items.
</p><p dir="ltr">We<a href="https://github.com/elastic/elasticsearch/pull/42412"> fixed</a> a<a href="https://github.com/elastic/elasticsearch/issues/31908"> long-standing issue</a> where reindex from remote was not working with external versioning.
</p><h4 dir="ltr">Distributed: Investigating test failures
</h4><p dir="ltr">We are working hard at reducing the number of open test failures (~50 beginning of May, ~30 last week, 22 now). Addressing some of these failures takes significant effort. To name a few:
</p><ul>
	<li dir="ltr">We found a<a href="https://github.com/elastic/elasticsearch/pull/42600"> test bug</a> that was causing a lot of noise in our CI.</li>
	<li dir="ltr">We investigated a<a href="https://github.com/elastic/elasticsearch/issues/38331"> test failure</a> which resulted in an<a href="https://github.com/elastic/elasticsearch/pull/42454"> adaptation</a> of the cluster state APIs to always return a cluster state that has a master.</li>
	<li dir="ltr">We investigated a<a href="https://github.com/elastic/elasticsearch/issues/42244"> test failure</a> where the test could hang as our simulated network disruption of type "unresponsive" would silently swallow requests, breaking the premise that all transport requests eventually respond, either successfully or with a failure. We proposed a number of solutions to make the simulated network disruptions more realistic,.</li>
	<li dir="ltr">We investigated a<a href="https://github.com/elastic/elasticsearch/issues/39172"> test failure</a> where a cluster would get stuck in a state where state recovery would not be triggered, and<a href="https://github.com/elastic/elasticsearch/pull/42576"> fixed</a> a bug which prevented state recovery to run after a very unfortunate series of events.</li>
</ul><h2 dir="ltr">Apache Lucene</h2><h4 dir="ltr">Geo-Land
</h4><p dir="ltr">We are working on several improvements, all in review at this point:
</p><ul>
	<li dir="ltr">Update tessellator logic to label if a triangle edge belongs to the original polygon. This is necessary to implement CONTAINS for LatLonShape. </li>
	<li dir="ltr"><a href="https://github.com/apache/lucene-solr/pull/587">Adding</a> LatLonShape distance query.</li>
	<li dir="ltr"><a href="https://github.com/apache/lucene-solr/pull/627">Refactoring</a> of EdgeTree class to support different types of components. This will allow to support geometry collections in one query,</li>
	<li dir="ltr"><a href="https://github.com/apache/lucene-solr/pull/652">Fixing</a> some corner cases in the tessellator, in particular some failures when holes share a vertex with the polygon.</li>
</ul><h4 dir="ltr">Analysis
</h4><p dir="ltr">We collaborated on a<a href="https://issues.apache.org/jira/browse/LUCENE-8784"> patch</a> to add an option in the Korean Tokenizer to preserves punctuation tokens
</p><p dir="ltr">This option already exists in the Japanese tokenizer to allow subsequent token filter to normalize Japanese numbers (kansūji) into decimal numbers. Since Korean numbers are similar, we added this new option in the Tokenizer and we are now iterating on<a href="https://issues.apache.org/jira/browse/LUCENE-8812"> adding</a> a KoreanNumberFilter similar to the JapaneseNumberFilter:
</p><h4 dir="ltr">Performance
</h4><p dir="ltr">We are also<a href="https://issues.apache.org/jira/browse/LUCENE-8806"> working</a> on supporting two-phase iterators in the WANDScorer (boolean disjunctions). Disjunctions of clauses that use two-phase iterators (phrase queries, script, ...) are slower in Lucene 8 so this change should fix the regression.
</p><h4 dir="ltr">Bugs
</h4><p dir="ltr">One of our CI builds tripped an<a href="https://issues.apache.org/jira/browse/LUCENE-8813"> assertion</a> that looked critical but turned out to be a rather theoretical where two consecutive commits are faster than applying an empty delete queue. We<a href="https://github.com/apache/lucene-solr/pull/688"> fixed</a> the issue and improved correctness on applying deletes to protect from any future issues in this area since it's crucial for data integrity.
</p><h4 dir="ltr">Misc
</h4><ul>
	<li dir="ltr">The<a href="https://issues.apache.org/jira/browse/LUCENE-8362"> work</a> on docvalues support for  {Int|Long|Float|Double}Range fields continues.</li>
</ul><h2 dir="ltr">Changes in Elasticsearch</h2><p dir="ltr">Changes in 8.0:
</p><ul>
	<li dir="ltr">BREAKING: Remove client jar support from build<a href="https://github.com/elastic/elasticsearch/pull/42640"> #42640</a></li>
	<li dir="ltr">Remove SecurityClient from x-pack<a href="https://github.com/elastic/elasticsearch/pull/42471"> #42471</a></li>
	<li dir="ltr">Add explicit build flag for experimenting with test execution cacheability<a href="https://github.com/elastic/elasticsearch/pull/42649"> #42649</a></li>
	<li dir="ltr">Removes types from SearchRequest and QueryShardContext<a href="https://github.com/elastic/elasticsearch/pull/42112"> #42112</a></li>
	<li dir="ltr">BREAKING: Remove "nodes/0" folder prefix from data path<a href="https://github.com/elastic/elasticsearch/pull/42489"> #42489</a></li>
	<li dir="ltr">BREAKING: Remove support for chained multi-fields.<a href="https://github.com/elastic/elasticsearch/pull/42333"> #42333</a></li>
</ul><p dir="ltr">Changes in 7.3:
</p><ul>
	<li dir="ltr">Deprecate CommonTermsQuery and cutoff_frequency<a href="https://github.com/elastic/elasticsearch/pull/42619"> #42619</a></li>
	<li dir="ltr">Remove unused "messy-test" plugin<a href="https://github.com/elastic/elasticsearch/pull/42684"> #42684</a></li>
	<li dir="ltr">Prevent merging nodes' data paths<a href="https://github.com/elastic/elasticsearch/pull/42665"> #42665</a></li>
	<li dir="ltr">Remove usage of deprecated compare gradle builds plugin<a href="https://github.com/elastic/elasticsearch/pull/42687"> #42687</a></li>
	<li dir="ltr">[ML] [Data Frame] add support for weighted_avg agg<a href="https://github.com/elastic/elasticsearch/pull/42646"> #42646</a></li>
	<li dir="ltr">Fix Class Load Order in Netty4Plugin<a href="https://github.com/elastic/elasticsearch/pull/42591"> #42591</a></li>
	<li dir="ltr">Log leader and handshake failures by default<a href="https://github.com/elastic/elasticsearch/pull/42342"> #42342</a></li>
	<li dir="ltr">Detect when security index is closed<a href="https://github.com/elastic/elasticsearch/pull/42191"> #42191</a></li>
	<li dir="ltr">Allow aggregations using expressions to use _score<a href="https://github.com/elastic/elasticsearch/pull/42652"> #42652</a></li>
	<li dir="ltr">Lazily compute Java 8 home in reindex configuration<a href="https://github.com/elastic/elasticsearch/pull/42630"> #42630</a></li>
	<li dir="ltr">Validate routing commands using updated routing state<a href="https://github.com/elastic/elasticsearch/pull/42066"> #42066</a></li>
	<li dir="ltr">Improve how internal representation of pipelines are updated<a href="https://github.com/elastic/elasticsearch/pull/42257"> #42257</a></li>
	<li dir="ltr">Upgrade to Netty 4.1.36<a href="https://github.com/elastic/elasticsearch/pull/42543"> #42543</a></li>
	<li dir="ltr">Shard CLI tool always check shards<a href="https://github.com/elastic/elasticsearch/pull/41480"> #41480</a></li>
	<li dir="ltr">Cluster state from API should always have a master<a href="https://github.com/elastic/elasticsearch/pull/42454"> #42454</a></li>
</ul><p dir="ltr">Changes in 7.2:
</p><ul>
	<li dir="ltr">Fix refresh remote JWKS logic<a href="https://github.com/elastic/elasticsearch/pull/42662"> #42662</a></li>
	<li dir="ltr">Propagate version in reindex from remote search<a href="https://github.com/elastic/elasticsearch/pull/42412"> #42412</a></li>
	<li dir="ltr">[ML-DataFrame] rewrite start and stop to answer with acknowledged<a href="https://github.com/elastic/elasticsearch/pull/42589"> #42589</a></li>
	<li dir="ltr">[ML Data Frame] Set DF task state to stopped when stopping<a href="https://github.com/elastic/elasticsearch/pull/42516"> #42516</a></li>
	<li dir="ltr">Avoid loading retention leases while writing them<a href="https://github.com/elastic/elasticsearch/pull/42620"> #42620</a></li>
	<li dir="ltr">Reset state recovery after successful recovery<a href="https://github.com/elastic/elasticsearch/pull/42576"> #42576</a></li>
	<li dir="ltr">[ML-DataFrame] add support for fixed_interval, calendar_interval, remove interval<a href="https://github.com/elastic/elasticsearch/pull/42427"> #42427</a></li>
	<li dir="ltr">Remove IndexStore and DirectoryService<a href="https://github.com/elastic/elasticsearch/pull/42446"> #42446</a></li>
	<li dir="ltr">Build cache init script<a href="https://github.com/elastic/elasticsearch/pull/42484"> #42484</a></li>
</ul><p dir="ltr">Changes in 6.8:
</p><ul>
	<li dir="ltr">Percolator: Exclude nested documents<a href="https://github.com/elastic/elasticsearch/pull/42554"> #42554</a></li>
	<li dir="ltr">Deprecation info for joda-java migration<a href="https://github.com/elastic/elasticsearch/pull/41956"> #41956</a></li>
	<li dir="ltr">un-mute Watcher rolling upgrade tests and bump up logging<a href="https://github.com/elastic/elasticsearch/pull/42377"> #42377</a></li>
	<li dir="ltr">Added param ignore_throttled=false when indicesOptions.ignoreThrottle…<a href="https://github.com/elastic/elasticsearch/pull/42393"> #42393</a></li>
	<li dir="ltr">Only index into "doc" type in security index<a href="https://github.com/elastic/elasticsearch/pull/42563"> #42563</a></li>
	<li dir="ltr">fixed ignoring name parameter for percolator queries<a href="https://github.com/elastic/elasticsearch/pull/42598"> #42598</a></li>
	<li dir="ltr">Do not use ifSeqNo for update requests on mixed cluster<a href="https://github.com/elastic/elasticsearch/pull/42596"> #42596</a></li>
	<li dir="ltr">Address test failures for SmokeTestWatcherWithSecurityIT<a href="https://github.com/elastic/elasticsearch/pull/42092"> #42092</a></li>
	<li dir="ltr">Fix sorting on nested field with unmapped<a href="https://github.com/elastic/elasticsearch/pull/42451"> #42451</a></li>
</ul>

This Week in Elasticsearch and Apache Lucene - 2019-05-31

This Week in Elasticsearch and Apache Lucene - 2019-05-24

This Week in Elasticsearch and Apache Lucene - 2019-04-26

This Week in Elasticsearch and Apache Lucene - 2019-05-05

Week in Elasticsearch and Apache Lucene - 2019-04-12

This Week in Elasticsearch and Apache Lucene - 2019-04-05

This Week in Elasticsearch and Apache Lucene - 2019-03-29

<html><head></head><body><h3>Elasticsearch</h3><h4>Remote clusters UI
	</h4><p>Work continues on the remote clusters UI. We <a href="https://github.com/elastic/kibana/pull/25695">have</a> <a href="https://github.com/elastic/kibana/pull/25773">finished</a> <a href="https://github.com/elastic/kibana/pull/25833">the UI</a> <a href="https://github.com/elastic/kibana/pull/25837">layer</a>, and have been working on the <a href="https://github.com/elastic/kibana/pull/25632">app layer</a> <a href="https://github.com/elastic/kibana/pull/25688">to integrate</a> <a href="https://github.com/elastic/kibana/pull/25782">the UI</a> <a href="https://github.com/elastic/kibana/pull/25835">with endpoints</a>. So far the work has been done based on the assumption that add/edit/delete are all persistent settings. Next, we will work on handling transient settings as well as settings that come from elasticsearch.yml.
</p><h4>Index lifecycle management UI
</h4><p>Work on this UI is progressing nicely.  The flows from the index management app to add a policy to and remove a policy from an index are completed. We also added the ability to see <a href="https://github.com/elastic/kibana/pull/25553">the error stack trace for lifecycle errors and to see the phase definition</a>. Remaining tasks include adding the ability to disable the UI, adding a button to add a policy to an index template from the policy CRUD UI, adding filter buttons in index management to filter on managed/unmanaged and lifecycle phase, and <a href="https://github.com/elastic/kibana/pull/22768">updating autocomplete rules for console to include new ILM APIs</a>.
</p><h4>Query Script Query
</h4><p>There is a new query called <a href="https://github.com/elastic/elasticsearch/pull/34533">script_score</a> that allows users to manipulate the score of documents through a Painless script. The query also includes a script context which exposes a number of useful methods including the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#function-decay">Decay functions</a> of the function_score query.  This new query is the first part of the work to <a href="https://github.com/elastic/elasticsearch/issues/27588">deprecate and remove the function_score query</a> in favour of easier to use alternatives.
</p><h4>Rare terms aggregation
</h4><p>We have a PR for a new <a href="https://github.com/elastic/elasticsearch/pull/35718">aggregation</a> that is designed to surface rare terms. This is designed to be used instead of ordering a terms agg by count ascending which has an unbounded error. The rare terms aggregation uses a different strategy to merge the results from different shards to ensure a better precision.
</p><h4>Build
</h4><p>As every PR against our repo is built on our CI infrastructure as part of the review process, it is very important to have fast feedback cycles. By <a href="https://github.com/elastic/elasticsearch/pull/35473">splitting the build</a> and running it on multiple worker nodes we were able to reduce the total PR build time by 50% down to 90 minutes. We have also enabled <a href="https://gradle.com/build-scans/">Gradle build scans</a> so we can find more room for improvements.
</p><h4>Backup of Security
</h4><p>It&#x2019;s currently possible to backup the .security index, but it&#x2019;s not an ideal experience. We have <a href="https://github.com/elastic/elasticsearch/issues/34454">a plan of tasks</a> we want to complete to improve this experience and we have started working through this list.
</p><p>We have opened <a href="https://github.com/elastic/elasticsearch/pull/35820">a PR for a &#x201C;snapshot&#x201D; privilege</a> and have been thinking about how to allow some low-privileged users the ability to list the .security index, and potentially see some metadata about it, but not necessarily be able to search it, or get documents from it.
</p><h4>Zen 2 
</h4><p>We now have a storage layer that can <a href="https://github.com/elastic/elasticsearch/pull/35049">atomically write cluster states</a>, and the <a href="https://github.com/elastic/elasticsearch/issues/33958">remaining follow-up work</a> will focus on integrating the new storage layer with the existing Zen2 components.
</p><p>We also fixed a long-standing issue where nodes miss applying a cluster state update, indefinitely lagging behind the master in terms of applied cluster states. This can potentially impact other operations in the cluster, for example block indexing operations to wait on mapping updates. The solution is for the master to <a href="https://github.com/elastic/elasticsearch/pull/35685">track applied cluster state versions</a> of each node and to kick a node out of the cluster if it doesn&apos;t apply a published cluster state or any of its follow-up published states within some timeout, giving us a strong bound on how stale the cluster state applied on every node in the cluster can be.
</p><h4>Cross Cluster Replication
</h4><p>We are also continuing to <a href="https://github.com/elastic/elasticsearch/pull/35594">investigate</a> and benchmark update- and refresh-heavy use cases, to see if the number of refreshes (and merges) can be improved without ruining the indexing throughput. This work has led to two Lucene changes to improve lock contention when indexing happens concurrently to refresh / open reader calls: 1) <a href="https://issues.apache.org/jira/browse/LUCENE-8569">Improve merge performance</a> in the common case where soft- and hard-deletes are not mixed and 2) <a href="https://issues.apache.org/jira/browse/LUCENE-8571">prevent blocking on resolving deletes</a>.
</p><p>The work to implement &quot;recovery from remote&quot; by <a href="https://github.com/elastic/elasticsearch/pull/35719">bootstrapping</a> the creation of the follower index through the Snapshot/Restore functionality has started by prototyping a CCR Snapshot/Restore repository that gets <a href="https://github.com/elastic/elasticsearch/pull/35801">registered based on available remote clusters</a>.
</p><h4>Changes
</h4><p>Changes in 6.4:
</p><ul>
	<li>[CI] Reactivate 3rd party tests on CI [branch 6.4] <a href="https://github.com/elastic/elasticsearch/pull/35733">#35733</a></li>
</ul><p>Changes in 6.5:
</p><ul>
	<li>Avoid NPE in follower stats when no tasks metadata <a href="https://github.com/elastic/elasticsearch/pull/35802">#35802</a></li>
	<li>[Scripting] Use Number as a return value for BucketAggregationScript <a href="https://github.com/elastic/elasticsearch/pull/35653">#35653</a></li>
	<li>SQL: Improve validation of unsupported fields <a href="https://github.com/elastic/elasticsearch/pull/35675">#35675</a></li>
</ul><p>Changes in 6.6:
</p><ul>
	<li>BREAKING: Forbid negative scores in functon_score query <a href="https://github.com/elastic/elasticsearch/pull/35709">#35709</a></li>
	<li>SQL: Implement NVL(expr1, expr2) <a href="https://github.com/elastic/elasticsearch/pull/35794">#35794</a></li>
	<li>TransportResyncReplicationAction should not honour blocks <a href="https://github.com/elastic/elasticsearch/pull/35795">#35795</a></li>
	<li>GEO: More robust handling of ignore_malformed in geoshape parsing <a href="https://github.com/elastic/elasticsearch/pull/35603">#35603</a></li>
	<li>SQL: Implement ISNULL(expr1, expr2) <a href="https://github.com/elastic/elasticsearch/pull/35793">#35793</a></li>
	<li>HLRC: ML Delete event from Calendar <a href="https://github.com/elastic/elasticsearch/pull/35760">#35760</a></li>
	<li>[HLRC] Fix issue in equals impl for GlobalOperationPrivileges <a href="https://github.com/elastic/elasticsearch/pull/35721">#35721</a></li>
	<li>[GEO] Add support to ShapeBuilders for building Lucene geometry <a href="https://github.com/elastic/elasticsearch/pull/35707">#35707</a></li>
	<li>SQL: Implement IFNULL variant of COALESCE <a href="https://github.com/elastic/elasticsearch/pull/35762">#35762</a></li>
	<li>SQL: Perform lazy evaluation of mismatched mappings <a href="https://github.com/elastic/elasticsearch/pull/35676">#35676</a></li>
	<li>[HLRC] Add support for get application privileges API <a href="https://github.com/elastic/elasticsearch/pull/35556">#35556</a></li>
	<li>[Tests] Fix SimpleQueryStringBuilderTests <a href="https://github.com/elastic/elasticsearch/pull/35784">#35784</a></li>
	<li>manage_token privilege for kibana_system <a href="https://github.com/elastic/elasticsearch/pull/35751">#35751</a></li>
	<li>SQL: Introduce INTERVAL support <a href="https://github.com/elastic/elasticsearch/pull/35521">#35521</a></li>
	<li>[HLRC] Added support for CCR Resume Follow API <a href="https://github.com/elastic/elasticsearch/pull/35638">#35638</a></li>
	<li>Clean up PutLicenseResponse <a href="https://github.com/elastic/elasticsearch/pull/35689">#35689</a></li>
	<li>Clean up StartBasicResponse <a href="https://github.com/elastic/elasticsearch/pull/35688">#35688</a></li>
	<li>Add a prebuilt ICU Analyzer <a href="https://github.com/elastic/elasticsearch/pull/34958">#34958</a></li>
	<li>[HLRC] Added support for CCR Unfollow API <a href="https://github.com/elastic/elasticsearch/pull/35693">#35693</a></li>
	<li>Fix problem with MatchNoDocsQuery in disjunction queries <a href="https://github.com/elastic/elasticsearch/pull/35726">#35726</a></li>
	<li>HLRC: ML Get Calendar Events <a href="https://github.com/elastic/elasticsearch/pull/35747">#35747</a></li>
	<li>HLRC ML Add Event To Calendar API <a href="https://github.com/elastic/elasticsearch/pull/35704">#35704</a></li>
	<li>HLRC: ML Delete job from calendar <a href="https://github.com/elastic/elasticsearch/pull/35713">#35713</a></li>
	<li>Add a _freeze / _unfreeze API <a href="https://github.com/elastic/elasticsearch/pull/35592">#35592</a></li>
	<li>HLRC: Add &quot;_has_privileges&quot; API to Security Client <a href="https://github.com/elastic/elasticsearch/pull/35479">#35479</a></li>
	<li>HLRC for _mtermvectors <a href="https://github.com/elastic/elasticsearch/pull/35266">#35266</a></li>
	<li>HLRC: ML Add Job to Calendar API <a href="https://github.com/elastic/elasticsearch/pull/35666">#35666</a></li>
	<li>Build: Fix official plugins list <a href="https://github.com/elastic/elasticsearch/pull/35661">#35661</a></li>
	<li>HLRC: Add ML delete filter action <a href="https://github.com/elastic/elasticsearch/pull/35382">#35382</a></li>
	<li>SQL: Move internals from Joda to java.time <a href="https://github.com/elastic/elasticsearch/pull/35649">#35649</a></li>
	<li>Fix phrase_slop in query_string query <a href="https://github.com/elastic/elasticsearch/pull/35533">#35533</a></li>
</ul><p>Changes in 7.0:
</p><ul>
	<li>Add a new query type - ScriptScoreQuery <a href="https://github.com/elastic/elasticsearch/pull/34533">#34533</a></li>
	<li>Deprecate types in validate query requests. <a href="https://github.com/elastic/elasticsearch/pull/35575">#35575</a></li>
	<li>Deprecate types in count and msearch. <a href="https://github.com/elastic/elasticsearch/pull/35421">#35421</a></li>
</ul><h3>Lucene
</h3><h4>BM25F
</h4><p>A <a href="https://issues.apache.org/jira/browse/LUCENE-8216">first implementation of BM25F</a> has been pushed to sandbox. BM25F is an extension to BM25 that supports scoring across multiple fields with different degrees of importance. The way it works is quite simple, for instance say that you want to give <code>title</code> a weight of 5 and <code>body</code> a weight of 1 then BM25F will score across these two fields as if there was a single field that consisted of the concatenation of the body field and 5 times the title field. While this first implementation is already helpful, this is currently implemented via a custom query that ignores the similarity that is configured on IndexSearcher. There is still work needed in order to make it more flexible, eg. by accepting different values of b (the BM25 parameter that controls length normalization) per field and making it easier to use. We hope to eventually leverage this functionality to improve scoring in Elasticsearch when using the <code>cross_fields</code> way of scoring across multiple fields.
</p><h4>Benchmarks for LatLonShape
</h4><p>While we would eventually want to benchmark LatLonShape - our new BKD-based support for shapes - with realistic datasets of shapes (polygons, lines, etc), we started simple by running our current benchmark for points with LatLonShape. This already gives us a good indication of how much overhead LatLonShape adds compared with LatLonPoint when only indexing points. See the purple line at <a href="http://people.apache.org/~mikemccand/geobench.html">http://people.apache.org/~mikemccand/geobench.html</a> if you are curious. LatLonShape is still at an early stage of its development and we hope to bridge some of the gap with LatLonPoint in the near future.
</p><h4>Faster merging when indexing geo shapes
</h4><p>We recently enhanced BKD trees so that only a subset of dimensions is indexed, which in-turn helps use BKD trees as a R-tree. We noticed that we are still doing work on non-indexed dimensions which is <a href="https://issues.apache.org/jira/browse/LUCENE-8562">not necessary</a>. Fixing it is expecting to make merging faster for fields that index shapes. We will hopefully soon see a bump in the indexing speed charts (see above paragraph).
</p><h4>Less contention when applying deletes
</h4><p>Deletes may be applied either while indexing or refreshing. Today this is a blocking call, which might block some indexing threads. However blocking is only required when refreshing, in order to make sure that all deletes are reflected in the new point-in-time view of the index. <a href="https://issues.apache.org/jira/browse/LUCENE-8571">an optimization that only makes it blocking for refreshes</a> has been implemented to solve this.
</p><h4>Normalization as a first-class citizen
</h4><p>The ability for analysis chains to only normalize content and skip tokenization, stemming and all other components that don&apos;t work with partial terms (think of wildcard queries) always felt a bit like a second-class citizen as you had to check for whether a component implemented a special marker interface to know whether it should be applied for normalization. This was fixed by <a href="https://issues.apache.org/jira/browse/LUCENE-8497">moving normalization directly to the CharFilter and TokenFilter factory classes</a>.
</p><h4>Resources around Lucene
</h4><p>As highlighted in a <a href="https://twitter.com/jpountz/status/1065534715120226304">recent Twitter conversation</a>, the Lucene wiki has a useful <a href="https://wiki.apache.org/lucene-java/LucenePapers">page</a> that collects various resources that either describe Lucene&apos;s internals or have been influential.
</p><h4>Other
</h4><p> - The Lucene 7.6 release process is on pause during Thanksgiving.
</p><p> - Nori, the korean tokenizer, could hit a <a href="https://issues.apache.org/jira/browse/LUCENE-8570">NullPointerException if the part-of-speech attribute is not set</a>.
</p><p> - Nori doesn&apos;t do a good job with <a href="https://issues.apache.org/jira/browse/LUCENE-8548">non-CJK tokens</a>.
</p><p> - The &quot;flexible&quot; query parser can fail with <a href="https://issues.apache.org/jira/browse/LUCENE-8572">index-out-of-bounds exceptions</a> with non-default locales.
</p><p> - In the general case, counting soft deletes upon merge requires a linear scan. Simon <a href="https://issues.apache.org/jira/browse/LUCENE-8569">optimized the common special case that a segment consists only of soft deletes</a>, as in that case the number of soft deletes is already known via IndexReader#numDocs.
</p></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-11-23

<html><head></head><body><h2 dir="ltr" id="docs-internal-guid-97f4d933-7fff-b05e-c4c1-945fb33fd0f8">Elasticsearch<br></h2><h3 dir="ltr">Highlights
</h3><h4 dir="ltr">Structured Audit Logging
</h4><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/31931">merged structured audit logging</a> to master at the end of last week, and have been <a href="https://github.com/elastic/elasticsearch/pull/33894">working on the backport for 6.5</a>. For 6.x, we have decided to maintain the old (non-structured) audit log alongside the new format, so the backport requires replicating the old format on top of the new code.
</p><h4 dir="ltr">Append only indexing and cross-cluster replication
</h4><p dir="ltr">Users of time based indices usually index documents once and never change them again (metrics, logs etc.), relying on ES to generate ids for them. In ES jargon, this indexing pattern is called append only indexing and we have various optimizations in place to support it. For cross-cluster replication, those optimizations are not available to the &#x201C;following&#x201D; index because from  its perspective the documents already had IDs when they came. Without these optimizations the following index won&apos;t be able to keep up.
</p><p dir="ltr">We have had numerous discussions on how to support those optimizations on the following index and came up with an<a href="https://github.com/elastic/elasticsearch/issues/33656"> exciting plan</a> to automatically detect operations that don&apos;t change any existing documents on the primary, capture this information and transfer it to the replicas and secondarily to the following shards. This means that this optimizations will automatically be enabled on the following indices, even if the indexing operation originally had an ID and thus following indices may be able to catch up faster.
</p><h4 dir="ltr">Java Time migration
</h4><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/33486">reworked Joda time backwards compatibility</a> in scripts so both the Joda and the Java time API can be used at the same time in 6.x, and users get an explicit deprecation warning for each method that will disappear on upgrade to 7.0. We <a href="https://github.com/elastic/elasticsearch/pull/33467">introduced a date formatter</a> interface to be able to parse milliseconds since epoch properly; also each date formatter now has a name which is needed for mapping changes.
</p><h4 dir="ltr">Performance
</h4><p dir="ltr">We have <a href="https://discuss.elastic.co/t/rally-1-0-1-released/148439">released Rally 1.0.1</a> with lots of stability fixes but also a few features like a new shrink index API.
</p><h4 dir="ltr">Types removal
</h4><p dir="ltr">We have resumed work on types removal, coming up with a <a href="https://github.com/elastic/elasticsearch/issues/15613#issuecomment-422833360">plan for REST tests migration</a>, and moving <a href="https://github.com/elastic/elasticsearch/pull/33934">tests</a> and <a href="https://github.com/elastic/elasticsearch/pull/33953">docs</a> to typeless APIs. 
</p><h4 dir="ltr">Annotated text field type
</h4><p dir="ltr">We have merged work on a new field type, annotated_text. The purpose of this field type is to allow identification of entities in documents, without the use of synonyms. This is done as a plugin to elasticsearch, includes a highlighter.
</p><h4 dir="ltr">Token Filters
</h4><p dir="ltr">We have <a href="https://github.com/elastic/elasticsearch/pull/31958">added two scriptable token filters</a>: condition, which will only apply a given filter subchain if the current token matches a predicate; and <a href="https://github.com/elastic/elasticsearch/pull/33431">predicate_token_filter</a>, which will strip out tokens that don&#x2019;t match a predicate. We will  add a further scriptable filter that allows you to edit the byte content of a token, which should reduce the need to write Java plugins for custom analysis chains.
</p><p dir="ltr">We have also reworked how token filters that refer to other filters, or to their predecessors in the token chain, are constructed.  This means that synonym filters now <a href="https://github.com/elastic/elasticsearch/pull/33702">work correctly</a> with conditional or multiplexer filters.
</p><h2 dir="ltr">Changes
</h2><h4 dir="ltr">Changes in 6.4:
</h4><ul>
	<li dir="ltr">Allow to clear the fielddata cache per field<a href="https://github.com/elastic/elasticsearch/pull/33807"> #33807</a></li>
	<li dir="ltr">Watcher: Ensure triggered watch deletion is sync<a href="https://github.com/elastic/elasticsearch/pull/33799"> #33799</a></li>
	<li dir="ltr">Do not override named S3 client credentials<a href="https://github.com/elastic/elasticsearch/pull/33793"> #33793</a></li>
	<li dir="ltr">Skip rebalancing when cluster_concurrent_rebalance threshold reached<a href="https://github.com/elastic/elasticsearch/pull/33329"> #33329</a></li>
</ul><h4 dir="ltr">Changes in 6.5:
</h4><ul>
	<li dir="ltr">Introduce a search_throttled threadpool<a href="https://github.com/elastic/elasticsearch/pull/33732"> #33732</a></li>
	<li dir="ltr">Restore local history from translog on promotion<a href="https://github.com/elastic/elasticsearch/pull/33616"> #33616</a></li>
	<li dir="ltr">HLRC: Add support for reindex rethrottling<a href="https://github.com/elastic/elasticsearch/pull/33832"> #33832</a></li>
	<li dir="ltr">Update geolite2 database in ingest geoip plugin<a href="https://github.com/elastic/elasticsearch/pull/33840"> #33840</a></li>
	<li dir="ltr">ingest: support simulate with verbose for pipeline processor<a href="https://github.com/elastic/elasticsearch/pull/33839"> #33839</a></li>
	<li dir="ltr">SQL: Fix ANTL4 Grammar ambiguities.<a href="https://github.com/elastic/elasticsearch/pull/33854"> #33854</a></li>
	<li dir="ltr">HLRC: Reindex should support requests_per_seconds parameter<a href="https://github.com/elastic/elasticsearch/pull/33808"> #33808</a></li>
	<li dir="ltr">SQL: TRUNCATE and ROUND functions<a href="https://github.com/elastic/elasticsearch/pull/33779"> #33779</a></li>
	<li dir="ltr">Add contains method to LocalCheckpointTracker<a href="https://github.com/elastic/elasticsearch/pull/33871"> #33871</a></li>
	<li dir="ltr">Create a WatchStatus class for the high-level REST client.<a href="https://github.com/elastic/elasticsearch/pull/33527"> #33527</a></li>
	<li dir="ltr">Move CompletionStats into the Engine<a href="https://github.com/elastic/elasticsearch/pull/33847"> #33847</a></li>
	<li dir="ltr">Fix potential NPE in _cat/shards/ with partial CommonStats<a href="https://github.com/elastic/elasticsearch/pull/33858"> #33858</a></li>
	<li dir="ltr">Allow TokenFilterFactories to rewrite themselves against their preceding chain<a href="https://github.com/elastic/elasticsearch/pull/33702"> #33702</a></li>
	<li dir="ltr">[Tests] Nudge wait time in RemoteClusterServiceTests<a href="https://github.com/elastic/elasticsearch/pull/33853"> #33853</a></li>
	<li dir="ltr">Fixing assertions in integration test<a href="https://github.com/elastic/elasticsearch/pull/33833"> #33833</a></li>
	<li dir="ltr">SQL: Fix issue with options for QUERY() and MATCH().<a href="https://github.com/elastic/elasticsearch/pull/33828"> #33828</a></li>
	<li dir="ltr">Move DocsStats into Engine<a href="https://github.com/elastic/elasticsearch/pull/33835"> #33835</a></li>
	<li dir="ltr">Use the global doc id to generate random scores<a href="https://github.com/elastic/elasticsearch/pull/33599"> #33599</a></li>
	<li dir="ltr">Profiler: Don&#x2019;t profile NEXTDOC for ConstantScoreQuery.<a href="https://github.com/elastic/elasticsearch/pull/33196"> #33196</a></li>
	<li dir="ltr">SQL: Better handling of number parsing exceptions<a href="https://github.com/elastic/elasticsearch/pull/33776"> #33776</a></li>
	<li dir="ltr">Ensure fully deleted segments are accounted for correctly<a href="https://github.com/elastic/elasticsearch/pull/33757"> #33757</a></li>
	<li dir="ltr">SQL: Grammar tweak for number declarations<a href="https://github.com/elastic/elasticsearch/pull/33767"> #33767</a></li>
	<li dir="ltr">Watcher: Use Bulkprocessor in HistoryStore/TriggeredWatchStore<a href="https://github.com/elastic/elasticsearch/pull/32490"> #32490</a></li>
	<li dir="ltr">Dependencies: Update javax.mail in watcher to 1.6.2<a href="https://github.com/elastic/elasticsearch/pull/33664"> #33664</a></li>
	<li dir="ltr">Add create rollup job api to high level rest client<a href="https://github.com/elastic/elasticsearch/pull/33521"> #33521</a></li>
	<li dir="ltr">Implement xpack.monitoring.elasticsearch.collection.enabled setting<a href="https://github.com/elastic/elasticsearch/pull/33474"> #33474</a></li>
	<li dir="ltr">[Monitoring] Removing unused version.* fields<a href="https://github.com/elastic/elasticsearch/pull/33584"> #33584</a></li>
	<li dir="ltr">DiskThresholdDecider#canAllocate can report negative free bytes<a href="https://github.com/elastic/elasticsearch/pull/33641"> #33641</a></li>
	<li dir="ltr">Only notify ready global checkpoint listeners<a href="https://github.com/elastic/elasticsearch/pull/33690"> #33690</a></li>
	<li dir="ltr">Structured audit logging<a href="https://github.com/elastic/elasticsearch/pull/31931"> #31931</a></li>
	<li dir="ltr">[Kerberos] Add realm name &amp; UPN to user metadata<a href="https://github.com/elastic/elasticsearch/pull/33338"> #33338</a></li>
	<li dir="ltr">Deprecate negative weight in Function Score Query<a href="https://github.com/elastic/elasticsearch/pull/33624"> #33624</a></li>
	<li dir="ltr">SQL: Return functions in JDBC driver metadata<a href="https://github.com/elastic/elasticsearch/pull/33672"> #33672</a></li>
</ul><h4 dir="ltr">Changes in 7.0:
</h4><ul>
	<li dir="ltr">BREAKING: Add minimal sanity checks to custom/scripted similarities.<a href="https://github.com/elastic/elasticsearch/pull/33564"> #33564</a></li>
	<li dir="ltr">BREAKING: Core: Default node.name to the hostname<a href="https://github.com/elastic/elasticsearch/pull/33677"> #33677</a></li>
	<li dir="ltr">add RemoveCorruptedShardDataCommand<a href="https://github.com/elastic/elasticsearch/pull/32281"> #32281</a></li>
	<li dir="ltr">TESTS: Set SO_LINGER = 0 for MockNioTransport<a href="https://github.com/elastic/elasticsearch/pull/32560"> #32560</a></li>
	<li dir="ltr">Ensure realtime _get and _termvectors don&#x2019;t run on the network thread<a href="https://github.com/elastic/elasticsearch/pull/33814"> #33814</a></li>
	<li dir="ltr">New Annotated_text field type<a href="https://github.com/elastic/elasticsearch/pull/30364"> #30364</a></li>
	<li dir="ltr">Don&#x2019;t count hits via the collector if the hit count can be computed from index stats.<a href="https://github.com/elastic/elasticsearch/pull/33701"> #33701</a></li>
</ul><h2 dir="ltr">Apache Lucene
</h2><h4 dir="ltr">Indexed shapes
</h4><p dir="ltr">After<a href="https://github.com/mikemccand/luceneutil/pull/20"> benchmarking</a><a href="https://issues.apache.org/jira/browse/LUCENE-8396"> BKD-based geo shapes</a>, we were a bit disappointed by their performance, but we looked into it and found out that we had quite some room for improvement!
</p><p dir="ltr">One issue is that BKD trees are designed to index points. The goal of the BKD tree is to split the space into cells that contain equal numbers of points in such a way that filtering by any dimension or combination of dimensions is efficient. When indexing shapes and ranges, a better goal is to minimize intersection between cells, which is what R-trees do. Unfortunately, naively indexing coordinates of a triangle as a 6D point in a BKD tree doesn&apos;t split the space efficiently. Nick is proposing to improve BKD trees so that<a href="https://issues.apache.org/jira/browse/LUCENE-8496"> only a subset of their dimensions could be indexed</a>. The underlying idea would be that instead of indexing triangles as 6D points, we would index the minimum bounding rectangle of the triangle as a 4D point, and remaining information that would allow to reconstruct the triangle in 3 additional dimensions that wouldn&apos;t be indexed. This makes the BKD tree essentially behave like a R-tree and gave a 46% speedup to shape search in the benchmark.
</p><p dir="ltr">Another issue is that the bounding box of points for a sub-tree that the BKD tree records in its index is sometimes larger than the minimum bounding rectangle of those points, especially if there is correlation between dimensions which is common when storing ranges or shapes. Ignacio addressed this performance issue by making sure to<a href="https://issues.apache.org/jira/browse/LUCENE-7862"> record the minimum bounding rectangle that contains all points on every leaf of the BKD tree</a>. Again this brought significant speedups, up to 10x in some cases.
</p><h4 dir="ltr">Other
</h4><ul><li> We are contributing an<a href="https://issues.apache.org/jira/browse/LUCENE-8462"> Arabic stemmer</a>
</li><li>We are exploring<a href="https://issues.apache.org/jira/browse/LUCENE-8490"> separating features from queries</a>.
</li><li>We propose to <a href="https://issues.apache.org/jira/browse/LUCENE-8498">remove LowercaseTokenizer</a>, which would in-turn help<a href="https://issues.apache.org/jira/browse/LUCENE-8497"> improve the normalization API</a>.
</li></ul></body></html>

This Week in Elasticsearch and Apache Lucene - 2018-09-21

In Elasticsearch 6.3, we've made it ever easier to build and deploy your own X-Pack security extensions and custom realms.

full-bleed-2k.jpg

thumb.jpeg

How to Develop Your Own Security Extensions and Custom Realms for Elasticsearch

Elasticsearch 6.0 will enforce strict content-type checking. All REST requests that include a body must also provide the correct Content-Type header.

<html><head></head><body><p>The Elasticsearch engineering team is busy working on features for Elasticsearch 6.0. One of the changes that is coming in Elasticsearch 6.0 is <em>strict content-type checking</em>.
</p><span id="docs-internal-guid-b6da973f-7205-496b-cae6-0234ce14987a"><h1 dir="ltr">What&#x2019;s changing?</h1><p dir="ltr">Starting from Elasticsearch 6.0, all REST requests that include a body must also provide the correct content-type for that body.
</p><p dir="ltr">In earlier releases of Elasticsearch, the content-type was optional, and if it was missing or not recognised, then the server would <em>sniff</em> the content and make an educated guess regarding the content-type. That will no longer happen in Elasticsearch 6.0 - every incoming request needs to have the correct content-type for the body it contains.
</p><p dir="ltr">This ability to enforce strict content-type checking has existed since Elasticsearch 5.3 via the <code>http.content_type.required</code> <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.3/modules-http.html#_settings_2">configuration setting</a>. In 5.x it is optional, and defaults to <tt>false</tt>, in Elasticsearch 6.0, that setting defaults to <tt>true</tt>, and there is no way to disable it.
</p><h1 dir="ltr">Why are we changing this?</h1><p dir="ltr">We know that the content-type sniffing has been quite convenient when using basic HTTP tools such as curl. Many of us are quite accustomed to searching a cluster by running something like this:
</p><pre class="prettyprint">curl &apos;http://localhost:9200/_search&apos; -d&apos;
{
    &quot;query&quot; : {
        &quot;match_all&quot; : {}
    }
}&apos;
</pre><p dir="ltr">But, we need to make that sort of operation slightly more verbose, and include the content-type, in the interests of <strong><em>clarity</em></strong> and <strong><em>security</em></strong>.
</p><h2 dir="ltr">Clarity</h2><p dir="ltr">As Elasticsearch has evolved we&#x2019;ve made a conscious decision to favour reliability and predictability over leniency. And while being lenient with content-types has been convenient, it also produced some surprising results.
</p><p dir="ltr">For example, if you tried to send plain text content to an API that didn&#x2019;t support it, then you would usually receive a clear error like this:
</p><blockquote style="font-family:monospace;font-style:normal;font-size:100%">Content-Type header [text/plain] is not supported
</blockquote><p dir="ltr">But under the covers Elasticsearch was doing its best to try and guess what you might have meant. So, if your body started with &#x201C;<tt>{</tt>&#x201D; then it would guess that your content was actually JSON, but when it tried to parse that, it would fail and the error message would look more like:
</p><blockquote style="font-family:monospace;font-style:normal;font-size:100%">Unexpected character (&apos;a&apos; (code 97)): was expecting double-quote to start field name
</blockquote><p dir="ltr">And, while most of our APIs support YAML formatted requests, the content-type sniffing required that the body start with a start-of-document marker (&#x201C;<tt>---</tt>&#x201D;), which is not what users expected.
</p><p dir="ltr">When it comes to content-type, we&#x2019;ve come to the conclusion that &#x201C;Say what you mean&#x201D; provides a more reliable and predictable outcome, than guessing. Being explicit is the safer, clearer and more consistent approach.
</p><h2 dir="ltr">Security</h2><p dir="ltr">Strict checking of content-type is also useful as a layer of protection against <em>Cross Site Request Forgery</em> attacks.
</p><p dir="ltr">Because the Elasticsearch REST API uses simple HTTP requests, what&#x2019;s easy to do with curl, is often easy to do with your web browser. If your internal network allows it, you can point your favourite browser at the <tt>/_cluster/settings</tt> endpoint on one of your Elasticsearch nodes and see the settings for your cluster.
</p><p dir="ltr">Unfortunately, if an attacker has the right knowledge about your internal network and Elasticsearch cluster, they can craft a malicious webpage that would use that same technique to perform unwanted updates to your cluster. Web browsers implement a number of security policies that help protect from such attacks, and part of that protection is to place limits on the content-types that may be used when sending data to remote servers.
</p><p dir="ltr">As an example, consider this very simple web page:
</p><pre class="prettyprint">&lt;html&gt;
  &lt;body&gt;
    &lt;script src=&quot;https://code.jquery.com/jquery-3.2.1.min.js&quot;
            type=&quot;text/javascript&quot;&gt;&lt;/script&gt;
    &lt;script type=&quot;text/javascript&quot;&gt;
      $(function() {
        $.ajax({
          url: &quot;http://localhost:9200/visitors/doc/&quot;,
          type:&apos;POST&apos;,
          data: JSON.stringify({ browser: navigator.userAgent,
                                 date: new Date() }),
          contentType: &apos;text/plain&apos;
        });
      });
    &lt;/script&gt;
  &lt;/body&gt;
&lt;/html&gt;
</pre><p dir="ltr">If you run an out-of-the-box install of Elasticsearch 5 on your local computer, and then open up that page in your web browser, it will add a new document in your Elasticsearch cluster that stores your browser&#x2019;s <em>User Agent</em> and the time that you loaded the page. That&#x2019;s not so scary, but with minimal changes we could make that same page overwrite or delete existing documents.
</p><p dir="ltr">If you try run that code in your browser, you will find that there an error message is displayed in the development console. Here&#x2019;s what it looks like in Google Chrome:
</p><blockquote style="font-family:monospace;font-style:normal;font-size:100%">XMLHttpRequest cannot load http://localhost:9200/visitors/doc/. No &apos;Access-Control-Allow-Origin&apos; header is present on the requested resource.
</blockquote><p dir="ltr">That error is due to the <em>Same Origin Policy</em><sup id="footref-1">&#xA0;<a href="#footnote-1">1</a></sup> that is one of the security features of the web. By default, a web page loaded from one site may only access content from that same site. This policy prevented our sample web page from reading the JSON that was sent as a response when it stored the document in Elasticsearch.
</p><p dir="ltr">But why does the browser even allow us to send data to the Elasticsearch server if we&#x2019;re not allowed to read the result? The answer lies in a second browser feature called <em>Cross Origin Resource Sharing</em> (CORS)<sup id="footref-2">&#xA0;<a href="#footnote-2">2</a></sup>. While the <em>Same Origin Policy</em> acts as a very useful default to securing the web, there are also many times where it is helpful for two otherwise independent sites to be able to share resources. CORS defines a mechanism by which a site can optionally grant other sites access to its resources.
</p><p dir="ltr">Due to the history of the web, and the way it has evolved over time, CORS assumes that some types of requests can always be <em>sent</em> safely. For example, web browsers have always allowed cross-origin form submission - a form on my web page can be configured to send its data to your server. If the browser determines that a cross-origin request meets certain requirements, then it will declare it to be <em>safe</em> and will send that request off to the third-party server. It is only when the response comes back from that server, that the browser checks to see whether the original web page is allowed to access the provided content.
</p><p dir="ltr">In our example above, the request sets the content-type to be <tt>text/plain</tt> which browsers treat as a safe value<sup id="footref-3">&#xA0;<a href="#footnote-3">3</a></sup>, so the requests is sent off to the Elasticsearch server. When Elasticsearch responds, the browser looks for special CORS-related headers, so that it can decide whether the calling script is allowed to process the content of the response. By default, an Elasticsearch server does not include any of those CORS headers in the response, so the cross-origin request fails, and our web-page is prevented from seeing the results of the POST. But by then the damage has already been done - the request was sent to the Elasticsearch cluster and the document has been stored.
</p><p dir="ltr">The strict content-type checking in Elasticsearch 6.0 helps prevent that damage. The Index API that is being used in this example does not support a content-type of <tt>text/plain</tt>, so Elasticsearch 6.0 will reject the request without performing any updates.
</p><p dir="ltr">We might attempt to work around those content-type checks by changing our test page to send a valid Content-Type such as JSON.
</p><pre class="prettyprint">&lt;html&gt;
  &lt;body&gt;
    &lt;script src=&quot;https://code.jquery.com/jquery-3.2.1.min.js&quot;
            type=&quot;text/javascript&quot;&gt;&lt;/script&gt;
    &lt;script type=&quot;text/javascript&quot;&gt;
      $(function() {
        $.ajax({
          url: &quot;http://localhost:9200/visitors/doc/&quot;,
          type:&apos;POST&apos;,
          data: JSON.stringify({ browser: navigator.userAgent,
                                 date: new Date() }),
          contentType: &apos;application/json&apos;
        });
      });
    &lt;/script&gt;
  &lt;/body&gt;
&lt;/html&gt;
</pre><p dir="ltr">However, the CORS security policy does not treat <tt>application/json</tt> as a <em>safe</em> content-type, so the browser performs what is known as a <em>preflight</em> request. That request is sent to the same URL on the Elasticsearch server, but the HTTP method is set to <tt>OPTIONS</tt> and no data is sent in the request body. Once again the web browser looks for the special CORS response headers, and since Elasticsearch doesn&#x2019;t send them, the cross-origin request is refused and the <tt>POST</tt> body is never sent to the Elasticsearch server.
</p><p dir="ltr">By enforcing strict content-type checks in Elasticsearch 6.0, we reduce the risk of <em>Cross Site Request Forgery</em> attacks and help protect against accidental or malicious destruction of data.
</p><h1 dir="ltr">What do I need to do?</h1><p dir="ltr">For most users there&#x2019;s nothing you need to do - everything has been taken care of.
</p><p dir="ltr">All the components of the Elastic Stack, as well as our official REST clients will send the correct content-type for each request - just make sure that you&#x2019;re on a recent version. If you are using a third-party client, or one that you built yourself, then you&#x2019;ll need to check whether that client sends a valid content-type for each request.
</p><p dir="ltr">If you regularly use <tt>curl</tt> or another command line tool to send data into Elasticsearch, you&#x2019;ll need to add the <code>Content-Type</code> header to any request that contains a body. For curl, that means adding <code style="white-space:nowrap">-H&apos;Content-Type: application/json&apos;</code> to the command line of any request that has a JSON body<sup id="footref-4">&#xA0;<a href="#footnote-4">4</a></sup>.
</p><p dir="ltr">If you&#x2019;re interested in testing your application or client library, we&#x2019;ve released <a href="https://www.elastic.co/blog/elasticsearch-6-0-0-alpha2-released">6.0.0-alpha2</a>, and it includes content-type checking. Alternatively, you can turn on strict content-type checking in recent versions of Elasticsearch 5 (see below).
</p><h1 dir="ltr" rel="line-height:1.38;margin-top:20pt;margin-bottom:6pt;">What about Elasticsearch 5.x?</h1><p dir="ltr">I mentioned earlier that you can enable strict content-type checking in recent releases of Elasticsearch 5 by enabling the <code>http.content_type.required</code> configuration option. Given the security reasons mentioned above, you should consider whether that is something that would be of value to you right now.
</p><p dir="ltr">If you&#x2019;re deploying a brand new Elasticsearch cluster, it&#x2019;s probably a good idea to require strict content-types from the start. It will be one less thing to worry about when you do upgrade to 6.x, and it gives you an added layer of protection against <em>Cross Site Request Forgery</em> attacks.
</p><p dir="ltr">If you have an existing Elasticsearch installation, then turning on that setting may be a little trickier - you need to know that all of your clients are sending the correct content-type. But if you can tackle that problem now that will get you one step closer to being able to migrate to Elasticsearch 6 when it is officially available.
</p><h1 dir="ltr">Conclusion</h1><p dir="ltr">This is not a decision that we made lightly. We had a lot of conversation about it, and considered various options. We recognise that the old content sniffing approach was convenient, but we feel strongly that this change is a necessary one to help provide stable, reliable and predictable features in Elasticsearch.
</p></span><hr><p id="footnote-1" style="margin:0 1em;"><a href="#footref-1" style="font-size:80%;min-width:1.2em;display:inline-block">1.</a>&#xA0;<a href="https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy">https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy</a>
</p><p id="footnote-2" style="margin:0 1em;"><a href="#footref-2" style="font-size:80%;min-width:1.2em;display:inline-block">2.</a>&#xA0;<a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS">https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS</a>
</p><p id="footnote-3" style="margin:0 1em;"><a href="#footref-3" style="font-size:80%;min-width:1.2em;display:inline-block">3.</a>&#xA0;<a href="https://fetch.spec.whatwg.org/#cors-safelisted-request-header">https://fetch.spec.whatwg.org/#cors-safelisted-request-header</a>
</p><p id="footnote-4" style="margin:0 1em;"><a href="#footref-4" style="font-size:80%;min-width:1.2em;display:inline-block" rel="font-size:90%;min-width:1.2em;display:inline-block">4.</a>&#xA0;If you&#x2019;re copying examples from our documentation, you&#x2019;ll find that the the <span style="color:#00a9e5">COPY AS CURL</span> button automatically includes this option.
</p></body></html>

Articles by Tim Vernum