Here's how we're making the date_histogram aggregation a ton faster in Elasticsearch 7.11 and how we're going to make it even faster-er in the future.

<p>Elasticsearch's <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/search-aggregations-bucket-datehistogram-aggregation.html">date_histogram</a> aggregation is the cornerstone of Kibana's <a href="https://www.elastic.co/guide/en/kibana/7.11/discover.html">Discover</a>. And the <a href="https://www.elastic.co/guide/en/observability/7.11/monitor-logs.html">Logs Monitoring UI</a>. I use it all the time to investigate trends in build failures, but when it is slow I get cranky. Four seconds to graph all of the failures of some test over the past six months! I don't have time for that! Who is going to give me my four seconds back?! So I spent the past six months speeding it up. On and off.
</p><h2>Round with our bare hands</h2><p>Back at the dawn of time (2018) there was a bug, "<a href="https://github.com/elastic/elasticsearch/issues/28727">time_zone option makes date histograms much slower</a>". Time zones that had a daylight savings time transition were four times slower. @jpountz <a href="https://github.com/elastic/elasticsearch/pull/30534">fixed</a> it by interpreting time ignoring daylight savings time on shards that don't contain any daylight savings transitions. It was good because time zones without a transition are easy! You just subtract their offset from UTC, round, add the offset back in, and do your aggregation. Well, you don't do it. The CPU does it. But still. It happens. You get the idea.
</p><p>So <code>date_histogram</code> was fast. But every six months it slowed down! It's pretty common to have about a day's worth of data in an index. If you have to run a <code>date_histogram</code> on one of the indices with the daylight savings time transition it'd be slow. By the time I got wind of it the date rounding itself was about 1400% slower than on shards with a daylight savings time transition.
</p><p>It turns out that we were using the <code>java.util.time</code> APIs which are lovely and accurate and cover everything but they allocate objects. And you really want to avoid making new objects for every single value in an aggregation. So we took the gloves off and <a href="https://github.com/elastic/elasticsearch/pull/61467">implemented</a> our own daylight savings time conversion code customized specifically for rounding times the way that <code>date_histogram</code> does it. Now instead of allocating we get to build an array with all of the daylight savings time transitions visible to the shard and just binary search it. That's fast! Even when the shard has thousands of transitions. Logarithmic time is a wonderful thing.
</p><h2>Stop rounding</h2><p>But if you are going to precompute all of the daylight savings time transitions in all the data on the shard, then why couldn't you precompute all of the "rounding points"? All of the keys for each bucket that a <code>date_histogram</code> could produce. When we were doing our "gloves off" rounding API we did all of the work to flow the minimum and maximum dates in the index into all of the right spots, so you could just start at the minimum and then get the next rounding value until you are past the maximum, stuff it into an array and binary search *that*. Then you wouldn't need to call the date rounding code at all. It <a href="https://github.com/elastic/elasticsearch/pull/61467">turns out</a> that this is always faster. Well, almost always. If you have a hand full of documents over a huge time range then it isn't. But it's pretty fast then too. Again, logarithmic time is a wonderful thing.
</p><h2>Start filtering</h2><p>Slight tangent: @polyfractal had the <a href="https://github.com/elastic/elasticsearch/pull/47712">idea</a> that you could speed up the range aggregation by looking at the search index instead of the doc values. It showed a pretty compelling speed bump but we didn't want to merge the prototype because it would be expensive to maintain and folks don't use the range aggregation all that much.
</p><p>But we realized that if you've precomputed all of the "rounding points" for a <code>date_histogram</code> you can <a href="https://github.com/elastic/elasticsearch/pull/63643">turn it into a range aggregation</a>. And if you use the search index for that range aggregation just as @polyfractal's prototype did you get an 8x speed boost. And it's worth maintaining the optimization for the range aggregation now that it's powering an optimization for the <code>date_histogram</code> aggregation.
</p><p>This is the first time we've turned one aggregation into another one to execute it more efficiently. And we're actually doing it twice. We turn the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/search-aggregations-bucket-datehistogram-aggregation.html">date_histogram</a> into a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/search-aggregations-bucket-range-aggregation.html">range</a> aggregation and then we turn the range aggregation into a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/search-aggregations-bucket-filters-aggregation.html">filters</a> aggregation. The filters aggregation has never been super duper quick. So we wrote a "filter by filter" execution mode for it that produces the correct results in some cases and we just don't use it in the other cases. So the sequence of events goes like this:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1943197c2ccb61a2/600f3ffe6e12171027249ecd/blog-faster-date-histograms.png" data-sys-asset-uid="blt1943197c2ccb61a2" alt="When it doubt, diagram it out." style="display: block; margin: auto; width:500px;"><br>
</p><h2>Why this works</h2><p>The neat thing about this is that you can get on the "optimization train" at any stop along the way. range aggregations will check if they can run "filter by filter". So will filter aggregations.
</p><p>It doesn't have the maintenance burden that the range specific optimization had because we only have to maintain the new "filter by filter" execution mechanism and the aggregation rewrites. And we can probably speed up more aggs with other rewrites. Could we rewrite terms aggregations into filters aggregations and get the same optimization? Probably! Can we optimize a terms aggregation inside of a <code>date_histogram</code> aggregation by thinking of it as a filters aggregation instead of another filters aggregation? Maybe. Could we do the geo-distance aggregation as <a href="https://en.wikipedia.org/wiki/Annulus_(mathematics)">ring shaped</a> filters? Probably, but it might not actually be faster. Would it be worth doing even if it isn't inherently faster just to reduce the working set? It'll be fun to find out.
</p><p>So this brave new world needs <a href="https://github.com/elastic/rally-tracks/pull/113">brave</a> <a href="https://github.com/elastic/rally-tracks/pull/118">new</a> <a href="https://github.com/elastic/rally-tracks/pull/126">benchmarks</a>. We rely on <a href="https://github.com/openjdk/jmh">JMH</a> for <a href="https://github.com/elastic/elasticsearch/tree/master/benchmarks">microbenchmarks</a> and <a href="https://esrally.readthedocs.io/en/stable/">Rally</a> for our <a href="https://github.com/elastic/rally-tracks/">macrobenchmarks</a>. We run Rally nightly and <a href="https://benchmarks.elastic.co/index.html">publish</a> the results. But that is a story for another blog post.
</p>Anway, it's fun to see your work make the graphs look better. It's been a fun trip overall. I've <a href="https://github.com/elastic/elasticsearch/commit/1af8d9f228960a62c9b29e920a184a6b6a8f8cf2#diff-576c27794bae284efc278336a16d0649aaf599973a64d4b65dc67bc99bb7f000">read</a> more assembly in the past year than in the past 15 years combined. We always welcome <a href="https://github.com/elastic/elasticsearch/blob/master/CONTRIBUTING.md">contributions</a> and if you want to get paid to contribute full time we're <a href="https://jobs.elastic.co/jobs/elasticsearch/distributed-amer-distributed-emea/elasticsearch-principal-search-engineer/2533438?lang=en_us#/">hiring</a>. And you too can <a href="https://www.elastic.co/cloud">experience the exhilarating speed of "filter by filter" aggregations on Elastic Cloud</a> the day Elasticsearch 7.11.0 is released.

blog-banner-elasticsearch-gears-light-blue.png

blog-thumb-elasticsearch-gears-light-blue.png

How we made date_histogram aggregations faster than ever in Elasticsearch 7.11

How we're making date_histogram aggregations faster than ever in Elasticsearch 7.11

Version 6.2 of the Elastic stack will officially support running on Java 9 or Java 8. Your choice. Supporting Java 9 has been a pretty wild ride!

<html><head></head><body><p>
Version 6.2 of the Elastic Stack will officially support running on Java 9 or Java 8. Your choice.
Supporting Java 9 has been a pretty wild ride! Back in March of 2015 when we
<a href="https://github.com/elastic/elasticsearch/pull/10228">first</a> got the
Elasticsearch tests passing against Java 9 we didn&apos;t know what we were starting.
After almost three years Java 9 has become an integral part of how we
build Elasticsearch.

</p><h2>Following Development</h2><p>
<a href="https://github.com/elastic/elasticsearch/pull/10228">We&apos;ve</a>
<a href="https://github.com/elastic/elasticsearch/pull/13353">been</a>
<a href="https://github.com/elastic/elasticsearch/pull/14497">keeping</a>
<a href="https://github.com/elastic/elasticsearch/pull/14726">Elasticsearch</a>
<a href="https://github.com/elastic/elasticsearch/pull/16986">compatible</a>
<a href="https://github.com/elastic/elasticsearch/pull/18047">with</a>
<a href="https://github.com/elastic/elasticsearch/pull/18496">Java</a>
<a href="https://github.com/elastic/elasticsearch/pull/18527">9</a>
<a href="https://github.com/elastic/elasticsearch/pull/18767">for</a>
<a href="https://github.com/elastic/elasticsearch/pull/18789">a</a>
<a href="https://github.com/elastic/elasticsearch/pull/20260">long</a>
<a href="https://github.com/elastic/elasticsearch/pull/24070">time</a>
<a href="https://github.com/elastic/elasticsearch/pull/25859">now</a>.
You&apos;ve been able to build Elasticsearch against Java 9, on and off, since
March of 2015. We added Java 9 nodes to our test rotation in May of 2016
and with Java 9 build 118 we periodically upgraded those nodes to new
preview releases as they were released. We wanted to keep current so that
we understood the changes going into Java 9 and in case any of those
changes to Java 9 had any unintended consequences for us. Just in case.
</p><p>
On upgrading to JDK 9 build 140 our testing paid off: we encountered a fairly
significant performance regression. It hit our tests particularly hard
with one jumping from 7 seconds to 31 seconds. Elasticsearch is fairly
unique because it runs inside the
<a href="https://github.com/elastic/elasticsearch/pull/10717">Java Security Manager</a>
and its tests can generate a lot of files as it creates several in-process
Elasticsearch nodes and acts on them to test the clustering mechanisms.
</p><p>
In what feels like a poster child for &quot;open source development makes
better software&quot;, the author of the JDK change that caused
the performance regression joined our
<a href="https://github.com/elastic/elasticsearch/issues/21534">discussion</a>
of the issue and used our tests as a basis to propose a
<a href="https://bugs.openjdk.java.net/browse/JDK-8177969">fix</a> for Java 9
late in Java 9&apos;s release process.

</p><h2>Using Java 9</h2><p>We&apos;ve been excited to use the new Java 9 features and Painless has been&#xA0;taking advantage of Java 9&apos;s&#xA0;<a href="http://openjdk.java.net/jeps/280">indified string concatenation</a>&#xA0;since&#xA0;<a href="https://github.com/elastic/elasticsearch/pull/18400">May of 2016</a>&#xA0;by generating bytecode with the Java 9 specific optimizations when running in&#xA0;Java 9 to speed up code like&#xA0;<code>ctx._source[&apos;field&apos;] + &apos; &apos; + ctx._source[&apos;other&apos;]</code>.</p><p>
Painless is somewhat special because it generates bytecode that it can
customize based on the Java version. The rest of Elasticsearch didn&apos;t get to
use Java 9 specific features until we
<a href="https://github.com/elastic/elasticsearch/pull/28051&gt;added&lt;/a&gt; support
for building &lt;a href=" http:="" openjdk.java.net="" jeps="" 238"="">multi-release JARs</a>.
</p><p>
To build a multi-release JAR we compile most of the code for Java 8 and compile
a second API compatible version of some classes for Java 9 and bundle them into
a single JAR with the Java 9 classes tucked into a special directory that Java 9
reads and Java 8 ignores. This is the standard for multi-release JARs and seems
like a pretty clever hack to me &#x2014; not &quot;UTF-8 is backwards compatible with 7 bit
ASCII&quot; <a href="https://youtu.be/MijmeoH9LT4">clever</a>, but still pretty
clever.
</p><p>
Anyway, at first we proposed requiring developers to have both Java 8
installed to compile most of Elasticsearch and Java 9 installed to compile
the Java 9 substituted classes. This felt like the most logical approach
but we abandoned it because it forced all contributors to install both
JDKs and set up two environment variables. That is just too much to expect
from a first time contributor.
</p><p>
Luckily, Java 9 has wonderful
<a href="http://openjdk.java.net/jeps/247">support</a> for compiling against
Java 8. So we
<a href="https://github.com/elastic/elasticsearch/pull/28071">use</a> it!
Every build of Elasticsearch is built by Java 9&apos;s JDK. We still need to test
against Java 8 <strong>and</strong> Java 9 but we don&apos;t require every
contributor to do that. We added an optional system property,
<code>RUNTIME_JAVA_HOME</code>, that you can set to a Java 8 JVM and the
build will use that JVM for all of its testing. If you don&apos;t define
<code>RUNTIME_JAVA_HOME</code> then the build will run the tests using the
Java 9 JDK that compiled the code. Elastic will set this to a Java 8 JVM in
some CI builds to make sure Elasticsearch has sufficient coverage, but
contributors don&apos;t have to.

</p><h3>Plugin Developments</h3><p>
The build tools that Elasticsearch uses to build its plugins are published
available as a
<a href="http://search.maven.org/#artifactdetails%7Corg.elasticsearch.gradle%7Cbuild-tools%7C6.2.0%7Cjar">Gradle plugin</a>
and they &quot;come along for the ride&quot; so to speak. The Java version checking
and <code>RUNTIME_JAVA_HOME</code> finding, and static analysis changes
pick up these changes in the 6.2.0 release and anyone using the Gradle
plugin will also need Java 9 to build their Elasticsearch plugin. The
philosophy of this Gradle plugin has always been to mirror the process
used to build the plugins inside the Elasticsearch source code tree as
closely as possible.
</p><p>
This may not be desired but we figure that anyone with super strong
opinions about the build process used to build an Elasticsearch plugin
likely is building their plugin using their own process rather than this
plugin.

</p><h2>The Future</h2><p>
Java&apos;s
<a href="http://www.oracle.com/technetwork/java/javase/eol-135779.html">change</a>
to six month release cycles has forced us to reconsider how we work with
new Java versions. We very much want the Elastic Stack to be compatible
with the new non-LTS releases as fast as possible because those releases
will contain new features that can make Elasticsearch and the rest of the Elastic
Stack faster. We also want to continue to support LTS releases so users
are not constantly forced to upgrade the JVM that runs Elasticsearch.
</p><p>
Assuming Java continues its twice yearly
<a href="http://openjdk.java.net/jeps/322">JEP 322</a> release plan,
we think that we will release a version of the Elastic Stack that is compatible
with both the newest LTS release and the newest release. For example, we
expect that the upcoming 6.2 will be compatible with Java 8 (the LTS
release) and Java 9 (the newest release). Once Java 10 is released we&apos;d
like to release a version of Elasticsearch that is compatible with Java 8
and Java 10. Then a version that supports just Java 11 because it&apos;ll be
both the newest LTS release <strong>and</strong> the newest release.
</p><p>
This compatibility plan is a long way from becoming a support policy. We
can&apos;t guarantee that we&apos;re going to do what we planned. We followed Java 9&apos;s
development fairly closely but were not compatible with it in 6.0 even though
we very much wanted to be. That was the first Elastic Stack release after
Java 9&apos;s release but we weren&apos;t able to get the compatibility until 6.2. We
expect we&apos;ll be able to support six month releases of Java more quickly
because, hopefully, they&apos;ll be smaller. And because we already support
building a multi-release JAR. But we can&apos;t predict the future.
</p><p>
We are absolutely committed to making the best Elastic Stack possible but
Java six month release cycle is going to have all kinds of surprises! We
wanted to make our plans for compatibility public even if they aren&apos;t
complete because we think users and contributors deserve to know what we&apos;re
thinking. We really want to make the Elastic Stack compatible with the new versions
as fast as possible so folks can use them and benefit from all the work put
into the JVM. We also really want to keep the Elastic Stack compatible with the LTS
releases of Java so folks can continue to use them without having to worry
about keeping on the bleeding edge. We&apos;re not sure what this means regarding
support yet.</p></body></html>

elasticsearch-java-9-and-beyond-full.jpg

elasticsearch-java-9-and-beyond.jpg

Elasticsearch: Java 9 and Beyond!

Check out the ten new bootstrap checks that are coming along with Elasticsearch 5.0 and how they will help prevent major heartache for you.

<html><head></head><body><p>Elasticsearch 5.x has ten new <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.0/bootstrap-checks.html">&quot;bootstrap checks&quot;</a> which run when Elasticsearch starts up to check for configuration problems that might cause exciting failures after a node sees serious use. If any of these check fail, the node will abort during startup if it is bound to a non-local IP address. We earnestly hope that these checks are just a minor annoyance and that they prevent major heartache.</p><h2>Serial Killer</h2><p>Concrete example time! We recently added a <a href="https://github.com/elastic/elasticsearch/blob/7eb6260131aeabf25ff8301b2e3b9ada6297e692/core/src/main/java/org/elasticsearch/bootstrap/BootstrapCheck.java#L502-L532">check</a> that fails if we detect that the JVM that Elasticsearch is running in is using the &quot;serial&quot; collector. The serial collector is designed for single-threaded applications or applications running with extremely small heaps. Elasticsearch is very much a multi-threaded application and will not fare well with an extremely small heap so the collector isn&#x2019;t a good fit for it. This collector can cause nodes to time out of the cluster because this collector stops the world while it is running and those pauses can be quite substantial for large heaps because the collection process is single-threaded. If these pauses run longer than ninety seconds the master node might time out its periodic pings to the node and remove it from the cluster. When the node finishes its long GC cycle it will rejoin the cluster, causing the node to &quot;bounce&quot; in and out of the cluster.</p><p>We hope the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.0/_use_serial_collector_check.html">serial collector check</a> saves someone a lot of trouble down the line. In fact we&#x2019;re fairly sure it will because we&#x2019;ve seen the serial collector in action. One thing buying a subscription from Elastic entitles you to is support and &quot;one of my nodes is periodically leaving the cluster and coming back&quot; is a thing that comes up from time to time. Sometimes we track the failures back to the serial garbage collector. Users won&#x2019;t see those failures in production on 5.x because Elasticsearch will refuse to start with the serial garbage collector at all.

</p><p>Since the startup scripts explicitly set up a different collector we expect this check to trigger very rarely but given how insidious the failure mode is we thought it was worth checking.</p><h2>The File Limits are Too Damn Low</h2><p>Another issue that we see very frequently is &quot;too many open files&quot;. This happens when a user (often unknowingly) has their file descriptor limits set far too conservatively, like 4096. Elasticsearch needs lots of file descriptors to open many indices composed of many segments (and, on Unix-based operating systems, things like network sockets are file descriptors too). If this limit is set too low and Elasticsearch&#xA0;bumps up against the limit, things like shard allocation can fail and disaster can ensue.</p><p>So, we added the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.0/_file_descriptor_check.html">file descriptor bootstrap check</a>. It&#x2019;s the original bootstrap check, inspired by <a href="https://github.com/elastic/elasticsearch/issues?utf8=&#x2713;&amp;q=is%3Aissue%20is%3Aclosed%20%22too%20many%20open%20files%22%20">issues</a> opened on our <a href="https://github.com/elastic/elasticsearch">GitHub repository</a>, and <a href="https://discuss.elastic.co/search?q=%22too%20many%20open%20files%22%20category%3A6">posts</a> on our <a href="https://discuss.elastic.co/c/elasticsearch">discussion forum</a>. It simply checks at startup the current file descriptor limit and screams loudly if it&#x2019;s below 65536. It can take a few minutes to track down exactly the right incantation to increase the limit on your operating system and this is super annoying but it sure beats losing data in production because you never knew the limit should be higher.</p><h2>These Warnings go to Eleven</h2><p>A core value of the Elasticsearch team is &quot;it should be easy to start exploring Elasticsearch&quot;. In the past few years this <a href="https://www.elastic.co/blog/elasticsearch-the-server">has to compete</a> with values like &quot;Elasticsearch should be secure&quot; and &quot;Elasticsearch should be stable&quot; but it is still important. In that vein, we still want Elasticsearch to start for folks that just download a distribution and start it without configuring anything. Every failing bootstrap check will simply print a <code>WARN</code>.</p><pre class="prettyprint">[2016-09-23T12:19:25,588][WARN ][o.e.b.BootstrapCheck     ] [DOteo05] max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]</pre><p>and Elasticsearch will start as though nothing was wrong. We hope these warnings makes a first time user think &quot;Elasticsearch cares enough about stability to warn me when something is poorly configured.&quot;</p><p>The trouble is that people ignore warnings. We really want first time users to scan the list of warnings, realize that they should go fix those things before going to production, and keep playing with Elasticsearch. We hope they play long and hard and fall in love with Elasticsearch like we all did. After all that, we expect folks that&#xA0;saw the warnings to forget that there are things to clean up. And, of course, there are folks who start Elasticsearch in the background and don&#x2019;t read the logs at all, missing the warnings entirely.</p><p>So for these bootstrap checks to really be effective they have to change from warnings to actual errors at some point. They have to stop Elasticsearch from starting and <strong>make</strong> you fix them. We want to be sure that any &quot;real&quot; installation of Elasticsearch passes all of these checks so we don&#x2019;t have to help anyone recover from a disaster caused by something like running out of file descriptors. The best measure we could find for what makes a &quot;real&quot; installation is binding to a non-local IP address. We briefly toyed with the idea of an explicit &quot;production mode&quot; flag but decided against it because it would be too easy for folks to turn it off to &quot;work around&quot; one of the bootstrap checks.</p><p>Binding to a non-local IP address is a fairly good way to detect a &quot;real&quot; installation because you can&#x2019;t form a multi-node cluster without doing so. And you can&#x2019;t serve requests from servers other than the one running Elasticsearch either. Well, you could play tricks with tunnels but that is fairly arcane. Anyway, we think that 99% of folks that have &quot;real&quot; &quot;production&quot; installation of Elasticsearch will bind to a non-local IP address. We also think that <strong>most</strong> folks that bind to a non-local IP address are either setting up a &quot;real&quot; installation of Elasticsearch <strong>or</strong> are running Elasticsearch in a container or VM, presumably set up using a configuration management system like Puppet/Chef/Ansible/Salt/bash-scripts-and-duct-tape.</p><p>So that is the bootstrap checks, annoying you now so you don&#x2019;t get paged on a Saturday morning at 3 AM because a bulk import caused a long GC which caused a node to drop out of the cluster, all because the wrong garbage collector is configured.</p></body></html>

hand-smaller.jpg

bootstrap-checks-thumbnail.jpg

Bootstrap checks: Annoying you now instead of devastating you later!

How to live and work with the asynchronous process Elasticsearch uses to make documents visible for search called "refresh".

<html><head></head><body><p>When you send Elasticsearch a request that modifies or creates documents and
it replies with 
	<code>200 OK</code> or <code>201 CREATED</code> it has synced
the changes to disk on all active shards
	<sup><a href="#fn1" id="r1">1</a></sup>.
That means that the changes will survive catastrophic system shutdown but it doesn&apos;t mean that the changes are available for search. The process that
makes changes available for search is called a &quot;refresh&quot; and it is the topic of
this post.
</p>
<p>Refreshes are performed periodically
(
	<a href="/guide/en/elasticsearch/reference/current/index-modules.html#dynamic-index-settings"><code>index.refresh_interval</code></a>),
when the
	<a href="/guide/en/elasticsearch/reference/current/indexing-buffer.html">indexing buffer</a>
	is full, and on demand (<code>?refresh</code>)<sup><a href="#fn2" id="r2">2</a></sup>.
On demand refreshing is rarely used outside of testing because it creates small
index segments which are inefficient to create and search and must later be merged
into larger segments. Waiting for the indexing buffer to be full is
unpredictable so we can&apos;t rely on it either. That means that we mostly think of
the index as being refreshed every 
	<code>index.refresh_interval</code>, which
defaults to 1 second.
</p>
<h2>The problem</h2>
<p>Refreshing every second is fine if you are indexing something like logs
where you expect to be some amount of time behind real time, but if you are
indexing blog posts or comments or calendars then it can be a bit difficult. For
anything where a user might expect to make a change and immediately be able to
search for that change (blog, forum, scheduling app) your application needs some
way to 
	<strong>know</strong> that the change is visible for search. This is
doubly true for applications that want to use search for something
interesting after the user&apos;s change (think scheduling or aggregations). In
those cases you have a few options all of which have interesting tradeoffs:
</p>
<ol>
	<li>Wait for the refresh, perhaps polling to check that it is there.</li>
	<li>Force a refresh with <code>?refresh</code></li>
	<li>Wait for the refresh to occur with <code>?refresh=wait_for</code>&#xA0;(<strong>coming in 5.0-alpha4!</strong>)
	</li>
</ol>
<h2>Wait for the refresh</h2>
<p>You could just wait for the refresh interval to pass. This has the advantage
of being something you can do totally asynchronously. The disadvantage is that you have
to wait for the whole one second and even then it is not guaranteed. Refresh 
isn&apos;t instant. Usually it is pretty quick but
some refreshes will be slower than others so you can&apos;t really predict it. For
applications where you can tolerate not knowing for sure if something is
available for search then this is totally the right choice. But this blog post
really isn&apos;t about those applications. So, for the sake of this blog post,
we&apos;re going to assume this option isn&apos;t good enough for you.
</p>
<h2>Force a refresh with <code>?refresh</code></h2>
<p>You could force an immediate refresh. This has the advantage of being pretty
quick. Like I said a few paragraphs up, it has the disadvantage of creating
small segments that are inefficient to create, search, and merge. For plenty of
use cases this inefficiency is worth the speed. Don&apos;t be afraid to force a
refresh if it makes sense for your use case.
</p>
<p>For example, say you are loading something into Elasticsearch and plan to
analyze the results. This search index is just for you so you know when you are
done loading documents. At that point you shouldn&apos;t hesitate to refresh the
index. Waiting isn&apos;t going to help.
</p>
<p>I should mention that adding <code>?refresh</code> to an index, update,
delete, or bulk request is subtly different than performing a
	<a href="/guide/en/elasticsearch/reference/current/indices-refresh.html">refresh</a>
	API call. Refresh API calls will refresh all the shards on the index.
	<code>?refresh</code> will only refresh the shards that have been modified. So
for index, update, and delete requests that is just the shard to which the
document was routed. For bulk requests that is all shards to which any document
was routed.
</p>
<p><code>?refresh</code> might also be a bad choice because it affects other
indexing in the same index. Say you have a bulk loading process that works
quite well. But now you want to start inserting a few documents into the same
index interactively. If you do it with 
	<code>?refresh</code> then, suddenly,
you&apos;ve started refreshing documents outside of whatever refresh interval you
were using for the bulk load. If you do that frequently enough that&apos;ll change
the search and index performance of the bulk loading process.
</p>
<h2>Wait for the refresh with <code>?refresh=wait_for</code>&#xA0;<strong>(coming in 5.0-alpha4!)</strong></h2>
<p>Elasticsearch 5.0 brings a hybrid approach between the two options. Adding
	<code>?refresh=wait_for</code> to index, update, delete, or bulk request will
cause the request to wait until its changes have been made visible for search
before returning to the user. This has the advantage of being correct without
creating inefficient segments. It has the disadvantage of having to wait for
the refresh. You don&apos;t have to wait for as long as the &quot;wait for the refresh&quot;
option because Elasticsearch signals you as soon as the document is ready for
search. So if the change comes half way through the refresh interval you only
have to wait for half of the time.
</p>
<p>Unlike <code>?refresh</code>, <code>?refresh=wait_for</code> won&apos;t affect
concurrent indexing on the same index. It has no
effect on segment size because it doesn&apos;t force a refresh immediately. If you
must know when the refresh happens, you can wait for the refresh, and you plan
to upgrade to 5.0&#xA0;
	<sup><a href="#fn3" id="r3">3</a></sup>, then this
is the right choice!
</p>
<p>Even if you are super excited to upgrade to 5.0-alpha4 to get this feature
keep in mind that Elasticsearch&apos;s alphas and betas are for testing purposes
only because they aren&apos;t compatible with the GA release. We are still
finalizing the wire level communications and on disk layout so 5.0&apos; alphas and
betas aren&apos;t guaranteed to upgrade to properly to 5.0.0, either with rolling
restarts or a full cluster restart. Please test this feature to see if it fits
for you but don&apos;t upgrade production clusters to alphas or betas.
</p>
<p>Back to the feature, there is a limit to the number of
	<code>?refresh=wait_for</code> API calls that can be waiting on any one shard: 
	<code>index.max_refresh_listeners</code> which defaults to <code>1000</code>.
If a request with 
	<code>?refresh=wait_for</code> comes in while all the slots
are full then Elasticsearch will refresh the shard and reply to the request
immediately.
</p>
<p>What does <code>?refresh=wait_for</code> do if you set the
	<code>index.refresh_interval</code> to <code>-1</code>, disabling periodic
refreshes, you may ask? Well the answer is that 
	<code>?refresh=wait_for</code> will honor whatever refresh interval you
  configure. The request will only return when you fill the indexing buffer,
force an explicit refresh, or try to wait on more than
	<code>index.max_refresh_listeners</code> requests in the same shard.
</p>
<p><code>index.refresh_interval</code> is just about the maximum number of time
that 
	<code>?refresh=wait_for</code> will have to wait for the changes to become
visible. If you use 
	<code>?refresh=wait_for</code>, raising the refresh
interval will make indexing feel slower and slower to your users. And
lowering it will make indexing feel faster and faster. So it might be tempting
to lower the refresh interval. Doing so will make less and less efficient
segments. Making the refresh interval the same as the write rate is as
inefficient as using 
	<code>?refresh</code> on every request.
</p>
<h2>Pick the refresh strategy that makes sense for you</h2>
<p>Ultimately there is no silver bullet for refreshes. Elasticsearch&apos;s
	<code>index.refresh_interval</code> is a useful because it coalesces several
changes into one big change to the search index, making a more efficient index.
You either wait for the refresh interval, potentially slowing down your users,
or you force an immediate refresh and pay the price at search and merge time.
	<code>?refresh=wait_for</code> gives you a tool to make waiting for the refresh
interval interactive so you can make whatever tradeoffs make sense for you.
</p><section>
<h2>Footnotes</h2>
<p id="fn1"><sup><a href="#r1">1</a>&#xA0;</sup>This is not always true but it is
  the recommended configuration. See
	<a href="/guide/en/elasticsearch/reference/current/index-modules-translog.html#_translog_settings">
	<code>index.translog.durability</code></a>.
</p>
<p id="fn2"><sup><a href="#r2">2</a>&#xA0;</sup>Refreshes also occur during
  recovery, the process that moves shards between nodes. This ought to be rare
  enough not to factor into most thinking about refreshes.
</p>
<p id="fn3"><sup><a href="#r3">3</a>&#xA0;</sup>You can cobble together something
  that works alright in older versions of Elasticsearch using the steps
	<a href="https://github.com/elastic/elasticsearch/issues/1063#issuecomment-148722039">here</a>
	and/or <a href="http://stackoverflow.com/questions/28111805/elasticsearch-block-until-refresh-wait-for-doc-to-be-searchable-alternatives">here</a>
	. The trouble is that it doesn&apos;t work well with bulk and or replicas. It is
  far from perfect.
</p>
</section></body></html>

dove-nest.jpg

dove-nest-720x420.jpg

Where are my documents?<br>Refreshing news...

Reindex and update-by-query are coming to Elasticsearch!

<html><head></head><body><p>
	<code>_reindex</code> and <code>_update_by_query</code> are coming to
Elasticsearch 2.3.0 and 5.0.0-alpha1! Hurray!
</p>
<p>
	<a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/docs-reindex.html">_reindex</a>
	reads documents from one index and writes them to another index. It can be used
to copy documents from one index to another, enrich documents with fields, or
recreate the index to change settings that are locked when the index is created.
</p>
<p>
	<a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.x/docs-update-by-query.html">_update_by_query</a>
	reads documents from an index and writes them back to the same index. It can be
used to update fields in many documents at once or to pick up mapping changes
that can be made online.
</p>
<h2><code>_reindex</code> copies documents</h2>
<p>
	The <code>_reindex</code> API is really just a convenient way to copy documents
from one index to another. Everything else that it can do is an outgrowth of
that. If all you want to do is to copy all the documents from the
	<code>src</code> index into the <code>dest</code> index you invoke
	<code>_reindex</code> like this:
</p>
<pre class="prettyprint">curl -XPOST localhost:9200/_reindex?pretty -d&apos;{
  &quot;source&quot;: {
    &quot;index&quot;: &quot;src&quot;
  },
  &quot;dest&quot;: {
    &quot;index&quot;: &quot;dest&quot;
  }
}&apos;
</pre>
<p>
	If you want to be a little more selective and, say, only copy docments tagged
with 
	<code>bananas</code> you invoke <code>_reindex</code> like this:
</p>
<pre class="prettyprint">curl -XPOST localhost:9200/_reindex?pretty -d&apos;{
  &quot;source&quot;: {
    &quot;index&quot;: &quot;src&quot;,
    &quot;query&quot;: {
      &quot;match&quot;: {
        &quot;tags&quot;: &quot;bananas&quot;
      }
    }
  },
  &quot;dest&quot;: {
    &quot;index&quot;: &quot;dest&quot;
  }
}&apos;
</pre>
<p>
	If you want to copy documents tagged with <code>bananas</code> but you want to
add the 
	<code>chocolate</code> tag to all copied documents you invoke
	<code>_reindex</code> like this:
</p>
<pre class="prettyprint">curl -XPOST localhost:9200/_reindex?pretty -d&apos;{
  &quot;source&quot;: {
    &quot;index&quot;: &quot;src&quot;,
    &quot;query&quot;: {
      &quot;match&quot;: {
        &quot;tags&quot;: &quot;bananas&quot;
      }
    }
  },
  &quot;dest&quot;: {
    &quot;index&quot;: &quot;dest&quot;
  },
  &quot;script&quot;: {
    &quot;inline&quot;: &quot;ctx._source.tags += \&quot;chocolate\&quot;&quot;
  }
}&apos;
</pre>
<p>
	That requires that you have dynamic scripts enabled but you can do the same
thing with non-
	<code>inline</code> scripts.
</p>
<p>
	Recreating an index to change settings that are locked at index creations is a
bit more involved but still simpler than before 
	<code>_reindex</code>:
</p>
<pre class="prettyprint"># Say you have an old index that you made like this
curl -XPUT localhost:9200/test_1 -d&apos;{
  &quot;aliases&quot;: {
    &quot;test&quot;: {}
  }
}&apos;
for i in $(seq 1 1000); do
  curl -XPOST localhost:9200/test/test -d&apos;{&quot;tags&quot;: [&quot;bananas&quot;]}&apos;
  echo
done
curl -XPOST localhost:9200/test/_refresh?pretty
# But you don&apos;t like having the default number of shards
# You can make a copy of it with the new number of shards
curl -XPUT localhost:9200/test_2 -d&apos;{
  &quot;settings&quot;: {
    &quot;number_of_shards&quot;: 1
  }
}&apos;
curl -XPOST &apos;localhost:9200/_reindex?pretty&amp;refresh&apos; -d&apos;{
  &quot;source&quot;: {
    &quot;index&quot;: &quot;test&quot;
  },
  &quot;dest&quot;: {
    &quot;index&quot;: &quot;test_2&quot;
  }
}&apos;
# Then just swing the alias to the new index
curl -XPOST localhost:9200/_aliases?pretty -d&apos;{
  &quot;actions&quot;: [
    { &quot;remove&quot;: { &quot;index&quot;: &quot;test_1&quot;, &quot;alias&quot;: &quot;index&quot; } },
    { &quot;add&quot;: { &quot;index&quot;: &quot;test_2&quot;, &quot;alias&quot;: &quot;index&quot; } }
  ]
}&apos;
# Then when you are good and sure you are done with it you can
curl -XDELETE localhost:9200/test_1?pretty
</pre>
<h2><code>_update_by_query</code> modifies documents</h2>
<p>
	The simplest way to invoke update by query isn&apos;t particularly useful on its own:
</p>
<pre class="prettyprint">curl -XPOST localhost:9200/test/_update_by_query?pretty
</pre>
<p>
	That will just increment the document version number on each document in the
	<code>test</code> index and fail if you modify a document while it is running.
A more interesting example is adding the 
	<code>chocolate</code> tag to all
documents with the 
	<code>bananas</code> tag:
</p>
<pre class="prettyprint">curl -XPOST &apos;localhost:9200/test/_update_by_query?pretty&amp;refresh&apos; -d&apos;{
  &quot;query&quot;: {
    &quot;bool&quot;: {
      &quot;must&quot;: [ {&quot;match&quot;: {&quot;tags&quot;: &quot;bananas&quot;}} ],
      &quot;must_not&quot;: [ {&quot;match&quot;: {&quot;tags&quot;: &quot;chocolate&quot;}} ]
    }
  },
  &quot;script&quot;: {
    &quot;inline&quot;: &quot;ctx._source.tags += \&quot;chocolate\&quot;&quot;
  }
}&apos;
</pre>
<p>
	Like the last version this will fail if any documents are changed while it is
running, but it is written in such a way that you can just retry it and it&apos;ll
pick up from where it left off. If you&apos;ve already modified whatever application
is making the concurrent updates to add the 
	<code>chocolate</code> tag whenever
it sees 
	<code>bananas</code> then you can safely ignore version conflicts in
the 
	<code>_update_by_query</code>. You can tell it to do so by setting
	<code>conflicts=proceed</code>. It will just count the version conflicts and
continue performing updates. Now the command looks like this:
</p>
<pre class="prettyprint">curl -XPOST &apos;localhost:9200/test/_update_by_query?pretty&amp;refresh&amp;conflicts=proceed&apos; -d&apos;{
  &quot;query&quot;: {
    &quot;bool&quot;: {
      &quot;must&quot;: [ {&quot;match&quot;: {&quot;tags&quot;: &quot;bananas&quot;}} ],
      &quot;must_not&quot;: [ {&quot;match&quot;: {&quot;tags&quot;: &quot;chocolate&quot;}} ]
    }
  },
  &quot;script&quot;: {
    &quot;inline&quot;: &quot;ctx._source.tags += \&quot;chocolate\&quot;&quot;
  }
}&apos;
</pre>
<p>
	Finally, you can use <code>_update_by_query</code> to suck up mapping changes
that only take effect when the document is modified like adding a new field to
an existing field. For example:
</p>
<pre class="prettyprint"># Say I made an index with tags not_analyzed because, you know, they are tags after all
curl -XPUT localhost:9200/test_3?pretty -d&apos;{
  &quot;mappings&quot;: {
    &quot;test&quot;: {
      &quot;properties&quot;: {
        &quot;tags&quot;: {
          &quot;type&quot;: &quot;string&quot;,
          &quot;index&quot;: &quot;not_analyzed&quot;
        }
      }
    }
  }
}&apos;
for i in $(seq 1 1000); do
  curl -XPOST localhost:9200/test_3/test -d&apos;{&quot;tags&quot;: [&quot;bananas&quot;]}&apos;
  echo
done
curl -XPOST localhost:9200/test_3/_refresh?pretty
# But now I want to search on tags using the standard analyzer so I can search for banana and find bananas
curl -XPUT localhost:9200/test_3/_mapping/test?pretty -d&apos;{
  &quot;properties&quot;: {
    &quot;tags&quot;: {
      &quot;type&quot;: &quot;string&quot;,
      &quot;index&quot;: &quot;not_analyzed&quot;,
      &quot;fields&quot;: {
        &quot;analyzed&quot;: {
          &quot;type&quot;: &quot;string&quot;,
          &quot;analyzer&quot;: &quot;standard&quot;
        }
      }
    }
  }
}&apos;
# This doesn&apos;t take effect immediately
curl &apos;localhost:9200/test_3/_search?pretty&apos; -d&apos;{
  &quot;query&quot;: {
    &quot;match&quot;: {
      &quot;tags.analyzed&quot;: &quot;bananas&quot;
    }
  }
}&apos;
# :(
# But we can _update_by_query to pick up the new mapping on all documents
curl -XPOST &apos;localhost:9200/test_3/_update_by_query?pretty&amp;conflicts=proceed&amp;refresh&apos;
# And now the new mapping has been applied to the whole index!
curl &apos;localhost:9200/test_3/_search?pretty&apos; -d&apos;{
  &quot;query&quot;: {
    &quot;match&quot;: {
      &quot;tags.analyzed&quot;: &quot;bananas&quot;
    }
  }
}&apos;
</pre>
<h2>Getting the status</h2>
<p>
	<code>_reindex</code> and <code>_update_by_query</code> can touch millions of
documents so they can take a long time. You can fetch their status with:
</p>
<pre class="prettyprint">curl localhost:9200/_tasks?pretty&amp;detailed&amp;actions=*reindex,*byquery
</pre>
<p>That will contain a field that looks like:
</p>
<pre class="prettyprint">&quot;BHgHr0cETkOehwqZ2N_-aQ:28295&quot; : {
  &quot;node&quot; : &quot;BHgHr0cETkOehwqZ2N_-aQ&quot;,
  &quot;id&quot; : 28295,
  &quot;type&quot; : &quot;transport&quot;,
  &quot;action&quot; : &quot;indices:data/write/reindex&quot;,
  &quot;start_time_in_millis&quot; : 1458767149108,
  &quot;running_time_in_nanos&quot; : 5475314,
  &quot;status&quot; : {
    &quot;total&quot; : 6154,
    &quot;updated&quot; : 3500,
    &quot;created&quot; : 0,
    &quot;deleted&quot; : 0,
    &quot;batches&quot; : 36,
    &quot;version_conflicts&quot; : 0,
    &quot;noops&quot; : 0,
    &quot;retries&quot;: 0,
    &quot;throttled_millis&quot;: 0
  }
}
</pre>
<p>
	You can read the <a href="/guide/en/elasticsearch/reference/2.x/docs-update-by-query.html#docs-update-by-query-task-api">docs</a>
	for more, but the gist is that _reindex plans to do <code>total</code>
	operations and has already done <code>updated + created + deleted + noops</code>
	of them. So you can estimate how complete the request is by dividing those
numbers.
</p>
<h2>Cancelling</h2>
<p>
	<code>_reindex</code> was so long in coming because Elasticsearch lacked a way
to cancel running tasks. For short running tasks like 
	<code>_search</code> and
indexing that is fine. But, like I wrote above, 
	<code>_reindex</code> and
	<code>_update_by_query</code> can touch millions of documents are take a long
time. The tasks themselves are ok with that, but you may not be. Say you realize
ten minutes into a three hour long 
	<code>_update_by_query</code> that you made
a mistake in the script. There isn&apos;t a way to rollback the changes that the
reindex already made but you can cancel it so it won&apos;t make any more such
changes:
</p>
<pre class="prettyprint">curl -XPOST localhost:9200/_task/{taskId}/_cancel
</pre>
<p>
	And where do you get the taskId? It is the name of the object returned by the
task listing API in the last section of this blog post. The one in the example
return is 
	<code>BHgHr0cETkOehwqZ2N_-aQ:28295</code>.
</p>
<p>
	In Elasticsearch task cancelation is opt in. It kind of has to be that way in
any Java application. Anyway, tasks that can be canceled like
	<code>_reindex</code> and <code>_update_by_query</code> periodically check to
see if they have been canceled and then shut themselves down. This means that
you might see the task if you immediately list its status after it has been
canceled. It will go away on its own and you can&apos;t cancel it any harder without
stopping the node it is running on.
</p>
<h2>Remember that Elasticsearch is a search engine</h2>
<p>
	Every update has to mark the document as deleted and index the entire new
document. The deleted documents have to then be merged out of the index.
	<code>_reindex</code> and <code>_update_by_query</code> don&apos;t save anything in
that process. They work just as though you performed a scroll query and indexed
all the results. Running a zillion 
	<code>_reindex</code>s or
	<code>_update_by_query</code>s is unlikely to be the most efficient use of
computer resources to accomplish some task. You will almost always be better off
making changes to the application that adds data to Elasticsearch rather than
updating the data after the fact. 
	<code>_reindex</code> and
	<code>_update_by_query</code> are most useful for turning the data that you
already have in Elasticsearch into the data that you want to be in
Elasticsearch.
</p></body></html>

flower-166156.jpg

Reindex is coming!

Load Wikipedia's real search index and experiment with it! These instructions have been updated for Elastic Stack 6.3.1.

<html><head></head><body><table style="background: #FFFFD2;">
<tbody>
<tr>
<td>A while back I published a <a href="https://www.elastic.co/blog/elasticsearch-queries-or-term-queries-are-really-fast">blog</a> post testing query speeds with data from English Wikipedia and mentioned that the loading process deserved its own blog post. Well here it is! These instructions have been tested against the version 6.3.1 of the Elastic Stack.</td>
</tr>
</tbody>
</table>
<p></p>
<h2>Step 0: Why</h2>
<p>This gets you a copy of the search index used to power on site search for Wikipedia. The index is quite large
  because it&apos;s used to support all the funky things that Elasticsearch is used for there. It may have mistakes, old
  data, strange leftovers from previous versions of the indexing code that have yet to be cleaned up. In short: it&apos;s
  imperfect, just like any real production deploy.
</p>
<h2>Step 1: Download a smaller wiki</h2>
<p>The Wikimedia Foundation makes everything about Wikipedia and the other wikis it powers&#xA0;public. Well, everything that is safe
  to make public anyway. That includes a dump of the search index. Head
	<a href="https://dumps.wikimedia.org/other/cirrussearch/">here</a> and you&apos;ll get a list of dates when the dump runs
  began. Click on the latest date and download the file that looks like
	<code>enwikiquote-$date-cirrussearch-content.json.gz</code>. The naming scheme is
	<code>$wiki-$date-cirrussearch-$indexType.json.gz</code> where:
</p>
<ul>
	<li>
	<code>$wiki</code> is usually made of two parts: <code>$langCode$project</code>
	<ul>
		<li><code>$langCode</code> is the language - mostly
		<a href="https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes">ISO 639-1</a>
		codes. I use English because it is nice and large and I can read it.</li>
		<li>
		<code>$project</code> is the kind of wiki:
		<ul>
			<li><code>wiki</code> is <a href="https://www.wikipedia.org/">Wikipedia</a>, an encyclopedia</li>
			<li><code>wiktionary</code> is <a href="https://www.wiktionary.org/">Wikitionary</a>, a dictionary</li>
			<li><code>wikisource</code> is <a href="https://wikisource.org/">Wikisource</a>, a library of free
            content
			</li>
			<li><code>wikivoyage</code> is <a href="https://www.wikivoyage.org/">Wikivoyage</a>, a collaborative travel
            guide
			</li>
			<li><code>wikibooks</code> is <a href="https://www.wikibooks.org/">Wikibooks</a>, a repository of open
            content text books
			</li>
			<li><code>wikiquote</code> is <a href="https://www.wikiquote.org/">Wikiquote</a>, a collection of
            quotations
			</li>
			<li><code>wikiversity</code> is <a href="https://www.wikiversity.org/">Wikiversity</a>, an open learning
            community
			</li>
			<li><code>wikinews</code> is <a href="https://www.wikinews.org/">Wikinews</a>, a collaborative news site</li>
		</ul>
		</li>
		<li>There are some special wiki codes like <code>commonwiki</code> which is for
		<a href="http://commons.wikimedia.org/">Wikimedia Commons</a>, a repository of free multimedia stuff.</li>
	</ul>
	</li>
	<li><code>$indexType</code> is the type of search&#xA0;index. Normally it&apos;s either&#xA0;<code>content</code>, pages someone reading the
    wiki usually wants to search, and 
	<code>general</code>, pages that are useful for building the wiki like templates
    discussions, and scripts. Wikimedia Commons, being a multimedia repository, has it a 
	<code>file</code>
	<code>$indexType</code> that contains data about the files like description. Sometimes it contains text that can be
    extracted from the files, like
	<a href="https://commons.wikimedia.org/wiki/File:Scientific_American_-_Series_1_-_Volume_001_-_Issue_01.pdf?action=cirrusdump">this</a>
	issue of Scientific American. Some of these extracts are fairly large like
	<a href="https://commons.wikimedia.org/wiki/File:Thorton_book.pdf?action=cirrusdump">this</a> biography.
	</li>
</ul>
<p>By now you&apos;ve figured out that I didn&apos;t have you download English Wikipedia&apos;s search index. I had you fetch English
  Wikiquote. You can follow my instructions below to load English Wikiquote and then&#xA0;start over to load any other
  wiki you&apos;d like from the dumps page. If&#xA0;one particular wiki caught your eye, go and start downloading it now
  while you follow along to the end of the blog post using English Wikiquote. If all goes well it&apos;ll be ready by the time you want it.
</p>
<h2>Step 2: Get the index ready</h2>
<p>While the dump downloads you can setup Elasticsearch to handle it the index. You&apos;ll need the
	<code>analysis-icu</code> plugin, and a plugin that wikimedia uses for similarity. You can do that by this into your bash shell:
</p>
<p><code>bin/elasticsearch-plugin install analysis-icu</code>
</p>
<p><code>bin/elasticsearch-plugin install org.wikimedia.search:extra:6.3.1.2</code>
</p>
<p>You&apos;ll have to restart Elasticsearch after installing that for it to take.
</p>
<p>Then you&apos;ll need <a href="https://stedolan.github.io/jq/">jq</a> for some of the json-foo you&apos;ll do next. For me
  it&apos;s just 
	<code>sudo apt-get install jq</code> but your command may vary.
</p>
<p>Finally you can create you index with these bash shell commands:
</p>
<pre class="prettyprint">export es=localhost:9200
export site=en.wikipedia.org
export index=enwikiquote
curl -XDELETE $es/$index?pretty
curl -H &apos;Content-Type: application/json&apos; -s &apos;https://&apos;$site&apos;/w/api.php?action=cirrus-settings-dump&amp;format=json&amp;formatversion=2&apos; |
  jq &apos;{
    settings: { 
        index: { 
            analysis: .content.page.index.analysis, 
            similarity: .content.page.index.similarity 
        } 
    }
  }&apos; |
curl -H &apos;Content-Type: application/json&apos; -XPUT $es/$index?pretty -d @-
curl -H &apos;Content-Type: application/json&apos; -s &apos;https://&apos;$site&apos;/w/api.php?action=cirrus-mapping-dump&amp;format=json&amp;formatversion=2&apos; |
  jq .content |
  sed &apos;s/&quot;index_analyzer&quot;/&quot;analyzer&quot;/&apos; |
  sed &apos;s/&quot;position_offset_gap&quot;/&quot;position_increment_gap&quot;/&apos; |
  curl -H &apos;Content-Type: application/json&apos; -XPUT $es/$index/_mapping/page?pretty -d @-
</pre>
<p>Let me walk you through that:
</p>
<ul>
	<li><code>export es=localhost:9200</code> sets up <code>$es</code> to be Elasticsearch&apos;s address. I do that so I
    don&apos;t have to type 
	<code>localhost:9200</code> over and over again. And it&apos;ll make copying and pasting my text
    easier if your Elasticsearch doesn&apos;t listen on 9200. Win/win.
	</li>
	<li><code>export site=en.wikipedia.org</code> sets up <code>$site</code> to be the hostname of the MediaWiki instance
    that you want to use. For English Wikipedia you&apos;d use 
	<code>en.wikipedia.org</code>. For German Wikipedia you&apos;d use
	<code>de.wikipedia.org</code>. For Wikimedia Commons, you&apos;d use <code>commons.wikimedia.org</code>.</li>
	<li><code>export index=enwikiquote</code> just sets <code>$index</code> to the name of the index you&apos;ll be loading. I
    usually use the same name as the wiki.
	</li>
	<li><code>curl -XDELETE $es/$index?pretty</code> deletes the index if it already exists. It fails if the index
    doesn&apos;t exist but we don&apos;t care. You&apos;ll see this at the start of most curl issue reproductions filed&#xA0;with Elasticsearch because it
    ensures a clean slate.
	</li>
	<li>The next sequence of pipes fetches the settings that en.wikiquote.org uses for its index and uses them as a
    template to create a new index in our test cluster.&#xA0;
	</li>
	<li>The last sequence of pipes fetches that mapping for the content index, upgrades it from 2.x style to 6.x style,
    and creates a 
	<code>page</code> type with that mapping in our test index.</li>
</ul>
<p>Clear as mud? Ok.
</p>
<h2>Step 3: Prepare the wiki for loading</h2>
<p>Crack open the file with zless and have a look at it. Don&apos;t worry, you can do that before it finishes downloading.
  Its contents are conveniently in the format that Elasticsearch uses for bulk loading. Hurray! We&apos;ll unzip and cut
  this file into smaller chunks so it&apos;ll work properly with curl and the 
	<code>_bulk</code> API. It&apos;ll also you a way
  to pause and resume the process. After it has finished downloading you can do that with these bash shell commands:
</p>
<pre class="prettyprint">export dump=enwikiquote-20160201-cirrussearch-content.json.gz
export index=enwikiquote
mkdir chunks
cd chunks
zcat ../$dump | split -a 10 -l 500 - $index
</pre>
<ul>
	<li>The first <code>export</code> line just names the file that you downloaded. Change it to whatever file you
    fetched. The second export line is the same as one used in the previous section.
	</li>
	<li>The <code>mkdir</code> and <code>cd</code> lines make a directory to hold the files we&apos;re going to make.</li>
	<li>The last line cuts the file into 500 line chunks. 250 of those lines are metadata lines for the
	<code>_bulk</code> api. 250 lines are the actual documents.</li>
	<li>If zcat doesn&apos;t work on your system, `gunzip -c` is a drop in replacement.</li>
</ul>
<p>For English Wikiquote that should finish in a few seconds. English Wikipedia takes longer than a coffee break.
</p>
<h2>Step 4: Load the wiki</h2>
<p>The last step is to load the actual data with these bash commands:
</p>
<pre class="prettyprint">export es=localhost:9200
export index=enwikiquote
cd chunks
for file in *; do
  echo -n &quot;${file}:  &quot;
  took=$(curl -s -H &apos;Content-Type: application/x-ndjson&apos; -XPOST $es/$index/_bulk?pretty --data-binary @$file |
    grep took | cut -d&apos;:&apos; -f 2 | cut -d&apos;,&apos; -f 1)
  printf &apos;%7s\n&apos; $took
  [ &quot;x$took&quot; = &quot;x&quot; ] || rm $file
done
</pre>
<p>The first three lines should be familiar from above. The loop loads each file and deletes it after it&apos;s loaded. If
  the file fails to load it isn&apos;t deleted and the loop moves on to the next file.
</p>
<p>I find setting the <code>refresh_interval</code> to <code>-1</code> will speed this process up some. You can apply
  it by running this in a different terminal:
</p>
<pre class="prettyprint">curl -XPUT &quot;$es/$index/_settings?pretty&quot; -d &apos;{
    &quot;index&quot; : {
        &quot;refresh_interval&quot; : -1
    }
}&apos;
</pre>
<p>You can monitor the progress with:
</p>
<pre class="prettyprint">date; curl $es/$index/_refresh?pretty; curl $es/$index/_count?pretty
</pre>
<p>The <code>_refresh</code> makes the current process visible regardless of the <code>refresh_interval</code>. The
	<code>_count</code> just counts how many documents you&apos;ve loaded.
</p>
<p>Loading English Wikiquote is an excellent opportunity to have a coffee break. Loading English Wikipedia takes all
  night. When it is all done you should flush the index with 
	<code>curl $es/$index/_flush</code> to make sure that the
  next time you restart Elasticsearch it doesn&apos;t have a large translog to replay.
</p>
<h2>Step 5: Now do it with a bigger wiki</h2>
<p>Now that you&apos;ve loaded a smaller wiki you can just repeate the process with other <code>export</code> statements to
  load a larger wiki. While loading English Wikiquote took something like 20 minutes total you&apos;ll find loading English
  Wikipedia to be an overnight job.
</p>
<p>Regarding space: I find that gzip gives English Wikipedia about a 1:6 compession ratio and when the Elasticsearch
  index is comlete it&apos;s only slightly bigger. Since the load process removes that files after they are loaded into
  Elasticsearch it should be safe if you budget 10 time the file size for this process. When I last loaded English
  Wikipedia the Elasticsearch index ended up being 7 times the size of the gzip and I kept the gzip on disk in case I
  wanted to load it again. Its small enough not to be a big deal on a spinning disk but large enough to be a pain on an
  SSD.
</p>
<h2>Step 6: Now what?</h2>
<p>After you&apos;ve loaded the index you can do whatever you want with it: presumably you&apos;ll want to test some queries or
  something. You can do that now. You&apos;ll notice a huge performance boost if you
	<code>_optimize?max_num_segments=1</code> but that is cheating to some degree because it causes trouble if you ever
  modify the index. For testing maybe it&apos;s fine but it doesn&apos;t simulate a system that is constantly being updated.
</p></body></html>

book-1798.jpg

Loading Wikipedia's Search Index For Testing

Comparing the performance of different kinds of queries in Elasticsearch.

<html><head></head><body><p>I remember a couple years ago I was at a convention talking to folks about Elasticsearch or Lucene and one of them said something like &quot;Everyone knows that if you really want to search fast you need to just use term queries.&quot; He was certainly right about term queries being fast but I really wanted to run some numbers and see just how right he was. And that felt like a good time to write a blog post to help with the &quot;everyone knows&quot; part.
</p>
<h1>Term queries are really fast</h1><center><img alt="AND vs OR: Searchers Per Second" src="https://api.contentstack.io/v2/assets/575e4d1c3dc542cb38c09b08/download?uid=blt460a7e935b471f4b?uid=blt460a7e935b471f4b" style="display:inline" data-sys-asset-uid="blt18139356edc0e047"> <img alt="AND vs OR: Hits Per Search" src="https://api.contentstack.io/v2/assets/575e4d1c12798d8e381fede6/download?uid=blt2f085f4fc774452f?uid=blt2f085f4fc774452f" style="display:inline" data-sys-asset-uid="bltb75941d28e01303a"></center>
<p>Term queries are pretty fast! On my development desktop with a single spinning disk I get 8,000 searches containing a single term query a second. Once you start adding multiple terms to the query it gets faster if they are <code>AND</code>ed together and slower if they are <code>OR</code>ed together. This is because OR queries find more hits and scoring and counting those hits takes more time.
</p>
<p>We can control for this using <code>terminate_after</code>. <img alt="Term Queries: terminate_after" src="https://api.contentstack.io/v2/assets/575e4d1c58208ba076e298a8/download?uid=blt5be7a13e202b7dcc?uid=blt5be7a13e202b7dcc" style="float:right; margin:0 0 20px 20px; max-width:100%;" data-sys-asset-uid="blted9cac52c9a42e11"> If <code>terminate_after</code> is set then each shard stops searching as soon as it has hit that many results. If the document that would have had the best score wasn&apos;t hit because the shard terminated early then it&apos;s just not returned. If you are sorting by something other than score this is still a problem because the documents aren&apos;t encountered in a useful order. It also breaks the total hit count because each shard stops counting after <code>terminate_after</code> hits. On the other hand it really improves the performance of unselective queries so it&apos;s worth thinking about even though it&apos;ll only be appropriate for somewhat niche use cases. Values of <code>terminate_after</code> less than 10,000 make <code>OR</code>s faster than <code>AND</code>s and values greater than 10,000 make <code>AND</code>s faster than <code>OR</code>s. In my dataset. With my two term queries. This is because <code>OR</code>s accept every candidate document so they have to find fewer candidates.
</p>
<h1>Testing methodology</h1>
<p>Now a note about testing methodology: this isn&apos;t super scientific. I&apos;m using a rather old dump of English Wikipedia&apos;s search index for its content namespaces (available <a href="https://dumps.wikimedia.org/other/cirrussearch/">here</a>, the file looks like <code>enwiki-$date-cirrussearch-content.json.gz</code>). How to load your own copy would be a great topic for another <a href="/blog/loading-wikipedia">blog post</a>.... Anyway! I&apos;m using <a href="https://github.com/wg/wrk">wrk</a> for generating the actual load which uses a Lua script to generate the queries. The terms I&apos;m using for the queries come from <code>/usr/share/dict/american-english</code>. I&apos;m running both wrk and Elasticsearch on the same machine. I&apos;m using a single shard and all the default arguments for Elasticsearch 2.1.1 including its 1GB heap. So I have about 30GB of free memory for Linux to use for the page cache. Every query I run has <code>size=0</code> so I don&apos;t spend time loading the <code>_source</code>, highlighting, or any of that stuff that a real search has to do. After loading the index I <code>_optimize</code>d it which is totally cheating. A real production system can&apos;t optimize an index that is always changing and Wikipedia is always changing. But the index is 124GB and my development desktop&apos;s SSD only has 101GB of space total so I&apos;m having to use my spinning drive, a 1TB WD Blue spinning at 7200RPM. If I don&apos;t optimize I very quickly saturate the poor disk and my tests just show that I need to buy a better disk. Which doesn&apos;t make for a great blog post. I want to talk about the CPU overhead of these queries and optimize does a good job of making that possible. But take everything here with a grain or two of salt. Benchmark for yourself. My goal is to put you in the right ballpark.
</p>
<p>I use <code>query_string</code> for all of my queries because it&apos;s convenient. I don&apos;t believe you should use <code>query_string</code> for queries sent by users because it&apos;s too powerful and too brittle but it <strong>is</strong> super convenient for my lua query generator to just generate a string query rather than worry about generating JSON.
</p>
<h1>Various query types</h1><center><img alt="Query Types" src="https://api.contentstack.io/v2/assets/575e4d1c0342dfd738264a17/download?uid=blt949357b7563d9230?uid=blt949357b7563d9230" style="display:inline" data-sys-asset-uid="bltd6e33348fca89fff"></center>
<p>All being said, have a look at the graph above which compares some different types of queries. A couple of things: first and foremost I&apos;ve shifted this graph to log scale. Sorry! Without doing that the fast queries just overwhelm the slow queries and you can&apos;t see anything. The horizontal axis is how frequently the term comes from a list of &quot;common&quot; words. All queries contain just two words. This doesn&apos;t use <code>terminate_after</code>. Finally, I&apos;m using a home grown notation for the query types. Its short and super readable if you are me but in case you aren&apos;t:
</p>
<ul>
	<li><code>AND</code> means <a href="https://gist.github.com/nik9000/009929d3a0b777e8c5a3"><code>AND</code>ing term queries together</a>.</li>
	<li><code>OR</code> means <a href="https://gist.github.com/nik9000/942047d2465981e63cf3"><code>OR</code>ing term queries together</a>.</li>
	<li><code>AND*$p</code> means <a href="https://gist.github.com/nik9000/1726ed9bbb720d4b03e3"><code>AND</code>ing prefix queries where the length is <code>$p</code></a>.</li>
	<li><code>AND~$f</code> means <a href="https://gist.github.com/nik9000/e013006fb92136997133"><code>AND</code>ing fuzzy queries with fuzziness <code>$f</code></a>.</li>
	<li><code>PHRASE</code> means terms are wrapped in a <a href="https://gist.github.com/nik9000/9c8a7c8decdc93f2fd18">phrase query</a>.</li>
</ul>
<p>A couple things jump out at me:
</p>
<ul>
	<li>As the terms get more and more common the whole thing gets slower and slower. This is caused by the Elasticsearch having to do more work to either hit the documents or reject them.</li>
	<li>Phrase queries are really fast when all the terms are uncommon - almost as fast as just <code>AND</code>ing the terms together. But they suffer quite a bit as the parts of the phrase get more and more common. This shouldn&apos;t be shocking: phrase queries only visit the term positions if the document contains all the terms so if the terms are rare they have to visit very few documents. When the document does have all the terms they have walk the positions lists of each of the terms in parallel which requires quite a bit of work, including reading those lists from disk. If you are trying at home to estimate how expensive phrase queries will be I&apos;d err closer to the right hand side of the graph. Real people are more likely to do phrase queries for things that might form a phrase. My test script is as likely to search for <code>&quot;blighted icepicks&quot;</code> as it is to search for <code>&quot;good job&quot;</code>. Do your own benchmarks with your own data and your own query logs if possible.</li>
	<li>Other than unselective phrase queries there really is no competition for <code>AND</code>ing terms together. Prefix queries of at least 5 terms is about an order of magnitude slower and everything else is slower still.</li>
	<li>Fuzzy queries with <code>fuzziness=AUTO</code> are surprisingly zippy. I don&apos;t know why they become progressively slow as their terms get more and more common. <code>fuzziness=2</code> is devastatingly slow, potentially because English has so many short words and applying a fuzziness of 2 to a 3 letter word finds <strong>tons</strong> of terms. Its worth mentioning that I run these queries with all the default settings and that the <code>rewrite</code> parameter might affect their their run times.</li>
	<li>Prefix queries are really dependent on the length of the prefix. Obviously short prefixes find more documents, but they have more overhead in general because they have to walk more of the terms dictionary. Prefix queries where the prefix is just a single letter are so slow I had to leave them out of the graph. I have yet to measure it but anything that allows a leading wildcard is going to be pretty nasty as well! I have yet to test it but it stands to reason that the actual prefix matters as well. More common prefixes will generate more terms to lookup and more hits, taking more time.</li>
</ul>
<h1><code>terminate_after</code> makes them faster</h1><center><img alt="Query Types" src="https://api.contentstack.io/v2/assets/575e4d1c3dc542cb38c09afe/download?uid=blt4a7cb6fe6b99c3a1?uid=blt4a7cb6fe6b99c3a1" style="max-width:100%;" data-sys-asset-uid="blt5875c28ad4788522"></center>
<p>Above is the same chart of different query types, this time with <code>terminate_after</code> set to <code>10000</code>. I picked <code>10000</code> because its pretty that inflection point between <code>AND</code> and <code>OR</code> which just seemed fun. I also kept the same range on the horizontal axis. Here is what I noticed:
</p>
<ul>
	<li>Performance is better all over the place. Not in every query but in a ton of them.</li>
	<li>As expected <code>OR</code> ties with <code>AND</code> 100% uncommon terms but is faster than <code>AND</code> for more common terms. This is because it has to do less work to hit the 10,000 results.</li>
	<li>Phrase queries suffer the most as the frequency of common terms goes up. They spend a ton of effort rejecting documents so they get the least benefit from <code>terminate_after</code> because they are very selective.</li>
</ul>
<h1>Configured queries</h1>
<p><img alt="Phrase Slop" src="https://api.contentstack.io/v2/assets/575e4d1c12798d8e381fedd5/download?uid=blted7342e7cc30e0d4?uid=blted7342e7cc30e0d4" style="float:right; margin:0 0 20px 20px; max-width:100%;" data-sys-asset-uid="blt36a284f278756628">You&apos;ll notice that I ignored phrase slop when talking about phrase queries. Its because the slop value doesn&apos;t really matter. A slop of 0 is <em>slightly</em> faster than a slop of 1 but beyond that all slops are just as fast as one another.
</p>
<p><img alt="Fuzziness" src="https://api.contentstack.io/v2/assets/575e4d1c58208ba076e298a0/download?uid=blt18352b8cb9385237?uid=blt18352b8cb9385237" style="float:right; margin:0 0 20px 20px; max-width:100%;" data-sys-asset-uid="blt344473872071e256"> i also skipped over <code>fuzziness</code> of <code>1</code> in the graphs above and only had <code>AUTO</code> and <code>2</code>. Its not super clear to me why, but <code>1</code> has the same performance as <code>AUTO</code>.
</p>
<p><img alt="Prefix Queries" src="https://api.contentstack.io/v2/assets/575e4d1c0342dfd738264a06/download?uid=blt4017d04a34202bda?uid=blt4017d04a34202bda" style="float:right; margin:0 0 20px 20px; max-width:100%;" data-sys-asset-uid="bltaf7999f2d7ad27bb"> As expected the longer the prefix on the prefix query the faster it finishes. Here is as good a place as any to describe how I build these prefix queries because its important that you know: I use the random word picker to get a word, slice it down to the prefix length or smaller if it was fewer code points than the prefix length, and then I make a prefix query out of it. This mixes in shorter prefix queries with longer ones which artificially depresses the scores, especially closer to the right hand side of these graphs which have more common, shorter words. For a while I was using the word picker over and over again until I got a long enough word. This artificially increased the scores of long prefixes because they found far fewer documents than the term queries I was comparing them against because the term queries contained more shorter, more common words. The moral of this story is that graphs hide lots of complexity.
</p>
<h1>Conclusion</h1>
<p>So my advice:
</p>
<ul>
	<li>Query the terms you want directly if possible. That means to use queries like <code>match</code> and <code>term</code> instead of <code>match_phrase</code> and <code>prefix</code>.
	</li>
	<li>Prefer <code>AND</code> to <code>OR</code> if you can get away with it.</li>
	<li>Use fuzzy queries if you must but prefer <code>fuzziness=AUTO</code> over other fuzziness values. <code>fuzziness=2</code> is super slow.</li>
	<li>Prefix and wildcard style queries are OK if the prefix is long. They get progressively worse as the prefix gets shorter. 5 characters is about the same as <code>fuzziness=AUTO</code>, 3 as <code>fuzziness=1</code>, and 1 is just plain horrible.</li>
	<li>Stay away from phrase queries if at all possible. Their worst case performance is the worst.</li>
	<li>If you must use phrase queries then use whatever phrase slop is appropriate for your use case. A slop of 0 is only marginally faster than any other slop so its not worth worrying about it.</li>
	<p>It&apos;s worth thinking about whether <code>terminate_after</code> is appropriate for your use case. While it provides a nice boost it does so at the cost of not necessarily returning the best result and not getting an exact count of results. It <strong>stops</strong> looking at the results on each shard once it hits the limit so it&apos;s quite possible that the highest scoring result is won&apos;t have been inspected. Its probably limited to the niche use cases where you can tell users &quot;Sorry but your query was too general so we aren&apos;t sure that we found the best results for it. Please make it more specific by adding more terms.&quot;
	</p>
	<p>There are other options for limiting the performance impact of slow queries. Wrapping phrase queries in a <code>rescore</code> is quite common and probably worth yet another blog post with more pretty graphs. Another option is to use the <code>shingle</code> token filter to create a query with phrase-ish characteristics. You can also limit the number of hits when there are many <code>OR</code> clauses using <code>minimum_should_match</code>, <code>common_terms_query</code>, and stop words which are all knobs worth tweaking that I simply ran out of time to cover in this post.
	</p>
</ul></body></html>

Articles by Nik Everett

How we're making date_histogram aggregations faster than ever in Elasticsearch 7.11

Elasticsearch: Java 9 and Beyond!

Bootstrap checks: Annoying you now instead of devastating you later!

Where are my documents?<br>Refreshing news...

Reindex is coming!

Loading Wikipedia's Search Index For Testing

Elasticsearch Queries, or Term Queries are Really Fast!

Products & Solutions

Company

Resources

Featured topics

News

Articles by Nik Everett

How we're making date_histogram aggregations faster than ever in Elasticsearch 7.11

Elasticsearch: Java 9 and Beyond!

Bootstrap checks: Annoying you now instead of devastating you later!

Where are my documents?<br>Refreshing news...

Reindex is coming!

Loading Wikipedia's Search Index For Testing

Elasticsearch Queries, or Term Queries are Really Fast!