Hosted Elasticsearch by Found gives you an easy way to clone your cluster. Find out why you want to do this in the following article.

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to&#xA0;our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>One of the advantages of using a SaaS offering is flexibility for on-demand scaling. With Found this means you can create and delete clusters as you like and only pay for the hour APIs they&#x2019;re running. Building on the Snapshot and restore API of Elasticsearch, Found makes it easy to duplicate a cluster. In this blog post we will see how this works and how it may be used in a few different scenarios. Finally we will have a brief look at how it is implemented which will also serve to show why the clone operation has absolutely no performance impact on the source cluster.
</p><h2>Use cases</h2><p style="line-height: 22.4px;">A common denominator in all of these use cases is that they benefit from the isolation that a separate cluster provides, whether that is isolation of hardware resources or data resources.
</p><h3>Ad hoc analytics</h3><p style="line-height: 22.4px;">For most logging and metrics use cases it is cost prohibitive to have all the data in memory, even if that is what provides the best performance for aggregations. Cloning the relevant data to an ad hoc analytics cluster that can be discarded after use can be a very cost effective way to experiment with your data, and at the same time without risk of hurting performance in production.
</p><h3>Test upgrades</h3><p style="line-height: 22.4px;">The safest way to check that both your indexes and your application is ready for the next Elasticsearch version is to copy the indexes to a new cluster and test the entire upgrade path by first upgrading Elasticsearch and then making sure that your application works.
</p><h3>Enable your developers</h3><p style="line-height: 22.4px;">Realistic test data is crucial for uncovering unexpected errors early in the development cycle. What can be more realistic than actual data from the production cluster? Giving the developer team access to experiment with real production data is not only great for breaking down silos, but may also provide new insights about the domain. A safe and isolated playground is the essential benefit for this use case.
</p><h3>Test Mapping changes</h3><p style="line-height: 22.4px;">Mapping changes almost always require reindexing. Unless your data volume is trivial it also takes some time. Tweaking the parameters to achieve best performance usually takes a little trial and error. While this use case could also be handled by running the scan and scroll query directly against the source cluster, it is worth noting that a long lived scroll has the side effect of blocking merges even if the scan query is very light weight.
</p><h3>Integration testing</h3><p style="line-height: 22.4px;">Test your application against a real live Elasticsearch instance with actual data. If you automate this, you could also aggregate performance metrics from the tests and use those metrics to detect if a change in your application has introduced a performance degradation.
</p><h2>How to clone a cluster</h2><p style="line-height: 22.4px;">The steps required:
</p><ol>
	<li>Prepare target cluster (the cluster which the snapshot will be copied to).</li>
	<li>Select snapshot from source cluster</li>
	<li>Issue restore</li>
</ol><h3>Prepare the target cluster</h3><p style="line-height: 22.4px;">In order for the restore to be successful the target cluster must meet the following conditions:
</p><ul>
	<li>Same region</li>
	<li>Compatible Elasticsearch version</li>
	<li>Index resources used
	<ul>
		<li>Plugins for custom types</li>
		<li>Synonym files used by analyzer</li>
		<li>Dictionary files used by analyzer</li>
	</ul></li>
	<li>No conflicting indexes</li>
</ul><p style="line-height: 22.4px;">Both clusters have to be in the same region in order to have access to each others backups. In most cases any Elasticsearch version equal to or higher than the version of the source cluster will be compatible. The exception is if upgrading more than one major version at a time as Elasticsearch then might not have a Lucene version capable of reading the indexes. Same thing happens when an index originally created on an Elasticsearch version older than the current version have segments written in an old Lucene format, but this is usually not a problem as the the format is upgraded automatically during merges.
</p><p style="line-height: 22.4px;">The target cluster is not required to have all the plugins and dictionaries of the source cluster, just the ones required to open the cloned indexes. In practice this means any dictionary or synonym file referenced by a custom analyser in any of the indexes, aka user bundles on the cluster config page. Slightly less common, but just as hard a requirement is if any of the indexes use a custom field type provided by a plugin like mapper-attachments or murmur3.
</p><p style="line-height: 22.4px;">The final requirement of the target cluster is that it doesn&#x2019;t have any conflicting indexes defined. If any of the indexes to be copied already exist in the target cluster they must either be deleted / closed before issuing the restore. Optionally you may use a rename pattern when issuing the restore to have the indexes restored with a different name.
</p><h3>Selecting the snapshot</h3><p style="line-height: 22.4px;">The cluster cloning capabilities on Found is really an extension of the automated backup service that issues a snapshot every 30 minutes. To select a snapshot, sign in to the console on&#xA0;<a href="https://found.elastic.co/">https://found.elastic.co</a>&#xA0;and go to the clusters page.
</p><p><img src="https://api.contentstack.io/v2/assets/575e4d239e7a83165490d9e3/download?uid=blt2b53e45f666bd2aa?uid=blt2b53e45f666bd2aa" data-sys-asset-uid="blt39e607b28712410c" alt="clone-cluster-cluster-overview.png">
</p><p style="line-height: 22.4px;">There you will find a &#x201C;Snapshots&#x201D; item for each of your clusters in the left hand menu. The snapshots page lists each snapshot by timestamp. Click on one and it takes you to the restore snapshot page.
</p><h3>Issue a restore</h3><p style="line-height: 22.4px;">Once you&#x2019;ve selected a snapshot to restore from, you should see a form with the fields indexes, rename pattern, rename replacement and cluster.
</p><p style="line-height: 22.4px;"><img src="https://api.contentstack.io/v2/assets/575e4d2312798d8e381fef02/download?uid=blt9b85a9f420f843bb?uid=blt9b85a9f420f843bb" data-sys-asset-uid="blt7360420a474a5142" alt="clone-cluster-restore.png" width="401" height="250" style="width: 401px; height: 250px;">
</p><p style="line-height: 22.4px;"><span style="line-height: 22.4px; background-color: initial;">The first three correspond to the similar parameters of the&#xA0;</span><a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.1/modules-snapshots.html#_restore">restore command in Elasticsearch</a><span style="line-height: 22.4px; background-color: initial;">. The cluster parameter specifies the target cluster of the restore operation. If you don&#x2019;t specify a cluster, the default behavior is to restore to the same cluster the snapshot was made from. If you don&#x2019;t see your cluster in the dropdown list, make sure that it running a suitable Elasticsearch version and has had time to start.</span><br>
</p><h2>Things included and not included</h2><p style="line-height: 22.4px;">Indexes and index settings are included, but cluster settings are not. This decision is a consequence of the fact that the indexes may be restored to a cluster with a very different plan, but it can be an important distinction to be aware of if performance testing reveals different results between different clusters.
</p><p style="line-height: 22.4px;">Another consideration for performance comparisons is that copying indexes from one cluster to another with the snapshot and restore API is not the same as copying indexes with a scan/scroll query and bulk indexing. The former copies Lucene segments and the latter JSON documents. If the source index has segments in an old Lucene format, these are not modified, but copied as they are.
</p><h2>Under the hood</h2><p style="line-height: 22.4px;">For every restore to a different cluster issued by the found console, three things happen:
</p><ul>
	<li>The settings of the&#xA0;<code style="font-size: 12.6px;">found-snapshots</code>&#xA0;repository in the source cluster are inspected</li>
	<li>A snapshot repository with the same S3 bucket and path is created in the target cluster, but the repository&#x2019;s id is the id of the source cluster and not&#xA0;<code style="font-size: 12.6px;">found-snapshots</code>.</li>
	<li>The new repository is used to issue a restore on the target cluster.</li>
</ul><p style="line-height: 22.4px;">One important takeaway here is that the data is not transferred from the source cluster when the restore is issued, but copied from the backup in S3. This means that issuing a restore to a different cluster will not have any performance implication on the source cluster.
</p><p style="line-height: 22.4px;">After the restore is complete, the repository in the target cluster is not removed, making it easy issue another restore through the Elasticsearch API. If you would like to include this initialization in a script, the steps required are described in our&#xA0;<a href="https://www.elastic.co/guide/en/found/current/snapshot-restore.html#_sharing_a_repository_across_clusters">documentation on sharing a repository across clusters</a>.
</p><h2>Conclusion</h2><p style="line-height: 22.4px;">At the Found team we aim to provide all the cool features in Elasticsearch in combination with the ease and flexibility of a good SaaS offering. If you have any suggestions, we would love to hear them on&#xA0;<a href="https://discuss.elastic.co/c/found">discuss</a>.
</p></body></html>

clone-cluster-lambs.jpg

Cluster cloning in the cloud

Packetbeat taps into the network stack, analyzes traffic for known protocols, extracts useful info from each packet, and ships it into Elasticsearch.

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to&#xA0;our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c940342dfd738263238/download?uid=blta99cb2bb37a700fc?uid=blta99cb2bb37a700fc" data-sys-asset-uid="blt6d136a7f6ddd8e98" alt="" width="258" height="206" style="line-height: 1.6em; float: right; width: 258px; height: 206px; margin: 0px;">
</p><p>Packetbeat is the tool that lets you ship your network data into Elasticsearch so you can visualize them in Kibana.<br>
</p><h2>How Does It Work?</h2><p>Packetbeat taps into the network stack on the machine it&apos;s running on, analyzes all the traffic for known protocols and extracts useful information from each packet, before shipping that information into Elasticsearch. Currently it supports HTTP, PostgreSQL, MySQL, Redis and Thrift.
</p><p>Packetbeat is a competitor to Logstash, you say? Well, yes and no. There is some overlap between Packetbeat and Logstash, they both feed data into Elasticsearch, but their use cases are still slightly different. Packetbeat focuses on network traffic while Logstash focuses on logs. Both are capable of extracting metrics and Packetbeat also has a sibling called Filebeat that can consume log files. Filebeat can either feed logs directly into Elasticsearch or forward them to Logstash for centralised preprocessing.
</p><h2>Installing</h2><p>Installing Packetbeat is very straightforward. Download the package matching your platform from <a href="https://www.elastic.co/downloads/beats/packetbeat">https://www.elastic.co/downloads/beats/packetbeat</a> and install it with your package system or simply extract the archive if you opted for one of the <code>.tar.gz</code> or <code>.tgz</code> files.
</p><h2>Configuring</h2><p>Once Packetbeat is installed, it is time to edit <code>packetbeat.yml</code>. The Debian and the RPM packages install this file directly to <code>/etc/packetbeat/packetbeat.yml</code>, but the generic packages just leave it in the directory you extracted it to.
</p><h3>Shipper</h3><p>There are three noteworthy properties in this section, <code>name</code>, <code>tags</code> and <code>ignore_outgoing</code>.
</p><p>The <code>name</code> attribute is meant to uniquely identify the server Packetbeat is monitoring and it defaults to the host name. You can safely leave it blank or just give a name that makes sense to you.
</p><p>The <code>tags</code> attribute allows you to specify tags that will be applied to all data originating from this server. Use it for what makes sense to you, but tags like test, staging, production, database or application are not uncommon. In the end the tags are useful when you want to filter data from different groups of servers.
</p><h3>Interfaces</h3><p>A short section, but no less important. This is where you specify how Packetbeat will integrate with the networking stack on the server. At minimum, you should specify which devices to monitor. On my Mac, I used something as simple as this:
</p><pre class="prettyprint lang-yaml">interfaces:
 device: en0
</pre><p>The device names are platform specific and you should choose what matches your platform. Run <code>ifconfig</code> in your terminal for a list of devices. On Macs your Wi-Fi and your ethernet card are usually named either <code>en0</code> or  <code>en1</code> and Linux <code>eth0</code> and <code>eth1</code> are common. There is also a special name called <code>any</code> where Packetbeat will connect with all available devices.
</p><p>Packetbeat supports a few more options in this section, each with different trade-offs that are out of scope for this article, but before deploying into production, have a look at <a href="/guide/en/beats/packetbeat/current/_configuration.html#configuration-interfaces">the interfaces section in the guide</a>.
</p><h3>Protocols</h3><p>Packetbeat currently supports monitoring HTTP, PostgreSQL, MySQL, Redis and Thrift. For all protocols there is one mandatory parameter, <code>ports</code>. For demo purposes, declaring ports and protocols in use is usually sufficient, but in particular for the HTTP protocol there are a few things to be aware of.
</p><h4><code>hide_keywords</code></h4><p>The <code>hide_keywords</code> parameter allows you to specify form fields in GET or POST requests that should not be indexes. This is important since some fields may contain sensitive data like passwords in clear text.
</p><h4><code>real_ip_header</code></h4><p>A very common setup is to have a standard HTTP server, like Apache or Nginx in front of your application server and then let this server handle things like SSL termination. One disadvantage of this is that it hides the client IP that request originated from. Luckily, both Apache and Nginx and most other servers you would want to use in such a scenario support adding the client IP in a header on each request. Packetbeat can then be configured with the <code>real_ip_header</code> parameter to extract this information.
</p><p>Terminating SSL before it reaches the application server is also the only way to let Packetbeat monitor services provided over HTTPS.
</p><h3>Output</h3><p>Like Logstash, Packetbeat supports multiple outputs and most frequently people use the Elasticsearch output. In this example I&apos;ve defined the necessary parameters for connecting to a cluster on Found. Specifically this includes: <code>host</code>, <code>port</code>, <code>protocol</code>, <code>username</code> and <code>password</code>.
</p><pre class="prettyprint lang-yaml">output:
  elasticsearch:
    enabled: true
    host: 20ea7fisk409df2c3b523a2b131d.eu-west-1.aws.found.io
    port: 9243
    save_topology: true
    protocol: https
    username: readwrite
    password: &lt;password&gt;
</pre><h3>Procs</h3><p>This section is optional and it is disabled by default, but on Linux systems it will allow monitoring traffic between processes on the same host. It will also populate the <code>proc</code> and <code>client_proc</code> fields. All that is required is to give the process a name and a grep filter so that Packetbeat can identify the process id.
</p><p>The complete reference for <code>packetbeat.yml</code> is available in <a href="/guide/en/beats/packetbeat/current/_configuration.html">The Guide</a>.
</p><h2>Running Packetbeat</h2><p>The Deb and RPM packages include launch scripts so you can launch Packetbeat like any other service in your distribution. For the generic packages you might have to specify the location of <code>packetbeat.yml</code>, like this:
</p><pre class="prettyprint lang-sh">sudo ./packetbeat -c ./packetbeat.yml -v
</pre><p><code>-v </code> for verbose is also useful the first time after a change to <code>packetbeat.yml</code>. Privileged access (sudo) is also required for sniffing packets.
</p><h2>Visualizing The Data</h2><p>Packetbeat feeds data into Elasticsearch in a format that is timestamped and suitable for visualizing with Kibana. You can use either Kibana 3 or 4.
</p><p>Packetbeat does not come with any Kibana dashboards out of the box, but in all likelihood, you most likely want to build your own, a dashboard targeting your data and the questions you have. If you have used Kibana previously, all you really need to know to get started is the index pattern to use and that is: &quot;[packetbeat-]YYYY.MM.DD&quot;. If you&apos;ve never used Kibana before, I recommend <a href="/webinars/introduction-elk-stack">An Introduction to the ELK stack</a>. There is also a <a href="http://demo.elastic.co/packetbeat/">live demo of Kibana 4 with data fed by packetbeat</a>.
</p></body></html>

Packetbeat on Found

<p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p><p>There are many buzzwords that may be applied to Elasticsearch. Multi-tenancy is one. Getting started with a multi-tenant use case can be deceptively easy - there are some pitfalls that will require a little careful design.
</p><section id="what-is-multi-tenancy" class="level2">
<h2><a href="#what-is-multi-tenancy">What is Multi-Tenancy?</a></h2>
<p>Multi-tenancy means that one has a system with multiple tenants. Depending on the project you might refer to a tenant as a user, an organisation, a project, an application, a client or something different. If each of your tenants use Elasticsearch for entirely different things then what you’re actually doing is providing Elasticsearch as service with a shared cluster model - not a good idea. If you wonder why we think shared clusters is a bad idea, please see our article on <a href="/blog/found-elasticsearch-in-production">Elasticsearch in production</a>.
</p>
<p>If, however, each tenant has an instance of the same application that relies on Elasticsearch, there is a commonality between the tenants, both in terms of queries executed and how data is structured. Even stronger commonality exists when the application itself is multi-tenant, like a software as a service scenario. In either case, an important criteria of multi-tenant solutions is to ensure that one tenant is not able to observe any data belonging to another tenant. I will simply refer to this as the data separation constraint.
</p>
</section><section id="example-blog-hosting-as-a-service" class="level2">
<h2><a href="#example-blog-hosting-as-a-service">Example: Blog Hosting as a Service</a></h2>
<p>As an example througout this article we use a fictive blog hosting provider. It is fictive in the sense that it I’m not thinking of anybody in particular - and any similarity is accidental - however I wouldn’t be suprised if you find a similar story on the tech blog of an actual blog hosting provider. As their competitors, they provide a freemium model, where the smallest package is free.
</p>
</section><section id="using-one-index-per-tenant" class="level2">
<h2><a href="#using-one-index-per-tenant">Using One Index Per Tenant</a></h2>
<p>For their proof of concept they started out with separate indexes per tenant. By simply appending the unique ID of each tenant to the index name, their proof of concept was up and running in no time and the data separation constraint was satisified.
</p>
<p>Once the proof of concept was completed they wanted to put Elasticsearch to the test and started loading one percent of their production data into a test cluster. Lucky that they actually did this test, as the results where no good. The test cluster ran out of memory half way during the import. The engineers where anxious about these results because they had convinced the management that Elasticsearch would be so much faster.
</p>
<p>After some analysis the engineers figured out that when the the first out-of-memory error happened, the amount of data indexed actually corresponded to only a fraction of the memory available to the instance. Clearly, there was an issue with configuration here, and they started reading <a href="/blog/found-sizing-elasticsearch/">Sizing Elasticsearch</a> only to learn that there is a base cost of memory for each shard and that this base cost is constant, even if the shard contains no documents. They did the test again with only one shard per index (default is five) and this helped, but their total shard count per Elasticsearch instance was still too high. Since every index has to have at least one shard, their freemium model made it impossible to have one index per tenant. Or as one of the team members put it: “We cannot reserve memory for idle customers that don’t pay us any money.”
</p>
</section><section id="all-tenants-share-one-index" class="level2">
<h2><a href="#all-tenants-share-one-index">All Tenants Share One Index</a></h2>
<p>Having realised that tenants need to share indexes, they quickly rewrote the queries by wrapping them in a filtered query and filtering on tenant ID. Similarily, they changed the data migration script to target one index and include the tenant ID in every document. This fixed the memory usage and they where able to get the performance numbers required for a go ahead from management.
</p>
<p>Further down the line, the initial version of the application with Elasticsearch support was nearing release and the operations team now started testing out a full-scale data migration. After the usual performance tweaks, importing data went smoothly and they started integration testing the application with this full-scale cluster. At this time they discovered that the first time a tenant’s data was accessed, page loads were really slow. They realized that this had to do with warming up caches - which was confirmed by testing out cache warmers - but the metrics showed that this would require too much CPU to scale out for all their freemium customers. Yet another case of reserving hardware for idle customers who don’t pay.
</p>
</section><section id="custom-routing" class="level2">
<h2><a href="#custom-routing">Custom Routing</a></h2>
<p>The fact that this was not an issue with the one percent dataset led them to realize that the CPU cost for warming the cache for a single tenant was not related to the amount of data that tenant had, but in fact it was related to the size of the entire data set and the total number of shards. Both of these parameters were quite different between the initial performance test and the full-scale test. After reading <a href="/blog/found-elasticsearch-top-down/#routing">Elasticsearch from the top down</a> they understood more about the way routing works in Elasticsearch and the problem became obvious: “For every tenant cache that needed to be populated, no matter how small the tenant actually is, every shard in the cluster was queried.”
</p>
<p>Being used to the advanced query optimizers common in relational databases they felt they had moved to an inferior technology, but they soon discovered that Elasticsearch has support for defining custom routing. They changed the index settings to use the tenant ID as a routing key, reindexed the full data set and added the new routing key to all the queries in the application as well. Now they finally got similar performance regardless of the number of tenants in the data set.
</p>
<p>They realized, of course, that some tenants are larger than others and wanted to reduce the probability of two large tenants ending up in the same shard and possibly making that shard too large for a single instance. This led them to choose a higher than otherwise necessary number of shards for the indexes.
</p>
</section><section id="tenants-grow-both-in-data-and-traffic" class="level2">
<h2><a href="#tenants-grow-both-in-data-and-traffic">Tenants Grow Both in Data and Traffic</a></h2>
<p>After about a year in production, the entire organization was very happy with the move to Elasticsearch. The developers used new aggregations features and the operations team had good routines for replacing and adding instances as necessary. Elasticsearch even took the size of shards into consideration during allocation so that the total amount of data on each instance remained fairly even.
</p>
<p>Of course, things are never that easy in the long run… The largest tenant kept pushing the size of the shard it was in and operations realized that it was getting more and more expensive and difficult to scale up the instance that shard was residing on. This led to a hotfix of the application with a bunch of special cases for that tenant ID so that it could have an index of its own with multiple shards for that tenant. After a while they also realized that this enabled them to increase the number of replicas for just this tenant, which helped to distribute the search load incurred by this tenant over more instances.
</p>
</section><section id="nobody-can-tell-the-future" class="level2">
<h2><a href="#nobody-can-tell-the-future">Nobody Can Tell the Future</a></h2>
<p>The developers were not happy about the special cases in their code relating to a specific tenant, but they realized that the extreme differences in size required them to do so. It probably did not hurt that the revenue from this one customer paid a huge chunck of their salaries.
</p>
<p>After a while though, they realized that bad code could end up hurting all customers and not just their own job satisfaction, wich led them to revisit the issue once more. This time, somebody asked the key question: “What happens when our second largest customer outgrows a single shard, and which customer will that be?”
</p>
<p>This question led them to the realization that they had to make it configurable for the operations team to decide which index or perhaps even indexes a tenant has data in. They ended up with a config scheme that looked something like this:
</p>
<pre class="sourceCode json prettyprint">[
  {
    "tenant-id-of-large-tenant":{
      "write":"index1-largetenant",
      "read":[
        "index*-largetenant"
      ]
    }
  },
  {
    ".*":{
      "write":"index-main",
      "read":[
        "index-main"
      ]
    }
  }
]
</pre>
</section><section id="regarding-transitions-and-moving-tenants-around" class="level2">
<h2><a href="#regarding-transitions-and-moving-tenants-around">Regarding Transitions and Moving Tenants Around</a></h2>
<p>Elasticsearch has a nifty feature called index aliases. They’re great when migrating a tenant from an old to a new index. Not only do they support targeting multiple indexes, either as a list or with a pattern, but they allow you to define filters to be included when querying.
</p>
<p>It is even possible to use them in such a way that your application might think there is only one index per tenant, when in reality there is one index alias per tenant. In the long run however, there will be a scaling issue with this as well. The reason is that the list of aliases is part of the cluster state that is replicated to all nodes and a large cluster state can threaten the stability of your cluster. Still, it is a simple solution that can take you pretty far.
</p>
</section><section id="conclusion" class="level2">
<h2><a href="#conclusion">Conclusion</a></h2>
<p>In the long run, it’s a huge benefit for multitenant systems to have independence between shards and tenants. In order to scale well, some tenants need to share a shard and others need multiple shards. By designing your application so that tenants can be moved out to a different index you not only make it possible to add the hardware required for the largest tenants, but you also ensure that the existence of a large tenant does not increase the cost of adding a small tenant.
</p>
<p>The final configuration scheme in this story may seem overengineered for many applications and you may feel that you would prefer your application not having to know about this. Mitigating the latter is best done by encapsulating it in the data access layer of your application. Regarding overengineering, you will probably not need to be this flexible in the beginning when your total data volume is low, but you should still have this situation in mind and consider how you can transition to such a scheme. I would also recommend that you once in a while check the size of your top 10 tenants and compare that to the average size of the other tenants. It is not uncommon with distributions where the top 10% of the largest tenants make up 90% of the total volume.
</p>
<p>The trickiest part is probably the fact that you don’t know which tenants that will grow, requiring you to be able to make changes along the way. The exact way to do so, can be very different from project to project. How much downtime is acceptable for a tenant? Is Elasticsearch your primary storage? Is it acceptable to see duplicates in searches while transitioning or is it better that new documents don’t show up until the transition is complete? These and more are questions you will have to ask yourself.
</p>
<p>I hope you enjoyed the read!
</p>
</section>

Discovering the Need for an Indexing Strategy in Multi-Tenant Applications

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>We all know memory is critical to Elasticsearch, but when should you add more? In the <a href="https://found.elastic.co/#/authentication/login/">Found</a> console we&apos;ve included a memory pressure indicator so that you can easily check where your cluster is at in terms of memory usage and capacity. For those familiar with how the JVM garbage collector works: &quot;The indicator uses the fill percentage of the old generation pool&quot;. In this article I will cover the basics of the old pool and why we chose that as the indicator.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>If you monitor the total memory used on the JVM you will typically see a sawtooth pattern where the memory usage steadily increases and then drops suddenly.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c9112798d8e381fd737/download?uid=blt1afdccc0ea7031a4?uid=blt1afdccc0ea7031a4" data-sys-asset-uid="blt60a41f852ebdc5bf" style="max-width:100%;" alt="Sawtooth">
</p><figcaption>Sawtooth</figcaption>
<p>The reason for this sawtooth pattern is that the JVM continously needs to allocate memory on the heap as new objects are created as a part of the normal program execution. Most of these objects are however short lived and quickly become available for collection by the garbage collector. When the garbage collector finishes you&#x2019;ll see a drop on the memory usage graph. This constant state of flux makes the current total memory usage a poor indicator of memory pressure.
</p>
<p>The JVM garbage collector is designed such that it draws on the fact that most objects are short lived. There are separate pools in the heap for new objects and old objects and these pools are garbage collected separately. After the collection of the new objects pool, surviving objects are moved to the old objects pool, which is garbage collected less frequently. This is due to the fact that it&#x2019;s less likely to be any substantial amount of garbage there. If you monitor each of these pools separately, you will see the same sawtooth pattern, but the old pool is fairly steady while the new pool frequently moves between full and empty. This is why we we have based our memory pressure indicator on the fill rate of the old pool.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c929e7a83165490c22b/download?uid=blt7f28e47e049bdfb1?uid=blt7f28e47e049bdfb1" data-sys-asset-uid="blte3f3b147f7b759bc" style="max-width:100%;" alt="Eden (new objects) and Oldgen (objects having survived collection in Eden)">
</p><figcaption>Eden (new objects) and Oldgen (objects having survived collection in Eden)</figcaption>
<p>So when do you need to upgrade your cluster? Well, this of course depends on your performance and stability needs, and hence the decision is yours, but I&#x2019;ll provide you with a guideline below.
</p>
</section><section id="green-is-good" class="level2">
<h2><a href="#green-is-good">Green is Good</a></h2>
<p>Anything below 75% - the green range of the indicator - is good and there is no need to upgrade unless you expect an increase in requests. In fact, there might be a substantial amount of garbage in the old pool, but there is no way to tell how much until the garbage collector starts, and that does not happen until the indicator reaches 75%. In other words, for a reported amount of up to 75% used heap the actual amount of used data might be less.
</p>
<p>Above 75% percent and you should start paying closer attention to what is going on, both in terms of indexing requests and data stored. Consider if there are appropriate optimizations that you can do. Check your mappings, your shard count and your cache utilization. If for some reason Elasticsearch seems to use more heap than you have data, you probably should check your mappings and your sharding strategy. More on that in <a href="/blog/found-sizing-elasticsearch">Sizing Elasticsearch</a> and <a href="/blog/found-crash-elasticsearch">Six ways to crash Elasticsearch</a>.
</p>
</section><section id="heavy-memory-pressure-consumes-cpu" class="level2">
<h2><a href="#heavy-memory-pressure-consumes-cpu">Heavy Memory Pressure Consumes CPU</a></h2>
<p>If the old pool is still above 75% after the collector finishes, the Java virtual machine will schedule a new collection, expecting to finish just before the pool runs out of memory. This means that higher fill rate in the old pool will result in more frequent collections and thus more CPU will be spent on garbage collections as the fill rate approaches 100%. This is the behaviour of the ConcurrentMarkSweep (CMS) collector that Elasticsearch uses.
</p>
</section><section id="is-high-memory-pressure-always-a-problem" class="level2">
<h2><a href="#is-high-memory-pressure-always-a-problem">Is High Memory Pressure Always a Problem?</a></h2>
<p>If you are happy with your current search performance, then there is no big cause for concern that your cluster has around 80% fill rate, but you need to be aware of that you do not have much to go on in case of spikes. The likelihood of a spike depends a lot on how much control you have over the data flow in and out of your cluster. Common sources are:
</p>
<ul>
	<li>A burst in new documents for indexing - If you use bulk indexing, do you have a cap on the bulk size?</li>
	<li>A burst in searches - Are searches triggered by users on from the internet? Do you have good cache utilization?</li>
	<li>Ad hoc analytics - Trying out some new aggregations or selecting a wide time range in Kibana can consume supprisingly big amounts of memory.</li>
</ul>
<p>High memory usage does not necessarily mean that your cluster cannot accept any more documents either. This depends on the memory profile of your searches. Does the memory requirement of a search grow with index size? If you use field data it will, but then doc values can be your saviour.
</p>
</section><section id="approaching-100" class="level2">
<h2><a href="#approaching-100">Approaching 100%</a></h2>
<p>If for some reason the Concurrent Mark Sweep collector was not started or did not finish in time to free up enough memory for a needed allocation, we get an allocation failure. An allocation failure means that there was not enough free memory at the time it was requested and execution has to wait for the garbage collector. When this happens the JVM gives up on using the CMS collector and triggers the old style stop-the-world collector. It is actually faster, but at the cost of blocking everything. All threads on the JVM will be halted until the collector finishes. This blocks Elasticsearch from receiving and sending network traffic as well. If the garbage collector takes too long, other nodes might think the node is down and the stability of your cluster could be at risk.
</p>
<p>If the collector is able to free up enough memory for the allocation, the cluster continues from exactly where it left off. Before learning of the amount of time passed during the pause and actions taken by the rest of the cluster during that time, the node might just be able to send off a few messages based on outdated information. The resilience of Elasticsearch has improved a lot in the latest versions, but concurrency and distribution is not only difficult to program with, it can be a challenge to test properly as well. In my experience, long garbage collection pauses is a repeated source of the more obscure concurrency bugs.
</p>
</section><section id="running-out-of-memory" class="level2">
<h2><a href="#running-out-of-memory">Running out of Memory</a></h2>
<p>If the collector is not able to free up enough memory then an allocation error is thrown and whichever thread that tried to allocate memory is killed. The problem is that the heap is a shared resource among all threads and thus there is no guarantee that the thread causing the spike is the one receiving the allocation error. Most of the threads in Elasticserach belong to pools that are able to recreate them. But it is very hard to know for sure that an instance is an healthy state after an out of memory error has occurred. It&#x2019;s for this reason that we at Found automatically reboot instances after out of memory errors. When that happens you will receive an email notification. For repeated alerts the emails are throttled and aggregated. We don&#x2019;t want to overload your inbox and we certainly don&#x2019;t want spam filters to start sorting out these alerts.
</p>
</section><section id="conclusion" class="level2">
<h2><a href="#conclusion">Conclusion</a></h2>
<p>Our recomendation is the following:
</p>
<ul>
	<li>Below 75%: No worries.</li>
	<li>75% to 85%: Acceptable if performance is satisfactory and load is expected to be stable.</li>
	<li>Above 85%: Now is the time for action. Either reduce memory consumption or add more memory.</li>
</ul>
<p>The 75% limit matches the <code>CMSInitiatingOccupancyFraction</code> setting that Elasticsearch uses. The 85% limit is based on our experience. There are no hard limits between 75% and 100%, but the performance will gradually worsen as the fill rate increases until you either run out of memory or the cluster falls apart due to long garbage collection pauses.
</p>
</section></body></html>

Understanding the Memory Pressure Indicator

Elasticsearch Upgrade Guide as Seen by the Client

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p>
<p>As much as we love Elasticsearch, at Found we&apos;ve seen customers crash their clusters in numerous ways. Mostly due to simple misunderstandings and usually the fixes are fairly straightforward. In our quest to enlighten new adopters and entertain the experienced users of Elasticsearch, we&apos;d like to share this list of pitfalls.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>Any datastore can be brought to its knees, especially if it contains a substantial amount of data compared to the available hardware or if the traffic is simply too high. This article will however focus on ways of taking down an Elasticsearch cluster that are not related to insufficient hardware. Some are quite common and real, while others less so. At any rate, it&#x2019;s good to be aware of them. These also emphasize why we only provide dedicated clusters at Found: you can shoot yourself in the foot, but not your neighbors&#x2019;.
</p>
</section><section id="mapping-explosion" class="level2">
<h2><a href="#mapping-explosion">Mapping Explosion</a></h2>
<p>One quick way of making Elasticsearch run out of memory and struggle to recover is by ignoring to consider the difference between keys and values when indexing documents. This has been explained in great detail in the article <a href="/blog/found-beginner-troubleshooting/#keyvalue-woes">Troubleshooting Elasticsearch searches, for Beginners</a>, but the essence is that the keys in the documents change, with the result that the mappings for the index is evergrowing. On a small scale, this will require a disproportionate large amount of heap space and for larger clusters it will typically cause problems with distributing the cluster state.
</p>
<p>Let&#x2019;s make this more clear with an example. We create a script that indexes documents with a single field, the key is a growing integer and the value is the square of the key. The first documents look like this:
</p>
<pre class="sourceCode json prettyprint">{&quot;1&quot;: 1}
{&quot;2&quot;: 4}
{&quot;3&quot;: 9}
{&quot;4&quot;: 16}
{&quot;5&quot;: 25}
{&quot;6&quot;: 36}
</pre>
<p>You can find the entire script in this <a href="https://gist.github.com/beiske/c784ed0cd5247aa58e07">gist</a>. On a single node cluster with 128MB the script managed to index about 30 000 documents before the node went down, but the index on disk is less than 7 MB. Depending on the use case it is not uncommon to have an index size between one and sixteen times the heap size in a normal system.
</p>
<p>I terminated the script after the first out-of-memory exception and our system restarted the instance. It managed to recover, but even at idle status it is doing heavy garbage collection. If you imagine what the mappings will look like for this index, with more than 30000 different fields, it&#x2019;s not difficult to understand that the cluster is struggling. Especially considering that this mapping is included in the cluster state that is broadcast to all nodes on several occasions. Put differently, we have not reached a limit on documents, but a limit on the metadata for an index.
</p>
<p>The proper way of indexing this data would have been something like this:
</p>
<pre class="sourceCode json prettyprint">{&quot;key&quot;: 1, &quot;value&quot;: 1}
{&quot;key&quot;: 2, &quot;value&quot;: 4}
{&quot;key&quot;: 3, &quot;value&quot;: 9}
{&quot;key&quot;: 4, &quot;value&quot;: 16}
{&quot;key&quot;: 5, &quot;value&quot;: 25}
{&quot;key&quot;: 6, &quot;value&quot;: 36}
</pre>
</section><section id="too-many-shards-or-the-gazillion-shards-problem" class="level2">
<h2><a href="#too-many-shards-or-the-gazillion-shards-problem">Too Many Shards or the Gazillion Shards Problem</a></h2>
<p>Too many shards or the gazillion shards problem, as some of the Elasticsearch developers like to refer to it, comes from the fact that there is a base cost to every shard and index even if it does not contain any documents.
</p>
<p>In this rather extreme example we create an index with 1073741824 shards.
</p>
<pre class="sourceCode bash prettyprint">curl -XPOST https://&lt;cluster_id&gt;-region.foundcluster.com:9243/toomanyshards -d &apos;{&quot;settings&quot;:{&quot;index&quot;:{&quot;number_of_shards&quot;:&quot;1073741824&quot;,&quot;number_of_replicas&quot;:&quot;1&quot;}}}&apos;
</pre>
<p>A few seconds after issuing this request the connection was closed and a few seconds after that I got an email saying the instance ran out of memory. This without indexing a single document. Now, the next time you set up Logstash with its default index template of one index per day, you might want to have a look at the size of those indexes after a few days. Why? Well, each index implies at least one shard and if you don&#x2019;t have that many log entries, you will probably do better with one index per week or per month. For Elasticsearch it is the same thing (in terms of performance) if you are having one index with two shards or two indexes with one shard. Both cases have two Lucene indexes.
</p>
</section><section id="arbitrary-large-size-parameter" class="level2">
<h2><a href="#arbitrary-large-size-parameter">Arbitrary Large Size Parameter</a></h2>
<p>The classic example: a developer knows that a search will not return that many hits, and he wants all of them without paging, so he simply sets the size parameter to an arbitrarily large size like Integer.MAX_VALUE (2147483647).
</p>
<p>When optimizing code like Elasticsearch developers do, there is no getting around making some assumptions. One of the assumptions is that a query usually has more hits than the number of elements to be returned. This implies that they will choose to prepare internal data structures for the number of documents specified in the size. In other words, if the size parameter is ridiculously large, then the Elasticsearch will create a ridiculously large internal data structure. And the one who pays the price is you, waiting forever - or so it seems! - for an insignificant result.
</p>
<p>In earlier versions of Elasticsearch this was perhaps the quickest trick to wreak havoc on somebody&#x2019;s cluster. Now, in recent versions of Elasticsearch this is not as big a problem as it used to be, but it&#x2019;s still a good reason to use the scan and scroll API instead. Even if you have tested it before and you know that there is lots of free heap on the instance, it can blow up simply because one of those structures needs contiguous memory on the heap, and your instance is not able to provide that without doing a major garbage collection.
</p>
</section><section id="scripting-vs-halting-problem" class="level2">
<h2><a href="#scripting-vs-halting-problem">Scripting vs Halting Problem</a></h2>
<p>In academia it is a known problem that one cannot determine if a running process will terminate or not, and since Elasticsearch does not have a timeout for long running scripts the following script will never halt. This results in the script keeping one search thread that will never be released back to the pool. Subsequent executions of such scripts will eventually consume all search threads and put all future searches in the queue until the queue reaches its maximum and all future searches are halted. This is also the case for the new sandboxed scripts.
</p>
<pre class="prettyprint">{
    &quot;script_fields&quot;: {
        &quot;test1&quot;: {
            &quot;lang&quot;: &quot;groovy&quot;,
            &quot;script&quot;: &quot;while (true) {print &apos;Hello world&apos;}&quot;
        }
    }
}
</pre>
<p>This script does however have one benevolent purpose: testing how your client behaves when the search queue fills up. Neeless to say, don&#x2019;t do this in production, use a staging or test cluster instead. By starting this script repeatedly until all the search threads are taken, the normal searches issued by your client will be put in the queue. This will simulate the effect of an overloaded cluster where searches take longer time to process than the rate at which they are received. Another entry into this pit is if your client mistakes an expensive and thus slow query for a failed query and executes it again. This is particularly bad if the query for some reason is not capable of utilizing caches, like queries calculating date ranges based on the current time.
</p>
</section><section id="too-deep-aggregations" class="level2">
<h2><a href="#too-deep-aggregations">Too Deep Aggregations</a></h2>
<p>Deep aggregations can still be an issue, but the following extreme example has been fixed in the 1.2 release of Elasticsearch. In Elasticsearch 1.1 we could create and index like this:
</p>
<pre class="sourceCode bash prettyprint">curl -XPOST https://&lt;cluster&gt;-region.foundcluster.com:9243/toodeepaggs -d &apos;{
   &quot;settings&quot;:{
      &quot;number_of_shards&quot;:1
   },
   &quot;mappings&quot;:{
      &quot;logs&quot;:{
         &quot;_all&quot;:{
            &quot;enabled&quot;:false
         },
         &quot;properties&quot;:{
            &quot;cust_id&quot;:{
               &quot;type&quot;:&quot;long&quot;
            },
            &quot;devi_id&quot;:{
               &quot;type&quot;:&quot;long&quot;
            },
            &quot;evt_date&quot;:{
               &quot;type&quot;:&quot;date&quot;,
               &quot;format&quot;:&quot;yyyy-MM-dd&quot;
            },
            &quot;hr&quot;:{
               &quot;type&quot;:&quot;string&quot;
            },
            &quot;type&quot;:{
               &quot;type&quot;:&quot;string&quot;
            },
            &quot;action&quot;:{
               &quot;type&quot;:&quot;string&quot;
            }
         }
      }
   }
}&apos;
</pre>
<p>And then, without indexing a single document we could make the following aggregation:
</p>
<pre class="sourceCode json prettyprint">{
    &quot;aggregations&quot;: {
        &quot;date&quot;: {
            &quot;terms&quot;: {
                &quot;field&quot;: &quot;evt_date&quot;
            },
            &quot;aggregations&quot;: {
                &quot;customer&quot;: {
                    &quot;terms&quot;: {
                        &quot;field&quot;: &quot;cust_id&quot;
                    },
                    &quot;aggregations&quot;: {
                        &quot;device&quot;: {
                            &quot;terms&quot;: {
                                &quot;field&quot;: &quot;devi_id&quot;
                            },
                            &quot;aggregations&quot;: {
                                &quot;type&quot;: {
                                    &quot;terms&quot;: {
                                        &quot;field&quot;: &quot;type&quot;
                                    },
                                    &quot;aggregations&quot;: {
                                        &quot;action&quot;: {
                                            &quot;terms&quot;: {
                                                &quot;field&quot;: &quot;action&quot;
                                            },
                                            &quot;aggregations&quot;: {
                                                &quot;hr&quot;: {
                                                    &quot;terms&quot;: {
                                                        &quot;field&quot;: &quot;hr&quot;
                                                    }
                                                }
                                            }
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
</pre>
<p>Even without a single document this way of nesting aggregations resulted in the instance running out of memory. I have tested this for instances with heaps as large as 8GB.
</p>
<p>This example is actually based on a real world problem where a developer most likely had misunderstood the aggregations API and created nested aggregations instead of sibling aggregates, but it still shows that the temporary structures created inside Elasticsearch when processing nested aggregations can be quite large.
</p>
<p>Even if this contrived example without a single document in the index has been fixed in newer versions of Elasticsearch it can be good idea to just snapshot the indexes you need over to a separate cluster when doing ad hoc analytics. Personally, I often do that when analyzing logs, our Logstash cluster is simply not tuned for heavy analytics. The safety of steering clear of ad hoc queries on the production cluster is not the only benefit. It also allows me to use a cluster with more memory, since it will only be running for the few hours that I need it.
</p>
</section><section id="long-garbage-collection-pauses" class="level2">
<h2><a href="#long-garbage-collection-pauses">Long Garbage Collection Pauses</a></h2>
<p>At Found we have experienced both customers and ourselves pushing clusters to their limits. Failing to take action when a cluster reaches its limits can be dangerous to the integrity of your data.
</p>
<p>To demonstrate this I added a second node to the previous cluster, making it redundant (2 x 128MB heap). I recreated the index with one shard and one replica and started the same script used for demonstrating the bad mappings, but instead of stopping on the first out-of-memory error I simply restarted the script and kept pushing more data.
</p>
<p>The first thing I noticed was a dramatic slowdown in indexing speed in the script. I then looked up the cluster in our metrics console and as expected it was collecting garbage continously. Elasticsearch configures the JVM to use the ConcurrentMarkSweep collector and since it is concurrent it avoids stop-the-world pauses as much as possible, but it cannot avoid them entirely, and with small instances like these the effect of the collector consuming CPU is also very much noticeable.
</p>
<p>Looking at the cluster in Kopf I also noticed that the replicas where out of sync, at least that&#x2019;s what the document counts reported. Elasticsearch is usually pretty quick to discard a replica if it does not match the master, but in this case the nodes were slowed down to a point where they had not been able to do so yet, or they were too slow to respond with an updated statistic to Kopf.
</p>
<p>In the end, both nodes went out of memory at the same time and I stopped the script. This allowed the nodes to come back online. The replica was then discarded and the master shard recovered from the local gateway of that node. This time the recovery mechanism worked and having a high availaility setup did make the cluster handle the load better, but the high load made the entire cluster unavailable, and for a while the redundancy was lost. In other words, if one does not respond to a situation like this, then it&#x2019;s just a matter of time before another error comes along and data loss is a fact.
</p>
</section><section id="conclusion" class="level2">
<h2><a href="#conclusion">Conclusion</a></h2>
<p>Despite the oddities and pitfalls described in this article, our opinion is that Elasticsearch is a really great product. Most of these issues can easily be avoided or will be addressed in future versions as similar issues have been adressed previously, but at the moment their consequences can be destructive.
</p>
<p>Personally, I am careful with who I allow access to my cluster and I would never use a shared cluster environment, at least not until there have been major changes to how Elasticsearch balances resources between different tasks. Even if the circuit breakers are constantly improving and considering more and more resources, can you be sure they will always be able to keep up with the new features?
</p>
</section></body></html>

Six Ways to Crash Elasticsearch

ZooKeeper - The King of Coordination

Getting Started with LIRE and Elasticsearch

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p>
<h2>How to Add Another Scripting Language</h2>
<p>You can use this post as a starting point to make a language plugin for any scripting engine that implements the JSR-223 interfaces.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>Currently Elasticsearch supports several different kinds of scripting languages. The core distribution supports MVEL, which is also the default language, but you can add more by installing plugins. For Closure, Groovy, Javascript and Python there are official plugins maintained by Elasticsearch Inc. In Elasticsearch 1.3 the default is planned to be changed to Groovy. In this article we will look closer at what it takes to create a language plugin and to demonstrate we will create a plugin targeting the JSR-223 API.
</p>
<p>The JSR-223 API, or the <code>javax.script</code> package, is not language specific, but an API created to be used by all scripting engines implemented for the JVM. The emphasis will be on the extension points of the scripting module and not so much <a href="/blog/found-writing-a-plugin/">how to write a plugin</a>.
</p>
</section><section id="the-scriptengineservice-interface" class="level2">
<h2><a href="#the-scriptengineservice-interface">The ScriptEngineService Interface</a></h2>
<p>The <code>ScriptEngineService</code> interface is the key component to implement. The rest is just bindings to register our implementation with the <code>ScriptService</code>.
</p>
<pre class="sourceCode java prettyprint">public interface ScriptEngineService {

    String[] types();

    String[] extensions();

    Object compile(String script);

    ExecutableScript executable(Object compiledScript, @Nullable Map<string, object=""> vars);

    SearchScript search(Object compiledScript, SearchLookup lookup, @Nullable Map<string, object=""> vars);

    Object execute(Object compiledScript, Map<string, object=""> vars);

    Object unwrap(Object value);

    void close();
}
</string,></string,></string,></pre>
<p>The <code>types()</code> and <code>extensions()</code> methods are where we declare the script types and file extensions supported by our engine. This information is used by the <code>ScriptService</code> to decide which engine to use for a given script, whether it be a script stored in a file or a dynamic one where you have declared a scripting language with the lang field.
</p>
<p>The <code>compile()</code> method should compile the given script, but simply returning the string would also work, a sensible choice if the script is for a non compiled language like Bash. What is worth noting about the method signature is that it is an example of the mnemonic pattern, as the actual type of the return value is hidden for the caller. The only thing the caller can assume about the return value is that it may be passed to the <code>executable</code>, <code>search</code>and <code>execute</code> methods of the engine that made it. Just like the mnemonic pattern where an external service may persist an objects internal state without being able to mutate it, the <code>ScriptingService</code> in Elasticsearch can cache compiled scripts. Hence the method should be implemented as a proper function in the sense that a given input will always produce the same output and the returned objects should be reusable.
</p>
</section><section id="the-scriptservice-gives-you-a-lot-for-free" class="level2">
<h2><a href="#the-scriptservice-gives-you-a-lot-for-free">The ScriptService Gives You a Lot for Free</a></h2>
<p>A key part of the scripting module is the ScriptService. It is the entry point for clients of the scripting module and acts as a facade that delegates to each <code>ScriptEngineService</code> instance. In addition to deciding which engine to invoke for a given script it also caches compiled scripts for later reuse. Threading however is left to be decided by the caller so make sure your <code>ScriptEngineService</code> is thread safe. This actually makes a lot of sense too when considering that Elasticsearch has dedicated thread pools for searches, indexing and so on.
</p>
<p>Once you have an implementation of <code>ScriptEngineService</code> you hook it up to the scripting module by including a method like this in your implementation of <code>AbstractPlugin</code>.
</p>
<pre class="sourceCode java prettyprint">public void onModule(ScriptModule module) {
    module.addScriptEngine(MyScriptEngineServiceImpl.class);
</pre>
<p>You can find more info on the <code>AbstractPlugin</code> in <a href="/blog/found-writing-a-plugin/#letting-elasticsearch-know-about-our-plugin">the writing a plugin article</a>. For this to work you must also remember to annotate the constructor of your implementation with <code>@inject</code>so Guice can do its magic and instantiate it.
</p>
</section><section id="the-jsr-223-api" class="level2">
<h2><a href="#the-jsr-223-api">The JSR-223 API</a></h2>
<p>Getting started using a JSR-223 engine can be as simple as adding to your class path and looking it up by its extension like this:
</p>
<pre class="sourceCode java prettyprint">ScriptEngineManager manager = new ScriptEngineManager();
ScriptEngine engine = manager.getEngineByExtension(&quot;lua&quot;)
</pre>
<p>In the above example I used Luaj which you can get with this Maven dependency:
</p>
<pre class="sourceCode xml prettyprint"><dependency>
  <groupid>org.luaj</groupid>
  <artifactid>luaj-jse</artifactid>
  <version>3.0-beta2</version>
</dependency>
</pre>
<p>The <code>javax.script.ScriptEngine</code> interface exposes among other the following methods which are useful to us:
</p>
<pre class="sourceCode java prettyprint">public interface ScriptEngine  {

    public Object eval(String script, Bindings n) throws ScriptException;

    public Bindings createBindings();

}
</pre>
<p>Using only those two methods we&#x2019;re able to create a valid implementation of <code>ScriptEngineService</code>, but it would not be very performant for compiled languages. In order to make compilations cacheable we check if the <code>ScriptEngine</code> also implements the <code>javax.script.Compilable</code> interface. This interface lets us transform a string into a <code>CompiledScript</code> that may be executed multiple times with different parameters.
</p>
<section id="registering-any-number-of-engines" class="level3">
<h3><a href="#registering-any-number-of-engines">Registering Any Number of Engines</a></h3>
<p>When it comes to registering the plugin, there is one key difference between a plugin adapting JSR-223 and one targeting a specific engine. The normal way of registering the implementation by the class name does not work when we need to register one instance of the class per engine found on the class path. The solution is to use the <code>multibinder.addBinding().toInstance()</code> feature in Guice.
</p>
<pre class="sourceCode java prettyprint">public class JSR223Module extends AbstractModule {

    @Override
    protected void configure() {
        Multibinder<scriptengineservice> multibinder = Multibinder.newSetBinder(binder(), ScriptEngineService.class);
        getClass().getClassLoader();
        ScriptEngineManager manager = new ScriptEngineManager();
        ESLogger logger = Loggers.getLogger(getClass());
        for (ScriptEngineFactory factory : manager.getEngineFactories()) {
            logger.info(&quot;Registering JSR223 engine: [{}-{}] for type: [{}] and extensions: [{}]&quot;,
                    factory.getEngineName(), factory.getEngineVersion(), factory.getLanguageName(),
                    factory.getExtensions());
            multibinder.addBinding().toInstance(new JSR223Adapter(factory));
        }
    }

}
</scriptengineservice></pre>
</section>
</section><section id="hello-world" class="level2">
<h2><a href="#hello-world">Hello World!</a></h2>
<p>The complete source is available on <a href="https://github.com/foundit/elasticsearch-jsr223">Github</a>. To test it we build it with <code>maven package</code> and use
</p>
<pre class="sourceCode bash prettyprint">bin/plugin --url file:///path/to/checkout/target/releases/elasticsearch-lang-jsr223-0.1.0-1.1.1-SNAPSHOT.zip --install elasticsearch-lang-jsr223
</pre>
<p>with the resulting zip file. Once Elasticsearch starts up, take a look at the log output to see which engines are detected. On my computer this was the result:
</p>
<pre class="prettyprint">[2014-05-22 15:02:30,937][INFO ][node                     ] [Hazmat] version[1.1.1], pid[24820], build[f1585f0/2014-05-03T16:01:42Z]
[2014-05-22 15:02:30,938][INFO ][node                     ] [Hazmat] initializing ...
[2014-05-22 15:02:30,964][INFO ][plugins                  ] [Hazmat] loaded [river-twitter, JSR-223-plugin], sites []
ScriptEngineManager providers.next(): javax.script.ScriptEngineFactory: Provider scala.tools.nsc.interpreter.IMain$Factory could not be instantiated
[2014-05-22 15:02:31,527][INFO ][no.found.elasticsearch.plugin.jsr223.JSR223Module] Registering JSR223 engine: [Luaj-Luaj-jse 3.0-beta2] for type: [lua] and extensions: [[lua, .lua]]
[2014-05-22 15:02:31,534][INFO ][no.found.elasticsearch.plugin.jsr223.JSR223Module] Registering JSR223 engine: [AppleScriptEngine-1.1] for type: [AppleScript] and extensions: [[scpt, applescript, app]]
[2014-05-22 15:02:31,579][INFO ][no.found.elasticsearch.plugin.jsr223.JSR223Module] Registering JSR223 engine: [Mozilla Rhino-1.7 release 3 PRERELEASE] for type: [ECMAScript] and extensions: [[js]]
[2014-05-22 15:02:32,814][DEBUG][discovery.zen.ping.multicast] [Hazmat] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
</pre>
<p>The AppleScript and Rhino engines are bundled with my JDK and since I launched Elasticsearch from Eclipse, which also adds the test scope of the project (technically not correct, but very convenient), it also added Luaj and tried to launch the Scala engine which failed to run for some reason I have yet to figure out.
</p>
<p>To test if the engines are launchable we can do the traditional &#x201C;Hello world&#x201D;. Any index with at least one document will do fine and then post the below query to the _search endpoint:
</p>
<pre class="sourceCode json prettyprint">{
    &quot;aggs&quot;: {
        &quot;script&quot;: {
            &quot;terms&quot;: {
                &quot;lang&quot;: &quot;lua&quot;,
                &quot;script&quot;: &quot;return &apos;Hello World!&apos;&quot;
            }
        }
    },
    &quot;size&quot;: 0
}
</pre>
<p>And once more for Rhino:
</p>
<pre class="sourceCode json prettyprint">{
    &quot;aggs&quot;: {
        &quot;script&quot;: {
            &quot;terms&quot;: {
                &quot;lang&quot;: &quot;ECMAScript&quot;,
                &quot;script&quot;: &quot;\&quot;Hello World!\&quot;&quot;
            }
        }
    },
    &quot;size&quot;: 0
}
</pre>
</section><section id="conclusion-and-future-work" class="level2">
<h2><a href="#conclusion-and-future-work">Conclusion and Future Work</a></h2>
<p>The implementation works, but it is just a proof of concept and will probably remain as such since most scripting engines will require a little more adaption and configuration to allow for good integration with Elasticsearch. That said, it should be a good starting point to make a plugin for any scripting engine that implement the JSR-223 interfaces. Feel free to check it out on <a href="https://github.com/foundit/elasticsearch-jsr223">Github</a>.
</p>
</section></body></html>

Extending the Scripts Module

Snapshot and Restore

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p>
<p>A follow up on the similarity article where we compare the precision and recall of the two models using Wikipedia articles.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>In the <a href="/blog/found-similarity-in-elasticsearch">previous article</a> I looked into the difference between BM25 and the tf-idf similarity as they are defined. The default similarity in Lucene and hence also Elasticsearch is however not a pure tf-idf implementation. It does actually have a form of document length normalization too. Thanks to<span class="citation" data-cites="ZacharyTong"><a href="#fn1" class="footnoteRef" id="fnref1"><sup>1</sup></a></span> for mentioning this! This makes it a little more difficult to compare it with the solutions described in academic papers and the conclusion that BM25 is mostly superior to tf-idf might not be applicable to the tf-idf implemation in Lucene. To investigate this issue I will do a comparison of precision and recall using Wikipedia articles.
</p>
</section><section id="indexing-the-data" class="level2">
<h2><a href="#indexing-the-data">Indexing the Data</a></h2>
<p>Using the <a href="https://github.com/elasticsearch/elasticsearch-river-wikipedia">Wikipedia river</a> indexing the articles is pretty straight forward, but if you plan on indexing the entire dataset there are a few gotchas. Due to the sheer size of the dataset just downloading the data takes a few hours and you will likely have to leave it working over night. At my first attempt I used the <a href="http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2">default image</a>, but this of course was updated to a new image midway in the download and it failed. The solution was simple, use a specific image. This also had the added benefit of making it easier to reproduce the exact same documents for indexing them with a different similarity model. Should you want to do something similar, I recommend you start querying right away as the river does not overload your cluster and it&#x2019;s better to figure out that you should have changed the mappings or some other index setting before having wasted all that time waiting.
</p>
<p>The river settings:
</p>
<pre class="sourceCode json prettyprint">{
    &quot;type&quot; : &quot;wikipedia&quot;,
    &quot;index&quot; : {
        &quot;index&quot;: &quot;wikipedia&quot;,
        &quot;type&quot;: &quot;page&quot;,
        &quot;url&quot; : &quot;http://dumps.wikimedia.org/enwiki/20131202/enwiki-20131202-pages-articles.xml.bz2&quot;,
        &quot;bulk_size&quot; : 1000,
        &quot;flush_interval&quot; : &quot;1s&quot;,
        &quot;max_concurrent_bulk&quot; : 2
    }
}
</pre>
<p>Index settings with default similarity:
</p>
<pre class="sourceCode json prettyprint">{
   &quot;settings&quot;:{
      &quot;number_of_shards&quot;:2,
      &quot;number_of_replicas&quot;:0
   },
   &quot;mappings&quot;:{
      &quot;page&quot;:{
         &quot;properties&quot;:{
            &quot;category&quot;:{
               &quot;type&quot;:&quot;string&quot;
            },
            &quot;disambiguation&quot;:{
               &quot;type&quot;:&quot;boolean&quot;
            },
            &quot;link&quot;:{
               &quot;type&quot;:&quot;string&quot;
            },
            &quot;redirect&quot;:{
               &quot;type&quot;:&quot;boolean&quot;
            },
            &quot;redirect_page&quot;:{
               &quot;type&quot;:&quot;string&quot;
            },
            &quot;special&quot;:{
               &quot;type&quot;:&quot;boolean&quot;
            },
            &quot;stub&quot;:{
               &quot;type&quot;:&quot;boolean&quot;
            },
            &quot;text&quot;:{
               &quot;type&quot;:&quot;string&quot;
            },
            &quot;title&quot;:{
               &quot;type&quot;:&quot;string&quot;
            }
         }
      }
   }
}
</pre>
<p>Index settings BM25 similarity:
</p>
<pre class="sourceCode json prettyprint">{
   &quot;settings&quot;:{
      &quot;number_of_shards&quot;:2,
      &quot;number_of_replicas&quot;:0
   },
   &quot;mappings&quot;:{
      &quot;page&quot;:{
         &quot;properties&quot;:{
            &quot;category&quot;:{
               &quot;type&quot;:&quot;string&quot;,
               &quot;similarity&quot;:&quot;BM25&quot;
            },
            &quot;disambiguation&quot;:{
               &quot;type&quot;:&quot;boolean&quot;,
               &quot;similarity&quot;:&quot;BM25&quot;
            },
            &quot;link&quot;:{
               &quot;type&quot;:&quot;string&quot;,
               &quot;similarity&quot;:&quot;BM25&quot;
            },
            &quot;redirect&quot;:{
               &quot;type&quot;:&quot;boolean&quot;
            },
            &quot;redirect_page&quot;:{
               &quot;type&quot;:&quot;string&quot;,
               &quot;similarity&quot;:&quot;BM25&quot;
            },
            &quot;special&quot;:{
               &quot;type&quot;:&quot;boolean&quot;
            },
            &quot;stub&quot;:{
               &quot;type&quot;:&quot;boolean&quot;
            },
            &quot;text&quot;:{
               &quot;type&quot;:&quot;string&quot;,
               &quot;similarity&quot;:&quot;BM25&quot;
            },
            &quot;title&quot;:{
               &quot;type&quot;:&quot;string&quot;,
               &quot;similarity&quot;:&quot;BM25&quot;
            }
         }
      }
   }
}
</pre>
<p>As you can see changing the similarity for a specific field is as simple as changing the mapping.
</p>
<section id="versions" class="level4">
<h4><a href="#versions">Versions</a></h4>
<p>For this test I used Elasticsearch 0.90.7 and elasticsearch-river-wikipedia 1.3.0.
</p>
</section>
</section><section id="the-query-used" class="level2">
<h2><a href="#the-query-used">The Query Used</a></h2>
<p>I started out experimenting with a basic match query, adressing all fields and comparing the results I got with what I could find in the search box on wikipedia.org. Two things became clear, the search on wikipedia.org mainly tries to match your keywords with titles and the document collection also has many documents that simply represent alternate titles and a redirect link to another document. Considering Wikipedia&#x2019;s encyclopedic style, the matching of search queries with titles is a natural corollary of the strong aboutness that the title has for such articles. The redirect titles are a good source of alternate titles and spelling, but the fact that the redirects appear as pages in the document collection is more of an implementation detail regarding mediawiki and when indexed with Elasticsearch, it would better fit as a bundle with the targeted document.
</p>
<p>The Wikipedia river actually detects the redirects and marks them with the boolean redirect field making it easy to filter them out. I decided to filter them away as for some documents the redirect document ended up with a better score as it contained the desired terms and where that much shorter. To better get to know the data I also tried some queries on just the titles and it was easy to find the articles I where looking for. If I were to create a new searchbox for Wikipedia with Elasticsearch I would definitely give a boost to a title or redirect title match.
</p>
<p>Now to address the real issue at hand. I needed a way to create a query set where each query has a preknown answer. Having witnessed the aboutness that the titles represent for their articles I constructed my test by simply drawing 1000 random articles and using the title as the query keywords and the document text as the answer.
</p>
<p>I then wrote a <a href="https://gist.github.com/beiske/8145331">script</a> that executed the below query replacing &lt;title&gt; for each chosen article.
</p>
<pre class="sourceCode json prettyprint">{
&quot;query&quot;: {
    &quot;filtered&quot;: {
        &quot;query&quot;: {
            &quot;match&quot;: {
                &quot;text&quot;: &quot;&lt;title&gt;&quot;
            }
        },
        &quot;filter&quot;: {
            &quot;bool&quot;: {
                &quot;must_not&quot;: [
                    {
                        &quot;term&quot;: { &quot;redirect&quot;: true }
                    },
                    {
                        &quot;regexp&quot;: { &quot;text&quot;: &quot;redirect[^\\s]+&quot; }
                    }
                ]
            }
        }
    },
    &quot;explain&quot;: true
}
</pre>
<p>For this test the redirects would ideally not have been indexed at all, but the above filters handles this. The term filter uses the redirect field created by the river and the regexp catches some remaining redirects that where not detected by the river.
</p>
<p>The query is limited to the 10 top ranking hits (default in Elasticsearch). Usually similarity models are compared by precision, <em>the percentage of relevant queries within the result</em>, and recall, <em>the percentage of all relevant documents that are included in the result</em>, but the limit of 10 hits and only one relevant document for each query makes these values a little skewed as all queries will either have a recall of zero or one hundred percent and a precision of either zero or ten percent. Instead of the usual academic criteria I will for this non-academic test use percentage of queries that did not find the desired document as an indicator of recall and the average rank of the desired document for those queries who did find it as an indicator of precision.
</p>
<p>The result of the script is an indication of precision and recall for both similarities when matching Wikipedia document titles with document text (titles excluded).
</p>
</section><section id="results-and-conclusion" class="level2">
<h2><a href="#results-and-conclusion">Results and Conclusion</a></h2>
<table width="55%">
<thead>
<tr class="header">
	<th style="text-align: left;">Similarity
	</th>
	<th style="text-align: left;">Average rank
	</th>
	<th style="text-align: left;">Documents not found
	</th>
</tr>
</thead>
<tbody>
<tr class="odd">
	<td style="text-align: left;">BM25
	</td>
	<td style="text-align: left;">2,07
	</td>
	<td style="text-align: left;">16,0%
	</td>
</tr>
<tr class="even">
	<td style="text-align: left;">Default
	</td>
	<td style="text-align: left;">2,44
	</td>
	<td style="text-align: left;">57,7%
	</td>
</tr>
</tbody>
</table>
<p>Clearly BM25 performed far better than the default similarity for this case, but it is important to keep in mind the 10 hits limit on the result size when interpreting these results. If one where to retrieve more hits, it is likely that the percentage of documents not found would drop for both similarities, but then the average rank would also grow.
</p>
<p>By processing the hits of every query until the desired number of hits is found or not one could calculate the actual precision and recall, but this would still only be circumstantial proof as the result would only be in regards to matching titles to articles in the English version of Wikipedia. As this experiment would be more in line with practical search than a formal proof I chose to also use the 10 hit limit as this is about as many hits as I would expect the average user to bother to consider reading before refining his query if the desired document was not found.
</p>
<p>In other words the result of this experiment is not a general proof that BM25 is always better than the default similarity, but it does suggest there can be a significant potential in using BM25 over the default similarity, at the very least for some cases. I strongly recommend taking the time to test both, with documents from your own use of Elasticsearch.
</p>
</section></body></html>

BM25 vs Lucene Default Similarity

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><h2>A Brief Introduction to the Similarity Models Available</h2><p>Elasticsearch now supports replacement of the default similarity model. In this article we will look into what a similarity model is with a special focus on tf-idf and bm25.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>A similarity model is a set of abstractions and metrics to define to what extent things are similar. That&#x2019;s quite a general definition. In this article I will only consider textual similarity. In this context, the uses of similarity models can be divided into two categories: classification of documents, with a finite set of categories where the categories are known; and information retrieval where the problem can be defined as &#x2018;find the the most relevant documents to a given query&#x2019;. In this article I will look into the latter category.
</p>
<p>Elasticsearch provides the following similarity models: default, bm25, drf and ib. I have limited the scope of this article to default and bm25. The divergence from randomness and information based similarities may feature in a future article.
</p>
</section><section id="default-similarity" class="level2">
<h2><a href="#default-similarity">Default Similarity</a></h2>
<p>The default similarity model in Elasticsearch is an implementation of <strong>tf/idf</strong>. Tf/idf is the most common vector space model. A vector space model is a model where each term of the query is considered a vector dimension. This allows for defining one vector for the query and another for the document considered. The scalar product of the two vectors is then considered the relevance of the document to the query. This implies that the positions of the words within a document are not used - a document is just a bag of words.
</p>
<p>A simple vector space model would be to set each coordinate in the document vector to one if the document has the word for the dimension, otherwise set it to zero. The query vector would be all one&#x2019;s if each word has the same weight. In such a model the scalar product would then be the sum of the words which are common for the document and the query. However, such a model would rank a document referring once to a query term in a side note the same as a document using the term repeatedly throughout the text. Which of the the two documents&#x2019; topics is identical to the term? The tf part in tf/idf helps us resolve this: tf stands for term frequency, and the solution is to use the term frequency instead of a simple zero or one in the document vector.
</p>
<p>In its simplest form you could calculate the term frequency by counting the number of occurences of the term in the document. This simplistic approach has one inherent problem. Assuming a query with the terms: <em>fire fox</em> generates the document vectors D1{15, 0} and D2{5, 5}. With a query vector of Q{1, 1}, document D1 will have a score of 15 and document D2 will have a score of 10. Let&#x2019;s say D1 is a lengthy document about fire hazards and D2 is a tutorial on how to install the browser Firefox. Which document do you think corresponds best to the query? To get the best of both worlds, by rewarding documents for specifically being about a term and also being about all the terms, tf-idf uses the logarithm of the term frequency for the vector value. Those who remember their calculus know that the logarithm of zero is infinitely negative while zero is the logarithm to one. To get the desired behaviour tf-idf simply adds one to all term frequencies before taking the logarithm. Using natural logarithms the document vectors become D1{2.772588722239781, 0} and D2{1.791759469228055, 1.791759469228055} with ranks 2.772588722239781 and 3.58351893845611 respectively. We now have a model that values having all terms without disregarding that a document can be more about a term than another.
</p>
<p>The other part of tf-idf yet to be mentioned is IDF, the Inverse Document Frequency. The definition of idf is the total number of documents in the collection (or index) divided by the number of documents that contain the word. It is used to address the fact that some words are more common in the language than others. In fact some words are that common they are considered noise. If a word is included in every document of a collection, how can it be helpful in choosing documents? Why don&#x2019;t we use stop words you say? Well, Elasticsearch does use stop words, however using stop words can prove to be inaccurate and of course, stop words are not only language specific, but also domain specific. IDF is a measure of the significance of a word&#x2019;s occurrence in a document. Similarly to TF, it is common to use the logarithm of IDF. The base number of the logarithm is not important, the crux of the matter is that it requires an exponential increase in the frequency to increase the IDF.
</p>
<p>As division by zero is somewhat problematic, it&#x2019;s recommended to add 1 to the denominator of the fraction. To avoid a negative IDF it is then common to add a +1 to the idf outside of the logarithm as in Lucene&#x2019;s <a href="http://lucene.apache.org/core/4_5_1/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html">TFIDFSimilarity</a>.
</p>
</section><section id="bm25" class="level2">
<h2><a href="#bm25">BM25</a></h2>
<p>BM25 belongs to the probabilistic models while TF-IDF is a vector space model, but their formulas are not as different as you might expect. Both models define a weight for each term as a product of some idf-function and some tf-function and then summarize that term weight as the score for the whole document towards the given query.
</p>
<p><span class="math">$\mathrm{Score}_{i} = A(tf_{i}) * B(idf_{i})$</span>
</p>
<p>For those who would like a thorough summary of the theoretical basis of BM25 I recommend <a href="http://staff.city.ac.uk/~sb317/papers/foundations_bm25_review.pdf">The Probabilistic Relevance Framework: BM25 and Beyond</a>. In this article I will rather try to explain the practical difference.
</p>
<section id="saturation" class="level3">
<h3><a href="#saturation">Saturation</a></h3>
<p>Both tf-idf and bm25 acknowledge that for highly frequent terms a further increase in term frequency has little significance for the relevance. The difference between them is that BM25 takes this a bit further. The saturation function of BM25 asymptotically approaches a limit for high term frequencies, while the logarithms of tf-idf has no boundary. For tf-idf, multiplying the the term frequency by the base of the logarithm, usually <span class="math">$e \approx 2,72$</span>, will always increase the logarithm by one.
</p>
<p>This difference in growth implies that for any fixed set of tuning parameters, the base of the logarithm included, you can create a document that will get a higher relative increase in term frequency score in tf-idf than in bm25.
</p>
<p>The following functions describe how term frequency contributes to score in BM25 and tf-idf:
</p>
<ul>
	<li><span class="math">$\mathrm{Saturation}_{BM25}(tf_{i}) = \frac{tf_{i}}{k_{1} + tf_{i}}$</span></li>
	<li><span class="math">$\mathrm{Saturation}_{tf-idf}(tf_{i}) = \mathrm{ln}\left(tf_{i} + 1\right)$</span></li>
</ul>
<p>If we plot these functions for typical values of k1 and term frequency from 0 to 100 we get:
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4c8c3dc542cb38c08256/download?uid=blt9b2d2d963d3bc6c6?uid=blt9b2d2d963d3bc6c6" data-sys-asset-uid="blt3aee6ed9aae8e413" style="max-width:100%;" alt="Saturation"><figcaption>Saturation</figcaption>
</figure>
<p>When reading this chart it&#x2019;s important to note that the absolute height of each graph is irrelevant as it could easily be adjusted with a simple boost factor. What is interesting is the relative growth of each curve. There are two things to note from this graph. Firstly, it supports the fact that the logarithm has no upper boundary and secondly, the differences between <span class="math">$k_{1}$</span> in the range from 1,2 to 2,0 are subtle. The latter observation has the practical consequence that if you don&#x2019;t have any idea what to choose for <em><span class="math">$k_{1}$</span></em>, the default value will most likely not be too far off.
</p>
</section>
<section id="average-document-length" class="level3">
<h3><a href="#average-document-length">Average Document Length</a></h3>
<p>The second major difference between tf-idf and BM25 is the use of document length in BM25. BM25 uses document length to compensate for the fact that a longer document in general has more words and thus is more likely to have a higher term frequency without necessarily being more pertinent to the term and thus no more relevant to the query. But it&#x2019;s not all black and white, some documents actually have a wider scope which makes the longer text justified. BM25 adjusts the term factor by a factor <span class="math">$B$</span>, which is calculated from a tuning parameter <span class="math">$b$</span> <span class="math">$\left(0 \leq b \leq 1\right)$</span>, the document length <span class="math">$dl$</span> and the average document length <span class="math">$avdl$</span>, like this:
</p>
<p><span class="math">$B = \left((1-b)+b\frac{dl}{avdl}\right)$</span>
</p>
<p>By defining <span class="math">$tf_{adjusted} = tf * B$</span> and substituting <span class="math">$tf$</span> for <span class="math">$tf_{adjusted}$</span> in previous weight defintion we get:
</p>
<p><span class="math">$W_{i}^{TF} = \frac{tf_{i}}{tf_{i} + k1\left(\left(1 - b\right) + b\frac{dl}{avgdl}\right)}$</span>
</p>
<p>This is how term frequency is used in BM25.
</p>
</section>
<section id="the-complete-term-weight" class="level3">
<h3><a href="#the-complete-term-weight">The Complete Term Weight</a></h3>
<p>To get the full weight for a term in BM25 we need to multiply <span class="math">$W_{i}^{TF}$</span> with <span class="math">$IDF_{i}$</span>. However BM25 defines idf slightly differently than the default similarity and this difference is reflected in Lucene&#x2019;s implementation. The definition used is:
</p>
<p><span class="math">$IDF_{i} = \mathrm{log}\left(1 + \frac{D - d_{i} + 0.5}{d_{i} + 0.5}\right)$</span>
</p>
<p>Where <span class="math">$D$</span> is the total number of documents int the collection and <span class="math">$d_{i}$</span> is the number documents containing the i&#x2019;th term of the query.
</p>
</section>
<section id="how-lucene-does-bm25" class="level3">
<h3><a href="#how-lucene-does-bm25">How Lucene Does BM25</a></h3>
<p>Lucene uses a variation of BM25 where the numerator of the fraction is multiplied with <span class="math">$\left(k_{1} + 1\right)$</span>. That&#x2019;s a common trick to make the formula more similar to other ranking functions and hence also easier to compare the resulting values. As this expression is a constant, it has not consequence for the ranking produced.
</p>
<p><span class="math">$W_{i}^{\mathrm{Lucene BM25}} = idf_{i} * boost * \frac{(k_{1} + 1)tf_{i}}{tf_{i} + k_{1}\left(\left(1 - b\right) + b\frac{dl}{avgdl}\right)}$</span>
</p>
<p>As Lucene aims for performance, the formula is not calculated entirely at query time. Lucene uses the similarity model chosen both while indexing and querying. This reflects on Elasticsearch as well.
</p>
</section>
<section id="tuning-bm25" class="level3">
<h3><a href="#tuning-bm25">Tuning BM25</a></h3>
<p>Regarding the tuning parameters, <span class="math">$k_{1}$</span> and <span class="math">$b$</span> there is a reason they are defined as parameters. The ideal value is dependent on the document collection. Unfortunately, there is no silver bullet and trial and error is the way to go, but it does help to have an idea of which documents are relevant to the queries used and to have a methodological approach.
</p>
</section>
</section><section id="conclusion" class="level2">
<h2><a href="#conclusion">Conclusion</a></h2>
<p>There is a reason why TF-IDF is as widespread as it is. It is conceptually easy to understand and implement while also performing pretty well. That said, there are other, strong candidates. Typically, they offer more tuning flexiblity. In this article we have delved into one of them, BM25. In general, it is known to perform just as good or even better than TF-IDF, especially on collections with short documents. Bearing that in mind, don&#x2019;t forget that similarity rank is not the only ranking contributor in Elasticsearch. To create a good search experience it is key to combine textual similarity rank with metadata suiting the given case, such as the last time document was updated or if there is some proximity between the authors of the query and the document. Read our follow-up article where we compare the precision and recall of the two models using Wikipedia articles: <a href="/blog/found-bm-vs-lucene-default-similarity">BM25 vs Lucene Default Similarity</a>.
</p>
</section></body></html>

Similarity in Elasticsearch

City Bikes Part two

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>Leader election is one of the most tricky things to do in distributed systems. At same time, understanding how a leader is elected and the responsibilities of the leader is key to understanding a distributed system.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>All distributed systems have to deal with maintaining consistency in one way or another. To simplify, we may divide the strategies into two groups. The ones that attempt to avoid inconsistencies from occurring and the ones that define how to reconcile them. The latter is very powerful for applicable problems, but imposes unacceptable constraints on the data model in others. In this article we will look into one example of the first category and how it copes with network failures.
</p>
</section><section id="why-use-a-leader" class="level2">
<h2><a href="#why-use-a-leader">Why Use a Leader?</a></h2>
<p>Having a node designated as a leader in a distributed system is actually quite similar to the keyword synchronized in Java. The classic example for Java synchronized is when two threads attempt to increment the same integer. For instance, when two threads attempt to deposit $100 each to the same account. Each thread has to read the balance, increment it and write it back. Given a starting balance of $100 and no thread synchronization the following may occur:
</p>
<ul>
	<li>Thread A reads balance ($100)</li>
	<li>Thread B reads balance ($100)</li>
	<li>Thread A calculates new balance ($200)</li>
	<li>Thread B calculates new balance ($200)</li>
	<li>Thread A writes new balance ($200)</li>
	<li>Thread B writes new balance ($200)</li>
</ul>
<p>Clearly the account started with $100, a total of $200 was deposited, and the final balance should have been $300. However, thread A&#x2019;s deposit is overwritten as thread B has done its calculation on data that is stale by the time it writes its updated balance.
</p>
<p>The solution in Java is of course to use the keyword synchronized on the code doing the incrementation and it will make one thread finish its work with the balance before the other gets to read the balance.
</p>
<p>If you consider a thread as a node, it&#x2019;s easy to imagine the same problem in a distributed system, but there is no synchronized keyword to the rescue. The reason being that synchronized is implemented using semaphores, and in turn, they are backed by the underlying operating system and the hardware it&#x2019;s running on. But if we look at what synchronized does, we might adapt the concept to a distributed setting. When declaring a block of code as synchronized in Java, it is always synchronized to the monitor of a specific object. The size of this monitor is one. This means that only one thread may be inside the monitor at any given time. When a thread requests the monitor and the monitor is available, it is allowed to enter instantly. If the monitor is occupied, the thread is put in the waiting pool and suspended until the monitor is released.
</p>
<p>How can we adapt this to a distributed setting? Well, we do have the option to create as many monitors as we want on each node, but for a given resource that requires protection there may only exist one monitor. In other words, there may only be one monitor for each account balance. This transforms the problem of maintaining data consistency to the problem of ensuring all nodes agree on which node is responsible for implementing the monitor for each balance. In our simple example this could be handled by the leader, and all that remains to be done is to elect a leader.
</p>
<p>If you have a peer to peer background like myself you would probably suggest using a distributed hash table rather than a leader to decide which node implements the monitor for each resource. In fact, there are some really great DHT-implementations out there that deal with thousands of nodes leaving and joining every hour. Given that they also work in a heterogenous network with no prior knowledge of underlying network topology, query response times of four to ten message hops is pretty good, but in a grid setting where the nodes are homogenous and the network is fairly stable, we can do better.
</p>
<p>Another simplification in the typical scenario for Elasticsearch is that there are not that many nodes in the cluster. Typically, the number of nodes is far less than the number of connections a single node is capable of maintaining, and a grid environment does not have to deal with nodes joining and leaving that frequently. This is why the leader approach is a better fit for Elasticsearch.
</p>
</section><section id="the-zen-way" class="level2">
<h2><a href="#the-zen-way">The Zen Way</a></h2>
<p>So we have a leader that regularly pushes the current version of the global cluster state to each node. What happens if the leader goes down? Just because our nodes are fairly reliable does not imply that we can accept a single point of failure. Provided that only the leader node has crashed and all other nodes are still capable of communicating, Elasticsearch will handle this gracefully as the remaining nodes will detect the failure of the leader - or rather the absence of messages from the leader - and initiate leader election. If you are using ZenDiscovery (default) then the process is like this:
</p>
<ul>
	<li>Each node calculates the lowest known node id and sends a vote for leadership to this node</li>
	<li>If a node receives <em>sufficiently many votes</em> and the node also voted for itself, then it takes on the role of leader and starts publishing cluster state.</li>
</ul>
<p>The definition of <em>sufficiently many votes</em> to win an election is what is known as a quorum. In Elasticsearch the quorum size is a configurable parameter. A node in a distributed system is not able to determine whether another node is dead or whether the network is not able to deliver its messages to it, it is therefore common to have a previously agreed upon threshold - the quorum size - to represent the other party&#x2019;s votes. Let&#x2019;s exemplify with the following scenario: A cluster has nodes A, B, C, D, E and quorum size 3. A is the leader. It so happens that A and B is on one network and C, D and E on another. The networks are connected through a link.
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4c9b9985d58976376ca5/download?uid=blt74a77a580f55964a?uid=blt74a77a580f55964a" data-sys-asset-uid="blt8bdce1626de819c6" alt="Initial network" style="max-width:100%;"><figcaption>Initial network</figcaption>
</figure>
<p>When this link fails, A and B are still able to communicate with each other, but not with C, D and E. Similarly, on the other network C, D and E may communicate with each other, but not with A and B.
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4c9b58208ba076e282c2/download?uid=bltaa821b8fd8a647ed?uid=bltaa821b8fd8a647ed" data-sys-asset-uid="blta2af94513dd52d7c" alt="Partitioned network" style="max-width:100%;"><figcaption>Partitioned network</figcaption>
</figure>
<p>What happens next is this: nodes C, D and E detect that they no longer have contact with the leader A and subsequently initiate leader election by sending votes to C. Once C has received three votes it takes on the role of leader and starts publishing to C, D and E. On the other network, the leader A detects that it no longer has contact with nodes C, D and E. Leader A calculates that the new cluster size is less than the quorum size and gives up the leader role and effectively stops nodes A and B from responding to queries until the link is restored.
</p>
<p>In real life it&#x2019;s unlikely that someone trips over on a crucial network cable, but networks are more complex than my example above, and network partitions are actually not that uncommon. It is not hard to imagine a network split when a system is relying on multiple datacenters; another likely culprit of tricky network errors is a wrongly configured router or switch. As mentioned in <a href="http://aphyr.com/posts/288-the-network-is-reliable">the network is reliable</a>, it does occur that network cards do things like dropping all inbound packets while still delivering outbound packets, with the result that a server still sends heartbeats but is unable to service any requests.
</p>
</section><section id="avoid-split-brain" class="level2">
<h2><a href="#avoid-split-brain">Avoid Split Brain</a></h2>
<p>The concept of a quorum size has two immediate advantages. Firstly it simplifies the election process as votes only need to be delivered to the node they&#x2019;re voting for. Secondly and more importantly it is the only way to avoid a split brain. Imagine the same cluster and the same network split, but with a quorum size of two. C would still be elected leader, but A would not give up its leader role. This would result in two leaders, with clusters unknown to each other. This is what is known as a split brain. Both clusters would be accepting read and write operations, and as expected, they would be out of sync. Without human intervention they would probably never recover. Depending on the data model it might be impossible to unify the two data versions, forcing one to simply discard all data on one of the clusters.
</p>
</section><section id="dont-reinvent-the-wheel" class="level2">
<h2><a href="#dont-reinvent-the-wheel">Don&#x2019;t Reinvent the Wheel</a></h2>
<p>As you might expect, leader election has been an intriguing topic in academic circles for many years and quite a few smart people have done a great deal of contemplation. If you are more keen on getting your distributed system working than putting a massive effort into research and development, you&#x2019;re probably better off with having a go at implementing a well known algorithm. Some day in the future when you have finally figured out that crazy bug in your system, it&#x2019;s more likely to be just a bug in the implementation than a major design flaw in the algorithm. Of course, the same argument applies to adapting or integrating an existing implementation rather than implementing from scratch, especially if you want one of the more advanced algorithms. <a href="https://github.com/elasticsearch/elasticsearch/issues/2488">This bug report</a> illustrates how tricky it can be to create a good leader election implementation, even when you have a sound algorithm.
</p>
<section id="the-bully-algorithm" class="level3">
<h3><a href="#the-bully-algorithm">The Bully Algorithm</a></h3>
<p>The bully algorithm is one of the basic algorithms for leader election. It assumes that all nodes are given a unique ID that imposes a total ordering of the nodes. The current leader at any time is the node with the highest id participating in the cluster. The advantage of this algorithm is an easy implementation, but it does not cope well with a scenario whereby the node with the largest id is flaky. Especially in a situation where the node with the largest id tends to be overloaded by the chores of the leader role. Consequently, it will crash as the leader; the node with the second largest id will be elected; and the largest id node will recover - as it&#x2019;s no longer overloaded - and subsequently initiate leader election again, only to be elected and crash yet again. However, in low level hardware interfaces the bully algorithm is quite common. It might be tempting to avoid thrashing by postponing election until the current leader fails, but that will easily lead to a split brain.
</p>
</section>
<section id="paxos" class="level3">
<h3><a href="#paxos">Paxos</a></h3>
<p>Paxos is actually a much more than a leader election algorithm. Paxos is a family of different protocols for maintaining consensus in a distributed system. I will not go into detail about the varieties of Paxos, but rather discuss the concepts and qualities of Paxos in general. The data model used in Paxos is a growing list of statements where each statement has a unique number. The mathematically proven guarantees of Paxos are that two nodes will never have different statements for the same number, and as long as there is a quorum of nodes participating the algorithm will make progress. This translates to the nodes never being inconsistent and the algorithm never going into deadlock, but it may halt if there are not enough nodes online. These guarantees are in fact quite similar to what we want for a leader election algorithm. We want all nodes participating in the cluster to agree on which node is the leader, and we do not want any other nodes to be able to run off and create their own cluster, resulting in a split brain. Performing leader election with Paxos then becomes as simple as proposing the statement &#x201C;I am the leader&#x201D;. If it is accepted, then you are the leader until another node proposes to be the leader. However, this alone is not sufficient to avoid a split brain. In the partitioned network example above the existing leader A will not be notified of the election of Node C on the other switch. There needs to be another criteria to end the leadership. One option is that when initiating leader election the statement used is on the form: &#x201C;I am leader until <some criteria="">&#x201D;, but a more preferred option is that the leader is able to detect that it no longer has contact with a quorum and then demotes itself.</some>
</p>
<p>Given the previous example of a network partition, let&#x2019;s imagine that a janitor trips on the network cable to node C and then reconnects the cable to the other switch, resulting in a new partition of the network:
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4c9b58208ba076e282c0/download?uid=blt2233ba1c8a54fe38?uid=blt2233ba1c8a54fe38" data-sys-asset-uid="blt205a7b37646ce0f7" alt="Moved node" style="max-width:100%;"><figcaption>Moved node</figcaption>
</figure>
<p>The interesting part here is that the quorum now consists of nodes A, B and C instead of the previous C, D and E. If the nodes where running Paxos, their data could look something like this:
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4c9c0342dfd738263371/download?uid=blte70b54df709027b8?uid=blte70b54df709027b8" data-sys-asset-uid="blt62f71e91c7af6e1f" alt="Paxos example" style="max-width:100%;"><figcaption>Paxos example</figcaption>
</figure>
<p>Depending on the exact implementation of leader election through Paxos, a number of outcomes are possible, but they all have two properties in common. Nodes D and E will not be able to elect a new leader as they do not have a majority. Nodes A and B will not elect a new leader before learning that node C was elected in round 2. Assuming that the end criterion of each leader term is a timeout, there are two possible outcomes. If node C establishes contact with nodes A and B before its term is ended, it will reestablish a quorum, resume its leader role, and no leader election is necessary. If its term is ended, then either node A, B or C may be elected. Assuming node A is the first to discover node C on the network, it will try to get re-elected as leader for term 2, but node C will reject this as it already has a leader for term 2. Node A then has the option to attempt to be elected for term 3. If it does, it will then also inform node B about node C&#x2019;s time as leader in term 2. Now, what if node C and A try to get elected at the same time? The full answer to that question requires delving into the specifics of Paxos, which out of scope for this article, but the short answer is that election will fail. In fact, the original paxos paper suggests using a leader election algorithm to decide which node should be the one initiating new proposals, but since the Paxos protocol guarantees that there will not be inconsistencies even if there are multiple nodes proposing conflicting statements, this leader election algorithm could be just a simple heuristic to avoid consecutive failed rounds. For instance, a node might decide to wait a randomized amount of time before reattempting to propose a failed statement. This flexibility of Paxos in terms of when and how to do leader election can be a great advantage over the simple bully algorithm, having in mind that in real life there are far more failure modes than a dead network link.
</p>
</section>
</section><section id="conclusion" class="level2">
<h2><a href="#conclusion">Conclusion</a></h2>
<p>In this article I have tried to give a brief insight into some of the work that has gone into leader election in academic circles and the importance of doing it right. The condensed version is this:
</p>
<ul>
	<li>Quorum size is really important if you care about your data.</li>
	<li>Paxos is really robust, but not as easy to implement.</li>
	<li>It&#x2019;s better to implement a well known algorithm where the strengths and flaws are known rather than getting a big surprise further down the road. There are certainly plenty of options out there.</li>
</ul>
</section></body></html>

Leader Election, Why Should I Care?

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p>
<p>The Zen discovery plugin might seem like magic, but replacing it is actually not that hard. Getting down and dirty with Zen might very well pay off in your particular setup.
</p><section id="so-what-does-it-do-and-how" class="level2">
<h2><a href="#so-what-does-it-do-and-how">So What Does it Do, and How?</a></h2>
<p>The purpose and responsibility of the discovery plugin is to provide up to date information about the cluster to an Elasticsearch node. This boils down to the following behaviour:
</p>
<ul>
	<li>On non-master nodes:
	<ul>
		<li>Elect a master, when no known master.</li>
		<li>Receive cluster state from master and update local ClusterService</li>
	</ul></li>
	<li>On the master:
	<ul>
		<li>Publish changes to the cluster state</li>
		<li>Detect nodes leaving and joining the cluster and update cluster state accordingly.</li>
	</ul></li>
</ul>
<p>This is done by providing an implementation of the discovery interface and communicating with the ClusterService. The essential parts of the discovery interface are:
</p>
<pre class="sourceCode java prettyprint">DiscoveryNode localNode();
void publish(ClusterState clusterState);
</pre>
<p>The localNode-method does not really do any magic beyond what you would expect, it returns the DiscoveryNode representing the local instance of Elasticsearch. The publish-method is where half the fun happens: when the ClusterService on the master node receives an updated state it uses the publish-method to notify other nodes. How and where to publish the ClusterState is not specified, but Elasticsearch relies on all other nodes participating in the cluster to receive the update. Exemplified by the ZenDiscovery, the publish-method first updates internal state with the new cluster state - e.g. which nodes to send heartbeats to - then transmits the cluster state as a message via the transport protocol to all other nodes. Depending on the motivation for replacing the ZenDiscovery one could for instance update a global third party directory with the cluster state instead of transmitting it directly. Bear in mind that it&#x2019;s important that all nodes have the same view of the cluster; regardless of method used it must be able to transmit the cluster state reliably and efficiently.
</p>
<p>The other half of the fun is what the discovery plugin does on its own. As the discovery interface extends LifecycleComponent<discovery>, it&#x2019;s provided with start- and-stop methods allowing it to register and release handlers and other resources as part of the instance lifecycle. Again, dependent on the motivation for creating a new discover plugin you may do different things, however the lifecycle start-method is where you would set it all up. As a minimum you would have to listen for cluster state updates from the master, but you would also most likely want:</discovery>
</p>
<ul>
	<li>node discovery in the event of becoming the master</li>
	<li>master election in case the master goes offline</li>
	<li>network/node fault detection to ensure other nodes are reachable</li>
</ul>
</section><section id="why-create-your-own-plugin" class="level2">
<h2><a href="#why-create-your-own-plugin">Why Create Your Own Plugin?</a></h2>
<p>We have now briefly discussed what a discovery plugin might do and what is expected of it, or what you can refer to as the what and how. Let&#x2019;s discuss the why. The short answer to why implement another discovery plugin is that you are not happy with ZenDiscovery. But why are you not happy with it? After all, it&#x2019;s the one most commonly used discovery plugin. Why do you think that you can make a better plugin than the core developers did? The answer lies in the question. The solution for everybody has to work with any network in any environment. Hence <em>the solution for everybody</em> can only do discovery through multicast or a preconfigured list, it has to handle leader election on its own (which is one of the tough problems in distributed systems), and node failure detection is limited to ping timeouts. In your environment you may have resources that could be utilized to greatly simplify one or more of these challenges. In addition to ZenDiscovery, Elasticsearch is bundled with LocalDiscovery which assumes all nodes run on the same VM and hence does node discovery through a singleton object and avoids the network altogether for cluster state publishing. By installing the cloud-aws plugin you can use <a href="/guide/en/elasticsearch/plugins/2.0/cloud-aws-discovery.html">EC2Discovery</a> which does node discovery through the EC2 API. And finally there is the <a href="https://github.com/sonian/elasticsearch-zookeeper">sonian-zookeeper-plugin</a> that uses ZooKeeper for node discovery and leader election. At Found, we have done something similar to the sonian plugin to improve leader election and node discovery using ZooKeeper, but it&#x2019;s currently not publicly available as it&#x2019;s too tightly integrated with our other infrastructure.
</p>
</section><section id="a-note-on-leader-election" class="level2">
<h2><a href="#a-note-on-leader-election">A Note on Leader Election</a></h2>
<p>If you are curious about leader election in general and the challenges related to achieving consensus in a distributed system, you should read up on <a href="http://en.wikipedia.org/wiki/Paxos_%28computer_science%29">Paxos</a>. That being said, it is usually best not to reinvent the wheel and, depending on the resources available in your environment, you can usually find someone who have already made a good implementation like <a href="http://curator.incubator.apache.org/curator-recipes/leader-latch.html">the curator frameworks leader latch</a>, if you are already using ZooKeeper.
</p>
</section></body></html>

AUTHOR

Articles by Konrad Beiske

Cluster cloning in the cloud

Packetbeat on Found

Discovering the Need for an Indexing Strategy in Multi-Tenant Applications

Understanding the Memory Pressure Indicator

Elasticsearch Upgrade Guide as Seen by the Client

Six Ways to Crash Elasticsearch

ZooKeeper - The King of Coordination

Getting Started with LIRE and Elasticsearch

Extending the Scripts Module

Sign up for Elastic Cloud free trial

Products & Solutions

Company

We're hiring

Resources

Solutions

Elastic (ELK) Stack

New

Featured topics

News

AUTHOR

Articles by Konrad Beiske

Cluster cloning in the cloud

Packetbeat on Found

Discovering the Need for an Indexing Strategy in Multi-Tenant Applications

Understanding the Memory Pressure Indicator

Elasticsearch Upgrade Guide as Seen by the Client

Six Ways to Crash Elasticsearch

ZooKeeper - The King of Coordination

Getting Started with LIRE and Elasticsearch

Extending the Scripts Module

Sign up for Elastic Cloud free trial

Company

We're hiring

Resources