Interested in how our leadership manages the teams making the Elastic Stack? In this edition: Kevin Kluge, Senior Vice President of Engineering at Elastic.

I joined Elastic six years ago. How I came to Elastic is an interesting story, and one I’d like to share, because I think it’s revealing in terms of our engineering culture and what it means to be a company that is distributed.
Six years ago I was working at a company called Citrix, which had acquired a company that I worked for called Cloud.com. At the time I was happy at my job. I’d managed about 100 people a few times, and I felt capable of that. As the next step in my career seemed imminent, I was eager to work somewhere that I could build something bigger, somewhere that lasted more than a few years before being acquired, which had been the history of my career so far.
This is where Elastic came in. I knew Peter Fenton and Mike Volpi, who had invested early on in Elastic. They told me to take a look at Elastic, and since I respected their opinion, I downloaded <a href="https://www.elastic.co/elasticsearch">Elasticsearch</a>. I had heard of the product, but had never used it, and I really didn’t know much about what it was capable of. Within 15 minutes I had a cluster up and running and had ingested some data. It was clear to me from the start that this was an incredibly powerful tool.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt36db4d183b18e069/5dd73ea28fa7cb3b40be5a7d/Kevin_Kluge_Presenting.png" data-sys-asset-uid="blt36db4d183b18e069" alt="" style="display: block; margin: auto;"><figcaption>Me presenting at Elastic{ON} about engineering culture at Elastic.</figcaption>While interviewing I was impressed with the team and Shay’s approach to management. When Elastic made an offer, I was excited about the opportunity, but I was still apprehensive. I worked a lot of long hours at Cloud.com, and subsequently at Citrix. Sometimes I worked until dinner, and sometimes through it. Sometimes I went home for dinner and then went back to work until 11 p.m. My wife and I had recently had our third child and there was no way I was going to miss out on raising her. Working long hours, typical of so many tech companies, would mean picking work over my family — again.
So I went to Shay and Steven Schuurman (who was CEO at the time) and raised my concern. I remember Shay saying to me, to my surprise:&nbsp;Why don’t you leave at 4 p.m.? That simple allowance — being able to go home at 4 p.m. — seemed incomprehensible to me. It felt so against the grain of the typical Silicon Valley mentality, and of my own experience, that I immediately took the job.
When I joined Elastic there were only 14 engineers. We knew that we needed to scale to create something that excelled in the long term. The challenge was — how could we do that while retaining the values that made me join?
<h2>Hiring a self-motivated team</h2>The first step was building a good team of engineers. The first 14 engineers at Elastic when I joined were geographically dispersed, meaning we were distributed from the start. I’ve reflected on what this humble beginning as a small, distributed team meant for our engineering culture in those early days and how it’s shaped who we are today, and I believe this distributed nature attracted engineers who knew that they needed to be self-motivated to be successful. This is something that’s still important to Elastic today. If you're 500 miles and multiple time zones away from the nearest coworker, you have to come up with a lot of your own direction.
This is what drove us to hire more senior engineers in those earlier days and what inspires us to hire self-motivated engineers — no matter their experience — today. In the beginning, we didn’t have the infrastructure to be like Google or Facebook who could take on thousands of people straight out of school and make them successful. But now that we have bigger teams&nbsp;and better organization, we've been able to hire motivated people just out of school and ensure that they, too, are successful.
<h2>Giving engineers the focus they need to dive deep</h2>In the early days, if somebody was going to join Elastic, they had to be willing, as an engineer, to do all kinds of things that weren't in the job description. This is completely natural in a company that’s just starting out. Some people really enjoyed having a lot of different responsibilities. (I did.)&nbsp;But as we grew, we ended up specializing and building specific teams to take over some of the roles that the engineers were performing in the beginning. Now we’re proud to have sales engineers, trainers, support engineers, and more&nbsp;making sure our Elasticians are able to do the job they were hired to do&nbsp;—&nbsp;and to excel in those areas.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8ad00775a6416681/5dd70b396aec2738d0786441/GAH_Kevin_Deep_Dive.jpg" data-sys-asset-uid="blt8ad00775a6416681" alt="" style="display: block; margin: auto;">
<figcaption>Providing a deep dive presentation at Elastic Global All Hands, 2019.</figcaption>Hiring distributed was incredibly important when building those small teams. We were trying to run 24/7 support, and being able to hire across the world allowed us to make coverage in different time zones without asking our Elasticians to change their sleeping patterns (too much). <a href="https://www.elastic.co/about/our-source-code">Finding balance in life</a>, even in the early days, was of the utmost importance to the Elastic team.
Beyond focused roles, we wanted our engineers to feel free to create in the manner they work best. To feel empowered. Having distributed engineers made this an easy one to solve. Managing distributed Elasticians makes it impossible to micromanage people, even if you wanted to. So we embrace that. I don't want to know where somebody is on a Tuesday at 2 p.m. Instead, regardless of what we’re producing as a team — a blog post, lines of code, closing a support case — we need to be able to look back at what we achieved at the end of the week and be proud of it. If you can't look back at a week's worth of work and be proud of what you did, something has gone wrong, and you need to self-reflect on what might have happened. The work should speak for itself.
<h2>Organizing our teams</h2>Org structure has also been an important component for making a strong, distributed team. When organizing, we wanted to ensure there were no advantages or disadvantages to being in a certain region, that everyone was able to respect each other’s cultures and time zones, and that we were creating an environment where our engineers could be successful.
To make this work, we organized our teams around individual products (for example, Elasticsearch) and their source code, regardless of the geographical location of each team member. This meant we ended up with a mixture of people from all over the world working on a given product, encouraging open and thoughtful communication across time zones.
<h2>Communicating effectively</h2>This presents another challenge: How do we handle communication in teams that don’t share a physical location? With our communications charter we’ve been able to create rules and common practices around using various communication platforms. A big one is that we try to make decisions in GitHub, or an email — rarely in more synchronous channels like Slack. This ensures everyone feels as though they’re a part of the conversation, and they never feel excluded because of offline times and time zones.
<h2>Keeping things Elastic and continuing our growth</h2>Though our teams are bigger, not much has changed since the early days except our capacity to keep maturing. With Elastic, our distributed nature offers so much flexibility and freedom — both in how we develop as a company and how our Elasticians grow as individuals. We're set up in a way that supports parents, families, lifelong students — <a href="https://www.elastic.co/about/our-source-code">as you are</a>. We continue to offer pathways for our engineers to grow their careers, and since everyone is distributed, there’s no advantage going into an office every day over those folks who work from home. We ask people to communicate via GitHub, email, or Slack, even if they sit next to each other. This gives everyone the same information, no matter where they’re located. We offer job tracks for both technical and managerial roles, and those work regardless of where you are&nbsp;and who you are.
Looking towards the future, Elastic has reached a different level of scale, which has pushed me to develop new skills for the scaling of our engineering teams. I’ve managed teams in the past that were a few management layers deep, but they were small enough groups that if I need to effect some change, I could talk with each manager individually. Evolving to larger teams means I’ve had to work to teach other managers how to teach their managers&nbsp;—&nbsp;and to develop my written communication skills to be as clear as possible across cultures and perspectives. I’ve also had to resort to a broadcast communications approach with dozens of engineering managers since I can’t work enough hours for 1:1 communication to be effective, even though I enjoy that style the most.We’ve almost finished a handbook that we will use as an onboarding tool for new managers, which has had the side benefit of pushing those of us that have been around a bit longer to codify all the values we want our new leaders to know, including the empowerment to teach people that join later. And fundamentally, those values that we're&nbsp;handing over to these new managers in our handbook have been here from the beginning, and I love that it’s continuing to be nurtured.
That’s why I stay.
Want to work with the engineering team at Elastic? Check out our <a href="https://www.elastic.co/about/careers/">up-to-date job postings</a>.

kevin_elasticglobalallhands.jpg

Kevin_journey.jpg

Leadership @ Elastic | Kevin Kluge: Distributed for the better

Kibana 5.2.0 has been released.  It includes a heatmap visualization and the return of tribe node querying via Kibana.

<html><head></head><body>Kibana 5.2.0 has just been released and it has several cool features for you to check out. We&apos;ve made good progress on visualizations, included a huge community contribution, and brought back Tribe functionality.&#xA0;<h2>Return of Tribe</h2>We know a lot of our users had used a hack to enable Kibana to query multiple clusters through tribe node. This worked with Kibana 4.x but <a href="https://github.com/elastic/elasticsearch/issues/20544">broke</a> with Kibana 5.0. And we know that caused some pain and blocked upgrades for people. We&apos;ve <a href="https://github.com/elastic/kibana/issues/8495">fixed that</a> with 5.2.0 and gotten to a much better place for the testing and maintenance of tribe node with Kibana.
5.2.0 introduces the separation of a &quot;data&quot; cluster from an &quot;admin&quot; cluster. The data cluster is the one you typically think of when you use Kibana, as it&apos;s the cluster where the data you see is coming from. And that data cluster can be a tribe node. 5.2.0 allows you to configure Kibana to use a different cluster for its administrative functions. These are things like storing your dashboards and visualizations in the .kibana index. This second cluster is called the &quot;admin&quot; cluster, and it cannot refer to a tribe node.
By default, the data cluster and the admin cluster are assumed to be the same cluster, and must not be a tribe node.
There are still a couple of shortcomings that we are investigating: the Console (formerly known as Sense) does not work with separate data and admin clusters. Similarly, we disable the editing of X-Pack users and roles when tribe is configured to avoid any ambiguity about where changes are being made. We don&apos;t yet have an ETA on addressing these things, but they&apos;re on our radar.
<h2>Heatmaps FTW</h2>We have been experimenting with heatmap visualizations for a while, and we now feel confident enough to <a href="https://github.com/elastic/kibana/pull/9403">include them</a> in the 5.2.0 release! Heatmaps are great when you want to pick out an area of high or low volume in time series data. Heatmaps have been cited as <a href="http://queue.acm.org/detail.cfm?id=1809426">helping people</a> see trends that they might otherwise miss.
One common pattern for users will be to create date histograms (with a date histogram aggregation on the X-axis) and then do a terms agg on the y-axis. The buckets in the heatmap are colored based on the metric of your choice. For example, you can choose the count of items in the bucket to influence the color. The heatmap visualization is quite general and you can place any buckets you want on the X and Y axes.
The heatmap feature even allows for custom range settings. For example, you could configure that a given heatmap should show red for places where the count exceeds 95, yellow for 75-94, and so on.
<img src="https://api.contentstack.io/v2/uploads/5894ea2068c81886565dfe38/download?uid=blt9ff483806ac4d098" data-sys-asset-uid="blt9ff483806ac4d098" alt="heatmap_5.2.png"><h2>The start of internationalization</h2>The initial groundwork for internationalization support in Kibana has landed. Thanks to some <a href="https://github.com/elastic/kibana/pull/7545">work from IBM</a> Kibana is able to load JSON translation files from plugins, which will in the future be used to translate any and all text labels in Kibana. While this is only the groundwork for the overall internationalization support in Kibana, it is a huge step forward and work on the next phase of development has already begun. We wanted to give a big thanks to the contributors here as it was a considerable amount of work with multiple iterations over the course of many months to get right, and we&apos;re thrilled with the results so far.
<h2>Tile service enhancements</h2>We introduced the Elastic Tile Service a few months ago. Initially, we offered 10 levels of zoom on the tiles. Now, with 5.2.0, X-Pack Basic users will get up to 12 levels of zoom. Additionally, we are exploring how to increase available tilemap zoom levels to 18 levels of zoom. Starting with the 5.2 release of Kibana, we can make these types of service level adjustments dynamically, without releasing new versions of Kibana, so look for further improvements.
<h2>Monitoring&#xA0;Elasticsearch in containers</h2>Running Elasticsearch in containers? Want to track CPU usage and throttle time? X-Pack monitoring 5.2.0 adds the ability to monitor the container&apos;s utilization metrics that are reported from each of your Elasticsearch instances.
<img src="https://api.contentstack.io/v2/uploads/588f934968f1bbf20be05580/download?uid=bltb10d1dffb6277557" data-sys-asset-uid="bltb10d1dffb6277557" alt="Monitoring Elasticsearch in Containers">
If you are running Elasticsearch in a container and would prefer to see these metrics in place of the overall system&apos;s CPU metrics, then you can use the <a href="https://www.elastic.co/guide/en/x-pack/5.2/monitoring-settings.html#monitoring-ui-settings">container setting</a> to display them in the X-Pack monitoring UI under each Node&apos;s Advanced tab, which was added to display more Elasticsearch metrics in 5.1.1.
<h2>And more</h2>There are even more features in 5.2.0, including a better experience saving dashboards and visualizations as well as the ability to delete Timelion sheets. The <a href="https://www.elastic.co/guide/en/kibana/5.2/release-notes-5.2.0.html">release notes</a> describe all the features plus bug fixes. The downloads are in the <a href="https://www.elastic.co/downloads/kibana">usual place</a>. Please let us know at our <a href="https://github.com/elastic/kibana">repo</a> if you see anything else we need to fix!
</body></html>

heatmap2.png

just-heatmap.png

Kibana 5.2.0 released

We released Kibana 5.1.1 today.  It adds profiler UI, tagcloud visualization, more monitoring charts, and precious performance improvements.

blog-kibana-banner.jpg

blog-kibana-thumb.png

Kibana 5.1.1 released

Today we released Kibana 5.0.2 with a few important fixes.   Users are advised to upgrade.

Kibana 5.0.2 released

<html><head></head><body>Today, we released Elasticsearch 1.4.4 and 1.3.9. This is a small bug fix release, mainly including a packaging fix for our RPM and DEB packages if you are using Lucene expression scripts . You can <a href="/downloads/past-releases/elasticsearch-1-4-4">download Elasticsearch 1.4.4 here</a>.
<h2>fixes</h2>
The 1.4.3 RPM and DEB packages were <a href="https://github.com/elasticsearch/elasticsearch/pull/9696">missing dependencies</a> for Antlr and ASM. These dependencies are required to use Lucene expression scripts with Elasticsearch. Since the <a href="/blog/elasticsearch-1-4-3-and-1-3-8-released/">changes in 1.4.3</a> for Groovy, we expect a lot more people will be using expressions scripts, so we quickly released 1.4.4.
The release also fixes a few bugs around cluster pending tasks. And it fixes a bug where a negative interval on date histograms could cause an <code>OutOfMemoryError</code>.
The <a href="/downloads/past-releases/elasticsearch-1-4-4">1.4.4 release notes</a> as well as <a href="/downloads/past-releases/elasticsearch-1-3-9">1.3.9 release notes</a> the list out all the changes.
<h2>feedback</h2>
As always, we would love to hear from you. Please let us know what you think on Twitter via <a href="https://twitter.com/elasticsearch">@elasticsearch</a> or by filing an <a href="https://github.com/elasticsearch/elasticsearch/issues">issue on GitHub</a>.</body></html>

Elasticsearch 1.4.4 and 1.3.9 Released

<html><head></head><body>We&apos;re happy to announce the release of Shield 1.0.1!This is a bug fix release. We recommend that all users of Shield 1.0.0 upgrade to Shield 1.0.1. This version of Shield is certified to work with Elasticsearch 1.4.2 and 1.4.3.<h3>Fixes</h3>Most importantly, this release fixes an issue where Shield 1.0.0 was incompatible with Elasticsearch 1.4.3. Even if you&apos;re currently using Elasticsearch 1.4.2, we recommend upgrading to Shield 1.0.1. This will protect you from encountering this incompatibility when you upgrade to Elasticsearch 1.4.3 or later in the future.We also fixed a bug with parsing the roles to users mapping. In some cases, Shield would not allow the user to receive privileges from all the roles it was a member of.<h3>Enhancements</h3>Kibana 4 has been evolving, and you may have seen <a href="/blog/kibana-4-rc1-is-now-available">our RC1 release</a> yesterday(!). We have updated the default roles available with Shield to work with new features added to Kibana 4 RC1.We have also added a changelog to the documentation.<h3>Upgrading</h3>To upgrade, just uninstall the current Shield and install Shield 1.0.1. Your configuration will be preserved. You can do this with a rolling upgrade of Elasticsearch.For each node, after you have stopped it, run<pre class="programlisting prettyprint lang-json pln">bin/plugin -r shield
bin/plugin -i elasticsearch/shield/latest</pre>Then restart the node. Larger deployments should follow the steps in the <a href="/guide/en/elasticsearch/reference/current/setup-upgrade.html#_1_0_and_later">rolling upgrade section</a> in order to ensure the recovery is as quick as possible.On upgrade, your current config files will keep their names and content. The config files provided by Shield 1.0.1 will be added with a &#x201C;.new&quot; extension. If you are a Kibana 4 user or might become one, you&apos;ll want to copy the role additions for the Kibana user, found in roles.yml.new, to roles.yml.</body></html>

Shield 1.0.1 Released

<html><head></head><body>We released Logstash 1.4.2 today. This is a bug fix release that includes an important fix for a security vulnerability that was present in previous versions of Logstash. We recommend that users of Logstash&apos;s zabbix or nagios_nsca outputs upgrade immediately. Deployments that do not use the zabbix or the nagios_nsca outputs are not vulnerable and do not need to upgrade for this reason.
<h2>the vulnerability</h2>
The vulnerability impacts deployments that use the either the zabbix or the nagios_nsca outputs. In these cases, an attacker with an ability to send crafted events to any source of data for Logstash could execute operating system commands with the permissions of the Logstash process.
Deployments that do not use the zabbix or the nagios_nsca outputs are not vulnerable and do not need to upgrade for this reason.
We have added this vulnerability to our <a href="http://www.elasticsearch.org/community/security/">CVE page</a> and are working on filling out the CVE.
We would like to thank Jan Karwowski and Danila Borisiuk for reporting the issue and working with us on the resolution.
<h2>remediations</h2>
An upgrade to Logstash 1.4.2 will address the issue. This is our recommended path.
Some deployments may be able to remove the zabbix and nagios_nsca outputs from their configuration. This is a viable option to remediate until an upgrade can be performed.
We have also released
<link>a patch for the 1.3.x series of Logstash releases. This patch can be applied to address the vulnerability. This patch is available as an option to upgrade to 1.4.2. If you apply the patch, you do not need to upgrade to 1.4.2 to fix the vulnerability.
In order to apply the patch for the 1.3.x series, do the following on each Logstash host. &#xA0;This example uses 1.3.3, but you can also use these steps for 1.3.0 - 1.3.2.
<pre class="prettyprint noescap" style="overflow-x: auto !important;"># mkdir -p /tmp/logstash-patch/logstash/outputs

# wget -O /tmp/logstash-patch/logstash/outputs/zabbix.rb https://github.com/elasticsearch/logstash-contrib/raw/v1.4.2/lib/logstash/outputs/zabbix.rb

# wget -O /tmp/logstash-patch/logstash/outputs/nagios_nsca.rb https://github.com/elasticsearch/logstash/raw/v1.4.2/lib/logstash/outputs/nagios_nsca.rb

# jar uf logstash-1.3.3-flatjar.jar -C /tmp/logstash-patch/ logstash/outputs/zabbix.rb -C /tmp/logstash-patch/ logstash/outputs/nagios_nsca.rb</pre>
<h2>other fixes</h2>
The 1.4.2 release includes a number of other fixes. For Logstash core:
<ul>
<li>the nagios_nsca output plugin external command invocation works correctly</li>
<li>Logstash can now be invoked outside of its home directory</li>
<li>the generator input plugin with the stdin option works correctly</li>
<li>the file input plugin now works correctly on Debian 7</li>
</ul>
And for logstash-contrib:
<ul>
<li>the zabbix output external command invocation works correctly</li>
<li>added translate filter dictionary file auto refresh and improved its documentation</li>
</ul>
You can read the full <a href="https://github.com/elasticsearch/logstash/blob/master/CHANGELOG">changelog</a>, or jump right to the <a href="http://www.elasticsearch.org/overview/logstash/download/">Logstash downloads</a> page.
</body></html>

Logstash 1.4.2 released; vulnerability fixed

<html><head></head><body><article>
			<h1>introduction</h1>
In the 1.2.1 release we fixed a routing bug that had been introduced in the 1.2.0 release. This was described in the&#xA0;<a href="http://www.elasticsearch.org/blog/elasticsearch-1-2-1-released/">1.2.1 blog post</a>. The contents of this post apply only to users that ran 1.2.0.
Before we can discuss the tool we developed to help fix damage from the bug, we have to understand the problem the bug created. When Elasticsearch stores a document it has to decide which shard to put the document in. It does this by computing a hash of the document&#x2019;s &#x201C;UID&#x201D;, which is by default the type#id tuple of the document. Elasticsearch then uses that hash modulo the number of shards to pick a shard. We have to make sure that this hash function works the same across different versions of Elasticsearch. If it doesn&#x2019;t, version X could expect to find a given document in shard 1 and version X+1 could expect the same document in shard 2. In this case, version X+1 could not directly get the document if it were indexed while running version X.
The bug in 1.2.0 broke this consistent, cross-version hashing. 1.2.1 restored the same hash operation as used in previous versions (except 1.2.0). Once you&#x2019;ve upgraded to 1.2.1 there are a few potential problems:
<ul>
<li>documents indexed in 1.2.0 are now in the &#x201C;wrong&#x201D; shard. You may not be able to retrieve (get) documents by ID when they were indexed in the wrong shard.</li>
<li>documents that were updated in 1.2.0 may now exist in two places: the correct shard (as in 1.2.1) and the wrong shard (as in 1.2.0).</li>
</ul>
The percentage of documents in the wrong shard varies by the number of shards in the index. With 2 shards, all documents are routed correctly. With 4 shards, 50% of the documents will be routed incorrectly. With 5 shards, 80% of documents will be routed incorrectly. 
Since you have probably continued to update documents, Elasticsearch cannot tell which of two documents with the same UID is the correct (newer) one. To be clear, it cannot tell if the document in the correct shard was put there recently and is the newer version, or if it was put there some time ago and is the older version. We&#x2019;ll discuss this more later in this post.
<h1>how the tool helps</h1>
We&#x2019;ve developed the&#xA0;<a href="http://download.elasticsearch.org/elasticsearch/elasticsearch-fix-routing/elasticsearch-fix-routing-1.0.zip">fix-routing tool</a>&#xA0;to give you as much help as we can in addressing the issue. We&#x2019;ll walk through a few specific use cases to illustrate the use of the tool.
<h2>re-indexing the data may be the simplest solution</h2>
We have provided the tool as a way to help you. After you read the options below you may conclude that re-indexing the data is simpler. That is a valid option, which will work fine. Just be sure that you have upgraded to 1.2.1 before re-indexing.
You will also either want to delete the index you are replacing or index into a new index. If you re-index into the existing index you will not fix the problem; the re-index will not change the location of the documents in the wrong shard.
There is no need to use this tool for any index that you choose to re-index.
<h2>tool prerequisites and preparation</h2>
The tool has a number of requirements to run correctly:
<ul>
<li>JDK 1.7 installed on the host running the tool</li>
<li>_source must have been stored</li>
<li>Your cluster is running version 1.2.1</li>
<li>You are using the default hashing algorithm (most users are)</li>
<li>You are not using the _type in the computation of the routing key (very few users do)</li>
</ul>
If you do not store the _source in Elasticsearch you can still run the count and delete operations described below. But, the tool will not be able to help with any other parts of conflict resolution.
You can run the tool on a running cluster. You do not have to block client access while running it. If you continue to allow updates to the cluster while the tool runs, you will have to repeat the procedure, with the second run of the procedure cleaning up duplicates created while the first instance ran.
Some operations of the tool will delete documents from an index. As a result we recommend taking a backup of each index before running the tool on that index. See&#xA0;<a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html">the upgrade instructions</a>&#xA0;for a backup procedure.
The tool uses a MVEL script as part of its execution. This script, called bad_routes.mvel, is included in the Zip file download. You need to place it in the&#xA0;<code>config/scripts</code>&#xA0;directory of all data nodes. There is no need to restart the nodes after putting this script in the directory. By default, it will take one minute for Elasticsearch to load the script and make it available via the API.
Note that the tool will work even if dynamic scripting is disabled.
<h1>using the tool</h1>
Be sure you have&#xA0;<a href="http://download.elasticsearch.org/elasticsearch/elasticsearch-fix-routing/elasticsearch-fix-routing-1.0.zip">downloaded the tool</a>. After you unzip the download, you will have a jar file and a MVEL script.
<h2>understanding the scope of the problem</h2>
The tool can give you a count on a per-index basis of the number of documents in the wrong shard:
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 count</pre>
You can run this for each index that may have had writes while running 1.2.0 to understand how many documents were impacted.
You can also get an Elasticsearch query that will find the misplaced documents. This can be helpful if you want to write your own conflict resolution program.
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 query</pre>
<h2>if you do not perform updates</h2>
If you do not update existing documents in an index, you can be sure there are no duplicates. Usually, people that are using Elasticsearch for the storage of logs and other machine generated data will want to follow these steps.
In this case the tool can be used to move misplaced documents from the wrong shard to the correct shard. Moving documents with the tool requires first copying them and then deleting them from the original (incorrect) shard.
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 copy_if_missing</pre>
At this point you should be able to get &#x2013;&#xA0;not search for&#xA0;&#x2013; one of the misplaced documents. You should check a few to make sure the copy worked. Then delete the docs in the wrong shard.
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 delete</pre>
Searches may return duplicate documents if you choose to not delete the documents in the wrong shard (as above).
<h2>if you have done updates and use external versioning</h2>
In general, Elasticsearch cannot tell which version of two duplicate documents is the correct one. However, if you use external versioning, it can. The newer document will have the higher external version number. If this is the case for your system, you can run:
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 copy_version</pre>
This command will make sure that the higher version document is in the correct shard. Then, after you check that you now have two copies of the documents that were in the wrong shard, delete the one copy of the document that was in the wrong shard:
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 delete</pre>
<h2>if you have done updates and do not use external versioning</h2>
This is the most difficult case. As mentioned earlier, the fix-routing program cannot tell which copy of the document should be preserved. In this case, you will have to use your domain knowledge of your documents to effect a fix. Start by getting the list of documents that have duplicates and then get the source of the documents that are in the wrong shard. The&#xA0;<code>get</code>&#xA0;command below will produce one JSON file per shard with the contents of misplaced documents for that shard.
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 get_uids &gt; wrong_shard_uids.txt &gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 get</pre>
Then iterate over each document returned and record the UID of those documents for which you want to keep the copy that was in the wrong shard. Call this list wrong_shard_restore. You can delete all the documents in the wrong shard since you have extracted them from the index. To delete all the documents that are in the wrong shard:
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 delete</pre>
Then, for each document in wrong_shard_restore, re-index it using the document in one of the files output by the&#xA0;<code>get</code>&#xA0;command above. This will overwrite the version that was in the correct shard with the version that was in the wrong shard.
The documents that were in the correct shard and that you want to preserve will not be impacted by these operations. Your ability to correctly put only the documents to restore in wrong_shard_restore insures this.
<h1>other comments</h1>
You can run a count operation after you delete documents. This will help you verify that the delete did what was expected. Be careful here, as if you are still adding documents to the index the counts from the beginning of the procedure may be different from those at the end.
We&#x2019;re investigating our testing procedures to understand how this bug escaped our notice. We hope to publish a postmortem with the results of our analysis.
You should think about the backups you have made with 1.2.0 and 1.2.1 prior to performing this procedure. You may want to take a snapshot after performing this procedure, then delete the backups that had documents in the wrong shard.
			<aside id="post_tags">
							</aside>
		</article></body></html>

A tool to help with routing issues from Elasticsearch 1.2.0

<html><head></head><body>
	We released <a href="http://www.elasticsearch.org/downloads/1-2-1/">Elasticsearch 1.2.1</a> today. This is a bug fix release that includes the fix for a severe bug in Elasticsearch 1.2.0. We recommend that anyone using version 1.2.0 upgrade immediately to this 1.2.1 release. We removed the 1.2.0 release from our download site and repositories due to the severity of the bug.
<h3>A Fix for Document Routing and Duplicate Documents</h3>
	There was&#xA0;<a href="https://github.com/elasticsearch/elasticsearch/issues/6393">a routing bug</a>&#xA0;in Elasticsearch 1.2.0 that could have a number of bad side effects on the cluster. Possible side effects include:
<ul>
	<li>documents that were indexed prior to the upgrade to 1.2.0 may not be accessible via get. A search would find these documents, but not a direct get of the document by ID.</li>
	<li>documents that were updated after the upgrade to 1.2.0 may be duplicated, with one copy from pre-1.2.0 and a second copy updated since the upgrade to 1.2.0.</li>
	<li>if a document is duplicated as above, and versioning is in use, the document added after the upgrade to 1.2.0 will have its version reset.</li>
</ul>
	The 1.2.1 release fixes this bug. The bug fix will restore access via get to the documents that were indexed prior to the upgrade to 1.2.0.

	An upgrade to 1.2.1 will not correct duplicate documents. Also, an upgrade to 1.2.1 will break access via get to documents that were indexed since the upgrade to 1.2.0. &#xA0;We are investigating what tools we can create to help with the diagnosis and correction of these problems. We will post an update as soon as possible with this information.
<h3>Other Fixes</h3>
	Elasticsearch 1.2.1 also fixes a&#xA0;<a href="https://github.com/elasticsearch/elasticsearch/issues/6332">bug with the new aggregations circuit breaker</a>. We found that the&#xA0;<a href="https://github.com/elasticsearch/elasticsearch/pull/6050">enhancement to constrain aggregations memory usage</a>&#xA0;was too conservative and could prevent memory requests that should have been allowed. We have effectively disabled this new circuit breaker for 1.2.1. The fielddata circuit breaker continues to work as it has in previous releases.

	There is also a&#xA0;<a href="https://github.com/elasticsearch/elasticsearch/issues/6304">bug fix for mapping parsing</a>. A mapping that contained the parameter <code>include_in_all</code>&#xA0;in the root type would produce a MapperParsingException. For example:
<pre class="prettyprint noescape">PUT test1
{
 &quot;mappings&quot;: {
 &quot;type_name&quot;: {
 &quot;include_in_all&quot;: false,
 &quot;properties&quot;: {}
 }
 }
}
</pre>
	would trigger the bug, effectively removing this feature from 1.2.0. If such a mapping is present when upgrading to 1.2.0, then the mapping for this index can be corrupted once a document with a new field is indexed. 1.2.1 restores the correct behavior of&#xA0;<code>include_in_all</code>.
<h3>Conclusion</h3>
	Users with Elasticsearch 1.2.0 should immediately upgrade to 1.2.1. We will be working to see what tools we can create to help users that have experienced problems from 1.2.0 and we will update on our progress soon. Please&#xA0;<a href="http://www.elasticsearch.org/downloads/1-2-1/">download 1.2.1</a>&#xA0;and let us know what you think. You can report any issues on our&#xA0;<a href="https://github.com/elasticsearch/elasticsearch/issues">GitHub page</a>.
</body></html>

Elasticsearch 1.2.1 Released

<html><head></head><body>Today we are happy to announce the release of&#xA0;Elasticsearch 1.0.1&#xA0;and&#xA0;Elasticsearch 0.90.12.
<h2>elasticsearch 1.0.1</h2>
Thanks to everyone for the positive feedback on 1.0.0, including the bug reports! &#xA0;We have responded to the quick feedback with the 1.0.1 release, which is now the current stable release in the 1.0 series. &#xA0;Details of the changes are available in the&#xA0;<a href="http://www.elasticsearch.org/downloads/1-0-1">release notes</a>. Some of the notable changes are:
<ul>
<li>Memory usage statistics on older, Lucene 3.x-based segments were <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/5201">using excessive CPU</a>, and they have been disabled. &#xA0;This will impact node stats results. &#xA0;We hope to enable them once Lucene 4.7 ships.</li>
<li>Fixed a bug where, in rare circumstances, queries could cause an <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/5043">infinite loop</a>.</li>
<li>Improved <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/5139">processing of large cluster states</a>, as would happen with a large number of mappings, on non-master nodes.</li>
<li>Fixed a bug where the get field mapping API call could<a href="http://github.com/elasticsearch/elasticsearch/issues/issue/5177"> return incorrect results</a>.</li>
<li>Restored snapshots will now correctly <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/5210">restore the mappings</a> and index settings.</li>
<li><a href="http://github.com/elasticsearch/elasticsearch/issues/issue/5057">Time zone parsing</a> in the date_histogram aggregation works correctly.</li>
<li><a href="http://github.com/elasticsearch/elasticsearch/issues/issue/3820">Fixed the file permissions</a> in the Debian package.</li>
</ul>
We recommend that users of Elasticsearch from all versions upgrade to 1.0.1 and not 1.0.0. &#xA0;This path will avoid the issue where reading Lucene memory usage could cause excessive CPU usage.
You can&#xA0;<a href="http://www.elasticsearch.org/downloads/1-0-1">download Elasticsearch 1.0.1 here</a>.
<h2>elasticsearch 0.90.12</h2>
This is the current stable release in the 0.90 series and contains a number of &#xA0;bug fixes and small enhancements that you can read about in the&#xA0;<a href="http://www.elasticsearch.org/downloads/0-90-12">release notes</a>. Please note that we are decreasing the rate of development of the 0.90.x series of releases so that we can focus our attention on 1.0.x and future releases. &#xA0;Some of the notable changes are:
<ul>
<li>Memory usage statistics on older, Lucene 3.x segments were <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/5201">using excessive CPU</a>, and they have been disabled. &#xA0;This will impact node stats results. &#xA0;We hope to enable them once Lucene 4.7 ships.</li>
<li>Fixed a bug where, in rare circumstances, queries could cause an <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/5043">infinite loop</a>.</li>
<li>Improved <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/5139">processing of large cluster states</a>, as would happen with a large number of mappings, on non-master nodes.</li>
<li>Fixed a bug where the get field mapping API call could <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/5177">return incorrect results</a>.</li>
</ul>
We recommend that users of Elasticsearch from the 0.20.x series or earlier upgrade to 0.90.12 and not 0.90.11, or 1.0.1. &#xA0;This path will avoid the issue where getting Lucene memory usage could cause excessive CPU usage.
You can&#xA0;<a href="http://www.elasticsearch.org/downloads/0-90-12">download Elasticsearch 0.90.12 here</a>.
&#xA0;
<h2></h2>
</body></html>

AUTHOR

Articles by Kevin Kluge

Leadership @ Elastic | Kevin Kluge: Distributed for the better

Kibana 5.2.0 released

Kibana 5.1.1 released

Kibana 5.0.2 released

Elasticsearch 1.4.4 and 1.3.9 Released

Shield 1.0.1 Released

Logstash 1.4.2 released; vulnerability fixed

A tool to help with routing issues from Elasticsearch 1.2.0

Elasticsearch 1.2.1 Released

Sign up for Elastic Cloud free trial

Products & Solutions

Company

We're hiring

Resources

Solutions

Elastic (ELK) Stack

New

Featured topics

News

AUTHOR

Articles by Kevin Kluge

Leadership @ Elastic | Kevin Kluge: Distributed for the better

Kibana 5.2.0 released

Kibana 5.1.1 released

Kibana 5.0.2 released

Elasticsearch 1.4.4 and 1.3.9 Released

Shield 1.0.1 Released

Logstash 1.4.2 released; vulnerability fixed

A tool to help with routing issues from Elasticsearch 1.2.0

Elasticsearch 1.2.1 Released

Sign up for Elastic Cloud free trial

Company

We're hiring

Resources