<html><head></head><body><p>The Elasticsearch dev team are pleased to announce the release of E<strong>lasticsearch 0.90.1</strong>, which is based on Lucene 4.3. You can download it&#xA0;<a title="0.90.1" href="http://www.elasticsearch.org/downloads/0-90-1/">here</a>.</p><p>We highly recommend upgrading to 0.90.1 from 0.90.0 as this release includes some key bug fixes, notably a fix to the bool filter which could produce incorrect results (<a href="https://github.com/elasticsearch/elasticsearch/issues/2979">#2979</a>) and a fix to the DFS search types (<a href="http://github.com/elasticsearch/elasticsearch/issues/issue/3012">#3012</a>).</p><p>Besides the bug fixes, 0.90.1 also has some nice new features and enhancements, especially:</p><ul>
<li><span style="line-height: 13px;" rel="line-height: 13px;">New APIs for aliases, including the ability to retrieve specific aliases more efficiently (<a href="http://github.com/elasticsearch/elasticsearch/issues/issue/3075">#3075</a>, <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/3076">#3076</a>, <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/3077">#3077</a> , <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/3100">#3100</a>)&#xA0;</span></li>
<li>The Bulk API now supports update requests (<a href="http://github.com/elasticsearch/elasticsearch/issues/issue/2982">#2982</a>)</li>
<li>Parent-child documents require much less memory than before - only the parent IDs need to be loaded to memory, not the child IDs (<a href="http://github.com/elasticsearch/elasticsearch/issues/issue/3028">#3028</a>)</li>
<li>You can now GET just the _source field (without the _index, _type, _id metadata) using GET /index/type/id/_source (<a href="http://github.com/elasticsearch/elasticsearch/issues/issue/2993">#2993</a>, <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/2995">#2995</a>)</li>
</ul><p>There are many more changes which you can read about on the <a href="https://github.com/elasticsearch/elasticsearch/issues?labels=v0.90.1&amp;sort=created&amp;direction=desc&amp;state=closed&amp;page=1">issues list</a>. Please download and use 0.90.1, and let us know what you think.</p></body></html>

Elasticsearch 0.90.1 Released

<html><head></head><body><p><strong>WARNING: This article contains outdated information. We no longer recommend taking its advice.</strong></p><section id="post_content">
<article>
<p>When building queries in Elasticsearch, you&#x2019;ll often find yourself composing sets of filters. Say you need to filter users that are:
</p>
<ul>
	<li>Gender: Male</li>
	<li>Age: 23-26</li>
	<li>Language: English</li>
</ul>
<p>Generally you should place these filters inside of a <a href="http://www.elasticsearch.org/guide/reference/query-dsl/bool-filter/">Boolean Filter</a>.
</p>
<p>But wait&#x2026;isn&#x2019;t there an <a href="http://www.elasticsearch.org/guide/reference/query-dsl/and-filter/">And Filter</a>? What about the <a href="http://www.elasticsearch.org/guide/reference/query-dsl/or-filter/">Or Filter</a> and <a href="http://www.elasticsearch.org/guide/reference/query-dsl/not-filter/">Not Filters</a>?  Are they simply an alternative syntax to the Bool filter?
</p>
<p>The short answer is: <strong>No, they are very distinct</strong>. More importantly, they can have a big impact on your query performance.
</p>
<h2>BitSets</h2>
<p>The key to understanding the difference will require a detour into BitSets.  Fundamentally, a <a href="http://en.wikipedia.org/wiki/Bit_array">BitSet</a> is just an array of bits that represent state.  A position in the bit array is either a 1 or a 0.
</p>
<p>Filters don&#x2019;t score documents &#x2013; they simply include or exclude.  If a document matches a filter, it is represented with a one in the BitSet; otherwise a zero. This means that Elasticsearch can store an entire segment&#x2019;s filter state (&#x201C;<em>who matches this particular filter?&#x201D;</em>) in a single, compact BitSet.
</p>
<p>The first time Elasticsearch executes a filter, it parses Lucene segment data structures to determine what matches your filter. Instead of throwing away this information, it caches it inside a BitSet.
</p>
<p>The next time the same filter is executed, Elasticsearch can reference the compact BitSet instead of the Lucene segments.  This has huge performance benefits.
</p>
<h2>How do I love thee BitSets? Let me count the ways&#x2026;</h2>
<p>It is hard to overstate how fast bitwise operations are.
</p>
<p>Bitwise operations are such a basic unit of computation that a CPU has dedicated bitwise circuitry. Performing an in-memory bitwise AND is orders of magnitude faster than parsing Lucene data structures and performing the intersection manually.
</p>
<p>BitSets play nicely with other filter BitSets too.  Have three filters?  Elasticsearch will do a bitwise AND on all three bitsets to derive your final matching set of documents.
</p>
<p>Even better, BitSets are cached independent of the query itself. A complex query could have a dozen filters, but each of those filter BitSets are independant and entirely reusable in other contexts. This allows Elasticsearch to be very efficient with filter reuse.
</p>
<p>Furthermore, since BitSets are stored on a per-segment basis, Elasticsearch can do some cool performance tricks.  Lucene segments are immutable &#x2013; once written to disk, they will never change.
</p>
<p>If a particular filter doesn&#x2019;t match any documents in a segment (the BitSet is all zeros), in the future Elasticsearch can ignore that entire BitSet when performing it&#x2019;s filtering operation.
</p>
<p>Similarly, as new segments are added, cached filter BitSets do not need to be invalidated.  If you are indexing new documents into a MySQL table, for example, the B-Tree index is constantly updating due to the nature of B-Trees (they are sorted data structures).
</p>
<p>With Elasticsearch filter caches, only newly created segments need to build filter BitSet; old BitSets can be reused without modification.
</p>
<h2>Boolean vs. And/Or/Not</h2>
<p><em>&#x201C;Ok, we get it. BitSets are cool! Why does this concern me?&#x201D; </em>you may be asking yourself.
</p>
<p>It matters because the Bool filter utilizes BitSets while the And/Or/Not filters do not. If you put a Terms Filter inside of an And&#x2026;no BitSet will be used, even though it exists.
</p>
<p>Why?
</p>
<p>And/Or/Not work on a doc-by-doc basis. They first load whatever data is needed into the Field Data memory pool then iterate over the docs, excluding as they go. BitSets are not used and there is no composability from cached filters.  Elasticsearch is simply scanning down the list of documents and checking each one individually
</p>
<p>If you have multiple Filters, And/Or/Not will &#x201C;short-circuit&#x201D; the operation, only passing the matching docs to the next filter.
</p>
<p>This reduces the amount of work that each subsequent filter needs to perform. For this reason, your &#x201C;heaviest&#x201D; filter should always be placed last &#x2013; typically Geo filters since they can perform some heavy computations to determine distance.
</p>
<h2>When to use And/Or/Not</h2>
<p>It may seem that Bool filters are superior in every regard. Is there a time to use And/Or/Not?
</p>
<p>And/Or/Not are more performant when you are working with filters that <em>do not</em> return a bitset. These are operations that must iterate over every document anyway. For example, a custom script is not BitSet-able because it performs computation on every document.
</p>
<p>In these cases, And/Or/Not is a better choice than the Bool. The list of <strong>Non-Bitset </strong>filters is very small, so it is easy to remember:
</p>
<ul>
	<li>Geo* filters</li>
	<li>Scripts</li>
	<li>Numeric_range</li>
</ul>
<p>That&#x2019;s it! Every other filter should be placed inside of a Bool.
</p>
<h2>Combining Bool with And/Or/Not</h2>
<p>You&#x2019;ll often run into situations where you need several BitSet filters and several Non-BitSet filters. In these situations, combine Bool with And/Or/Not where appropriate. Always wrap the entire composition with an And/Or/Not.  So for example, if you have:
</p>
<ul>
	<li>Gender: Male</li>
	<li>Age: 23-26</li>
	<li>Language: English</li>
	<li>Custom Script</li>
	<li>Geo</li>
</ul>
<p>Your filter list may look something like this:
</p>
<pre class="prettyprint lang-json">{
  &quot;and&quot; : [
    {
      &quot;bool&quot; : {
        &quot;must&quot; : [
          { &quot;term&quot; : {} },
          { &quot;range&quot; : {} },
          { &quot;term&quot; : {} }
        ]
      }
    },
    { &quot;custom_script&quot; : {} },
    { &quot;geo_distance&quot; : {} }
  ]
}
</pre>
<h2>Conclusion</h2>
<p>Filters enable you to find documents that you want, but also boost query performance by offloading simple exclusion tasks to fast BitSet operations. When composing filters, make sure you spend a few extra minutes organizing them to use the appropriate class of aggregating filter:
</p>
<ul>
	<li><strong>Geo, Script</strong> or <strong>Numeric_range</strong> filter: Use And/Or/Not Filters</li>
	<li><strong>Everything else</strong>: Use Bool Filter</li>
</ul>
<h3>Some additional sources:</h3>
<ul>
	<li><a href="https://groups.google.com/d/msg/elasticsearch/2SusmSoShlo/qlOAbWmDRqwJ">Filter execution order</a></li>
	<li><a href="https://groups.google.com/d/msg/elasticsearch/PS12RcyNSWc/I1PX1r0RfFcJ">Bool vs And/Or/Not</a></li>
</ul>
<aside id="post_tags">
</aside>
</article>
</section>
</body></html>

All About Elasticsearch Filter BitSets

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Load More

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Author

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Articles by

Share this story

Share by Email

Read less

Search Integrations

All Solutions

Thank you for registering. We will send you a confirmation email soon.

Thank you for your interest!

Follow us on Youtube

Follow us on Twitter

Follow us on LinkedIn

Follow us on Facebook

Headshot of

Reset all

Filters

Global Virtual Event

View more posts

Print

Continue reading

Share on Facebook

Share on LinkedIn

Share on Twitter

Share

Small image for

Video for

Explore similar demos

Register now

View next

Upcoming webinar

On-demand webinar

Featured webinar

Highlights

See when this webinar starts in my time zone

Related workshops

Hosted by

Agenda

Location

Date

More

View more learning opportunities

Load more press releases

Load more news

What to explore next...

See All Posts

Contact Info

Register to Watch

Sign In to Attend

Register to Attend

Facebook & Elasticsearch: for Your Holiday Viewing Pleasure

<html><head></head><body><p>Hello again friends!</p>
<p><a href="http://logstash.net/docs/1.3.2/">Logstash 1.3.2</a> is now available!</p>
<p>We recently implemented a new feature to help make Logstash and Elasticsearch get along more usefully (the new indexing template, it is awesome!). However, 1.3.0 had a bug that prevented it from actually working, and 1.3.1 had a different bug that caused the data to be stored in a way that made searches work inconsistently with previous releases. The 1.3.2 release has a few bug fixes and a big performance improvement for date processing (json codec, date filter, etc will all benefit here).  You can <a title="view the full changelog" href="https://github.com/logstash/logstash/blob/v1.3.2/CHANGELOG#L1-L21" target="_blank">view the full changelog</a> to see the whole list.  </p>
<p>Some highlights:</p>
<ul>
<li>We fixed the default template.  The easiest way to get the updated template is to set the following in the <code>elasticsearch</code> or <code>elasticsearch_http</code> output plugin section of your logstash configuration:
<pre class="prettyprint lang=js">    template_overwrite =&gt; true
</pre>
<p>If you&apos;re using the <code>elasticsearch</code> output plugin, the included template works with version 0.90.5+. If you&apos;re using an older version it will now indicate this with an error message.  The template will work with older versions of Elasticsearch if you are using the <code>elasticsearch_http</code> output plugin, but we highly recommend updating your Elasticsearch cluster to the current stable release to take advantage of performance enhancements and other fixes.</p></li>
<li>We also added some test Elasticsearch queries to make sure expected behavior continues with future updates.</li>
<li>The embedded Elasticsearch version has been bumped to 0.90.9.</li>
<li>A new <a href="http://logstash.net/docs/1.3.2/outputs/csv" title="csv output plugin documentation" target="_blank">csv output plugin</a> has been added!  This feature will transition into a codec in the future, but you can take advantage of the awesome in this output right now!</li>
</ul>
<p>That&apos;s it for now! Happy Logstashing!</p>
</body></html>

Logstash 1.3.2 Released!

<html><head></head><body><p>Today, just in time for the holidays, we are happy to announce the release of E<strong>lasticsearch 0.90.9</strong>, which is based on Lucene 4.6. This is the current stable release in the 0.90 series. You can download it <a href="http://www.elasticsearch.org/downloads/0-90-9/">here</a>.</p><p>Unfortunately, 0.90.8 had a bug in the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-health.html"><code>cluster-health</code></a> API which might report too few shards if one of your indices is status `red`. While this doesn&apos;t put your data at risk, it might cause sysadmins to suffer a moment of extreme panic.</p><p>While we were about it, we found a bug in the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-terms-filter.html#_terms_lookup_mechanism">terms lookup filter</a> which could, in rare cases, cause search requests to hang. And lastly, a bug was introduced in 0.90.8 which prevented index templates from being loaded when placed under the config directory.</p><p>Apologies for the quick re-release, but we hope it makes your holiday more festive and relaxed. Thanks to those who reported these bugs so promptly. Please <a href="http://www.elasticsearch.org/downloads/0-90-9/">download 0.90.9</a> and let us know what you think.</p></body></html>

Elasticsearch 0.90.9 released

<html><head></head><body><section id="post_content">
		<article>
			<p>We are happy to announce that the frequently requested DEB and RPM repositories are now available.</p>
<p>Due to big differences between major versions, we have decided to setup different repository URLs for each major version.&#xA0;We wanted to make this process simple and make sure you don&#x2019;t accidentally upgrade between major versions and break stuff.&#xA0;But don&#x2019;t forget to check the changelog of new versions before installing them.</p>
<h4>Public key</h4>
<p>Our public key can be found <a href="http://packages.elasticsearch.org/GPG-KEY-elasticsearch">here</a>.</p>
<p>For Debian/Ubuntu systems you can do the following:</p>
<pre class="prettyprint lang=js">wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -</pre>
<p>For CentOS/Red Hat systems:</p>
<pre class="prettyprint lang=js">rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch</pre>
<h4>Repositories</h4>
<p>On APT based systems add the following to your sources.list for using the repositories:</p>
<p>Elasticsearch 0.90.x:</p>
<pre class="prettyprint lang=js">deb http://packages.elasticsearch.org/elasticsearch/0.90/debian stable main</pre>
<p>Elasticsearch 1.0.x:</p>
<pre class="prettyprint lang=js">deb http://packages.elasticsearch.org/elasticsearch/1.0/debian stable main</pre>
<p>Logstash 1.2.x:</p>
<pre class="prettyprint lang=js">deb http://packages.elasticsearch.org/logstash/1.2/debian stable main</pre>
<p>Logstash 1.3.x:</p>
<pre class="prettyprint lang=js">deb http://packages.elasticsearch.org/logstash/1.3/debian stable main</pre>
<p>Run <code>apt-get update</code> and you will be able to use the repositories</p>
<p>Even though the distribution is specified as stable in the <code>sources.list</code>, you should be able to install the debian packages on any distribution.</p>
<p>On YUM based systems add these to your <code>/etc/yum.repos.d/</code> directory:</p>
<p>Elasticsearch 0.90.x:</p>
<pre class="prettyprint lang=js">[elasticsearch-0.90]
name=Elasticsearch repository for 0.90.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/0.90/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1</pre>
<p>Elasticsearch 1.0.x:</p>
<pre class="prettyprint lang=js">[elasticsearch-1.0]
name=Elasticsearch repository for 1.0.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.0/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1</pre>
<p>Logstash 1.2.x:</p>
<pre class="prettyprint lang=js">[logstash-1.2]
name=logstash repository for 1.2.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.2/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1</pre>
<p>Logstash 1.3.x:</p>
<pre class="prettyprint lang=js">[logstash-1.3]
name=logstash repository for 1.3.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.3/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1</pre>
<p>For more information on setting up Elasticsearch, take a look at our <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html">setup documentation</a></p>
<p>Got feedback about our repositories? Let us know via <a href="http://twitter.com/elasticsearch">Twitter</a>, <a href="https://groups.google.com/forum/#!forum/elasticsearch">Google Groups</a> or <a href="https://github.com/elasticsearch/elasticsearch/issues">Github</a></p>
			<aside id="post_tags">
							</aside>
		</article>
			</section></body></html>

Announcing DEB and RPM Repositories

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p><p>Elasticsearch&apos;s faceting feature has made it extremely popular not just for realtime search, but also for analytics. With its new aggregations framework, it&apos;ll take you even further.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>Facets come in many forms and can be processed and visualized in many ways. Superfast faceting and filtering underpins usage of search engines for analytical purposes. We are not necessarily interested in finding the top <em>n</em> documents, but instead in what we can learn from filtering sets of results and seeing how e.g. histograms and charts change. The massively popular combination of Elasticsearch and Kibana for data crunching and visualization is an example of this kind of usage.
</p>
<p>While facets are great and people are finding all sorts of creative ways of using them, they have some limitations. To deal with these limitations, Elasticsearch has engineered the new aggregations framework, which brings a whole new level of awesome.
</p>
</section><section id="what-facets-cannot-do" class="level2">
<h2><a href="#what-facets-cannot-do">What Facets Cannot Do</a></h2>
<p>The illustrations and examples that follow are simplified. They are intended to explain how faceting and aggregation can be used, and not necessarily how they are implemented under the covers. A facet accepts a set of documents (as determined by the query) and produces some sort of summary, such as the how many documents have a certain tag, or how many documents fall in a certain &#x201C;bucket&#x201D; in a date histogram. While crunching these numbers, it does <em>not</em> retain information about <em>which</em> documents fall into which buckets: just the tallies. So it isn&#x2019;t possible to easily say for example &#x201C;for every store, show me a histogram of sales performance over time&#x201D; with this model, just &#x201C;show me sales performance per store&#x201D; or &#x201C;show me sales performance over time&#x201D;.
</p>
</section><section id="aggregations" class="level2">
<h2><a href="#aggregations">Aggregations</a></h2>
<p>Aggregations can retain information about which documents go into which bucket. A &#x201C;bucketing&#x201D; aggregator takes a set of documents and outputs one or more buckets. These buckets also have their own sets of documents. Therefore, one aggregations&#x2019; <em>output</em> can be used as another one&#x2019;s <em>input</em>! The resulting possibilities are vast. You can now compose aggregations, creating trees of aggregators. Before we jump into building vast forrests of aggregation trees we&#x2019;ll start with something simple.
</p>
<p>Let&#x2019;s start by combining aggregators to create something like the existing <code>terms_stats</code> facet. The terms stats <em>facet</em> is equivalent to a combination of <code>terms</code> and <code>stats</code> aggregators. It makes it possible to get statistics about one field per value of another, e.g. price statistics per tag. There is no corresponding <code>terms_stats</code>-aggregation. It&#x2019;s not necessary, as we can now achieve the same goal by combining <code>terms</code> and <code>stats</code> aggregations. Our query will first perform a <code>terms</code>-aggregation and <em>then</em> perform a <code>stats</code>-aggregation. For every bucket the <code>terms</code>-aggregation produces, a <code>stats</code>-aggregation is run on the documents in that bucket.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c980342dfd7382632ed/download?uid=blt8bdc17e1c3b9730a" data-sys-asset-uid="blt8bdc17e1c3b9730a" alt="Facets vs. aggregations" style="max-width:100%;">
</p><figcaption>Facets vs. aggregations</figcaption>
</section><section id="buckets-vs.metrics" class="level2">
<h2><a href="#buckets-vs.metrics">Buckets vs. Metrics</a></h2>
<p>The <code>stats</code>-aggregation is an example of what&#x2019;s called a <em>metric</em> aggregator, and sometimes also a &#x201C;calculator&#x201D;. Example metric aggregators are <code>min</code>, <code>max</code>, <code>sum</code>, <code>avg</code>, <code>value_count</code>, <code>stats</code>, and <code>extended_stats</code>. These take a set of documents as input and crunch some numbers for the provided field. The output of metric aggregators does not include any information linked to individual documents, just the statistical data. i.e. metric aggregators can calculate that the minimum <code>price</code> is <code>10</code>, but will not tell you which documents have that value.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c999e7a83165490c344/download?uid=blt70c03bd113fca098?uid=blt70c03bd113fca098" data-sys-asset-uid="bltc62de4db3e2cd65b" alt="stats is a metric aggregator" style="max-width:100%;">
</p><figcaption><code>stats</code> is a metric aggregator</figcaption>
<p>The other kind of aggregator is a <em>bucketing</em> aggregator. These aggregators produce buckets, where a bucket has an associated value and a set of documents. For example, the <code>terms</code>-aggregator produces a bucket per term for the field it&#x2019;s aggregated on. A document can end up in multiple buckets if the document has multiple values for the field being aggregated on. If a bucket aggregator has one or more &#x201C;downstream&#x201D; (i.e. child) aggregators, these are run on every bucket produced.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c993dc542cb38c08486/download?uid=blt562ba996699531eb" data-sys-asset-uid="blt562ba996699531eb" alt="terms is an aggregator that produces buckets" style="max-width:100%;">
</p><figcaption><code>terms</code> is an aggregator that produces <em>buckets</em></figcaption>
</section><section id="bucket-ordering" class="level2">
<h2><a href="#bucket-ordering">Bucket Ordering</a></h2>
<p>By default, the buckets for a <code>terms</code> aggregator are ordered by the number of documents within them, high to low. You can make the order depend on the value of a metrics sub-aggregator as well. For example, you may want to produce a list of products and order them by the revenue generated, and not just the sales count. Similarily, the <code>histogram</code>-aggregation orders buckets by the implied <em>key</em> by default, but this can be changed as well.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c99d8edd48f76935f9f/download?uid=blt6403fdaa3250d609" data-sys-asset-uid="blt6403fdaa3250d609" alt="Ordering the buckets by computed metrics" style="max-width:100%;">
</p><figcaption>Ordering the buckets by computed metrics</figcaption>
</section><section id="real-world-example" class="level2">
<h2><a href="#real-world-example">Real World Example</a></h2>
<p>Honza Kr&#xE1;l presents an excellent example of a more involved aggregation in his <a href="https://www.youtube.com/watch?v=yZu4jQtBUPg">PyCon PL 2013-presentation &#x201C;Explore Your Data&#x201D;</a>. Using <a href="http://stackoverflow.com/">Stackoverflow</a> data as an example, he shows how to structure a query that can answer the question: <em>&#x201C;What&#x2019;s the number of questions per tag? And per tag, what&#x2019;s the average number of comments? Who&#x2019;s commenting the most, and what&#x2019;s the highest score of a comment per author?&#x201D;</em>
</p>

<section id="summary" class="level2">
<h2><a href="#summary">Summary</a></h2>
<p>Aggregations bring a lot of new possibilities to the table. They will undoubtedly further strengthen Elasticsearch&#x2019;s position as an analytics engine. In a future post we will look more into how they are implemented under the covers, in order to get a better understanding of the performance and memory complexity involved.
</p>
<p>While the facet framework will continue to be supported for a while, it will be deprecated as of 1.0. Thus, you should plan on moving to the aggregations framework in the not too distant future.
</p>
</section></section></body></html>

Elasticsearch's New Aggregations

<html><head></head><body><section id="post_content">
		<article>
			<p>Today we are happy to announce the release of <strong>Elasticsearch 0.90.8</strong>, which is based on Lucene 4.6. This is the current stable release in the 0.90 series. You can download it <a href="http://www.elasticsearch.org/downloads/0-90-8/" target="_top">here</a>.</p>
<p>There are not many big new features in this release, but it contains a number of important bug fixes and stability improvements. We highly recommend upgrading, especially if you are currently running 0.90.6 or 0.90.7, or if you are using parent-child relationships.</p>
<h2><a id="_new_features" href="#_new_features"></a>New Features</h2>
<h3><a id="_cluster_stats_api" href="#_cluster_stats_api"></a>Cluster Stats API</h3>
<p>In addition to the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html" target="_top"><code>nodes-stats</code></a>, <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-info.html" target="_top"><code>nodes-info</code></a> and <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-stats.html" target="_top"><code>indices-stats</code></a> APIs, we add the new <code>cluster-stats</code> API, which returns useful summary information from a cluster-wide perspective. It includes basic index metrics and important information about the nodes in the cluster.</p>
<p><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-stats.html" target="_top">Read more about the <code>cluster-stats</code> API.</a></p>
<h3><a id="_simple_query_string_query" href="#_simple_query_string_query"></a>Simple Query String Query</h3>
<p>The <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html" target="_top"><code>query_string</code> query</a> is very powerful but problematic. It supports a complex dense mini-language for expressing queries, but any syntax error will result in an error message instead of results. It allows users to query any field in your index and potentially run very heavy queries. It is not a suitable query to expose directly to your users.</p>
<p>Enter the new <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html" target="_top"><code>simple_query_string</code> query</a>. It supports a much simpler syntax:</p>
<ul>
<li><code>&amp;</code> : and</li>
<li><code>|</code> : or</li>
<li><code>-</code> : not</li>
<li><code>(...)</code> : grouping or precedence</li>
<li><code>&quot;quick brown fox&quot;</code> : phrase query</li>
<li><code>foo*</code> : prefix query</li>
</ul>
<p>But the best part about it is that it is immune to syntax errors. If the syntax is not quite right, it will try to do the right thing anyway!</p>
<p><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html" target="_top">Read more about the <code>simple_query_string</code> query here</a>.</p>
<h3><a id="_disable_fielddata_loading" href="#_disable_fielddata_loading"></a>Disable Fielddata Loading</h3>
<p>When faceting or sorting on field values, Elasticsearch needs instant access to the value for each document in order to perform well. Fielddata is the magic potion that makes these functions blazing fast, by loading field values into memory. However, load the wrong field into memory and you could run out of memory and bring your cluster down.</p>
<p>We are working on circuit-breakers which will prevent you from damaging your cluster, but in the meantime we allow you to disable fielddata loading for specific fields, such as the body of an email:</p>
<pre class="prettyprint lang-js"> {
    &quot;body&quot;: {
        &quot;type&quot;:       &quot;string&quot;,
        &quot;fielddata&quot;: {
            &quot;format&quot;: &quot;disabled&quot;
        }
    }
}</pre>
<p><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/fielddata-formats.html#_fielddata_loading">Read more about disabling fielddata here</a>.</p>
<h3><a id="_geo_point_compression" href="#_geo_point_compression"></a>Geo-Point Compression</h3>
<p>A geo-point consists of a latitude and a longitude, and these values need to be loaded into fielddata memory to perform filtering by geo-location or geo-distance. By default, a geo-point takes up 16 bytes of memory and is extremely precise. We can easily sacrifice a little precision for big memory savings:</p>
<table>
<thead>
<tr>
<th>Precision</th>
<th>Bytes per point</th>
<th>Size reduction</th>
</tr>
</thead>
<tbody>
<tr>
<td>1km</td>
<td>4</td>
<td>75%</td>
</tr>
<tr>
<td>3m</td>
<td>6</td>
<td>62.5%</td>
</tr>
<tr>
<td>1cm</td>
<td>8</td>
<td>50%</td>
</tr>
<tr>
<td>1mm</td>
<td>10</td>
<td>37.5%</td>
</tr>
</tbody>
</table>
<p><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html#_field_data">Read more about setting geo-point <code>precision</code> here</a>.</p>
<h3><a id="_token_count" href="#_token_count"></a>Token Count</h3>
<p>Quite often we not only want to make a field searchable, we also want to know how many words or tokens that the field contains. We have added a new field type called <code>token_count</code> which will index the number of tokens in the field automatically:</p>
<pre class="prettyprint lang-js">{
    &quot;message&quot;: {
        &quot;type&quot;: &quot;multi_field&quot;,
        &quot;fields&quot;: {
            &quot;message&quot;:    { &quot;type&quot;: &quot;string&quot;      },
            &quot;word_count&quot;: { &quot;type&quot;: &quot;token_count&quot; }
        }
    }
}</pre>
<p>A filter on <code>message.word_count</code> would allow you to find documents that contain the tokens <code>foo</code>, <code>bar</code> and <code>baz</code>, but no other tokens.</p>
<p><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#token_count" target="_top">Read more about <code>token_count</code> here</a></p>
<h2><a id="_bug_fixes_and_enhancements" href="#_bug_fixes_and_enhancements"></a>Bug Fixes and Enhancements</h2>
<p>This release contains a number of important optimizations and bug fixes which will improve the stability of your cluster, especially if you have a very large cluster.</p>
<ul type="disc">
<li>In rare cases, it is possible that shards could be deleted incorrectly and that dead nodes continue to show up as members of the cluster. This fix alone is sufficient reason to upgrade. See <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/4503" target="_top">#4503</a>.</li>
<li>The logic in the shard allocation deciders has been greatly improved&#x2009;&#x2014;&#x2009;deciding where to allocate thousands of shards can now be completed in seconds instead of minutes. See <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/4459" target="_top">#4459</a>, <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/4458" target="_top">#4458</a> and <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/4454" target="_top">#4454</a>.</li>
<li>Recovery of local primary shards (fast) is now done before relocating primary shards from one node to another (slow). <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/4237" target="_top">#4237</a>.</li>
<li>Frequent mapping updates on clusters with very large mappings will now complete much more quickly&#x2009;&#x2014;&#x2009;only the latest mapping is processed instead of each mapping change. See <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/4373" target="_top">#4373</a>.</li>
<li>Cluster state changes now wait for an <code>ack</code> response from the nodes in the cluster. Usually these changes complete quickly, but on very large clusters they can take more time. The <code>ack</code> mechanism ensures that changes are in place before returning success to the client. See <a href="https://github.com/elasticsearch/elasticsearch/search?q=ack&amp;ref=cmdform&amp;state=closed&amp;type=Issues" target="_top"><code>ack</code> related issues</a>.</li>
<li>Various bugs were fixed in the <code>has_child</code> and <code>has_parent</code> queries, which could occasionally return incorrect results. See <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/4341" target="_top">#4341</a>, <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/4313" target="_top">#4313</a>, <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/4306" target="_top">#4306</a> and <a href="http://github.com/elasticsearch/elasticsearch/issues/issue/4291" target="_top">#4291</a>.</li>
<li>Ensuring that the <code>bootstrap.mlockall</code> setting has been applied correctly is both very important and difficult to do. Now you can use the <code>nodes-info</code> API to verify, with:
<pre class="prettyprint lang-js">curl localhost:9200/_nodes/process?pretty</pre>
</li>
</ul>
<p>We hope you enjoy this new release. Please <a href="http://www.elasticsearch.org/downloads/0-90-8/" target="_top">download 0.90.8</a>, and let us know what you think.</p>
			<aside id="post_tags">
							</aside>
		</article>
			</section></body></html>

Elasticsearch 0.90.8 Released

<html><head></head><body><p>
	Welcome to&#xA0;<i>This Week in Elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
</p>
<h2>Elasticsearch core</h2>
<ul>
	<li>Batch processing mapping updates can cause missed merged mappings when batching multiple types (<a href="https://github.com/elasticsearch/elasticsearch/commit/10cdb0ae221a672bbc348140a30454d5aab29554">commit</a>, 0.90 and master)</li>
	<li>Bulk response has errors indication + status per item (<a href="https://github.com/elasticsearch/elasticsearch/issues/4002">#4002</a>, master)</li>
	<li>Unify default ack timeout to 30 seconds (used to acknowledge operations) (<a href="https://github.com/elasticsearch/elasticsearch/issues/4395">#4395</a>, 0.90 and master)</li>
	<li>Never throw IndexMissingException in indices query and filter (<a href="https://github.com/elasticsearch/elasticsearch/issues/3428">#3428</a>, 0.90 and master)</li>
	<li><code> MapperBuilders</code> did not contain builders for all mappers (<a href="https://github.com/elasticsearch/elasticsearch/issues/4423">#4423</a>, 0.90 and master)</li>
	<li>Remove <code>IndexFieldData.getHighestNumberOfSeenUniqueValues()</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/4426">#4426</a>, 0.90 and master)</li>
	<li>Added new <code>simple_query_string</code> query type (<a href="https://github.com/elasticsearch/elasticsearch/issues/4159">#4159</a>, 0.90 and master)</li>
	<li>Refresh the mapping source directly into a compressed buffer (<a href="https://github.com/elasticsearch/elasticsearch/commit/0f4d81fcd43eae034655f63428cc3a1f475da170">commit</a>, 0.90 and master)</li>
	<li>Print nice error in <code>bin/elasticsearch</code> if user needs to run maven first<br>
 (
	<a href="https://github.com/elasticsearch/elasticsearch/issues/2954">#2954</a>, 0.90 and master)</li>
	<li>Expose number of queries in percolator stats (<a href="https://github.com/elasticsearch/elasticsearch/issues/4440">#4440</a>, master)</li>
	<li>Improving performance of allocation deciders by stopping if one part returns <code>NO</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/4454">#4454</a>, 0.90 and master)</li>
	<li>Added the <code>force_source</code> option to highlighting that enforces to use of <code>_source</code> even if there are stored fields (<a href="https://github.com/elasticsearch/elasticsearch/issues/4348">#4348</a>, master)</li>
	<li>Aggregations: Ordinals-based string bucketing support (<a href="https://github.com/elasticsearch/elasticsearch/issues/4350">#4350</a>, master)</li>
	<li>Library upgrades: Jackson 2.3.0, jsr166 (commits <a href="https://github.com/elasticsearch/elasticsearch/commit/59d85bc22ae597709ad28d8b2a487b518d111ff0">1</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/80ab75e54ef2116d20abded2c1464eb312a7f4a8">2</a>, 0.90 and master)</li>
	<li>Resolve potential deadlock state during <code>EsThreadPoolExecutor</code> shutdown (<a href="https://github.com/elasticsearch/elasticsearch/issues/4334">#4334</a>, 0.90 and master)</li>
	<li>CAT Api: Aliases endpoint (<a href="https://github.com/elasticsearch/elasticsearch/issues/4414">#4414</a>, master)</li>
	<li>Improve <code>RoutingNodes</code> API, made it more immutable and dont expose state (<a href="https://github.com/elasticsearch/elasticsearch/issues/4458">#4458</a>, 0.90 and master)</li>
	<li>A cancelled shard allocation fails to reset the state of the source shard when moving (<a href="https://github.com/elasticsearch/elasticsearch/issues/4457">#4457</a>, 0.90 and master)</li>
	<li><code>IndicesLifecycle.Listener</code> to support listening for any index shard state change (<a href="https://github.com/elasticsearch/elasticsearch/issues/4413">#4413</a>, 0.90 and master)</li>
	<li>Term suggest: Throw <code>IllegalArgumentException</code> if a field has different analysis chains across indices (<a href="https://github.com/elasticsearch/elasticsearch/issues/3196">#3196</a>, 0.90 and master)</li>
	<li>Removed <code>text</code> and <code>field</code> queries (<a href="https://github.com/elasticsearch/elasticsearch/issues/4033">#4033</a>, master)</li>
	<li>Renamed top level <code>filter</code> to <code>post_filter</code>. (<a href="https://github.com/elasticsearch/elasticsearch/issues/4119">#4119</a>, 0.90 and master)</li>
	<li>Take field data changes immediately into account and add the ability to disallow field data loading (<a href="https://github.com/elasticsearch/elasticsearch/issues/4430">#4430</a>, <a href="https://github.com/elasticsearch/elasticsearch/issues/4431">#4431</a>, 0.90 and master)</li>
	<li>Elasticsearch is now running in the foreground by default and supports a parameter to daemonize, so change your custom init script for 1.0 (<a href="https://github.com/elasticsearch/elasticsearch/issues/4392">#4392</a>, master)</li>
	<li>Elasticsearch properties can now be set using getopt long style parameter (<a href="https://github.com/elasticsearch/elasticsearch/issues/4393">#4393</a>, master)</li>
	<li>Geo point field data can now be compressed (<a href="https://github.com/elasticsearch/elasticsearch/issues/4386">#4386</a>, master)</li>
	<li>A new <code>/_cluster/stats</code> API has been added (<a href="https://github.com/elasticsearch/elasticsearch/issues/4460">#4460</a>, 0.90 and master)</li>
	<li>Fix parsing of file based template loading (<a href="https://github.com/elasticsearch/elasticsearch/issues/4411">#4411</a>, 0.90 and master)</li>
	<li>The rest specification test suite used by all official clients is now also executed in the core (<a href="https://github.com/elasticsearch/elasticsearch/issues/4469">#4469</a>, 0.90 and master)</li>
	<li>Don&apos;t delete local shard data when its allocated on a node that doesn&apos;t exist (<a href="https://github.com/elasticsearch/elasticsearch/issues/4502">#4502</a>, 0.90 and master)</li>
	<li>Improve shard allocation performance by about five times (<a href="https://github.com/elasticsearch/elasticsearch/commit/314499cee078e42353b3373901368152d7c4fa4e">commit</a>, 0.90 and master)</li>
	<li>Added cluster health status to the Cluster Stats API (relates to <a href="https://github.com/elasticsearch/elasticsearch/issues/4460">#4460</a>, 0.90 and master)</li>
	<li>On node join, evict existing node(s) with the same transport address (<a href="https://github.com/elasticsearch/elasticsearch/issues/4503">#4503</a>, 0.90 and master)</li>
</ul>
<h2>Elasticsearch Ecosystem</h2>
<p>
	Here&apos;s some more information about what is happening in the ecosystem we are maintaining around Elasticsearch, including plugin and driver releases, as well as news about Logstash and Kibana.
</p>
<ul>
	<li>Elasticsearch Javascript client: A brand new <a href="http://www.elasticsearch.org/blog/client-for-node-js-and-the-browser/">official Javascript client has been released</a> yesterday, mainly driven by <a href="https://twitter.com/salger">Spencer Alger</a>, who is eager to get some feedback </li>
	<li><a href="https://twitter.com/theuntergeek">Aaron Mildenstein</a> created a <a href="http://www.elasticsearch.org/blog/logstash-collectd-input-plugin/">logstash collectd plugin</a> </li>
	<li>Check out this 17 minute interview with <a href="http://blog.qbox.io/rackspace-startup-evangelist-robert-scoble-gets-search-and-analytics-demonstration-from-qbox-ceo">Robert Scoble from Rackspace and Mark Brandon, CEO from qbox.io</a></li>
	<li><a href="http://blog.clojurewerkz.org/blog/2013/12/12/elastisch-1-dot-3-0-is-released/">Elastisch 1.3.0</a>, a Clojure client for Elasticsearch, has been released </li>
	<li>The makers of the <a href="https://twitter.com/exoplatform">eXo platform</a> wrote about <a href="http://blog.exoplatform.com/en/2013/12/12/use-elasticsearch-search-use-external-sources-like-wikipedia-inside-exo-platform">using Elasticsearch in their eXo platform</a> </li>
	<li><a href="https://groups.google.com/forum/#!topic/kundera-discuss/2DtT2d9Oqt4">Kundera 2.9</a>, a NoSQL mapping library, has been released </li>
	<li>The European Environment Agency released a <a href="https://github.com/eea/eea.elasticsearch.river.rdf">RDF river</a> </li>
	<li>A blog post about <a href="http://www.found.no/foundation/fuzzy-search/">how to use fuzzy searching with elasticsearch</a> </li>
	<li>The new talend platform also leverages <a href="http://www.kai-waehner.de/blog/2013/12/17/realtime-event-logging-complex-event-processing-cep-and-monitoring-with-talends-unified-platform-5-4-di-esb-dq-mdm-bpm-using-elasticsearch-logstash-and-kibana/">Elasticsearch, logstash and Kibana</a> </li>
	<li><a href="https://twitter.com/simplechris">Chris Simpson</a> wrote a nice blog post giving an <a href="http://chrissimpson.co.uk/elasticsearch-aggregations-overview.html">aggregations overview</a></li>
	<li>French content: <a href="https://twitter.com/Elrykk">Antoine Rey</a> writes about his experiences when <a href="http://www.technologies-ebusiness.com/solutions/les-problematiques-elastisearch">getting started with Elasticsearch</a> </li>
	<li>French content: <a href="https://twitter.com/vspiewak">Vincent Spiewak</a> write a <a href="http://blog.xebia.fr/2013/12/12/logstash-elasticsearch-kibana-s01e02-analyse-orientee-business-de-vos-logs-applicatifs/">second blog post about Elasticsearch, Logstash and Kibana</a></li>
</ul>
<h2>Slides</h2>
<ul>
	<li>French content: <a href="https://twitter.com/vspiewak">Vincent Spiewak</a> also presented at the Paris Elasticsearch meetup group about <a href="https://github.com/vspiewak/ElasticsearchFR-Meetup-4/raw/master/slides/Xebia%20-%20VSPIEWAK%20-%20ES%20Meetup%20-%20Logstash%20%2B%20ES%20%2B%20Kibana.pdf">Using Elasticsearch, Kibana and Logstash</a> </li>
</ul>
<h2>Where to find Us</h2>
<h4>Israel</h4>
<p>
	The first ever <a href="http://www.meetup.com/Elasticsearch-Tel-Aviv-Meetup/events/152066232/">Elasticsearch Tel Aviv meetup</a> will take place on December 30th, beginning at 7 PM. Join Elasticsearch core developer <a href="http://twitter.com/uboness">Uri Boness</a>, who will present on what&apos;s new in Elasticsearch 1.0, and Itamar Syn-Hershko, who will discuss Monitoring online conversation at scale using Elasticsearch.
</p>
<h4>France</h4>
<p>
	<a href="http://twitter.com/dadoonet">David Pilato</a> will talk about <a href="http://humantalks.com/talks/318-make-sense-of-your-big-data">Making sense of your big data</a> in Angers at the 14th of January.
</p>
<h4>Netherlands</h4>
<p>
	<a href="http://twitter.com/bleskes">Boaz Leskes</a> will talk about Elasticsearch at <a href="http://www.meetup.com/010DEV/events/155663882/">010dev</a> in Rotterdam at the 9th of January.
</p>
<h4>United Kingdom</h4>
<p>
	Mark Harwood will talk about Elasticsearch 1.0 at <a href="https://secure.trifork.com/london-2014/freeevent/index.jsp?eventOID=5882">QCon night London</a> in London at the 15th of January.
</p>
<h2>Where to Find You</h2>
<p>
	Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, <a href="https://docs.google.com/forms/d/1qBZ6Fwq4JWw7FH3s6C9Q3KeGCvUOmWYOvNMsxNqkCNM/viewform">take a moment to let her know</a>.
</p>
<p>
	Oh yeah, we&apos;re also&#xA0;<a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you&apos;d like us to find you for employment purposes, just&#xA0;<a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &#xA0;We care more about your skill set and passion for Elasticsearch, Kibana and Logstash than where you rest your head.
</p>
<h2>Training</h2>
<p>
	If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
</p>
<ul>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/2014-01-13">Amsterdam</a>&#xA0;- January 13, 2014</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/seattle/2014-01-16">Seattle</a>&#xA0;- January 16, 2014</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/hamburg/2014-01-22">Hamburg</a>&#xA0;- January 22, 2014</li>
	<li><a href="http://purchases.elastic.co" target="_blank">San Jose</a>&#xA0;- January 23, 2014</li>
</ul></body></html>

This Week in Elasticsearch - December 18, 2013

<p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>Elasticsearch's Fuzzy query is a powerful tool for a multitude of situations. Username searches, misspellings, and other funky problems can oftentimes be solved with this unconventional query. In this article we clarify the sometimes confusing options for fuzzy searches, as well as dive into the internals of Lucene's FuzzyQuery.</p><section id="introduction" class="level2"><h2><a href="#introduction" target="_self">Introduction</a></h2><p>Searching natural language is inherently imprecise. Since computers can't comprehend natural language, there are a plethora of approaches to search, each with its own advantages and drawbacks. Lucene, the technology underlying Elasticsearch, is a swiss-army knife composed of many text processing tools. Each tool within it is a heuristic, an algorithmic shortcut in lieu of true linguistic comprehension. Some of these tools, like the Snowball stemmer and the Metaphone phonetic analyzer, are quite sophisticated. These tools mimic grammatical and phonetic aspects of language comprehension respectively. Other tools are very basic, like the prefix query type, which simply matches the beginning letters of words. Fuzzy queries sit somewhere in the middle of this toolchest in terms of sophistication; they find words that need at most a certain number of character modifications, known as 'edits', to match the query. For instance, a fuzzy search for 'ax' would match the word 'axe', since only a single deletion, removing the 'e', is required to match the two words.</p></section><section id="a-simple-fuzzy-match" class="level2"><h2><a href="#a-simple-fuzzy-match" target="_self">A Simple Fuzzy Match</a></h2><p>Fuzzy queries can most easily be performed through additional arguments to the match query type, as seen in the example below this paragraph. In the example, the final request, a search for 'vaacuum', which has an extra A, should still turn up the 'Vacuum' product. The fuzziness argument specifies that the results match with a maximum edit distance of 2. It should be noted that fuzziness should only be used with values of 1 and 2, meaning a maximum of 2 edits between the query and a term in a document is allowed. Larger differences are far more expensive to compute efficiently and are not processed by Lucene. The official documentation still refers to setting fuzziness to float values, like 0.5, but these values are in-fact deprecated, and are harder to reason about.</p><pre class="prettyprint"># Create the index<br/>PUT /fuzzy_products<br/># Create the product mapping<br/>PUT /fuzzy_products/product/_mapping<br/>{<br/>  "product": {<br/>    "properties": {<br/>      "name": {<br/>        "type": "string",<br/>        "analyzer": "simple"<br/>      }<br/>    }<br/>  }<br/>}<br/># Upload some documents<br/>PUT /fuzzy_products/product/1<br/>{"name": "Vacuum Cleaner"}<br/>PUT /fuzzy_products/product/2<br/>{"name": "Turkey Baster"}<br/># Perform a fuzzy search!<br/>POST /fuzzy_products/product/_search<br/>{<br/>  "query": {<br/>    "match": {<br/>      "name": {<br/>        "query": "Vacuummm",<br/>        "fuzziness": 2,<br/>        "prefix_length": 1<br/>      }<br/>    }<br/>  }<br/>}<br/></pre></section><section id="determining-edit-distance" class="level2"><h2><a href="#determining-edit-distance" target="_self">Determining Edit Distance</a></h2><p>The metric used by fuzzy queries to determine a match is the Damerau-Levenshtein distance formula. Simply put, the Damerau-Levenshtein distance between two pieces of text is the number of insertions, deletions, substitutions, and transpositions needed to make one string match the other. As an example, the Levenshtein distance between the words “ax” and “axe” is 1 due to the single deletion required.</p><p>The Damerau-Levenshtein distance formula is a modification of the classic Levenshtein distance formula, altering it by adding transposition as a valid operation. Both formulas are supported, with Damerau-Levenshtein being the default, and classic Levenshtein being selectable by setting transpositions to false in the query. The utility of transpositions can be seen in the case of comparing the strings 'aex' and 'axe'. When using the classic Levenshtein distance formula, 'aex' is not one, but two edits away; the 'e' must be deleted, after which a new 'e' is inserted in the proper place, while in Damerau-Levenshtein, a single operation, swapping the 'e' and the 'x', suffices. Extrapolating out from this, using classic Levenshtein would mean that 'aex' is as far away from 'axe' as 'faxes' is; an example showing why Damerau-Levenshtein makes greater intuitive sense in most cases.</p><p>When dealing with fuzzy searches in particular, it is vital to understand that in elasticsearch text is first run through an analyzer before being made available for search. When data is indexed it is processed into what are known as 'terms', the actual searchable units in the database. It is the analyzed terms (what terms are is covered in <a href="/blog/found-elasticsearch-from-the-bottom-up" target="_self">Elasticsearch from the Bottom Up</a>), not the actual stored documents that are searched. That means that when performing fuzzy queries, the query text may be compared to an unanticipated term value as a result of analysis, leading to sometimes confusing results. This also means that if <a href="/guide/en/elasticsearch/reference/current/analysis-synonym-tokenfilter.html" target="_self">synonyms</a> are enabled on a field the synonyms may be matched, even if that word does not appear at all in the source text. For instance, if one were to use a fuzzy query over an ngram analyzed field, the results would likely be bizarre, as ngrams break words up into many small letter combinations, many of which are only an edit or two away, though the actual words involved are quite dissimilar. This also means that if using say, a snowball analyzer, a fuzzy search for 'running', will be stemmed to 'run', but will not match the misspelled word 'runninga', which stems to 'runninga', because 'run' is more than 2 edits away from 'runninga'. This can cause quite a bit of confusion, and for this reason, it often makes sense only to use the simple analyzer on text intended for use with fuzzy queries, possibly disabling synonyms as well. To clarify this, a diagram showing a fuzzy query running against a snowball analyzed document can be seen below.</p><p><img src="https://api.contentstack.io/v2/assets/575e4c8b43e9adc538713ab9/download?uid=blt19ff87a32a904536?uid=blt19ff87a32a904536" data-sys-asset-uid="blt9f4c5c9fe47cabdb" style="max-width: 100%;" alt="A fuzzy query using a snowball analyzer"/></p><figcaption>A fuzzy query using a snowball analyzer</figcaption></section><section id="the-different-types-of-fuzzy-searches" class="level2"><h2><a href="#the-different-types-of-fuzzy-searches" target="_self">The Different Types of Fuzzy Searches</a></h2><p>Multiple types of fuzzy search are supported by elasticsearch and the differences can be confusing. The list below attempts to disambiguate these various types.</p><ul><li><a href="/guide/en/elasticsearch/reference/current/query-dsl-match-query.html" target="_self"><strong>match</strong></a><strong> query + </strong><strong>fuzziness</strong><strong> option:</strong> Adding the fuzziness parameter to a match query turns a plain match query into a fuzzy one. Analyzes the query text before performing the search.</li><li><a href="/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-query.html" target="_self"><strong>fuzzy</strong></a><strong> query:</strong> The elasticsearch fuzzy query type should generally be avoided. Acts much like a term query. Does <em>not</em> analyze the query text first.</li><li><a href="/guide/en/elasticsearch/reference/current/query-dsl-flt-query.html" target="_self"><strong>fuzzy_like_this</strong></a><strong>/</strong><a href="/guide/en/elasticsearch/reference/1.7/query-dsl-flt-field-query.html" target="_self"><strong>fuzzy_like_this_field</strong></a><strong>:</strong> A <a href="/guide/en/elasticsearch/reference/current/search-more-like-this.html" target="_self">more_like_this</a> query, but supports fuzziness, and has a tuned scoring algorithm that better handles the characteristics of fuzzy matched results.*</li><li><a href="/guide/en/elasticsearch/reference/current/search-suggesters.html" target="_self"><strong>suggesters</strong></a><strong>:</strong> Suggesters are not an actual query type, but rather a separate type of operation (internally built on top of fuzzy queries) that can be run either alongside a query, or independently. Suggesters are great for 'did you mean' style functionality.</li></ul><p>A match query with the fuzziness parameter set is perhaps the most versatile of the fuzzy queries. The fuzzy query type supports the exact same behavior, except it does not allow for any analysis for the query text. Additionally, the fuzzy query type is a subset of the functionality of a match query makes it more confusing than useful.</p><p>The fuzzy_like_this, or FLT, queries are useful for providing recommendations based on a large piece of source text that may contain misspellings or other inaccuracies separated by edit distance. There are two different queries in this category fuzzy_like_this and fuzzy_like_this_field, which both provide the same functionality, the latter query simplifying the syntax for the case where only a single field is used. These queries take a parameter, like_text, consisting of a large piece of text, say the body of an article, and try to find documents 'like' that one. The text in the article is checked, and query terms are weighted based their frequency in the like_text with special tweaks (disabling the coord factor, source text based IDF) for combining the various terms from the source text. FLT queries work best for cases where the corpus contains a large number of misspellings, otherwise a standard more_like_this query will have better performance. More detail can be found on the <a href="/guide/en/elasticsearch/reference/current/query-dsl-flt-query.html" target="_self">official documentation for Fuzzy More Like This</a>.</p><p>Suggesters are fantastic in the case where a user types in 'New Yrok', for a search, having intended to type in 'New York'. Applications can suggesters to present a 'did you mean' box in the UI recommending a potential correction. Beware that for the specific case of type-ahead search the <a href="/guide/en/elasticsearch/reference/current/search-suggesters-completion.html" target="_self">completion API</a> offers better performance given the latency sensitive nature of type-ahead searches. More information on suggesters can be found on the <a href="/guide/en/elasticsearch/reference/current/search-suggesters.html" target="_self">search suggesters</a> page.</p></section><section id="performance-considerations" class="level2"><h2><a href="#performance-considerations" target="_self">Performance Considerations</a></h2><p>Even though Lucene's Levenshtein distance implementation is state of the art, and quite fast, it is still much slower than a plain match query. The runtime of the query grows with the number of unique terms in the index. That is to say, when performing a fuzzy search the main criteria is not how many documents will be returned, but how many unique terms across the cluster there are for the fields being searched. If there are 100 documents with 10,000 unique words apiece, searching that index will be slower than searching 10,000 documents where the field being searched only has 100 unique words.</p><p>The primary reason for this slowness is that a standard match query can quickly check the terms index, an internal datastructure used by Lucene, for a match and find documents extremely quickly using binary search. This process is fast even for large dictionaries since binary searches scale well. Fuzzy queries, on the other hand, use a more advanced algorithm involving a <a href="http://en.wikipedia.org/wiki/Deterministic_finite_automaton" target="_self">DFA</a> which must process a large number of terms. Processing the much larger number of terms required for a fuzzy search is always slower than a simple binary search. As an example, running a fuzzy search on the English language Wikipedia dataset takes approximately 320ms for the term “history” given a min_similarity of 2, while a plain match search for the same term runs in a mere 35ms, an order of magnitude difference. Given a min_similarity of 1, the search time is only 150ms. It should be noted that these tests were not conducted in a statistically sound manner, and these times can be affected significantly by factors particular to individual applications. The Wikipedia dataset in particluar, has many more terms than a large number of common use cases.</p><p>Performance can be significantly improved by requiring that matches have exact prefix matches with the query. This dramatically shortens the search space at the expense of not finding words with a misspelling toward the front. The longer the prefix length, the faster the speedup. Specifying a prefix_length of 1 cuts the 320ms query mentioned above down to a mere 104ms. Increasing this number yielded further speed gains. Generally, it is advisable to require a prefix for datasets with large numbers of terms or performance will be poor, oftentimes in the 100s of milliseconds.</p><p>The max_expansions setting, which defines the maximum number of terms the fuzzy query will match before halting the search, can also have dramatic effects on the performance of a fuzzy query. Cutting down the query terms has a negative effect, however, in that some valid results may not be found due to early termination of the query. It is important to understand that the max_expansions query limit works at the shard level, meaning that even if set to 1, multiple terms may match, all coming from different shards. This behavior can make it seem as if max_expansions is not in effect, so beware that counting unique terms that come are returned is not a valid way to determine if max_expansions is working.</p></section><section id="alternatives-to-fuzzy-matching" class="level2"><h2><a href="#alternatives-to-fuzzy-matching" target="_self">Alternatives to Fuzzy Matching</a></h2><p>Fuzzy matching isn't always the right tool for the job, oftentimes imprecise matches can be found through other techniques. The <a href="https://github.com/elasticsearch/elasticsearch-analysis-phonetic" target="_self">phonetic analysis plugin</a> contains a number of fascinating tools for approximating matches, such as the <a href="http://en.wikipedia.org/wiki/Metaphone" target="_self">metaphone</a> analyzer, which finds words that sound similar to other words. For instance, if what is required is making sure words like run and ran are both considered equivalent, a smart analyzer like a Snowball analyzer is preferable. Alternatively, for checking misspellings, N-gram analysis <a href="https://web.archive.org/web/20140209084956/http://exploringelasticsearch.com/book/searching-natural-language/searching-non-word-text.html" target="_self">as described in this short tutorial</a> can run quite a bit faster at query time, depending on the dataset. N-Grams do, however, come at the cost of additional storage / memory usage, slightly more index-time processing, and a long tail of false positives after good matches.</p></section><section id="more-reading" class="level2"><h2><a href="#more-reading" target="_self">More Reading</a></h2><ul><li>For a more in-depth look at the performance of Fuzzy Queries in elasticsearch, Lucene core contributor Mike McCandless has written a fascinating <a href="http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html" target="_self">blog post</a> on the topic. There is also a video, Finite State Automata in Lucene and Solr, of a presentation by Dawid Weiss that covers Levenshtein and other automata in Lucene.</li><li>If you would like to learn more about Levenshtein Automata, Nick Johnson has a great post <a href="http://blog.notdot.net/2010/07/Damn-Cool-Algorithms-Levenshtein-Automata" target="_self">Damn Cool Algorithms: Levenshtein Automata</a>.</li></ul></section>

How to Use Fuzzy Searches in Elasticsearch

<html><head></head><body><section id="post_content">
		<article>
			<p>A few months ago we released client libraries for PHP, Ruby, Python, and Perl and today we add another to the family, JavaScript! This new client runs in Node.js and modern browsers, and aims to solve the same problems that the others do:</p>
<ul>
<li>provide access to the entire Elasticsearch REST API</li>
<li>play nice with clusters</li>
<li>automatically discover nodes when desired</li>
<li>intelligently handle node failure</li>
<li>be easily extendable, so that you can really have it behave just the way you want</li>
</ul>
<p>If you didn&#x2019;t know that we are developing our own clients you should check out this <a title="Unleash the clients!  Ruby, Python, PHP, Perl" href="/blog/unleash-the-clients-ruby-python-php-perl/">post</a> from the first round of releases. In that post Clint explains why we decided to go down this road, as well as some of the goals of the project.</p>
<ul>
<li><a href="https://github.com/elasticsearch/elasticsearch-js">Github Repo</a></li>
<li><a href="https://github.com/elasticsearch/elasticsearch-js/issues">Issue Tracker</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/client/javascript-api/current/index.html">Documentation</a></li>
</ul>
<p>Just like our other clients, Elasticsearch.js is licensed under Apache 2.0 and hosted on <a href="https://github.com/elasticsearch/elasticsearch-js">Github</a>. Getting started with Elasticsearch.js is almost easier than getting started with Elasticsearch itself.</p>
<h2>Getting the Node.js module</h2>
<p>To install the module into an existing Node.js project use npm:</p>
<pre class="prettyprint noescape lang=shell">npm install elasticsearch</pre>
<h2>Getting the browser client</h2>
<p>For a browser-based projects, builds for modern browsers are available <a title="Elasticsearch.js - Browser Builds" href="http://www.elasticsearch.org/guide/en/elasticsearch/client/javascript-api/current/about.html#_browser_builds">here</a>. Download one of the archives, and extract it. Inside you&#x2019;ll find three files, pick the one that best matches your environment:</p>
<ul>
<li>elasticsearch.jquery.js &#x2013; use this build if you are already using jQuery</li>
<li>elasticsearch.angular.js &#x2013; use this build in Angular projects</li>
<li>elasticsearch.js &#x2013; in all other cases, use this build</li>
</ul>
<p>Each of the library specific builds tie into the AJAX and Promise creation facilities provided by their respective libraries. This is an example of how Elasticsearch.js can be extended to provide a more opinionated approach when appropriate.</p>
<h2>Setting up the client</h2>
<p>Now you are ready to get busy! First thing you&#x2019;ll need to do is create an instance of <code>elasticsearch.Client</code>. Here are several examples of configuration parameters you can use when creating that instance. For a full list of configuration options see <a href="http://www.elasticsearch.org/guide/en/elasticsearch/client/javascript-api/current/configuration.html">the configuration docs</a>.</p>
<pre class="prettyprint noescape lang=javascript">var elasticsearch = require(&apos;elasticsearch&apos;);

// Connect to localhost:9200 and use the default settings
var client = new elasticsearch.Client();

// Connect the client to two nodes, requests will be
// load-balanced between them using round-robin
var client = elasticsearch.Client({
  hosts: [
    &apos;elasticsearch1:9200&apos;,
    &apos;elasticsearch2:9200&apos;
  ]
});

// Connect to the this host&apos;s cluster, sniff
// for the rest of the cluster right away, and
// again every 5 minutes
var client = elasticsearch.Client({
  host: &apos;elasticsearch1:9200&apos;,
  sniffOnStart: true,
  sniffInterval: 300000
});

// Connect to this host using https, basic auth,
// a path prefix, and static query string values
var client = new elasticsearch.Client({
  host: &apos;https://user:password@elasticsearch1/search?app=blog&apos;
});
</pre>
<h2>Setting up the client in the browser</h2>
<p>The params accepted by the Client constructor are the same in the browser versions of the client, but how you access the Client constructor is different based on the build you are using. Below is an example of instantiating a client in each build.</p>
<pre class="prettyprint noescape lang=javascript">// elasticsearch.js adds the elasticsearch namespace to the window
var client = elasticsearch.Client({ ... });

// elasticsearch.jquery.js adds the es namespace to the jQuery object
var client = jQuery.es.Client({ ... });

// elasticsearch.angular.js creates an elasticsearch
// module, which provides an esFactory
var app = angular.module(&apos;app&apos;, [&apos;elasticsearch&apos;]);
app.service(&apos;es&apos;, function (esFactory) {
  return esFactory({ ... });
});
</pre>
<h2>Using the client instance to make API calls.</h2>
<p>Once you create the client, making API calls is simple.</p>
<pre class="prettyprint noescape lang=javascript">// get the current status of the entire cluster.
// Note: params are always optional, you can just send a callback
client.cluster.health(function (err, resp) {
  if (err) {
    console.error(err.message);
  } else {
    console.dir(resp);
  }
});

// index a document
client.index({
  index: &apos;blog&apos;,
  type: &apos;post&apos;,
  id: 1,
  body: {
    title: &apos;JavaScript Everywhere!&apos;,
    content: &apos;It all started when...&apos;,
    date: &apos;2013-12-17&apos;
  }
}, function (err, resp) {
  // ...
});

// search for documents (and also promises!!)
client.search({
  index: &apos;users&apos;,
  size: 50,
  body: {
    query: {
      match: {
        profile: &apos;elasticsearch&apos;
      }
    }
  }
}).then(function (resp) {
  var hits = resp.body.hits;
});</pre>
<h2>Entering Construction Zone</h2>
<p>This release of Elasticsearch.js should be considered beta. Please install it, try it out and give us feedback. What doesn&#x2019;t work? What is missing? What can we do better? We are excited to receive your issues and pull requests!</p>
<p>We will be actively working to squash bugs as soon as they pop-up, and plan to do a full performance audit (it&#x2019;s quick already, but we want to be certain that it&#x2019;s also efficient as possible). We are also interested in creating wrappers for this library which might include plugins for other ORM modules like Mongoose or Bookshelf, helpers for some of the more complex parts of the API. If you would like to see anything specific let us know in a <a href="https://github.com/elasticsearch/elasticsearch-js/issues">Github issue</a>!</p>
<h3>Happy searching!</h3>
			<aside id="post_tags">
							</aside>
		</article>
			</section></body></html>

Announcing Elasticsearch.js for Node.js and the Browser

<html><head></head><body><p>A few weeks ago I finished the initial release of the Logstash collectd input plugin.  I&apos;m really excited about this new feature!
</p>
<p>Some of you may be wondering why we would go to this effort, seeing how collectd data isn&apos;t exactly, well, <em>logs</em>.  We consider any data that has a corresponding timestamp to be an event in Logstash.  The data collectd sends is also timestamp + metric data.  It will fit right alongside your other log data as a valuable companion.  Now you can see if there&apos;s a corresponding change in disk I/O or CPU load when you see certain log entries, or vice versa!  The possibilities are vast!  I can&apos;t wait to hear how you use this plugin to help visualize your data!
</p>
<h4>collectd configuration</h4>
<p>The simplest way to get started is to configure collectd.   You may or may not already have a collectd setup where you are, but your configuration file could be as simple as this:
</p>
<pre class="prettyprint lang=js">Hostname    &quot;host.example.com&quot;
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin network
&lt;Plugin interface&gt;
    Interface &quot;eth0&quot;
    IgnoreSelected false
&lt;/Plugin&gt;
&lt;Plugin network&gt;
    &lt;Server &quot;10.0.0.1&quot; &quot;25826&quot;&gt;
    &lt;/Server&gt;
&lt;/Plugin&gt;
</pre>
<p>These options will send collectd information consisting of cpu load, memory stats, and network traffic information via UDP to the IP 10.0.0.1 on port 25826.  For your own setup just replace 10.0.0.1 with the IP or hostname of your Logstash instance.  The collectd plugin will populate the Logstash event host field with whatever is in the &#x201C;Hostname&quot; directive rather than what reverse lookup finds.  If otherwise unset, the default configuration will send values every 10 seconds.  You can learn more about collectd configuration <a title="collectd configuration documentation" href="https://collectd.org/documentation.shtml" target="_blank">here</a>.
</p>
<p>This example is only a tiny sample of the kind of plugins and configurations collectd has to offer.  A comprehensive list can be found <a title="List of collectd plugins" href="https://collectd.org/wiki/index.php/Table_of_Plugins" target="_blank">here</a>.  As with Logstash, you can write your own plugins for collectd so there are virtually endless possibilities!
</p>
<h4>Logstash configuration</h4>
<p>Now that we have our collectd ready to send let&apos;s configure our Logstash instance
</p>
<pre class="prettyprint lang=js">input {
  collectd {}
}
</pre>
<p>Yep, that&apos;s it.  Pretty crazy, right?  We try to come up with sane defaults for everything.  The full configuration explanations are <a title="Logstash collectd input plugin documentation" href="http://logstash.net/docs/1.3.1/inputs/collectd" target="_blank">here</a>.  Let me explain some important ones.
</p>
<ul>
	<li><code>buffer_size</code> The default is 1452, which is what the official collectd server process expects too.</li>
	<li><code>port</code> The default is 25826, which is also what the official collectd server process expects.</li>
	<li><code>prune_intervals</code> The default is true.  This one is a bit trickier to explain as it is part of the way collectd sends data.  In order to allow some applications to properly gauge frequency a special field is sent to indicate how frequently that field will be updating.  Another is subsequently sent each time that interval is reached.  In Logstash we typically do not care about this field, but if you want it anyway, you can enable it by setting prune_intervals to false.</li>
	<li><code>typesdb</code> The typesdb is the definition file for collectd data.  I included the typesdb from the most recent (5.4.0 at this writing) version of collectd so you won&apos;t need to use this option unless you have one or more custom typesdb files. This option is simply an array of paths, &lt;br&gt;e.g. &lt;code&gt;[ &quot;/path/to/typesdb1&quot;, &quot;/path/to/typesdb2&quot;, &#x2026; ]</li>
</ul>
<p>So, what is the result of this little example?  I ran this test on my 2013 MacBook Air, with the data coming from a nearby computer with 2 Ethernet ports and 16G of memory.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d87d8edd48f76938858/download?uid=bltb30fb5b045a8a491?uid=bltb30fb5b045a8a491" style="max-width:100%;">
</p>
<p>This is traffic from a single box over a 1 hour period.
</p>
<p>As you can see from this graph the memory is broken down into blocks: <b>Inactive</b>, <b>Free</b>, <b>Wired</b>, <b>Active</b>.<br>
	The CPU load histogram should be fairly self-explanatory, as are the network I/O charts.  You can clearly see peaks every 5 minutes in network traffic on EN1 with little other traffic usually.
</p>
<p><strong>Caveat:</strong> The version of Kibana I am using here (as of 16 Dec 2013) was downloaded straight from the master branch on GitHub and is not an officially released version. I needed this version as it enabled me to do derivative graphing, where each subsequent point is the difference between the current and previous values.  This is necessary with network values as collectd measures them as it simply sends the counter values from the kernel store (similar to SNMP network data). If you need this feature now you too can use the most current development version.  The release of Kibana coming in Jan 2014 (coinciding with ES 1.0 release) will have this feature.
</p>
<p><br>
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d87d8edd48f7693885d/download?uid=blt673086edaa18b036?uid=blt673086edaa18b036" style="max-width:100%;">
</p>
<p>The stats themselves are unimpressive, seeing how they are from a personal server on home network with limited traffic.  So I fired up <code style="line-height: 1.6em;">collectd-tg</code>, the collectd traffic generator.  This was the result:
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d870342dfd738265ced/download?uid=blt6254b10d30bdd81e?uid=blt6254b10d30bdd81e" style="max-width:100%;">
</p>
<p>Keeping in mind that this was tested on a MacBook Air I thought it was a pretty good show: 3000 events per second.  If I had configured servers to send an average of 30 events every 10 seconds (3 events per second) that amounts to my laptop being able to process a continuous stream of events from 1000 servers!
</p>
<p>So, there&apos;s a brief introduction to the collectd plugin.  Happy Logstashing!
</p></body></html>

Logstash Collectd Input Plugin

<html><head></head><body><p>For a long time in the Logstash community we&#x2019;ve been advising users to apply an index mapping template.  There are a number of compelling reasons to do this, including:</p>
<ul>
<li>Elasticsearch tuning (<code>&quot;index.refresh_interval&quot; : &quot;5s&quot;</code>, for example, reduces disk I/O)</li>
<li>Custom definitions for fields (numbers, strings, IP, etc.)</li>
<li>Custom lists of stopwords</li>
<li>Custom analyzers</li>
<li>Your fields can be set to <code>not_analyzed</code>, so terms faceting won&#x2019;t see useless or non-existing data (see Jordan&#x2019;s awesome screen caps of this <a title="Logstash 1.3.1: Awesome new default template" href="/blog/logstash-1-3-1-released/" target="_blank">here</a>)</li>
</ul>
<p>Not having that last option set has resulted in a near-constant stream of similar questions, &#x201C;Why do the terms break up on hyphens?&#x201D; and others of this sort.  To answer these questions and prevent future questions like it from being asked, it became one of the first tasks I was assigned to work on as an employee at Elasticsearch.</p>
<h4>Important note</h4>
<p>The <code><a href="/guide/en/logstash/current/index.html" title="elasticsearch output plugin" target="_blank">elasticsearch</a></code> plugin uses Java API calls to manage the template, while the <code><a href="/guide/en/logstash/current/index.html" title="elasticsearch_http output plugin" target="_blank">elasticsearch_http</a></code> plugin uses REST API calls.  With this comes an important caveat.</p>
<p>In order to use template management with the <code>elasticsearch</code> plugin you must be using version 0.90.5 or newer.  The Java API calls did not exist prior to that.  If you attempt to use Logstash v1.3+ with a version of Elasticsearch older than 0.90.5 with the <code>elasticsearch</code> output plugin the template management features will not work and there will be a stack trace in the log files indicating the absence of those API calls.</p>
<p>The REST API has no such constraints.  If you are using the <code>elasticsearch_http</code> output with an older version of Elasticsearch it will still attempt to assign the template.  Some new template options may not exist in very old versions of Elasticsearch, so be sure to upgrade.  Upgrading is good.  You want all of the performance benefits and new features that come with new releases, right?</p>
<h3>Configuration options.</h3>
<p>Common to both the <code>elasticsearch</code> and <code>elasticsearch_http</code> plugin are the following options <em>(with their defaults)</em>:</p>
<ul>
<li><code>manage_template</code> <em>(true)</em></li>
<li><code>template_name</code> <em>(&#x201C;logstash&#x201D;)</em></li>
<li><code>template</code> <em>(if unset, use internal)</em></li>
<li><code>template_overwrite</code> <em>(false)</em></li>
</ul>
<h4><code>manage_template</code></h4>
<p>The <code>manage_template</code> option is boolean and is only used to disable the automatic template feature since it is on by default.  Who would use this feature?  Why wouldn&#x2019;t you want the awesomeness of automatic template management?  One such reason might be that you have dynamically named indices.  For example, if you want a different index name for production logs than staging, but in the same Elasticsearch cluster, you could configure that in Logstash:</p>
<pre class="prettyprint lang=js">output {
  elasticsearch { 
    cluster =&gt; &quot;mycluster&quot; 
    manage_template =&gt; false 
    index =&gt; &quot;%{segment_name}-logstash-%{+YYYY.MM.dd}&quot; 
  }
}</pre>
<p>In this example the index is determined in part by date, and in part by a variable, <code>segment_name</code>.  When using complex index names we recommend setting the template manually.</p>
<h4><code>template_name</code></h4>
<p>The <code>template_name</code> option determines what name the template will be stored as in Elasticsearch.  The default is <code>logstash</code>.  There is an important caveat to note with this setting.  If you change the <code>template_name</code> option in a fully configured and running system you&#x2019;ll still have a template stored under that name.  In this case you may want to clean out the old template so it&#x2019;s not left around.</p>
<pre class="prettyprint lang=js">curl -XDELETE http://localhost:9200/_template/OLD_template_name?pretty</pre>
<p> <br>
where <code>OLD_template_name</code> is the previous template name.</p>
<h4><code>template</code></h4>
<p>The <code>template</code> configuration option is useful if you want to use the template engine but provide your own template instead of the included one.  An example might be:</p>
<pre class="prettyprint lang=js">output {
  elasticsearch { 
    cluster =&gt; &quot;mycluster&quot; 
    template =&gt; &quot;/path/to/mytemplate.json&quot;
  }
}</pre>
<p>Because the default is to manage templates, this configuration would read the JSON from the indicated file and attempt to upload the template if there isn&#x2019;t one already there.  This brings me to&#x2026;</p>
<h4><code>template_overwrite</code></h4>
<p>The <code>template_overwrite</code> option will always overwrite the indicated template in Elasticsearch with either the one indicated by <code>template</code> or the included one.  This option is set to <code>false</code> by default.  If you always want to stay up to date with the template provided by Logstash, this option could be very useful to you.  Likewise, if you have your own template file managed by puppet, for example, and you wanted to be able to update it regularly, this option could help there as well.</p>
<h3>Conclusion</h3>
<p>Hopefully this helps make the configuration options more clear.  Happy Logstashing!</p></body></html>

New in Logstash 1.3: Elasticsearch Index Template Management

<html><head></head><body><p>Hello friends!</p>
<p>We have released <a href="/guide/en/logstash/current/index.html">logstash 1.3.1</a> with lots of new fixes and features. You can <a title="view the full changelog" href="https://github.com/logstash/logstash/blob/v1.3.1/CHANGELOG#L1-L89" target="_blank">view the full changelog</a>, but I&#x2019;d like to highlight two of the new features, both of which were implemented by <a href="https://twitter.com/theuntergeek">Aaron Mildenstein</a>.</p>
<p>First, in our mission to make it easy to integrate with lots of tools, we have a new input plugin, collectd. This plugin lets you receive metrics from <a title="collectd" href="http://collectd.org/" target="_blank">collectd</a> agents with logstash and ship them anywhere you want. Want to know more? Check out the <a href="/guide/en/logstash/current/index.html" target="_blank">documentation for the collectd input</a>.</p>
<p>Second, as a way to make logstash have the best possible default behavior for the most users, we now have logstash providing its own <a href="/guide/en/elasticsearch/reference/current/indices-templates.html">index template</a>. This index template was built to try and solve some common problems for users by using better default analyzers and mappings specific to logging use cases.</p>
<p>To demonstrate how this new feature works, let&#x2019;s look at some apache logs in Kibana. I&#x2019;ve parsed these apache logs with the following logstash config:</p>
<pre class="prettyprint lang=js">filter {
  grok {
    match =&gt; { &quot;message&quot; =&gt; &quot;%{COMBINEDAPACHELOG}&quot; }
  }

  date {
    match =&gt; [ &quot;timestamp&quot;, &quot;dd/MMM/yyyy:HH:mm:ss Z&quot; ]
  }
}</pre>
<p>The filter configuration above uses grok with the built-in apache log pattern to parse apache logs into separate fields such as the request path, http response code, bytes sent, user agent, etc. The second filter takes the original timestamp field in the apache log and parses it to be used as the canonical timestamp of the event &#x2013; this gives you more accurate search results over time and also lets you ingest old log data correctly.</p>
<p>Now, a common search pattern is to ask for the top N of something. In Kibana, you can either use the &#x2018;top N&#x2019; query or you can use a pie chart, depending on your goals. In this example, I&#x2019;ll just use a pie chart. Adding a pie chart with mode &#x2018;terms&#x2019; on the &#x2018;request&#x2019; field gets me this:</p>
<p><img alt="pie chart!" src="https://api.contentstack.io/v2/assets/575e4c8e9985d58976376a71/download?uid=blta3ac55be3c8629ac?uid=blta3ac55be3c8629ac" data-sys-asset-uid="blt40571d4bfab9d99f" width="300" height="263"></p>
<p>Most folks, in this situation, sit and scratch their heads, right? I know I did the first time. I&#x2019;m pretty certain &#x201C;docs&#x201D; and &#x201C;centralized&#x201D; aren&#x2019;t valid paths on the logstash.net website! The problem here is that the pie chart is built from a <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-facets-terms-facet.html" target="_blank">terms facet</a>. With the default text analyzer in elasticsearch, a path like &#x201C;/docs/1.3.1/filters/&#x201D; becomes 3 terms {docs, 1.3.1, filters}, so when we ask for a terms facet, we only get individual terms back!</p>
<p>Index templates to the rescue! The <a href="https://github.com/logstash/logstash/blob/v1.3.1/lib/logstash/outputs/elasticsearch/elasticsearch-template.json">logstash index template</a> we provide adds a &#x201C;.raw&#x201D; field to every field you index. These &#x201C;.raw&#x201D; fields are set by logstash as &#x201C;not_analyzed&#x201D; so that no analysis or tokenization takes place &#x2013; our original value is used as-is! If we update our pie chart above to instead use the &#x201C;request.raw&#x201D; field, we get the following:</p>
<p><img alt="pie chart with &apos;request.raw&apos; field" src="https://api.contentstack.io/v2/assets/575e4c8d0342dfd73826311f/download?uid=blt78ea9a2d4aa94fc6?uid=blt78ea9a2d4aa94fc6" data-sys-asset-uid="bltee67415a32194870" width="300" height="292"></p>
<p>Much better!</p>
<p>And because we still index both the terms and the not-analyzed parts for each field, you can still do simple term queries like &#x201C;request:docs&#x201D; to find all requests with &#x2018;docs&#x2019; in the text.</p>
<p>I hope this helps explain the new feature. Happy logstashing!</p></body></html>

Logstash 1.3.1 Released!

<html><head></head><body><p>
	Welcome to&#xA0;<i>This week in Elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project. Also, do not miss our <a href="http://www.elasticsearch.org/webinars/introduction-to-logstash/">Introduction to logstash</a> webinar tomorrow!
</p><h3>Elasticsearch core</h3><ul>
	<li><code>has_child</code> query can yield in wrong results if parent type has nested inner objects (<a href="https://github.com/elasticsearch/elasticsearch/issues/4341">#4341</a>, 0.90 and master)</li>
	<li>Fix <code>_all</code> boosting, which did not work correctly under certain conditions (<a href="https://github.com/elasticsearch/elasticsearch/issues/4315">#4315</a>, 0.90 and master)</li>
	<li>Allow to disable sending a refresh-mapping to master node, using the <code>indices.cluster.send_refresh_mapping</code> setting (<a href="https://github.com/elasticsearch/elasticsearch/issues/4342">#4342</a>, 0.90 and master)</li>
	<li>FsStats: Added a total section, that sums up infos for all data paths (<a href="https://github.com/elasticsearch/elasticsearch/issues/4343">#4343</a>, 0.90 and master)</li>
	<li>FsStats: Added IO operation information, which sums op reads and writes (<a href="https://github.com/elasticsearch/elasticsearch/issues/4344">#4344</a>, 0.90 and master)</li>
	<li>Fixed inconsistent proximity search results (<a href="https://github.com/elasticsearch/elasticsearch/issues/4356">#4356</a>, 0.90 and master)</li>
	<li>Improve processing time for indices with large updates of mappings (<a href="https://github.com/elasticsearch/elasticsearch/issues/4373">#4373</a>, 0.90 and master)</li>
	<li>CpuStats: Add a new <code>usage</code> metric (<a href="https://github.com/elasticsearch/elasticsearch/issues/4374">#4374</a>, 0.90 and master)</li>
	<li>Support YAML as output format of the analyze API (<a href="https://github.com/elasticsearch/elasticsearch/issues/4311">#4311</a>, master only)</li>
	<li>Query String caching could cause matched_filters not working (<a href="https://github.com/elasticsearch/elasticsearch/issues/4361">#4361</a>, 0.90 and master)</li>
	<li>Support postings highlighter in percolate API (<a href="https://github.com/elasticsearch/elasticsearch/issues/4385">#4385</a>, 0.90 and master)</li>
	<li>The <code>BalancedShardsAllocator</code> accidentally overwrote some of its own set variables in specific cases  (<a href="https://github.com/elasticsearch/elasticsearch/issues/4390">#4390</a>, 0.90 and master)</li>
	<li>Remove <code>term_index_interval</code> and <code>term_index_divisor</code> settings (<a href="https://github.com/elasticsearch/elasticsearch/issues/3912">#3912</a>, 0.90 and master)</li>
	<li>Fix bug in explain for <code>function_score</code> queries. (<a href="https://github.com/elasticsearch/elasticsearch/commit/1037d071bfc2b6e535ec1f532681afdd837b2502">commit</a>, 0.90 and master)</li>
	<li>A search preference for an unknown node returns a client error now (<a href="https://github.com/elasticsearch/elasticsearch/commit/c4f3da2b9d6294a1871a87b3e489f9896b95f72e">commit</a>, 0.90 and master)</li>
	<li>Introduce <code>BytesTransportRequest</code>, which allows to share request bodies for several requests instead of allocating new objects (like in the cluster state) (<a href="https://github.com/elasticsearch/elasticsearch/commit/a9e259d438c3cb1d3bef757db2d2a91cf85be609">commit</a>, 0.90 and master)</li>
	<li>Wildcard support to field resolving in Get Field Mapping API (<a href="https://github.com/elasticsearch/elasticsearch/issues/4367">#4367</a>, 0.90 and master)</li>
	<li>Packaging: Ensure setting of sysctl <code>vm.max_map_count</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/4397">#4397</a>, 0.90 and master)</li>
	<li>Geodistance calculations now make use of the faster <code>SloppyMathUtils</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/3862">#3862</a>, master)</li>
</ul><h3>Elasticsearch Ecosystem</h3><p>
	Here&apos;s some more information about what is happening in the ecosystem we are maintaining around Elasticsearch, including plugin and driver releases, as well as news about Logstash and Kibana.
</p><ul>
	<li>The fine folks at <a href="https://twitter.com/docker">docker.io</a> also <a href="http://blog.docker.io/2013/12/improved-index-search-engine-with-suggestions-and-better-search-results/">switched to elasticsearch for searching their repository</a> </li>
	<li><a href="https://twitter.com/aglover">Andrew Glover</a> took the time to provider an Elasticsearch vagrant image and <a href="http://thediscoblog.com/blog/2013/11/25/elasticsearch-in-a-box/">blogged about it</a> </li>
	<li><a href="http://www.bazaarvoice.com/">Bazaarvoice</a> explains their <a href="http://highscalability.com/blog/2013/12/2/evolution-of-bazaarvoices-architecture-to-500m-unique-users.html">technology stack</a>, which also includes elasticsearch </li>
	<li><a href="https://twitter.com/mozilla">Mozilla</a> has relaunched the Mozilla Developer Network using elasticsearch <a href="https://twitter.com/jezdez/status/410327871497203712"></a> </li>
	<li><a href="https://twitter.com">Netflix</a> talks about <a href="http://techblog.netflix.com/2013/12/announcing-suro-backbone-of-netflixs.html">their data pipeline called Suro</a>, which includes elasticsearch as part of their stack </li>
	<li>French: <a href="https://twitter.com/vspiewak">Vincent Spiewak</a> writes about <a href="http://blog.xebia.fr/2013/12/06/logstash-elasticsearch-kibana-s01e01-analyse-de-reputation-sur-twitter/">Logstash, ElasticSearch, Kibana &#x2013; S01E01 &#x2013; Analyse d&apos;e-r&#xE9;putation sur Twitter</a> </li>
	<li>Thomas Sutton wrote about his takeaways from the first <a href="http://passingcuriosity.com/2013/sydney-elasticsearch-meetup/">Elasticsearch Meetup in Sydney</a> </li>
	<li>Greg Brown offers some advice about <a href="http://gibrown.wordpress.com/2013/12/05/managing-elasticsearch-cluster-restart-time/">minimizing cluster startup time</a> due to shard allocation.</li>
</ul><h3>Slides</h3><ul>
	<li>One of the last elasticsearch meetups at Atlassian was recorded and shows how <a href="http://zuhaiblog.com/2013/12/04/hipchat-elasticsearch-meetup-video-slides/">HipChat scaled to 1 Billions messages using elasticsearch</a> </li>
	<li><a href="https://twitter.com/kimchy">Shay Banon</a> talked last week in Amsterdam about, <a href="https://speakerdeck.com/elasticsearch/why-we-built-elasticsearch">why we built elasticsearch</a></li>
	<li>French: <a href="https://twitter.com/jpountz">Adrien Grand</a> published his slides from the last meetup in Paris: <a href="https://speakerdeck.com/elasticsearch/dessine-moi-un-index-fr">Dessine moi un index</a> </li>
</ul><h3>Where to find Us</h3><h4>France</h4><p>
	<a href="http://twitter.com/dadoonet">David Pilato</a> will speak at the <a href="http://toursjug.cloud.xwiki.com/xwiki/bin/view/Meetings/20131211">Tours Java Users&apos; Group on December 11th</a>. David&apos;s talk will cover all things Elasticsearch, including new features in 1.0. Doors open at 7:15 PM.
</p><h4>Israel</h4><p>
	The first ever <a href="http://www.meetup.com/Elasticsearch-Tel-Aviv-Meetup/events/152066232/">Elasticsearch Tel Aviv meetup</a> will take place on December 30th, beginning at 7 PM. Join Elasticsearch core developer <a href="http://twitter.com/uboness">Uri Boness</a>, who will present on what&apos;s new in Elasticsearch 1.0, and Itamar Syn-Hershko, who will discuss Monitoring online conversation at scale using Elasticsearch.
</p><h4>Netherlands</h4><p>
	<a href="http://twitter.com/bleskes">Boaz Leskes</a> will talk about Elasticsearch at <a href="http://www.meetup.com/010DEV/">010dev</a> in Rotterdam at the 9th of January.
</p><h3>Where to Find You</h3><p>
	Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, <a href="https://docs.google.com/forms/d/1qBZ6Fwq4JWw7FH3s6C9Q3KeGCvUOmWYOvNMsxNqkCNM/viewform">take a moment to let her know</a>.
</p><p>
	Oh yeah, we&apos;re also&#xA0;<a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you&apos;d like us to find you for employment purposes, just&#xA0;<a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &#xA0;We care more about your skill set and passion for Elasticearch, Kibana and Logstash than where you rest your head.
</p><h3>Trainings</h3><p>
	If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
</p><ul>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/2014-01-13">Amsterdam</a>&#xA0;- January 13, 2014</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/seattle/2014-01-16">Seattle</a>&#xA0;- January 16, 2014</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/hamburg/2014-01-22">Hamburg</a>&#xA0;- January 22, 2014</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/san-jose/2014-01-23">San Jose</a>&#xA0;- January 23, 2014</li>
</ul></body></html>

This Week in Elasticsearch - December 11, 2013

Helpful tips for securing your Elasticsearch clusters and keeping your data safe.

blog-banner-generic-teal.png

Securing Your Elasticsearch Cluster

<html><head></head><body><p>
	Welcome to&#xA0;<i>This week in Elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
</p><h3>Elasticsearch core</h3><ul>
	<li><a href="http://www.elasticsearch.org/downloads/1-0-0-beta2/">Elasticsearch 1.0.0-Beta2 has been released</a>, featuring aggregation, snapshot/restore API and the cat API. Try it out, and give us feedback</li>
	<li>Aggregations now have support for the <code>shard_size</code> parameter (<a href="https://github.com/elasticsearch/elasticsearch/issues/4242">#4242</a>, master)</li>
	<li>Ensuring correct fielddata format for a field in <code>IndexFieldDataService.getForField()</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/4270">#4270</a>, master)</li>
	<li>Catch <code>AlreadClosedException</code> in <code>RobinSearcher#release()</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/4273">#4273</a>, 0.90 &amp; master)</li>
	<li>Maven plugin updates: shade 2.2, assembly plugin 2.4, compiler plugin 3.1, dependency plugin 2.8, eclipse plugin 2.9, jar plugin 2.4, resources plugin 2.6, rpm plugin 2.1-alpha-3, source plugin 2.1.1, surefire plugin 2.16 (<a href="https://github.com/elasticsearch/elasticsearch/pull/4266">#4266</a>, commits <a href="https://github.com/elasticsearch/elasticsearch/commit/79d92dffa6dfd2b48b9dac1db2e23de5481b124e">1</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/b1c62b02b0a0f49d4217c522501ee9df694139e2">2</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/eac602ec70714cfeb40ecc4d63b3f3e4a33d5f00">3</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/2b32121e348314eff84fb74626079582a059fbce">4</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/fa0a5e58441aa66c552c41add5597aa8c2daacfe">5</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/2b32121e348314eff84fb74626079582a059fbce">6</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/d18f0a28d923a07bf4e2ccc5f50658608cd6cbec">7</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/975c43f6d55e0f994f9e9f29a2d63f5abd5a2448">8</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/90c3fce4fe721ad8b2cd964efaa9b2a958a0e246">9</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/baaa1a6aa2d08ac5b6b8f00c9f76db0c3f601fd9">10</a>, 0.90 &amp; master)</li>
	<li><code>has_child</code> query with <code>score_mode=avg</code> can emit infinity as score (<a href="https://github.com/elasticsearch/elasticsearch/issues/4291">#4291</a>, 0.90 &amp; master)</li>
	<li>The percolate API now has support for aggregations (<a href="https://github.com/elasticsearch/elasticsearch/issues/4245">#4245</a>, master)</li>
	<li><code>_cat/indices</code> API waits for cluster health timeout on missing index (<a href="https://github.com/elasticsearch/elasticsearch/issues/4296">#4296</a>, 0.90 &amp; master)</li>
	<li>Cat API: Add <code>/_cat/pending_tasks</code> endpoint (<a href="https://github.com/elasticsearch/elasticsearch/issues/4251">#4251</a>, 0.90 &amp; master)</li>
	<li>Increase priority of shard started task to <code>URGENT</code> (<a href="https://github.com/elasticsearch/elasticsearch/commit/d2f91173b7b5979eccf98a508a5fcebd3cf72657">commit</a>, 0.90 &amp; master)</li>
	<li>Put mapping API was moved to new acknowledgement mechanism (<a href="https://github.com/elasticsearch/elasticsearch/issues/4228">#4228</a>, 0.90 &amp; master)</li>
	<li>Aggregations: Terms aggregator supports filtering terms by regex (<a href="https://github.com/elasticsearch/elasticsearch/issues/4267">#4267</a>, master)</li>
	<li>Fixed <code>AllTermQuery</code> explanations (<a href="https://github.com/elasticsearch/elasticsearch/issues/4298">#4298</a>, 0.90 &amp; master)</li>
	<li>Changed the minScore comparator from <code>&gt;</code> to <code>&gt;=</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/4303">#4303</a>, 0.90 &amp; master)</li>
	<li>Fixed <code>NullPointerException</code> in <code>FsTranslog</code> when reverting transient log (<a href="https://github.com/elasticsearch/elasticsearch/issues/4223">#4223</a>, 0.90 &amp; master)</li>
	<li><code>has_child</code> filter and query could yield inconsistent results (<a href="https://github.com/elasticsearch/elasticsearch/issues/4306">#4306</a>, 0.90 &amp; master)</li>
	<li>Changed the <code>script_lang</code> parameter to <code>lang</code> in all value source based aggs to be consistent with all other script based APIs. (<a href="https://github.com/elasticsearch/elasticsearch/commit/cdc7dfbb2c33c3b243837b39c6a63c271a99940f">commit</a>, master)</li>
	<li>Allow string fields to store token counts (<a href="https://github.com/elasticsearch/elasticsearch/issues/3920">#3920</a>, master)</li>
	<li>Removed bash specific features in shell scripts (<a href="https://github.com/elasticsearch/elasticsearch/issues/4314">#4314</a>, 0.90 &amp; master)</li>
	<li>The <code>FastVectorHighlighter</code> now can combine results from different fields (<a href="https://github.com/elasticsearch/elasticsearch/issues/3750">#3750</a>, 0.90 &amp; master)</li>
	<li>A parsing bug in the new percolator was fixed (<a href="https://github.com/elasticsearch/elasticsearch/issues/4325">#4325</a>, master)</li>
	<li><code>FieldDataSource.Bytes.SortedAndUnique</code> returned wrong hash codes (<a href="https://github.com/elasticsearch/elasticsearch/issues/4330">#4330</a>, master)</li>
	<li>Fail <code>geohash_cell</code> filter if geohash prefix is not enabled (<a href="https://github.com/elasticsearch/elasticsearch/issues/4335">#4335</a>, 0.90 &amp; master)</li>
</ul><h3>Elasticsearch Ecosystem</h3><p>
	Here&apos;s some more information about what is happening in the ecosystem we are maintaining around Elasticsearch, including plugin and driver releases, as well as news about Logstash and Kibana.
</p><ul>
	<li><a href="https://twitter.com/ZacharyTong/">Zachary Tong</a> released <a href="https://github.com/elasticsearch/elasticsearch-php">version 0.4.3</a> of the elasticsearch PHP client</li>
	<li><a href="https://twitter.com/Mpdreamz/">Martijn Laarman</a> released <a href="https://github.com/Mpdreamz/NEST/releases/tag/0.12.0.0">NEST 0.12.0.0</a>, a .NET client for elasticsearch</li>
	<li>Charly Clairmont <a href="http://altic.org/blog/48-ccharly/459-elasticsearch-kibana3-une-nouvelle-approche-de-la-bi">authored an article</a> on taking a new approach to business intelligence using Kibana. (en fran&#xE7;ais)</li>
	<li>The team at XOOM shared their experiences using Elasticsearch <a href="http://dev-blog.xoom.com/2013/12/01/natural-language-and-targeted-search-using-elastic-search/">on their Developer Blog</a>.</li>
	<li>Niels Henrik Hagen authored an article on <a href="http://nhhagen.wordpress.com/2013/11/28/query-log-analysis-using-logstash-elasticsearch-and-kibana/">Query Log Analysis &#x2013; using Logstash, Elasticsearch and Kibana</a>.</li>
	<li>Larry Smith published a post on setting up an <a href="http://everythingshouldbevirtual.com/ubuntu-logstash-server-kibana3-front-end-autoinstall">Ubuntu Logstash Server with Kibana 3 Front End Autoinstall</a>.</li>
	<li>Oliver Eilhard created an <a href="https://github.com/olivere/elastic">elasticsearch client for go</a></li>
	<li>A blog post about <a href="http://found.no/foundation/similarity/">Similarity in Elasticsearch</a> by Konrad G. Beiske</li>
</ul><h3>Slides</h3><ul>
	<li>You can watch <a href="http://twitter.com/spinscale">Alexander Reelsen&apos;s</a> talk on <a href="http://www.developer-conference.eu/session_post/elasticsearch-beyond-full-text-search/">Elasticsearch - beyond full-text search</a> from Developer Conference in Hamburg last month. (auf Deutsch)</li>
	<li><a href="http://twitter.com/ralphm">Ralph Meijer</a> shared his slides from the Elasticsearch <a href="http://www.meetup.com/ElasticSearch-NL/events/150664602/">Netherlands November Meetup</a>, on <a href="http://ralphm.net/publications/elasticsearch_2013/#/">Elasticsearch at Mailgun, by Rackspace</a>.</li>
	<li>Luca Bonmassar from Gild <a href="http://www.youtube.com/watch?v=H81IiLUJavI">gave a presentation</a> on Mastering Elasticsearch with Ruby at RubyConf 2013.</li>
</ul><h3>Where to find Us</h3><h4>France</h4><ul>
	<li>The fourth <a href="http://www.meetup.com/elasticsearchfr/events/138099582/">Elasticsearch France Meetup</a> takes place tonight in Paris from 7:30-9:30 PM. There will be a number of great talks on offer, including <a href="http://twitter.com/jpountz">Adrien Grand</a>, Elasticsearch core developer, on What is a Lucene Index?</li>
	<li><a href="http://twitter.com/dadoonet">David Pilato</a> will speak at the <a href="http://toursjug.cloud.xwiki.com/xwiki/bin/view/Meetings/20131211">Tours Java Users&apos; Group on December 11th</a>. David&apos;s talk will cover all things Elasticsearch, including new features in 1.0. Doors open at 7:15 PM.</li>
</ul><h4>Israel</h4><p>
	The first ever <a href="http://www.meetup.com/Elasticsearch-Tel-Aviv-Meetup/events/152066232/">Elasticsearch Tel Aviv meetup</a> will take place on December 30th, beginning at 7 PM. Join Elasticsearch core developer <a href="http://twitter.com/uboness">Uri Boness</a>, who will present on what&apos;s new in Elasticsearch 1.0, and Itamar Syn-Hershko, who will discuss Monitoring online conversation at scale using Elasticsearch.
</p><h4>Netherlands</h4><p>
	<a href="http://twitter.com/kimchy">Shay Banon</a> will present <a href="http://www.opensourceconference.nl/3736">Why We Built Elasticsearch</a> at the Open Source Conference in Amsterdam on December 6th. Shay&apos;s talk will start at 4:05 PM.
</p><h4>United States</h4><ul>
	<li>The <a href="http://www.meetup.com/Elasticsearch-Boston/events/152284762/">December Elasticsearch Boston Meetup</a> will take place on Wednesday the 4th in Cambridge, Massachusetts. Speakers will cover how Elasticsearch is used at GetHuman and Swoop. Doors open at 6 PM.</li>
	<li>The <a href="http://www.meetup.com/Elasticsearch-Washington-DC/events/152052972/">December Elasticsearch Washington, D.C. Meetup</a> will take place on Tuesday the 10th at NGP VAN&apos;s offices. Craig A. Lebowitz, NGP VAN&apos;s Enterprise Architect, will cover Adding Elasticsearch To An Existing .NET / SQL Server Application. The meeting will begin at 6:30 PM.</li>
</ul><h3>Where to Find You</h3><p>
	Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, <a href="https://docs.google.com/forms/d/1qBZ6Fwq4JWw7FH3s6C9Q3KeGCvUOmWYOvNMsxNqkCNM/viewform">take a moment to let her know</a>.
</p><p>
	Oh yeah, we&apos;re also&#xA0;<a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you&apos;d like us to find you for employment purposes, just&#xA0;<a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &#xA0;We care more about your skill set and passion for Elasticearch, Kibana and Logstash than where you rest your head.
</p><h3>Trainings</h3><p>
	If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
</p><ul>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/paris/2013-12-05">Paris</a>&#xA0;- December 05, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/london-uk/2013-12-09">London</a>&#xA0;- December 12, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/san-francisco/2013-12-19">San Francisco</a>&#xA0;- December 19, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/2014-01-13">Amsterdam</a>&#xA0;- January 13, 2014</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/seattle/2014-01-16">Seattle</a>&#xA0;- January 16, 2014</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/hamburg/2014-01-22">Hamburg</a>&#xA0;- January 22, 2014</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/san-jose/2014-01-23">San Jose</a>&#xA0;- January 23, 2014</li>
</ul></body></html>

This Week in Elasticsearch - December 04, 2013

<p>It's been just over a year since Elasticsearch Inc was born, and about 3.5 years since our CTO, Shay Banon, first released Elasticsearch. From that time to now, we've brought almost 40 developers on board to grow and maintain the ecosystem around Elasticsearch, Logstash and Kibana, and we've seen our software downloaded almost half a million times per month. Just a week ago, we hit the memorable milestone of 5,000,000 downloads, an impressive number by anyone's standard. The fact that nearly 4,000,0000 of those were downloaded over the last 12 months, and that number is growing at a rate of nearly 500,000 a month makes it even more special.
</p>
<p>Elasticsearch, Logstash and Kibana are on fire, baby!
</p>
<p>Members of all startups will consistently tell you how proud they are of their world-changing technology, phenomenal growth and market traction, and we are - of course - no exception. However, we'd like you to hear these things from our customers and our users, not from us.
</p>
<p>To that end, we've had a robust case study program going for the past few months, leading to some pretty amazing stories of how Elasticsearch has helped companies create new revenue streams, improve their customer interaction, create stellar new application functionality and save money. Today, we'd like to share some of these case studies with you and, more importantly, to ask for you to let us know how you're using Elasticsearch.
</p>
<div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/izbGWPRGrxy2PZxpKdLrQn.jpg" data-uuid="izbGWPRGrxy2PZxpKdLrQn" data-v="4" data-type="inline">
</div>
<p><strong>Watch how Elasticsearch powers Loggly's Cloud Based Log Management Service</strong>
</p>
<h2>Elasticsearch Powers Social, ECommerce, Education and Much More…</h2>
<p>We've seen a number of compelling use cases for Elasticsearch across the full spectrum of social companies, including:
</p>
<h3>Klout</h3>
<p>If you're one of the many household names using Klout to create campaigns that target social influencers, you're using Elasticsearch. <a href="/use-cases/klout">Take a look</a> at this case study to find out how Elasticsearch made it possible for Klout to provide their forthcoming self-service option to their customers, which Klout predicts will allow them to at least double their current revenues.
</p>
<h3>XING</h3>
<p>XING is the leading business social network in Europe, with half its users located in Germany and the other half throughout the rest of Europe, Asia and Australia. XING has called their relationship with Elasticsearch a strategic partnership, far beyond a simple customer and service provider relationship. <a href="/use-cases/xing">Learn more</a> about how we've forged these deep ties with our customer by enabling XING to keep their users' updates flowing in real-time.
</p>
<h3>GitHub</h3>
<p>Elasticsearch empowers GitHub's 4 million 'social coders' through providing search across GitHub's 8 million + code repositories. The GitHub team also makes use of Elasticsearch to monitor for abuse using some fairly clever logging hacks.  You can get all the details in our <a href="/use-cases/github">GitHub case study</a>.
</p>
<p><strong>See how Elasticsearch Enables Stack Overflow's Users to Find the Answers They Need</strong>
</p>
<div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/P1DjhRq3QpED1u4LaehKrX.jpg" data-uuid="P1DjhRq3QpED1u4LaehKrX" data-v="4" data-type="inline">
</div>
<h3>Focusmatic</h3>
<p>Focusmatic provides its customers with brand insights from across all aspects of the social web: blogs and micro-blogging sites, video content, news outlets and social networks. Thanks to Elasticsearch, Focusmatic was able to scale to meet the needs of their current customers and, more significantly, be prepared to meet the needs of many more new customers, all at 75% less cost. <a href="/use-cases/focusmatic">Read all about</a> how Elasticsearch made it possible for Focusmatic to “continue their business."
</p>
<h3>Cogenta</h3>
<p>Cogenta provides retail market data to high-end brand retailers in Europe, with plans currently underway to expand into 20 countries worldwide. Their real-time pricing intelligence platform tracks and analyzes competitive data on more than 20 million products. <a href="/use-cases/cogenta">Take a look</a> at how Elasticsearch helped Cogenta prepare for this customer base expansion at a 40% lower total cost of ownership.
</p>
<h3>Datadog</h3>
<p>Datadog is a SaaS monitoring service startup for IT, operations and development teams that enables these teams to better analyze metrics and events, ensuring their operations continue smoothly. By using Elasticsearch, Datadog was able to scale to take on larger customers who produce 500x more events than Datadog could originally handle. You can <a href="/use-cases/data-dog">learn all about</a> how Elasticsearch helped Datadog increase their event handling capacity by 11x.
</p>
<h2>How Are You Using Elasticsearch?</h2>
<p>We are always excited to hear from anyone using Elasticsearch: we want to know why you chose Elasticsearch, plus how it has made life easier for you and better for your customers. <a href="http://twitter.com/elasticsearch">Drop us a line</a> anytime and share your story.
</p>

How the World is Using Elasticsearch

<html><head></head><body><p>Today we are delighted to announce the release of <b>elasticsearch 1.0.0.Beta2</b>, the second beta release on the road to 1.0.0 GA. The new features we have planned for 1.0.0 have come together more quickly than we expected, and this beta release is chock full of shiny new toys. Christmas has come early!</p>
<p>We have added:</p>
<ul type="disc">
<li>
the <a href="#snapshot" title="Snapshot / Restore">snapshot/restore API</a>
</li>
<li>
the <a href="#cat" title="cat API">cat API</a>
</li>
<li>
the new <a href="#aggs" title="Aggregations">aggregations</a> framework
</li>
</ul>
<p>Please <a href="/downloads/past-releases/elasticsearch-1-0-0-beta2">download elasticsearch 1.0.0.Beta2</a>, try it out, break it, figure out what is missing and <a href="https://github.com/elasticsearch/elasticsearch/issues">tell us about it</a>. Our next release will focus on cleaning up inconsistent APIs and usability, plus fixing any bugs that are reported in the new functionality, so your early bug reports are an important part of ensuring that 1.0.0 GA is solid.</p>
<p><b>WARNING:</b> This is a beta release &#x2013; it is not production ready, features are not set in stone and may well change in the next version, and once you have made any changes to your data with this release, it will no longer be readable by older versions!</p>
<h2><a id="snapshot" href="#snapshot"></a>Snapshot / Restore</h2>
<p>While it has always been possible to backup a live index on Elasticsearch, the process was a bit finicky.  The long awaited snapshot/restore API makes it easy.</p>
<p>First, define a <code>repository</code>&#x2009;&#x2014;&#x2009;the place where your backups will live:</p>
<pre class="prettyprint lang=javascript">curl -XPUT &apos;http://localhost:9200/_snapshot/my_backup&apos; -d &apos;{
    &quot;type&quot;: &quot;fs&quot;,
    &quot;settings&quot;: {
        &quot;location&quot;: &quot;/mount/backups/my_backup&quot;
    }
}&apos;</pre>
<p>Currently we support the shared filesystem (<code>fs</code>) repository, which needs to be writable by all nodes.  In the future we will add support for S3, HDFS, GlusterFS, Google Compute Engine and Microsoft Azure.</p>
<p>With the repository in place, you can tell Elasticsearch to create a snapshot named <code>snapshot_1</code> with:</p>
<pre class="prettyprint lang=javascript">curl -XPUT localhost:9200/_snapshot/my_backup/snapshot_1</pre>
<p>You can snapshot the whole cluster or just specific indices.  The best part is that <strong>snapshots are incremental</strong>&#x2009;&#x2014;&#x2009;it only copies the segments that have changed since the last snapshot was made.  This makes the snapshotting process faster and lighter (you can snapshot every 5 minutes if you want to) and uses up less storage in the repository.</p>
<p>You can restore the whole cluster, with or without persistent cluster settings, or just individual indices:</p>
<pre class="prettyprint lang=javascript">curl -XPOST localhost:9200/_snapshot/my_backup/snapshot_1/_restore</pre>
<p>Later on we plan on making cross data-center replication possible by adding the ability to do incremental restores into a read-only index.</p>
<p>You can <a href="/guide/en/elasticsearch/reference/master/modules-snapshots.html">read more about snapshot/restore in the reference docs</a>.</p>
<h2><a id="cat" href="#cat"></a><code>cat</code> API</h2>
<p>JSON is a great&#x2026; for computers.  But at 3 AM when you&#x2019;re trying to figure out what is happening in your cluster, we humans prefer a simpler text format, easier to read and use with command line tools like <code>sort</code> and <code>awk</code>.  The new <code>cat</code> API is the system administrator&#x2019;s friend: &#x2009;&#x2014;&#x2009;it formats the results in a columnar fashion for easy reading and parsing.</p>
<p>For instance, to find out which indices are status <code>yellow</code>:</p>
<pre class="prettyprint lang=sh">curl localhost:9200/_cat/indices | grep ^yell
yellow foo          5 1   4 0    17kb    17kb</pre>
<p>Column headers are not returned by default, but you can request them by adding the <code>v</code> parameter to the query string:</p>
<pre class="prettyprint lang=sh">curl localhost:9200/_cat/recovery?v
index shard   target recovered     % ip            node
wiki1 2     68083830   7865837 11.6% 192.168.56.20 Adam II
wiki2 1      2542400    444175 17.5% 192.168.56.20 Adam II</pre>
<p>Above we can see that two shards are recovering after a node failure, and that they are 12% and 18% complete, respectively.</p>
<p>There are <code>cat</code> endpoints for many admin APIs, including allocation, indices, nodes, and shards.  See the <a href="/guide/en/elasticsearch/reference/master/cat.html"><code>cat</code> reference docs</a> for details.</p>
<h2><a id="aggs" href="#aggs"></a>Aggregations</h2>
<p>Aggregations are &#x201C;facets reborn&#x201D;.  Facets are amazingly powerful&#x2009;&#x2014;&#x2009;they allow you to summarise vast amounts of data in the context of a user&#x2019;s query on the fly, without the need for slow batch precalculations.  However, the way they are implemented limits them in two important ways:</p>
<ul type="disc">
<li>
You can&#x2019;t combine two facet types without writing new code.  For instance, to generate statistics on popular terms, we had to write a new <code>terms_stats</code> facet instead of just being able to plug the <code>statistical</code> facet into the <code>terms</code> facet
</li>
<li>
They are one level deep only.  You can ask for &#x201C;counts of popular hashtags&#x201D;, or &#x201C;counts of tweets per day&#x201D;, but you can&#x2019;t ask for &#x201C;counts of popular hashtags per day&#x201D;.
</li>
</ul>
<p>Aggregations change all this.</p>
<p>There are two types of aggregation (or <code>aggs</code>): <strong><code>bucket</code></strong> aggregators which allow you to divide documents up into separate buckets eg <code>terms</code>, <code>range</code>, <code>histogram</code>, <code>date_histogram</code>, <code>filter</code>, etc and <strong><code>metric</code></strong> aggregators which perform some calculation on the documents in each bucket, eg <code>count</code>, <code>avg</code>, <code>stats</code> etc.</p>
<p>Buckets can be sub-divided into smaller buckets, and metrics can be calculated for any bucket at any level.  In the following example, we run a query looking for all tweets which mention <code>elasticsearch</code>, count the most popular hashtags overall, count the number of tweets per day <strong>and</strong>  calculate the most popular hashtags per day:</p>
<pre class="prettyprint lang=javascript">GET /tweets/_search
{
    &quot;query&quot;: {
        &quot;match&quot;: { &quot;tweet&quot;: &quot;elasticsearch&quot; }
    },
    &quot;aggs&quot;: {
        &quot;popular_hashtags_overall&quot;: {
            &quot;terms&quot;: { &quot;field&quot;: &quot;hashtags&quot; }
        },
        &quot;per_day&quot;: {
            &quot;date_histogram&quot;: {
                &quot;field&quot;:    &quot;date&quot;,
                &quot;interval&quot;: &quot;day&quot;
            },
            &quot;aggs&quot;: {
                &quot;popular_hashtags&quot;: {
                    &quot;terms&quot;: { &quot;field&quot;: &quot;hashtags&quot; }
                }
            }
        }
    }
}</pre>
<p>Doing this with facets would have been unthinkable&#x2009;&#x2014;&#x2009;it would have required a separate query for every day!</p>
<p>Now that this framework is in place, it makes it much easier to add new aggregations.  And as soon as a new aggregation has been implemented, it can used immediately in combination with any of the other aggregations.</p>
<p>Facets are not going away just yet.  We want to give you time to migrate from facets to aggregations so both implementations will be available for the foreseeable future. Also facets have had many micro-optimizations added over the years to make them perform as well as they do.   Aggregations are not yet as fast for the simple case, but we are working hard to squeeze more performance out of them.</p>
<p>You can <a href="/guide/en/elasticsearch/reference/master/search-aggregations.html">read more about aggregations in the reference docs</a>.</p>
<h2><a id="_still_to_come" href="#_still_to_come"></a>Still to come</h2>
<p>Before the next release, we are going to be looking at tidying up inconsistencies and warts that have crept into our API.  We want to make it clean, consistent, and as Do-What-I-Mean as possible.  You can take a look at the issues we are considering <a href="https://github.com/elasticsearch/elasticsearch/issues?labels=v1.0.0.RC1&amp;state=open">here</a>.  We welcome further discussion.</p>
<p>Please download <a href="/downloads/past-releases/elasticsearch-1-0-0-beta2">Elasticsearch 1.0.0.Beta2</a>, try it out, and <a href="http://twitter.com/elasticsearch">let us know what you think</a>.</p></body></html>

1.0.0.Beta2 Released

<html><head></head><body><p>Elasticsearch is not just about full-text search, and many users are actually not using Elasticsearch for full-text search at all but for analytics though facets. This approach works well, but, as you probably know, faceting or sorting on a field requires loading field values into in-memory data structures that we call <em>field data</em>. It is very common that <em>field data</em> takes several (tens of) gigabytes of memory. Memory is rather cheap, so it is usually not a problem to get boxes with enough memory. However, this can raise issues at the JVM level: major garbage collections on a heap of several tens of gigabytes can easily take several seconds during which your application will be unresponsive. Careful JVM tuning can help prevent this issue, but ideally field data should be stored <em>off-heap</em>.</p><h2>Doc Values to the Rescue</h2><p><em>Doc values</em> is a feature that will be available in the forthcoming Elasticsearch 1.0 release. You can already check it out in our <a href="http://www.elasticsearch.org/downloads/1-0-0-Beta1/">1.0 beta release</a>.  </p><p><em>Doc values</em> are a Lucene 4.x feature which allow for storing field values on disk in a column stride fashion, which is filesystem cache friendly and suitable for custom scoring, sorting or faceting, exactly like field data. It was only natural to build a new field data backend based on <em>doc values</em>, and this new implementation has several benefits compared to the traditional field data implementations that Elasticsearch builds by uninverting the inverted index:</p><ul>
<li>Memory is managed at the OS level through the filesystem cache instead of the JVM, just like the rest of the index.</li>
<li>Since <em>doc values</em> data structures are computed at indexing time, <em>_refresh</em> will be faster.</li>
<li>The way <em>doc values</em> are computed provides more efficient compression.</li>
</ul><p>On the other hand, <em>doc values</em> are going to make indices bigger &#x2014; unless it allows for not indexing the field, eg. if the field is used solely for sorting &#x2014; and <em>field data</em> intensive work loads such as faceting will be slightly slower.</p><h2>When Should I Use Doc Values?</h2><p><em>Doc values</em> can be used as a drop-in replacement for uninverted <em>field data</em> in most cases, but there are a few cases where they can be particularly helpful:</p><ul>
<li><b>Constrained hardware</b>: The in-memory field data data structures take up a lot of space, and sometimes you just can&apos;t add more and more RAM to your nodes in order to sustain the growth of your field data.</li>
<li><b>Daily jobs</b>: Some users have queries which are run daily in order to get point-in-time aggregated views on their data. In those cases, the most important thing is not query latency or throughput, rather to not take all the resources that your cluster may need for other work loads. <em>Doc values</em> can help here since they won&apos;t fill up memory with data structures which won&apos;t be used again in the future.</li>
<li><b>Memory management</b>: Having the Elasticsearch data structure that takes the most memory moved to disk means that it is now possible to start an Elasticsearch process with only 1G of memory, and to let the OS handle memory through the file system cache with almost no risk of out-of-memory error. Another nice aspect to this type of setup is that there are usually many less garbage collection issues on heaps of 1G as compared to heaps of tens of gigabytes.</li>
</ul><h2>How to Enable Doc Values</h2><p><em>Doc values</em> are an index time decision, so they need to be enabled in the mappings before indexing the first document. Here is an example of a string field definition that can be used for sorting and faceting, but not searching:</p><pre class="prettyprint noescape lang=javascript">&quot;my_field&quot;: {
  &quot;type&quot;: &quot;string&quot;,
  &quot;index&quot;: &quot;no&quot;,
  &quot;fielddata&quot;: {
    &quot;format&quot;: &quot;doc_values&quot;
  }
}
</pre><p>For now, doc values are supported on non-analyzed string fields and numeric fields (byte, short, integer, long, float, double, date).</p><p>As you can see, fields don&apos;t need to be indexed to enable <em>doc values</em>. And once <em>doc values</em> are enabled, all operations working on top of <em>field data</em> like sorting or faceting will transparently use <em>doc values</em> under the hood.</p><p>We would love to get your feedback on <em>doc values</em>. Try out our latest beta release and <a href="http://twitter.com/elasticsearch">tell us about your experience</a>!</p></body></html>

Disk-Based Field Data a.k.a. Doc Values

<p>
	Welcome to&nbsp;
	<i>This week in Elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
</p><h2>Elasticsearch core</h2><ul>
	<li>Aggregations have landed in master! (<a href="https://github.com/elasticsearch/elasticsearch/issues/3300">#3300</a>, master)</li>
	<li>Update to Lucene 4.6.0 (<a href="https://github.com/elasticsearch/elasticsearch/issues/4241">#4241</a>, 0.90 & master)</li>
	<li>Importing and running the elasticsearch code base into eclipse is now easier<code></code> (<a href="https://github.com/elasticsearch/elasticsearch/commit/76c5f53dfa1ae510e76ada99e521a1e6fc943d25">commit</a>, 0.90 & master)</li>
	<li>Library upgrades: forbiddenapis 1.4, randomized test runner maven plugin (commits <a href="https://github.com/elasticsearch/elasticsearch/commit/a949a8056b4eefb6e0dcf20df0c76ac559474a43">1</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/eb55458e44d26be44c1e73d27ecbc79f010fe949">2</a>, 0.90 & master)</li>
	<li>Snapshot/Restore: Prevent modification or deletion of repositories during snapshot <code></code> (<a href="https://github.com/elasticsearch/elasticsearch/commit/1d0bae0f507915e4aa60d87fdb95e6944450bd9b">commit</a>, master)</li>
	<li>Added <code>mlockall</code> setting to process info output (<a href="https://github.com/elasticsearch/elasticsearch/issues/">#4233</a>, 0.90 & master)</li>
	<li>During node startup local primaries are now preferred to relocating primaries (<a href="https://github.com/elasticsearch/elasticsearch/issues/4237">#4237</a>, 0.90 & master)</li>
	<li><code>RobinEngine.acquireSearcher()</code> should fail with state failure when not set/closed (<a href="https://github.com/elasticsearch/elasticsearch/issues/4232">#4232</a>, 0.90 & master)</li>
	<li>Bulk requests support shard timeout like the index api (<a href="https://github.com/elasticsearch/elasticsearch/issues/4220">#4220</a>, 0.90 & master)</li>
	<li>Deprecate <code>numeric_range</code> filter, add a field data execution option to range filter (<a href="https://github.com/elasticsearch/elasticsearch/issues/4034">#4034</a>, 0.90 & master)</li>
	<li>Ongoing effort to reduce excessive object allocation due to copying maps around by switching to <code>ImmutableOpenMap</code> (commits <a href="https://github.com/elasticsearch/elasticsearch/commit/036febe110f0ea87d96bfe2dce71f97469d5f317">1</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/9f5d01ca4cc06bf7affdb5a656dc838c383e0c4c">2</a>, 0.90 & master)</li>
	<li>Allow to parse directly the compressed mapping (<a href="https://github.com/elasticsearch/elasticsearch/commit/6f90a3e39ae59fcb91334a99e3881198d6b53e65">commit</a>, 0.90 & master)</li>
	<li>The completion suggester code has been updated to the current lucene release (<a href="https://github.com/elasticsearch/elasticsearch/commit/bf74f49fdd79061ae55bc60175d005ac1b8e0edb">commit</a>, 0.90 & master)</li>
	<li>Cat API: Add <code>h</code> parameter to apis, allowing to return columns and descriptions for them (<a href="https://github.com/elasticsearch/elasticsearch/issues/">#4262</a>, master only)</li>
	<li>MultiTermVector API and TermVector API were unified (<a href="https://github.com/elasticsearch/elasticsearch/commit/3be5f3345edbe9868b073a7653cd67973c3fbf1e">commit</a>, master only)</li>
	<li>Multi term vector request never returned if shards for all requested documents were non existent. (<a href="https://github.com/elasticsearch/elasticsearch/issues/3989">#3989</a>, master)</li>
	<li>Fixed file-based template loading via <code>config/templates</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/4235">#4235</a>, 0.90 & master only)</li>
</ul><h3>Elasticsearch Ecosystem</h3><p>
	Here's some more information about what is happening in the ecosystem we are maintaining around Elasticsearch, including plugin and driver releases, as well as news about Logstash and Kibana.
</p><ul>
	<li>Lisa Sensmeier with <a href="http://elasticsearch.com/about/partners/">Hortonworks, a partner of Elasticsearch Inc</a>, penned a post on <a href="http://hortonworks.com/blog/fast-search-and-analytics-on-hadoop-with-elasticsearch-and-hdp/">Fast Search and Analytics on Hadoop with Elasticsearch</a>.</li>
	<li>Ben Hundley with <a href="http://elasticsearch.com/about/partners/">StackSearch, also an Elasticsearch partner</a>, shared a blog post <a href="http://blog.qbox.io/launching-and-scaling-elasticsearch">Launching and Scaling Elasticsearch</a>.</li>
	<li>Daniel Guo and other volunteers in China have created new resources for users who speak Chinese, including the start of translations of the Elasticsearch website and a wiki for users to help each other use Elasticsearch. You can find out more on the <a href="http://www.elasticsearch.org/community/">Elasticsearch community page</a>.</li>
	<li>Sarath Pillai created a <a href="http://slashroot.in/logstash-tutorial-linux-central-logging-server">tutorial on Logstash</a>.</li>
	<li>Florian Hopf shared a how to artcile on <a href="http://blog.florian-hopf.de/2013/11/reindexing-content-in-elasticsearch.html">Reindexing Content in Elasticsearch</a>.</li>
	<li>Doug McClure shared his experiences <a href="http://dougmcclure.net/blog/2013/11/event-analysis-using-smartcloud-analytics-log-analysis-scala-v1103-deploying-logstash/">performing event analysis using Logstash and SCALA</a>.</li>
	<li>Vojtěch Hýža released a plugin for the <a href="https://bitbucket.org/hlavki/jlemmagen">JLemmaGen</a> project <a href="https://github.com/vhyza/elasticsearch-analysis-lemmagen">on github</a>, which supports dictionary based lemmatization for 15 languages.</li>
	<li>Terry Patterson wrote up his experiences using Elasticsearch, Logstash and Kibana in conjunction with his <a href="http://www.blackboardguru.com/2013/11/how-logstash-and-kibana-empower-blackboard-learn-logs/">Blackboard installation</a>.</li>
	<li>The <a href="https://github.com/elasticsearch/elasticsearch-metrics-reporter-java">metrics-elasticsearch-reporter</a>, which reports data generated by the <a href="http://metrics.codahale.com/">metrics library</a>, has been released as version 1.0 to maven central.</li>
	<li><a href="http://twitter.com/mvgroningen">Martijn van Groningen</a> wrote about the new <a href="http://www.elasticsearch.org/blog/percolator-redesign-blog-post/">distributed percolator</a> coming in elasticsearch 1.0 - which you can already test with the current beta version.</li>
	<li>Jelle Smet created a how to on <a href="http://smetj.net/submit-elasticsearch-metrics-to-graphite.html">submitting Elasticsearch metrics to Graphite</a>.</li>
	<li>Alex Brasetvik with Found shared more <a href="http://www.found.no/foundation/presenting-play/">information on Play</a> as service that allows users to test drive features of Elasticsearch.</li>
	<li>Rossi Oddet wrote up a great summary of <a href="http://twitter.com/dadoonet">David Pilato's</a> presentation, Elasticsearch: <a href="http://blog.roddet.com/2013/11/nantesjug-novembre-part2/">From SQL to NoSQL in Less than an Hour, at the November Nantes JUG meeting</a>. (in french)</li>
	<li>The <a href="https://github.com/elasticsearch/elasticsearch-cloud-aws">AWS cloud plugin</a> was released in version 1.16.0.</li>
	<li>The <a href="https://github.com/elasticsearch/elasticsearch-river-wikipedia">wikipedia river</a> was released in version 1.3.0.</li>
</ul><h3>Slides</h3><ul>
	<li>Tomas Jansson <a href="http://www.slideshare.net/mastoj/getting-started-with-elasticsearch-and-net">shared the slides</a> from his presentation Getting Started with Elasticsearch on Windows and .Net with NEST.</li>
	<li>Gérald Croes and Julien Salleyron posted the slides from their presentation <a href="http://www.croes.org/gerald/conf/php/paris/2013/ElasticSearch.pdf">Look no further than Elasticsearch</a> as presented at the <a href="http://www.afup.org/pages/forumphp2013/sessions.php">Forum PHP Paris 2013</a> conference. (in french)</li>
</ul><h3>Where to find Us</h3><h4>France</h4><p>
	David Pilato will speak at Drupagora on December 5th, covering 
	<a href="http://www.drupagora.com/2013/programme-drupagora#Présentation des solutions de recherche">search solutions for Drupal using Elasticsearch</a>. David's talk will begin at 3 PM.
</p><h4>Netherlands</h4><p>
	<a href="http://twitter.com/kimchy">Shay Banon</a> will present <a href="http://www.opensourceconference.nl/3736">Why We Built Elasticsearch</a> at the Open Source Conference in Amsterdam on December 6th. Shay's talk will start at 4:05 PM.
</p><h4>Spain</h4><p>
	<a href="http://twitter.com/clintongormley">Clinton Gormley</a> will speak at the NoSQL matters conference in Barcelona on <a href="http://2013.nosql-matters.org/bcn/abstracts/#abstract_287512263">Getting Down and Dirty with Elasticsearch</a>. Clinton's talk takes place on Saturday, November 30th at 10:30 AM, and the conference runs November 29-30th.
</p><h4>United Kingdom</h4><ul>
	<li><a href="http://twitter.com/s1m0nw">Simon Willnauer</a> will present <a href="http://skillsmatter.com/podcast/nosql/with-a-hammer-in-your-hand-elasticsearch">Elasticsearch: With a Hammer in Your Hand</a> at the NOSQL eXchange in London on November 28th. Simon's presentation begins at 1 PM.</li>
	<li>Simon will also speak at the <a href="http://codemesh.io/">CodeMesh conference</a> from the 4th to the 5th of December.</li>
</ul><h4>United States</h4><ul>
	<li><a href="http://twitter.com/dakrone">Lee Hinman</a> will discuss all the <a href="http://www.meetup.com/Elasticsearch-Denver/events/149926022/">elasticsearch 1.0 features at the Denver meetup</a> on December 3rd</li>
	<li>The <a href="http://www.meetup.com/Elasticsearch-Boston/events/152284762/">December Elasticsearch Boston Meetup</a> will take place on Wednesday the 4th in Cambridge, Massachusetts. Speakers will cover how Elasticsearch is used at GetHuman and Swoop. Doors open at 6 PM.</li>
	<li>The <a href="http://www.meetup.com/Elasticsearch-Washington-DC/events/152052972/">December Elasticsearch Washington, D.C. Meetup</a> will take place on Tuesday the 10th at NGP VAN's offices. We'll be sharing more talk details soon, but a draft agenda is available now and registration is open. The meeting will begin at 6:30 PM.</li>
</ul><h3>Where to Find You</h3><p>
	Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, 
	<a href="https://docs.google.com/forms/d/1qBZ6Fwq4JWw7FH3s6C9Q3KeGCvUOmWYOvNMsxNqkCNM/viewform">take a moment to let her know</a>.
</p><p>
	Oh yeah, we're also&nbsp;
	<a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you'd like us to find you for employment purposes, just&nbsp;<a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &nbsp;We care more about your skill set and passion for Elasticearch, Kibana and Logstash than where you rest your head.
</p><h3>Trainings</h3><p>
	If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
</p><ul>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/paris/2013-12-05">Paris</a>&nbsp;- December 05, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/london-uk/2013-12-09">London</a>&nbsp;- December 12, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/los-angeles/2013-12-16">Los Angeles</a>&nbsp;- December 16, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/2014-01-13">Amsterdam</a>&nbsp;- January 13, 2014</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/seattle/2014-01-16">Seattle</a>&nbsp;- January 16, 2014</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/hamburg/2014-01-22">Hamburg</a>&nbsp;- January 22, 2014</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/san-jose/2014-01-23">San Jose</a>&nbsp;- January 23, 2014</li>
</ul>

This week in elasticsearch - November 27, 2013

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><h2>A Brief Introduction to the Similarity Models Available</h2><p>Elasticsearch now supports replacement of the default similarity model. In this article we will look into what a similarity model is with a special focus on tf-idf and bm25.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>A similarity model is a set of abstractions and metrics to define to what extent things are similar. That&#x2019;s quite a general definition. In this article I will only consider textual similarity. In this context, the uses of similarity models can be divided into two categories: classification of documents, with a finite set of categories where the categories are known; and information retrieval where the problem can be defined as &#x2018;find the the most relevant documents to a given query&#x2019;. In this article I will look into the latter category.
</p>
<p>Elasticsearch provides the following similarity models: default, bm25, drf and ib. I have limited the scope of this article to default and bm25. The divergence from randomness and information based similarities may feature in a future article.
</p>
</section><section id="default-similarity" class="level2">
<h2><a href="#default-similarity">Default Similarity</a></h2>
<p>The default similarity model in Elasticsearch is an implementation of <strong>tf/idf</strong>. Tf/idf is the most common vector space model. A vector space model is a model where each term of the query is considered a vector dimension. This allows for defining one vector for the query and another for the document considered. The scalar product of the two vectors is then considered the relevance of the document to the query. This implies that the positions of the words within a document are not used - a document is just a bag of words.
</p>
<p>A simple vector space model would be to set each coordinate in the document vector to one if the document has the word for the dimension, otherwise set it to zero. The query vector would be all one&#x2019;s if each word has the same weight. In such a model the scalar product would then be the sum of the words which are common for the document and the query. However, such a model would rank a document referring once to a query term in a side note the same as a document using the term repeatedly throughout the text. Which of the the two documents&#x2019; topics is identical to the term? The tf part in tf/idf helps us resolve this: tf stands for term frequency, and the solution is to use the term frequency instead of a simple zero or one in the document vector.
</p>
<p>In its simplest form you could calculate the term frequency by counting the number of occurences of the term in the document. This simplistic approach has one inherent problem. Assuming a query with the terms: <em>fire fox</em> generates the document vectors D1{15, 0} and D2{5, 5}. With a query vector of Q{1, 1}, document D1 will have a score of 15 and document D2 will have a score of 10. Let&#x2019;s say D1 is a lengthy document about fire hazards and D2 is a tutorial on how to install the browser Firefox. Which document do you think corresponds best to the query? To get the best of both worlds, by rewarding documents for specifically being about a term and also being about all the terms, tf-idf uses the logarithm of the term frequency for the vector value. Those who remember their calculus know that the logarithm of zero is infinitely negative while zero is the logarithm to one. To get the desired behaviour tf-idf simply adds one to all term frequencies before taking the logarithm. Using natural logarithms the document vectors become D1{2.772588722239781, 0} and D2{1.791759469228055, 1.791759469228055} with ranks 2.772588722239781 and 3.58351893845611 respectively. We now have a model that values having all terms without disregarding that a document can be more about a term than another.
</p>
<p>The other part of tf-idf yet to be mentioned is IDF, the Inverse Document Frequency. The definition of idf is the total number of documents in the collection (or index) divided by the number of documents that contain the word. It is used to address the fact that some words are more common in the language than others. In fact some words are that common they are considered noise. If a word is included in every document of a collection, how can it be helpful in choosing documents? Why don&#x2019;t we use stop words you say? Well, Elasticsearch does use stop words, however using stop words can prove to be inaccurate and of course, stop words are not only language specific, but also domain specific. IDF is a measure of the significance of a word&#x2019;s occurrence in a document. Similarly to TF, it is common to use the logarithm of IDF. The base number of the logarithm is not important, the crux of the matter is that it requires an exponential increase in the frequency to increase the IDF.
</p>
<p>As division by zero is somewhat problematic, it&#x2019;s recommended to add 1 to the denominator of the fraction. To avoid a negative IDF it is then common to add a +1 to the idf outside of the logarithm as in Lucene&#x2019;s <a href="http://lucene.apache.org/core/4_5_1/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html">TFIDFSimilarity</a>.
</p>
</section><section id="bm25" class="level2">
<h2><a href="#bm25">BM25</a></h2>
<p>BM25 belongs to the probabilistic models while TF-IDF is a vector space model, but their formulas are not as different as you might expect. Both models define a weight for each term as a product of some idf-function and some tf-function and then summarize that term weight as the score for the whole document towards the given query.
</p>
<p><span class="math">$\mathrm{Score}_{i} = A(tf_{i}) * B(idf_{i})$</span>
</p>
<p>For those who would like a thorough summary of the theoretical basis of BM25 I recommend <a href="http://staff.city.ac.uk/~sb317/papers/foundations_bm25_review.pdf">The Probabilistic Relevance Framework: BM25 and Beyond</a>. In this article I will rather try to explain the practical difference.
</p>
<section id="saturation" class="level3">
<h3><a href="#saturation">Saturation</a></h3>
<p>Both tf-idf and bm25 acknowledge that for highly frequent terms a further increase in term frequency has little significance for the relevance. The difference between them is that BM25 takes this a bit further. The saturation function of BM25 asymptotically approaches a limit for high term frequencies, while the logarithms of tf-idf has no boundary. For tf-idf, multiplying the the term frequency by the base of the logarithm, usually <span class="math">$e \approx 2,72$</span>, will always increase the logarithm by one.
</p>
<p>This difference in growth implies that for any fixed set of tuning parameters, the base of the logarithm included, you can create a document that will get a higher relative increase in term frequency score in tf-idf than in bm25.
</p>
<p>The following functions describe how term frequency contributes to score in BM25 and tf-idf:
</p>
<ul>
	<li><span class="math">$\mathrm{Saturation}_{BM25}(tf_{i}) = \frac{tf_{i}}{k_{1} + tf_{i}}$</span></li>
	<li><span class="math">$\mathrm{Saturation}_{tf-idf}(tf_{i}) = \mathrm{ln}\left(tf_{i} + 1\right)$</span></li>
</ul>
<p>If we plot these functions for typical values of k1 and term frequency from 0 to 100 we get:
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4c8c3dc542cb38c08256/download?uid=blt9b2d2d963d3bc6c6?uid=blt9b2d2d963d3bc6c6" data-sys-asset-uid="blt3aee6ed9aae8e413" style="max-width:100%;" alt="Saturation"><figcaption>Saturation</figcaption>
</figure>
<p>When reading this chart it&#x2019;s important to note that the absolute height of each graph is irrelevant as it could easily be adjusted with a simple boost factor. What is interesting is the relative growth of each curve. There are two things to note from this graph. Firstly, it supports the fact that the logarithm has no upper boundary and secondly, the differences between <span class="math">$k_{1}$</span> in the range from 1,2 to 2,0 are subtle. The latter observation has the practical consequence that if you don&#x2019;t have any idea what to choose for <em><span class="math">$k_{1}$</span></em>, the default value will most likely not be too far off.
</p>
</section>
<section id="average-document-length" class="level3">
<h3><a href="#average-document-length">Average Document Length</a></h3>
<p>The second major difference between tf-idf and BM25 is the use of document length in BM25. BM25 uses document length to compensate for the fact that a longer document in general has more words and thus is more likely to have a higher term frequency without necessarily being more pertinent to the term and thus no more relevant to the query. But it&#x2019;s not all black and white, some documents actually have a wider scope which makes the longer text justified. BM25 adjusts the term factor by a factor <span class="math">$B$</span>, which is calculated from a tuning parameter <span class="math">$b$</span> <span class="math">$\left(0 \leq b \leq 1\right)$</span>, the document length <span class="math">$dl$</span> and the average document length <span class="math">$avdl$</span>, like this:
</p>
<p><span class="math">$B = \left((1-b)+b\frac{dl}{avdl}\right)$</span>
</p>
<p>By defining <span class="math">$tf_{adjusted} = tf * B$</span> and substituting <span class="math">$tf$</span> for <span class="math">$tf_{adjusted}$</span> in previous weight defintion we get:
</p>
<p><span class="math">$W_{i}^{TF} = \frac{tf_{i}}{tf_{i} + k1\left(\left(1 - b\right) + b\frac{dl}{avgdl}\right)}$</span>
</p>
<p>This is how term frequency is used in BM25.
</p>
</section>
<section id="the-complete-term-weight" class="level3">
<h3><a href="#the-complete-term-weight">The Complete Term Weight</a></h3>
<p>To get the full weight for a term in BM25 we need to multiply <span class="math">$W_{i}^{TF}$</span> with <span class="math">$IDF_{i}$</span>. However BM25 defines idf slightly differently than the default similarity and this difference is reflected in Lucene&#x2019;s implementation. The definition used is:
</p>
<p><span class="math">$IDF_{i} = \mathrm{log}\left(1 + \frac{D - d_{i} + 0.5}{d_{i} + 0.5}\right)$</span>
</p>
<p>Where <span class="math">$D$</span> is the total number of documents int the collection and <span class="math">$d_{i}$</span> is the number documents containing the i&#x2019;th term of the query.
</p>
</section>
<section id="how-lucene-does-bm25" class="level3">
<h3><a href="#how-lucene-does-bm25">How Lucene Does BM25</a></h3>
<p>Lucene uses a variation of BM25 where the numerator of the fraction is multiplied with <span class="math">$\left(k_{1} + 1\right)$</span>. That&#x2019;s a common trick to make the formula more similar to other ranking functions and hence also easier to compare the resulting values. As this expression is a constant, it has not consequence for the ranking produced.
</p>
<p><span class="math">$W_{i}^{\mathrm{Lucene BM25}} = idf_{i} * boost * \frac{(k_{1} + 1)tf_{i}}{tf_{i} + k_{1}\left(\left(1 - b\right) + b\frac{dl}{avgdl}\right)}$</span>
</p>
<p>As Lucene aims for performance, the formula is not calculated entirely at query time. Lucene uses the similarity model chosen both while indexing and querying. This reflects on Elasticsearch as well.
</p>
</section>
<section id="tuning-bm25" class="level3">
<h3><a href="#tuning-bm25">Tuning BM25</a></h3>
<p>Regarding the tuning parameters, <span class="math">$k_{1}$</span> and <span class="math">$b$</span> there is a reason they are defined as parameters. The ideal value is dependent on the document collection. Unfortunately, there is no silver bullet and trial and error is the way to go, but it does help to have an idea of which documents are relevant to the queries used and to have a methodological approach.
</p>
</section>
</section><section id="conclusion" class="level2">
<h2><a href="#conclusion">Conclusion</a></h2>
<p>There is a reason why TF-IDF is as widespread as it is. It is conceptually easy to understand and implement while also performing pretty well. That said, there are other, strong candidates. Typically, they offer more tuning flexiblity. In this article we have delved into one of them, BM25. In general, it is known to perform just as good or even better than TF-IDF, especially on collections with short documents. Bearing that in mind, don&#x2019;t forget that similarity rank is not the only ranking contributor in Elasticsearch. To create a good search experience it is key to combine textual similarity rank with metadata suiting the given case, such as the last time document was updated or if there is some proximity between the authors of the query and the document. Read our follow-up article where we compare the precision and recall of the two models using Wikipedia articles: <a href="/blog/found-bm-vs-lucene-default-similarity">BM25 vs Lucene Default Similarity</a>.
</p>
</section></body></html>

Similarity in Elasticsearch

<html><head></head><body><p>The percolator is essentially search in reverse, which can by confusing initially for many people. This post will help to solve that problem and give more information on the redesigned percolator. We have added a lot more features to it to help users work with percolated documents/queries more easily.</p>

<p>In normal search systems, you store your data as documents and then send your questions as queries. The search results are a list of documents that matched your query.</p>

<p>With the percolator, this is reversed. First, you store the queries and then you send your &apos;questions&apos; as documents. The percolator results are a list of queries that matched the document.</p>

<p>So what can do percolator do for you? The percolator can be used for a number of use cases, but the most common is for alerting and monitoring. By registering queries in Elasticsearch, your data can be monitored in real-time. If data with certain properties is being indexed, the percolator can tell you what queries this data matches.</p>

<p>For example, imagine a user &quot;saving&quot; a search. As new documents are added to the index, documents are percolated against this saved query and the user is alerted when new documents match. The percolator can also be used for data classification and user query feedback.</p>

<p>So how does it work in Elasticsearch? Data and queries are two separate things, but are both expressed in JSON. By using this common property, we can send queries to Elasticsearch and they will be stored as documents...but that alone isn&apos;t enough. Elasticsearch needs to treat it as a query, not a document. The current percolator system stores queries in a dedicated <code>_percolator</code> index. Whenever queries are registered with the percolator index, the query part is extracted and turned into a real query, which is kept around for later usage via the percolate api. The percolate api is the equivalent to the search api for percolating. Also the percolate api is one of the APIs that is fully real-time, meaning that as soon as a query is registered with the percolator it available for use. Let take a look at the percolator already available in many Elasticsearch releases:</p>

<pre class="prettyprint lang-js"># Register a query for &lt;= 0.90.x percolator.
curl -XPUT &apos;localhost:9200/_percolator/my-index/my-id&apos; -d &apos;{
    &quot;query&quot; : {
        &quot;match&quot; : {
            &quot;body&quot; : &quot;coffee&quot;
        }
    }
}&apos;

# &lt;= 0.90.x percolate api
curl -XPUT &apos;localhost:9200/my-index/my-type/_percolate&apos; -d &apos;{
    &quot;doc&quot; : {
        &quot;title&quot; : &quot;Coffee percolator&quot;,
        &quot;body&quot; : &quot;A coffee percolator is a type of ...&quot;
    }
}&apos;
</pre>

<p>The <code>_percolator</code> system index that is used to register a query as shown in the first request in the above sample is a single primary shard index that has replica shards on each data node and these settings are fixed. The last request in the above sample sends a percolate request to Elasticsearch and will yield the following result:</p>

<pre class="prettyprint lang-js">{
    &quot;ok&quot; : true
    &quot;matches&quot; : [&quot;my-id&quot;, ...]
}
</pre>

<p>This image below illustrates how a client executes a percolate request. It doesn&apos;t matter where the percolate request is executed, because each data node has a _percolate shard (p1 squares), that sits next to all the other shards (other squares) a data node may have.</p>

<p><img src="https://api.contentstack.io/v2/assets/575e4d9612798d8e38200280/download?uid=blt5b49a9d406f5ca5f?uid=blt5b49a9d406f5ca5f"></p>

<p>The current percolator has been around since ES version <code>0.15.0</code> and since then more people have started using it with more and more queries. The percolator works fine until a certain amount of queries are store in it, but then the percolate execution time begins to increase to a level where people are less comfortable using it in production. It scales roughly linear to the number of registered queries.</p>

<p>In order to get around this limitation, queries can be partitioned against multiple indices or the percolate query mechanism can be used in order to reduce the percolator execution time. Even with these workarounds, the current percolator has fundamental scaling limits, which becomes obvious when people are percolating massive amounts of queries. So we had to go back to the drawing board and come up with a different solution for how to scale the percolator. The good news is that we did and came up with an alternative solution which we already included in our first 1.0 beta release!</p>

<p>So what has changed in the percolator? The core of the percolator didn&apos;t change that much. You still register percolator queries as in the old system, and percolate documents with the percolate api. What has changed is where we store the percolate queries. We dropped the reserved <code>_percolator</code> index and instead percolate queries are now be stored in any index under the <code>.percolator</code> type. This means that any index can be used as a percolate index, which allows for more flexibility when it comes to scaling. The single shard index restriction has been removed.</p>

<p>Each percolator index can be configured with the number of shards necessary to hold your percolator queries. The percolate api has been changed to execute on any index and parallelize the execution between all the shards within that index. The percolate api can now percolate documents against multiple indices and aliases. The percolate API also now supports <code>routing</code> and <code>preference</code> support, just like the search API.</p>

<p><img src="https://api.contentstack.io/v2/assets/575e4d97d8edd48f76938b2d/download?uid=blt3115962a34656351?uid=blt3115962a34656351"></p>

<p>This image illustrates how the redesigned percolator works. Here the index <code>a</code> has a <code>.percolate</code> type and holds registered queries. It is just an ordinary index with 3 primary shards (green squares) and a replica shard for each primary shard (white squares). The registered queries are divided between shards and then the percolate request is executed in parallel on each node that holds these shards.</p>

<p>Obviously, this large-scale redesign means breaking backwards compatibility with the currently released percolator. Breaking changes are a good time to make improvements to existing structures; for example, the percolator response has changed with some new usability improvements. If you&apos;re already using the percolator, you can just upgrade to 1.0 and re-import your queries from the <code>_percolator</code> index to an index of your choice. Here&apos;s some code to illustrate the new functionality:</p>

<pre class="prettyprint lang-js"># Register a query with the redesigned percolator
curl -XPUT &apos;localhost:9200/my-index1/.percolator/my-id&apos; -d &apos;{
    &quot;query&quot; : {
        &quot;match&quot; : {
            &quot;body&quot; : &quot;coffee&quot;
        }
    }
}&apos;

# Execute a percolate request with the redesigned percolator
curl -XGET &apos;localhost:9200/my-index1/my-type/_percolate&apos; -d &apos;{
    &quot;doc&quot; : {
        &quot;title&quot; : &quot;Coffee percolator&quot;,
        &quot;body&quot; : &quot;A coffee percolator is a type of ...&quot;
    }
}&apos;
</pre>

The last request in the above sample to the percolate api will yield the following result:

<pre class="prettyprint lang-js">{
    &quot;took&quot; : 19,
    &quot;_shards&quot; : {
        &quot;total&quot; : 2,
        &quot;successful&quot; : 2,
    &quot;failed&quot; : 0
    },
    &quot;count&quot; : 4,
    &quot;matches&quot; : [
        {
            &quot;_index&quot; : &quot;my-index1&quot;,
            &quot;_id&quot; : &quot;my-id&quot;
        },
        ...
    ]
}
</pre>

<p>The first major change to the percolate response is that we now include a response header. Just like with the search api, we show you how many shards the request should have gone to, how many shards were actually visited and how many shards failed. The biggest change in the response is how percolate query matches are represented. Instead of list all query ids that have matches in a string array, there is now a match object that encapsulates the query id and the concrete index the query resides in. The <code>_index</code> part is important now because the same query id can be used in multiple indices. We also include a total count in the response, which just tells how many queries matched the percolated document. By default all matching query ids are returned, but that is controllable.</p>

<p>The new features are described in our <a title="Percolator documentation" href="/guide/en/elasticsearch/reference/master/search-percolate.html" target="_blank">reference guide</a>. So, are you curious about the redesigned percolator? Try out our latest beta release and <a href="http://twitter.com/elasticsearch" target="_blank">tell us about it</a>!</p></body></html>

Redesigned Percolator

<p>
	Welcome to&nbsp;
	<i>This week in Elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
</p><h3>Elasticsearch core</h3><ul>
	<li><code>multi_match</code> query now also accepts a string for the <code>fields</code> in addition to an array (<a href="https://github.com/elasticsearch/elasticsearch/issues/4164">#4164</a>, 0.90 & master)</li>
	<li>Indices query/filter now skips parsing for irrelevant indices when possible (<a href="https://github.com/elasticsearch/elasticsearch/issues/2416">#2416</a>, 0.90 & master)</li>
	<li>Added support for <code>_name</code> parameter in indices filter (<a href="https://github.com/elasticsearch/elasticsearch/issues/4166">#4166</a>, 0.90 & master)</li>
	<li>Indices aliases api uses the new generic ack mechanism (<a href="https://github.com/elasticsearch/elasticsearch/issues/4114">#4114</a>, 0.90 & master)</li>
	<li>The cat API got documentation and a help (commit <a href="https://github.com/elasticsearch/elasticsearch/commit/8fabeb1c0b4931f142b668707958f1e01813b239">1</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/7a348cd2d890092f3bdde5dddb6a38be01668dd6">2</a>, master only)</li>
	<li>Native scripts can set script values to primitive arrays (<a href="https://github.com/elasticsearch/elasticsearch/issues/4175">#4175</a>, 0.90 & master)</li>
	<li>A deep pagination with only a few results could lead to overallocation of an array (<a href="https://github.com/elasticsearch/elasticsearch/issues/4156">#4156</a>, 0.90 & master)</li>
	<li>The FilterBuilder interface now is consistent with the QueryBuilder<br>
interface (
	<a href="https://github.com/elasticsearch/elasticsearch/commit/9c15b86b892d16c8c1865806091f06b27eda6930">commit</a>, 0.90 & master)</li>
	<li>The <code>date_format</code> inside of the date mapper can now be updated at runtime (<a href="https://github.com/elasticsearch/elasticsearch/issues/3727">#3727</a>, 0.90 & master)</li>
	<li>A <code>SearchRequest</code> in any <code>PutWarmerRequest</code> is now mandatory and properly validated (<a href="https://github.com/elasticsearch/elasticsearch/issues/4196">#4196</a>, 0.90 & master)</li>
	<li>A field data based <code>TermsFilter</code> has been added (<a href="https://github.com/elasticsearch/elasticsearch/issues/4209">#4209</a>, 0.90 & master)</li>
	<li>Running the node stats api while a shard is moving onto the node, now does not log an exception anymore (<a href="https://github.com/elasticsearch/elasticsearch/issues/4203">#4203</a>, 0.90 & master)</li>
	<li>Listing plugins on a directory with wrong permissions could lead to a <code>NullPointerException</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/4186">#4186</a>, 0.90 & master)</li>
	<li>Refactored open/close API to use the newly introduced generic ack mechanism (<a href="https://github.com/elasticsearch/elasticsearch/issues/4169">#4169</a>, 0.90 & master)</li>
</ul><h3>Elasticsearch Ecosystem</h3><p>
	Here's some more information about what is happening in the ecosystem we are maintaining around Elasticsearch, including plugin and driver releases, as well as news about Logstash and Kibana.
</p><ul>
	<li>Barnaby Gray has written a blog post describing how to use <a href="http://loads.pickle.me.uk/2013/11/12/spark-and-elasticsearch.html">Elasticsearch and Spark</a>.</li>
	<li><a href="https://github.com/petrjasek/eve-elastic">Eve-elastic</a> is a data layer using Elasticsearch for the python-based <a href="http://python-eve.org/&quot;&quot;">Eve framework</a>, which allows building REST APIs easily.</li>
	<li>The creators of <a href="http://www.arangodb.org/">ArangoDB</a> released <a href="https://github.com/triAGENS/elasticsearch-river-arangodb">an ArangoDB river</a> to make data searchable via Elasticsearch.</li>
	<li>The <a href="https://github.com/elasticsearch/puppet-elasticsearch/">Elasticsearch Puppet Module</a> has been majorly updated and partly rewritten.</li>
	<li>The <a href="https://github.com/johtani/elasticsearch-extended-analyze">Extended Analyze Plugin</a>, which adds some more information into the analyze API, has been released for version 0.6.0.</li>
	<li><a href="https://github.com/dadoonet/spring-elasticsearch/">Spring Elasticsearch 0.3</a> has been released</li>
	<li>Version 1.1.0 of the <a href="https://github.com/jprante/elasticsearch-analysis-baseform">Baseform Analysis Plugin</a> has been released</li>
	<li><a href="http://twitter.com/costinl">Costin Leau</a>, the driving force behind the Elasticsearch-Hadoop integration, was interviewed about <a href="http://www.infoq.com/articles/costin-elasticsearch-bigdata">Elasticsearch, BigData and Hadoop</a> on infoq</li>
	<li>Hendrik Saly has updated his <a href="https://github.com/salyh/elasticsearch-msi-installer/releases/tag/v0.90.7.3">Windows MSI Installer for Elasticsearch</a>, which now installs EA 0.90.7 (with Oracle JRE 7 Update 45).</li>
	<li>Several developers from Endgame have authored a post on <a href="http://blog.cloudera.com/blog/2013/11/binarypig-scalable-static-binary-analysis-over-hadoop/">BinaryPig: Scalable Static Binary Analysis Over Hadoop</a>, which includes information on their use of Elasticsearch</li>
	<li>Jurriaan Persyn has written an <a href="http://www.jurriaanpersyn.com/archives/2013/11/18/introduction-to-elasticsearch/">Introduction to Elasticsearch</a> based on presentations he has given at two events. The post includes links to the talk videos and slides.</li>
	<li>Thomas Ardal <a href="http://thomasardal.com/implementing-date-histogram-facets-with-elasticsearch-and-nest/">shared his experiences</a> Implementing Date Histogram Facets with Elasticsearch and NEST.</li>
	<li>Michael Bouvy <a href="http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/">penned an article</a> on how to Collect & visualize your logs with Logstash, Elasticsearch & Redis.</li>
	<li>Jason Wilder has <a href="http://jasonwilder.com/blog/2013/11/19/fluentd-vs-logstash/">written up an article</a> comparing Logstash and Fluentd.</li>
	<li>Smile Open Source Solutions have authored an article <a href="http://blog.smile.fr/Elasticsearch-fait-de-l-ombre-a-solr">Elasticsearch Overshadows Solr</a> (en français)</li>
	<li>Hugo Lassiege has <a href="http://www.eventuallycoding.com/index.php/comparaison-de-moteurs-de-base-de-donnees-pour-du-stockage-de-logs/">authored an article</a> comparing different tools for log and storage analysis, including Elasticsearch (en français)</li>
	<li>Antoine Rey authored a post on <a href="http://javaetmoi.com/2013/11/musicbrainz-elasticsearch-angularjs-openshift/">Elastifying a MusicBrainz database on OpenShift</a> (en français)</li>
</ul><h3>Slides</h3><ul>
	<li>You can watch <a href="http://twitter.com/honzakral">Honza Kral's</a> talk from PyCon Poland 2013: Explore Your Data with Elasticsearch<br>
	<iframe src="https://www.youtube.com/embed/yZu4jQtBUPg" allowfullscreen="" width="504" height="283" frameborder="0">
	</iframe>
	</li>
	<li>Andrew Clegg, Head Data Geek at Pearson Technology, presented at last week's <a href="http://www.meetup.com/London-ElasticSearch-User-Group/events/144492702/">Elasticsearch London Meetup</a> about <a href="http://www.slideshare.net/AndrewClegg1/scaling-elasticsearch-for-multiterabyte-analytics">Tuning Elasticsearch for Multi-Terabyte Analytics</a>.</li>
	<li>Zuhaib Siddique, Infrastructure Engineer at Atlassian, <a href="https://speakerdeck.com/zsiddique/hipchat-elasticsearch-meetup">posted his slides</a> from last week's <a href="http://www.meetup.com/Elasticsearch-San-Francisco/events/141698772/">Elasticsearch San Francisco Meetup</a>. The presentation covers how HipChat uses Elasticsearch in its infrastructure to provide near real-time search and chat history to the hundreds of thousands of active chatters using HipChat, scaling to over 1 billion messages.</li>
	<li>Sharif Olorin, Developer with Anchor Systems, has <a href="https://speakerdeck.com/fractalcat/monitoring-elasticsearch-for-fun-profit-and-not-getting-woken-up-at-3am">shared his slides</a> from his presentation at the first Sydney Elasticsearch Meetup, Monitoring Elasticsearch for Fun, Profit and Not Getting Woken Up at 3am.</li>
	<li><a href="http://twitter.com/theuntergeek">Aaron Mildenstein</a> presented an <a href="https://speakerdeck.com/elasticsearch/introduction-to-logstash-1">Introduction to Logstash</a> - Or: How to wring useful data from those files just lying around using disk space at the <a href="http://www.meetup.com/Elasticsearch-User-Group-Austin/events/149180112/">Elasticsearch Austin Meetup</a>.</li>
	<li><a href="http://twitter.com/dadoonet">David Pilato's</a> presentations from Devoxx 2013 are both available; check out <a href="https://speakerdeck.com/elasticsearch/elastify-your-app-from-sql-to-nosql-in-less-than-one-hour">Elastify Your Application</a>: from SQL to NoSQL in Less than One Hour, presented with Tugdual Grall, and <a href="https://speakerdeck.com/elasticsearch/make-sense-of-your-big-data-1">Make Sense of Your (BIG) Data</a>.</li>
	<li>Xebia France created a presentation on <a href="http://fr.slideshare.net/XebiaFrance/open-xke-tirer-le-meilleur-de-ses-donnes-avec-elasticsearch-par-sven-le-mesle">Open XKE - Making the most of your data with ElasticSearch</a> (en français)</li>
</ul><h3>Where to find Us</h3><p><br></p>
<h4>France</h4><p>
	David Pilato will speak at Drupagora on December 5th, covering 
	<a href="http://www.drupagora.com/2013/programme-drupagora#Présentation des solutions de recherche">search solutions for Drupal using Elasticsearch</a>. David's talk will begin at 3 PM.
</p><h4>Netherlands</h4><ul>
	<li>
	Elasticsearch will be at the <a href="http://cloudstackcollab.org/">CloudStack Collaboration Conference EU</a> on November 20-22nd in Amsterdam. Stop by our table to say hello.</li>
	<li><a href="http://twitter.com/kimchy">Shay Banon</a> will present <a href="http://www.opensourceconference.nl/3736">Why We Built Elasticsearch</a> at the Open Source Conference in Amsterdam on December 6th. Shay's talk will start at 4:05 PM.</li>
</ul><h4>Norway</h4><p>
	<a href="http://twitter.com/bleskes">Boaz Leskes</a> and <a href="http://twitter.com/karmiq">Karel Minarik</a> will speak at the <a href="http://www.meetup.com/elasticsearch-oslo/events/146914192/">Elasticsearch Oslo Meetup</a> on November 26th. Full details on their talks and other presentations by community members in Oslo will be published next week.
</p><h4>Spain</h4><p>
	<a href="http://twitter.com/clintongormley">Clinton Gormley</a> will speak at the NoSQL matters conference in Barcelona on <a href="http://2013.nosql-matters.org/bcn/abstracts/#abstract_287512263">Getting Down and Dirty with Elasticsearch</a>. Clinton's talk takes place on Saturday, November 30th at 10:30 AM, and the conference runs November 29-30th.
</p><h4>United Kingdom</h4><ul>
	<li><a href="http://twitter.com/s1m0nw">Simon Willnauer</a> will present <a href="http://skillsmatter.com/podcast/nosql/with-a-hammer-in-your-hand-elasticsearch">Elasticsearch: With a Hammer in Your Hand</a> at the NOSQL eXchange in London on November 28th. Simon's presentation begins at 1 PM.</li>
	<li>Simon will also speak at the <a href="http://codemesh.io/">CodeMesh conference</a> from the 4th to the 5th of December.</li>
	<p>
		<strong><br></strong>
	</p>
</ul><h3>Where to Find You</h3><p>
	Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, 
	<a href="https://docs.google.com/forms/d/1qBZ6Fwq4JWw7FH3s6C9Q3KeGCvUOmWYOvNMsxNqkCNM/viewform">take a moment to let her know</a>.
</p><p>
	Oh yeah, we're also&nbsp;
	<a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you'd like us to find you for employment purposes, just&nbsp;<a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &nbsp;We care more about your skill set and passion for Elasticearch, Kibana and Logstash than where you rest your head.
</p><h3>Trainings</h3><p>
	If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
</p><ul>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/new-york/2013-12-03">New York</a>&nbsp;- December 02, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/paris/2013-12-05">Paris</a>&nbsp;- December 05, 2013</li>
	<li><a href="&quot;http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/london-uk/2013-12-09">London</a>&nbsp;- December 12, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/los-angeles/2013-12-16">Los Angeles</a>&nbsp;- December 16, 2013</li>
</ul>

This Week in Elasticsearch - November 21, 2013

Keeping Elasticsearch in Sync

<html><head></head><body><p>Welcome to&#xA0;<i>This Week in Elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.</p><h2>Elasticsearch Core</h2><ul>
<li>We <a href="http://www.elasticsearch.org/blog/0-90-7-released/">released 0.90.7</a> yesterday, the current stable release of the 0.90 series. Upgrading is recommended.</li>
<li>The initial Snapshot/Restore implementation has landed! (<a href="https://github.com/elasticsearch/elasticsearch/issues/3826">#3826</a>, master only)</li>
<li>Setting <code>rootTypeParsers</code> in <code>DocumentMapperParser</code> was incorrect (<a href="https://github.com/elasticsearch/elasticsearch/issues/4113">#4113</a>, 0.90 &amp; master)</li>
<li>Guarantee sorted order for Double/Long|Bytes values (<a href="https://github.com/elasticsearch/elasticsearch/commit/a9239f81473bdc1b1fb1c28bdf8d6207b75ce0bb">commit</a>, 0.90 &amp; master)</li>
<li>Highlight hit object cache is not resetted per hit anymore (<a href="https://github.com/elasticsearch/elasticsearch/issues/4106">#4106</a>, 0.90 &amp; master)</li>
<li>Postings highlighter doesn&apos;t return snippets when using <code>&quot;path&quot;:&quot;just_name&quot;</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/4116">#4116</a>, 0.90 &amp; master)</li>
<li><code>_default_</code> mapping not applied when using separate master/data nodes (<a href="https://github.com/elasticsearch/elasticsearch/issues/4124">#4124</a>, 0.90 &amp; master)</li>
<li>Creating a nested filtered alias on a dataless master failed (<a href="https://github.com/elasticsearch/elasticsearch/issues/4112">#4112</a>, 0.90 &amp; master)</li>
<li>Added support for external query in postings highlighter (<a href="https://github.com/elasticsearch/elasticsearch/issues/4121">#4121</a>, 0.90 &amp; master)</li>
<li>Fixed issues with <code>JAVA_OPTS/ES_JAVA_OPTS</code> in <code>service.bat</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/4127">#4127</a>, 0.90 &amp; master)</li>
<li>We are working on exposing all the nifty test classes used in the elasticsearch tests to use in your own project (<a href="https://github.com/elasticsearch/elasticsearch/commit/a824f69d21c709bcd578981a78a592200c999dbc">commit</a>, 0.90 &amp; master)</li>
<li>Library upgrades: Netty 3.8.0, hppc 0.5.3 (commits <a href="https://github.com/elasticsearch/elasticsearch/commit/b17732ea5630a9ed6a35e636ac16bce0c06c44a8">1</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/9fdcb0ad0514d1556150abb0622779f77d9b3ed2">2</a>, 0.90 &amp; master)</li>
<li>Make <code>mmapfs</code> the default store for 64-bit Linux (<a href="https://github.com/elasticsearch/elasticsearch/issues/4134">#4134</a>, master only)</li>
<li>Fixed a <code>NullPointerException</code> when percolating a document that contains a completion field (<a href="https://github.com/elasticsearch/elasticsearch/issues/4028">#4028</a>, 0.90 &amp; master)</li>
<li>Problems with non-starting rivers have been fixed (<a href="https://github.com/elasticsearch/elasticsearch/issues/4089">#4089</a>, 0.90 &amp; master)</li>
<li>Expose maximum heap settings and heap usage in percent in JVM memory stats (<a href="https://github.com/elasticsearch/elasticsearch/issues/4145">#4145</a>, 0.90 &amp; master)</li>
<li>IndexShardGatewayService should not call post recovery if shard is in <code>STARTED</code> state (<a href="https://github.com/elasticsearch/elasticsearch/issues/4147">#4147</a>, 0.90 &amp; master)</li>
<li>NPE for <code>has_child</code> query if number of results exceed certain limit (<a href="https://github.com/elasticsearch/elasticsearch/issues/4135">#4135</a>, 0.90 &amp; master)</li>
<li>Fixed a deadlock in <code>BulkProcessor</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/4153">#4153</a>, 0.90 &amp; master)</li>
<li>Wrong result on bool filter with <code>must_not</code> and <code>geo_distance</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/4130">#4130</a>, 0.90 &amp; master)</li>
</ul><h2>Elasticsearch Ecosystem</h2><p>Here&apos;s some more information about what is happening in the ecosystem we are maintaining around Elasticsearch, including plugin and driver releases, as well as news about Logstash and Kibana.</p><ul>
<li>A Scala client named <a href="https://github.com/gphat/wabisabi">wabisabi</a> has been released</li>
<li>A couple of official analysis plugins have been updated: <a href="https://github.com/elasticsearch/elasticsearch-analysis-kuromoji">kuromoji</a>, <a href="https://github.com/elasticsearch/elasticsearch-analysis-phonetic">phonetic</a>, <a href="https://github.com/elasticsearch/elasticsearch-analysis-smartcn">smartcn</a>, <a>icu</a>, <a href="https://github.com/elasticsearch/elasticsearch-analysis-stempel">stempel</a></li>
<li>We had another community blog post about <a href="http://www.elasticsearch.org/blog/building-enterprise-content-management-systems-on-elasticsearch/">building enterprise content management systems on Elasticsearch</a></li>
<li>A <a href="https://github.com/englishtown/vertx-mod-elasticsearch">vert.x elasticsearch module</a> has been created</li>
<li>Version 0.4.2 of the <a href="https://github.com/elasticsearch/elasticsearch-php/">elasticsearch-php client</a> was released (<a href="https://github.com/elasticsearch/elasticsearch-php/blob/master/CHANGELOG.md">changelog</a>)</li>
<li><a href="http://twitter.com/fhopf">Florian Hopf</a> created a nice blog post about the <a href="http://www.found.no/foundation/Sense-Elasticsearch-interface/">Chrome Sense extension</a>, an awesome in-browser interface to use elasticsearch</li>
</ul><h2>Slides</h2><p>Check out <a href="http://twitter.com/jpountz">Adrien Grand&apos;s</a> presentations from Lucene revolutions, talking about <a href="https://speakerdeck.com/elasticsearch/index-sorting-with-lucene">Index sorting with Lucene</a> and <a href="https://speakerdeck.com/elasticsearch/what-is-in-a-lucene-index">What is a in a Lucene index</a>. Also <a href="http://twitter.com/kevinkluge">Kevin Kluge</a> <a href="https://speakerdeck.com/elasticsearch/elasticsearch-in-20-minutes">introduced Elasticsearch</a> at the ruby user group in San Francisco. Last, <a href="http://twitter.com/spinscale">Alexander Reelsen</a> talked about using <a href="https://speakerdeck.com/elasticsearch/elasticsearch-beyond-full-text-search">Elasticsearch - beyond full-text search</a>.</p><h2>Where to Find Us</h2><h4>Australia</h4><p>Clinton Gormley will be speaking at the inaugural <a href="http://www.meetup.com/Elasticsearch-Sydney-Meetup/events/149068632/">Sydney Elasticsearch Meetup</a> on November 18th and the <a href="http://www.meetup.com/melbourne-search/events/148371692/">Melbourne Search Users Group</a> on November 21st. Clinton will cover new features forthcoming in 1.0 and will be joined by other speakers from the Melbourne and Sydney community. We&apos;ll have full details for you next week.</p><h4>Belgium</h4><p>If you&apos;re heading to Devoxx Belgium 2013, be sure to check out <a href="http://twitter.com/dadoonet">David Pilato</a> on Elastify Your App: From SQL to NoSQL in Less than One Hour. David will speak on Wednesday, November 13th at 3:10 PM in Room 6. David will also give a lightning talk, Make Sense of Your (BIG) Data on Thursday, November 14th at 1:10 PM in Room 9. Devoxx Belgium runs from November 11-15 at the Metropolis Business Center in Antwerp.</p><h4>France</h4><ul>
<li>David Pilato wil speak on <a href="http://nightclazz-es-avance.eventbrite.fr/">Advanced Elasticsearch and Kibana</a> at Zenika&apos;s Night Class on November 7th. The class begins at 7:00 PM.</li>
<li>David will also speak at <a href="http://www.drupagora.com/2013/">Drupagora</a> at the 5th of December</li>
</ul><h4>Germany</h4><p><a href="http://twitter.com/spinscale">Alexander Reelsen</a> organizes the <a href="http://spinscale.github.io/munich-search.html">Munich Search Meetup group</a>, and will be hosting a meetup on November 20th. The meetup begins at 6:45 PM.</p><h4>Netherlands</h4><p>Elasticsearch will be at the <a href="http://cloudstackcollab.org/">CloudStack Collaboration Conference EU</a> on November 20-22nd in Amsterdam. Stop by our table to say hello, and make sure to catch <a href="http://twitter.com/bleskes">Boaz Leskes</a> and Gerard de Vos&apos; workshop on Elasticsearch, taking place on Wednesday, November 20th. Also you can easily join the <a href="http://www.meetup.com/ElasticSearch-NL/events/150664602/">Elasticsearch Amsterdam Meetup</a> on the 20th as well in the evening. Even if you attend the Cloudstack Collaboration Conference, feel free to come over, as the event starts a bit later.</p><h4>Norway</h4><p><a href="http://twitter.com/bleskes">Boaz Leskes</a> and <a href="http://twitter.com/karmiq">Karel Minarik</a> will speak at the <a href="http://www.meetup.com/elasticsearch-oslo/events/146914192/">Elasticsearch Oslo Meetup</a> on November 26th. Full details on their talks and other presentations by community members in Oslo will be published next week.</p><h4>Spain</h4><p><a href="http://twitter.com/clintongormley">Clinton Gormley</a> will speak at the NoSQL matters conference in Barcelona on <a href="http://2013.nosql-matters.org/bcn/abstracts/#abstract_287512263">Getting Down and Dirty with Elasticsearch</a>. Clinton&apos;s talk takes place on Saturday, November 30th at 10:30 AM, and the conference runs November 29-30th.</p><h4>United Kingdom</h4><ul>
<li><a href="http://twitter.com/s1m0nw">Simon Willnauer</a> will present <a href="http://skillsmatter.com/podcast/nosql/with-a-hammer-in-your-hand-elasticsearch">Elasticsearch: With a Hammer in Your Hand</a> at the NOSQL eXchange in London on November 28th. Simon&apos;s presentation begins at 1 PM.</li>
<li>Simon will also speak at the <a href="http://codemesh.io/">CodeMesh conference</a> from the 4th to the 5th of December.</li>
</ul><h4>United States</h4><ul>
<li><a href="https://twitter.com/thnetos">Lee</a> and <a href="https://twitter.com/drewr">Drew</a> are both attending <a href="http://clojure-conj.org/">Clojure Conj</a> in Alexandria this week, as elasticsearch is also a conference sponsor</li>
<li>You can learn all about how Atlassian uses Elasticsearch for <a href="http://www.meetup.com/Elasticsearch-San-Francisco/events/141698772/">Heavy lifting in HipChat</a> at the San Francisco Bay Area meetup on Monday, November 18th. Atlassian will be hosting the meetup at their offices and doors open at 6:30 PM.</li>
<li><a href="http://twitter.com/uboness">Uri Boness</a> and <a href="http://twitter.com/zacharytong">Zachary Tong</a> will be presenting at the <a href="http://www.meetup.com/Elasticsearch-User-Group-Chicago/events/148062252/">Chicago Elasticsearch Meetup</a> on November 18th. Further details coming soon!</li>
</ul><h2>Where to Find You</h2><p>Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, <a href="https://docs.google.com/forms/d/1qBZ6Fwq4JWw7FH3s6C9Q3KeGCvUOmWYOvNMsxNqkCNM/viewform">take a moment to let her know</a>.</p><p>Oh yeah, we&apos;re also&#xA0;<a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you&apos;d like us to find you for employment purposes, just&#xA0;<a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &#xA0;We care more about your skill set and passion for Elasticearch, Kibana and Logstash than where you rest your head.</p><h2>Trainings</h2><p>If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:</p><ul>
<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/chicago/2013-11-18">Chicago</a>&#xA0;- November 18, 2013</li>
<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/san-francisco-california-united-states/2013-11-21">San Francisco</a>&#xA0;- November 21, 2013</li>
<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/paris/2013-12-05">Paris</a>&#xA0;- December 05, 2013</li>
<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/london-uk/2013-12-09">London</a>&#xA0;- December 12, 2013</li>
<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/los-angeles/2013-12-16">Los Angeles</a>&#xA0;- December 16, 2013</li>
</ul></body></html>

This Week in Elasticsearch - November 14, 2013

<html><head></head><body><p><strong>UPDATE:</strong> <em>This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</em></p><p><strong>Update May 1, 2017:</strong>&#xA0;<em>At the time of this writing, Sense existed purely as a Chrome plugin. &#xA0;It was later made into a Kibana plugin and has now been integrated directly into Kibana as &quot;Console&quot; in what is&#xA0;known as the &quot;Dev Tools&quot; section. &#xA0;The Dev Tools&#xA0;can be accessed in Kibana without the need to install a&#xA0;browser plugin. &#xA0;See <a href="/guide/en/kibana/5.3/console-kibana.html">the Console docs in Kibana</a> for more information on that functionality. &#xA0;As a result of the migration into Kibana, the Chrome plugin is no longer maintained, and the blog below is retained mostly for legacy purposes, though you may find that many of the usage patterns contained on this page (including autocomplete and the request/response panes) are directly applicable in Kibana&apos;s Dev Tools Console.</em><br></p><p>This article will introduce you to Sense, a Chrome plugin for Elasticsearch. It offers autocompletion, code highlighting and formatting and can help you with exploring Elasticsearch.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>HTTP and JSON are the backbone for any work with Elasticsearch. Indexing, searching, administrative operations: Everything is done with HTTP and JSON. Most examples you will see are using <a href="http://curl.haxx.se/">cURL</a>, the http client widely available on different systems. With cURL you pass the HTTP method and the body of a request as command line parameters.
</p>
<p>While because of its availability on different systems cURL is great to be used for examples you might long for something more comfortable for your daily work. Enter <a href="https://github.com/bleskes/sense">Sense</a>, a <a href="http://www.google.com/chrome">Chrome</a> plugin for Elasticsearch.
</p>
</section><section id="basics" class="level2">
<h2><a href="#basics">Basics</a></h2>
<p>After installing Sense, it will allow you to use a simpler query request format, offers code completion and a highlighted response section.
</p>
<p>Like any extension, it can be started from the menu on the top right of Chrome. The UI consists of two larger panes, a white one on the left that is used to enter your requests to Elasticsearch and a black response section on the right. The host to be used can be configured in the top line and defaults to a local installation.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8a12798d8e381fd5fd/download?uid=bltccab886e4a31cd63?uid=bltccab886e4a31cd63" data-sys-asset-uid="bltde7dc0c68917467d" alt="Sense consists of two panes: One for the request and one for the response" style="max-width:100%;">
</p><figcaption>Sense consists of two panes: One for the request and one for the response</figcaption>
<p>You will see that there already is an example query that you can immediately execute using the green arrow on the top right of the query pane. It queries the current instance for all documents of all types in all indices. If you are starting with a fresh Elasticsearch installation this will return an empty result list but you can immediately start with some examples from the Elasticsearch guide. Let&#x2019;s index some data.
</p>
</section><section id="copy-and-paste" class="level2">
<h2><a href="#copy-and-paste">Copy and Paste</a></h2>
<p>As so many examples are already using cURL Sense offers a nice copy and paste feature that translates cURL requests to the proper syntax to use for Sense. Just copy the following snippet to your clipboard:
</p>
<pre class="prettyprint">curl -XPUT &apos;http://localhost:9200/twitter/tweet/1&apos; -H &apos;Content-Type: application/json&apos; -d &apos;{
    &quot;user&quot; : &quot;kimchy&quot;,
    &quot;post_date&quot; : &quot;2009-11-15T14:12:12&quot;,
    &quot;message&quot; : &quot;trying out Elastic Search&quot;
}&apos;
</pre>
<p>Mark all of the existing query in the left pane and press Ctrl-V to paste the text. You will notice that Sense automatically removes the cURL parameters and transforms the query to its internal syntax.
</p>
<pre class="prettyprint">PUT /twitter/tweet/1
{
    &quot;user&quot; : &quot;kimchy&quot;,
    &quot;post_date&quot; : &quot;2009-11-15T14:12:12&quot;,
    &quot;message&quot; : &quot;trying out Elastic Search&quot;
}
</pre>
<p>Once you click the green arrow to execute the indexing query you will see the familiar response from Elasticsearch that your document has been indexed.
</p>
<p>You can also go the other way around. If you&#x2019;d like to get the cURL equivalent of a query you can mark it in the pane, press the little tool icon and choose Copy as cURL. You can then paste the result to a shell and execute the same command using cURL. The keyboard shortcut for this is Ctrl + Shift + C.
</p>
</section><section id="autocompletion" class="level2">
<h2><a href="#autocompletion">Autocompletion</a></h2>
<p>Instead of copying and pasting we can of course also create the query ourselves. Remove all the existing text from the query pane and start typing GET. Sense will immediately offer a small tooltip with possible completions. Continue typing the / and you will notice that the popup has some internal knowledge of our Elasticsearch instance: It offers our index, twitter, that we just created.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8a43e9adc538713aa0/download?uid=bltf312c168e5c212f2?uid=bltf312c168e5c212f2" data-sys-asset-uid="bltbe74fa336c406c11" alt="Autocompletion works on indexes" style="max-width:100%;">
</p><figcaption>Autocompletion works on indexes</figcaption>
<p>We can continue to enter our query, e.g. a simple term query on the user:
</p>
<pre class="prettyprint">GET /twitter/tweet/_search
{
    &quot;query&quot;: {
        &quot;term&quot;: {
           &quot;user&quot;: {
              &quot;value&quot;: &quot;kimchy&quot;
           }
        }
    }
}
</pre>
<p>If during any part of typing the query you are unsure what is allowed or wonder about the correct spelling you can always press Ctrl-Space to get a list of possible options. This powerful context sensitive completion can help a lot when experimenting with Elasticsearch.
</p>
</section><section id="history" class="level2">
<h2><a href="#history">History</a></h2>
<p>When working with Sense you might notice that you did something exciting a while ago but can&#x2019;t remember what you did. No need to undo all your edits; just open the history that can be accessed on the top right. It will show you the last 500 edits and you can choose one that you would like to execute again.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8a9985d589763769d8/download?uid=blt8f4ebfa5f3270d60?uid=blt8f4ebfa5f3270d60" data-sys-asset-uid="bltf46bc349e11f4a2e" alt="The history will show you your recent queries" style="max-width:100%;">
</p><figcaption>The history will show you your recent queries</figcaption>
</section><section id="structuring-json" class="level2">
<h2><a href="#structuring-json">Structuring JSON</a></h2>
<p>Sometimes requests and responses can become rather large with different indentation levels. To structure them you can use two features of Sense: Code indentation and code folding. If you don&#x2019;t insert well indented code at first Sense can do the indentation for you: Just press Ctrl-I to have it formatted. And if your query or response really gets too long you can use the code folding feature. In the query as well as the response pane you can see little arrows at some line numbers. Those can be used to fold or unfold a code block. This can help a lot to make the JSON more approachable.
</p>
</section><section id="conclusion" class="level2">
<h2><a href="#conclusion">Conclusion</a></h2>
<p>Sense is a fantastic tool for everyday work with Elasticsearch as well as for learning the query syntax. It will guide you with the queries, format everything correctly and do syntax highlighting on the query and response for you. If you&#x2019;re not using it yet you should at least give it a try.
</p><p><em>Editor&#x2019;s Note (May 1, 2017): Starting with 6.0, any curl command to Elasticsearch containing content will require a valid content type header. As a result, this post has been updated to reflect this change and to set readers of this post up for success with future versions.</em><br></p>
</section></body></html>

Sense - a Cool JSON Aware Interface to Elasticsearch

<html><head></head><body><p>Today we are happy to announce the release of <strong>elasticsearch 0.90.7</strong>, which is based on Lucene 4.5.1. This is the current stable release in the 0.90 series and we recommend upgrading. You can download it <a href="http://www.elasticsearch.org/downloads/0-90-7/" target="_top">here</a>.</p>
<p>This is largely a bug fix release. Most changes deal with edge cases (you can read about them in the <a href="http://www.elasticsearch.org/downloads/0-90-7/" target="_top">release notes</a>) but there are a few important fixes which are worth mentioning:</p>
<h2><a id="_norms_and_memory_use" href="#_norms_and_memory_use"></a>Norms and memory use</h2>
<p>In an effort to make Elasticsearch as responsive as possible, 0.90.6 introduced a change which loaded all <em>field norms</em> into memory whenever a new merged segment was created. For some users, this turned out to be an optimization too far. Field norms provide statistics about field length and are an important variable in TF/IDF, Lucene&#x2019;s default similarity algorithm.</p>
<p>However, they are only useful when you actually run a full-text query on a field. Many users, especially our Logstash users, store many text fields in Elasticsearch which they never search on, which means that we never need to load the norms for these fields. This &#x201C;eager&#x201D; loading of norms used up much more RAM than before.</p>
<p>In 0.90.7 we reverted this change (see <a href="https://github.com/elasticsearch/elasticsearch/issues/4078" target="_top">#4078</a>) and in a later version we will add a per-field option which will allow you to opt in to eager norms loading, where it makes sense for you.&#xA0;See <a href="https://github.com/elasticsearch/elasticsearch/issues/4079" target="_top">#4079</a>.</p>
<h2><a id="_lucene_and_memory_use" href="#_lucene_and_memory_use"></a>Lucene and memory use</h2>
<p>A <a href="https://issues.apache.org/jira/browse/LUCENE-5330" target="_top">bug in Lucene</a> was causing a delay in cleaning up old files which manifested itself in increased memory use, especially when using the <code>memory</code> store. This bug has been fixed in Lucene 4.6 and the fix has been backported to this release as well.&#xA0;See <a href="https://github.com/elasticsearch/elasticsearch/issues/4093" target="_top">#4093</a>.</p>
<h2>Postings highlighter</h2>
<p>The newly added Postings highlighter got off to a false start. Highlighting worked just fine... until there was more than one segment. (Note: you can never have too many tests). See <a href="https://github.com/elasticsearch/elasticsearch/issues/4103" target="_top">#4103</a>. It also failed on fields whose index name and field name were different. See <a href="https://github.com/elasticsearch/elasticsearch/issues/4116" target="_top">#4116</a>. And finally, an enhancement! The postings highlighter now also supports a separate <code>highlight_query</code>. See <a href="https://github.com/elasticsearch/elasticsearch/issues/4121" target="_top">#4121</a>.</p>
<h2><a id="_other" href="#_other"></a>Other</h2>
<ul>
<li>The <code>has_child</code> filter would occasionally throw an error when it matched many parents. See <a href="https://github.com/elasticsearch/elasticsearch/issues/4135" target="_top">#4135</a>.</li>
<li>A <code>bool</code> filter which combined a <code>must_not</code> clause with a <code>geo</code> filter could produce too few results. See <a href="https://github.com/elasticsearch/elasticsearch/issues/4144" target="_top">#4144</a>.</li>
<li>The improvements in cluster state changes resulted in a race condition which caused newly added rivers to fail. See <a href="https://github.com/elasticsearch/elasticsearch/issues/4089" target="_top">#4089</a>.</li>
</ul>
<p>Details of other changes are available in the <a href="http://www.elasticsearch.org/downloads/0-90-7/" target="_top">release notes</a>.</p>
<p>Many thanks to everybody who submitted bug reports and pull requests. Your help is greatly appreciated.</p>
</body></html>

0.90.7 released

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p><a href="https://www.found.no/play/">Play</a> is a tool that makes it easy to, well, play with Elasticsearch&#x2019;s vast feature set. It&#x2019;s inspired by tools like <a href="http://jsfiddle.net/">jsfiddle</a>. jsfiddle lets you easily throw together HTML+CSS+Javascript demos. Play lets you do that for Elasticsearch: a combination of sample documents, text analyzers, mappings and searches constitute a runnable demo.
</p>
<p>To develop the perfect search, you need to structure the documents correctly, configure the right text processing, wire up mappings and do some searches to see how well it works.
</p>
<p>This is typically an iterative process. &#x201C;Oh, it would be great if I could search like this. But that requires a change to that analyzer, and tweaking the mappings a bit, and [etc., etc.&#x2026;]&#x201D;
</p>
<p>While changing mappings and analyzers for an existing index is a lot of work with plain Elasticsearch, Play makes it easy by doing those things for you. Change an analyzer configuration, run, and a split second later you can see how it affects your searches. It also assists you with configuring mappings and searches as well, with context aware autocompletion and documentation.
</p>
<p>When you play, it creates the indexes with the right mappings and analyzers for you, indexes all your sample documents and then runs all the sample searches. It can do this very quickly because we impose one limitation: you cannot have a huge set of sample documents.
</p>
<p><strong>Note</strong>: <a href="https://www.found.no/play/">Play</a> is neither feature complete nor bug free. This is a preview to show where we&#x2019;re headed, and to get some early feedback. Expect bugs!
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4ca10342dfd738263484/download?uid=blt6d469b897f9b25a1" data-sys-asset-uid="blt6d469b897f9b25a1" style="max-width:100%;">
</figure>
</section><section id="overview" class="level2">
<h2><a href="#overview">Overview</a></h2>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4ca19985d58976376da3/download?uid=blt754f23fbfa62099d" data-sys-asset-uid="blt754f23fbfa62099d" alt="Overview" style="max-width:100%;"><figcaption>Overview</figcaption>
</figure>
<p><a href="https://www.found.no/play/">Play</a>&#x2019;s user interface is structured into five editor tabs.
</p>
<p>In addition to an Overview tab, there are supplementary tabs designed to optimize the screen estate for the task at hand. They are different views on the same data, and are kept in sync.
</p>
<ul>
	<li>The <em>Documents</em> tab uses most of the screen estate on the editor for sample documents, with the rest spent on showing the results.</li>
	<li><em>Analysis</em> shows an editor for configuring analyzers, tokenizers and so on, with a view that shows how text is processed step by step.</li>
	<li><em>Mappings</em> uses most of the space for the mappings editor, with some space left for documentation for the type of mapping you&#x2019;re doing, a small view that shows how text is processed for the currently selected field, and lastly, the resulting mapping.</li>
	<li><em>Searches</em> uses a lot of space for editing searches and for showing results. There&#x2019;s also a view for showing documentation for the search/filter/facet/parameter your cursor is at.</li>
</ul>
</section><section id="thats-not-json" class="level2">
<h2><a href="#thats-not-json">That&#x2019;s Not JSON!</a></h2>
<p>Play prefers <a href="http://yaml.org/">YAML</a> over JSON. YAML is easier to read and edit (by humans), and you can comment it.
</p>
<p>While JSON is valid YAML, to get the most out of Play (i.e. context-aware autocompletion and documentation), you should use the supported subset of YAML. This is documented in Play&#x2019;s help section.
</p>
<p>In <a href="#can-i-run-it-on-my-cluster">Can I Run it on My Cluster?</a> we describe how you can export things as JSON, though!
</p>
</section><section id="your-first-play" class="level2">
<h2><a href="#your-first-play">Your first Play</a></h2>
<p>Before we look further into the various features of Play, let&#x2019;s start out with a simple Hello World.
</p>
<p>In this exercise we want to index three documents and run two searches.
</p>
<ol type="1">
	<li>First, open <a href="https://www.found.no/play/">Play</a> in a separate tab or window. Click the &#x201C;Clear&#x201D;-button in the top right corner to clear out the introductory text.</li>
	<li>Paste this text in the &#x201C;Documents&#x201D; editor, i.e. the top left editor:</li>
</ol>
<pre class="sourceCode yaml prettyprint">&lt;span class=&quot;co&quot;&gt;# This is the first document&lt;/span&gt;
&lt;span class=&quot;fu&quot;&gt;quote:&lt;/span&gt; Man had always assumed that he was more intelligent than dolphins because he had achieved so much - the wheel, New York, wars and so on - whilst all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man - for precisely the same reasons.
&lt;span class=&quot;fu&quot;&gt;author:&lt;/span&gt; Douglas Adams
&lt;span class=&quot;ot&quot;&gt;---&lt;/span&gt;
&lt;span class=&quot;co&quot;&gt;# This is the second document. The three dashes (---) separate them.&lt;/span&gt;
&lt;span class=&quot;fu&quot;&gt;quote:&lt;/span&gt; The ships hung in the sky in much the same way that bricks don&apos;t.
&lt;span class=&quot;fu&quot;&gt;author:&lt;/span&gt; Douglas Adams
&lt;span class=&quot;ot&quot;&gt;---&lt;/span&gt;
&lt;span class=&quot;co&quot;&gt;# And the third...&lt;/span&gt;
&lt;span class=&quot;fu&quot;&gt;quote:&lt;/span&gt; Winter is coming.
&lt;span class=&quot;fu&quot;&gt;author:&lt;/span&gt; George R. R. Martin
</pre>
<ol start="3" type="1">
	<li>Paste this into the &#x201C;Searches&#x201D; editor, i.e. the top right editor:</li>
</ol>
<pre class="sourceCode yaml prettyprint">&lt;span class=&quot;co&quot;&gt;# This is the first search&lt;/span&gt;
&lt;span class=&quot;fu&quot;&gt;query:&lt;/span&gt;
    &lt;span class=&quot;fu&quot;&gt;match:&lt;/span&gt;
        &lt;span class=&quot;fu&quot;&gt;quote:&lt;/span&gt; dolphins
&lt;span class=&quot;ot&quot;&gt;---&lt;/span&gt;
&lt;span class=&quot;co&quot;&gt;# Second search.&lt;/span&gt;
&lt;span class=&quot;fu&quot;&gt;facets:&lt;/span&gt;
    &lt;span class=&quot;fu&quot;&gt;words:&lt;/span&gt;
        &lt;span class=&quot;fu&quot;&gt;terms:&lt;/span&gt; 
            &lt;span class=&quot;fu&quot;&gt;field:&lt;/span&gt; author
</pre>
<ol start="4" type="1">
	<li>Click on &#x201C;Run&#x201D; in the top right menu, or press Ctrl+Enter to run the play.</li>
	<li>Results appear in the bottom right window. There&#x2019;s one tab for the resulting mappings, and one for each search.</li>
</ol>
</section><section id="autocompletion-documentation-and-linting" class="level2">
<h2><a href="#autocompletion-documentation-and-linting">Autocompletion, Documentation and Linting</a></h2>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4ca158208ba076e283b7/download?uid=blt30c04086b66f74f5" data-sys-asset-uid="blt30c04086b66f74f5" alt="Context aware autocompletion and documentation" style="max-width:100%;"><figcaption>Context aware autocompletion and documentation</figcaption>
</figure>
<p>While Play is neither all-encompassing nor feature complete, we have spent a lot of time making sure we can add context aware autocompletion, documentation and linting.
</p>
<p>The various editors know where your cursor is, and whether the cursor is in a filter in a query in a facet. They already suggest many things:
</p>
<ul>
	<li>Most of the search structures, like query, filter and facet DSLs.</li>
	<li>Fields and types - both when mapping, modifying sample documents and specifying them as parameters in queries, facets or filters.</li>
	<li>Available analyzers, when configuring mappings and searches.</li>
</ul>
<p>We will eventually teach Play to autocomplete everything, including suggesters, <a href="https://github.com/elasticsearch/elasticsearch/issues/3300">aggregations</a>, etc.
</p>
<p>Knowing the location and the context of the cursor is also used e.g. to highlight the results for the search you are currently working on, or show how existing text in your sample documents is currently being tokenized when editing mappings.
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4ca20342dfd738263492/download?uid=blta607ec907a3cd19f?uid=blta607ec907a3cd19f" data-sys-asset-uid="bltffa02ede4d42e4f7" alt="Example sanity checking &#x2013; before it becomes a performance problem" style="max-width:100%;"><figcaption>Example sanity checking &#x2013; before it becomes a performance problem</figcaption>
</figure>
<p>Furthermore, we want to be able to highlight warnings and errors as they happen. The knowledge base is not comprehensive for the time being, though. You will get a little warning if you try to do inefficient filters, such as when using a top level filter without any facets. Or when you want to be using a <code>bool</code> filter and not an <code>and</code> as explained in Zachary Tong&#x2019;s article on <a href="http://www.elasticsearch.org/blog/all-about-elasticsearch-filter-bitsets/">filter bitsets</a>.
</p>
</section><section id="working-with-text" class="level2">
<h2><a href="#working-with-text">Working with Text</a></h2>
<p>As explained in our article on <a href="https://www.found.no/foundation/elasticsearch-from-the-bottom-up/">Elasticsearch from the bottom up</a>, getting the text processing right is a very important part of working with search.
</p>
<p>To make it easy to work with analyzers, tokenizers, token filters and so on, we&#x2019;ve made an analysis editor that shows step by step how text is processed.
</p>
<p>The image below shows how the input e.g. &#x201C;John Smith&#x201D; is first tokenized into <code>[John, Smith]</code> by the <code>standard-tokenizer</code> before each term is subsequently filtered by the <code>double_metaphone</code>-filter. Tokens are displayed such that overlapping tokens will be displayed as such. You can also hover them to see other overlapping tokens.
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4ca19985d58976376daf/download?uid=blt8f2f9bc24b047065" data-sys-asset-uid="blt8f2f9bc24b047065" alt="Sample text processing output" style="max-width:100%;"><figcaption>Sample text processing output</figcaption>
</figure>
<p>For a dive into analyzers, here&#x2019;s a great read: <a href="https://www.found.no/foundation/text-analysis-part-1/">All About Analyzers, Part One</a>
</p>
</section><section id="how-it-works" class="level2">
<h2><a href="#how-it-works">How it Works</a></h2>
<p>Here&#x2019;s exactly what happens when you run/execute a Play: An API-request consisting of all sample documents, searches, analyzers and mappings is sent to a backend running on Found&#x2019;s servers. This backend uses a pool of in-memory Elasticsearch servers, and for every request &#x2026;
</p>
<ol type="1">
	<li>Creates index templates with the correct analysis and mappings configurations.</li>
	<li>Indexes all the documents. If a document causes the creating of an index, the index is created after the template made in the previous step.</li>
	<li>Gets the resulting mapping, which is a combination of what Elasticsearch guesses and what&#x2019;s specified by the user.</li>
	<li>Runs all the searches and gets the results.</li>
	<li>Deletes all the indexes.</li>
</ol>
<p>This all happens in memory and usually takes just a few tenths of a second.
</p>
</section><section id="can-i-run-it-on-my-cluster" class="level2">
<h2><a href="#can-i-run-it-on-my-cluster">Can I Run it on My Cluster?</a></h2>
<p>Not yet.
</p>
<p>We are committed to open source Play, with the same license used by Elasticsearch, i.e. <a href="https://github.com/elasticsearch/elasticsearch/blob/master/LICENSE.txt">APLv2</a>.
</p>
<p>That said, the Play environment is not something you&#x2019;d want to have running on a live cluster. The &#x201C;create lots of indexes and then delete them&#x201D; process causes lots of changes to the cluster state, and it will not be very fast when things need to hit disk.
</p>
<p>However, the search view with mapping aware autocompletion and documentation is very useful on a live cluster. We intend to provide that functionality as a separate plugin that you can run on your cluster.
</p>
<p>What you can do is export the resulting state of Play to a live cluster, using the save/export-pane. That&#x2019;ll provide you with a shell-script you can run.
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4ca10342dfd738263478/download?uid=blt61e3b55d847ca819" data-sys-asset-uid="blt61e3b55d847ca819" alt="Exporting the state of Play to another cluster" style="max-width:100%;"><figcaption>Exporting the state of Play to another cluster</figcaption>
</figure>
</section><section id="limitations" class="level2">
<h2><a href="#limitations">Limitations</a></h2>
<p>There are some things you cannot do with Play, at least not at present:
</p>
<ul>
	<li>Dynamic scripts are disabled.</li>
	<li>The sum of all documents, searches, mappings and analyzers cannot exceed 2 MiB.</li>
	<li>Shards and custom routing cannot be specified at this time.</li>
</ul>
</section><section id="sharing" class="level2">
<h2><a href="#sharing">Sharing</a></h2>
<p>If you want to save your work, it will be saved as a <a href="https://gist.github.com/">Gist</a>. You can chose whether it should be private (which is the default) or public. Public gists can be found by other users through the &#x201C;Explore&#x201D;-tab.
</p>
<p>Play lets you authenticate with Github. Once signed in you can easily find your own private gists, as well as save any changes to your own gists. If you edit an existing gist you do not own, a fork will be made first.
</p>
</section><section id="roadmap" class="level2">
<h2><a href="#roadmap">Roadmap</a></h2>
<p>We have many plans for Play. This is merely an initial preview release. There are known bugs, but with the amount of &#x201C;When can I get to use it?!&#x201D; feedback we have received, we wanted to make a preview available sooner rather than later.
</p>
<p>As mentioned, Play will be available as an open source project. There&#x2019;s some work ahead to decouple it from internal tools and some general cleanups that needs to happen before we can publish it.
</p>
<p>Having said that, here&#x2019;s a few things we plan to do:
</p>
<ul>
	<li>Integrate the official documentation, both for the search, view and mapping views.</li>
	<li>Add more autocompletion, such as suggesters, decay scoring, and the upcoming aggregation feature.</li>
	<li>Make things commentable. We&#x2019;re considering using gist&#x2019;s commenting feature for this.</li>
	<li>Enable viewing of a gist&#x2019;s history and loading an old version right from Play.</li>
	<li>Refactor the search view and turn it into an Elasticsearch plugin you can use on your own cluster.</li>
	<li>Collaboration with <a href="https://togetherjs.com/">TogetherJS</a></li>
</ul>
<p>If you have any comments, feedback, thoughts, ideas or constructive criticism, don&#x2019;t hesitate to reach out: tweet<span class="citation" data-cites="elastic"><a href="#fn1" class="footnoteRef" id="fnref1"><sup>1</sup></a></span>(https://twitter.com/elastic) or mail <a href="mailto:play@found.no">play@found.no</a>!
</p>
</section></body></html>

Presenting Play: A Preview of an Elasticsearch Playground

<html><head></head><body><p>Today we&#x2019;re bringing you another story of Elasticsearch&#x2019;s use in the field: Vodori and Pepper, their home grown enterprise content management system based on Alfresco and Elasticsearch. <span style="text-decoration: underline;"><a href="http://www.vodori.com/index.html">Vodori</a></span> is a full service digital agency that blends Strategy, Design, Technology, and Product to deliver high-velocity digital global marketing solutions. They are based in Chicago, Illinois, US and they&#x2019;re the gracious hosts of the <span style="text-decoration: underline;"><a href="http://www.meetup.com/Elasticsearch-User-Group-Chicago/">Chicago Elasticsearch Meetup</a></span>. Grant Gochnauer Vice President and Co-Founder of Vodori was kind enough to share their story with us.</p>
<p>Two and a half years ago, Vodori set out to redesign its enterprise content management system, Pepper, to delivery high-velocity digital marketing to our customers. With a global scale and complex content management and distribution requirements, Pepper had to be flexible, scalable, and of course provide rich search capabilities. In 2010, we evaluated technologies that would enable us to meet our product goals and decided on two core technologies: Alfresco and Elasticsearch.</p>
<h2>Why Elasticsearch?</h2>
<p>Before diving into our decision to use Elasticsearch, it helps to understand a bit more about some of our requirements. Content management systems base their data storage on an <a href="http://alistapart.com/article/content-modelling-a-master-skill">&#x201C;object model&#x201D; or &#x201C;content model&#x201D;</a> allowing developers to specify how content types are defined and relate to one another in a system. For example, we might have an object called &#x201C;Document&#x201D; that contains properties such as &#x201C;title&#x201D; and &#x201C;filename&#x201D;. We can then extend &#x201C;Document&#x201D; with a child object in our object model &#x2013; a &#x201C;WebDocument&#x201D;. It has additional properties such as &#x201C;seoDescription&#x201D; and &#x201C;url&#x201D;. &#x201C;WebDocument&#x201D; inherits the properties defined on &#x201C;Document&#x201D;.</p>
<p>Pepper provides an object model as part of the core product but allows (and encourages) implementation teams to extend this model depending on the business requirements for the solution being delivered. Alfresco provides its own &#x201C;object model&#x201D; out of box and we extend this concept significantly for Pepper in our searching paradigm. Because project implementations can contain widely varied content types and properties, we have to support the myriad ways our customers may want to search and filter content.</p>
<p>While Alfresco implements CMIS, the industry standard for interacting with content data (creating, updating, searching), the performance did not meet Pepper&#x2019;s specific needs.  Our challenge: how to get content out of Alfresco (and by extension Pepper) in such a way that:</p>
<ul>
<li>Implementation teams could write complex queries based on client-specific business requirements</li>
<li>The Pepper team could support more robust control on our searches including sorting, faceting, and paging.</li>
<li>We could loosely couple Pepper&#x2019;s user interface and business logic to the underlying data stores (Alfresco and other relational stores), enabling more elegant evolution of each tier.</li>
</ul>
<p>Enter Elasticsearch. After evaluating a number of technology options, we selected Elasticsearch because it met our requirements and also because of the trust we had in the people and product gained through the successful use of Shay Banon&#x2019;s previous technology, Compass. By introducing this facade layer on top of Alfresco, we not only introduced a high-performance and scalable way to interact with our data, it also gave us complete control of how we search content.</p>
<h2>Let&#x2019;s look at an example: Pepper Library</h2>
<p>One of the many ways users can find and interact with content is through the Pepper Library.</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4c923dc542cb38c0836a/download?uid=bltf525438b27a8acc5?uid=bltf525438b27a8acc5" data-sys-asset-uid="bltc71083056410dff0" width="421" style="max-width:100%;"></p>
<p>We allow customers to toggle a number of parametric filters as the results appear through an endless scroll list. Elasticsearch provides everything we need to implement this user experience out of the box.</p>
<p>When users toggle filters along the left panel, we simply construct the appropriate Elasticsearch query using the TermsFilter combined with a BoolFilter and Elasticsearch immediately provides the data we need.</p>
<p>We also provide for free-text search by using the search box in the upper right corner of the Library. Thankfully this is also extremely easy to do with Elasticsearch &#x2013; even when searching with the left panel filters applied. If a user types in a search query, we simply append a Match Query to the existing BoolFilter in our previous query. Elasticsearch happily returns what we asked for. We can further customize the way free-text search works by implementing custom Index and Search analyzers for each property.</p>
<h2>Modelling Content in Elasticsearch</h2>
<p>One of the challenges we faced when building our architecture was making it easy for product engineering to create content models without worrying about the details of Elasticsearch. Similarly, we wanted to provide a layer of abstraction so that our implementation teams could focus on the client solution and not on the intricacies of Elasticsearch mapping and index creation.</p>
<p>We built a framework that allows us to annotate our Java content object model in such a way that generates Elasticsearch mappings that abstract the complexities from our engineers. Our framework provides the contract between our product API&#x2019;s and Elasticsearch.</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4c929e7a83165490c239/download?uid=blta7cb49036a039db1?uid=blta7cb49036a039db1" data-sys-asset-uid="bltb6f2fb1af45872e9" width="608" style="max-width:100%;"></p>
<p>In this example, &#x201C;logicalName&#x201D; represents the filename of a document exposed to the end-user which is different than the system generated physical filename we store in Alfresco. With our @ElasticSearchProperty annotation, we can specify the following:</p>
<ul>
<li>type: This is the field type that Elasticsearch uses to define how this value is indexed</li>
<li>filterable: Whether we want to be able to use Elasticsearch Filters on this value. When set to true, we don&#x2019;t want to analyze the values.</li>
<li>freeTextSearchable: Whether we want to index the analyzed value for free text searching.</li>
<li>searchAnalyzer: The custom analyzer we use when searching.</li>
<li>indexAnalyzer: The custom analyzer we use when indexing the document.</li>
</ul>
<p>When using a combination of &#x201C;filterable&#x201D; and &#x201C;freeTextSearchable&#x201D; we index the value in a &#x201C;multi_field&#x201D; type in Elasticsearch so that we can use either the analyzed or non-analyzed value. Our framework is smart enough to know when to use each.</p>
<h2>Parent child relationships</h2>
<p>The other Elasticsearch modelling feature we use heavily is Parent/Child relationships. We needed to support the ability to manage and distribute a single document in many different distribution channels (Website, iPad, social media, etc). By leveraging the Parent/Child relationships, we can index a single source document which represents the canonical binary asset and an &#x201C;index card&#x201D; that represents a pointer back to the source document with additional descriptor metadata. We can then create additional index card objects in each channel of distribution, all pointing back to the same source document. The Index card becomes the parent document with the source document becoming the child document.</p>
<p>Our framework also extends this idea with another annotation on the child source document:</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4c92d8edd48f76935e90/download?uid=bltf5bfb53420b0db54?uid=bltf5bfb53420b0db54" data-sys-asset-uid="bltace37fa150ccf40c" width="780" style="max-width:100%;"></p>
<p>So why would we want to use a parent/child relationship in this way? Among other things, it allows us to write really interesting queries to fetch all content for a distribution channel while also filtering on values in the source document. Let&#x2019;s look at two examples.</p>
<p>Our customers store product marketing and catalog data within Pepper. As we mentioned earlier, this content is leveraged across many different distribution channels. As a global marketing manager, I may need to search across the entire content ecosystem and find where a particular set of documents are being used. With Elasticsearch and our Parent/Child relationships, we can execute queries that answer this question by leveraging the <i>has_child</i> filter. We simply search for child documents with a specific attribute value (e.g. productCatalogId) and Elasticsearch will return all the parent documents. Parent documents represent our &#x201C;index card&#x201D; which is specific to a distribution channel and contains specific usage metadata for that channel.</p>
<pre class="prettyprint noescape lang=json">{
   &quot;from&quot;:0,
   &quot;size&quot;:2,
   &quot;filter&quot;:{
      &quot;has_child&quot;:{
         &quot;type&quot;:&quot;com.vodori.pepper.content.model.document.PepperWebpage&quot;,
         &quot;query&quot;:{
            &quot;term&quot;:{
               &quot;productCatalogId&quot;:&quot;CRX1234A&quot;
            }
         }
      }
   }
}</pre>
<p>We also need to be able to serve and query content by distribution channel. For example, when our iPad application, Pepper Mobile, needs to fetch all the content approved for consumption from within the iOS application, we can perform this search using the <i>has_parent </i>filter. In this case, we want to find child &#x201C;source documents&#x201D; where the parent object has a specific attribute value such as a distributionChannelId value of 5.</p>
<pre class="prettyprint noescape lang=json">{
   &quot;from&quot;:0,
   &quot;size&quot;:2,
   &quot;filter&quot;:{
      &quot;has_parent&quot;:{
         &quot;type&quot;:&quot;com.vodori.pepper.content.model.descriptor.PostDescriptor&quot;,
         &quot;query&quot;:{
            &quot;term&quot;:{
               &quot;distributionChannel&quot;:&quot;5&quot;
            }
         }
      }
   }
}</pre>
<h2>What&#x2019;s next for Pepper&#x2019;s usage in Elasticsearch?</h2>
<p>We have a number of new features on our roadmap that will be based on Elasticsearch. A few examples include:</p>
<ul>
<li><span style="text-decoration: underline;">Index Aliasing</span>: This will allow us to rebuild our Elasticsearch Index while the existing index remains in service. Once complete, we can simply update the alias so that Pepper points to the newly rebuilt index. Zero downtime.</li>
<li><span style="text-decoration: underline;">Updating Mappings API</span>: Elasticsearch provides capabilities to update its mappings on the fly and so when we release new versions of Pepper, it will be smart enough to inspect the current mappings and apply an update to the index based on changes in the Product version requirements.</li>
<li><span style="text-decoration: underline;">Distributed Percolator:</span> We are very excited about this 1.0 feature because it will allow us to build a mechanism for our customers to subscribe to content updates in the system and be notified of any kind of relevant changes or compliance issues that they specify. It allows our products to be proactive about what&#x2019;s going on in the system based on arbitrary business criteria defined by our customers. Very exciting.</li>
</ul>
<h2>Wrapping up</h2>
<p>It has been really exciting to see the Elasticsearch product and team grow over the last two years. The Elasticsearch community has been fantastic and really reinforces the fact that we made the right decision on our choice to use Elasticsearch. We think that this technology will continue its growth into new industries and applications as more companies realize the power and flexibility of Elasticsearch. We are excited to be a part of that growth.</p>
<p><em>Many thanks to Grant Gochnauer for sharing their experiences with us. We are always excited to share stories from the community about how they are using Elasticsearch, so please <a href="http://twitter.com/lhawthorn">reach out to me</a> any time you&#x2019;d like to share you story.</em></p>
<p>&#xA0;</p></body></html>

Building Enterprise Content Management Systems on Elasticsearch

<p>
	Welcome to&nbsp;
	<i>This week in Elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.</p><h3>Elasticsearch core</h3><ul>
	<li>We released <a href="http://www.elasticsearch.org/blog/1-0-0-beta1-released/">Elasticsearch 1.0.0.beta1</a> mere hours ago. Check it out and <a href="http://twitter.com/elasticsearch">let us know</a> what you think!</li>
	<li>Monday, we <a href="http://www.elasticsearch.org/blog/0-90-6-released/">released 0.90.6</a>, the current stable release of the 0.90 series. Upgrade is recommended.</li>
	<li>
	The default analyzer will not remove stopwords in <code>1.0</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/3775">#3775</a> , master only)
	</li>
	<li>Lots of improvements in the Cat API (commits <a href="https://github.com/elasticsearch/elasticsearch/commit/f16eb7a243185a0db423f83174f4644b8e2a5e0b">1</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/5d46e6915437a41ccbabe31053e24d80dda60710">2</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/f9a17263007afd43d51e5dc7a82094d07e629c99">3</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/7d2c4afa9826dd84a156a0fd01247956ccd56962">4</a>, issues <a href="https://github.com/elasticsearch/elasticsearch/issues/4016">#4016</a>, <a href="https://github.com/elasticsearch/elasticsearch/issues/4020">#4020</a>, <a href="https://github.com/elasticsearch/elasticsearch/issues/4050">#4050</a>, master only)</li>
	<li>
	Removed index reader warmer introduced in 0.90.6 (<a href="https://github.com/elasticsearch/elasticsearch/issues/4078">#4078</a>, 0.90 & master)
	</li>
	<li>
	Fixed a Lucene regression that prevented documents deletions from happening (<a href="https://github.com/elasticsearch/elasticsearch/issues/4093">#4093</a>, 0.90 & master)
	</li>
	<li>A new API to get the mapping for specific fields has been added (<a href="https://github.com/elasticsearch/elasticsearch/issues/3941">#3941</a>, 0.90 & master)</li>
	<li>Performance of adding new fields to the mapping has been improved dramatically (<a href="https://github.com/elasticsearch/elasticsearch/commit/5cd56e5cea5ac3a41527b5185449d78baad85d18">commit</a>, 0.90 & master)</li>
	<li>Introduced a new <code>ImmutableOpenMap</code> class inspired by the optimizations added to mapping processing (commits <a href="https://github.com/elasticsearch/elasticsearch/commit/c63741db04b14fb73562c18a15abefc4781b17ad">1</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/f7205313283312456fd84845137b78e3c668422d">2</a>, 0.90 & master)
	</li>
	<li>
	Put mappings requests are now rejected if the root node does not have the same name as the type specifed in the URI (<a href="https://github.com/elasticsearch/elasticsearch/issues/4038">#4038</a>, master only)
	</li>
	<li>The FieldData API has been simplified (<a href="https://github.com/elasticsearch/elasticsearch/commit/f01d3a18adcab2f8a9a229300943f31046db581a">commit</a>, 0.90 & master)</li>
	<li>Improvements to the multi percolate api (commits <a href="https://github.com/elasticsearch/elasticsearch/commit/65fc90728e4d7ab82e38ea4df4bc33bc63431005">1</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/d660b18298fcdd13343494d2492bff8be7727253">2</a>, master only)</li>
	<li>
	The distributed percolator stores now queries in the <code>.percolator</code> instead of the <code>_percolator</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/4090">#4090</a>, master only)
	</li>
	<li>
	Scripts are now automatically reloaded from the <code>config/scripts</code> directory (<a href="https://github.com/elasticsearch/elasticsearch/issues/4062">#4062</a>, 0.90 & master)
	</li>
	<li>
	Added support for the routing URI parameter to Bulk api and Multi search api  (<a href="https://github.com/elasticsearch/elasticsearch/issues/4053">#4053</a>, <a href="https://github.com/elasticsearch/elasticsearch/issues/4058">#4058</a>, 0.90 & master)
	</li>
	<li>
	Improvements to the recently introduced postings highlighter (<a href="https://github.com/elasticsearch/elasticsearch/issues/4042">#4042</a>, <a href="https://github.com/elasticsearch/elasticsearch/issues/4103">#4103</a>, 0.90 & master)
	</li>
	<li>
	Improvements to our release process (<a href="https://github.com/elasticsearch/elasticsearch/commit/35f33c4bbc1e1026b277c0c10f434faac8d7d51c">commit</a>, 0.90 & master)
	</li>
	<li>
	<code>indices.recovery.concurrent_small_file_streams</code> is now a dynamic settings (<a href="https://github.com/elasticsearch/elasticsearch/issues/4094">#4094</a>, 0.90 & master)
	</li>
	<li>
	Improved behaviour when updating the cluster state and no acknowledgement is needed or the request timeout is set to 0 (<a href="https://github.com/elasticsearch/elasticsearch/commit/afa0ab62261d9a46c77a8bea47679ffe5e743ddc">commit</a>, 0.90 & master)
	</li>
	<li><code>RegexpQueryBuilder</code> now implements <code>MultiTermQueryBuilder</code>, in order to support span multi term queries (<a href="https://github.com/elasticsearch/elasticsearch/issues/3392">#3392</a>, 0.90 & master)</li>
	<li>The +index notation without a wildcard in the index list was handled inconsistently (<a href="https://github.com/elasticsearch/elasticsearch/issues/3979">#3979</a>, 0.90 & master)</li>
	<li>
	Fixed alias resolution issue using the more like this query (<a href="https://github.com/elasticsearch/elasticsearch/issues/4045">#4045</a>, 0.90 & master)
	</li>
	<li>Improved timeout handling when waiting for primary shard to be active for indexing (<a href="https://github.com/elasticsearch/elasticsearch/commit/c68016bb83e2aad815b4bf2fe2b374a06b7fd9e2">commit</a>, 0.90 & master)</li>
	<li>Fixed concurrency issue in simple id cache (<a href="https://github.com/elasticsearch/elasticsearch/commit/81089fb22848313324c8f5b8e96abde90f1f6029">commit</a>, 0.90 & master)</li>
</ul><h3>Elasticsearch Ecosystem</h3><p>
	Here's some more information about what is happening in the ecosystem we are maintaining around Elasticsearch, including plugin and driver releases, as well as news about Logstash and Kibana.
</p><ul>
	<li>Check out the latest <a href="https://github.com/Mpdreamz/NEST/releases/tag/0.11.7.0">NEST client release</a>, which fixes some performance problems.</li>
	<li>ES Nozzle, a framework for connecting repositories like file systems or mail servers to Elasticsearch, has been <a href="http://brainbot.com/es-nozzle/doc/">released in version 0.3.11</a>.</li>
	<li><a href="http://twitter.com/rashidkpc">Rashid Khan</a> treated us to an update on all the latest and greatest in the release of <a href="http://www.elasticsearch.org/blog/kibana-3-milestone-4/">Kibana 3: Milestone 4</a>.</li>
	<li><a href="http://twitter.com/clintongormley">Clinton Gormley</a> has just released a <a href="https://github.com/elasticsearch/elasticsearch-perl-netcurl">NetCurl Cxn class for Elasticsearch</a>.</li>
	<li>Hendrik Saly has <a href="https://github.com/salyh/elasticsearch-security-plugin">created a plugin</a> to add security functionality to Elasticsearch.</li>
	<li>JoliCode has <a href="http://jolicode.com/blog/elasticsearch-comme-nouveau-standard-de-la-recherche-applicative">written an article</a> on Elasticsearch as the new standard for application research. (en français)</li>
	<li>Thomas Ardal <a href="http://thomasardal.com/running-elasticsearch-in-a-cluster-on-azure/">created a how to</a> on running Elasticsearch on Azure for Windows users.</li>
	<li>Mogens Heller Grabe has <a href="http://mookid.dk/oncode/archives/3518">written up instructions</a> for running Elasticsearch on Azure for Ubuntu users.</li>
	<li>Bogdan Dumitrescu <a href="http://blog.trifork.com/2013/10/31/java-clients-behavior-during-creating-a-split-brain-situation-in-elasticsearch/">authored a post</a> on Java Clients Behavior During a Split-Brain Situation in Elasticsearch.</li>
	<li>Jordi Llonch <a href="https://github.com/llonchj/django-tastypie-elasticsearch">created a Django application</a> to provide Elasticsearch support for Tastypie.</li>
	<li>Local Measure <a href="http://blog.getlocalmeasure.com/elasticsearch-for-metrics/">shared details</a> on how they use Elasticsearch for metrics collection.</li>
	<li>NET-A-PORTER wrote up details on how they're using Logstash and Kibana on their <a href="http://techblog.net-a-porter.com/2013/11/making-sense-of-your-logs-with-logstash-1-2-2-and-kibana-3/">technical blog</a>.</li>
	<li><a href="http://twitter.com/zacharytong">Zachary Tong</a> wrote on article on how to make <a href="http://webandphp.com/stretching-your-data-with-elastisearch-php-driver">best use of the Elasticsearch PHP client</a>.</li>
	<li><a href="https://twitter.com/thnetos">Lee Hinman</a> <a href="http://writequit.org/articles/literate-es-scripts.html">created a tutorial</a> on Writing literate-programming style Elasticsearch shell scripts with Emacs.</li></ul><h3>Slides</h3><p>
	Check out 
	<a href="http://twitter.com/imotov">Igor Motov's</a> presentation on <a href="https://speakerdeck.com/elasticsearch/new-features-in-elasticsearch-v1-dot-0">New Features in Elasticsearch 1.0</a> and <a href="http://twitter.com/drewr">Drew Raines'</a> slides on <a href="https://speakerdeck.com/elasticsearch/consistency-and-elasticsearch">Elasticsearch and Data Consistency</a>. Both presentations were delivered at this week's <a href="http://www.meetup.com/Elasticsearch-Boston/events/146252052/">Boston Meetup</a>.
</p><h3>Where to find Us</h3><h4>Australia</h4><p>
	Clinton Gormley will be speaking at the inaugural 
	<a href="http://www.meetup.com/Elasticsearch-Sydney-Meetup/events/149068632/">Sydney Elasticsearch Meetup</a> on November 18th and the <a href="http://www.meetup.com/melbourne-search/events/148371692/">Melbourne Search Users Group</a> on November 21st. Clinton will cover new features forthcoming in 1.0 and will be joined by other speakers from the Melbourne and Sydney community. We'll have full details for you next week.
</p><h4>Belgium</h4><p>
	If you're heading to Devoxx Belgium 2013, be sure to check out 
	<a href="http://twitter.com/dadoonet">David Pilato</a> on <a href="http://www.devoxx.be/dv13-david-pilato.html">Elastify Your App: From SQL to NoSQL in Less than One Hour</a>. David will speak on Wednesday, November 13th at 3:10 PM in Room 6. David will also give a lightning talk, <a href="http://www.devoxx.be/dv13-david-pilato.html?presId=3282">Make Sense of Your (BIG) Data</a> on Thursday, November 14th at 1:10 PM in Room 9. Devoxx Belgium runs from November 11-15 at the Metropolis Business Center in Antwerp.
</p><h4>Estonia</h4><p>
	<a href="http://twitter.com/HonzaKral">Honza Kral</a> will explore <a href="http://www.topconf.com/Conference/Abstracts2013/BeyondSearch/tabid/261/language/en-US/Default.aspx">Beyond Search</a> at <a href="http://www.topconf.com/">Topconf</a> on Thursday, November 7th at 3:30 PM. Topconf runs November 6th and 7th in Tallinn.
</p><h4>France</h4><ul>
	<li>David Pilato wil speak on <a href="http://nightclazz-es-avance.eventbrite.fr/">Advanced Elasticsearch and Kibana</a> at Zenika's Night Class on November 7th. The class begins at 7:00 PM.</li>
</ul><h4>Germany</h4><ul>
	<li>Several Elasticsearchers will be attending the <a href="http://jax.de/wjax2013/">W-JAX Conference</a> in Munich from November 4th to the 8th. You can stop by our booth to visit with <a href="http://twitter.com/spinscale">Alexander Reelsen</a> and <a href="https://twitter.com/a2tirb">Britta Weber</a>, and make sure to catch Alex's talk on <a href="http://jax.de/wjax2013/sessions/elasticsearch-beyond-full-text-search">Elasticsearch: Beyond full-text search</a> on Wednesday, November 6th at 10:15 AM.</li>
	<li>From W-JAX, Alex and Britta will head to Hamburg for <a href="http://developer-conference.eu">Developer-Conference.eu</a> on Thursday and Friday, November 7th and 8th. Alex will once again present on <a href="http://www.developer-conference.eu/session_post/elasticsearch-beyond-full-text-search/">Elasticsearch: Beyond full-text search</a> on Thursday at 1 PM in the Cinema Three room.</li>
	<li>Alex also organizes the <a href="http://spinscale.github.io/munich-search.html">Munich Search Meetup group</a>, and will be hosting a meetup on November 20th. The meetup begins at 6:45 PM.</li>
</ul><h4>Ireland</h4><p>
	<a href="http://twitter.com/jpountz">Adrien Grand</a> will present <a href="http://lucenerevolution.org/sessions#Adrien_Grand">What is in a Lucene index?</a> at <a href="http://lucenerevolution.org/">Lucene Revolution</a> on Thursday, November 7th at 2:45 PM. Lucene Revolution takes place November 4th - 7th in Dublin.
</p><h4>Japan</h4><p>
	The Elasticsearch User Group in Japan will meet for a 
	<a href="http://elasticsearch.doorkeeper.jp/events/6532">study session</a> on November 12th in Tokyo. The meeting begins at 7 PM.
</p><h4>Netherlands</h4><p>
	Elasticsearch will be at the 
	<a href="http://cloudstackcollab.org/">CloudStack Collaboration Conference EU</a> on November 20-22nd in Amsterdam. Stop by our table to say hello, and make sure to catch <a href="http://twitter.com/bleskes">Boaz Leskes</a> and Gerard de Vos' workshop on Elasticsearch, taking place on Wednesday, November 20th.
</p><h4>Norway</h4><p>
	<a href="http://twitter.com/bleskes">Boaz Leskes</a> and <a href="http://twitter.com/karmiq">Karel Minarik</a> will speak at the <a href="http://www.meetup.com/elasticsearch-oslo/events/146914192/">Elasticsearch Oslo Meetup</a> on November 26th. Full details on their talks and other presentations by community members in Oslo will be published next week.
</p><h4>Spain</h4><p>
	<a href="http://twitter.com/clintongormley">Clinton Gormley</a> will speak at the NoSQL matters conference in Barcelona on <a href="http://2013.nosql-matters.org/bcn/abstracts/#abstract_287512263">Getting Down and Dirty with Elasticsearch</a>. Clinton's talk takes place on Saturday, November 30th at 10:30 AM, and the conference runs November 29-30th.
</p><h4>United Kingdom</h4><ul>
	<li>We'll be out in force at <a href="http://strataconf.com/strataeu2013">Strata London</a> from November 11-13th, with both <a href="http://twitter.com/kimchy">Shay Banon</a> and <a href="https://twitter.com/costinl">Costin Leau</a> attending the conference. You can stop by the Elasticsearch booth to meet with both of them during the conference. Shay will present at the <a href="http://strataconf.com/strataeu2013/public/schedule/detail/32102">Big Data London Meetup</a>, taking place on the first evening of Strata London.</li>
	<li>Shay and Costin will also be attending the <a href="http://www.meetup.com/London-ElasticSearch-User-Group/events/144492702/">London Elasticsearch Meetup</a> on November 11th, where they'll be available for audience Q&A following presentations from <a href="https://twitter.com/tackers">Graham Tackley</a> with The Guardian and <a href="https://twitter.com/andrew_clegg">Andrew Clegg</a> with Pearson. The meetup will start at 6:30 PM at Red Badger's offices.</li>
	<li><a href="http://twitter.com/s1m0nw">Simon Willnauer</a> will present <a href="http://skillsmatter.com/podcast/nosql/with-a-hammer-in-your-hand-elasticsearch">Elasticsearch: With a Hammer in Your Hand</a> at the NOSQL eXchange in London on November 28th. Simon's presentation begins at 1 PM.</li>
</ul><h4>United States</h4><ul>
	<li>Martijn will present at a <a href="http://www.meetup.com/Elasticsearch-Washington-DC/events/129113592/">Lunch and Learn session</a> on New Features in Elasticsearch 1.0 at Verizon's offices in Ashburn, Virginia on Thursday, November 7th. Our hosts, Verizon, will also treat the audience to information on their use of Elasticsearch. Doors open at 11:30 AM; please arrive early to give yourself plenty of time to check in at reception.</li>
	<li><a href="https://twitter.com/kevinkluge">Kevin Kluge</a> will be speaking at the <a href="http://www.meetup.com/sfruby/events/144512912/">San Francisco Ruby Meetup</a> on November 7th. Doors open at 7:00.</li>
	<li><a href="http://twitter.com/rashidkpc">Rashid Khan</a> will be attending <a href="http://nov2013.desertcodecamp.com/">Desert Code Camp</a> in Phoenix, Arizona on Saturday, November 9th. If you'll also be there, have a chat with him on all things Kibana.</li>
	<li>You can learn all about how Atlassian uses Elasticsearch for <a href="http://www.meetup.com/Elasticsearch-San-Francisco/events/141698772/">Heavy lifting in HipChat</a> at the San Francisco Bay Area meetup on Monday, November 18th. Atlassian will be hosting the meetup at their offices and doors open at 6:30 PM.</li>
	<li><a href="http://twitter.com/uboness">Uri Boness</a> and <a href="http://twitter.com/zacharytong">Zachary Tong</a> will be presenting at the <a href="http://www.meetup.com/Elasticsearch-User-Group-Chicago/events/148062252/">Chicago Elasticsearch Meetup</a> on November 18th. Further details coming soon!</li>
</ul><h3>Where to Find You</h3><p>
	Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, 
	<a href="https://docs.google.com/forms/d/1qBZ6Fwq4JWw7FH3s6C9Q3KeGCvUOmWYOvNMsxNqkCNM/viewform">take a moment to let her know</a>.
</p><p>
	Oh yeah, we're also&nbsp;
	<a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you'd like us to find you for employment purposes, just&nbsp;<a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &nbsp;We care more about your skill set and passion for Elasticearch, Kibana and Logstash than where you rest your head.
</p><h3>Trainings</h3><p>
	If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
</p><ul>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/amsterdam/2013-11-11">Amsterdam</a>&nbsp;- November 11, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/chicago/2013-11-18">Chicago</a>&nbsp;- November 18, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/san-francisco-california-united-states/2013-11-21">San Francisco</a>&nbsp;- November 21, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/oslo/2013-11-26">Oslo</a>&nbsp;- November 26, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/paris/2013-12-05">Paris</a>&nbsp;- December 05, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/london-uk/2013-12-09">London</a>&nbsp;- December 12, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/los-angeles/2013-12-16">Los Angeles</a>&nbsp;- December 16, 2013</li>
</ul>

This Week in Elasticsearch - November 06, 2013

<html><head></head><body><p>Today we are delighted to announce the release of <strong>elasticsearch 1.0.0.Beta1</strong>, the first public release on the road to 1.0.0. The countdown has begun!</p>
<p>You can <a href="http://www.elasticsearch.org/downloads/1-0-0-Beta1/" target="_top">download Elasticsearch 1.0.0.Beta1 here</a>.</p>
<p>In each beta release we will add one major new feature, giving you the chance to try it out, to break it, to figure out what is missing and to tell us about it. Your use cases, ideas and feedback is essential to making Elasticsearch awesome.</p>
<p>The main feature we are showcasing in this first beta is <strong>Distributed Percolation</strong>.</p>
<p><strong>WARNING:</strong> This is a beta release - it is not production ready, features are not set in stone and may well change in the next version, and once you have made any changes to your data with this release, it will no longer be readable by older versions!</p>
<h2><a id="_distributed_percolation" href="#_distributed_percolation"></a>Distributed percolation</h2>
<p>For those of you who aren&#x2019;t familiar with percolation, it is &#x201C;search reversed&#x201D;. Instead of running a query to find matching docs, percolation allows you to find queries which match a doc. Think of people registering alerts like: <em>tell me when a newspaper publishes an article mentioning &quot;Elasticsearch&quot;</em>.</p>
<p>Percolation has been supported by Elasticsearch for a long time. In the current implementation, queries are stored in a special <code>_percolator</code> index which is replicated to all nodes, meaning that all queries exist on all nodes. The idea was to have the queries alongside the data.</p>
<p>But users are using it at a scale that we never expected, with hundreds of thousands of registered queries and high indexing rates. Having all queries on every node just doesn&#x2019;t scale.</p>
<p>Enter <em>Distributed Percolation</em>.</p>
<p>In the new implementation, queries are registered under the special <code>.percolator</code> type within the same index as the data. This means that queries are distributed along with the data, and percolation can happen in a distributed manner across potentially all nodes in the cluster. It also means that an index can be made as big or small as required. The more nodes you have the more percolation you can do.</p>
<p>We have removed the ability to percolate while indexing a document, as that didn&#x2019;t scale either. But we&#x2019;ve added a host of new features that makes percolation awesome.</p>
<ul type="disc">
<li>You can percolate a document that has not yet been indexed, or you can percolate an indexed document by specifying the index, type and id.</li>
<li>The multi-percolate API allows you to percolate docs in bulk, saving on network overhead.</li>
<li>Percolation queries can highlight the documents that they match, providing more context to why a particular query matched.</li>
<li>The percolation count API tells you how many queries match the current document.</li>
<li>Facet-support allows you to facet on the metadata of &#xA0;matching queries.</li>
</ul>
<p>Check out the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-percolate.html" target="_top">distributed percolation docs here</a>.</p>
<h2><a id="_doc_values" href="#_doc_values"></a>Doc Values</h2>
<p>The most common problem that users have with Elasticsearch is with fielddata. In order to sort or facet on field values, those values need to be easily accessible. By far the fastest way of accessing field values is by loading them into memory, which is known as fielddata. Especially string fields can use large amounts of RAM and can cause slow garabage collection and even out of memory exceptions.</p>
<p>The other big new feature in this release is the addition of doc values. With this new functionality, field values can be stored on disk at index time, instead of in memory. It&#x2019;s not as fast as holding fielddata in memory - facets take longer because they need to hit disk - but this is achieved with a fraction of the memory, meaning that you can now calculate facets, sort on fields or access field values in scripts for much larger datasets than was previously possible.</p>
<p>Doc values need to be setup in the field mapping at index time, and work on numeric, geo and <code>not_analyzed</code> string values, both single and multi-valued. They cannot be enabled on analyzed string fields as that would require a second analysis phase. This may change in the future.</p>
<p>Check out the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/mapping-core-types.html#_doc_values_format" target="_top">docvalues docs here</a>.</p>
<h2>Stopwords disabled by default</h2>
<p>The <code>standard</code> analyzer, the default analyzer used by Elasticsearch, comes with stopwords enabled. Not only that, it uses English stopwords by default. It doesn&apos;t matter what language your text is in, or which full text field you&apos;re indexing, it&apos;ll have stopwords removed. This can produce unexpected confusion, like wondering why <code>{ &quot;country_code&quot;: &quot;no&quot; }</code> doesn&apos;t match anything, or why you can&apos;t find any plays called &quot;To be or not to be&quot;. With modern hardware and queries like the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-common-terms-query.html">common terms query</a>, stopwords are less useful than they used to be.</p>
<p>This release sets the default stopwords list for the <code>standard</code> analyzer to empty, meaning that no stopwords are removed by default. You can still enable stopwords where you want them, but you can do that where you choose to. For the sake of backwards compatibility, indices created with a previous version of Elasticsearch will continue to use the English stopwords list.</p>
<h2><a id="_still_to_come" href="#_still_to_come"></a>Still to come</h2>
<p>There are three major new features still to come:</p>
<ul type="disc">
<li>snapshot/restore for backing up the whole cluster or individual indices, see <a href="https://github.com/elasticsearch/elasticsearch/issues/3826" target="_top">#3826</a></li>
<li>aggregations, or <em>facets reborn</em>, see <a href="https://github.com/elasticsearch/elasticsearch/issues/3300" target="_top">#3300</a></li>
<li>the <code>_cat</code> api, for easy console-based insight into your cluster (documentation to follow)</li>
</ul>
<p>Please <a href="http://www.elasticsearch.org/downloads/1-0-0-Beta1/" target="_top">download Elasticsearch 1.0.0.Beta1</a>, try it out, and let us know what you think.</p>
</body></html>

1.0.0.Beta1 released

<html><head></head><body><p>Kibana 3: Milestone 4 has been released and brings with it a wealth of performance, usability and visualization enhancements. Let&#x2019;s take a look at a few of the significant changes. If you&#x2019;re still on Milestone 3, check out <a href="/blog/this-week-in-kibana/">this post</a> for even more new features.
</p>
<h3>An all new look</h3>
<p>Kibana&#x2019;s panels have been overhauled to feature more prominent labeling, easier to use buttons and links, and an entirely new style. The changes result in improved usability, as well as a space efficient design that allows for great data density and a more consistent UI.
</p>

	<p><img src="https://api.contentstack.io/v2/assets/575e4c9212798d8e381fd758/download?uid=bltf2329ee8e485a2cd?uid=bltf2329ee8e485a2cd" style="max-width:100%;" data-sys-asset-uid="blta0264c9922c4230d"></p><figcaption>Kibana&#x2019;s new look</figcaption>
	<p></p>

<h3>Consistent query and filter placement</h3>
<p>To improve UI harmony, the query and filter panels get their own collapsible, pull-down sections right under the navigation bar. There is no longer a need to place these essential panels yourself, as they&#x2019;re included in every dashboard by default. Like many of Kibana&#x2019;s features, they can be disabled entirely from the dashboard configuration dialog.
</p>
<h3>100% new time range selector</h3>
<p>If you&#x2019;re familiar with Kibana&#x2019;s 2 year history, you might know that there have been several time picker evolutions. The new time picker is a total rewrite that not only uses less space than the old one, but is simpler and easier to use. By moving this important component off the main dashboard, Kibana now has more room to devote to important data and charts. Plus, the new filter format implements Elasticsearch&#x2019;s <a href="/guide/en/elasticsearch/reference/current/mapping-date-format.html#date-math">date math</a> so there&#x2019;s no need to reselect a time frame to move your time window; every search will update the window automatically.
</p>

	<p><img src="https://api.contentstack.io/v2/assets/575e4c929985d58976376b37/download?uid=blt57afbae29307747c?uid=blt57afbae29307747c" style="max-width:100%;" data-sys-asset-uid="blt122f4252003d45e7"></p><figcaption>An all new time picker</figcaption>
	<p></p>

<h3>Filterable field list</h3>
<p>Find fields quickly and easily with the type-to-filter feature of the table.<br>
</p>

	<p><img src="https://api.contentstack.io/v2/assets/575e4c9243e9adc538713bfb/download?uid=blt90886a1ca9622b89?uid=blt90886a1ca9622b89" style="max-width:100%;" data-sys-asset-uid="blt87ca82aa6e55fa35"></p><figcaption>Filterable field lists</figcaption>
	<p></p>

<h3>Ad-hoc facets</h3>
<p>And, once you&#x2019;ve found those field,s analyze them quickly with ad-hoc terms facets. Simply click a field and select a visualization to see the top 10 term matches for that field.
</p>

	<p><img src="https://api.contentstack.io/v2/assets/575e4c93d8edd48f76935e9f/download?uid=bltbce7e633e096dc2a?uid=bltbce7e633e096dc2a" style="max-width:100%;" data-sys-asset-uid="blt3137b9522031f523"></p><figcaption>Exploration is even easier</figcaption>
	<p></p>


	<p><img src="https://api.contentstack.io/v2/assets/575e4c933dc542cb38c0837c/download?uid=blt6508b4c6554970b6?uid=blt6508b4c6554970b6" style="max-width:100%;" data-sys-asset-uid="blt16f9548c747f16db"></p><figcaption>No need to add a panel. Pie chart on the fly!</figcaption>
	<p></p>

<h3>Dynamic dashboards and URL parameters</h3>
<p>Kibana 3: Milestone 4 can now take input via URL parameters! This much anticipated feature comes in two flavors: templated dashboards and scripted dashboards. Kibana 3: Milestone 4 ships with 2 examples designed to work flawlessly with Logstash and gives you a foundation to build your own dashboards with. Templated dashboards are easy to create by simply exporting your dashboard schema to a file, editing it and adding it to your app/dashboards directory. For example, take this section from logstash.json (<a href="https://github.com/elasticsearch/kibana/blob/v3.0.0milestone4/src/app/dashboards/logstash.json">on github</a>)
</p>
<pre><code>  &quot;0&quot;: {
    &quot;query&quot;: &quot;{{ARGS.query || &apos;*&apos;}}&quot;,
    &quot;alias&quot;: &quot;&quot;,
    &quot;color&quot;: &quot;#7EB26D&quot;,
    &quot;id&quot;: 0,
    &quot;pin&quot;: false
  }
</code>
</pre>
<p>Templated dashboards use &#x201C;handlebar syntax&#x201D; to add dynamic section to the JSON based dashboard scheme. Here we&#x2019;ve replaced the contents of the query key with an expression that translates to: <i>Use the query parameter from the URL, if one is not present, use *</i>. Now we could access this dashboard with a URL such as
</p>
<pre><code>http://kibana.example.com/index.html#/dashboard/<strong>file</strong>/logstash.json?query=extension:zip</code>
</pre>
<h4>More complex scripted dashboards</h4>
<p>Scripted dashboards are an even more powerful method of taking URL parameters, using all of the power of JavaScript to compose a complete dashboard object. An example, logstash.js (<a href="https://github.com/elasticsearch/kibana/blob/v3.0.0milestone4/src/app/dashboards/logstash.js">on GitHub</a>) is included in app/dashboards. Because scripted dashboards are entirely JavaScript, we can perform complex operations, like splitting URL parameters. For example, in this URL we are searching for <em>HTML, CSS or PHP</em>, in the <em>last 2 days</em>, and showing <em>request, response and user agent</em> in the table. Note the important change from <em>file</em> to <em>script</em> in the URL:
</p>
<pre><code>http://localhost:8000/index.html#/dashboard/<strong>script</strong>/logstash.js?query=html,css,php&amp;from=2d&amp;fields=request,response,agent</code>
</pre>
<h3>Go get it now</h3>
<p>Milestone 4 is a big step forward for both implementers and users. It is simultaneously more powerful and more simple. And, of course. Kibana continues to integrate seamlessly with Logstash, including the recently released <a href="/guide">Logstash 1.2.2</a>. Kibana can be downloaded directly from elastic.co <a href="/products/kibana">right here</a>.
</p>
<h4>Also, we&#x2019;re hiring</h4>
<p>Do you want to get paid to work on open source? Are you passionate about JavaScript, with a head full of data visualization ideas? We&#x2019;re way more interested in what you know and what you can do than where you live. <a href="/about/careers">Check out our open positions</a>.
</p></body></html>

Kibana 3: Milestone 4

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>This article introduces the networking part of Elasticsearch. We look at the network topology of an Elasticsearch cluster, which connections are established between which nodes and how the different Java clients works. Finally, we look a bit closer on the communication channels between two nodes.
</p><section id="node-topology" class="level2">
<h2><a href="#node-topology">Node Topology</a></h2>
<p>Elasticsearch nodes in a cluster form what is known as a <a href="http://en.wikipedia.org/wiki/Network_topology#Mesh">full mesh topology</a>, which means that each Elasticsearch node maintains a connection to each of the other nodes.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8d9e7a83165490c152/download?uid=blt2173f39afdb94e27?uid=blt2173f39afdb94e27" data-sys-asset-uid="blt292169d40755cd62" alt="Full mesh topology with 6 cluster nodes">
</p><figcaption>Full mesh topology with 6 cluster nodes</figcaption>
<p>In order to simplify the code base, the connections are used as one-way connections, so the connection topology actually ends up looking like this:
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8d58208ba076e28077/download?uid=blt3058d4e681869e36?uid=blt3058d4e681869e36" data-sys-asset-uid="blt561990b45246d265" alt="6 cluster nodes with connections, each line represents a connection">
</p><figcaption>6 cluster nodes with connections, each line represents a connection</figcaption>
</section><section id="adding-a-node" class="level2">
<h2><a href="#adding-a-node">Adding a Node</a></h2>
<p>When a node starts up, it reads a list of seed nodes from its configuration using <a href="/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#unicast">unicast</a>, and sends <a href="/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#multicast">multicast</a> messages looking for other nodes in the same cluster.
</p>
<p>As the node discovers more instances in the same cluster, it connects to them one by one, asks them for information about all the nodes they know and then attempts to connect to them all and officially join the cluster. In this way, previously running instances assist in quickly getting fresh instances up to speed on the current nodes in a cluster.
</p>
</section><section id="node-clients" class="level2">
<h2><a href="#node-clients">Node Clients</a></h2>
<p>Even <a href="/guide/en/elasticsearch/client/java-api/current/client.html#node-client">&#x201C;client&#x201D; Elasticsearch nodes</a> (i.e nodes configured with <code>node.client: true</code> or build with <code>NodeBuilder.client(true)</code>) connect to the cluster this way.
</p>
<p>This implies that the client node becomes a full participant in the full mesh network. In other words, once it starts joining the cluster, all the existing nodes in the cluster will connect back to the instance. This means that both the client and server firewalls and <a href="/guide/en/elasticsearch/reference/current/modules-network.html">publish hosts</a> must be properly configured in order to allow this. Additionally, whenever a node client joins, leaves or experiences networking issues, it causes extra work for all the other nodes in the cluster, such as opening / closing network connections and updating the cluster state with the information about the node.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8d12798d8e381fd677/download?uid=blt08a91ac57ff257cb?uid=blt08a91ac57ff257cb" data-sys-asset-uid="blt2da54d75934a625e" alt="Node client connecting to a cluster">
</p><figcaption>Node client connecting to a cluster</figcaption>
</section><section id="transport-clients" class="level2">
<h2><a href="#transport-clients">Transport Clients</a></h2>
<p>On the other hand, <a href="/guide/en/elasticsearch/client/java-api/current/client.html#transport-client">&#x201C;transport&#x201D; clients</a> work differently.
</p>
<p>When a Transport client connects to one or more instances in a cluster, the instances do not connect back, nor is the existence of the transport client part of the cluster state. This means that a joining / leaving transport client causes minimum extra work for the other nodes in the cluster.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8d9985d58976376a51/download?uid=bltb810f4f10f5305aa?uid=bltb810f4f10f5305aa" data-sys-asset-uid="bltba355f9bfb2a37a6" alt="Transport client connecting to a cluster">
</p><figcaption>Transport client connecting to a cluster</figcaption>
</section><section id="connections-and-channels" class="level2">
<h2><a href="#connections-and-channels">Connections and Channels</a></h2>
<p>What do we mean when we talk about the <code>connection</code> to a Node in Elasticsearch? In the sections above, we refrained from being specific about it and only used the term as a logical connection that allows communication between two nodes. Let&apos;s go into more detail.
</p>
<p>Usually, when we talk about connections in the context of networks, we refer to <a href="http://en.wikipedia.org/wiki/Transmission_Control_Protocol">TCP</a>-connections, which provide a reliable line of communication between two nodes.
</p>
<p>Elasticsearch uses (by default) TCP for communication between nodes, but in order to prevent important traffic such as fault-detection and cluster state changes from being affected by far less important, slower moving traffic such as query/indexing requests, it creates multiple TCP connections between each node. For each of these connections, Elasticsearch uses the term <code>channel</code>, which encapsulates the serialization / deserialization of messages over a particular connection.
</p>
<p>In earlier Elasticsearch versions there used to be three different classes of channels: <code>low</code>, <code>med</code> and <code>high</code>. After a while, <code>ping</code> was added, and a <a href="https://github.com/elasticsearch/elasticsearch/commit/35b573ff24c1462de77252f53cd60b4dcab23eda">recent change</a> renamed these channels such that they are more descriptive about their intention. At of the time of writing, the following channel classes exist:
</p>
<ul>
	<li><code>recovery</code>: 2 channels for the recovery of indexes</li>
	<li><code>bulk</code>: 3 channels for low priority bulk based operations such as bulk indexing. Previously called <code>low</code>.</li>
	<li><code>reg</code>: 6 channels for medium priority regular operations such as queries. Previous called <code>med</code>.</li>
	<li><code>state</code>: 1 channel dedicated to state based operations such as changes to the cluster state. Previously called <code>high</code>.</li>
	<li><code>ping</code>: 1 channel dedicated to pings between the instances for fault detection.</li>
</ul>
<p>The default number of channels in each of these class are configured with the configuration prefix of <code>transport.connections_per_node</code>.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8c0342dfd7382630f5/download?uid=blt8ac9e7f8ca891434?uid=blt8ac9e7f8ca891434" data-sys-asset-uid="blt09cfbb880e12caf9" alt="All channels in a three-node cluster. Each line represents a single channel">
</p><figcaption>All channels in a three-node cluster. Each line represents a single channel</figcaption>
<p>Elasticsearch has support for <a href="http://tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/">TCP keepalive</a> which is not explicitly set by default. This prevents unused or idle channels from being closed, which would otherwise cascade into a complete disconnect-reconnect cycle as explained above.
</p>
<section id="adding-an-instance" class="level3">
<h3><a href="#adding-an-instance">Adding an Instance</a></h3>
<p>A consequence of the above is that adding a new instance to an existing cluster causes it to establish 13 connections to each node, and each node establishes 13 connections back to the new instance.
</p>
</section>
<section id="linked-channels" class="level3">
<h3><a href="#linked-channels">Linked Channels</a></h3>
<p>If one of the 13 channels to a particular node is closed due to intermittent network errors for example, all the other channels to the same node are closed and the connections to the node is reconnected. This is done in order to maintain some internal invariants and to ensure the internal consistency of the communication between the nodes.
</p>
</section>
</section><section id="ending-remarks" class="level2">
<h2><a href="#ending-remarks">Ending Remarks</a></h2>
<p>The fact that all the channels between two nodes in a cluster are linked makes it extra vulnerable to network issues, and this is one of the reasons why people are discouraged from trying to create a cluster between data centers that are far apart (and thus adding more sources of failure).
</p>
<p>In this article we&apos;ve shown the basic network topology of an Elasticsearch cluster and introduced the concept of <code>channels</code> that are used for communication between nodes. In a later article we&apos;ll take a closer look at what goes on inside each of these channels.
</p>
</section></body></html>

Elasticsearch Internals: Networking Introduction

<html><head></head><body><p>Today we are happy to announce the release of <strong>elasticsearch 0.90.6</strong>, which is based on Lucene 4.5.1.  This is the current stable release in the 0.90 series and we recommend upgrading. You can download it <a class="ulink" href="/downloads/past-releases/elasticsearch-0-90-6" target="_top">here</a>. </p>
<p>In this release there have been big improvements to highlighting including the new <em>postings</em> highlighter, automatic script reloading, better handling of edge cases associated with cluster-level changes, and many small bug fixes and enhancements which you can read about in the <a class="ulink" href="/downloads/past-releases/elasticsearch-0-90-6" target="_top">release notes</a>.</p>
<h2><a id="_postings_highlighter"></a>Postings highlighter</h2>
<p>The new <code class="literal">postings</code> highlighter is faster, requires less disk space than the fast-vector-highlighter, and is &#x201C;sentence aware&#x201D; which should result in more meaningful snippets.</p>
<p>In order to use it, the field mapping must be configured with <code class="literal">index_options</code> set to <code class="literal">&quot;offsets&quot;</code> instead of the default <code class="literal">&quot;positions&quot;</code>. For instance:</p>
<pre class="programlisting prettyprint lang-js">curl -XPUT localhost:9200/my_index -d &apos;
{
    &quot;mappings&quot;: {
        &quot;my_type&quot;: {
            &quot;properties&quot;: {
                &quot;title&quot;: {
                    &quot;type&quot;:          &quot;string&quot;,
                    &quot;analyzer&quot;:      &quot;english&quot;,
                    &quot;index_options&quot;: &quot;offsets&quot;
                }
            }
        }
    }
}
&apos;</pre>
<p>With the mapping setup correctly, we can index some documents:</p>
<pre class="programlisting prettyprint lang-js">curl -XPOST localhost:9200/my_index/my_type/_bulk -d &apos;
{&quot;index&quot;:{&quot;_id&quot;: 1}}
{&quot;title&quot;: &quot;The quick brown fox jumped over the lazy dog&quot;}
{&quot;index&quot;:{&quot;_id&quot;: 2}}
{&quot;title&quot;: &quot;Brown foxes do love jumping, especially over dogs&quot;}
&apos;</pre>
<p>And search them:</p>
<pre class="programlisting prettyprint lang-js">curl -XGET localhost:9200/my_index/my_type/_search?pretty -d &apos;
{
    &quot;query&quot;: {
        &quot;match&quot;: { &quot;title&quot;: &quot;Jumping brown foxes&quot;}
    },
    &quot;highlight&quot;: {
        &quot;fields&quot;: {
            &quot;title&quot;: {}
        }
    }
}
&apos;</pre>
<p>Each result from the above request is accompanied by a nicely highlighted<br>
snippet:</p>
<ul>
<li>&#x201C;The quick <em>brown</em> <em>fox</em> <em>jumped</em> over the lazy dog&#x201D;</li>
<li>&#x201C;<em>Brown</em> <em>foxes</em> do love <em>jumping</em>, especially over dogs&#x201D;</li>
</ul>
<h2><a id="_other_highlighting_improvements"></a>Other highlighting improvements</h2>
<p>Thanks go to Nik Everett, a frequent contributor to Elasticsearch, who has added the ability to specify a separate query just for highlighting (see <a class="ulink" href="/guide/en/elasticsearch/reference/current/search-request-highlighting.html#_highlight_query" target="_top">highlight query</a>) and the ability to return a simple excerpt when there are no words that can be highlighted (see <code class="literal">no_match_size</code> in <a class="ulink" href="/guide/en/elasticsearch/reference/current/search-request-highlighting.html#_highlighted_fragments" target="_top">highlighted fragments</a>).</p>
<h2><a id="_more_accurate_terms_facet"></a>More accurate terms facet</h2>
<p>Because terms facets are calculated by combining the results from multiple shards, it is possible that each shard has a different <em>top-10</em> list, resulting in inaccurate global counts. This release introduces the <code class="literal">shard_size</code> parameter which allows you to fetch more results from each shard, while still returning only <code class="literal">size</code> (default <code class="literal">10</code>) results to the user.  Pulling more results from each shard reduces the inaccuracy in global counts.</p>
<h2>Reload scripts automatically</h2>
<p>Scripts are used in many APIs in Elasticsearch, eg for scoring, script fields, faceting etc.  A script can either be specified in the request itself, or named scripts can be loaded from the <code>config/scripts/</code> directory on each node. Previously, changing configured scripts was a tiresome process which involved updating and restarting all nodes.  Now, a new watcher will check for changes in the scripts directory every 60 seconds (configurable with <code>watcher.interval</code>) and load new scripts, reload changed scripts or delete removed scripts automatically. See <a href="/guide/en/elasticsearch/reference/current/modules-scripting.html#_automatic_script_reloading">Automatic script reloading</a> for more.</p>
<h2><a id="_pretty_is_prettier"></a>Pretty is prettier</h2>
<p>This is a very simple change, but removes a common source of annoyance. Pretty-printed results now have a newline <code class="literal">\n</code> character appended to make console output easier to read.</p>
<p>We hope you enjoy this new release. Please <a class="ulink" href="/downloads/past-releases/elasticsearch-0-90-6" target="_top">download and use 0.90.6</a>, and let us know what you think.</p></body></html>

0.90.6 Released

<html><head></head><body><h2>What You Need to Know</h2><p>We&apos;ve got some exciting news to share around Elasticsearch and Hadoop. Elasticsearch and <a href="http://hortonworks.com/">Hortonworks</a> have strengthened their relationship by <a href="http://hortonworks.com/partner/elasticsearch/">formally partnering up</a>, which has resulted in Elasticsearch now being the first certified <a href="http://hortonworks.com/hadoop/yarn/">Hortonworks Apache YARN</a> search vendor. With both Elasticsearch and Hortonworks being leaders in the open source space, this partnership will lead to a valuable integration between Elasticsearch and Hadoop that anyone using the open source products will benefit from. </p><p>This is what Hortonworks said about our newly formed partnership:</p><p>&#x201C;The over 400,000 downloads a month is a testament to the level of trust put into Elasticsearch. Since Elasticsearch and Hadoop are such critical elements to the data analytics infrastructure for a huge number of users, we&apos;re excited by this opportunity to join together and reach beyond our users&apos; expectations.&quot; Shaun Conolly, Vice President, Corporate Strategy at Hortonworks</p><p>Elasticsearch&apos;s real-time data exploration, analytics, logging and search features combine really well with Hadoop and make for a powerful combination; very useful to anyone handling large volumes of data on a day to day basis. </p><h2>Why This Matters</h2><p>Our customers can now enhance their Hortonworks Hadoop based workflows with our rich query language, designed to help businesses ask better questions, get clearer answers and better analyze their business metrics, all in real-time. Elasticsearch plays well with all Hadoop distributions, including MapR, Cloudera, Pivotal HD and Amazon EMR, and we have plans to provide specific integration with Hortonworks&apos; HDP platform. </p><p>This is what Shay had to say about our partnership with Hortonworks:</p><p>&#x201C;We&apos;re moving quickly here at Elasticsearch. Just a few weeks ago, we announced <a href="http://www.elasticsearch.org/blog/elasticsearch-and-hadoop/">Elasticsearch for Apache Hadoop</a>, so it&apos;s exciting to announce Elasticsearch now works with Hortonworks&apos; HDP Hadoop distribution. We&apos;re constantly working to improve our product so we can provide the most value to all of our users.&quot; Shay Banon, founder and CTO of Elasticsearch</p><h2>How Your Hadoop Infrastructure Can Benefit by integrating with Elasticsearch</h2><p>Often, data stores integrated into Hadoop can become a bottleneck due to the number of requests generated by the tasks running in the cluster for each job. The distributed nature of the Map/Reduce model fits really well on top of Elasticsearch because we correlate the number of Map/Reduce tasks with the number of Elasticsearch shards for a particular query. So every time a query is run, the system dynamically generates a number of Hadoop splits proportional to the number of shards available so that the jobs are run in parallel - your Hadoop cluster scales easily alongside Elasticsearch and vice-versa.</p><p>Finally, Elasticsearch provides near real-time responses (think milliseconds) that significantly improve a Hadoop job&apos;s execution and the cost associated with it, especially when running on &apos;rented resources&apos; such as Amazon EMR. </p><p>We&apos;ve got even more news in the works about Elasticsearch&apos;s integration with all things Hadoop. Follow this blog for future details! And, <a href="http://twitter.com/elasticsearch">let us know</a> how you&apos;re using Elasticsearch with Hadoop. We would love to hear about it.</p></body></html>

Elasticsearch and Hortonworks Partner

<html><head></head><body><p>
	Welcome to&#xA0;<i>This week in Elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
</p>
<h3>Elasticsearch core</h3>
<ul>
	<li>We upgraded to Lucene 4.5.1 (<a href="https://github.com/elasticsearch/elasticsearch/commit/3a34aa735e97aeb9c421be1484442d27a4d2e65c">commit</a>, 0.90 &amp; master)</li>
	<li>The Transport mechanism now has a separate channel for recovery, so it wont collide with bulk requests (<a href="https://github.com/elasticsearch/elasticsearch/issues/3954">#3954</a>, 0.90 &amp; master)</li>
	<li>Fielddata was refactored, simplified and therefore now is a bit faster (<a href="https://github.com/elasticsearch/elasticsearch/commit/7867de4f5bb7bed1641e6be8d58118d3bc0c5ce3">commit</a>, 0.90 &amp; master)</li>
	<li>The windows service script now detects JREs, supports a stop timeout and a startup mode (<a href="https://github.com/elasticsearch/elasticsearch/issues/3938">#3938</a>, <a href="https://github.com/elasticsearch/elasticsearch/issues/3962">#3962</a>, <a href="https://github.com/elasticsearch/elasticsearch/issues/3928">#3928</a>, 0.90 &amp; master)</li>
	<li>Highlighting can return excerpt with no highlights (<a href="https://github.com/elasticsearch/elasticsearch/issues/1171">#1171</a>, 0.90 &amp; master)</li>
	<li>Fixed a <code>NullPointerException</code> in <code>multi_match</code> query when using <code>lenient</code> and field weight (<a href="https://github.com/elasticsearch/elasticsearch/issues/3797">#3797</a>, 0.90 &amp; master)</li>
	<li>A new highlighter implementation called <code>PostingsHighlighter</code> is now supported (<a href="https://github.com/elasticsearch/elasticsearch/issues/3704">#3704</a>, 0.90 &amp; master)</li>
	<li>CompletionFieldMapper returns an error if weight is no integer (<a href="https://github.com/elasticsearch/elasticsearch/issues/3977">#3977</a>, 0.90 &amp; master)</li>
	<li>The <code>has_child</code> query could cause an infinite loop (100% CPU) when used in bool query (<a href="https://github.com/elasticsearch/elasticsearch/issues/3955">#3955</a>, 0.90 &amp; master)</li>
	<li>Rename and document <code>index.mapping.date.parse_upper_inclusive</code> setting for date fields (<a href="https://github.com/elasticsearch/elasticsearch/issues/3914">#3914</a>, 0.90 &amp; master)</li>
	<li>PreBuilt analyzers are now versioned on index creation (<a href="https://github.com/elasticsearch/elasticsearch/issues/3790">#3790</a>, 0.90 &amp; master)</li>
	<li>mget API supports global routing parameter (<a href="https://github.com/elasticsearch/elasticsearch/issues/3996">#3996</a>, 0.90 &amp; master)</li>
	<li>The cat API got a recovery information endpoint (<a href="https://github.com/elasticsearch/elasticsearch/issues/3969">#3969</a>, master only)</li>
	<li>Nodes stats: Path, dev and mount fields can now be retrieved using the Java API (<a href="https://github.com/elasticsearch/elasticsearch/issues/4004">#4004</a>, 0.90 &amp; master)</li>
	<li>When a node completes recovering a shard, a message is sent to master. If the master is not available, it now is ensured to be sent again (<a href="https://github.com/elasticsearch/elasticsearch/issues/4009">#4009</a>, 0.90 &amp; master)</li>
</ul>
<h3>Elasticsearch Ecosystem</h3>
<p>
	Here&apos;s some more information about what is happening in the ecosystem we are maintaining around Elasticsearch, including plugin and driver releases, as well as news about Logstash and Kibana.
</p>
<ul>
	<li>Featured blog post on elasticsearch.org: <a href="http://www.elasticsearch.org/blog/tiny-data-rapid-development-with-elasticsearch/">Using Elasticsearch for Rapid Development</a></li>
	<li>Florian Hopf wrote a summary about how <a href="http://blog.florian-hopf.de/2013/10/elasticsearch-at-scale-kiln-and-github.html">Kiln and GitHub are using elasticsearch</a></li>
	<li>The Scala client <a href="https://github.com/sksamuel/elastic4s">elastic4s</a> has been released as version 0.90.5.0, which is compatible with elasticsearch 0.90.5</li>
	<li>Eric LeBlond did a write up on <a href="https://home.regit.org/2013/10/logstash-and-suricata-for-the-old-guys/"> using Logstash and Suricata</a>.</li>
	<li>The Typo3 content management system project has begun <a href="https://github.com/Flowpack/Flowpack.ElasticSearch.ContentRepositoryAdaptor/">adding Elasticsearch support</a> to their redesigned CMS offering, Neos</li>
	<li>Alexander Cherednichenko created <a href="http://lexaux.blogspot.de/2013/10/nodejs-elasticsearch-mognodb-extjs-and.html">sample application</a> using Node.js, Elasticsearch, MongoDB and ext.js all running on Heroku.</li>
	<li>You can <a href="http://java.dzone.com/articles/devday-2013-full-text-search">watch the video</a> of Itamar Syn-hershko&apos;s recent presentation on Full-Text Search with Lucene and Elasticsearch.</li>
	<li>Bogdan Dumitrescu wrote a blog post on <a href="http://blog.trifork.com/2013/10/24/how-to-avoid-the-split-brain-problem-in-elasticsearch/">How to Avoid the Split-Brain Problem in Elasticsearch</a>.</li>
	<li>The Chef cookbook has been updated: Most important <a href="https://github.com/elasticsearch/cookbook-elasticsearch/compare/0.3.4...0.3.7">changes</a> include improving documentation and adding the `skip_restart` option.</li>
	<li>Joel Abrahamsson has written an article on <a href="http://joelabrahamsson.com/grouping-in-elasticsearch-using-child-documents/">Grouping in Elasticsearch Using Child Documents</a>.</li>
	<li>Pablo Godel has released <a href="https://github.com/pgodel/logstash-pusher">logstash-pusher</a> a CLI tool to send data to Logstash through Redis.</li>
	<li>Murhaf Feras created a how-to on <a href="http://blog.comperiosearch.com/blog/2013/10/24/instant-search-with-angularjs-and-elasticsearch/">Instant Search with AngularJS and Elasticsearch</a></li>
	<li>David Fauth wrote an excellent article on <a href="http://www.intelliwareness.org/?p=399">Creating an Elasticsearch index of United States&apos; Congress Bills Using Pig</a>.</li>
	<li>John Gifford from Loggly shares his <a href="http://www.loggly.com/confessions-of-a-search-logging-addict/">Confessions of a Search and Logging Addict</a> about Loggly&apos;s use of Elasticsearch in their 2nd generation product.</li>
	<li>Hendrik Saly released a <a href="https://github.com/salyh/elasticsearch-msi-installer">set of scripts</a> to build an MSI installer for installing Elasticsearch as a Windows service.</li>
</ul>
<h3>Upcoming Webinar</h3>
<p>
	<a href="https://twitter.com/honzakral">Honza Kral</a> will be presenting <a href="http://www.elasticsearch.org/webinars/the-why-and-what-about-python/">The &quot;What&quot; and &quot;Why&quot; About Python in Elasticsearch</a> on October 31st. You can watch live at 9 AM Pacific Time, 5 PM in the UK or 6 PM CET, followed by real-time Q&amp;A with Honza and other members of the Elasticsearch core developer team. For those who are not able to join the broadcast, it will be archived for later playback. Registration is required.
</p>
<h3>Slides</h3>
<ul>
	<li><a href="http://twitter.com/a2tirb">Britta Weber</a> shared her slides on the <a href="https://speakerdeck.com/elasticsearch/function-score">Elasticsearch function_score</a> as presented at last night&apos;s <a href="http://www.meetup.com/ElasticSearch-UG-Berlin/events/134717942/">Elasticsearch Berlin User Group meeting</a>.</li>
	<li><a href="http://twitter.com/dadoonet">David Pilato</a> has shared his slides from last week&apos;s presentation at SoftShake.ch on <a href="https://speakerdeck.com/elasticsearch/make-sense-of-your-big-data">Making Sense of your (BIG) Data</a></li>
	<li>We also have a <a href="https://speakerdeck.com/elasticsearch">Speaker Deck account</a> for all presentations given by Elasticsearch&apos;s employees that may be a useful reference resource for you.</li>
	<li>Valentino Gagliardi posted a presentation on <a href="http://www.slideshare.net/valentinogagliardi/log-analysisopensourcelogstashelasticsearchkibana">Open Source Log Analysis with Logstash, Elasticsearch and Kibana</a> (Italian)</li>
</ul>
<h3>Where to find Us</h3>
<h4>Australia</h4>
<p>
	We&apos;re working on scheduling meetups in Sydney on November 18th and Melbourne on November 21st to coincide with our Elasticsearch training courses. We&apos;ll update everyone once all details are confirmed.
</p>
<h4>Belgium</h4>
<p>
	If you&apos;re heading to Devoxx Belgium 2013, be sure to check out <a href="http://twitter.com/dadoonet">David Pilato</a> on <a href="http://www.devoxx.be/dv13-david-pilato.html">Elastify Your App: From SQL to NoSQL in Less than One Hour</a>. David will speak on Wednesday, November 13th at 3:10 PM in Room 6. David will also give a lightning talk, <a href="http://www.devoxx.be/dv13-david-pilato.html?presId=3282">Make Sense of Your (BIG) Data</a> on Thursday, November 14th at 1:10 PM in Room 9. Devoxx Belgium runs from November 11-15 at the Metropolis Business Center in Antwerp.
</p>
<h4>Brasil</h4>
<p>
	Elasticsearch community members in Brasil have created a Brasilian Portugese mailing list for folks to discuss Elasticsearch in their native tongue. You can <a href="https://groups.google.com/forum/#!forum/elasticsearch-pt">join the mailing list</a> now and we will have an update on their first meetup coming soon.
</p>
<h4>Estonia</h4>
<p>
	<a href="http://twitter.com/HonzaKral">Honza Kral</a> will explore <a href="http://www.topconf.com/Conference/Abstracts2013/BeyondSearch/tabid/261/language/en-US/Default.aspx">Beyond Search</a> at <a href="http://www.topconf.com/">Topconf</a> on Thursday, November 7th at 3:30 PM. Topconf runs November 6th and 7th in Tallinn.
</p>
<h4>France</h4>
<ul>
	<li>David Pilato will present <a href="http://nantesjug.org/#/events">Elastify your app</a> at the Nantes Java User Group meeting on November 4th. The meeting begins at 7:00 PM.</li>
	<li>You can also see David speak on <a href="http://nightclazz-es-avance.eventbrite.fr/">Advanced Elasticsearch and Kibana</a> at Zenika&apos;s Night Class on November 7th. The class begins at 7:00 PM.</li>
</ul>
<h4>Germany</h4>
<ul>
	<li>Several Elasticsearchers will be attending the <a href="http://jax.de/wjax2013/">W-JAX Conference</a> in Munich from November 4th to the 8th. You can stop by our booth to visit with <a href="http://twitter.com/spinscale">Alexander Reelsen</a> and <a href="https://twitter.com/a2tirb">Britta Weber</a>, and make sure to catch Alex&apos;s talk on <a href="http://jax.de/wjax2013/sessions/elasticsearch-beyond-full-text-search">Elasticsearch: Beyond full-text search</a> on Wednesday, November 6th at 10:15 AM.</li>
	<li>From W-JAX, Alex and Britta will head to Hamburg for <a href="http://developer-conference.eu">Developer-Conference.eu</a> on Thursday and Friday, November 7th and 8th. Alex will once again present on <a href="http://www.developer-conference.eu/session_post/elasticsearch-beyond-full-text-search/">Elasticsearch: Beyond full-text search</a> on Thursday at 1 PM in the Cinema Three room.</li>
	<li>Alex also organizes the <a href="http://spinscale.github.io/munich-search.html">Munich Search Meetup group</a>, and will be hosting a meetup on November 20th. The meetup begins at 6:45 PM.</li>
</ul>
<h4>Ireland</h4>
<p>
	<a href="http://twitter.com/jpountz">Adrien Grand</a> will present <a href="http://lucenerevolution.org/sessions#Adrien_Grand">What is in a Lucene index?</a> at <a href="http://lucenerevolution.org/">Lucene Revolution</a> on Thursday, November 7th at 2:45 PM. Lucene Revolution takes place November 4th - 7th in Dublin.
</p>
<h4>Japan</h4>
<p>
	The Elasticsearch User Group in Japan will meet for a <a href="http://elasticsearch.doorkeeper.jp/events/6532">study session</a> on November 12th in Tokyo. The meeting begins at 7 PM.
</p>
<h4>Netherlands</h4>
<p>
	Elasticsearch will be at the <a href="http://cloudstackcollab.org/">CloudStack Collaboration Conference EU</a> on November 20-22nd in Amsterdam. Stop by our table to say hello!
</p>
<h4>United Kingdom</h4>
<ul>
	<li>We&apos;ll be out in force at <a href="http://strataconf.com/strataeu2013">Strata London</a> from November 11-13th, with both <a href="http://twitter.com/kimchy">Shay Banon</a> and <a href="https://twitter.com/costinl">Costin Leau</a> attending the conference. You can stop by the Elasticsearch booth to meet with both of them during the conference. Shay will present at the <a href="http://strataconf.com/strataeu2013/public/schedule/detail/32102">Big Data London Meetup</a>, taking place on the first evening of Strata London.</li>
	<li>Shay and Costin will also be attending the <a href="http://www.meetup.com/London-ElasticSearch-User-Group/events/144492702/">London Elasticsearch Meetup</a> on November 11th, where they&apos;ll be available for audience Q&amp;A following presentations from <a href="https://twitter.com/tackers">Graham Tackley</a> with The Guardian and <a href="https://twitter.com/andrew_clegg">Andrew Clegg</a> with Pearson. The meetup will start at 6:30 PM at Red Badger&apos;s offices.</li>
</ul>
<h4>United States</h4>
<ul>
	<li><a href="http://twitter.com/drewr">Drew Raines</a> and <a href="http://twitter.com/imotov">Igor Motov</a> will both present at the <a href="http://www.meetup.com/Elasticsearch-Boston/events/146252052/">Elasticsearch Boston Meetup</a> on November 4th. Drew will discuss Elasticsearch and Consistency, and Igor will address What&apos;s New in Elasticsearch 1.0. Our Community Manager, <a href="http://twitter.com/lhawthorn">Leslie Hawthorn</a>, will also join the meetup. Doors open at 6:00 PM at Workbar in Cambridge, Massachusetts.</li>
	<li><a href="https://twitter.com/mvgroningen">Martijn van Groningen</a> will cover <a href="http://www.basistechweek.com/oss.html#speaker-martijn-van-groningen">Document Relations with Elasticsearch</a> at the Basis Open Source Search Conference in Chantilly, Virginia on Wednesday, November 6th. Martijn&apos;s presentation starts at 10:35 AM.</li>
	<li>Martijn will also present at a <a href="http://www.meetup.com/Elasticsearch-Washington-DC/events/129113592/">Lunch and Learn session</a> on New Features in Elasticsearch 1.0 at Verizon&apos;s offices in Ashburn, Virginia on Thursday, November 7th. OUr hosts, Verizon, will also treat the audience to information on their use of Elasticsearch. Doors open at 11:30 AM; please arrive early to give yourself plenty of time to check in at reception.</li>
	<li><a href="https://twitter.com/kevinkluge">Kevin Kluge</a> will be speaking at the <a href="http://www.meetup.com/sfruby/events/144512912/">San Francisco Ruby Meetup</a> on November 7th. Doors open at 7:00.</li>
	<li><a href="http://twitter.com/rashidkpc">Rashid Khan</a> will be attending <a href="http://nov2013.desertcodecamp.com/">Desert Code Camp</a> in Phoenix, Arizona on Saturday, November 9th. If you&apos;ll also be there, have a chat with him on all things Kibana.</li>
	<li>You can learn all about how Atlassian uses Elasticsearch for <a href="http://www.meetup.com/Elasticsearch-San-Francisco/events/141698772/">Heavy lifting in HipChat</a> at the San Francisco Bay Area meetup on Monday, November 18th. Atlassian will be hosting the meetup at their offices and doors open at 6:30 PM.</li>
	<li><a href="http://twitter.com/uboness">Uri Boness</a> and <a href="http://twitter.com/zacharytong">Zachary Tong</a> will be presenting at the <a href="http://www.meetup.com/Elasticsearch-User-Group-Chicago/events/148062252/">Chicago Elasticsearch Meetup</a> on November 18th. Further details coming soon!</li>
</ul>
<h3>Call for Speakers and Community Organizers</h3>
<p>
	Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, <a href="https://docs.google.com/forms/d/1qBZ6Fwq4JWw7FH3s6C9Q3KeGCvUOmWYOvNMsxNqkCNM/viewform">take a moment to let her know</a>.
</p>
<h3>Where to Find You</h3>
<p>
	Are you hosting an Elasticsearch meetup or giving a talk about Elasticsearch? We would love to know so we can feature that information in future editions of&#xA0;<em>This Week in Elasticsearch</em>. Just let our&#xA0;<a href="http://twitter.com/lhawthorn">Community Manager</a>&#xA0;know what you&apos;re up to and we&apos;re happy to help promote your efforts.
</p>
<p>
	Oh yeah, we&apos;re also&#xA0;<a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you&apos;d like us to find you for employment purposes, just&#xA0;<a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &#xA0;We care more about your skill set and passion for Elasticearch, Kibana and Logstash than where you rest your head.
</p>
<h3>Trainings</h3>
<p>
	If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
</p>
<ul>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/boston/2013-11-04">Boston</a>&#xA0;- November 04, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/amsterdam/2013-11-11">Amsterdam</a>&#xA0;- November 11, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/chicago/2013-11-18">Chicago</a>&#xA0;- November 18, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/san-francisco-california-united-states/2013-11-21">San Francisco</a>&#xA0;- November 21, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/oslo/2013-11-26">Oslo</a>&#xA0;- November 26, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/paris/2013-12-05">Paris</a>&#xA0;- December 05, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/london-uk/2013-12-09">London</a>&#xA0;- December 12, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/los-angeles/2013-12-16">Los Angeles</a>&#xA0;- December 16, 2013</li>
</ul></body></html>

This Week in Elasticsearch - October 30, 2013

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p>
<h2>What Does an Index Look Like?</h2>
<p>In the previous article we looked at how search engines parses, analyzes and tokenizes text. In this article we will explore in more detail what an index looks like.
</p><section id="terms-words-and-tokens" class="level2">
<h2><a href="#terms-words-and-tokens">Terms, Words and Tokens</a></h2>
<p>Read the previous article here: <a href="/blog/found-indexing-for-beginners-part2/" title="/blog/found-indexing-for-beginners-part2">Document parsing and tokenization</a>
</p>
<p>After we have tokenized the text, we are left with a list of tokens that the search engine will store in the index. In our cookbook example, the tokens are the actual words in the recipes that we want to make searchable. As you may have noticed, we often use the terms <em>word</em>, <em>term</em> and <em>token</em> interchangeably and in the broadest sense when talking about indexing. For example, we use these terms for actual words, numbers, abbreviations, paths (in the example with the path-tokenizer) and so on. Strictly speaking, a <em>token</em> is the name of a unit that we derive from the tokenizer, and the token therefore depends on the tokenizer. A token is not necessarily a word, but a <em>word</em> is normally a token when dealing with text. When we store the token in the index, it is usually called a <em>term</em>.
</p>
</section><section id="the-forward-index" class="level2">
<h2><a href="#the-forward-index">The Forward Index</a></h2>
<p>The most obvious way to build up an index, is to store a list of all terms for each document that we are indexing:
</p>
<table>
<thead>
<tr class="header">
	<th style="text-align: left;">Document
	</th>
	<th style="text-align: left;">Terms
	</th>
</tr>
</thead>
<tbody>
<tr class="odd">
	<td style="text-align: left;">Grandma&#x2019;s tomato soup
	</td>
	<td style="text-align: left;">peeled, tomatoes, carrot, basil, leaves, water, salt, stir, and, boil, &#x2026;
	</td>
</tr>
<tr class="even">
	<td style="text-align: left;">African tomato soup
	</td>
	<td style="text-align: left;">15, large, tomatoes, baobab, leaves, water, store, in, a, cool, place, &#x2026;
	</td>
</tr>
<tr class="odd">
	<td style="text-align: left;">Good ol&#x2019; tomato soup
	</td>
	<td style="text-align: left;">tomato, garlic, water, salt, 400, gram, chicken, fillet, cook, for, 15, minutes, &#x2026;
	</td>
</tr>
</tbody>
</table>
<p>This is called a <em>forward index</em>. The forward index is pretty fast when indexing, as new documents are appended to the index and no rebuilding of the index is required. It is, however, not very efficient when querying because querying requires the search engine to look through all entries in the index for a specfic term in order to return all documents containing the term.
</p>
</section><section id="inverted-index" class="level2">
<h2><a href="#inverted-index">Inverted Index</a></h2>
<p>To make the query process faster, it is much smarter to sort the index by terms, like this:
</p>
<table>
<thead>
<tr class="header">
	<th style="text-align: left;">Term
	</th>
	<th style="text-align: left;">Documents
	</th>
</tr>
</thead>
<tbody>
<tr class="odd">
	<td style="text-align: left;">baobaob
	</td>
	<td style="text-align: left;">African tomato soup
	</td>
</tr>
<tr class="even">
	<td style="text-align: left;">basil
	</td>
	<td style="text-align: left;">Grandma&#x2019;s tomato soup
	</td>
</tr>
<tr class="odd">
	<td style="text-align: left;">leaves
	</td>
	<td style="text-align: left;">African tomato soup, Grandma&#x2019;s tomato soup
	</td>
</tr>
<tr class="even">
	<td style="text-align: left;">salt
	</td>
	<td style="text-align: left;">African tomato soup, Good ol&#x2019; tomato soup
	</td>
</tr>
<tr class="odd">
	<td style="text-align: left;">tomato
	</td>
	<td style="text-align: left;">African tomato soup, Good ol&#x2019; tomato soup, Grandma&#x2019;s tomato soup
	</td>
</tr>
<tr class="even">
	<td style="text-align: left;">water
	</td>
	<td style="text-align: left;">African tomato soup, Good ol&#x2019; tomato soup, Grandma&#x2019;s tomato soup
	</td>
</tr>
<tr class="odd">
	<td style="text-align: left;">&#x2026;
	</td>
	<td style="text-align: left;">&#x2026;
	</td>
</tr>
</tbody>
</table>
<p>This type of index is called an <em>inverted index</em>, namely because it is an inversion of the forward index. With the inverted index, we only have to look for a term once to retrieve a list of all documents containing the term. As mentioned in the first article in this series, conventional textbook indexing is based on inverted index.
</p>
<p>Often both the forward and inverted index are used in search engines, where the inverted index is built by sorting the forward index by its terms.
</p>
<p>In some search engines the index includes additional information such as frequency of the terms, e.g. how often a term occurs in each document, or the position of the term in each document. The frequency of a term is often used to calculate the relevance of a search result, whereas the position is often used to facilitate searching for phrases in a document.
</p>
<p>In the next article, we will look at how you can build up more sophisticated indexes to speed up and improve the relevancy of your search results.
</p>
<p>Previous: <a href="/blog/found-indexing-for-beginners-part2/" title="/blog/found-indexing-for-beginners-part2">Document Parsing and Tokenization</a>
</p>
</section></body></html>

Indexing for Beginners, Part 3

<html><head></head><body><p><em>Today we&apos;re pleased to bring you the story of the creation of SeeMeSpeak, a Ruby application that allows users to record gestures for those learning sign language. <a href="https://twitter.com/Argorak">Florian Gilcher</a>, one of the organizers of the <a href="http://www.meetup.com/ElasticSearch-UG-Berlin/">Berlin Elasticsearch User Group</a> participated in a hackathon last weekend with three friends, resulting in this brand new open source project using Elasticsearch on the back end.</em>
</p>
<p>Last weekend we participated in <a href="http://railsrumble.com/">RailsRumble 2013</a>, a 48 hour hacking contest to build something fancy on top of Ruby (nowadays, Rails is not required). Beyond fancy, we wanted to build something that solves a real problem that we could continue developing after the hackathon concluded. It turns out that we found a common scratch to itch: our entire group was interested in <a href="http://en.wikipedia.org/wiki/Sign_language">sign language</a>.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4c9612798d8e381fd7f9/download?uid=blt6c9453d718003122?uid=blt6c9453d718003122" data-sys-asset-uid="bltff92b5092da048cd" style="max-width:100%;">
</p>
<h2>The Project</h2>
<p>Sadly, there are almost no good learning resources for sign language on the internet. If material is available, licensing is a hassle or both the licensing and the material is poorly documented. Documenting sign language yourself is also hard, because producing and collecting videos is difficult. You need third-party recording tools, video conversion and manual categorization. That&apos;s a sad state in a world where every notebook has a usable camera built in!
</p>
<p>Our idea was to leverage modern browser technologies to provide an easy recording function and a quick interface to categorize the recorded words. The result is <a href="http://rumble.seemespeak.org">SeeMeSpeak</a>.
</p>
<h2>We Have a Hundred Problems...</h2>
<p>Handling UserMedia (video and audio) in the browser is still in its infancy. Especially with recording, which mostly works through elaborate hacks like <a href="https://github.com/antimatter15/whammy">whammy.js</a> that need to be set up very carefully. Given only about 24 hours of full work - our team valued sleep and rest -  we expected recording and proper video conversion to take up most of our time.
</p>
<h2>... The Data Store Shouldn&apos;t Be One of Them</h2>
<p>Thus, we were looking for a data store that:
</p>
<ul>
	<li>is easy to set up for all developers</li>
	<li>is easy to handle on the server</li>
	<li>solves our immediate problems</li>
	<li>helps us to iterate quickly</li>
	<li>allows for future refinements in our problem space</li>
	<li>is resource-friendly</li>
</ul>
<p>The last point stems from the contest rules: all applications have to run on a Linode instance with 1GB of memory. Considering that we have to convert videos on the same machine, we cannot be wasteful.
</p>
<p>Our problem is language and searching, so Apache Lucene is an obvious pick. Three of our team members had already worked with Elasticsearch and the decision was made quickly; we used Elasticsearch and couldn&apos;t be happier with it.
</p>
<h2>Elasticsearch for Tiny Data</h2>
<p>Elasticsearch supported the goals of our fast prototype very well:
</p>
<p>Setting up a reasonably configured elasticsearch instance as a developer is as easy as:
</p>
<ul>
	<li>download and unzip</li>
	<li><code>cd elasticsearch; bin/elasticsearch -f</code></li>
</ul>
<p>With packages available for many platforms, Elasticsearch is quickly configured on the server as well. Also, contrary to popular belief, Elasticsearch can be run in a small memory space by setting <code>ES_HEAP_SIZE</code> to a small value. Given that we expected less than 5000 documents for our initial prototype, we were able to easily fit Elasticsearch in less around 256MB of memory.
</p>
<p>Setting up the new <a href="https://github.com/elasticsearch/elasticsearch-ruby">offical Ruby client</a> was also a breeze, it integrates well with the connection-less data models provided by <a href="https://github.com/solnic/virtus">Virtus</a>.
</p>
<p>On the content side, we had 3 main problems: tagging, flagging and transcription. The transcriptions to a video should be searchable. At the same time, we want to flag videos by a fixed set of criteria (if a word is vulgar or an insult, we want to know). Tagging is free-form and allows users to categorize words. All this is possible using Elasticsearch&apos;s dynamic mapping. Our main datatype ended up looking roughly like this:
</p>
<pre class="prettyprint lang-js">{
  &quot;transcription&quot;: &quot;hello from seemespeak&quot;,
  &quot;tags&quot;: [&quot;funny&quot;],
  &quot;flags&quot;: [&quot;casual&quot;],
  &quot;reviewed&quot;: false,
  &quot;language&quot;: &quot;ASL&quot;
}
</pre>
<p>This is a very workable application-side data format that Elasticsearch&apos;s dynamic mapping indexes very well. More specifically, it gives you all you need to search it as expected:
</p>
<ul>
	<li>The <code>match</code> query makes implementing a search box over the transcription field very easy.</li>
	<li>tags, flags and review status can be easily handled using the <code>term</code> and <code>terms</code> filters. A <code>bool</code> filter helps you combining those.</li>
	<li>For the random video on the frontpage and the full list, we used the newly introduced <a href="http://www.elasticsearch.org/blog/0-90-4-released/">function_score query</a> with a randomly changing seed, giving every user different, but stable results.</li>
</ul>
<p>We found that the Elasticsearch&apos;s query language makes is very easy to build queries iteratively. We started with a plain <code>match_all</code> query to give frontend developers enough content to start building and added filters and randomizations over the first day. By the end of the first day, we were done with all search scenarios.
</p>
<p>The current SeeMeSpeak is a prototype. We didn&apos;t venture into fine-tuning by implementing custom analyzers, e.g. for proper stemming and synonym searches. We were confident that we could do this on the second day, but wanted to invest our time in other things, like translating our German corpus to English for our English-speaking visitors and translating the whole site into two languages.
</p>
<p>All in all, Elasticsearch allowed us not to worry too much about data storage and push out a minimum viable version of our product with usable features over the course of a weekend.
</p>
<h2>Wrapping Up</h2>
<p>Elasticsearch is not only a great database for &quot;big data,&quot; it is also a great fit for rapid application development in early stages. We were able to validate all our scenarios and use-cases very quickly. As always, investing time and effort into your data store will improve results in the long run, but we were impressed by the number of features we got out of Elasticsearch so quickly without starting with fine tuning.
</p>
<h2>How You Can Help</h2>
<p>SeeMeSpeak is open source and solves a real issue: building an openly licenced and well tagged corpus for sign languages. You can contribute <a href="http://github.com/seemespeak">on GitHub</a>. Also, you can help us by voting or leaving feedback on the <a href="http://railsrumble.com/entries/39-see-me-speak">RailsRumble</a> site. Feel free to try out our recording function. Just try to say <a href="http://rumble.seemespeak.org/videos?transcription=hallo">hello</a>.
</p>
<p>Thank you!
</p>
<p><a href="http://twitter.com/argorak">Florian Gilcher</a>, Elasticsearch Berlin User Group Co-Organizer, <a href="http://twitter.com/kaeff">Klaus Fl</a>, <a href="http://twitter.com/jannilpferd">Jan Nietfeld</a>, <a href="http://twitter.com/bitboxer">Bodo Tasche</a>
</p>
<p><em>Many thanks to Florian, Klaus, Jan and Bodo for sharing their experiences with us. We are always excited to share stories from the community about how they are using Elasticsearch, so please <a href="http://twitter.com/lhawthorn">reach out to me</a> any time you&apos;d like to share you story.</em>
</p></body></html>

Tiny Data: Rapid Development with Elasticsearch

<html><head></head><body><p>
	Welcome to&#xA0;<i>This week in Elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
</p><h2>Elasticsearch core</h2><ul>
	<li>Settings <code>queue_size</code> on index/bulk TP can cause rejection failures when executed over network (<a href="https://github.com/elasticsearch/elasticsearch/issues/3929">#3929</a>, 0.90 &amp; master)</li>
	<li>Awareness attributes can now be reset (<a href="https://github.com/elasticsearch/elasticsearch/issues/3931">#3931</a>, 0.90 &amp; master)</li>
	<li>The new percolator now supports faceting (<a href="https://github.com/elasticsearch/elasticsearch/issues/3851">#3851</a>, master only)</li>
	<li>Java API: Setting track scores does not affect scan search type (<a href="https://github.com/elasticsearch/elasticsearch/issues/3949">#3949</a>, 0.90 &amp; master)</li>
</ul><h3>Elasticsearch Ecosystem</h3><p>
	Here&apos;s some more information about what is happening in the ecosystem we are maintaining around Elasticsearch, including plugin and driver releases, as well as news about Logstash and Kibana.
</p><ul>
	<li>The <a href="http://elasticsearch-py.readthedocs.org/en/latest/">Elasticsearch python client 0.4.3</a> has been released, as well as the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/client/php-api/current/index.html">PHP client 0.4.1</a> client release</li>
	<li><a href="https://github.com/dieswaytoofast/erlasticsearch">Erlasticsearch</a>: A client written in Erlang utilizing the Thrift interface.</li>
	<li>Jes: A Joomla extension using <a href="http://extensions.joomla.org/extensions/search-a-indexing/site-search/25449">elasticsearch as indexer</a></li>
	<li>Sage: A <a href="https://npmjs.org/package/sage">nodejs elasticsearch client</a></li>
	<li><a href="https://godoc.org/github.com/dustin/go-elasticsearch">Go-elasticsearch</a>: a new Go client for Elasticsearch</li>
	<li>The <a href="https://github.com/YannBrrd/elasticsearch-entity-resolution/">entity resolution plugin</a> has been updated to version 0.4</li>
	<li>The ruby integrations have been updated as well, these are the most important changes
	<ul>
		<li>The parameters to API methods are not only whitelisted, but verified -- passing an unsupported parameter will raise an exception, see <a href="https://github.com/elasticsearch/elasticsearch-ruby/commit/ff7e475">commit</a></li>
		<li>Incorrect behaviour of the Curb-based HTTP client has been fixed, see <a href="https://github.com/elasticsearch/elasticsearch-ruby/commit/8b22af4">commit</a></li>
		<li>Encoding of URL parts has been improved, see <a href="https://github.com/elasticsearch/elasticsearch-ruby/commit/9f65017">commit</a></li>
		<li>URL extraction in client constructor has been improved to allow more flexibility (`https`, authentication, URL prefixes, etc) , see <a href="https://github.com/elasticsearch/elasticsearch-ruby/commit/f91cec5">commit</a></li>
	</ul>
	</li>
</ul><h3>Slides</h3><p>
	While not slides per se, you can check out the interactive coder oriented Elasticsearch tutorial <a href="http://twitter.com/s1m0nw">Simon Willnauer</a> used for his presentation <a href="http://gotocon.com/berlin-2013/presentation/With%20a%20hammer%20in%20your%20hand%E2%80%A6%20Elasticsearch">Elasticsearch: With a Hammer in Your Hand</a> at last week&apos;s GOTO Berlin conference.
</p><p>
	We also now have a <a href="https://speakerdeck.com/elasticsearch">Speaker Deck account</a> for all presentations given by Elasticsearch&apos;s employees that may be a useful reference resource for you.
</p><h3>Where to find Us</h3><h4>Belgium</h4><p>
	If you&apos;re heading to Devoxx Belgium 2013, be sure to check out <a href="http://twitter.com/dadoonet">David Pilato</a> on <a href="http://www.devoxx.be/dv13-david-pilato.html">Elastify Your App: From SQL to NoSQL in Less than One Hour</a>. David will speak on Wednesday, November 13th at 3:10 PM in Room 6. David will also give a lightning talk, <a href="http://www.devoxx.be/dv13-david-pilato.html?presId=3282">Make Sense of Your (BIG) Data</a> on Thursday, November 14th at 1:10 PM in Room 9. Devoxx Belgium runs from November 11-15 at the Metropolis Business Center in Antwerp.
</p><h4>Estonia</h4><p>
	<a href="http://twitter.com/HonzaKral">Honza Kral</a> will explore <a href="http://www.topconf.com/Conference/Abstracts2013/BeyondSearch/tabid/261/language/en-US/Default.aspx">Beyond Search</a> at <a href="http://www.topconf.com/">Topconf</a> on Thursday, November 7th at 3:30 PM. Topconf runs November 6th and 7th in Tallinn.
</p><h4>France</h4><ul>
	<li>David Pilato will present <a href="http://nantesjug.org/#/events">Elastify your app</a> at the Nantes Java User Group meeting on November 4th. The meeting begins at 7:00 PM.</li>
	<li>You can also see David speak on <a href="http://nightclazz-es-avance.eventbrite.fr/">Advanced Elasticsearch and Kibana</a> at Zenika&apos;s Night Class on November 7th. The class begins at 7:00 PM.</li>
</ul><h4>Germany</h4><ul>
	<li>The <a href="http://www.meetup.com/ElasticSearch-UG-Berlin/events/134717942/">Berlin Elasticsearch User Group</a> will convene for their regular monthly meeting on October 29th.</li>
	<li><a href="http://twitter.com/lhawthorn">Leslie Hawthorn</a> will kick off the TYPO3 European Conference 2013 with a keynote address on <a href="http://t3con13de.typo3.org/program/community-day/hear/the-keeper-of-secrets-the-dance-of-community-leadership/">The Keeper of Secrets: The Dance of Community Leadership</a>. Her presentation takes place on Tuesday, October 29th at 9:30 AM and the conference runs through Thursday, October 31st in Stuttgart.</li>
	<li>Several Elasticsearchers will be attending the <a href="http://jax.de/wjax2013/">W-JAX Conference</a> in Munich from November 4th to the 8th. You can stop by our booth to visit with <a href="http://twitter.com/spinscale">Alexander Reelsen</a> and <a href="https://twitter.com/a2tirb">Britta Weber</a>, and make sure to catch Alex&apos;s talk on <a href="http://jax.de/wjax2013/sessions/elasticsearch-beyond-full-text-search">Elasticsearch: Beyond full-text search</a> on Wednesday, November 6th at 10:15 AM.</li>
	<li>From W-JAX, Alex and Britta will head to Hamburg for <a href="http://developer-conference.eu">Developer-Conference.eu</a> on Thursday and Friday, November 7th and 8th. Alex will once again present on <a href="http://www.developer-conference.eu/session_post/elasticsearch-beyond-full-text-search/">Elasticsearch: Beyond full-text search</a> on Thursday at 1 PM in the Cinema Three room.</li>
</ul><h4>Ireland</h4><p>
	<a href="http://twitter.com/jpountz">Adrien Grand</a> will present <a href="http://lucenerevolution.org/sessions#Adrien_Grand">What is in a Lucene index?</a> at <a href="http://lucenerevolution.org/">Lucene Revolution</a> on Thursday, November 7th at 2:45 PM. Lucene Revolution takes place November 4th - 7th in Dublin.
</p><h4>Switzerland</h4><p>
	David Pilato will be in Geneva on October 24th and 25th for the <a href="http://soft-shake.ch/2013/en/">SoftShake Conference</a>. You can visit with David in the Elasticsearch booth, and make sure to check out his presentation on <a href="http://soft-shake.ch/2013/program/sessions/13_bigdata/2013/10/24/07-make-sense-of-your-big-data.html">Make sense of your (BIG) data!</a> on Thursday at 4:30 PM. You may also want to join David for the NoSQL Fireside Chat, also taking place on Thursday, starting at 2:30 PM.
</p><h4>United Kingdom</h4><ul>
	<li><a href="https://twitter.com/Richardp82">Richard Pijnenburg</a>&#xA0;will give a talk at <a href="http://jaxlondon.com/">JAX London</a> about how to <a href="http://jaxlondon.com/sessions/level-your-logging">Level Up Your Logging</a> on Wednesday, October 30th. You can also stop by the Elasticsearch booth to say hello to him.</li>
	<li>We&apos;ll be out in force at <a href="http://strataconf.com/strataeu2013">Strata London</a>, with both <a href="http://twitter.com/kimchy">Shay Banon</a> and <a href="https://twitter.com/costinl">Costin Leau</a> attending the conference. You can stop by the Elasticsearch booth to meet with both of them during the conference.</li>
	<li>Shay and Costin will also be attending the <a href="http://strataconf.com/strataeu2013/public/schedule/detail/32102">November London Elasticsearch Meetup</a>, where they&apos;ll be available for audience Q&amp;A following presentations from <a href="https://twitter.com/tackers">Graham Tackley</a> with The Guardian and <a href="https://twitter.com/andrew_clegg">Andrew Clegg</a> with Pearson. The meetup will start at 6:30 PM.</li>
</ul><h4>United States</h4><ul>
	<li><a href="http://twitter.com/kimchy">Shay Banon</a> will be speaking at the inaugural <a href="http://www.meetup.com/Elasticsearch-Silicon-Valley/events/145034412/">Silicon Valley Elasticsearch Meetup</a> on Wednesday, October 23rd. Our meetup hosts, Facebook, will also share what they&apos;ve learned using Elasticsearch in a fast moving environment over the last few years.</li>
	<li><a href="http://www.linkedin.com/pub/steve-mayzak/2/300/508">Steve Mayzak</a> will be speaking on Getting Started with Elasticsearch at the second Chicago Elasticsearch meetup, scheduled for October 24th.</li>
	<li>Ralph Meijer from Mailgun will be talking at the San Francisco meetup scheduled for October 25th on&#xA0;<a href="http://www.meetup.com/Elasticsearch-San-Francisco/events/142398022/">The Business End of the [Mail]gun</a>.</li>
	<li><a href="http://twitter.com/drewr">Drew Raines</a> and <a href="http://twitter.com/imotov">Igor Motov</a> will both present at the <a href="http://www.meetup.com/Elasticsearch-Boston/events/146252052/">Elasticsearch Boston Meetup</a> on November 4th. Drew will discuss Elasticsearch and Consistency, and Igor will address What&apos;s New in Elasticsearch 1.0. The meetup begins at 6:00 PM at Workbar in Cambridge, Massachusetts.</li>
	<li><a href="https://twitter.com/mvgroningen">Martijn van Groningen</a> will cover <a href="http://www.basistechweek.com/oss.html#speaker-martijn-van-groningen">Document Relations with Elasticsearch</a> at the Basis Open Source Search Conference in Chantilly, Virginia on Wednesday, November 6th. Martijn&apos;s presentation starts at 10:35 AM.</li>
	<li>Martijn will also present at a <a href="http://www.meetup.com/Elasticsearch-Washington-DC/events/129113592/">Lunch and Learn session</a> on New Features in Elasticsearch 1.0 at Verizon&apos;s offices in Chantilly, Virginia on Thursday, November 7th. OUr hosts, Verizon, will also treat the audience to information on their use of Elasticsearch. Doors open at 11:30 AM; please arrive early to give yourself plenty of time to check in at reception.</li>
	<li><a href="https://twitter.com/kevinkluge">Kevin Kluge</a> will be speaking at the <a href="http://www.meetup.com/sfruby/events/144512912/">San Francisco Ruby Meetup</a> on November 7th. Doors open at 7:00.</li>
	<li><a href="http://twitter.com/rashidkpc">Rashid Khan</a> will be attending <a href="http://nov2013.desertcodecamp.com/">Desert Code Camp</a> in Phoenix, Arizona on Saturday, November 9th. If you&apos;ll also be there, have a chat with him on all things Kibana.</li>
	<li>You can learn all about how Atlassian uses Elasticsearch for <a href="http://www.meetup.com/Elasticsearch-San-Francisco/events/141698772/">Heavy lifting in HipChat</a> at the San Francisco Bay Area meetup on Monday, November 18th. Atlassian will be hosting the meetup at their offices and doors open at 6:30 PM.</li>
</ul><h3>Where to Find You</h3><p>
	Are you hosting an Elasticsearch meetup or giving a talk about Elasticsearch? We would love to know so we can feature that information in future editions of&#xA0;<em>This Week in Elasticsearch</em>. Just let our&#xA0;<a href="http://twitter.com/lhawthorn">Community Manager</a>&#xA0;know what you&apos;re up to and we&apos;re happy to help promote your efforts.
</p><p>
	Oh yeah, we&apos;re also&#xA0;<a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you&apos;d like us to find you for employment purposes, just&#xA0;<a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &#xA0;We care more about your skill set and passion for Elasticearch, Kibana and Logstash than where you rest your head.
</p><h3>Trainings</h3><p>
	If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
</p><ul>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/zurich/2013-10-29">Zurich</a>&#xA0;- October 29, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/boston/2013-11-04">Boston</a>&#xA0;- November 04, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/amsterdam/2013-11-11">Amsterdam</a>&#xA0;- November 11, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/chicago/2013-11-18">Chicago</a>&#xA0;- November 18, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/san-francisco-california-united-states/2013-11-21">San Francisco</a>&#xA0;- November 21, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/oslo/2013-11-26">Oslo</a>&#xA0;- November 26, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/paris/2013-12-05">Paris</a>&#xA0;- December 05, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/london-uk/2013-12-09">Oslo</a>&#xA0;- December 09, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/los-angeles/2013-12-16">Los Angeles</a>&#xA0;- December 16, 2013</li>
</ul></body></html>

This Week in Elasticsearch - October 23, 2013

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>This article will show how you can easily create a cluster, upgrade versions and make it a high availability cluster. I will also briefly describe what happens behind the scenes of various types of operations.
</p>
<p>Whether you need a small cluster for a few hours of testing, or a highly available mission critical production cluster, we work hard to make it easier to manage Elasticsearch clusters &#x2014; so our clients can focus on their core business: building cool stuff!
</p>
<p><strong>Note</strong>: This article mainly describes how Found&#x2019;s service works. <a href="/blog/found-elasticsearch-in-production/">Elasticsearch in Production</a> has many important details about running Elasticsearch &#x2013; whether you use Found or manage your own clusters. The aforementioned article goes into depth on why enough memory is key to Elasticsearch, and how high availability can be achieved in cloud environments.
</p>
<p><strong>Update August 30, 2015</strong>: Please note that some of the information in this article may be a little outdated. For up-to-date information about how to manage Elasticsearch with Found, see our <a href="https://www.elastic.co/guide/en/found/current/index.html">Documentation</a>.
</p>
</section><section id="making-a-cluster" class="level2">
<h2><a href="#making-a-cluster">Making a Cluster</a></h2>
<p>Assuming you have <a href="https://www.elastic.co/found">signed up for Found</a> and <a href="https://found.elastic.co/">signed in</a> to our administration console, you can quickly create a new cluster. Clusters are dedicated and completely isolated from each other.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c9812798d8e381fd837/download?uid=bltf6a40e016aff914e?uid=bltf6a40e016aff914e" data-sys-asset-uid="blt34ac151583369091" alt="Step 1: Choosing region and optionally specifying a name for your cluster" style="max-width:100%;">
</p><figcaption>Step 1: Choosing region and optionally specifying a name for your cluster</figcaption>
<p>First, we need to know in what region you want to host your cluster. It is recommended to chose a region with close geographical proximity to your systems, to lower latencies. It&#x2019;s optional to specify a name for your cluster, however it&#x2019;s wise if you have several clusters!
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c989e7a83165490c323/download?uid=blt28b96cc14f63cbcd?uid=blt28b96cc14f63cbcd" data-sys-asset-uid="blt7f24038efbd26fac" alt="Step 2: Choosing capacity and high availability" style="max-width:100%;">
</p><figcaption>Step 2: Choosing capacity and high availability</figcaption>
<p>You will now be asked to opt for the size of the cluster, as well as whether you need high availability. As opposed to the region you have selected, capacity and high availability can be changed later on for a running cluster, without downtime, during indexing and searching.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c9812798d8e381fd839/download?uid=blt582c7d8ae51e43ae?uid=blt582c7d8ae51e43ae" data-sys-asset-uid="blt95c88441deab50d6" alt="Step 3: Choosing Elasticsearch version and plugins" style="max-width:100%;">
</p><figcaption>Step 3: Choosing Elasticsearch version and plugins</figcaption>
<p>Lastly, you can specify which Elasticsearch version you want to provision and which plugins you want enabled. As we will see in the next section, this can also be changed without downtime, assuming you upgrade to a compatible version.
</p>
<p>When you click &#x201C;Create cluster&#x201D;, we will provision the cluster. This usually takes less than a minute. When the cluster is created, you will be redirected to the cluster&#x2019;s overview page, which will list the instance(s) making up the cluster as starting. After a few seconds, the instances will be listed as started, as in the following figure:
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c979985d58976376c18/download?uid=blt6d31fc20108e1f8f?uid=blt6d31fc20108e1f8f" data-sys-asset-uid="bltcfc9eae64476a1c7" alt="Cluster overview" style="max-width:100%;">
</p><figcaption>Cluster overview</figcaption>
<p>In the overview, you can see your Elasticsearch-endpoints, links to various dashboards and the status of the instances making up your cluster.
</p>
</section><section id="changing-a-cluster" class="level2">
<h2><a href="#changing-a-cluster">Changing a Cluster</a></h2>
<p>There are several ways to change a running cluster:
</p>
<ul>
	<li>Upgrade Elasticsearch version, where the two versions are compatible and can communicate with each other. This is the case when upgrading a <em>minor</em> version, e.g. 0.90.2 to 0.90.3.</li>
	<li>Major version upgrade of Elasticsearch, where the two versions are <em>not</em> compatible and cannot communicate with each other. This is the case when upgrading from e.g. 0.20.4 to 0.90.2.</li>
	<li>Version downgrade. This is not as common, and one has to be careful with the on-disk compatability of the Lucene indexes. For example, Elasticsearch 0.90.3 uses Lucene 4.4, while Elasticsearch 0.90.2 uses Lucene 4.3. Lucene 4.3 cannot read indexes modified by Lucene 4.4.</li>
	<li>Capacity upgrade or downgrade, where the amount of reserved memory is changed.</li>
	<li>Change available plugins.</li>
	<li>Adjust the number of data centers the cluster runs in, i.e. changing the high availability configuration.</li>
</ul>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c9743e9adc538713cd2/download?uid=blt59f2c09e846479de?uid=blt59f2c09e846479de" data-sys-asset-uid="blt55e734ee4da61a11" alt="Configuring a running cluster" style="max-width:100%;">
</p><figcaption>Configuring a running cluster</figcaption>
<p>You can change the configuration of a running cluster from the &#x201C;Configuration&#x201D;-pane.
</p>
<p>With the exception of major version upgrades, we can perform all these changes without having to interrupt your cluster - you can continue searching and indexing. The changes can also be done in bulk: in one action, you can add more memory, upgrade, adjust the number of plugins and adjust the number of availability zones.
</p>
<p>We perform all of these type of changes by making the cluster with the new configuration - in its entirety - join the existing cluster. After joining, the new nodes will recover the indexes. When they are done, they will start receiving requests. When all the new nodes are ready, we bring down the old ones.
</p>
<p>By doing it this way, we reduce the risk of doing any changes. If the new nodes have any problems, the old ones are still there, processing requests.
</p>
</section><section id="major-version-upgrades" class="level2">
<h2><a href="#major-version-upgrades">Major Version Upgrades</a></h2>
<p>For major version upgrades, we have to bring the cluster to a full stop before upgrading, as the nodes cannot communicate with each other. This is done by flushing all changes so we are sure we can recover them, then we start the cluster with the new version.
</p>
<p>While Elasticsearch is working on making upgrades across major versions possible, major version upgrades often include so many changes that upgrades can be risky. This is usually true for any kind of software. Our recommended approach for major version upgrades is to simply make a new cluster with the latest major version, reindex everything and make sure index requests are temporarily sent to both clusters. With the new cluster ready, you can then do a hot swap and send requests to the new cluster. Since you are only billed for the hours a cluster is running, the few extra dollars added to your bill for having an extra cluster running for a while is money well spent. Since the cluster with the version known to work well is already running, you can quickly roll back if the new version has errors.
</p>
<p>We make it easy to manage multiple clusters with different versions. We do not force customers to upgrade their clusters. If we need to end-of-life a very old version, you can expect to be notified in due time.
</p>
<p><strong>Note</strong>: If you use a Platform-as-a-Service-provider like Heroku, the administration console is slightly different and does not allow you to make changes that will affect the price. That must be done in the platform provider&#x2019;s addon system. You can still do things like change Elasticsearch version or -plugins.
</p>
</section><section id="tearing-it-down" class="level2">
<h2><a href="#tearing-it-down">Tearing it Down</a></h2>
<p>The cluster&#x2019;s configuration pane allows you to delete a cluster:
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c983dc542cb38c08464/download?uid=blt51d30badc2a46f29?uid=blt51d30badc2a46f29" data-sys-asset-uid="blt96565d37c005ff93" alt="Deleting a running cluster" style="max-width:100%;">
</p><figcaption>Deleting a running cluster</figcaption>
<p>Deleting is final and cannot be undone. Billing stops immediately when the cluster has been deleted, rounding up to the nearest hour. This means you can easily start a cluster, run some tests and tear it down again when you are done.
</p>
<p>In not too long, we will make our cluster management APIs available. This will enable you to automate the running of e.g. final integration tests before deploying to production, for a few cents. We are also looking into making it possible to <em>pause</em> a small running cluster. This can be quite useful when you only occasionally use a staging cluster.
</p>
</section><section id="on-high-availability" class="level2">
<h2><a href="#on-high-availability">On High Availability</a></h2>
<p>High availability is achieved by running a cluster with replicas in multiple availability zones, to prevent against downtime when inevitable infrastructure problems occur. Our article on <a href="/blog/found-elasticsearch-in-production/#networking">Elasticsearch in Production</a> covers this more extensively.
</p>
<p>We offer the options of running in one, two or three availability zones. Running in two AZ&#x2019;s is our default high availability configuration. It provides reasonably high protection against infrastructure failures and intermittent network problems. You might want three zones if you need even higher protection, or perhaps just one zone if the cluster is mainly used for testing or development.
</p>
<p>As mentioned above, this is something that you can change while the cluster is running. For example, when you prepare a new cluster for production use, you can first run it in a single zone, then add another zone right before deploying to production.
</p>
<p>While running in multiple zones increases a cluster&#x2019;s reliability, it does not protect against problematic searches causing nodes to run out of memory, for example. For a cluster to be highly reliable and available, it is also <a href="/blog/found-elasticsearch-in-production/#memory">important to have enough memory</a>.
</p>
</section><section id="accessing-the-logs" class="level2">
<h2><a href="#accessing-the-logs">Accessing the Logs</a></h2>
<p>We provide easy access to the logs of all the nodes in your cluster. You can browse and search through logs produced the last 14 days:
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c9712798d8e381fd82c/download?uid=blt14ee7b1721c4bce2?uid=blt14ee7b1721c4bce2" data-sys-asset-uid="blt3582ab3f1bf2c5e6" alt="Searching the logs" style="max-width:100%;">
</p><figcaption>Searching the logs</figcaption>
<p>Metadata such as level, logger and instance/zone can be clicked to filter on those types of logs.
</p>
</section><section id="access-control" class="level2">
<h2><a href="#access-control">Access Control</a></h2>
<p><strong>Note</strong>: We strongly advise configuring the access control for your cluster
</p>
<p>The default configuration implies that anyone knowing the cluster-ID has full access to your cluster (bear in mind that not all Elasticsearch clients support basic authentication).
</p>
<p>We highly recommend using the access control feature to at least require authentication. Authentication uses HTTP Basic-authentication. Most, but not all HTTP- and Elasticsearch-libraries support this.
</p>
<p>You can limit access based on path, source IP, method, username/password and whether SSL is used. The access control section of the dashboard has annotated samples to use as templates for your own ACLs.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c9712798d8e381fd827/download?uid=bltbafae31a7b1bae27?uid=bltbafae31a7b1bae27" data-sys-asset-uid="blt4819c557f3e76eca" alt="Modifying access control" style="max-width:100%;">
</p><figcaption>Modifying access control</figcaption>
</section><section id="summary-and-further-reading" class="level2">
<h2><a href="#summary-and-further-reading">Summary and Further Reading</a></h2>
<p>We have demonstrated how to create a cluster, upgrade it in various ways (while indexing and searching!) and tear it down. Furthermore, we have described how to access your cluster&#x2019;s logs, and limit who can access your cluster.
</p>
<p>Even though Found takes care of most tasks when it comes to managing clusters, it&#x2019;s advisable to have some knowledge about what is important when running an <a href="/blog/found-elasticsearch-in-production">Elasticsearch cluster in production</a>. We provide all user facing features of Elasticsearch without any restrictions, which also means there are many ways to run into problems. The article on <a href="/blog/found-elasticsearch-as-nosql">Elasticsearch as a NoSQL Database</a> goes through what expectations you can have of Elasticsearch when comparing it to other NoSQL stores. To get the most out of Elasticsearch, it helps to understand some of its internal datastructures as well. Our article series on <a href="/blog/found-elasticsearch-from-the-bottom-up">Elasticsearch from the Bottom up</a> covers exactly that.
</p>
<div class="references">
</div>
</section></body></html>

Managing Elasticsearch with Found

<html><head></head><body><p>
	Welcome to&#xA0;<i>This week in Elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
</p><h2>Elasticsearch core</h2><ul>
	<li>Support the <code>has_child</code>, <code>has_parent</code>, <code>top_children</code> queries and filters in count, explain (return dummy explanation) and delete by query APIs (<a href="https://github.com/elasticsearch/elasticsearch/issues/3822">#3822</a>, 0.90 &amp; master)</li>
	<li><code>QueryBuilders#fuzzyQuery</code> method accepts value parameter as an object (<a href="https://github.com/elasticsearch/elasticsearch/issues/3616">#3616</a>, 0.90 &amp; master)</li>
	<li>Added <code>IndexRequest#source(Object...)</code> input validation (<a href="https://github.com/elasticsearch/elasticsearch/commit/e14e2a6597d2d580e34d28851d5047739f451407">commit</a>, 0.90 &amp; master)</li>
	<li><code>DateFieldMapper</code> uses <code>Locale.ROOT</code> as default locale and does not rely on the system locale (<a href="https://github.com/elasticsearch/elasticsearch/issues/3852">#3852</a>, 0.90 &amp; master)</li>
	<li>Year units in date math expressions are now supported (<a href="https://github.com/elasticsearch/elasticsearch/issues/3828">#3828</a>, 0.90 &amp; master)</li>
	<li>Source filtering parameters of the Get Source API were inconsistent (<a href="https://github.com/elasticsearch/elasticsearch/issues/3886">#3886</a>, master only)</li>
	<li>Integration test for <code>PluginManager</code> to ensure downloading plugins was added (<a href="https://github.com/elasticsearch/elasticsearch/issues/3894">#3894</a>, 0.90 &amp; master)</li>
	<li>Support date math for `origin` decay function parsing (<a href="https://github.com/elasticsearch/elasticsearch/issues/3892">#3892</a>, 0.90 &amp; master)</li>
	<li>Fix <code>GeoShapeQueryBuilder#toXContent</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/3878">#3878</a>, 0.90 &amp; master)</li>
	<li>Query String query now supports multiple fields regexp queries (<a href="https://github.com/elasticsearch/elasticsearch/issues/3901">#3901</a>, 0.90 &amp; master)</li>
	<li>Set queue sizes by default on bulk/index thread pools (<a href="https://github.com/elasticsearch/elasticsearch/issues/3888">#3888</a>, 0.90 &amp; master)</li>
	<li>Fielddata can now go off-heap by storing it into Lucene DocValues on disk (configurable via an index option) (<a href="https://github.com/elasticsearch/elasticsearch/issues/3806">#3806</a>, master)</li>
	<li>Segments API: Support merge id on segments, which returns a <code>merge_id</code> element in each segment of the segments API, allowing to group segments that are being merged as part of a single merge and indicate which ones are being merged now (<a href="https://github.com/elasticsearch/elasticsearch/issues/3904">#3904</a>, 0.90 &amp; master)</li>
	<li>The <code>has_child</code> query&apos;s inner query now is wrapped in a filtered query with the child type as filter, this prevents other children from being returned as hit. (<a href="https://github.com/elasticsearch/elasticsearch/issues/3818">#3818</a>, 0.90 &amp; master)</li>
	<li>Add <code>match</code> query support for stacked tokens (<a href="https://github.com/elasticsearch/elasticsearch/issues/3881">#3881</a>, 0.90 &amp; master)</li>
	<li>A small bug in the <code>query string</code> query creation, using a word delimeter, using more than one field and containing a special char in the query was fixed (<a href="https://github.com/elasticsearch/elasticsearch/issues/3898">#3898</a>, 0.90 &amp; master)</li>
	<li>Naming in function score was fixed (<a href="https://github.com/elasticsearch/elasticsearch/issues/3872">#3872</a>, 0.90 &amp; master)</li>
	<li>The new percolator exposes more statistics (<a href="https://github.com/elasticsearch/elasticsearch/issues/3883">#3883</a>, master)</li>
	<li>Add support for Lucene <code>SuggestStopFilter</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/3913">#3913</a>, 0.90 &amp; master)</li>
	<li>A cluster state update ack mechanism has been added, which ensures that cluster states are properly ackknowledged. Support for warmer and delete APIs has been added already (<a href="https://github.com/elasticsearch/elasticsearch/issues/3786">#3786</a>, <a href="https://github.com/elasticsearch/elasticsearch/issues/3831">#3831</a>, <a href="https://github.com/elasticsearch/elasticsearch/issues/3833">#3833</a>, 0.90 &amp; master)</li>
	<li>A generic count down mechanism has been added instead of having to write this kind of functionality on your own (<a href="https://github.com/elasticsearch/elasticsearch/issues/3910">#3910</a>, 0.90 &amp; master)</li>
	<li>Index requests which specify a parent are rejected now, if no parent type is defined (<a href="https://github.com/elasticsearch/elasticsearch/issues/3848">#3848</a>, 0.90 &amp; master)</li>
	<li>Adding a parent mapping at runtime fails now (<a href="https://github.com/elasticsearch/elasticsearch/issues/3849">#3849</a>, 0.90 &amp; master)</li>
</ul><h3>Elasticsearch Ecosystem</h3><p>
	Here&apos;s some more information about what is happening in the ecosystem we are maintaining around Elasticsearch, including plugin and driver releases, as well as news about Logstash and Kibana.
</p><ul>
	<li>Last week Elasticsearch Inc held its second company All Hands meeting, with tons of technical discussion around future features and enhancements to Elasticsearch, Kibana and Logstash. Shay Banon provides a great in-depth summary of the conversations in this <a href="http://elasticsearch.com/blog/from-amsterdam-with-love-elasticsearchs-second-company-all-hands/">blog post</a>.</li>
	<li>Check out these interesting statistics and numbers from the <a href="http://www.elastichq.org/elasticsearchstats.php">ElasticHQ</a> plugin.</li>
	<li><a href="https://github.com/brainbot-com/es-nozzle">es-nozzle</a> is a scalable open source framework for connecting source content repositories like file systems or mail servers to elasticsearch</li>
	<li>Matthias Nehlsen wrote a nice summary of this tiny application <a href="http://matthiasnehlsen.com/blog/2013/09/10/birdwatch-explained/">BirdWatch</a>, which uses Play, AngularJS and Elasticsearch to show tweets in real-time.</li>
	<li>Shelly Cloud produced a <a href="https://shellycloud.com/blog/2013/10/adding-search-and-autocomplete-to-a-rails-app-with-elasticsearch?utm_source=hn&amp;utm_medium=free&amp;utm_campaign=Adding+Search+Blog+Post">tutorial</a> for adding search to Rails apps using Elasticsearch, Searchkick and typeahaed.js.</li>
	<li><a href="http://twitter.com/anystacker">Any Stacker</a> did a great write up on <a href="http://anystacker.com/2013/10/indexing-project-gutenberg-with-mongodb-and-elasticsearch/">Indexing Project Gutenberg with MongoDB and Elasticsearch</a>.</li>
	<li>Itamar Syn-Hershko <a href="http://code972.com/blog/2013/10/321-hebmorph-now-available-from-maven-central">blogged about searching Hebrew text</a> using Apache Lucene and Elasticsearch.</li>
	<li>Igor Motov <a href="http://elasticsearch.com/blog/hopper-hosts-hackathon-with-elasticsearch/">shared a write up</a> of the recent Hopper Elasticsearch hackathon and his experience as one of the event mentors.</li>
</ul><h3>Slides</h3><p>
	Last week, the Core Developer Team presented several lightning talks on Elasticsearch to more than 65 people at the Elasticsearch Netherlands User Group Meeting. You can check out <a href="https://speakerdeck.com/elasticsearch">all the presentations</a> on Speakerdeck and read <a href="http://twitter.com/zacharytong">Zach Tong&apos;s</a> excellent <a href="http://www.elasticsearch.org/blog/meet-the-devs-meetup-in-amsterdam">write up for more details</a> on the lightning talks.
</p><p>
	You might also want to check out the slides from <a href="https://speakerdeck.com/elasticsearch/explore-your-data-with-elasticsearch">Explore your data with Elasticsearch</a>, a presentation by <a href="http://twitter.com/honzakral">Honza Kral</a> at <a href="http://13.rupy.eu/">last week&apos;s RuPy conference</a>.
</p><h3>Where to find Us</h3><h4>Estonia</h4><p>
	<a href="http://twitter.com/HonzaKral">Honza Kral</a> will explore <a href="http://www.topconf.com/Conference/Abstracts2013/BeyondSearch/tabid/261/language/en-US/Default.aspx">Beyond Search</a> at <a href="http://www.topconf.com/">Topconf</a> on Thursday, November 7th at 3:30 PM. Topconf runs November 6th and 7th in Tallinn.
</p><h4>France</h4><p>
	<a href="http://twitter.com/dadoonet">David Pilato</a> will present <a href="http://nantesjug.org/#/events">Elastify your app</a> at the Nantes Java User Group meeting on November 4th. The meeting begins at 7:00 PM.
</p><h4>Germany</h4><ul>
	<li><a href="http://twitter.com/s1m0nw">Simon Willnauer</a> will speak at <a href="http://gotocon.com/berlin-2013/">GOTO Berlin</a> on &quot;With a hammer in your hand ... Elasticsearch&quot;. Simon&apos;s talk is scheduled for Friday, October 18th at 11:30 AM and he&apos;ll be hanging out with our Community Manager, <a href="http://twitter.com/lhawthorn">Leslie Hawthorn</a> in the exhibits hall throughout the conference. Stop by the Elasticsearch stand and say hello!</li>
	<li>The <a href="http://www.meetup.com/ElasticSearch-UG-Berlin/events/134717942/">Berlin Elasticsearch User Group</a> will convene for their regular monthly meeting on October 29th.</li>
	<li><a href="http://twitter.com/lhawthorn">Leslie Hawthorn</a> will kick off the TYPO3 European Conference 2013 with a keynote address on <a href="http://t3con13de.typo3.org/program/community-day/hear/the-keeper-of-secrets-the-dance-of-community-leadership/">The Keeper of Secrets: The Dance of Community Leadership</a>. Her presentation takes place on Tuesday, October 29th at 9:30 AM and the conference runs through Thursday, October 31st in Stuttgart.</li>
	<li>Several Elasticsearchers will be attending the <a href="http://jax.de/wjax2013/">W-JAX Conference</a> in Munich from November 4th to the 8th. You can stop by our booth to visit with <a href="http://twitter.com/spinscale">Alexander Reelsen</a> and <a href="https://twitter.com/a2tirb">Britta Weber</a>, and make sure to catch Alex&apos;s talk on <a href="http://jax.de/wjax2013/sessions/elasticsearch-beyond-full-text-search">Elasticsearch: Beyond full-text search</a> on Wednesday, November 6th at 10:15 AM.</li>
	<li>From W-JAX, Alex and Britta will head to Hamburg for <a href="http://developer-conference.eu">Developer-Conference.eu</a> on Thursday and Friday, November 7th and 8th. Alex will once again present on <a href="http://www.developer-conference.eu/session_post/elasticsearch-beyond-full-text-search/">Elasticsearch: Beyond full-text search</a> on Thursday at 1 PM in the Cinema Three room.</li>
</ul><h4>Ireland</h4><p>
	<a href="http://twitter.com/jpountz">Adrien Grand</a> will present <a href="http://lucenerevolution.org/sessions#Adrien_Grand">What is in a Lucene index?</a> at <a href="http://lucenerevolution.org/">Lucene Revolution</a> on Thursday, November 7th at 2:45 PM. Lucene Revolution takes place November 4th - 7th in Dublin.
</p><h4>Norway</h4><p>
	Alex Brasetvik from Found will present <a href="http://www.meetup.com/javaBin-Trondheim/events/140876832/">Elasticsearch: much more than just search</a> at the javaBin Trondheim User Group on October 22nd.
</p><h4>Poland</h4><p>
	Honza Kral will present <a href="http://pl.pycon.org/2013/en/agenda">Explore Your Data</a> at PyCon Poland on Friday, October 18th. The conference runs Thursday, October 17th through Sunday, October 20th in Szczyrk.
</p><h4>Switzerland</h4><p>
	David Pilato will be in Geneva on October 24th and 25th for the <a href="http://soft-shake.ch/2013/en/">SoftShake Conference</a>. You can visit with David in the Elasticsearch booth, and make sure to check out his presentation on <a href="http://soft-shake.ch/2013/program/sessions/13_bigdata/2013/10/24/07-make-sense-of-your-big-data.html">Make sense of your (BIG) data!</a> on Thursday at 4:30 PM.
</p><h4>United Kingdom</h4><p>
	<a href="https://twitter.com/Richardp82">Richard Pijnenburg</a>&#xA0;will give a talk at <a href="http://jaxlondon.com/">JAX London</a> about how to <a href="http://jaxlondon.com/sessions/level-your-logging">Level Up Your Logging</a> on Wednesday, October 30th.
</p><h4>United States</h4><ul>
	<li>You can learn all about how Atlassian uses Elasticsearch for <a href="http://www.meetup.com/Elasticsearch-San-Francisco/events/141698772/">Heavy lifting in HipChat</a> at the San Francisco Bay Area meetup on Monday, November 18th. Atlassian will be hosting the meetup at their offices and doors open at 6:30 PM.</li>
	<li><a href="http://twitter.com/uboness">Uri Boness</a> &#xA0;&amp; <a href="https://twitter.com/ZacharyTong">Zach Tong</a> will be presenting together at the Washington DC meetup on a very interesting combination of subjects-&#xA0;<a href="http://www.meetup.com/Elasticsearch-Washington-DC/events/142011702/">Elasticsearch&apos;&#xAD;s New Aggregations feature <em>and</em> Genomic Sequencing with Elasticsearch</a>. The meetup is scheduled for October 22nd.</li>
	<li><a href="http://twitter.com/kimchy">Shay Banon</a> will be speaking at the inaugural <a href="http://www.meetup.com/Elasticsearch-Silicon-Valley/events/145034412/">Silicon Valley Elasticsearch Meetup</a> on Wednesday, October 23rd. Our meetup hosts, Facebook, will also share what they&apos;ve learned using Elasticsearch in a fast moving environment over the last few years.</li>
	<li><a href="http://www.linkedin.com/pub/steve-mayzak/2/300/508">Steve Mayzak</a> will be speaking on Getting Started with Elasticsearch at the second Chicago Elasticsearch meetup, scheduled for October 24th.</li>
	<li>Ralph Meijer from Mailgun will be talking at the San Francisco meetup scheduled for October 25th on&#xA0;<a href="http://www.meetup.com/Elasticsearch-San-Francisco/events/142398022/">The Business End of the [Mail]gun</a>.</li>
	<li><a href="https://twitter.com/mvgroningen">Martijn van Groningen</a> will cover <a href="http://www.basistechweek.com/oss.html#speaker-martijn-van-groningen">Document Relations with Elasticsearch</a> at the Basis Open Source Search Conference in Chantilly, Virginia on Wednesday, November 6th. Martijn&apos;s presentation starts at 10:35 AM.</li>
	<li>Martijn will also present at a <a href="http://www.meetup.com/Elasticsearch-Washington-DC/events/129113592/">Lunch and Learn session</a> on New Features in Elasticsearch 1.0 at Verizon&apos;s offices in Chantilly, Virginia on Thursday, November 7th. OUr hosts, Verizon, will also treat the audience to information on their use of Elasticsearch. Doors open at 11:30 AM; please arrive early to give yourself plenty of time to check in at reception.</li>
	<li><a href="http://twitter.com/rashidkpc">Rashid Khan</a> will be attending <a href="http://nov2013.desertcodecamp.com/">Desert Code Camp</a> in Phoenix, Arizona on Saturday, November 9th. If you&apos;ll also be there, have a chat with him on all things Kibana.</li>
</ul><h3>Where to Find You</h3><p>
	Are you hosting an Elasticsearch meetup or giving a talk about Elasticsearch? We would love to know so we can feature that information in future editions of&#xA0;<em>This Week in Elasticsearch</em>. Just let our&#xA0;<a href="http://twitter.com/lhawthorn">Community Manager</a>&#xA0;know what you&apos;re up to and we&apos;re happy to help promote your efforts.
</p><p>
	Oh yeah, we&apos;re also&#xA0;<a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you&apos;d like us to find you for employment purposes, just&#xA0;<a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &#xA0;We care more about your skill set and passion for Elasticearch, Kibana and Logstash than where you rest your head.
</p><h3>Trainings</h3><p>
	If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
</p><ul>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/cologne/2013-10-17">Cologne</a>&#xA0;- October 17, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/washington-dc/2013-10-21">Washington, D.C.</a>&#xA0;- October 21, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/london/2013-10-21">London, U.K.</a> - October 22, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/zurich/2013-10-29">Zurich</a>&#xA0;- October 29, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/boston/2013-11-04">Boston</a>&#xA0;- November 04, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/amsterdam/2013-11-11">Amsterdam</a>&#xA0;- November 11, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/chicago/2013-11-18">Chicago</a>&#xA0;- November 18, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/sydney/2013-11-18">Sydney</a>&#xA0;- November 18, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/san-francisco-california-united-states/2013-11-21">San Francisco</a>&#xA0;- November 21, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/oslo/2013-11-26">Oslo</a>&#xA0;- November 26, 2013</li>
</ul></body></html>

This Week in Elasticsearch - October 16, 2013

<html><head></head><body><h1>Elasticsearch&apos;s Second Company All Hands</h1>
<p>This post was co-authored with <a href="http://twitter.com/kimchy">Shay Banon</a>.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbbd8edd48f769365a1/download?uid=blt214478aece4105b1?uid=blt214478aece4105b1" data-sys-asset-uid="bltd3c77b7d9ea83860" alt="Kicking Off Our Week in Amsterdam" width="300" height="200" style="float:right; padding:0 0 20px 20px;">Last week, the Elasticsearch crew all converged on our EU headquarters in Amsterdam, The Netherlands for five days of collaboration, hacking and community outreach. This meeting was the first time most of us had met in person, as the company has grown by more than 15 employees since our last All Hands in April 2013. We can&apos;t recount all the great stuff we did in one blog post, but we wanted to share some highlights of our meeting, particularly our development discussions.
</p>
<h2>Development Discussions</h2>
<p>We spent several days with all the developers talking about progress we have made thus far and brainstorming about where we should go across all our different products, Elasticsearch, Logstash, Kibana, Elasticsearch-Hadoop, and the various language clients.
</p>
<h3>Elasticsearch</h3>
<p>We checked status of features aimed at our 1.0 release planned for early next year. We chatted with <a href="http://twitter.com/imotov">Igor</a> about <a href="https://github.com/elasticsearch/elasticsearch/issues/3826">Snapshot/Restore</a>, <a href="http://twitter.com/mvgroningen">Martijn</a> talked about the <a href="https://github.com/elasticsearch/elasticsearch/issues/3173">distributed Percolator</a>, and <a href="http://twitter.com/uboness">Uri</a> discussed <a href="https://github.com/elasticsearch/elasticsearch/issues/3300">Aggregations</a>.
</p>
<p>We also brainstormed about some future work we would like to see in Elasticsearch, and we would love to share some of these initial thoughts with all of you:
</p>
<h4>Field Data</h4>
<p>Heavy brainstorming on how to improve the experience of users when using features that require loading all field data and the potential memory problems they run into. The discussions ranged from talking about what use cases &#x201C;doc values&quot; will help solve (which <a href="http://twitter.com/jpountz">Adrien</a> has implemented, watch out for a forthcoming blog post), improved memory usage, and potentially other storage means for the field data.
</p>
<h4>Machine Learning</h4>
<p>Britta led a wonderful session about ML, asking all of us what we would like to see in ML, and to see what directions might be interesting for us to pursue in the future. We ended up breaking things into several buckets, amongst them classification, predictive functions, sentiment analysis, NLP, and more. The session was mainly exploratory in terms of understanding what we have out there and what might apply to Elasticsearch.
</p>
<h4>Changes API</h4>
<p>One of the exciting features that we want to try and tackle post 1.0 release is registering for changes happening in an index. We mainly brainstormed about how something like this can be implemented - it is quite a complicated feature. We had a couple of different options for storing or virtually storing the change log, but the best choice isn&apos;t yet clear.
</p>
<h4>Backwards Compatibility</h4>
<p>The plan is to have wire level backwards compatibility in 1.0, and we also discussed what it means to have backwards compatibility on the index level over multiple Elasticsearch versions (for example, in keeping backward comp. on the analysis chain level).
</p>
<h4>Field Collapsing / Inner Hits</h4>
<p>We again fleshed out what is needed in order to properly support field collapsing in a distributed environment execution, as well as the ability to get inner hits (for nested / parent child cases). We have a good idea on the type of refactoring we need in our search execution infrastructure, and hope to tackle it post 1.0.
</p>
<h4>Tests</h4>
<p>If you haven&apos;t noticed, <a href="http://twitter.com/s1m0nw">Simon</a> and the rest of us have been hard at work at Elasticsearch to improve our testing, mainly around introducing randomized testing and improving our integration tests. We continued the discussion regarding how to move forward with our testing enhancements, including other places where we can benefit from creating an infrastructure for our tests.
</p>
<h3>Test Infrastructure</h3>
<p><a href="http://www.linkedin.com/pub/bill-hwang/0/b/777">Bill</a> has been working on setting up an extensive test infrastructure for all our products, including running all our tests over multiple JDK versions, multiple operating systems, multiple machine types and other variants. We test all our products using it, and we plan to open it up relatively soon for people to see it.
</p>
<h3>Performance Infrastructure</h3>
<p>We also discussed how to automate our benchmarking code and make it both consistent and applicable across different projects. We brainstormed on creating an infrastructure that all our projects can use, and creating performance reports streamed to Elasticsearch and visualized in Kibana, across different infrastructure variants (OS types, machines types, &#x2026;).
</p>
<h3>Documentation</h3>
<p><a href="http://twitter.com/clintongormley">Clint</a> discussed all the work that went into creating the infrastructure for all our documentation across all projects. If you haven&apos;t seen it, our reference guide has moved to live with the code, and we have a framework in place that slurps it up and builds it to be displayed on the web site. The same infrastructure can be used for docs across all our projects, and for example, elasticsearch-hadoop is already using it. The plan is to have the infrastructure also build our forthcoming book.
</p>
<h3>Logstash</h3>
<p><a href="http://twitter.com/nickethier">Nick</a> (with <a href="http://twitter.com/jordansissel">Jordan</a> participating remotely) gave a great introduction on how Logstash works to all the developers, and we bounced ideas around about how to make it even better. We have already, in the past couple of weeks, significantly improved the performance of the elasticsearch_http output, added multiple outputs to improve data throughput to elasticsearch (as an example output), and improved the performance of the grok filter.
</p>
<h3>Kibana</h3>
<p><a href="http://twitter.com/rashidkpc">Rashid</a> gave a demo on The State of Kibana, and we bounced several ideas around on how it should move forward. One of the exciting features that came out of this discussion is Annotations, with the ability to have annotations displayed on a histogram (for example) for important events during the (time) lifecycle of the data. Rashid didn&apos;t forget to mention how excited he is regarding Aggregations in Elasticsearch ;).
</p>
<h3>Clients</h3>
<p>Clint, <a href="http://twitter.com/honzakral">Honza</a>, <a href="http://twitter.com/karmiq">Karel</a> and <a href="http://twitter.com/zacharytong">Zach</a> talked about the recently open sourced set of language clients to Elasticsearch (PHP, Perl, Python and Ruby). The output of the effort was creating a spec of Elasticsearch APIs, and we discussed how we can automate generating the spec out of the Elasticsearch codebase. The other nice side effect is the fact that the spec now includes a generic YAML based test infrastructure that all the clients run, allowing us to write tests in a single place, and have them execute by all the different clients (which we want to also execute as part of the Elasticsearch tests as well).
</p>
<p>We also spent time with <a href="http://twitter.com/SpencerAlger">Spencer</a> around how to develop a javascript/node.js client, and Chander for a future .NET client.
</p>
<h3>Team Growth</h3>
<p><img src="https://api.contentstack.io/v2/assets/575e4cba0342dfd7382638e7/download?uid=blt1c8ba7d31dc752a5?uid=blt1c8ba7d31dc752a5" data-sys-asset-uid="blt55b8a27de53492c1" alt="The dev team at our Wednesday night company dinner, still talking future features while eating Everything on a Stick." width="300" height="200" style="float:right; padding:0 0 20px 20px;">We are not a small development team anymore, and we are getting more distributed by the day. We discussed how to communicate better within our team, how to properly develop features (feature branches, on dev own repo and making pull requests, &#x2026;), and the review process that goes with it. We also spent time to see how we can better manage our time between all the tasks our developers do, be it helping out on the mailing list and IRC, coding &amp; docs/books, talking at conferences and helping out customers (yea, support from our company means talking to the developers, which we are very proud about).
</p>
<h2>The Business of Our Business</h2>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbb9e7a83165490c8e0/download?uid=blt3a3af36195e89c60?uid=blt3a3af36195e89c60" data-sys-asset-uid="blt5ba5a7d13642d8e8" alt="The Elasticsearch sales team hard at work." width="300" height="200" style="float:right; padding:0 0 20px 20px;">While the majority of our heavy lifting was on the code side of the house, the rest of our team spent the week figuring out how to bring Elasticsearch to even more users and customers. Our marketing team focused on making sure that our meetup program rocks, that we&apos;re seeing folks at the right conferences and that information about Elasticsearch&apos;s product offerings is most robust. Our operations and administrative teams had extensive discussions on how to best improve our company processes so we can maintain our laser focus on producing Elasticsearch and serving our customers. Our sales team, of course, ran through &#x201C;the numbers,&quot; and things are looking great: lots of happy customers using our <a href="/subscriptions">support offerings</a> and great feedback from attendees of our <a href="/training">global training courses</a>.
</p>
<h2>Community Outreach</h2>
<p>As with all times when someone from the Elasticsearch team visits a particular city, we were excited to host a community meetup in Amsterdam. We welcomed more than 65 people to the Elasticsearch office for a full evening of talks, beer and pizza. While we usually have two shorter presentations on Elasticsearch at meetups - one on features, one on a particular use case - this time around we had the whole dev team talk about what they&apos;re working towards for 1.0 in a series of lightning talks.
</p>
<p>You can read even more about the Meet-the-Devs Meetup in <a href="/blog/meet-the-devs-meetup-in-amsterdam">Zachary Tong&apos;s blog post</a>.
</p>
<h2>Looking Forward</h2>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbb43e9adc53871429a/download?uid=blt6749c476dece23ac?uid=blt6749c476dece23ac" data-sys-asset-uid="bltba93a97fc7f1096f" alt="Our guests learning all about debugging performance quirks in GitHub&apos;s new cluster from Elasticsearch core developer Drew Raines." width="300" height="200" style="float:right; padding:0 0 20px 20px;">That&apos;s a wrap for our October All Hands, but we&apos;re all looking forward to all getting together again sometime in the Spring, likely at our brand new Silicon Valley headquarters in Los Altos, California, US. In the coming weeks, we&apos;ll be talking even more about the future of Elasticsearch, Logstash and Kibana, sharing more in-depth insights into the discussions we had in Amsterdam.
</p></body></html>

From Amsterdam with Love

<html><head><style>h2 {clear:left;}</style>
</head><body><p>Last Thursday, the <a href="http://www.meetup.com/ElasticSearch-NL/" target="_blank">Elasticsearch Netherlands</a> group held its eighth meetup. This meetup was a bit special, however, since it also coincided with the Elasticsearch company meeting. Rather than the traditional two presentation format, the Elasticsearch developers gave a number of lightning talks about what they have been working on.</p>
<p>A big thanks to <a href="http://www.trifork.com/">Trifork</a> for hosting the meetup! Around 65 people showed up, with lots of great discussion before, during and after the event.</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbb12798d8e381fde02/download?uid=blt55cefc87742319bc?uid=blt55cefc87742319bc" data-sys-asset-uid="bltd6b31df9900e524c" alt="dev-meetup-nl-2013-audience.jpg" style="max-width:100%;"></p>

<h2>Drew Raines - Life after EC2</h2>
<p>Elasticsearch talks are often at a very high level: a new feature, a new type of query, how to implement XYZ functionality. <a href="http://twitter.com/drewr">Drew&apos;s</a> talk went the other direction and discussed the ramifications of running on bare metal hardware. In particular, he discussed how he and <a href="http://github.com">GitHub</a> debugged a performance quirk with their new cluster.
</p>
<p>Long story short, they discovered that the default Linux I/O scheduler is atrocious when using SSDs, or RAID over fast disks. After switching their scheduler to Noop or Deadline, they saw a remarkable <strong>550x</strong> performance increase!
</p>
<p>Check out <a href="https://speakerdeck.com/elasticsearch/life-after-ec2" target="_blank">Drew&apos;s presentation</a> for more details (especially if you are using RAID or SSDs!)
</p>
<h2>Costin Leau - Real-time data with Hadoop</h2>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbbd8edd48f76936588/download?uid=blt38dab71f396c7f59?uid=blt38dab71f396c7f59" data-sys-asset-uid="blteecd143743f57377" alt="dev-meetup-nl-2013-costin-leau-hadoop.jpg" width="300" height="287" style="float:left; padding:0 20px 20px 0;"><a href="http://twitter.com/costinl">Costin</a> talked about his new <a href="/blog/elasticsearch-and-hadoop">Elasticsearch-Hadoop</a> integration. He highlighted what Hadoop is good at...and where it is lacking. Real-time processing, analytics, and geo operations are all difficult for Hadoop. Elasticsearch, on the other hand, excels at these problems.
</p>
<p>Costin explained how Elasticsearch can be integrated with an existing Hadoop installation to provide functionality that is difficult or slow in Hadoop. He then showed examples in several popular Hadoop frameworks and some benchmark data. Costin&apos;s slides are <a href="https://speakerdeck.com/elasticsearch/real-time-search-and-big-data-using-elasticsearch" target="_blank">available on Speakerdeck</a>.
</p>
<h2>Clinton Gormley, Karel Minarik, Honza Kral, Zachary Tong - Unleashing the Clients</h2>
<p>The client team gave a short overview of the <a href="/blog/unleash-the-clients-ruby-python-php-perl/">new language clients</a>. They talked about the motivations behind creating the clients, as well as the need for a unified interface, consistent testing framework and pluggable components.
</p>
<h2>Britta Weber - Function Score Query</h2>
<p><a href="http://twitter.com/a2tirb">Britta</a> talked about the new <a href="/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html">Function Score Query</a>. Partially known for its outrageously awesome <a href="https://github.com/elasticsearch/elasticsearch/issues/3423">ticket on GitHub</a>, the function score allows you to tweak scoring with complicated mathematical functions. Britta discussed how it is used and some example scenarios.
</p>
<h2>Igor Motov - Snapshot and Restore</h2>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbb9985d589763771fb/download?uid=blt04b22b3e1d5c9e8a?uid=blt04b22b3e1d5c9e8a" data-sys-asset-uid="bltab5d1372257ee061" alt="dev-meetup-nl-2013-igor-motov-snapshot-restore.jpg" width="300" height="238" style="float:left; padding:0 20px 20px 0;"><a href="http://twitter.com/imotov">Igor</a> presented the Snapshot and Restore functionality that he has been working on. Slated for version 1.0, Snapshot and Restore will allow users to snapshot their cluster to a shared repository (S3, shared FS, etc). Using another API, they can restore these incremental snapshots to the cluster. Snapshot and Restore will make backing up and delayed replication vastly easier in the future. You can get more detail from <a href="https://speakerdeck.com/elasticsearch/restore-in-elasticsearch-1-dot-0" target="_blank">Igor&apos;s slides</a>.
</p>
<h2>Alexander Reelsen - Completion Suggester</h2>
<p><img src="https://api.contentstack.io/v2/assets/575e4cba58208ba076e28813/download?uid=blt35aa64170548d195?uid=blt35aa64170548d195" data-sys-asset-uid="bltce177027b430ffa7" alt="dev-meetup-nl-2013-alex-reelsen-completion-suggester.jpg" width="254" height="300" style="float:left; padding:0 20px 20px 0;"><a href="http://twitter.com/spinscale">Alex</a> discussed the new <a href="/guide/en/elasticsearch/reference/current/search-suggesters-completion.html">Completion Suggester</a>. Using advanced Finite State Transducers, the completion suggester provides autocomplete-style suggestions in milliseconds. Importantly, it only requires one request, as opposed to approaches like ngrams which require multiple round-trips - users need suggestions <em>before</em> they finish typing. You can learn more from <a href="https://speakerdeck.com/elasticsearch/elasticsearchs-completion-suggest-feature" target="_blank">Alex&apos;s slides</a>.
</p>
<h2>Uri Boness - Aggregations</h2>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbc9e7a83165490c900/download?uid=bltb4b2a184a4a4a01f?uid=bltb4b2a184a4a4a01f" data-sys-asset-uid="blt0da19ed5d0e4574a" alt="dev-meetup-nl-2013-uri-boness-aggregations.jpg" width="300" height="249" style="float:left; padding:0 20px 20px 0;"><a href="http://twitter.com/uboness">Uri</a> demoed his much-anticipated <a href="https://github.com/elasticsearch/elasticsearch/issues/3300">Aggregations Feature</a>. Designed to replace facets, aggregations provide a framework where individual aggregations can be composed into nearly limitless combinations. He first built an example that mirrored a traditional facet, then proceeded to enhance it with more nested aggregations.</p>
<h2>Simon Willnauer - Lucene Pipeline</h2>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbb12798d8e381fde0d/download?uid=blt9162068db4e208a5?uid=blt9162068db4e208a5" data-sys-asset-uid="blt04f5eb008fda677f" alt="dev-meetup-nl-2013-simon-willnauer-lucene-pipeline.jpg" width="289" height="300" style="float:left; padding:0 20px 20px 0;"><a href="http://twitter.com/s1m0nw">Simon</a> wrapped up the meetup by talking about some low-level improvements that will be coming to Apache Lucene in the future. In particular, he discussed his frustrations with the standard Apache Lucene highlighters, as well as depreciating Span queries in favor of adding payloads to non-span queries. You&apos;ll enjoy <a href="https://speakerdeck.com/elasticsearch/lucene-is-awesome-but-dot-dot-dot" target="_blank">Simon&apos;s slides</a>, including the obligatory photo of his fellow Apache Lucene committer Uwe Schindler.
</p>
<h2>Wrap up</h2>
<p>Big thanks to everyone who showed up for the meetup, and everyone who stayed around to chat after the presentations. The Elasticsearch team had a great time presenting and really enjoyed fielding questions about their respective projects.
</p></body></html>

Meet-the-Devs Meetup in Amsterdam

City Bikes Part two

<html><head></head><body><p>Last week, I attended the first <a href="http://www.hackreduce.org/ai1ec_event/hopper-hackathon/?instance_id=" target="_blank">Hopper Elasticsearch Hackathon</a>. This all day event was co-organized by <a href="http://www.hopper.com/" target="_blank">Hopper</a>, a Cambridge, Massachusetts, US based company that is using Elasticsearch to plow through a massive amount of travel data. Also organizing the hackathon were the fine folks from the <a href="http://www.meetup.com/Elasticsearch-Boston/" target="_blank">Boston Elasticsearch Meetup</a>, a diverse group of Elasticsearch users in the Greater Boston area.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbb0342dfd73826390d/download?uid=bltbfae54ebcc925f34?uid=bltbfae54ebcc925f34" data-sys-asset-uid="blt1599c83fc10e8e05" alt="Boston Elasticsearch Hackathon" style="max-width:100%;">
</p>
<p>The hackathon gathered more than 40 software engineers, students and other search and open source software enthusiasts eager to learn more about Elasticsearch and share their knowledge with others. The goal of the Hackathon organizers was to create an inclusive event that would be interesting to both experts and novices alike, so we started with <a href="http://www.slideshare.net/imotov/boston-hopper-elasticsearch-hackathon" target="_blank">short introduction into Elasticsearch</a> to bring all attendees up to speed before getting started coding.
</p>
<p>During a typical one-day hackathon, attendees have only 5-7 hours to write code, and getting stuck during these hours can derail otherwise interesting project. In order to improve participants&#x2019; experiences and allow them to make as much progress as possible on their projects in just a few short hours, the Hackathon organizers convened a group of local Elasticsearch experts from Hopper, Traacker and Elasticsearch Inc as mentors. I was one of the six mentors during this hackathon, and it was an amazing experience.
</p>
<p>This event wasn&#x2019;t my first Elasticsearch-themed hackathon, but what made it very different for me was the number of new Elasticsearch users and the amount of progress that they made during a very short period of time. This Hackthon was the first where we introduced the <a href="/blog/unleash-the-clients-ruby-python-php-perl">new client API</a> and <a href="https://github.com/elasticsearch/kibana">Kibana</a>, and I think this two projects made all the difference. With the lightweight API, new users were able to very quickly translate concepts described in the elasticsearch.org user guide into working code. At the same time, Kibana made it simple to look at very complex data without requiring a lot of new code. In just seven hours by combining Twitter river with Kibana, one of the teams managed to create Twitter analytics solution for analyzing demographics of topics of interest (in their case tweets about Justin Bieber). Two winning teams analyzed Twitter to determine the popularity of TV shows and did sentiment analysis on political topics. But it wasn&apos;t all about Twitter; other hacks ranged from finding the shortest path between two Wikipedia articles to finding hidden relationships between Enron employees based on their email messages.
</p>
<p>Two teams won prizes in this hackathon, and it was wonderful to be a part of something where everyone was learning stuff, having fun and left with a sense of accomplishment.
</p>
<p>Many thanks to our sponsors: Elasticsearch Inc., SoftLayer and Traackr. You can read even more about the hackathon on the <a href="http://hoppertravel.tumblr.com/post/62831226107/congratulations-to-team-twittertv-jony-f-justin">Hopper Blog</a>.
</p></body></html>

Hopper Hosts Hackathon with Elasticsearch

<html><head></head><body><p>
	Welcome to&#xA0;<i>This week in Elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the Github repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
</p><h2>Elasticsearch core</h2><ul>
	<li>Upgrade to Lucene 4.5&#xA0;(<a href="https://github.com/elasticsearch/elasticsearch/commit/6b026119718a616fc76578aff3b20ceda023d403">commit</a>, 0.90 &amp; master)</li>
	<li>Elasticsearch now uses <a href="http://labs.carrotsearch.com/hppc.html">HPPC</a> instead of trove&#xA0;(<a href="https://github.com/elasticsearch/elasticsearch/commit/088e05b36849f0659d3b20cd0376c1a89e2b426b">commit</a>, 0.90 &amp; master)</li>
	<li>Better logic ignoring filters when parsing (there is a difference when a filter is empty, one cannot simply do a match all or none filter) &#xA0;(<a href="https://github.com/elasticsearch/elasticsearch/issues/3838">#3838</a>, 0.90 &amp; master)</li>
	<li>The <code>@Required</code> annotation has been removed&#xA0;(<a href="https://github.com/elasticsearch/elasticsearch/commit/a03013ea25db12573f94edc0593c24107291c722">commit</a>, 0.90 &amp; master)</li>
	<li>Parsing of completion field content got more strict&#xA0;(<a href="https://github.com/elasticsearch/elasticsearch/commit/19099ab3da45ea583f90e9b6deb0c5d8576387d1">commit</a>, 0.90 &amp; master)</li>
	<li>A race condition when indexing and merging mapping updates from master was fixed&#xA0;(<a href="https://github.com/elasticsearch/elasticsearch/issues/3844">#3844</a>, 0.90 &amp; master)</li>
	<li>Warm-up of merged segments can be improved without configuring warm-up queries&#xA0;(<a href="https://github.com/elasticsearch/elasticsearch/issues/3819">#3819</a>, 0.90 &amp; master)</li>
	<li>Calling <code>bin/plugin</code>&#xA0;with wrong parameters could lead to accidental removing of the <code>bin</code> directory (<a href="https://github.com/elasticsearch/elasticsearch/issues/3847">#3847</a>, 0.90 &amp; master)</li>
</ul><h3>Elasticsearch Ecosystem</h3><p>
	Here&apos;s some more information about what is happening in the ecosystem we are maintaining around Elasticsearch, including plugin and driver releases, as well as news about Logstash and Kibana.
</p><ul>
	<li>You asked, we answered: All the news you can use about the <a href="http://www.elasticsearch.org/blog/docs-docs-docs/">future of Elasticsearch documentation</a>.</li>
	<li>Check out <a href="http://jaxenter.com/spotlight-on-elasticsearch-48409.html">this interview</a> with <a href="http://twitter.com/kimchy">Shay Banon</a> about his vision of elasticsearch - both the product and the company.</li>
	<li>An interesting blog post about <a href="http://www.poudro.com/blog/building-an-elasticsearch-index-offline-using-hadoop-pig/">building an elasticsearch index offline using Hadoop Pig</a></li>
	<li>A drupal module for <a href="https://drupal.org/project/elastic_search_clients">managing elasticsearch clients</a> has been released.</li>
	<li>The clojure client <a href="http://blog.clojurewerkz.org/blog/2013/10/08/elastisch-1-dot-3-0-beta4-is-released/">Elastisch</a> has been released in version 1.3.0-beta4</li>
</ul><h3>Slides</h3><ul>
	<li><a href="https://twitter.com/dadoonet">David Pilato</a> delivered two presentations at Open World Forum. He gave an <a href="https://speakerdeck.com/dadoonet/elasticsearch-workshop">Elasticsearch workshop</a>&#xA0;as well as a introduction <a href="https://speakerdeck.com/dadoonet/open-world-forum-elastify-your-app-from-sql-to-nosql-in-40-mn">how to elastify</a> your app.</li>
	<li>Julien Pivotto gave a nice <a href="http://www.slideshare.net/roidelapluie/logstash-and-friends">introduction into logstash</a></li>
	<li><a href="http://twitter.com/lucacavanna">Luca Cavanna</a> uploaded his slides about <a href="https://speakerdeck.com/javanna/whats-new-in-percolator">what&apos;s up with the new percolator</a></li>
</ul><h3>Where to find Us</h3><h4>Germany</h4><ul>
	<li><a href="http://twitter.com/s1m0nw">Simon Willnauer</a> will speak at <a href="http://gotocon.com/berlin-2013/">GOTO Berlin</a> on &quot;With a hammer in your hand ... Elasticsearch&quot;. Simon&apos;s talk is scheduled for Friday, October 18th at 11:30 AM and he&apos;ll be hanging out with our Community Manager, <a href="http://twitter.com/lhawthorn">Leslie Hawthorn</a> in the exhibits hall throughout the conference. Stop by the Elasticsearch stand and say hello!</li>
	<li>The <a href="http://www.meetup.com/ElasticSearch-UG-Berlin/events/134717942/">Berlin Elasticsearch User Group</a> will convene for their regular monthly meeting on October 29th.</li>
</ul><h4>Hungary</h4><p>
	<a href="http://twitter.com/honzakral">Honza Kral</a>&#xA0;will talk about Elasticsearch at&#xA0;<a href="http://13.rupy.eu/">RuPy</a>&#xA0;in Budapest on October 12th. RuPy is a conference to bring together programmers and communities of different programming languages like Ruby, Python, Clojure or JavaScript.
</p><h4>Netherlands</h4><p>
	You can meet almost all of the Elasticsearch, Kibana and Logstash developers tomorrow in Amsterdam at the <a href="http://www.meetup.com/ElasticSearch-NL/">Elasticsearch Netherlands Meetup</a> and ask us whatever you want to know. In addition  to our Q&amp;A session, <a href="http://twitter.com/drewr">Drew Raines</a> will talk about running Elasticsearch on bare metal and <a href="http://twitter.com/costinl">Costin Leau</a> will talk about our Hadoop integration.
</p><h4>United Kingdom</h4><p>
	<a href="https://twitter.com/Richardp82">Richard Pijnenburg</a>&#xA0;will give a talk at <a href="http://jaxlondon.com/">JAX London</a> about how to <a href="http://jaxlondon.com/sessions/level-your-logging">Level Up Your Logging</a> on Wednesday, October 30th.
</p><h4>United States</h4><ul>
	<li><a href="http://twitter.com/uboness">Uri Boness</a> &#xA0;&amp; <a href="https://twitter.com/ZacharyTong">Zach Tong</a> will be presenting together at the Washington DC meetup on a very interesting combination of subjects-&#xA0;<a href="http://www.meetup.com/Elasticsearch-Washington-DC/events/142011702/">Elasticsearch&apos;&#xAD;s New Aggregations feature <em>and</em> Genomic Sequencing with Elasticsearch</a>. The meetup is scheduled for October 22nd.</li>
	<li><a href="http://www.linkedin.com/pub/steve-mayzak/2/300/508">Steve Mayzak</a> will be speaking on Getting Started with Elasticsearch at the second Chicago Elasticsearch meetup, scheduled for October 24th.</li>
	<li>Ralph Meijer from Mailgun will be talking at the San Francisco meetup scheduled for October 25th on&#xA0;<a href="http://www.meetup.com/Elasticsearch-San-Francisco/events/142398022/">The Business End of the [Mail]gun</a>.</li>
</ul><h3>Where to Find You</h3><p>
	Are you hosting an Elasticsearch meetup or giving a talk about Elasticsearch? We would love to know so we can feature that information in future editions of&#xA0;<em>This Week in Elasticsearch</em>. Just let our&#xA0;<a href="http://twitter.com/lhawthorn">Community Manager</a>&#xA0;know what you&apos;re up to and we&apos;re happy to help promote your efforts.
</p><p>
	Oh yeah, we&apos;re also&#xA0;<a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you&apos;d like us to find you for employment purposes, just&#xA0;<a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &#xA0;We care more about your skill set and passion for Elasticearch, Kibana and Logstash than where you rest your head.
</p><h3>Trainings</h3><p>
	If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
</p><ul>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/cologne/2013-10-17">Cologne</a>&#xA0;- October 17, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/washington-dc/2013-10-21">Washington, D.C.</a>&#xA0;- October 21, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/london/2013-10-21">London, U.K.</a> - Octoner 22, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/zurich/2013-10-29">Zurich</a>&#xA0;- October 29, 2013</li>
</ul></body></html>

This Week in Elasticsearch - October 09, 2013

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><h2>Document Parsing and Tokenization</h2><p>We continue the Indexing for beginners article with an introduction to the document parsing process and use some space to explain the concept of tokenization and tokens.
</p><section id="document-parsing" class="level2">
<h2><a href="#document-parsing">Document Parsing</a></h2>
<p>Read part 1 of this article here: <a href="/blog/found-indexing-for-beginners-part1/" title="/blog/found-indexing-for-beginners-part1/">What is an index?</a>
</p>
<p>Say you are running a recipe website where visitors can add their own recipes. Obviously, you want the recipes to be searchable, so you set up a search engine to take care of this. Every time someone creates or uploads a new recipe to the website, the search engine is tasked with making the new recipe searchable. The first step in this process is is to scan and process the text in the recipe. This process is generally called <em>document parsing</em>. Other common names for this process are <em>text processing</em>, <em>text analysis</em>, <em>text mining</em> and <em>content analysis</em>. In the document parsing step the search engine creates a list of the words - or <em>terms</em> - in the recipe, accompanied by a reference - or mapping - to the recipes containing these terms. This list is basically what makes up the index. Lastly, the search engine will save the index to disk to be able to retrieve it later. Also, parts of the index will reside in memory, to make the search process faster.
</p>
<section id="full-text-search-engines" class="level3">
<h3><a href="#full-text-search-engines">Full-Text Search Engines</a></h3>
<p>Search engines that scan and process the entire text in a document prior to indexing, are so-called full-text search engines. Lucene, which Elasticsearch and Solr are built on top of, is an example of a full-text search engine.
</p>
</section>
<section id="tokenization" class="level3">
<h3><a href="#tokenization">Tokenization</a></h3>
<p>When the recipe text is scanned and fed into our search engine, the first thing the search engine has to do is to make sense of the data it receives. This might seem trivial to us, but for a computer it&apos;s not that simple. When you and I read a text, we automatically break up the text into characters, words and sentences. This is a process we have learned and automated through years of practice. We know that words are separated by spaces, and that sentences are separated by punctuation marks. Computers, on the other hand, has no such built-in understanding of so-called <em>natural languages</em><a href="#fn1" class="footnoteRef" id="fnref1"><sup>1</sup></a>. For a computer, the text is just a stream of characters where a white space, a punctuation mark or the character &apos;t&apos; has no special meaning. Therefore, computers has to be programmed to break up text into their distinct elements, such as words and sentences. This process is called <em>tokenization</em> and the different chunks, usually words, that constitute the text are called <em>tokens</em>. Here is an example how a sentence is tokenized:
</p>
<p>The sentence
</p>
<pre>&lt;code&gt;Heat slowly, stir gently while adding twenty-five grams of flour.&lt;/code&gt;
</pre><br><br>
<p>is split into the following tokens
</p>
<pre>&lt;code&gt;[heat] [slowly] [stir] [gently] [while] [adding] [twenty-five] [grams] [of] [flour]&lt;/code&gt;
</pre><p><br></p>
<p><br>
</p>
<p>We see that the tokenizer used here knows how to separate the words from each other by looking for spaces and punctuation characters. The hyphen is a bit more tricky though; should we split <em>twenty-five</em> into two tokens: [twenty] and [five], or should it be only one token: [twenty-five]? And what if you want to match a search for 25?
</p>
<p>Tokenizers can be quite sophisticated, and they should be able to handle text constructs such as abbreviations (ca., kg., dr.), numbers (10 or ten, 3x10^9 or 3 billion), hyphens (twenty-five), end-of-line-hyphens (a word split across two lines), lexical hyphens (pre-, multi-), contractions (I&apos;m, it&apos;s), compound words (motorboat or Kraftfahrzeug), and so on. Additionally, in many cases the tokenizer needs to be language aware. This is especially true when tokenizing non-phonetic languages, where one grapheme<a href="#fn2" class="footnoteRef" id="fnref2"><sup>2</sup></a> might indicate an entire word or a syllable, for example in Chinese or Japanese.
</p>
<p>There are many specialized tokenizers, for example CamelCase tokenizer, URL tokenizer, path tokenizer and N-gram tokenizer. These are designed to handle special language constructs, especially in relation to artificial languages like mathematics or programming languages.
</p>
<p>Just to give you an idea how these tokenizers work, here is an example of a path tokenizer, where the path:
</p>
<pre>&lt;code&gt;C:\WINDOWS\system32\rundll32.exe&lt;/code&gt;
</pre><br><br>
<p>is split into the following tokens:
</p>
<pre>&lt;code&gt;[c:] [c:/windows] [c:/windows/system32] [c:/windows/system32/rundll32.exe]&lt;/code&gt;
</pre><br><br>
</section>
<section id="stop-words" class="level3">
<h3><a href="#stop-words">Stop Words</a></h3>
<p>Sometimes we want to avoid certain words from being indexed. For instance, in many cases it would make no sense to store the words on, for, a, the, us, who etc. in the index. Such terms are called stop words. However, in most modern search engines, stop words are not problematic, and by indexing stop words it&apos;s possible to search for the band &quot;The Who&quot; or exact phrases and even entire sentences.
</p>
</section>
<section id="relevancy" class="level3">
<h3><a href="#relevancy">Relevancy</a></h3>
<p>Essentially, all the tokens - in our case all the words in our Tomato recipes - are stored as terms in the index, together with the mapping to all occurrences of the terms. For example, if a visitor searches for the term &quot;tomato&quot;, the search request will return every recipes containing the word &quot;tomato&quot;, and it will certainly include more than just tomato soup recipes.
</p>
<p>This rudimentary way of indexing does not take into account the relevancy of the search results. In fact, one of the most important considerations when setting up a search engine is how you tell the search engine to process and analyze the text, and how to build the indexes, as this will heavily influence how you construct your queries and filters (explained later), which in turn will determine the relevancy of the search results. We will talk more about relevancy and relevance models in an upcoming article.
</p>
<p>Previous: <a href="/blog/found-indexing-for-beginners-part1/" title="/blog/found-indexing-for-beginners-part1/">What is an index?</a>
</p>
<p>Next: <a href="/blog/found-indexing-for-beginners-part3/" title="/blog/found-indexing-for-beginners-part3/">What does an index look like?</a>
</p>
<div class="references">
</div>
</section>
</section><section class="footnotes">
<hr>
<ol>
	<li id="fn1">Languages that we humans speak or write are generally called natural languages.<a href="#fnref1">&#x21A9;</a></li>
	<li id="fn2">A grapheme is the smallest unit in a written language, for instance the character &apos;w&apos; in the Latin alphabet, or the sign representing the word &apos;table&apos; in Chinese.<a href="#fnref2">&#x21A9;</a></li>
</ol>
</section></body></html>

Indexing for Beginners, Part 2

<html><head></head><body><p>There&apos;s not much point in having an amazing piece of software, if users can&apos;t figure out how to use it.  Over the years, we&apos;ve had several complaints about our documentation: it&apos;s difficult to navigate, it&apos;s difficult to know where to start, etc. This is not unusual -- documentation is the bugbear of most software projects. </p>
<p>But that doesn&apos;t mean we can&apos;t make it better! So let&apos;s tell you what we&apos;ve done so far and what we&apos;ve got planned.</p>
<h2>Docs and code, code and docs</h2>
<p>Documentation should be an accurate reflection of the code it describes.  Up until now, we&apos;ve had the code and the documentation in separate repositories.  Now, the code lives with the docs in the <a href="https://github.com/elasticsearch/elasticsearch/tree/master/docs">main Elasticsearch repository</a>. This is good news both for users and  developers.</p>
<p>Developers can include code and documentation in the same pull request and users can now access <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/index.html">the relevant documentation for the version of Elasticsearch</a> that they are using. We build a separate set of documentation for each major version, ie <code>0.90.*</code>, <code>1.00.*</code>, <code>master</code>.  Changes between point versions are marked up as in this example in the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#query-dsl-function-score-query"><code>function_score</code> query</a>.</p>
<p>So far, we have focussed on migrating the docs to the new build system. There are still a few niggles in the new docs. Please bear with us as we work on fixing them. Next we plan on improving them: making them more consistent, more complete, better explanations and more examples.  This is a long-term project but you should notice a steady improvement in the documentation over time. We welcome feedback, suggestions and pull-requests! </p>
<h2>One build system to rule them all</h2>
<p>All of our projects will be using the same framework to provide their documentation.  We have chosen to use <a href="http://www.methods.co.nz/asciidoc/userguide.html">AsciiDoc</a> as our markup language, because it is easy to read, very expressive and, via DocBook, gives us the assurance that we don&apos;t have any bad links.  Also, because we build all docs from each project together, we can ensure that cross-document links work and continue to work when edits are made later on.</p>
<p>Another good reason to use AsciiDoc and DocBook is that DocBook can produce output as HTML, PDF, EPUB and Mobi... and it is the preferred format for <b>books published by O&apos;Reilly Media</b>!</p>
<h2>The Definitive Guide</h2>
<p>Yes! We are in the process of writing <em>The Definitive Guide to Elasticsearch</em>, to be published by O&apos;Reilly Media. It was important for us to find the right partner for this book: we are an open-source company and we wanted the book to be open-source too.  O&apos;Reilly has a long track record of supporting the open-source world and we are thrilled to be working with them on this important project.</p>
<p>While our reference documentation is intended to explain how to use a particular API and what parameters it accepts, the book will focus on how to approach and solve problems: how to improve search relevance, how to handle multiple languages, how to tune Elasticsearch for your workload. It will be suitable for beginners who know nothing about search, but will have enough meat in it to satisfy the more experienced as well.</p>
<p>The book will be freely available online in HTML format both on our website and on O&apos;Reilly&apos;s site and, when finished, will be available to purchase in hard copy from O&apos;Reilly. The first part of the book is due to be published online in December this year. New chapters will be added as they are finished, and hardcopies will available for purchase in mid-2014.</p>
</body></html>

Docs! Docs! Docs

<section id="post_content">
		<article>
			<p>Several months ago, we announced on the <a href="https://groups.google.com/forum/#!searchin/elasticsearch/elasticsearch$20hadoop/elasticsearch/RW_yUgO5N4Q/nAjdPzBxcKUJ">mailing list</a> the transformation of <a href="https://github.com/elasticsearch/elasticsearch-hadoop">elasticsearch-hadoop</a> into the <i>go-to</i> place for all things related to Elasticsearch and Hadoop. Today, we have <a href="http://www.elasticsearch.org/overview/hadoop/">published</a> the first milestone (1.3.0.M1) based on the new code-base that has been in the works for the last few months.</p>
<h1>Native integration with Hadoop</h1>
<p>Using Elasticsearch in Hadoop has never been easier. Thanks to the deep API integration, interacting with Elasticsearch is similar to that of HDFS resources. And since Hadoop is more then just vanilla Map/Reduce, in elasticsearch-hadoop one will find support for <a href="http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html">Apache Hive</a>, <a href="http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/pig.html">Apache Pig</a> and <a href="http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/cascading.html">Cascading</a> in addition to plain <a href="http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html">Map/Reduce</a>.</p>
<p>To wit, writing data to Elasticsearch is just Hadoop as usual:</p>
<h3>Pig script</h3>
<pre class="prettyprint lang-default-markup">-- load data from HDFS into Pig using a schema
A = LOAD 'in/artists.dat' USING PigStorage()
    AS (id:long, name, url:chararray, picture: chararray);
-- ETL the data to your heart content
B = FOREACH A GENERATE name, TOTUPLE(url, picture) AS links;
-- save the result to Elasticsearch
STORE B INTO 'radio/artists' USING org.elasticsearch.hadoop.pig.ESStorage();
</pre>
<p>Or, if you prefer: </p>
<h3>Map/Reduce</h3>
<pre class="prettyprint lang-default-markup">JobConf conf = new JobConf();
// index used for storing data
conf.set("es.resource", "radio/artists");
// use dedicated output format
conf.setOutputFormat(ESOutputFormat.class);
...
JobClient.runJob(conf);
</pre>
<p>Reading is just as easy:</p>
<h3>Hive script</h3>
<pre class="prettyprint lang-default-markup">CREATE EXTERNAL TABLE artists (
    id BIGINT, name STRING, links STRUCT&lt;url:STRING, picture:STRING&gt;)
STORED BY 'org.elasticsearch.hadoop.hive.ESStorageHandler'
TBLPROPERTIES('es.resource' = 'radio/artists/_search?q=me*');

-- stream data from Elasticsearch
SELECT * FROM artists;
</pre>
<p>Note the use of the search query to retrieve only the data that we are interested in, which is all it takes to have a real-time engine available in Hadoop.</p>
<h2>Pure Map/Reduce Model</h2>
<p>The dedicated Elasticsearch <tt>Input</tt>/<tt>OutputFormat</tt>, <tt>Tap</tt>, <tt>Storage</tt> and <tt>SerDe</tt> take care of the heavy lifting so you do not have to fiddle with data conversion or communicating with Elasticsearch. But more importantly, the Map/Reduce model is ported over; by using Elasticsearch distributed architecture and shard allocation, elasticsearch-hadoop operations are executed in parallel.</p>
<p>In other words, no matter the environment, whenever a query is triggered, elasticsearch-hadoop will dynamically determine the number of shards used for the target index and for each one use a dedicated task to query the data in parallel. The results are aggregated (reduced) and returned back to the user transparently.</p>
<p>Moreover, Hadoop has full insight into the data topology used underneath so it can run the tasks with the data, a great performance boost in deployments where Elasticsearch and Hadoop clusters are co-located.</p>
<h2>Read / Write data transparently</h2>
<p>Whatever library is used, elasticsearch-hadoop automatically handles type conversion to and from JSON whether it’s Hadoop <tt>Writable</tt>, Hive <tt>Serde</tt> or Pig’s data type. And while the latest data types are supported (such as <tt>DateTime</tt> in Pig and <tt>Decimal</tt> in Hive), elasticsearch-hadoop is <a href="http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/requirements.html">backwards compatible</a> with older releases.</p>
<p>To minimize memory usage, type conversion happens on-the-fly and for efficiency data is written in bulk and read through streaming. By taking advantage of Elasticsearch capabilities, mappings are created automatically whether the data has or not a schema associated with it.</p>
<h2>Operational ease</h2>
<p>A job’s dependencies can easily increase its size to several megabytes, complicating its deployment; at only <strong>~150KB</strong>, elasticsearch-hadoop can be easily embedded and requires no extra JARs outside Hadoop itself. At runtime, it relies on HTTP/REST for communication so it can be used without any changes to existing networks.</p>
<h2>Road map</h2>
<p>For the upcoming releases we plan on adding support for advanced mapping (such as parent/child and nested documents), HDFS implementation for the new snapshot/restore feature in Elasticsearch 1.0 and push down operations to name but a few.</p>
<p>We are eager for feedback and in this post we only scratched the surface of the features available. Check out the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/">documentation</a> for in-depth details. </p>
<p>You can share your thoughts with us on the <a href="https://groups.google.com/forum/#!forum/elasticsearch">user’s mailing list</a> or on Twitter at <a href="https://twitter.com/elasticsearch">@elasticsearch</a>. You can also ping me directly on Twitter, where I’m <a href="https://twitter.com/costinl">@costinl</a>.</p>
<p>If you have spotted a <del>bug</del>undocumented features, let us know in the <a href="https://github.com/elasticsearch/elasticsearch-hadoop/issues">issue tracker</a> – or better yet, <a href="https://github.com/elasticsearch/elasticsearch-hadoop/fork">fork</a> the project and submit a pull request. We love contributions!</p>
<p>So head over, <a href="http://www.elasticsearch.org/overview/hadoop/">grab</a> the milestone (either as a <a href="http://download.elasticsearch.org/hadoop/elasticsearch-hadoop-latest.zip">ZIP</a> or <a href="http://search.maven.org/#artifactdetails|org.elasticsearch|elasticsearch-hadoop|1.3.0.M1|jar">JAR</a>) and let us know what you think.</p>
			<aside id="post_tags">
							</aside>
		</article>
			</section>

Elasticsearch and Hadoop

<html><head></head><body><p>Welcome to&#xA0;<i>This Week in Elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the Github repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.</p><h2>Elasticsearch core</h2><ul>
<li>The plugin script failed to report errors correctly when installing from local source files &#xA0;(<a title="3774" href="https://github.com/elasticsearch/elasticsearch/pull/3774">#3744</a>, 0.90 &amp; master).</li>
<li>A mapping update event was not sent to master when an indexing operation changed the mapping but later failed to complete (<a href="https://github.com/elasticsearch/elasticsearch/issues/3782">3782</a>, 0.90 &amp; master).</li>
<li>The new windows service.bat script failed to pass memory settings correctly (<a href="https://github.com/elasticsearch/elasticsearch/issues/3785">#3785</a>, 0.90 &amp; master).</li>
<li>Similar to <a href="https://github.com/elasticsearch/elasticsearch/issues/3778">#3778</a>&#xA0;from last week, the new index UUID is now used in mappings events as well (<a href="https://github.com/elasticsearch/elasticsearch/issues/3783">#3783</a>, 0.90 &amp; master).</li>
<li>The error reporting of the new completion feature was improved (<a href="https://github.com/elasticsearch/elasticsearch/commit/a18f47e62e426885de82584e5536f5f20d252d49">commit</a>,&#xA0;0.90 &amp; master).</li>
<li>The <a href="http://www.elasticsearch.org/guide/reference/index-modules/analysis/pattern-capture-tokenfilter/">pattern_capture</a> token filter now throws an error if you forget to specify the pattern parameter (<a href="https://github.com/elasticsearch/elasticsearch/issues/3808">#3808</a>, 0.90 &amp; master).</li>
<li>The get index templates API returned a 500 status code when called without a template name &#xA0;(<a href="https://github.com/elasticsearch/elasticsearch/issues/3812">#3812</a>, 0.90 &amp; master).</li>
<li>The same fix to the get index template API also makes sure it returns a proper empty JSON object and a 404 return code when the requested template is not found (<a href="https://github.com/elasticsearch/elasticsearch/issues/3817">#3817</a>, 0.90 &amp; master).</li>
<li><a href="http://www.elasticsearch.org/guide/reference/mapping/root-object-type/">Dynamic templates</a>&#xA0; do not require the <em>match</em> or <em>match path</em> options to be specified if they the <em>match type</em> criteria is used&#xA0;(<a href="https://github.com/elasticsearch/elasticsearch/issues/3814">#3814</a>, 0.90 &amp; master).</li>
<li>A lot of work has gone into fixing little things and improving our tests. Here is a selection:&#xA0;<a href="https://github.com/elasticsearch/elasticsearch/commit/4aa315974b6787ed0816f68e0b6ee7ed4101d4c8">commit</a>&#xA0;,&#xA0;&#xA0;<a href="https://github.com/elasticsearch/elasticsearch/commit/9cb55138c77d902a617a7ab5ca9d6b9f8950a5e8">commit</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/4ddfe89bdb9963a425189fefa5bf2855e306510e">commit</a>&#xA0;,&#xA0;<a href="https://github.com/elasticsearch/elasticsearch/commit/9976f6b9aa27b7e75fbb38f8edd4271fe6abedf7">commit</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/23c035da35bffbfd99091f1c349c10cf49906bfc">commit</a>&#xA0; (0.90 &amp; master).</li>
</ul><h2>Elasticsearch Ecosystem</h2><p>Here&apos;s some more information about what is happening in the ecosystem we are maintaining around Elasticsearch, including plugin and driver releases, as well as news about Logstash and Kibana.</p><ul>
<li>This week, we had our first Elasticsearch guest <a href="http://www.elasticsearch.org/blog/using-elasticsearch-and-logstash-to-serve-billions-of-searchable-events-for-customers/">blog post</a>, where&#xA0;<a href="https://twitter.com/ralphm">Ralph Meijer</a>&#xA0;wrote a nice overview of how <a href="http://www.mailgun.com/">Mailgun</a> is using Elasticsearch &amp; Logstash to track and offer insights into billions of events, allowing customers to exactly find out what happened to all the e-mails sent through their service. Thanks again to Ralph and Mailgun for sharing their experience.</li>
<li>As this blog post hit the presses, <a href="https://twitter.com/ZacharyTong">Zachary Tong</a> is holding a webinar entitled <a href="http://www.elasticsearch.org/webinars/elasticsearch-pre-flight-checklist/">Elasticsearch Pre-Flight Checklist</a>. Be quick to join or watch the video later - it will be published shortly after the webinar concludes at the same <a href="http://www.elasticsearch.org/webinars/elasticsearch-pre-flight-checklist/">place</a>.</li>
<li>Thomas Ardal took the time to write a nice blog post with step by step screenshots of how to run <a href="http://thomasardal.com/running-elasticsearch-in-a-cluster-on-azure/">Elasticsearch on Windows Azure</a></li>
<li>Version 0.3.4&#xA0;of the <a href="https://github.com/elasticsearch/cookbook-elasticsearch">Elasticsearch Cookbook</a> was released this week, updated to support the latest 0.90.5 version of Elasticsearch plus numerous <a href="https://github.com/elasticsearch/cookbook-elasticsearch/compare/0.3.3...0.3.4">bug fixes</a>.</li>
<li>Congratulations to the TwitterTV team for winning the <a href="http://elastichack.challengepost.com/">Hopper Elasticsearch Hackathon</a>. You can find out more about the event and the projects produced by all the hackathon teams&#xA0;on <a href="http://hoppertravel.tumblr.com/post/62831226107/congratulations-to-team-twittertv-jony-f-justin">the Hopper Blog</a>.</li>
</ul><h2>Slides</h2><ul>
<li><a href="https://twitter.com/karmiq">Karel Mina&#x159;&#xED;k</a> has put up the slides of his <a href="http://www.slideshare.net/karmi/elasticsearch-rubyshift-2013">Introducing Elasticsearch</a>&#xA0;talk at <a href="http://rubyshift.org/">Rubyshift 2013</a>.</li>
<li><a href="https://twitter.com/spinscale">Alexander Reelsen</a>&#xA0;gave a popular talk at <a href="http://gotocon.com/aarhus-2013/">GOTO Aarhus</a> on&#xA0;<a href="https://speakerdeck.com/spinscale/elasticsearch-beyond-full-text-search">Elasticsearch: Beyond full-text search</a>. (Check out his <a href="https://github.com/elasticsearch/elasticsearch-metrics-reporter-java">Metric&apos;s Elasticsearch reporter</a> as mentioned in his slides). Video will be published soon in the <a href="http://www.youtube.com/user/GotoConferences">GOTO YouTube channel</a>.</li>
<li><a href="https://twitter.com/mvgroningen">Martijn van Gronignen&apos;s</a> slides from the <a href="http://www.meetup.com/elasticsearch-oslo/events/137300022/">Oslo Elasticsearch meetup</a> covering upcoming features in Elasticsearch are available for <a href="https://speakerd.s3.amazonaws.com/presentations/3f4297b0097d0131a2ad4e918f15db03/oslo-meetup1.pdf">download as pdf</a>.</li>
</ul><h2>Where to find Us</h2><h4><strong>Belgium</strong></h4><p><a href="https://twitter.com/lucacavanna">Luca Cavanna</a> will be speaking tonight at the Belux Elasticsearch meetup about the <a href="http://www.meetup.com/ElasticSearch-User-Group-Belux-Belgium-Luxembourg/events/142527742/">new and shiny distributed percolator</a> which will be part of the 1.0 version.</p><h4><span rel="font-size: 1em;">France</span></h4><p>David Pilato will treat attendees of <a href="http://www.openworldforum.org/en/">Open World Foru</a>m to a his presentation&#xA0;<a href="http://2013.openworldforum.org/en/tracks/15#talk_79">Elastifiez Your Application: from SQL to NoSQL in in less than 40 mn</a>, as well as lead an&#xA0;<a href="http://2013.openworldforum.org/en/tracks/31#talk_82">Elasticsearch workshop</a>. Open World Forum takes place October 3rd-5th in Paris.</p><h4>Germany</h4><p>The <a href="http://www.meetup.com/ElasticSearch-UG-Berlin/events/134717942/">Berlin Elasticsearch User Group</a> will convene for their regular monthly meeting on October 29th.</p><h4><span rel="font-size: 1em;">Hungary</span></h4><p><a href="http://twitter.com/honzakral">Honza Kral</a>&#xA0;will talk about Elasticsearch at&#xA0;<a href="http://13.rupy.eu/">RuPy</a>&#xA0;in Budapest on October 12th. RuPy is a conference to bring together programmers and communities of different programming languages like Ruby, Python, Clojure or JavaScript.</p><h4>United Kingdom</h4><p><a href="https://twitter.com/Richardp82">Richard Pijnenburg</a>&#xA0;will give a talk at <a href="http://jaxlondon.com/">JAX London</a> about how to <a href="http://jaxlondon.com/sessions/level-your-logging">Level Up Your Logging</a> on Wednesday, October 30th.</p><h4>United States</h4><ul>
<li>If you&apos;re heading to<a href="http://dataweek.co/">&#xA0;DataWeek 2013</a>, make sure to check out <a href="http://twitter.com/kimchy">Shay Banon</a> on Elasticsearch&apos;s APIs:&#xA0;<a href="http://apiworld2013conferencefestival.sched.org/event/aacb9b4ea8f67ac841baec81b940e3fe?iframe=no&amp;w=760&amp;sidebar=yes&amp;bg=no#.Ui-joGSbh8E">Making It the Easy Way, Doing It the Hard Way</a>. Shay will be speaking on Wednesday, October 2nd; DataWeek 2013 runs from September 28th - October 2nd in San Francisco.</li>
<li><a href="http://twitter.com/drewr">Drew Raines</a> will be at&#xA0;<a href="http://devopsdays.org/events/2013-atlanta/">DevOpsDays Atlanta</a>&#xA0;on October 3-4th.</li>
<li><a href="http://twitter.com/uboness">Uri Boness</a> &#xA0;&amp; <a href="https://twitter.com/ZacharyTong">Zach Tong</a> will be presenting together at the Washington DC meetup on a very interesting combination of subjects-&#xA0;<a href="http://www.meetup.com/Elasticsearch-Washington-DC/events/142011702/">Elasticsearch&apos;&#xAD;s New Aggregations feature <em>and</em> Genomic Sequencing with Elasticsearch</a>. The meetup is scheduled for October 22nd</li>
<li>Ralph Meijer from Mailgun will be talking at the San Francisco meetup scheduled for October 25th on&#xA0;<a href="http://www.meetup.com/Elasticsearch-San-Francisco/events/142398022/">The Business End of the [Mail]gun</a>.</li>
</ul><h2>Where to Find You</h2><p>Are you hosting an Elasticsearch meetup or giving a talk about Elasticsearch? We would love to know so we can feature that information in future editions of&#xA0;<em>This Week in Elasticsearch</em>. Just let our&#xA0;<a href="http://twitter.com/lhawthorn">Community Manager</a>&#xA0;know what you&apos;re up to and we&apos;re happy to help promote your efforts.</p><p>Oh yeah, we&apos;re also&#xA0;<a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you&apos;d like us to find you for employment purposes, just&#xA0;<a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &#xA0;We care more about your skill set and passion for Elasticsearch, Kibana and Logstash than where you rest your head.</p><h2>Training</h2><p>If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:</p><ul>
<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/cologne/2013-10-17">Cologne</a>&#xA0;- October 17, 2013</li>
<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/washington-dc/2013-10-21">Washington, D.C.</a>&#xA0;- October 21, 2013</li>
<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/san-francisco/2013-10-24">San Francisco</a>&#xA0;- October 24, 2013</li>
<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/zurich/2013-10-29">Zurich</a>&#xA0;- October 29, 2013</li>
</ul></body></html>

This Week in Elasticsearch - October 02, 2013

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>This article gives an overview of the Elasticsearch internals. I will present a 10,000 foot view of the different modules that Elasticsearch is composed of and how we can extend or replace built-in functionality using plugins.
</p><section id="what-is-a-module" class="level2">
<h2><a href="#what-is-a-module">What is a Module?</a></h2>
<p>A module in Elasticsearch is a <a href="https://code.google.com/p/google-guice/">Guice</a> Module component that contributes configuration information and binds a specific implementation of the various interfaces of Elasticsearch. A standard Elasticsearch server without any plugins currently consists of over 100 modules:
</p>
<pre class="sourceCode bash prettyprint">$ &lt;span class=&quot;kw&quot;&gt;find&lt;/span&gt; src/main -name &lt;span class=&quot;dt&quot;&gt;\*&lt;/span&gt;Module&lt;span class=&quot;dt&quot;&gt;\*&lt;/span&gt; &lt;span class=&quot;kw&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;kw&quot;&gt;grep&lt;/span&gt; -v common &lt;span class=&quot;kw&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;kw&quot;&gt;wc&lt;/span&gt; -l
     &lt;span class=&quot;kw&quot;&gt;101&lt;/span&gt;
</pre>
<p>During startup, Elasticsearch collects different modules based on its configuration and runtime environment and creates what is called an <code>Injector</code>. To put it simply, an Injector is an object that can construct instances of classes without us providing the parameters required for their construction. The Injector will instead use its configured modules to locate all required dependencies and create them in a topologically sorted order for us. This both saves us a lot of time and assists us in creating a composable, modular system. And a system the size of Elasticsearch needs to be just that in order to be maintainable over time.
</p>
<p>In more technical terms, this is known as <a href="http://en.wikipedia.org/wiki/Dependency_injection">Dependency Injection</a>, which is explained shortly by <a href="http://www.jamesshore.com/Blog/Dependency-Injection-Demystified.html">James Shore</a> as:
</p>
<blockquote>
	Dependency injection means giving an object its instance variables
</blockquote>
<p>In the rest of this post we&#x2019;ll see how this makes Elasticsearch extremely pluggable and extendable. We start by taking a quick look at the HTTP side of Elasticsearch to see how Guice and dependency injection works in action.
</p>
</section><section id="http-server-modules" class="level2">
<h2><a href="#http-server-modules">HTTP Server Modules</a></h2>
<p>For example, Elasticsearch comes with a HTTP server, and to avoid coupling any component that works with HTTP operations in Elasticsearch, it has been abstracted to a <code>HttpServer</code> class, which during construction gets a <code>HttpServerTransport</code> that is by default bound to a concrete implementation called <code>NettyHttpServerTransport</code>. In other words, the <code>HttpServer</code> has no idea how the requests or responses are received and sent, it only has to deal with the application logic (i.e finding an appropriate request handler for incoming requests).
</p>
<p>If, down the line, Elasticsearch should want to switch HTTP layers due to any number of reasons such as performance, security, features etc, it&#x2019;s as simple as swapping out a configuration value (in this case <code>http.type</code>). As a matter of fact, <a href="http://www.sonian.com/">Sonian</a> has released a HTTP layer for Elasticsearch based on <a href="http://www.eclipse.org/jetty/">Jetty</a>, which does exactly this. It&#x2019;s available <a href="https://github.com/sonian/elasticsearch-jetty">here</a>. The Jetty-based implementation supports additional features like authentication, which might be a requirement in some deployments.
</p>
</section><section id="overview-with-namespaces" class="level2">
<h2><a href="#overview-with-namespaces">Overview with Namespaces</a></h2>
<p>Here&#x2019;s a bird&#x2019;s-eye view of modules and their namespaces:
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4c990342dfd73826330b/download?uid=blt786bc4f193251b8e?uid=blt786bc4f193251b8e" data-sys-asset-uid="blta6b35672f304fa74" alt="Modules overview" style="max-width:100%;"><figcaption>Modules overview</figcaption>
</figure>
<p>In this overview, multiple modules are put into different namespaces according to their location in the source code tree or usage in the code. Different aspects of Elasticsearch, such as the REST interface, the plugins, rivers and transport service are separate entities with no module level dependencies between them. Modules that otherwise would have been too large or non-composable are further separated into smaller modules in nested namespaces.
</p>
</section><section id="all-modules" class="level2">
<h2><a href="#all-modules">All Modules</a></h2>
<p>The full overview of all modules is rather large. Click on the thumbnail below to see it in full size.
</p>
<p>Legend: <span style="color: orange">Namespaces</span> &#x2013; Module names &#x2013; <span style="color: green">Bound classes</span>
</p>
<p><a href="/assets/blte08c27067bffe10a/modules-all.png"><img src="https://api.contentstack.io/v2/assets/575e4c993dc542cb38c0848e/download?uid=blt764b0f4d6bad5280?uid=blt764b0f4d6bad5280" data-sys-asset-uid="blt6410aae6b17f9c7f" alt="All modules" style="max-width:100%;"></a>
</p>
</section><section id="spawning-new-modules" class="level2">
<h2><a href="#spawning-new-modules">Spawning New Modules</a></h2>
<p>Most modules only provide bindings for one or more classes or interfaces, but some modules spawn new modules when they&#x2019;re used. These modules are usually conditionally spawned depending on the current <code>Settings</code> object, which is read from the <code>elasticsearch.yml</code> configuration file during start-up. This enables plugin authors to write plugins that replace or extend built-in functionality in Elasticsearch that can be enabled and configured via the default configuration system.
</p>
<p>One concrete example of this, is the <a href="https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/discovery/DiscoveryModule.java">Discovery Module</a>, which by default uses either the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html">Zen</a> or <a href="https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/discovery/DiscoveryModule.java">Local</a> discovery modules, but may have its implementation swapped by setting the <code>discovery.type</code> configuration value to the canonical name of another module. This is exactly how Sonian&#x2019;s <a href="https://github.com/sonian/elasticsearch-zookeeper">ZooKeeper Discovery</a> is used.
</p>
</section><section id="extending-and-swapping-implementations" class="level2">
<h2><a href="#extending-and-swapping-implementations">Extending and Swapping Implementations</a></h2>
<p>While it&#x2019;s a rather large task to completely swap out entire implementations of components in Elasticsearch, these modules provide us with an easy way of adding functionality to Elasticsearch. Since all classes instantiated by the module system in Elasticsearch may request a reference to any other class instanted by the same system, it&#x2019;s almost trivial to extend the functionality via well defined interfaces.
</p>
<p>One of the most important modules for many developers working with Elasticsearch and Lucene is the <code>AnalysisModule</code>. The Analysis module is responsible for providing the analyzers, tokenizers, character filters and token filters used by Elasticsearch during indexing and searching. Using the API provided by the Analysis module, it&#x2019;s trivial to add any Lucene compatible analyzer, tokenizer or filter from a plugin to Elasticsearch with just a few lines of code.
</p>
</section><section id="resources" class="level2">
<h2><a href="#resources">Resources</a></h2>
<p>The overviews were created with <a href="http://freemind.sourceforge.net/wiki/index.php/Main_Page">FreeMind</a>, and the source files can be downloaded <a href="/assets/bltf3c37cafc99dbe5f/modules-all.mm">here</a>.
</p>
</section></body></html>

Elasticsearch Internals: an Overview

<p><em>We're pleased to welcome our first guest blogger, <a href="https://twitter.com/ralphm">Ralph Meijer</a>, Software Developer at Rackspace working on Mailgun. Ralph graciously offered to send us a write up of Mailgun's use of Elasticsearch after we met up at <a href="http://monitorama.eu/">Monitorama EU</a>. He has been an enthusiastic member of the Elasticsearch community for some time now and frequently attends our <a href="http://www.meetup.com/ElasticSearch-NL/">Elasticsearch Netherlands Meetup</a>.</em>
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c75d8edd48f76935964/download?uid=bltbe28c350d98aaf02?uid=bltbe28c350d98aaf02" data-sys-asset-uid="bltb2405b77b1c447cb">
</p><p><img title="Needle in the Haystack" alt="Needle in the Haystack" src="https://api.contentstack.io/v2/assets/57e3ec127363b6f8741c104c/download?uid=blt3ec885c92f0314f1" data-sys-asset-uid="blt3ec885c92f0314f1" style="float:right;padding:0 0 1em 1em" height="204" width="275"><br>
	<a href="http://www.mailgun.com" target="_blank">Mailgun</a> sends and receives a lot of emails and we track and store every event that happens to every email. It adds up to billions of events per month that we need to make available to our customers, along with the ability easily parse through this data with full text search. Below is a detailed account of this challenge and how we solved it with the help of Elasticsearch and Logstash (note: we were delighted to hear about <a href="http://elasticsearch.com/blog/welcome-jordan-logstash/" target="_blank">Logstash joining Elasticsearch</a> shortly after completing this project).
</p><div id="events">
	<h2>Events</h2>
	<p>When thinking of events in Mailgun, consider the following: an incoming message is accepted or rejected, an outgoing message is delivered or rejected (spam? bounce?), the message is opened or a link in the message is clicked on, the recipient wants to unsubscribe. Beside the kind of event, each of these events can have meta data that helps our customers figure out what happened to their messages, why, and <em>when</em> it happened. This meta data includes message sender, recipient address, the message-id, SMTP error codes, link URLs, geographic information, etc.
	</p>
	<p>Every event basically comes down to a timestamp and a number of fields with their values. A typical representation of an event is an associative array, also known as dictionary or hash table.
	</p>
</div><div id="giving-access-to-events">
	<h2>Giving Access To Events</h2>
	<p>Assuming we have all these events, we need a way for our customer to use them in some way. In the Mailgun control panel, there is a logs tab, that shows log events in reverse chronological order and allows one to filter the view by domain and severity. An example log line is this:
	</p>
	<p style="text-align: center;"><img class="alignnone" title="Mailgun Logs UI" alt="Mailgun Logs UI" src="https://api.contentstack.io/v2/assets/57e3ec220ca44bba50267cf0/download?uid=bltbfbb29bc9220581c" data-sys-asset-uid="bltbfbb29bc9220581c" style="max-width:100%;">
	</p>
	<p>In this case, the event has severity 'warn' as the SMTP code signifies a temporary problem and we will reattempt delivery. There are two fields here, a timestamp and a pre-formatted, unstructured text message. In addition, we color code the event based on severity.
	</p>
	<p>Besides this web interface, we also have an API for retrieving logs programmatically, and way to set up webhooks for getting notifications when events happen. The latter notifications are already structured documents with many meta-data fields formatted as JSON. For example, the SMTP code is in its own field, as are the sender and recipient address and the message subject.
	</p>
	<p>Unfortunately, the old logs API was limited. It only returned events for message delivery (no tracking) and the contents were a pre-formatted string as shown in the control panel. There was no way to get or search in individual fields (like those in the webhook notifications) or do any kind of full text search. Similarly, the control panel lacked full text search.
	</p>
</div><div id="elasticsearch-for-storage-and-querying">
	<h2>Elasticsearch For Storage And Querying</h2>
	<p>To provide an API and access via the control panel, we wanted a new backend to improve on these shortfalls, along with several additional requirements:
	</p>
	<ul>
		<li>Allow filtering on most properties.</li>
		<li>Allow full text search.</li>
		<li>Capable of storing events for at least 30 days, but with limited retention.</li>
		<li>Easily extendable by adding nodes.</li>
		<li>Resilient to failing nodes.</li>
	</ul>
	<p>Enter Elasticsearch, a search engine that can index new documents in near real-time and make them immediately available for querying. Elasticsearch is based on Apache Lucene and allows for setting up clusters of nodes that store any number of indices in a distributed, fault-tolerant way. If a node disappears, the cluster will rebalance the (shards of) indices over the remaining nodes. You can configure how many shards make up each index and how many replicas of these shards there should be. If a master shard goes offline, one of the replicas is promoted to master and used to repopulate another node.
	</p>
	<p>Elasticsearch is document-oriented and optionally schema-free. This means that you can throw any JSON document at it and have it indexed as individual fields. This is a perfect match for our events.
	</p>
	<p>Elasticsearch also has a powerful querying/filtering interface for search in particular fields or do full text searches.
	</p>
</div><div id="getting-events-into-elasticsearch">
	<h2>Getting Events Into Elasticsearch</h2>
	<p>There are quite a bit of projects and services out there focussed on logging events. We ultimately picked <a href="/logstash">Logstash</a>, a tool for collecting, parsing, mangling and passing on logs.
	</p>
	<p>Internally, the events pushed out via our webhooks are also used in other parts of our system. We currently use Redis for this. Logstash has a Redis input plugin that retrieves log events from a Redis list. After some minor filtering, the events can then be sent out via an output plugin. A very commonly used output plugin is the Elasticsearch plugin.
	</p>
	<p>A great way to use Elasticsearch's very rich API is by setting up Kibana, a tool to "make sense of a mountain of logs". The new incarnation, <a href="http://three.kibana.org/" target="_blank">Kibana 3</a>, is fully client side JavaScript, and will be the default interface for Logstash. Unlike previous versions, it no longer depends on a Logstash-like schema, but is now usable for any Elasticsearch index.
	</p>
	<p><img title="Mailgun Kibana Dashboard" alt="Mailgun Kibana Dashboard" src="https://api.contentstack.io/v2/assets/57e3ec380ca44bba50267d14/download?uid=blt8e9a2c56f726cf68" data-sys-asset-uid="blt8e9a2c56f726cf68" style="max-width:100%;">
	</p>
</div><div id="authorization">
	<h2>Authorization</h2>
	<p>With this setup we have a full solution for putting our events in a central place and a rich API to dive into these logs. However, we don't want to expose all the events to just everybody, so we need a way to add authorization to the equation. Currently Elasticsearch nor Kibana offer authentication and authorization, so directly exposing the Elasticsearch API is not an option.
	</p>
	<p>We opted for building a two-staged proxy setup. One proxy for authorization and rate-limiting and one for translating requests to our new Events API to Elasticsearch requests. We have made the former proxy available under the Apache 2.0 license as <a href="https://github.com/mailgun/vulcan" target="_blank">vulcan</a> on Github. We also ported our old, now deprecated, Logs API to use Elasticsearch instead of the old system.
	</p>
</div><div id="index-design">
	<h2>Index Design</h2>
	<p>There are various ways to select how you organize your indexes, based on the amount of documents (per time period) and querying patterns.
	</p>
	<p>Logstash, by default, creates a new index per day, which then contains all events received from its inputs. You can configure it to change the rotation period (if any) or make separate indices per user or other property (like the type of event).
	</p>
	<p>In our case we have about 1500 events per second and we wanted retention periods configurable per account. The options:
	</p>
	<ul>
		<li>One big index.</li>
		<li>One index per day.</li>
		<li>One index per user account.</li>
	</ul>
	<p>Of course each these could be split up further by sharding on, for example, the event type, if needed.
	</p>
	<p>One way to manage retention in Elasticsearch is by setting <a href="/guide/reference/mapping/ttl-field/" target="_blank">TTLs</a> on each document. Elasticsearch will then periodically delete expired documents in bulk. This makes it easy to do custom retention per account. However, it turns out to be a rather expensive (repetitive) operation.
	</p>
	<p>A much lighter weight operation is dropping entire indices. This is why Logstash defaults to daily indices. When an index falls outside your retention period, you can simply drop the index from a cron job.
	</p>
	<p>However, this doesn't give us custom retentions. Looking at our numbers, it was more efficient for us to keep all entries and limit queries to older indices from the API layer. We have many user accounts, and keeping an index per day <em>and</em> per user would become impractical. Of course, keeping all logs for all accounts in daily indices means that we are storing all of that on disk. However, if an account has a two day retention, it does limit the amount of data kept in caches and the processing power for queries that span multiple days. On top of that, for our own internal uses, it is very helpful for debugging to still have older logs and access them through Kibana.
	</p>
</div><div id="mappings">
	<h2>Mappings</h2>
	<p>To define how a (fields in a) document should be massaged, indexed and stored in an index, Elasticsearch keeps a so-called <a href="/guide/reference/mapping/" target="_blank">mapping</a>. For each field for a it defines the type, how values are analyzed and tokenized for indexing and querying, whether the value needs to be stored and some other settings. By default, the mapping is dynamic, i.e. Elasticsearch will try to determine the type of the field from the first value it gets and applies settings to it.
	</p>
	<p>If you have homogeneous data, this works quite well. However, if you have data coming from different sources, or different types of logs, in our case, it might be that the value of a field with the same name represents something else and even has a different type. As Elasticsearch will reject an attempt to index a document with a type mismatch, we need to define a mapping.
	</p>
	<p>I defined a single mapping for the types of log events exposed via our <a href="http://documentation.mailgun.com/api-events.html" target="_blank">Events API</a>. Not all types events have all fields, but a field with a particular name always means the same.
	</p>
</div><div id="analysis">
	<h2>Analysis</h2>
	<p>By default, fields are set up with the <a href="/guide/reference/index-modules/analysis/standard-analyzer/" target="_blank">standard analyzer</a> as part of the mapping. In general, this means that a string value is put in lower case and split up in words. These <em>tokens</em> are then put in the index to point to this document for this field.
	</p>
	<p>In some cases, you want something different because it has undesirable effects. For example identifiers like account IDs, e-mail addresses or URLs, the default tokenizer will split on dashes and doesn't consider the domain name as a single token (besides its constituent domain labels). If you then facet on the domain name field, you get (parts of) the individual domain labels as a facet.
	</p>
	<p>To resolve this, can set the <span class="highlighted">index</span> attribute for such fields to <span class="highlighted">not_analyzed</span>. This causes the value to be indexed as is, without any case mapping or tokenization. When applying this to the <span class="highlighted">domain.name</span> field, for example, each domain will be a facet.
	</p>
	<p>If you still want to be able to have the document to be found by parts in such fields, you can use the so-called <a href="/guide/en/elasticsearch/reference/current/mapping-core-types.html#_multi_fields_3" target="_blank">multi-field type</a>. It allows you to map the same value to different core types an/or different mapping attributes and make them available under different names. We used this technique for IP addresses, where the <em>default</em> field (e.g. <span class="highlighted">sending-ip</span>) has the type <span class="highlighted">ip</span>, and a <em>non-default</em> field (e.g. <span class="highlighted">sending-ip.untouched</span>) is configured to be <span class="highlighted">not_analyzed</span> and of type <span class="highlighted">string</span>. You can then use the default field for doing range queries on IP addresses, and the <span class="highlighted">.untouched</span> one for faceting.
	</p>
	<p>For most fields, though, we haven't done this (yet). We are looking into applying the <a href="/guide/en/elasticsearch/reference/current/analysis-pattern-capture-tokenfilter.html" target="_blank">pattern capture tokenfilter</a> to some of these fields (like e-mail addresses) in the near future, in combination with the <span class="highlighted">multi-field</span> type trick.
	</p>
</div><div id="monitoring">
	<h2>Monitoring</h2>
	<p>To see how your cluster is holding up, you really want to set up monitoring for it. Elasticsearch has a great API for retrieving the <a href="/guide/en/elasticsearch/reference/current/cluster-state.html" target="_blank">cluster state</a> and <a href="/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html" target="_blank">node statistics</a>. We have set up <a href="http://graphite.wikidot.com/" target="_blank">Graphite</a> for storing these metrics and made comprehensive dashboards out of them, below is one of the panels.
	</p>
	<p style="text-align: center;"><img class="alignnone no-shadow no-border" title="Mailgun Graphite Monitoring Dashboard" alt="Mailgun Graphite Monitoring Dashboard" src="https://api.contentstack.io/v2/assets/57e3ec4dccfdfe6162a655df/download?uid=blt885060276ad65188" data-sys-asset-uid="blt885060276ad65188" style="max-width:100%;">
	</p>
	<p>To poll these metrics and pass them to Graphite I created <a href="http://github.com/mochi/vor" target="_blank">Vör</a>, which was made available under the MIT/X11 License by <a href="http://www.mochimedia.com/" target="_blank">Mochi Media</a>. An additional poller for keeping track of the size of Redis lists is also in the works.
	</p>
	<p>On top of that, we count things like messages received, clicks, opens, API requests and time such requests via <a href="https://github.com/etsy/statsd" target="_blank">StatsD</a> and have added these metrics to our Graphite dashboards, too.
	</p>
	<p style="text-align: center;"><img class="alignnone no-shadow no-border" title="Mailgun Graphite Event Dashboard" alt="Mailgun Graphite Event Dashboard" src="https://api.contentstack.io/v2/assets/57e3ec60189bd8fd612df82d/download?uid=bltb4adabc6b18fdd64" data-sys-asset-uid="bltb4adabc6b18fdd64" style="max-width:100%;">
	</p>
	<p>This is a great way to see what is happening. Graphite has a large number of functions you can apply to (series of) metrics before graphing or returning the result as a JSON document. For example, it is really easy to create a graph that shows the number of API requests against server load or indexing speed.
	</p>
</div><div id="current-setup">
	<h2>Current Setup</h2>
	<p>Some statistics on our set up:
	</p>
	<ul>
		<li>Between 40 to 60 million log events per day.</li>
		<li>30 days log retention.</li>
		<li>30 indexes</li>
		<li>5 shards per index</li>
		<li>1 replica for each shard</li>
		<li>2x 50 to 80 GB per index (times two because of the replica shards).</li>
	</ul>
	<p>For this we have set currently set up a total of 9 Rackspace cloud servers, in various configurations, based on our findings with the dashboards above:
	</p>
	<ul>
		<li>6x 30GB RAM, 8 vCPUs, 1TB disk: Elasticsearch data nodes.</li>
		<li>2x 8GB RAM, 4 vCPUs: Elasticsearch proxy node, Logstash, Graphite and StatsD.</li>
		<li>2x 4GB RAM, 2 core: Elasticsearch proxy node, Vulcan and API server</li>
	</ul>
	<p>Most of these will eventually move to dedicated boxes, while retaining the ability to create additional cloud servers to scale up.
	</p>
	<p>The Elasticsearch data nodes are configured to take and lock 16GB of RAM for the JVM heap. The rest of the configuration is pretty standard. We set the maximum fieldcache size to 40% of the heap to make sure the cluster does not break down on expensive and ill-advised faceting and sorting on fields with many different values. We also increased a few <a href="/guide/en/elasticsearch/reference/current/cluster-update-settings.html" target="_blank">cluster wide settings</a> for speeding up recovery and rebalancing. Especially the <span class="highlighted">indices.recovery.max_bytes_per_sec</span> is set very low by default, for the amount of documents we store.
	</p>
</div><div id="conclusion">
	<h2>Conclusion</h2>
	<p>We are very happy with Elasticsearch for keeping our log events and have been getting very positive feedback from our users about the new Events API and the improvements in the Logs tab of the Control panel. Digging through our logs by searching in almost any field is a significant improvement for them and ElasticSearch makes this fast and painless. Also, this tool chain of Logstash, Elasticsearch and Kibana is also ideal for internal application logging.
	</p>
	<p>If you want to know more about how we use Elasticsearch or questions about the Events API, be sure to drop us a line!  You can <a href="http://blog.mailgun.com/post/new-events-api-detailed-email-tracking-and-search/" target="_blank">read more details about the Events API itself</a> on the Mailgun blog.
	</p>
	<p>Happy logging, er, emailing!
	</p>
</div><p><em>Many thanks to Ralph for sharing his experiences with the wider community. If you're interested in being a guest blogger or have other Elasticsearch related write ups you'd like us to help you reshare, <a href="https://twitter.com/lhawthorn">let me know</a>.</em>
</p>

Using ElasticSearch and Logstash to Serve Billions of Searchable Events for Customers

<html><head></head><body><p>
	Welcome to&#xA0;
	<i>This week in elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in elasticsearch. We cover what happened in the Github repositories, as well as many elasticsearch events happening worldwide, and give you a small peek into the future of the project.
</p><h3>Elasticsearch core</h3><ul>
	<li>Do not return a 404 on template deletion using a match all regex, if no templates exist (<a href="https://github.com/elasticsearch/elasticsearch/issues/3723">#3723</a>, 0.90 &amp; master)</li>
	<li>Windows service startup can now cope with spaces in directory names (<a href="https://github.com/elasticsearch/elasticsearch/issues/3725">#3725</a>, 0.90 &amp; master)</li>
	<li>Windows service now also warns in case a JRE instead of a JDK is installed (<a href="https://github.com/elasticsearch/elasticsearch/issues/3739">#3739</a>, 0.90 &amp; master)</li>
	<li>Empty but existing filters in a query could cause a <code>NullPointerException</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/3724">#3724</a>, 0.90 &amp; master)</li>
	<li>Discovery to support a timeout waiting for other nodes to process a new cluster state (<a href="https://github.com/elasticsearch/elasticsearch/issues/3736">#3736</a>, 0.90 &amp; master)</li>
	<li>Rest Nodes Stats can now filter for completion fields as well (<a href="https://github.com/elasticsearch/elasticsearch/issues/3746">#3746</a>, 0.90 &amp; master)</li>
	<li>Pretty printing in a response now adds a newline add the end (<a href="https://github.com/elasticsearch/elasticsearch/issues/3748">#3748</a>, 0.90 &amp; master)</li>
	<li>The <code>_boost</code> field can now be indexed and stored (<a href="https://github.com/elasticsearch/elasticsearch/issues/3752">#3752</a>, 0.90 &amp; master)</li>
	<li>An internal origin has been added to a couple of internal actions like refresh, acquiring a searcher or delete-by-query (commits <a href="https://github.com/elasticsearch/elasticsearch/commit/167538ef0d8003e083af4f85da139c7020afcc57">1</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/78af818d72a1c08663e15578a425f2534911b036">2</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/ab6715b2923d6fcd141add1b04b5227622f751c1">3</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/e7e39936b854335b5a875bbecc3cbbdc28048bb3">4</a>, <a href="https://github.com/elasticsearch/elasticsearch/commit/5b1f263569a6d10fdec4adf4ad4fe7218309a9e2">5</a>, 0.90 &amp; master)</li>
	<li>A new shard state called <code>POST_RECOVERY</code> has been added, needed to solve potentation timing aspects when a shard is about to be marked as <code>STARTED</code> (<a href="https://github.com/elasticsearch/elasticsearch/commit/397f442c6dabf5cd2105ba01d6864fbbb22d11e2">commit</a>, 0.90 &amp; master)</li>
	<li>Update Operation might hang (rarely) when retrying on invalid shard state (<a href="https://github.com/elasticsearch/elasticsearch/issues/3769">#3769</a>, 0.90 &amp; master)</li>
	<li>If an index gets deleted and created rapidly, a new id-based mechanism has been ensured that events from the old and new index are not mixed up (<a href="https://github.com/elasticsearch/elasticsearch/issues/3778">#3778</a>, 0.90 &amp; master)</li>
</ul><h3>Elasticsearch ecosystem</h3><p>
	Here&apos;s some more information about what is happening in the ecosystem we are maintaining around elasticsearch, including plugin and driver releases, as well as news about logstash and kibana.
</p><ul>
	<li>As you might have seen the announcement yesterday, we have a released a set of <a href="http://www.elasticsearch.org/blog/unleash-the-clients-ruby-python-php-perl/">brand new elasticsearch clients</a> for different languages, namely perl, python, ruby and PHP. We are happy to get feedback about this!</li>
	<li>As we moved kibana news into an own blog post, you should check the latest <a href="http://www.elasticsearch.org/blog/this-week-in-kibana/">this week in kibana</a></li>
	<li><a href="http://twitter.com/fhopf">Florian Hopf</a> blogged about using <a href="http://blog.florian-hopf.de/2013/09/kibana-and-elasticsearch-see-what.html">Kibana and Elasticsearch for tweet analysis</a> during a conference</li>
	<li><a href="https://github.com/pagesjaunes/RestES">RestES</a> is a small configurable layer between your webapplication and elasticsearch, implemented as a nodejs application</li>
	<li>A long blog post about running <a href="http://www.found.no/foundation/elasticsearch-in-production/">running elasticsearch in production</a> by Alex Brasetvik</li>
	<li>The scala client <a href="https://github.com/sksamuel/elastic4s">Elastic4s</a> has been updated to 0.90.5</li>
	<li>Daniel Romero has invested the time to create <a href="https://github.com/infoslack/slackbuilds/blob/master/elasticsearch/elasticsearch.SlackBuild">Slackware packages</a> for elasticsearch</li>
</ul><h3>Slides</h3><ul>
	<li><a href="http://twitter.com/reyjrar">Brad Lothsky</a> from booking.com presented about using <a href="https://speakerdeck.com/reyjrar/elasticsearch-for-logging">elasticsearch for logging</a></li>
	<li><a href="http://twitter.com/honzakral">Honza Kral</a> managed to create a very concise 20 minute video about <a href="http://www.youtube.com/watch?v=UNsDUNWRWzg">managing your data with elasticsearch</a></li>
</ul><h3>Where to find Us</h3><h4>Denmark</h4><p>
	<a href="https://twitter.com/spinscale">Alexander Reelsen</a> will be attending to <a href="http://gotocon.com/aarhus-2013/">GOTO Aarhus</a> with a few other folks from the Elasticsearch team on September 30th - October 2nd. You can pop by our booth to meet us, or just head to Alex&apos;s talk <a href="http://gotocon.com/aarhus-2013/speaker/Alex+Reelsen">Elasticsearch - Beyond Full Text Search</a>.
</p><h4>France</h4><p>
	David will also give attendees of Open World Forum a taste of 
	<a href="http://www.openworldforum.org/en/tracks/15#talk_79">Elastifiez Your Application: from SQL to NoSQL in in less than 40 mn</a>, as well as lead an <a href="http://www.openworldforum.org/en/tracks/31#talk_82">Elasticsearch workshop</a>. Open World Forum takes place October 3rd-5th in Paris.
</p><h4>Norway</h4><p>
	The fine folks spearheading the 
	<a href="http://www.meetup.com/elasticsearch-oslo/events/137300022/">Oslo Elasticsearch Users Group</a> will host their inaugural meetup on Thursday, September 26th. <a href="https://twitter.com/mvgroningen">Martijn van Groningen</a> will join the festivities to talk about new features in the works for Elasticsearch.
</p><h4>Hungary</h4><p>
	<a href="http://twitter.com/honzakral">Honza Kral</a> will talk about elasticsearch at <a href="http://13.rupy.eu/">RuPy</a> in Budapest on 12th of October. RuPy is a conference to bring together programmers and communities of different programming languages like Ruby, Python, Clojure or JavaScript.
</p><h4>Ukraine</h4><p>
	<a href="http://rubyshift.org/">RubyShift 2013</a> kicks off on Friday, September 27th in Kiev. Karel Minarik will be there, giving a presentation on <a href="http://rubyshift.org/#agenda">Elasticsearch: Search and Analytics</a> the first day of the conference.
</p><h4>United States</h4><ul>
	<li>If you&apos;re heading to<a href="http://dataweek.co/"> DataWeek 2013</a>, make sure to check out Shay Banon on Elasticsearch&apos;s APIs: <a href="http://apiworld2013conferencefestival.sched.org/event/aacb9b4ea8f67ac841baec81b940e3fe?iframe=no&amp;w=760&amp;sidebar=yes&amp;bg=no#.Ui-joGSbh8E">Making It the Easy Way, Doing It the Hard Way</a>. Shay will be speaking on Wednesday, October 2nd; DataWeek 2013 runs from September 28th - October 2nd in San Francisco.</li>
	<li>Drew Raines will be at <a href="http://devopsdays.org/events/2013-atlanta/">DevOpsDays Atlanta</a> on October 3-4th.</li>
</ul><h3>Where to Find You</h3><p>
	Are you hosting an Elasticsearch meetup or giving a talk about Elasticsearch? We would love to know so we can feature that information in future editions of 
	<em>This Week in Elasticsearch</em>. Just let our <a href="http://twitter.com/lhawthorn">Community Manager</a> know what you&apos;re up to and we&apos;re happy to help promote your efforts.
</p><p>
	Oh yeah, we&apos;re also 
	<a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you&apos;d like us to find you for employment purposes, just <a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &#xA0;We care more about your skill set and passion for elasticearch, kibana and logstash than where you rest your head.
</p><h3>Trainings</h3><p>
	If you are interested in elasticsearch training we have courses taught by our core developers coming up in:
</p><ul>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/cologne/2013-10-17">Cologne</a> - October 17, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/washington-dc/2013-10-21">Washington, D.C.</a> - October 21, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/san-francisco/2013-10-24">San Francisco</a> - October 24, 2013</li>
	<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/zurich/2013-10-29">Zurich</a> - October 29, 2013</li>
</ul></body></html>

This Week in Elasticsearch - September 26, 2013

<html><head></head><body><p>Today, the Elasticsearch developer team is excited to announce the release of official Elasticsearch clients for Ruby, Python, PHP and Perl, with other languages to follow in the future. All of the clients are licensed under the Apache 2 open source license.</p>
<h2>Why are we providing official clients?</h2>
<p>The Elasticsearch community has done a great job over the years of providing clients for many languages. The feature set of these clients varies greatly: some are low level thin wrappers, some are high level with many abstractions. Some focus on the Elasticsearch API, while others focus on the networking layer. Some have extensive coverage of the Elasticsearch APIs and some just implement the few APIs that the developer needs.</p>
<p>And they are all implemented in the preferred style of the developer.</p>
<p>A number of our users have multiple languages in their codebase and want to use Elasticsearch from all of them. Differences between clients can make this more complicated than it need be.</p>
<h2>What do these clients provide?</h2>
<p>We have written these clients with the following goals in mind:</p>
<h3>No opinions!</h3>
<p>It should be as simple as possible to go from the <a href="/guide/en/elasticsearch/reference/current/index.html">Elasticsearch reference docs</a> directly to the client of your choice. All of the Elasticsearch APIs provided by these clients are direct translations of the native Elasticsearch REST interface.  There should be no guessing required.</p>
<h3>Play nicely with the cluster</h3>
<p>While it is easy to make HTTP calls to Elasticsearch, handling dynamic node detection or failover when nodes disappear is tricky.  These clients provide a solid networking base for playing nicely with the cluster.</p>
<h3>Full, consistent coverage of the APIs</h3>
<p>We have implemented the full REST interface.  Nothing is missing. And the method calls and parameters are consistent between languages: you can make the same calls from any of the supported clients.  That said, each client still feels like it belongs to the language it is written in.  It won&#x2019;t feel like you are programming Java in Ruby, or C++ in PHP.  </p>
<h3>Transport abstraction</h3>
<p>These clients are not tied to a particular networking module.  Different HTTP backends can be plugged in for different environments, or to match the HTTP client that you are already using. But this abstraction also allows us to plugin different transport protocols in the future, which are more efficient than HTTP.</p>
<h3>Extend and conquer</h3>
<p>The API that these clients provide is a thin wrapper around the REST interface.  Perhaps their style is different from your own, and you&#x2019;d prefer more abstractions.  We hear you, so we have built these clients to be extensible.  We have tried to do the hard stuff for you, to provide you with a solid foundation for building your own interface.This is the beauty of open source: our users will have great ideas that haven&#x2019;t even occurred to us.  Now you also have the tools to implement them, without having to start from scratch.</p>
<h3>Supported by Elasticsearch</h3>
<p>Because we wrote these clients and they are tested by us, we can provide official support for them.  When bugs are found, we can fix them quickly &#x2013; it&#x2019;s our job. </p>
<h2>Downloading the clients</h2>
<p>Each client is available from GitHub and also from the standard distribution network for each language, where one exists.</p>
<ul>
<li><a href="#perl">Elasticsearch for Perl</a></li>
<li><a href="#php">Elasticsearch for PHP</a></li>
<li><a href="#python">Elasticsearch for Python</a></li>
<li><a href="#ruby">Elasticsearch for Ruby</a></li>
</ul>
<p><b>Note:</b> This is the first release of these clients. Please download them, try them out and give us feedback.  What doesn&#x2019;t work? What is missing? What can we do better?</p>
<h2 id="perl">Elasticsearch for Perl</h2>
<p><b>GitHub repo:</b> <a href="https://github.com/elasticsearch/elasticsearch-perl">https://github.com/elasticsearch/elasticsearch-perl</a><br><b>Issues list:</b> <a href="https://github.com/elasticsearch/elasticsearch-perl/issues">https://github.com/elasticsearch/elasticsearch-perl/issues</a></p>
<h3>Installing</h3>
<pre class="prettyprint noescape lang=shell">cpanm Elasticsearch
</pre>
<h3>Example usage</h3>
<pre class="prettyprint noescape lang=perl">use Elasticsearch;

# Connect to localhost:9200
my $es = Elasticsearch-&gt;new();

# Round-robin between two nodes
my $es = Elasticsearch-&gt;new(
    nodes =&gt; [
        &apos;search1:9200&apos;,
        &apos;search2:9200&apos;
    ]
);

# Connect to cluster at search1:9200, sniff all nodes 
# and round-robin between them
my $es = Elasticsearch-&gt;new(
    nodes    =&gt; &apos;search1:9200&apos;,
    cxn_pool =&gt; &apos;Sniff&apos;
);

# Index a document
$es-&gt;index(
    index   =&gt; &apos;my_app&apos;,
    type    =&gt; &apos;blog_post&apos;,
    id      =&gt; 1,
    body    =&gt; {
        title   =&gt; &apos;Elasticsearch clients&apos;,
        content =&gt; &apos;Interesting content...&apos;,
        date    =&gt; &apos;2013-09-24&apos;
    }
);

# Get the document
my $doc = $es-&gt;get(
    index   =&gt; &apos;my_app&apos;,
    type    =&gt; &apos;blog_post&apos;,
    id      =&gt; 1
);

# Search
my $results = $es-&gt;search(
    index =&gt; &apos;my_app&apos;,
    body  =&gt; {
        query =&gt; {
            match =&gt; { title =&gt; &apos;elasticsearch&apos; }
        }
    }
);
</pre>
<h3>From the developer</h3>
<p>My name is Clinton Gormley and I&#x2019;ve been using Perl since 1999 and Elasticsearch from the first release back in early 2010.  At the time I was working for the largest obituary and family announcement website in Europe and we desperately needed a solution for full text search. </p>
<p>I kept putting it off because the options that existed back then made me want to cry. But eventually, I couldn&#x2019;t put it off any longer.  Before starting, I searched Google again, just in case a different solution had become available, and discovered an interview with Shay Banon discussing version 0.4 of Elasticsearch. I clicked through to the website and it looked too good to be true: simple, scalable, easy to start using.  I tried it out and found that it lived up to its promises!</p>
<p>The first thing that I needed was a Perl API so I wrote <a href="https://metacpan.org/module/ElasticSearch">ElasticSearch.pm</a> and have been maintaining it ever since.  ElasticSearch.pm is used by <a href="http://metacpan.org">metacpan.org</a> amongst others.</p>
<p>The new <a href="https://metacpan.org/module/Elasticsearch">Elasticsearch</a> client is a complete rewrite of the original codebase with a number of improvements, especially in extensibility, logging and handling node failover, that would have been difficult to add in the old version. I apologise in advance for the change in naming: ElasticSearch to Elasticsearch&#x2026;</p>
<p>Currently there is only support for using <a href="https://metacpan.org/module/HTTP::Tiny">HTTP::Tiny</a> as a backend, but I will soon be releasing backends for HTTP::Lite, LWP, libcurl and AnyEvent too.  Also, I will release an ElasticSearch::Compat module which will make it easy to migrate your existing code to the new Elasticsearch client.</p>
<p>In the meantime, please try out the new client and let me know of anything that is missing via the GitHub issues list.</p>
<h2 id="php">Elasticsearch for PHP</h2>
<p><b>GitHub repo:</b> <a href="https://github.com/elasticsearch/elasticsearch-php">https://github.com/elasticsearch/elasticsearch-php</a><br><b>Issues list:</b> <a href="https://github.com/elasticsearch/elasticsearch-php/issues">https://github.com/elasticsearch/elasticsearch-php/issues</a></p>
<h3>Installing</h3>
<p>Add `elasticsearch/elasticsearch` to your `composer.json`:</p>
<pre class="prettyprint noescape lang=json">{
    &quot;require&quot;: {
        &quot;elasticsearch/elasticsearch&quot;: &quot;~0.4&quot;
    }
}
</pre>
<p>Then install with composer</p>
<pre class="prettyprint noescape lang=shell">curl -s http://getcomposer.org/installer | php
php composer.phar install
</pre>
<h3>Example usage</h3>
<pre class="prettyprint noescape lang=php">require &apos;vendor/autoload.php&apos;;
use Elasticsearch;

// Connect to localhost:9200:
$es = new Elasticsearch\Client();

// Round-robin between two nodes:
$es = new Elasticsearch\Client(
    array(
        &apos;hosts&apos; =&gt; array(
            &apos;search1:9200&apos;,
            &apos;search2:9200&apos;
        )
    )
);

//Connect to cluster at search1:9200, sniff all nodes and round-robin between them:
$es = new Elasticsearch\Client(
    array(
        &apos;hosts&apos; =&gt; array(&apos;search1:9200&apos;),
        &apos;connectionPoolClass&apos; =&gt; &apos;\Elasticsearch\ConnectionPool\SniffingConnectionPool&apos;
    )
);


// Index a document:
$es-&gt;index(
    array(
        &apos;index&apos; =&gt; &apos;my_app&apos;,
        &apos;type&apos;  =&gt; &apos;blog_post&apos;,
        &apos;id&apos;    =&gt; 1,
        &apos;body&apos;  =&gt; array(
            &apos;title&apos;   =&gt; &apos;Elasticsearch clients&apos;,
            &apos;content&apos; =&gt; &apos;Interesting content...&apos;,
            &apos;date&apos;    =&gt; &apos;2013-09-24&apos;
        )
    )
);

//Get the document:
$doc = $es-&gt;get(
    array(
        &apos;index&apos; =&gt; &apos;my_app&apos;,
        &apos;type&apos;  =&gt; &apos;blog_post&apos;,
        &apos;id&apos;    =&gt; 1
    )
);

// Search:
$params = array(
    &apos;index&apos; =&gt; &apos;my_app&apos;,
    &apos;type&apos;  =&gt; &apos;blog_post&apos;
);
$params[&apos;body&apos;][&apos;query&apos;][&apos;match&apos;][&apos;title&apos;] = &apos;elasticsearch&apos;;
$results = $es-&gt;search($params);
</pre>
<h3>From the developer</h3>
<p>My name is Zachary Tong and I&#x2019;ve been working in PHP off and on for about 12 years.  I had the pleasure of meeting Elasticsearch about two years ago around release 0.18.6.1.  We&#x2019;ve been in a steady relationship ever since.</p>
<p>At the time I was working as a molecular biologist and was desperately dissatisfied with available biomedical search engines.  Always being an evening-hacker at heart, I obtained a lease to the NLM Medline database of biomedical abstracts and began trying to develop a better search solution.  Wouldn&#x2019;t you know it&#x2026;search is a Hard Problem!</p>
<p>I stumbled onto Elasticsearch at that point and was excited by the combination of features + performance (even on my modest laptop).  Although that particular search project never left my laptop, I&#x2019;ve continued using Elasticsearch for a variety of personal and professional projects since.</p>
<p>Coming from a non-information-retrieval background (and largely non-computer-science background), many of the concepts in Elasticsearch were foreign to me.  Shards, indexes, TF-IDF, analyzers, ngrams, what?  It can be overwhelming for newcomers to the search field.  I find that I learn best by teaching, so over the last two years I&#x2019;ve written a number of articles on my personal blog.</p>
<p>Although I&#x2019;ve been coding and writing about Elasticsearch for a long time, I have never really participated in the open source community until recently.  My first big forray into OSS was with a PHP client for Elasticsearch.  This caught the attention of developers at Elasticsearch and the rest is history.</p>
<p>I&#x2019;m very excited about the clients being released today.  My hope is to free developers from the drudgery of &#x201C;boring&#x201D; low-level interaction with an Elasticsearch cluster.  Using this client as a base we can start to build more interesting components, like bundles for popular frameworks, or advanced ORMs that abstract away the cluster entirely.</p>
<h2 id="python">Elasticsearch for Python</h2>
<p><b>GitHub repo:</b> <a href="https://github.com/elasticsearch/elasticsearch-py">https://github.com/elasticsearch/elasticsearch-py</a><br><b>Issues list:</b> <a href="https://github.com/elasticsearch/elasticsearch-py/issues">https://github.com/elasticsearch/elasticsearch-py/issues</a></p>
<h3>Installing</h3>
<pre class="prettyprint noescape lang=shell">pip install elasticsearch
</pre>
<h3>Example usage</h3>
<pre class="prettyprint noescape lang=python">import elasticsearch

# Connect to localhost:9200 by default:
es = elasticsearch.Elasticsearch()

# Round-robin between two nodes:
es = elasticsearch.Elasticsearch([&quot;search1:9200&quot;, &quot;search2:9200&quot;])

# Connect to cluster at search1:9200, sniff all nodes and round-robin between them
es = elasticsearch.Elasticsearch([&quot;search1:9200&quot;], sniff_on_start=True)

# Index a document:
es.index(
    index=&quot;my_app&quot;,
    doc_type=&quot;blog_post&quot;,
    id=1,
    body={
        &quot;title&quot;: &quot;Elasticsearch clients&quot;,
        &quot;content&quot;: &quot;Interesting content...&quot;,
        &quot;date&quot;: date(2013, 9, 24),
    }
)

# Get the document:
es.get(index=&quot;my_app&quot;, doc_type=&quot;blog_post&quot;, id=1)

# Search:
es.search(index=&quot;my_app&quot;, body={&quot;query&quot;: {&quot;match&quot;: {&quot;title&quot;: &quot;elasticsearch&quot;}}})
</pre>
<h3>From the developer</h3>
<p>My name is Honza Kr&#xE1;l, I chose Python as my primary language in 2006 and haven&#x2019;t looked back. Before I joined Elasticsearch in April of 2013 my job was to develop and maintain backends for a few bigger content websites based on the <a href="http://www.djangoproject.com/">Django</a> web framework. Before that I spent my days as an ETL guy working in data warehousing</p>
<p>I learned about Elasticsearch as part of my job when I was looking for a search engine to power our full-text search and was immediately captured by it&#x2019;s balance of ease-of-use and powerful feature set. It reminded me of my days as a data guy and dove right in.</p>
<p>Ever since the beginning of my programming career I was drawn towards open-source and, whenever possible, would contribute back to the project I am using. It was no surprise then that once I learned that there is an opening for a Python guy on the team to work on the new client I couldn&#x2019;t let it pass by.</p>
<p>Since the new clients we&#x2019;re releasing today are a very close mapping to the REST API it should be very easy for any elasticsearch user to learn and use, as well as flexible enough for any possible use-case. This should enable us to work well with existing clients and provide a clean migration path for people who would wish to migrate &#x2013; we will try and work with the maintainers of existing clients to port their code to be a layer on top of this new client, thus enabling developers to keep their code while getting the benefit of using our new client underneath.</p>
<h2 id="ruby">Elasticsearch for Ruby</h2>
<p><b>GitHub repo:</b> <a href="https://github.com/elasticsearch/elasticsearch-ruby">https://github.com/elasticsearch/elasticsearch-ruby</a><br><b>Issues list:</b> <a href="https://github.com/elasticsearch/elasticsearch-ruby/issues">https://github.com/elasticsearch/elasticsearch-ruby/issues</a></p>
<h3>Installing</h3>
<pre class="prettyprint noescape lang=shell">gem install elasticsearch
</pre>
<h3>Example usage</h3>
<pre class="prettyprint noescape lang=ruby">require &apos;elasticsearch&apos;

# Connect to localhost:9200 by default:
es = Elasticsearch::Client.new log: true

# Round-robin between two nodes:
es = Elasticsearch::Client.new hosts: [&apos;search1:9200&apos;, &apos;search2:9200&apos;]

# Connect to cluster at search1:9200, sniff all nodes and round-robin between them
es = Elasticsearch::Client.new hosts: [&apos;search1:9200&apos;], reload_connections: true

# Index a document:
es.index index: &apos;my_app&apos;,
         type:  &apos;blog_post&apos;,
         id: 1,
         body: {
          title:   &quot;Elasticsearch clients&quot;,
          content: &quot;Interesting content...&quot;,
          date:    &quot;2013-09-24&quot;
         }

# Get the document:
es.get index: &apos;my_app&apos;, type: &apos;blog_post&apos;, id: 1

# Search:
es.search index: &apos;my_app&apos;,
          body: { query: { match: { title: &apos;elasticsearch&apos; } } }
</pre>
<h3>From the developer</h3>
<p>My name is Karel Mina&#x159;&#xED;k, and I discovered Elasticsearch in late 2010, when I was researching alternatives for CouchDB-Lucene, which I was using on a project at the time. As I&#x2019;m quite often quoted, I initially considered the project a hoax; there just can&#x2019;t exist such a magical project with so many buzzwords (HTTP, JSON, schema-free, distributed, &#x2026;), I was saying to myself. Nevertheless, I installed the project, glanced on the source, and it not only did what it said on the tin &#x2014; it surpassed all expectations, up to the level where my colleagues didn&#x2019;t believed me when I was demoing it.</p>
<p>Our application was written in Ruby, and we had a pretty good idea how we would like to &#x201C;talk&#x201D; to Elasticsearch: we need a DSL for its DSL! One late evening, I have <a href="https://gist.github.com/karmi/813387">sketched</a> such Ruby DSL, and after some time, we had a brand new client as a Rubygem, called <i>Slingshot</i> and later renamed to <i>Tire</i>. Little did I know that <a href="https://github.com/karmi/tire">this project</a> will get nearly 1,500 watchers and 500 forks at Github, that I&#x2019;ll solve more then 700 issues, and the library will grow to be loved and hated by many users.</p>
<p><i>Tire</i> managed to solve many issues for a nice Ruby and Rails integration, and it also managed to make some seriously bad decisions. Today marks a fresh start for me, and for Ruby users of Elasticsearch. Today, we&#x2019;re releasing a suite of clients with common design and semantics, which aims to provide a solid foundation for further extensions, integrations and experiments.</p>
<p>If you liked <i>Tire</i>, you can be sure that all the convenience for painless Rails integration will come soon. If you hated <i>Tire</i>, I&#x2019;d like to invite you to kick the tires on the new Ruby client. I took great care to avoid all the bad decisions of Tire, and make the library &#x2014; or, in fact a collection of three libraries &#x2014; as modular, extensible and robust as much as I could. Your feedback, opinions and critique &#x2014; either via Github issues, via IRC or via e-mail &#x2014; is most welcome!</p></body></html>

Release the clients!  Ruby, Python, PHP, Perl

<p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>This article will give an introduction to the mapping feature of Elasticsearch. We'll define the key terms and take a closer look at what mapping is, when we specify it, how it is structured and how we can apply it to our data.</p><section id="what-is-a-schema" class="level2"><h2><a href="#what-is-a-schema">What is a Schema?</a></h2><p>A <em>schema</em> is a description of one or more fields that describes the document type and how to handle the different fields of a document.</p><p>The schema in Elasticsearch is a mapping that describes the fields in the JSON documents along with their data type, as well as how they should be indexed in the Lucene indexes that lie under the hood. Because of this, in Elasticsearch terms, we usually call this schema a “mapping”.</p><p>Conceptually, an Elasticsearch server contains zero or more indexes. An index is a container for zero or more types, which in turn has zero or more documents. To put it another way: a document has an identifier, belongs to a type, which belongs to an index. The figure below shows the documents A, B, X and Y of the type my_type, inside the index my_index.</p><p><img src="https://api.contentstack.io/v2/assets/575e4c8b0342dfd7382630ac/download?uid=bltf30eccd262213fec?uid=bltf30eccd262213fec" data-sys-asset-uid="blt4852179309023a90" style="max-width: 100%;" alt="Indexes, types and documents"/></p><figcaption>Indexes, types and documents</figcaption><p>The type called another_type and the index called another is shown in order to emphasize that Elasticsearch is multi-tenant, by which we mean that a single server can store multiple indexes and multiple types.</p><p>In the Elasticsearch documentation and related material, we often see the term “mapping type”, which is actually the name of the type inside the index, such as my_type and another_type in the figure above. When we talk about types in Elasticsearch, it is usually this definition of type. It is not to be confused with the type key inside each mapping definition that determines how the data inside the documents are handled by Elasticsearch.</p></section><section id="when-to-specify-a-custom-mapping" class="level2"><h2><a href="#when-to-specify-a-custom-mapping">When to Specify a Custom Mapping</a></h2><p>Elasticsearch has the ability to be schema-less, which means that documents can be indexed without explicitly providing a schema.</p><p>If you do not specify a mapping, Elasticsearch will by default <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping.html">generate one dynamically</a> when detecting new fields in documents during indexing. However, this dynamic mapping generation comes with a few caveats:</p><ol type="1"><li>Detected types might not be correct.</li><li>May lead to unnecessary duplication. (The <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping-source-field.html">_source field</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping-all-field.html#custom-all-fields">_all field</a> especially.)</li><li>Uses default analyzers and settings for indexing and searching.</li></ol><p>For example, a timestamp is often represented in JSON as a long, but Elasticsearch will be unable to detect the field as a date field, preventing date filters and facets such as <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/search-aggregations-bucket-datehistogram-aggregation.html">the date histogram facet</a> from working properly.</p><p>By explicitly specifying the schema, we can avoid these problems.</p></section><section id="what-does-a-mapping-look-like" class="level2"><h2><a href="#what-does-a-mapping-look-like">What Does a Mapping Look Like?</a></h2><p>The mapping is usually provided to Elasticsearch as JSON, and is a hierarchically structured format where the root is the name of the type the mapping applies to.</p><section id="the-mapping-root" class="level3"><h3><a href="#the-mapping-root">The Mapping Root</a></h3><p>At the root level of the mapping, right under the type name, Elasticsearch supports a few “special” fields to configure how we should treat metadata that is not part of the document being posted, such as its type, id, size and its <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping-all-field.html#custom-all-fields">fallback _all field</a>. For an extensive list of supported “special” fields, see the list of “fields” on the right-hand side of <a href="/guide/reference/mapping">Mapping Reference</a>.</p><p>The <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping.html">root object</a> can also have a few extra attributes, which set the default indexing and search analyzers for the type, the date formats for automatically parsing dates in the type, and dynamic templates - we’ll revisit these in a future article. Apart from the fields mentioned above, there is no difference between the root level and the other mapping levels for a nested JSON document.</p></section><section id="hierarchical-levels" class="level3"><h3><a href="#hierarchical-levels">Hierarchical Levels</a></h3><p>Each level usually defines a properties setting, which maps the keys of the document at that level to its mapping. This structure is hierarchical, which means that every level down to the leaf nodes may include properties settings for its child values. To visualize it, consider the following document and its mapping:</p><p>Document:</p><pre class="sourceCode json prettyprint">{<br />    "name": {<br />        "first": "John"<br />    }<br />}<br /></pre><p>Mapping:</p><pre class="sourceCode json prettyprint">{<br />  "my_type" : {<br />    "properties" : {<br />      "name" : {<br />        "properties" : {<br />          "first" : {<br />            "type" : "string"<br />          }<br />        }<br />      }<br />    }<br />  }<br />}<br /></pre><p>While the mapping is slightly more convoluted than the document, the structure of the mapping clearly follows the structure of the document, with the added “properties”-nodes.</p></section><section id="the-type-key" class="level3"><h3><a href="#the-type-key">The type Key</a></h3><p>In the above example, we see that the document field name.first differs from the rest of the structure in that it defines a type. The type key is used on the leaf levels to tell Elasticsearch how to handle the field at the given level in the document. If the type key is omitted, as in the case of non-leaf types, Elasticsearch assumes it is of the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping.html">object type</a>.</p><p>The string type is one of the built-in <a href="/guide/reference/mapping/core-types">core types</a>, and Elasticsearch comes with support for many different types, such as <a href="/guide/reference/mapping/geo-point-type.html">geo_point</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping.html">ip</a>, which can be used to effectively index and search geographical locations and IPv4 addresses respectively. Using the <a href="/guide/reference/mapping/multi-field-type.html">multi_field</a> type, we can even index a single document field into multiple virtual fields. We’ll elaborate on this in a future article.</p></section></section><section id="how-to-provide-a-mapping" class="level2"><h2><a href="#how-to-provide-a-mapping">How to Provide a Mapping</a></h2><p>There are two ways of providing a mapping to Elasticsearch. The most common way is during <a href="/guide/reference/api/admin-indices-create-index.html">index creation</a>:</p><pre class="sourceCode json prettyprint">curl -XPOST ...:9200/my_index -d '{<br />    "settings" : {<br />        # .. index settings<br />    },<br />    "mappings" : {<br />        "my_type" : {<br />            # mapping for my_type<br />        }<br />    }<br />}'<br /></pre><p>Another way of providing the mapping is using the <a href="/guide/reference/api/admin-indices-put-mapping">Put Mapping API</a>.</p><pre class="sourceCode json prettyprint">$ curl -XPUT 'http://localhost:9200/my_index/my_type/_mapping' -d '<br />{<br />    "my_type" : {<br />        # mapping for my_type<br />    }<br />}<br />'<br /></pre><p>Note that the type (my_type) is duplicated in the request path and the request body.</p><p>This API enables us to update the mapping for an already existing index, but with some limitations with regards to potential conflicts. New mapping definitions can be added to the existing mapping, and existing types may have their configuration updated, but changing the types is considered a conflict and is not accepted. It is, however, possible to pass ignore_conflicts=true as a parameter to the Mapping API, but doing so does not guarantee producing the expected result, as already indexed documents are not re-indexed automatically with the new mapping.</p><p>Because of this, specifying the mapping during creation of the indexes is recommended over using the Put Mapping API in most cases.</p></section><section id="closing-remarks" class="level2"><h2><a href="#closing-remarks">Closing Remarks</a></h2><p>I have now introduced you to the schema/mapping in Elasticsearch and demonstrated how the mapping is a hierarchical definition of data types. In a later article, I will go into more detail about <a href="/blog/found-mapping-workflow">a workflow I use when I explore new datasets with Elasticsearch</a>.</p></section>

An Introduction to Elasticsearch Mapping

<html><head></head><body><h3>Histogram zero filling</h3>
<p>The histogram panel received a makeover that brought proper light weight zero filling. This means that at intervals where a query should have 0 results, it will properly appear as such, instead of drawing a sloped line to the next point. The zero filling also means that histogram stacked bars will always appear in the same order top to bottom.
</p>
<p>In addition, the stacked tooltip now allows you to choose between cumulative and individual modes.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbdd8edd48f769365ee/download?uid=bltbab2eec0e5378500?uid=bltbab2eec0e5378500" data-sys-asset-uid="blt85bb86ffa1747848" alt="kibana-2013-09-19-histogram-zero-filling.png" style="max-width:100%;">
</p>
<h3>Micro analysis of array fields</h3>
<p>Array fields can now be handled as singles or groups in the micro analysis panel. For example, if I have an array of tags I could see either the 10 most common tags, or the 10 most common ways that tags appear together
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbd9985d58976377264/download?uid=bltf33b91d9dd2d56f7?uid=bltf33b91d9dd2d56f7" data-sys-asset-uid="blt23627892f4b4c182" alt="kibana-2013-09-19-micro-analysis-array-fields.png" style="max-width:100%;">
</p>
<h3>_source as the default table field</h3>
<p>If you don&apos;t select any fields for your table, Kibana will now show you the json source of your even by default until you select some fields to show
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbd3dc542cb38c08ac8/download?uid=bltb9a5237e0e7bb1ea?uid=bltb9a5237e0e7bb1ea" data-sys-asset-uid="blt2b39b71d6dddb7ae" alt="kibana-2013-09-19-source-as-default-table-field.png" style="max-width: 100%;">
</p>
<h3>Configurable field trimming</h3>
<p>Noticed the &apos;...&apos; at the end of the _source in the above screenshot? Table fields can be trimmed by a configurable &quot;factor&quot;. The factor is essentially the maximum length of a field, divided by the number of columns in the table, after which the field will be trimmed to nicely fit. For example, if my trim factor was 300, and I had 3 columns in the table, fields would be trimmed to a max of 100 character, after which &apos;...&apos; would be added. Of course the entirety of the field is still available in the expanded details view
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbd0342dfd73826396a/download?uid=bltd3d33f405d425fbc?uid=bltd3d33f405d425fbc" data-sys-asset-uid="blt7a90a3aa657b4b52" alt="kibana-2013-09-19-configurable-field-trimming.png" style="max-width:100%;">
</p>
<h3>About that details view</h3>
<p>You may know that you can see a table of event fields by click on a table row. Now you have a choice of how you&apos;d like to see the details of an event, including syntax highlighted JSON, as well as the raw unhighlighted JSON.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4cbd9e7a83165490c945/download?uid=blt3d90153ba40fd812?uid=blt3d90153ba40fd812" data-sys-asset-uid="blt0c6734ff59a29e1d" alt="kibana-2013-09-19-about-that-details-view.png" style="max-width:100%;">
</p>
<h3>Lighter, faster, smaller, better</h3>
<p>Kibana has an all new build system! This new system allows us to build an optimized, minified, awesomulated distribution of Kibana. It also has the benefit of cleanly clearing old caches when you upgrade. Regular builds of Kibana are being published at <a href="http://download.elastic.co/kibana/kibana/kibana-latest.zip">http://download.elastic.co/kibana/kibana/kibana-latest.zip</a> for your convenience. That zipball can be extracted right to your webserver.
</p>
<p>You can still run it right out of the <a href="https://github.com/elasticsearch/kibana">Github repository</a> if you want. However instead of copying the entire repo, you need only upload the src/ directory to your webserver. We recommend grabbing the built version however as its much faster.
</p></body></html>

This Week in Kibana

<html><head></head><body><p>Welcome to&#xA0;<i>This Week in Elasticsearch</i>. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the Github repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.</p><h2>Elasticsearch core</h2><ul>
<li>Elasticsearch 0.90.4 and 0.90.5 (fixing an issue to install site plugins from Github) have been released. Check out the blog posts for <a href="http://www.elasticsearch.org/blog/0-90-4-released/">0.90.4</a> and <a href="http://www.elasticsearch.org/blog/0-90-4-released/">0.90.5</a> as well as the accompanying release notes (<a href="http://www.elasticsearch.org/downloads/0-90-4/">0.90.4</a> and <a href="http://www.elasticsearch.org/downloads/0-90-5/">0.90.5</a>). Especially check out the new function scoring capability!</li>
<li>Since switching to randomized testing, which randomizes all kinds of stuff (number of nodes, number of shards, number of replicas to start with) we have found some tests need to be refined in order to reflect different states compared to the static tests before. We are currently investing some time to improve rarely failing tests since the switch. You will find quite a few test related commits (fixing, stabilizing, adding logging, log specific annotations, changing loglevels etc) over the past weeks in our commit log, as we are really keen to have the most stable tests possible, even in a distributed system.</li>
<li>The <code>CompletionFieldMapper</code>&#xA0;now takes the <code>path: just_name</code> option correctly into account (<a href="https://github.com/elasticsearch/elasticsearch/issues/3669">#3669</a>, 0.90 &amp; master)</li>
<li>The REST API now supports returning all index templates, also some deprecated methods were removed (<a href="https://github.com/elasticsearch/elasticsearch/issues/2532">#2532</a> and <a href="https://github.com/elasticsearch/elasticsearch/issues/3681">#3681</a>, 0.90 &amp; master)</li>
<li>The RPM and Debian packages now both support a configuration option to not restart on package upgrade. This should prevent data moving around during updates. This is also now configured as default. (<a href="https://github.com/elasticsearch/elasticsearch/issues/3685">#3685</a>, 0.90 &amp; master)</li>
<li>A new dedicated thread pool for suggestion has been added, to prevent competing against normal search operations inside of this pool (<a href="https://github.com/elasticsearch/elasticsearch/issues/3698">#3698</a>, 0.90 &amp; master)</li>
<li>The refresh flags have been removed from the Flush and Optimize API (<a href="https://github.com/elasticsearch/elasticsearch/issues/3689">#3689</a> and <a href="https://github.com/elasticsearch/elasticsearch/issues/3690">#3690</a>, 0.90 &amp; master)</li>
<li>The <code>RestActions#splitXXX(String)</code> methods have been removed in favor of using <code>Strings.splitStringByCommaToArray(String)</code>. Please check if your plugin needs to be updated (<a href="https://github.com/elasticsearch/elasticsearch/issues/3680">#3680</a>, 0.90 &amp; master)</li>
<li>On index removal a map with possibly waiting mappings to be applied was not cleaned correctly (<a href="https://github.com/elasticsearch/elasticsearch/issues/3697">#3697</a>, 0.90 &amp; master)</li>
<li>Library upgrades: Guava v15, (<a href="https://github.com/elasticsearch/elasticsearch/commit/cd9038296418d3a409055cc5406fa38925ed65a8">commit</a>), jackson 2.2.3, (<a href="https://github.com/elasticsearch/elasticsearch/commit/88173d762ae94d4e6ccd815f2dfc710ad9a443ce">commit</a>), netty 3.7.0, (<a href="https://github.com/elasticsearch/elasticsearch/commit/31097691af6a4e8d5ecbd8c6d7dc257817558465">commit</a>), 0.90 &amp; master</li>
<li>The Debian init script is now bourne shell compatible (<a href="https://github.com/elasticsearch/elasticsearch/issues/3691">#3691</a>, 0.90 &amp; master)</li>
<li>The Debian init script now also searches for JDKs of the ARM platform (<a href="https://github.com/elasticsearch/elasticsearch/issues/3659">#3659</a>, 0.90 &amp; master)</li>
<li>Completion Stats are now correctly included/excluded in REST responses (<a href="https://github.com/elasticsearch/elasticsearch/issues/3522">#3522</a>, 0.90 &amp; master)</li>
<li>Elasticsearch start-up shell script now copes with spaces in the directory name (<a href="https://github.com/elasticsearch/elasticsearch/issues/3712">#3712</a>, 0.90 &amp; master)</li>
<li>If a flush request hits an already ongoing flush, wait for the first one to finish (<a href="https://github.com/elasticsearch/elasticsearch/issues/3631">#3631</a>, 0.90 &amp; master)</li>
<li>Searching over all indices if none exists returns no results now instead of an exception (<a href="https://github.com/elasticsearch/elasticsearch/issues/3710">#3710</a>, 0.90 &amp; master)</li>
<li>New configuration parameter <code>node.mode</code> which supports <code>local</code> or <code>network</code> (<a href="https://github.com/elasticsearch/elasticsearch/issues/3713">#3713</a>, 0.90 &amp; master)</li>
<li>Site plugin copy mechanism was broken for 0.90.4 (<a href="https://github.com/elasticsearch/elasticsearch/issues/3707">#3707</a>, 0.90 &amp; master)</li>
<li>Elasticsearch can now be started as a service under windows (<a href="https://github.com/elasticsearch/elasticsearch/issues/3716">#3716</a>, 0.90 &amp; master)</li>
<li>Geo distance sorting now accepts <code>sort_mode</code> parameter (<a href="https://github.com/elasticsearch/elasticsearch/issues/3717">#3717</a>, 0.90 &amp; master)</li>
<li>Alias filters were not applied when using &apos;multi-index&apos; syntax with wild card in URL (<a href="https://github.com/elasticsearch/elasticsearch/issues/3677">#3677</a>, 0.90 &amp; master)</li>
</ul><h2>Elasticsearch ecosystem</h2><p>Here&apos;s some more information about what is happening in the ecosystem we are maintaining around Elasticsearch, including plugin and driver releases, as well as news about Logstash and Kibana.</p><ul>
<li>The <a href="https://github.com/elasticsearch/elasticsearch-metrics-reporter-java">Elasticsearch metrics reporter</a>, which reports data collected using the <a href="http://metrics.codahale.com/">metrics library</a> to Elasticsearch, is now under the official Elasticsearch umbrella</li>
<li>Found launched <a href="http://www.found.no/foundation/">Foundation</a>, a web site with articles about Elasticsearch</li>
<li>The Elasticsearch langdetect plugin has been <a href="https://github.com/jprante/elasticsearch-langdetect">updated to version 1.2.0</a></li>
<li>The Elasticsearch knapsack plugin has been <a href="https://github.com/jprante/elasticsearch-knapsack">updated to version 2.0.2</a></li>
<li>An interactive <a href="https://github.com/YannBrrd/elasticsearch-entity-resolution/">entity resolution plugin</a> using <a href="https://code.google.com/p/duke/">Duke</a> has been released</li>
</ul><h2>Slides</h2><ul>
<li>Chris Simpson talked at the London Elasticsearch meetup about his <a href="http://skillsmatter.com/podcast/nosql/chris-simpson-speaks-at-london-elasticsearch-meetup">experiences with Elasticsearch</a></li>
</ul><h2>Where to find Us</h2><h4>Australia</h4><p>On September 19th, the <a href="http://www.meetup.com/melbourne-search/events/135452552/">Melbourne Search Users Group</a> will have talk by RealEstate.com.au about their migration to Elasticsearch.</p><h4>Austria</h4><p><a href="http://twitter.com/spinscale">Alexander Reelsen</a> will be presenting <a href="http://community.ejug.at/networks/events/show_event.82703">Elasticsearch - Beyond full-text search</a> at EJUG Austria on September 25th.</p><h4>Czech Republic</h4><p>Should you find yourself in Prague for <a href="http://webexpo.net/prague2013/">WebExpo Prague 2013</a>, stop in to see <a href="https://twitter.com/karmiq">Karel Minarik</a> and <a href="https://twitter.com/HonzaKral">Honza Kral</a> co-present <a href="http://webexpo.net/prague2013/talk/explore-your-data-with-elasticsearch/">Elasticsearch: Beyond Full Text Search</a>.</p><h4>Denmark</h4><p><a href="https://twitter.com/spinscale">Alex Reelsen</a> will be attending to <a href="http://gotocon.com/aarhus-2013/">GOTO Aarhus</a> with a few other folks from the Elasticsearch team on September 30th - October 2nd. You can pop by our booth to meet us, or just head to Alex&apos;s talk <a href="http://gotocon.com/aarhus-2013/speaker/Alex+Reelsen">Elasticsearch - Beyond Full Text Search</a>.</p><h4>France</h4><ul>
<li>For folks in and around Strasbourg on Wednesday, September 18th, <a href="https://twitter.com/dadoonet">David Pilato</a> will present an overview of Elasticsearch at the <a href="http://www.meetup.com/elasticsearchfr/events/133734512/">Elsass Java Users Group</a>.</li>
<li>If you&apos;re heading to <a href="http://www.jugsummercamp.com/">JUG Summer Camp</a>, make sure to check out David&apos;s two sessions: <a href="http://www.jugsummercamp.com/edition/4/presentation/1059">Make Sense of Your (BIG) Data</a> and <a href="http://www.jugsummercamp.com/edition/4/presentation/1067">Elastifiez Your Application: SQL to NoSQL in Less Than an Hour</a>. David joins his fellow campers in La Rochelle on Friday, September 20th.</li>
<li>David will also give attendees of Open World Forum a taste of <a href="http://2013.openworldforum.org/en/tracks/15#talk_79">Elastifiez Your Application: from SQL to NoSQL in in less than 40 mn</a>, as well as lead an <a href="http://2013.openworldforum.org/en/tracks/31#talk_82">Elasticsearch workshop</a>.  Open World Forum takes place October 3rd-5th in Paris.</li>
</ul><h4>Germany</h4><ul>
<li><a href="http://twitter.com/lhawthorn">Leslie Hawthorn</a> will be attending Monitorama on September 19-20th in Berlin. Ping her on Twitter if you want to talk all things Elasticsearch over a tasty beverage.</li>
<li>The <a href="http://www.meetup.com/ElasticSearch-UG-Berlin/">Elasticsearch User Group Berlin</a> will meet September 24th. The theme for this month&apos;s meetup is <a href="http://www.meetup.com/ElasticSearch-UG-Berlin/events/134717872/">Beginner&apos;s Q&amp;A Edition</a>.</li>
</ul><h4>Norway</h4><p>The fine folks spearheading the <a href="http://www.meetup.com/elasticsearch-oslo/events/137300022/">Oslo Elasticsearch Users Group</a> will host their inaugural meetup on Thursday, September 26th. <a href="https://twitter.com/mvgroningen">Martijn van Groningen</a> will join the festivities to talk about new features in the works for Elasticsearch.</p><h4>Ukraine</h4><p><a href="http://rubyshift.org/">RubyShift 2013</a> kicks off on Friday, September 27th in Kiev. Karel Minarik will be there, giving a presentation on <a href="http://rubyshift.org/#agenda">Elasticsearch: Search and Analytics</a> the first day of the conference.</p><h4>United States</h4><ul>
<li><a href="http://twitter.com/thnetos">Lee Hinman</a> and <a href="http://twitter.com/drewr">Drew Raines</a> will be attending the <a href="https://thestrangeloop.com/">Strangeloop conference</a> in Saint Louis from September 18-20th. Stop by the Elasticsearch table in the Sponsor Exhibits area to say hello!</li>
<li><a href="http://loggly.com">Loggly</a> will host the <a href="http://www.meetup.com/Elasticsearch-San-Francisco/events/133202632/">Elasticsearch San Francisco</a> meetup on September 23rd, featuring a presentation by Lee Hinman on snapshot/restore and forthcoming features slated for our v 1.0 release. You&apos;ll also have the opportunity to talk to three other core developers, <a href="http://www.linkedin.com/profile/view?id=20107">Bill Hwang</a>, <a href="https://twitter.com/imotov">Igor Motov</a>, and <a href="https://twitter.com/SpencerAlger">Spencer Alger</a> - during the meetup&apos;s Q&amp;A hour.</li>
<li>If you&apos;re heading to<a href="http://dataweek.co/"> DataWeek 2013</a>, make sure to check out Shay Banon on Elasticsearch&apos;s APIs: <a href="http://apiworld2013conferencefestival.sched.org/event/aacb9b4ea8f67ac841baec81b940e3fe?iframe=no&amp;w=760&amp;sidebar=yes&amp;bg=no#.Ui-joGSbh8E">Making It the Easy Way, Doing It the Hard Way</a>. Shay will be speaking on Wednesday, October 2nd; DataWeek 2013 runs from September 28th - October 2nd in San Francisco.</li>
<li>Drew Raines will be at <a href="http://devopsdays.org/events/2013-atlanta/">DevOpsDays Atlanta</a> on October 3-4th.</li>
</ul><h2>Where to Find You</h2><p>Are you hosting an Elasticsearch meetup or giving a talk about Elasticsearch? We would love to know so we can feature that information in future editions of <em>This Week in Elasticsearch</em>. Just let our <a href="http://twitter.com/lhawthorn">Community Manager</a> know what you&apos;re up to and we&apos;re happy to help promote your efforts.</p><p>Oh yeah, we&apos;re also <a href="http://www.elasticsearch.com/about/jobs/">hiring</a>. If you&apos;d like us to find you for employment purposes, just <a href="mailto:jobs@elasticsearch.com">drop us a note</a>. &#xA0;We care more about your skill set and passion for elasticearch, kibana and logstash than where you rest your head.</p><h2>Training</h2><p>If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:</p><ul>
<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/cologne/2013-10-17">Cologne</a> - October 17, 2013</li>
<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/washington-dc/2013-10-21">Washington, D.C.</a> - October 21, 2013</li>
<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/san-francisco/2013-10-24">San Francisco</a> - October 24, 2013</li>
<li><a href="http://training.elasticsearch.com/class/elasticsearch/core-elasticsearch/zurich/2013-10-29">Zurich</a> - October 29, 2013</li>
</ul></body></html>

This Week in Elasticsearch - September 18, 2013

<html><head></head><body><p>The Elasticsearch dev team are pleased to announce the release of <strong>Elasticsearch 0.90.5</strong>, which is based on Lucene 4.4. You can download it&#xA0;<a title="0.90.5" href="http://www.elasticsearch.org/downloads/0-90-5/">here</a>.</p><p>We released 0.90.4 yesterday with great expectations, and a few hours later had our expectations dashed by a silly bug in the plugin install script, which installed site plugins (e.g. head, bigdesk, paramedic) incorrectly. We apologize for that. That bug has been fixed and we&apos;ve added the appropriate tests.  </p><p>However, that is not all.  Things move fast in the Elasticsearch world, so we&apos;ve taken advantage of this quick release to fix a few other bugs:</p><ul>
<li>Elasticsearch startup script doesn&apos;t work from directory with spaces in path <a href="https://github.com/elasticsearch/elasticsearch/issues/3712">#3712</a>
</li>
<li>A problem with sorting nested documents with <code>score_mode: avg</code> <a href="https://github.com/elasticsearch/elasticsearch/issues/3706">#3706</a>
</li>
<li>Alias filters were not being applied when indices were specified using a wildcard <code>*</code> <a href="https://github.com/elasticsearch/elasticsearch/issues/3677">#3677</a>
</li>
</ul><h3>Windows Support</h3><p>And finally, an enhancement for those of you using Elasticsearch on Windows: we have added support for running Elasticsearch as a service on Windows <a href="https://github.com/elasticsearch/elasticsearch/issues/3716">(see #3716)</a>, via the <code>service.bat</code> script. You can now do:</p><pre class="prettyprint lang-shell">&gt; service.bat
Usage: service.bat install|remove|start|stop|manager [SERVICE_ID]

&gt; service install
Installing service      :  &apos;elasticsearch-service-x64&apos;
Using JAVA_HOME (64-bit):  c:\jvm\jdk1.7
The service &apos;elasticsearch-service-x64&apos; has been installed.

&gt; service start
The service &apos;elasticsearch-service-x64&apos; has been started

&gt; service stop
The service &apos;elasticsearch-service-x64&apos; has been stopped

&gt; service remove
The service &apos;elasticsearch-service-x64&apos; has been removed

</pre><p>All the details are available in the <a href="http://www.elasticsearch.org/downloads/0-90-5/">release notes</a>.  If you are not affected by any of the above issues then there is no need for you to upgrade from 0.90.4.</p><p>Thank you for your feedback on the release of 0.90.4 yesterday - it allowed us to find and fix the plugin script error quickly. </p></body></html>

Elasticsearch 0.90.5 Released

Elasticsearch from the Bottom Up, Part 1

<html><head></head><body><section id="post_content">
		<article>
			<p>The Elasticsearch dev team are pleased to announce the release of E<strong>lasticsearch 0.90.4</strong>, which is based on Lucene 4.4. You can download it <a title="0.90.4" href="http://www.elasticsearch.org/downloads/0-90-4/">here</a>.</p>
<p>As usual, this release contains numerous optimizations and bug fixes, which you can read about in the <a href="http://www.elasticsearch.org/downloads/0-90-4/">release notes</a>, but there are some exciting new features which warrant a special mention in this blog post:</p>
<ul>
<li> <a href="#function_score">Function score query</a>
</li>
<li> <a href="#allocator">Disk-space aware shard allocation</a>
</li>
<li> <a href="#suggester">Suggester improvements</a>
</li>
<li> <a href="#other">Other smaller enhancements</a>
</li>
</ul>
<h2 id="#function_score">Function score</h2>
<p>Elasticsearch already had extensive support for custom manipulation of the relevance _score in the Query DSL with the:</p>
<ul>
<li> <code>custom_score</code> query: modify the relevance <code>_score</code> by running a script on fields in the document
</li>
<li><code>custom_filters_score</code> query: modify the <code>_score</code> of documents which match one or more filters using a boost value or script (which benefits from filter caching)
</li>
<li><code>custom_boost_factor</code> query: modify the <code>_score</code> by a multiplier, rather than using a `boost` value which gets normalized
</li>
</ul>
<p>In 0.90.4 we present the new <code>function_score</code> query which combines the functionality of the above three queries to make them more flexible and efficient, and adds support for predefined functions. These allow you to include <em>recency</em>, <em>distance-from-a-point</em>, <em>popularity</em> or even <em>randomness</em> in your scoring model.</p>
<h3>Decay functions:</h3>
<p>A common use case is the need to boost recently created documents over older results. You could achieve this by applying a <code>function_score</code> query to the <code>timestamp</code> field. The <code>scale</code> parameter allows you to control how quickly this recency effect should decay. In the following example, blog posts from one month ago will score half as much as blog posts from today:</p>
<pre class="noescape prettyprint lang-js">curl localhost:9200/blogposts/_search -d &apos;
&quot;query&quot;: {
  &quot;function_score&quot;: {
    &quot;query&quot;: {
      &quot;match&quot;: { &quot;title&quot;: &quot;elasticsearch&quot;}
    },
    &quot;gauss&quot;: {
      &quot;timestamp&quot;: {
          &quot;scale&quot;:  &quot;4w&quot;
      }
    }
  }
}&apos;
</pre>
<p>The above query would, however, reduce the score of relevant-but-very-old docs to almost zero, because the Gaussian decay function (like a bell curve) outputs a number between 0 and 1, which is then multiplied with the query <code>_score</code>.  If you want old-but-relevant docs to still be ranked highly, then a better recency calculation would output a multiplier between 1 and 2:</p>
<pre class="noescape prettyprint lang-js">curl localhost:9200/blogposts/_search -d &apos;
&quot;query&quot;: {
  &quot;function_score&quot;: {
    &quot;query&quot;: {
      &quot;match&quot;: { &quot;title&quot;: &quot;elasticsearch&quot;}
    },
    &quot;functions&quot;: [
      { &quot;boost&quot;:  1 },
      {
        &quot;gauss&quot;: {
          &quot;timestamp&quot;: {
            &quot;scale&quot;: &quot;4w&quot;
          }
        }
      }
    ],
    &quot;score_mode&quot;: &quot;sum&quot;
  }
}&apos;
</pre>
<p>The <code>score_mode</code> tells the <code>function_score</code> query that the output from both functions should be added together (ie <code>1 + gauss()</code>) before being multiplied with the <code>_score</code> from the query.</p>
<p>These decay functions can be applied to any <b>date</b>, <b>numeric</b> or <b>geo_point</b> field. For instance, if a user searches for a hotel in Central London which costs less than &#xA3;80 per night, and has received good ratings from other users, you may have a number of very highly rated hotels which almost match all of the criteria, but perhaps cost a bit more, or lie just outside Central London.</p>
<p>Instead of using filters, which would just exclude results which don&#x2019;t match, you can use a decay function for each of these criteria:</p>
<pre class="noescape prettyprint lang-js">curl localhost:9200/hotels/_search -d &apos;
&quot;query&quot;: {
  &quot;function_score&quot;: {
    &quot;functions&quot;: [
      { &quot;gauss&quot;:  { &quot;price&quot;: { &quot;origin&quot;: 0,      &quot;scale&quot;: 40    }}},
      { &quot;gauss&quot;:  { &quot;loc&quot;:   { &quot;origin&quot;: &quot;51,0&quot;, &quot;scale&quot;: &quot;5km&quot; }}},
      { &quot;linear&quot;: { &quot;stars&quot;: { &quot;origin&quot;: 5,      &quot;scale&quot;: 1     }}}
    ]
  }
}&apos;
</pre>
<p>The hotels which most closely match all of the criteria will receive the highest scores, but almost-matches will still appear in the results.</p>
<p>The <em>shape</em> of the decay can be controlled by choosing the most appropriate function (Gaussian, exponential and linear) and the <em>rate of decay</em> can be controlled with the `scale` and `decay` parameters.  See <a href="/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html">the documentation</a> for more information.</p>
<h3>Random sorting</h3>
<p>Occasionally, users have the need to sort documents randomly. Perhaps you have sponsored listings that you want to include in your results, but you want to ensure that all sponsors get equal exposure. One way to do this would be to sort results randomly.  However, you want a single user to see results in a consistent order, so that they won&#x2019;t see duplicate results on pages 2 or 3.  </p>
<p>This can now be achieved with the <code>function_score</code> query, using e.g.,&#xA0;the timestamp of the user&#x2019;s initial search as the <em>random seed</em>:</p>
<pre class="noescape prettyprint lang-js">curl localhost:9200/sponsors/_search -d &apos;
&quot;query&quot;: {
  &quot;function_score&quot;: {
    &quot;query&quot;: { &quot;match&quot;: { &quot;title&quot;: &quot;elasticsearch&quot;}}
    &quot;random_score&quot;: {
      &quot;seed&quot;: 1379333621000
    }
  }
}&apos;
</pre>
<p>As long as you use the same <em>seed</em> for each search, results will be sorted consistently. This allows you to choose whether you want to randomise results differently for every new search that a user runs, or to keep the same sort order for a single user throughout their session.</p>
<h2 id="allocator">Disk-space aware shard allocation</h2>
<p>The shard allocator can now take the amount of free space left on a disk into account when deciding where to allocate shards.  This can be enabled in the config file or dynamically on a live cluster by setting <code>cluster.routing.allocation.disk.threshold_enabled</code> to <code>true</code>. When enabled, the allocator takes two <em>watermarks</em> into account:</p>
<p>The low watermark (<code>cluster.routing.allocation.disk.watermark.low</code>, default <code>0.7</code>) will not allocate new shards to nodes whose disks usage is greater than 70%.  The high watermark (<code>cluster.routing.allocation.disk.watermark.high</code>, default <code>0.85</code>) tells the allocator to actively try to reallocate shards when a node&#x2019;s disk usage rises above 85%.  Both watermarks can also use absolute values like <code>&quot;500mb&quot;</code>.</p>
<h2 id="suggester">Suggester improvements</h2>
<p>The <a href="/guide/en/elasticsearch/reference/current/search-suggesters-completion.html">completion suggester</a> made its first appearance in 0.90.3, and this release adds:</p>
<ul>
<li> <b>fuzzy completion</b>, to allow matching of misspelled words <a href="https://github.com/elasticsearch/elasticsearch/issues/3465">#3465</a>
</li>
<li> <b>completion stats</b>, so that you can see how much memory is being used by each suggester <a href="https://github.com/elasticsearch/elasticsearch/issues/3522">#3522</a>
</li>
<li> <b>reduced memory usage</b> for payloads, which now accept scalar values instead of requiring JSON objects <a href="https://github.com/elasticsearch/elasticsearch/issues/3550">#3550</a>
</li>
</ul>
<p>We&#x2019;ve also added result highlighting to the <a href="/guide/en/elasticsearch/reference/current/search-suggesters-phrase.html">phrase suggester</a>, to highlight the parts of the phrase that the suggester has added or corrected. <a href="https://github.com/elasticsearch/elasticsearch/issues/3442">#3442</a></p>
<h2 id="other">Other improvements</h2>
<ul>
<li> The <code>clear scroll</code> API allows you to expire old scrolls as soon as you are finished with them, freeing up resources more quickly <a href="https://github.com/elasticsearch/elasticsearch/issues/3657">#3657</a>
</li>
<li> The <code>rest.action.multi.allow_explicit_index</code> setting allows you to restrict the bulk, multi-get and multi-search APIs to just the indices used in the URL, making it easier to use a URL-based security approach <a href="https://github.com/elasticsearch/elasticsearch/issues/3636">#3636</a>
</li>
<li> Named queries have been added to the existing named filter support, so that you can tell which query or filter clauses a document matched<br>
  <a href="https://github.com/elasticsearch/elasticsearch/issues/3581">#3581</a>
</li>
</ul>
<p>We hope you enjoy this new release. Please <a title="0.90.4" href="http://www.elasticsearch.org/downloads/0-90-4/">download and use 0.90.4</a>, and let us know what you think.</p>
			<aside id="post_tags">
							</aside>
		</article>
			</section></body></html>

Elasticsearch 0.90.4 released

<p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud. <a href="https://www.elastic.co/cloud/as-a-service/signup">14-day hosted Elasticsearch no-cost trials are available on Elastic Cloud</a>.</p><p>Can Elasticsearch be used as a "NoSQL"-database? NoSQL means different things in different contexts, and interestingly it's not really about SQL. We will start out with a "Maybe!", and look into the various properties of Elasticsearch as well as those it has sacrificed, in order to become one of the most flexible, scalable and performant search and analytics engines yet.</p><section id="what-is-a-nosql-database-anyway" class="level2"><h2><a href="#what-is-a-nosql-database-anyway">What is a NoSQL Database Anyway?</a></h2><p><a href="http://nosql-database.org/">NoSQL-database</a> defines NoSQL as "Next Generation Databases <em>mostly</em> addressing <em>some</em> of the points: being non-relational, distributed, open-source and horizontally scalable.". In other words, it's not a very precise definition.</p><p>It's not about SQL in particular. For example, <a href="http://hive.apache.org/">Hive</a>'s query language is clearly inspired by SQL. The same is true for Esper's query language, which operates on streams instead of relations. Also, did you know PostgreSQL was named "Postgres" and had "Quel" as its query language back in the days? While first and foremost an ORDBMS, it now also has many <a href="http://thebuild.com/blog/2013/07/02/postgresql-as-a-nosql-database/">features to make it viable as a schemaless document-store</a>.</p><p>It's not about ACID-ity either. Hyperdex is one example of a NoSQL-database that aims to provide ACID-transactions. MySQL, certainly an SQL-database, has a history of dubious interpretations of what ACID really means.</p><p>Relations? While most of the NoSQL-databases do not support joining in the same sense as traditional relational databases and leave that as an exercise for the user, there are those that do. <a href="http://www.rethinkdb.com/">RethinkDB</a>, <a href="http://hive.apache.org/">Hive</a> and <a href="http://pig.apache.org/">Pig</a>, to name a few. <a href="http://www.neo4j.org/">Neo4j</a>, a graph-oriented database, certainly deals with relations - it's excellent at traversing relations (i.e. edges) in graphs. Elasticsearch has a concept of "query time" joining with <a href="/guide/reference/mapping/parent-field">parent/child-relations</a> and "index time" joining with <a href="/guide/reference/mapping/nested-type">nested types</a>.</p><p>Distributed? While there are some distributed SQL-databases around, and <a href="https://github.com/petersirka/nosql">some</a> <a href="http://ejdb.org/">projects</a> aiming to be something like a NoSQLite, newer generation databases tend to be distributed in some way or another.</p><p>To summarize the summary, it neither makes sense to precisely define NoSQL, nor to simply say that Elasticsearch is a "document store"-type NoSQL-database. At the time of writing, <a href="http://nosql-database.org/">nosql-database.org</a> lists &gt;20 of those.</p><p>In the next sections, we'll have a look at some important properties and see how Elasticsearch does or does not implement them.</p></section><section id="transactions" class="level2"><h2><a href="#transactions">No Transactions</a></h2><p>Lucene, which Elasticsearch is built on, has a notion of transactions. Elasticsearch on the other hand, does not have transactions in the typical sense. There is no way to rollback a submitted document, and you cannot submit a group of documents and have either all or none of them indexed. What it does have, however, is a write-ahead-log to ensure the durability of operations without having to do an expensive Lucene-commit. You can also specify the consistency level of index-operations, in terms of how many replicas must acknowledge the operation before returning. This defaults to a quorum, i.e. <span class="math">$\lfloor\frac{n}{2}\rfloor + 1$</span>.</p><p>Visibility of changes is controlled when an index is refreshed, which by default is once per second, and happens on a shard-by-shard-basis.</p><p>Optimistic concurrency control is done by specifying the version of the submitted documents.</p><p>Elasticsearch is built for speed. Doing distributed transactions is a <em>lot</em> of work. Not providing them makes a lot of things easier. By accepting that what we read can be somewhat stale, and that everyone sees the same timeline, Elasticsearch can serve a lot of things from caches - which is paramount for the mind-boggling performance we love it for.</p></section><section id="schema-flexible" class="level2"><h2><a href="#schema-flexible">Schema Flexible</a></h2><p>Elasticsearch does not <em>require</em> you to specify a schema upfront. Throw a JSON-document at it, and it will do some educated guessing to infer its type. It does a good job at things like numerics, booleans and timestamps. For strings, it will use the "standard"-analyzer, which is usually good to get started.</p><p>While it's arguably "schema free", in the sense that you don't <em>have</em> to specify a schema, we like to think of it as "schema <em>flexible</em>" instead. To develop great search and/or analytics, you really need to tweak your schemas. Elasticsearch has an extensive set of powerful tools to help you, like dynamic templates, multi-field objects, etc. This is covered in more detail in our article on mapping.</p></section><section id="relations-and-constraints" class="level2"><h2><a href="#relations-and-constraints">Relations and Constraints</a></h2><p>Elasticsearch is a document oriented database. The entire object graph you want to search needs to be indexed, so before indexing your documents, they must be <em>denormalized</em>. Denormalization increases retrieval performance (since no query joining is necessary), uses more space (because things must be stored several times), but makes keeping things consistent and up-to-date more difficult (as any change must be applied to all instances). They're excellent for write-once-read-many-workloads, however.</p><p>For example, say you have set up database containing customers, orders and products, and you want to search for orders given the name of a product and user. This could be solved by indexing orders with all the necessary information about the user and the products. Searching is then easy, but what happens when you want to change the name of the product? In a relational design with proper normalization, you would simply update the product and be done. That's what they are really good at. With a denormalized document database, every order with the product would have to be updated.</p><p>In other words, with document oriented databases like Elasticsearch, we design our mappings and store our documents such that it's optimized for search and retrieval.</p><p>As mentioned in the introduction, Elasticsearch has a concept of "query time" joining with <a href="/guide/reference/mapping/parent-field">parent/child-relations</a>, and "index time" joining with <a href="/guide/reference/mapping/nested-type">nested types</a>. We'll probably cover this in more depth in a future article. In the meantime, we can recommend Martijn van Groningen's presentation "<a href="/videos/bbuzz2013-document-relations-with-elasticsearch">Document relations with Elasticsearch</a>".</p><p>Most relational databases also let you specify constraints to define what is and isn't consistent. For example, referential integrity and uniqueness can be enforced. You can require that the sum of account movements must be positive and so on. Document oriented databases tend not to do this, and Elasticsearch is no different.</p></section><section id="robustness" class="level2"><h2><a href="#robustness">Robustness</a></h2><p>A database should be robust, especially if it is your authoritative system of record. Ideally, a costly query should be possible to cancel, and you certainly don't want the database to stop working unless you tell it to.</p><p>Unfortunately, Elasticsearch (and the components it's made of) does not currently handle OutOfMemory-errors very well. We cover this in more depth in <a href="/blog/found-elasticsearch-in-production">Elasticsearch in Production, OutOfMemory-Caused Crashes</a>. It is very important to provide Elasticsearch with enough memory and be careful before running searches with unknown memory requirements on a production cluster.</p><p>While this is likely to improve as Elasticsearch matures, it's important to remember that Elasticsearch is built for <em>speed</em>, with the assumption that memory is abundant.</p></section><section id="distributed" class="level2"><h2><a href="#distributed">Distributed</a></h2><p>See also: <a href="/elasticsearch">Elasticsearch in Production, Networking</a>.</p><p>Before Shay Banon created Elasticsearch, he had been working on <a href="https://thedudeabides.com/articles/the_future_of_compass">Compass</a>. Realizing it would be hard to turn it into a distributed search engine, he started from scratch and created Elasticsearch<span class="citation" data-cites="future-of-compass"><a href="#fn1" class="footnoteRef" id="fnref1"><sup>1</sup></a></span>. Elasticsearch is designed to be distributed and easy to scale <em>out</em> to handle massive amounts of data on commodity hardware.</p><p>Elasticsearch is incredibly easy to use and get started with for a distributed system, but distributed systems are <em>complicated</em>. We cover this a bit more in <a href="/blog/found-elasticsearch-in-production">Elasticsearch in Production, Networking</a>, so what follows is a short summary.</p><p>The very nature of a distributed system implies a myriad of things that can go wrong. As such, different database systems focus on different strengths: some strive for strong guarantees, others on always being available, depite of being erroneous even some (or even most) of the time. Furthermore, what a database system <em>claims</em> to achieve when problems occur is rarely what it <em>actually</em> copes with, as <a href="http://aphyr.com/">Kyle Kingsbury</a> explores in his excellent series on the <a href="http://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-and-the-perils-of-network-partitions">perils of network partitions</a>. In short, he finds that while the distributed database works fine on a sunny day, most struggle when subjected to the <a href="http://aphyr.com/posts/288-the-network-is-reliable">vast amount of possible ways to fail</a>.</p><p>In terms of consistency, availability and partition tolerance, Elasticsearch is a CP-system, for a fairly weak definition of "consistent". If you have a read-only workload, Elasticsearch lets you achieve AP-behaviour by having a relaxed "minimum master nodes"-requirement, i.e. not requiring a quorum. Generally, however, you will need the majority of nodes in the cluster to be available. Writing to a misconfigured cluster without this majority, i.e. cluster with a "split brain", can result in irrecoverable dataloss. This is by no means specific to Elasticsearch.</p><p>In terms of scaling, an index is divided into one or more shards. This is specified when the index is created and cannot be changed. Thus, an index should be sharded proportionally with the anticipated growth. As more nodes are added to an Elasticsearch cluster, it does a good job at reallocating and moving shards around. As such, Elasticsearch is very easy to scale out.</p></section><section id="security" class="level2"><h2><a href="#security">Security</a></h2><p>See also: <a href="/security">Elasticsearch in Production, Security</a>.</p><p>Elasticsearch does not have any features for authentication or authorization. You should consider anyone who can connect to your Elasticsearch cluster to have "super user" rights, especially if Elasticsearch's powerful scripting capabilities are enabled.</p></section><section id="summary" class="level2"><h2><a href="#summary">Summary</a></h2><p>It is certainly possible to use Elasticsearch as a primary store, when the limitations described are not showstoppers. One good example is when using <a href="/logstash">Logstash</a>. Logstash is a fantastic tool for managing logs and shoving them into Elasticsearch, perhaps also archiving them somewhere else just in case. Logs are write once, read many. No updating, no need for transactions, integrity constraints, etc.</p><p>What about systems like Postgres, that come with <a href="http://www.postgresql.org/docs/current/static/textsearch.html">full-text search</a> <em>and</em> ACID-transactions? (Other examples are the full-text capabilities of MySQL, MongoDB, Riak, etc.) While you can implement basic search with Postgres, there's a huge gap both in possible performance, and in the features. As mentioned in the section on <a href="#transactions">transactions</a>, Elasticsearch can "cheat" and do a lot of caching, with no concern for multi version concurrency control and other complicating things. Search is also more than finding a keyword in a piece of text: it's about applying domain specific knowledge to implement good relevancy models, giving an overview of the entire result space, and doing things like spell checking and autocompletion. All while being <em>fast</em>.</p><p>Elasticsearch is commonly used <em>in addition</em> to another database. A database system with stronger focus on constraints, correctness and robustness, and on being readily and transactionally updatable, has the master record - which is then asynchronously pushed to Elasticsearch. (Or pulled, if you use one of Elasticsearch's "rivers".) Keeping things in sync is something we'll cover in depth in a future article. Here at Found, we typically use PostgreSQL and ZooKeeper as keeper of truths, which we feed into Elasticsearch for awesome searching.</p><p>Like with everything else, there's no silver bullet, no one database to rule them all. That's likely to always be the case, so know the strengths and weaknesses of your stores!</p><div class="references"><h2><a href="/">References</a></h2><p>Banon, Shay: <em>The future of compass &amp; elasticSearch</em> – <a href="https://thedudeabides.com/articles/the_future_of_compass">https://thedudeabides.com/articles/the_future_of_compass</a></p></div></section><section class="footnotes"><hr/><ol><li id="fn1">Shay Banon, <em>The future of compass &amp; elasticSearch</em> – <a href="https://thedudeabides.com/articles/the_future_of_compass">https://thedudeabides.com/articles/the_future_of_compass</a>.<a href="#fnref1">↩</a></li></ol></section>

Elasticsearch as a NoSQL Database

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>Leader election is one of the most tricky things to do in distributed systems. At same time, understanding how a leader is elected and the responsibilities of the leader is key to understanding a distributed system.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>All distributed systems have to deal with maintaining consistency in one way or another. To simplify, we may divide the strategies into two groups. The ones that attempt to avoid inconsistencies from occurring and the ones that define how to reconcile them. The latter is very powerful for applicable problems, but imposes unacceptable constraints on the data model in others. In this article we will look into one example of the first category and how it copes with network failures.
</p>
</section><section id="why-use-a-leader" class="level2">
<h2><a href="#why-use-a-leader">Why Use a Leader?</a></h2>
<p>Having a node designated as a leader in a distributed system is actually quite similar to the keyword synchronized in Java. The classic example for Java synchronized is when two threads attempt to increment the same integer. For instance, when two threads attempt to deposit $100 each to the same account. Each thread has to read the balance, increment it and write it back. Given a starting balance of $100 and no thread synchronization the following may occur:
</p>
<ul>
	<li>Thread A reads balance ($100)</li>
	<li>Thread B reads balance ($100)</li>
	<li>Thread A calculates new balance ($200)</li>
	<li>Thread B calculates new balance ($200)</li>
	<li>Thread A writes new balance ($200)</li>
	<li>Thread B writes new balance ($200)</li>
</ul>
<p>Clearly the account started with $100, a total of $200 was deposited, and the final balance should have been $300. However, thread A&#x2019;s deposit is overwritten as thread B has done its calculation on data that is stale by the time it writes its updated balance.
</p>
<p>The solution in Java is of course to use the keyword synchronized on the code doing the incrementation and it will make one thread finish its work with the balance before the other gets to read the balance.
</p>
<p>If you consider a thread as a node, it&#x2019;s easy to imagine the same problem in a distributed system, but there is no synchronized keyword to the rescue. The reason being that synchronized is implemented using semaphores, and in turn, they are backed by the underlying operating system and the hardware it&#x2019;s running on. But if we look at what synchronized does, we might adapt the concept to a distributed setting. When declaring a block of code as synchronized in Java, it is always synchronized to the monitor of a specific object. The size of this monitor is one. This means that only one thread may be inside the monitor at any given time. When a thread requests the monitor and the monitor is available, it is allowed to enter instantly. If the monitor is occupied, the thread is put in the waiting pool and suspended until the monitor is released.
</p>
<p>How can we adapt this to a distributed setting? Well, we do have the option to create as many monitors as we want on each node, but for a given resource that requires protection there may only exist one monitor. In other words, there may only be one monitor for each account balance. This transforms the problem of maintaining data consistency to the problem of ensuring all nodes agree on which node is responsible for implementing the monitor for each balance. In our simple example this could be handled by the leader, and all that remains to be done is to elect a leader.
</p>
<p>If you have a peer to peer background like myself you would probably suggest using a distributed hash table rather than a leader to decide which node implements the monitor for each resource. In fact, there are some really great DHT-implementations out there that deal with thousands of nodes leaving and joining every hour. Given that they also work in a heterogenous network with no prior knowledge of underlying network topology, query response times of four to ten message hops is pretty good, but in a grid setting where the nodes are homogenous and the network is fairly stable, we can do better.
</p>
<p>Another simplification in the typical scenario for Elasticsearch is that there are not that many nodes in the cluster. Typically, the number of nodes is far less than the number of connections a single node is capable of maintaining, and a grid environment does not have to deal with nodes joining and leaving that frequently. This is why the leader approach is a better fit for Elasticsearch.
</p>
</section><section id="the-zen-way" class="level2">
<h2><a href="#the-zen-way">The Zen Way</a></h2>
<p>So we have a leader that regularly pushes the current version of the global cluster state to each node. What happens if the leader goes down? Just because our nodes are fairly reliable does not imply that we can accept a single point of failure. Provided that only the leader node has crashed and all other nodes are still capable of communicating, Elasticsearch will handle this gracefully as the remaining nodes will detect the failure of the leader - or rather the absence of messages from the leader - and initiate leader election. If you are using ZenDiscovery (default) then the process is like this:
</p>
<ul>
	<li>Each node calculates the lowest known node id and sends a vote for leadership to this node</li>
	<li>If a node receives <em>sufficiently many votes</em> and the node also voted for itself, then it takes on the role of leader and starts publishing cluster state.</li>
</ul>
<p>The definition of <em>sufficiently many votes</em> to win an election is what is known as a quorum. In Elasticsearch the quorum size is a configurable parameter. A node in a distributed system is not able to determine whether another node is dead or whether the network is not able to deliver its messages to it, it is therefore common to have a previously agreed upon threshold - the quorum size - to represent the other party&#x2019;s votes. Let&#x2019;s exemplify with the following scenario: A cluster has nodes A, B, C, D, E and quorum size 3. A is the leader. It so happens that A and B is on one network and C, D and E on another. The networks are connected through a link.
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4c9b9985d58976376ca5/download?uid=blt74a77a580f55964a?uid=blt74a77a580f55964a" data-sys-asset-uid="blt8bdce1626de819c6" alt="Initial network" style="max-width:100%;"><figcaption>Initial network</figcaption>
</figure>
<p>When this link fails, A and B are still able to communicate with each other, but not with C, D and E. Similarly, on the other network C, D and E may communicate with each other, but not with A and B.
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4c9b58208ba076e282c2/download?uid=bltaa821b8fd8a647ed?uid=bltaa821b8fd8a647ed" data-sys-asset-uid="blta2af94513dd52d7c" alt="Partitioned network" style="max-width:100%;"><figcaption>Partitioned network</figcaption>
</figure>
<p>What happens next is this: nodes C, D and E detect that they no longer have contact with the leader A and subsequently initiate leader election by sending votes to C. Once C has received three votes it takes on the role of leader and starts publishing to C, D and E. On the other network, the leader A detects that it no longer has contact with nodes C, D and E. Leader A calculates that the new cluster size is less than the quorum size and gives up the leader role and effectively stops nodes A and B from responding to queries until the link is restored.
</p>
<p>In real life it&#x2019;s unlikely that someone trips over on a crucial network cable, but networks are more complex than my example above, and network partitions are actually not that uncommon. It is not hard to imagine a network split when a system is relying on multiple datacenters; another likely culprit of tricky network errors is a wrongly configured router or switch. As mentioned in <a href="http://aphyr.com/posts/288-the-network-is-reliable">the network is reliable</a>, it does occur that network cards do things like dropping all inbound packets while still delivering outbound packets, with the result that a server still sends heartbeats but is unable to service any requests.
</p>
</section><section id="avoid-split-brain" class="level2">
<h2><a href="#avoid-split-brain">Avoid Split Brain</a></h2>
<p>The concept of a quorum size has two immediate advantages. Firstly it simplifies the election process as votes only need to be delivered to the node they&#x2019;re voting for. Secondly and more importantly it is the only way to avoid a split brain. Imagine the same cluster and the same network split, but with a quorum size of two. C would still be elected leader, but A would not give up its leader role. This would result in two leaders, with clusters unknown to each other. This is what is known as a split brain. Both clusters would be accepting read and write operations, and as expected, they would be out of sync. Without human intervention they would probably never recover. Depending on the data model it might be impossible to unify the two data versions, forcing one to simply discard all data on one of the clusters.
</p>
</section><section id="dont-reinvent-the-wheel" class="level2">
<h2><a href="#dont-reinvent-the-wheel">Don&#x2019;t Reinvent the Wheel</a></h2>
<p>As you might expect, leader election has been an intriguing topic in academic circles for many years and quite a few smart people have done a great deal of contemplation. If you are more keen on getting your distributed system working than putting a massive effort into research and development, you&#x2019;re probably better off with having a go at implementing a well known algorithm. Some day in the future when you have finally figured out that crazy bug in your system, it&#x2019;s more likely to be just a bug in the implementation than a major design flaw in the algorithm. Of course, the same argument applies to adapting or integrating an existing implementation rather than implementing from scratch, especially if you want one of the more advanced algorithms. <a href="https://github.com/elasticsearch/elasticsearch/issues/2488">This bug report</a> illustrates how tricky it can be to create a good leader election implementation, even when you have a sound algorithm.
</p>
<section id="the-bully-algorithm" class="level3">
<h3><a href="#the-bully-algorithm">The Bully Algorithm</a></h3>
<p>The bully algorithm is one of the basic algorithms for leader election. It assumes that all nodes are given a unique ID that imposes a total ordering of the nodes. The current leader at any time is the node with the highest id participating in the cluster. The advantage of this algorithm is an easy implementation, but it does not cope well with a scenario whereby the node with the largest id is flaky. Especially in a situation where the node with the largest id tends to be overloaded by the chores of the leader role. Consequently, it will crash as the leader; the node with the second largest id will be elected; and the largest id node will recover - as it&#x2019;s no longer overloaded - and subsequently initiate leader election again, only to be elected and crash yet again. However, in low level hardware interfaces the bully algorithm is quite common. It might be tempting to avoid thrashing by postponing election until the current leader fails, but that will easily lead to a split brain.
</p>
</section>
<section id="paxos" class="level3">
<h3><a href="#paxos">Paxos</a></h3>
<p>Paxos is actually a much more than a leader election algorithm. Paxos is a family of different protocols for maintaining consensus in a distributed system. I will not go into detail about the varieties of Paxos, but rather discuss the concepts and qualities of Paxos in general. The data model used in Paxos is a growing list of statements where each statement has a unique number. The mathematically proven guarantees of Paxos are that two nodes will never have different statements for the same number, and as long as there is a quorum of nodes participating the algorithm will make progress. This translates to the nodes never being inconsistent and the algorithm never going into deadlock, but it may halt if there are not enough nodes online. These guarantees are in fact quite similar to what we want for a leader election algorithm. We want all nodes participating in the cluster to agree on which node is the leader, and we do not want any other nodes to be able to run off and create their own cluster, resulting in a split brain. Performing leader election with Paxos then becomes as simple as proposing the statement &#x201C;I am the leader&#x201D;. If it is accepted, then you are the leader until another node proposes to be the leader. However, this alone is not sufficient to avoid a split brain. In the partitioned network example above the existing leader A will not be notified of the election of Node C on the other switch. There needs to be another criteria to end the leadership. One option is that when initiating leader election the statement used is on the form: &#x201C;I am leader until <some criteria="">&#x201D;, but a more preferred option is that the leader is able to detect that it no longer has contact with a quorum and then demotes itself.</some>
</p>
<p>Given the previous example of a network partition, let&#x2019;s imagine that a janitor trips on the network cable to node C and then reconnects the cable to the other switch, resulting in a new partition of the network:
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4c9b58208ba076e282c0/download?uid=blt2233ba1c8a54fe38?uid=blt2233ba1c8a54fe38" data-sys-asset-uid="blt205a7b37646ce0f7" alt="Moved node" style="max-width:100%;"><figcaption>Moved node</figcaption>
</figure>
<p>The interesting part here is that the quorum now consists of nodes A, B and C instead of the previous C, D and E. If the nodes where running Paxos, their data could look something like this:
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4c9c0342dfd738263371/download?uid=blte70b54df709027b8?uid=blte70b54df709027b8" data-sys-asset-uid="blt62f71e91c7af6e1f" alt="Paxos example" style="max-width:100%;"><figcaption>Paxos example</figcaption>
</figure>
<p>Depending on the exact implementation of leader election through Paxos, a number of outcomes are possible, but they all have two properties in common. Nodes D and E will not be able to elect a new leader as they do not have a majority. Nodes A and B will not elect a new leader before learning that node C was elected in round 2. Assuming that the end criterion of each leader term is a timeout, there are two possible outcomes. If node C establishes contact with nodes A and B before its term is ended, it will reestablish a quorum, resume its leader role, and no leader election is necessary. If its term is ended, then either node A, B or C may be elected. Assuming node A is the first to discover node C on the network, it will try to get re-elected as leader for term 2, but node C will reject this as it already has a leader for term 2. Node A then has the option to attempt to be elected for term 3. If it does, it will then also inform node B about node C&#x2019;s time as leader in term 2. Now, what if node C and A try to get elected at the same time? The full answer to that question requires delving into the specifics of Paxos, which out of scope for this article, but the short answer is that election will fail. In fact, the original paxos paper suggests using a leader election algorithm to decide which node should be the one initiating new proposals, but since the Paxos protocol guarantees that there will not be inconsistencies even if there are multiple nodes proposing conflicting statements, this leader election algorithm could be just a simple heuristic to avoid consecutive failed rounds. For instance, a node might decide to wait a randomized amount of time before reattempting to propose a failed statement. This flexibility of Paxos in terms of when and how to do leader election can be a great advantage over the simple bully algorithm, having in mind that in real life there are far more failure modes than a dead network link.
</p>
</section>
</section><section id="conclusion" class="level2">
<h2><a href="#conclusion">Conclusion</a></h2>
<p>In this article I have tried to give a brief insight into some of the work that has gone into leader election in academic circles and the importance of doing it right. The condensed version is this:
</p>
<ul>
	<li>Quorum size is really important if you care about your data.</li>
	<li>Paxos is really robust, but not as easy to implement.</li>
	<li>It&#x2019;s better to implement a well known algorithm where the strengths and flaws are known rather than getting a big surprise further down the road. There are certainly plenty of options out there.</li>
</ul>
</section></body></html>

Leader Election, Why Should I Care?

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>The Missing Document on Search Engine Indexing
</p><section id="what-is-an-index" class="level2">
<h2><a href="#what-is-an-index">What is an Index?</a></h2>
<blockquote>
	&#x201C;If you don&#x2019;t find it in the index, look very carefully through the entire catalog.&#x201D; &#x2013; Sears, Roebuck, and Co., Consumers&#x2019; Guide 1897
</blockquote>
<p>The concept of an index is pretty straightforward. I&#x2019;m sure you are familiar with indexes from textbooks, so we will use this as a point of reference for the following introductory explanation.
</p>
<p>Here&#x2019;s an everyday scenario: you want to make tomato soup but don&#x2019;t know how. So you pick up that dusty cookbook you were given for Christmas. But hey, it happens to be a bible size cookbook covering all things foody. Surely you don&#x2019;t want to start on page 1 and read through the majority of the cookbook to find out how to make tomato soup, right? Luckily for you, some clever person in the 16th century came up with the headsmacking idea of indexing books!
</p>
<p>Here is a sample of what a cookbook index might look like:
</p>
<pre>&lt;code&gt;Tartar, beef tartar&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;page 67
Tomato chutney&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;page 645
Tomato soup&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;page 23, 78
Umami burger&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;&#x2026;page 378&lt;/code&gt;
</pre>
<p>An index in a textbook is basically a mapping between words or phrases in the book, for instance &#x201C;tomato soup&#x201D;, and the page or pages where you can find the word or phrase. By convention and most widely used is the back-of-the-book index, sorted alphabetically. Evidently, this makes it quick and easy to look up a word or phrase and subsequently locate the mapped content.
</p>
<p>Indexing by search engines works pretty much the same way. A search engine creates mappings between terms, in the above example recipe names, and occurrences of the terms, in the above example the pages in the cookbook where you can find the recipes.
</p>
<p>Just as it&#x2019;s a tedious and cumbersome task to read through an entire cookbook to find a specific recipe, it is equally inefficient for a computer to scan through a pile of documents or a database every time it is asked to find all occurrences of a term. For large databases or mountains of documents, this process could take several minutes, or even hours. Indexing has therefore been introduced to speed up the search process, or information retrieval (IR) process, as it is generally called.
</p>
<p>Somewhat akin to the alphabetical sorting of the index in a cookbook, a search engine&#x2019;s index is stored using specific formats - or data structures - to make it fast to locate and find the search term and their mappings. Moreover, storing the index in memory (RAM) is a sure-fire way to speed up the lookup process. Imagine if you could memorize all the indexes of your entire stack of textbooks!
</p>
<p>Next: <a href="/blog/found-indexing-for-beginners-part2/" title="/blog/found-indexing-for-beginners-part2">Document Parsing and Tokenization</a>
</p>
</section></body></html>

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

<

2013

<

may

>

Elasticsearch 0.90.1 Released

All About Elasticsearch Filter BitSets

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS