Data Visualization with ElasticSearch and Protovis

<html><head></head><body>elasticsearch version <code>0.16.1</code> has just been released. You can download it <a href="/download">here</a>. This is a minor release but it includes some really cool features and enhancements:
<h2>Text Query Family</h2>
A new family of queries called <a href="/guide/reference/query-dsl/text-query.html">text</a>. The aim with those is to simplify querying elasticsearch with regards to the analysis process (and should hopefully remove some confusion when using <code>term</code> query).
<h2>More Analysis Options</h2>
All the analyzers (mainly around language analyzers) part of Lucene <code>3.1</code> are now exposed. On top of that, several token filters were backported from Lucene 4.0 (synonym and word_delimiter).
<h2>Other</h2>
Minor additional enhancements (better global facets execution, custom boost per value for <code>string</code> type, and more) and important bug fixes are part of this release. Upgrade is highly recommended.
</body></html>

0.16.1 Released

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Load More

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Author

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Articles by

Share this story

Share by Email

Read less

Search Integrations

All Solutions

Thank you for registering. We will send you a confirmation email soon.

Thank you for your interest!

Follow us on Youtube

Follow us on Twitter

Follow us on LinkedIn

Follow us on Facebook

Headshot of

Reset all

Filters

Global Virtual Event

View more posts

Print

Continue reading

Share on Facebook

Share on LinkedIn

Share on Twitter

Share

Small image for

Video for

Explore similar demos

Register now

View next

Upcoming webinar

On-demand webinar

Featured webinar

Highlights

See when this webinar starts in my time zone

Related workshops

Hosted by

Agenda

Location

Date

More

View more learning opportunities

Load more press releases

Load more news

What to explore next...

See All Posts

Contact Info

Register to Watch

Sign In to Attend

Register to Attend

0.18.6 Released

<html><head></head><body>elasticsearch version <code>0.18.5</code> is out. You can download it <a href="/download">here</a>. It includes an upgraded Lucene version (3.5), featuring bug fixes and memory improvements, as well as more bug fixes in elasticsearch itself. Changes can be found <a href="https://github.com/elasticsearch/elasticsearch/issues?labels=v0.18.5&amp;sort=created&amp;direction=desc&amp;state=closed&amp;page=1">here</a>.
</body></html>

0.18.5 Released

<html><head></head><body>elasticsearch version <code>0.18.3</code> has just been released. You can download it <a href="/download">here</a>. Its an important bug fix release, including a fix for a <a href="https://github.com/elasticsearch/elasticsearch/issues/1455">major bug</a> and upgrade is highly recommended. An <code>0.17.10</code> version with the bug fixed has also been released.
The release also includes additional bug fixes, and some features including <a href="https://github.com/elasticsearch/elasticsearch/issues/1462">shard level slow search log</a>, and improved <a href="https://github.com/elasticsearch/elasticsearch/issues/1463">upper inclusive range</a> handling for dates.
[Update]: Heads up that this release includes a broken plugin script, more info <a href="https://github.com/elasticsearch/elasticsearch/issues/1474">here</a>.
[Update 2]: <code>0.18.4</code> has been released fixing the plugin script.
-shay.banon
</body></html>

0.18.3 Released

<html><head></head><body>elasticsearch version <code>0.18.0</code> has just been released. You can download it <a href="/download">here</a>. This is a major release, and includes the following major features:
UPDATE: <code>0.18.2</code> was quickly released to fix a bug with custom <code>path.data</code> location configuration and fetching inner fields from a source doc.
<h2>Shard Allocation Awareness and Filtering</h2>
Shard allocation within a cluster can now be &#x201C;aware&#x201D; of which node replicas are allocated on (similar to rack awareness). It allows to make sure that a shard and a replica will be allocated across logical grouping of nodes. The awareness can also be forced, by not over allocating replicas within the same logical node group. Search and Get operations will automatically prefer shards that are allocated within the same logical node group the search is executing on.
Filtering allows to explicitly control which nodes indices are allowed, or not allowed, to be allocated on, again, based on custom logical grouping of nodes (called node attributes).
Both settings can be updated in realtime using either the cluster wide or index level update settings API. More info can be found <a href="/guide/reference/modules/cluster.html">here</a>.
<h2>Cluster Update Settings API</h2>
The cluster update settings API allow to update node level settings (and not index level settings) in realtime on a live cluster. The settings allowed and more docs on the API are provided <a href="/guide/reference/api/admin-cluster-update-settings.html">here</a>.
<h2>Timestamp and TTL</h2>
A document can now automatically be indexed with a timestamp associated with it, and have a TTL (time to live), which, when expired, will cause the document to be deleted. More info can be found <a href="/guide/reference/mapping/timestamp-field.html">here</a> and <a href="/guide/reference/mapping/ttl-field.html">here</a>.
<h2>Improved Geo Execution</h2>
Geo distance filter and facet good a considerable performance boost by doing bounding box optimization. Also, an option to use indexed lat lon for the checks (must be enabled in the <code>geo_type</code> mapping) is provided which can provide even faster executing under certain conditions (compared to in memory checks).
<h2>More Statistics</h2>
More statistics are now provided out of the box, including a new index stats API (the status API should not be used for stats anymore). Statistics for APIs such as index, get, and search are also being aggregated, with search allowing to specific specific stats grouping aggregation (to further distinguish search &#x201C;types&#x201D;).
<h2>Multi Data locations</h2>
A node can now work with multiple data locations, spreading the index files across those locations in a RAID 0 like behavior.
<h2>Smaller improvements and bug fixes</h2>
Many more bug fixes and smaller improvement have went into this release. Full release notes can be found <a href="https://github.com/elasticsearch/elasticsearch/issues?sort=created&amp;direction=desc&amp;labels=v0.18.0&amp;state=closed">here</a>.
-shay.banon
</body></html>

0.18.0 Released

<html><head></head><body>elasticsearch version <code>0.17.8</code> has just been released. You can download it <a href="/download">here</a>. The release includes major bug fixes listed <a href="https://github.com/elasticsearch/elasticsearch/issues?labels=v0.17.8&amp;sort=created&amp;direction=desc&amp;state=closed">here</a>.
</body></html>

0.17.8 Released

<html><head></head><body>elasticsearch version <code>0.17.7</code> has just been released. You can download it <a href="/download">here</a>.
This release include the usual list of bug fixes, and also include an upgrade to Lucene 3.4.0 (fixes critical bugs, so make sure you upgrade), as well as improvements to the couchdb river (memory usage wise).
</body></html>

0.17.7 Released

<html><head></head><body>elasticsearch version <code>0.17.6</code> has just been released. You can download it <a href="/download">here</a>. This is is a quick release to fix a bug in automatic date detection when provided as <code>yyyy/MM/dd</code> or <code>yyyy/MM/dd HH:mm:ss</code>.
</body></html>

0.17.6 Released

<html><head></head><body>elasticsearch version <code>0.17.5</code> has just been released. You can download it <a href="/download">here</a>. This is a bug fix release and minor feature enhancements. Upgrade is highly recommended.
</body></html>

0.17.5 Released

<html><head></head><body>elasticsearch version <code>0.17.4</code> has just been released. You can download it <a href="/download">here</a>.
This is a quick release post <code>0.17.4</code> to fix a reported bug which probably doesn&#x2019;t affect most users, but its still an important one to get out there (for new users) as its a difficult one to track down&#x2026; . The upgrade is only really needed when using explicit stored fields that have multi values, which can result in not all the values being returned when asked.
</body></html>

0.17.4 Released

<html><head></head><body>elasticsearch version <code>0.17.3</code> has just been released. You can download it <a href="/download">here</a>. This is another bug fix release with minor enhancements (listed <a href="https://github.com/elasticsearch/elasticsearch/issues?sort=created&amp;labels=v0.17.3&amp;direction=desc&amp;state=closed">here</a>). This upgrade is highly recommended to all <code>0.17.x</code> users.
Note, when upgrading to this release, due to a bug in previous 0.17.x versions, it is recommended to run flush against all indices before shutting down the cluster (or doing a rolling restart) if &#x201C;delete_by_query&#x201D; is used against specific types.
</body></html>

0.17.3 Released

<html><head></head><body>I created the mailing list on a google apps account (paying one&#x2026;) with the hopes of not running into spam problems down the road. The problem with that is the fact that I did not know about the host of problems I will have with google account one&#x2026; . The latest one is a major problem, where google apps restrict the size of members to 1000 members, and we just hit that.
So, I am going to migrate the group to a new group in the &#x201C;public&#x201D; google groups, called <a href="https://groups.google.com/group/elasticsearch/">elasticsearch</a>. Sadly, I don&#x2019;t think migrating content is possible, but at least I will be able to subscribe all of you. I will automatically subscribe all under a &#x201C;Send email for each message and update&#x201D; option, so you will have to change that to your preference (sorry about that&#x2026;). Update: Google Groups seems to have spam limits on the number of users that can be added, will add them in batches over time, I guess&#x2026; . Make sure to <a href="https://groups.google.com/group/elasticsearch/">register</a> now if you can.
Luckily, the group has been mirrored on nabble, and it will remain as a mirror to the new group as well. This group will be closed (but not deleted) and old content will be searchable from both nabble and this interface, posting will not be allowed though. And posts should only go to the new group mailing address. I will post details on the new group shortly, but you should get it as well via direct mail once I invite you.
I apologize for that. If someone knows of a better way to migrate the mails, it would be great.
-shay.banon
</body></html>

Mailing List Migration

<html><head></head><body>elasticsearch version <code>0.17.2</code> and <code>0.16.5</code> have just been released. You can download it <a href="/download">here</a>.
A major concurrency bug has been fixed when executing search requests, which is the reason for the 0.16.5 release and also included in the <code>0.17.2</code> release.
The <code>0.17.2</code> release includes mainly bug fixes, and its highly recommended to upgrade to it if <code>0.17.x</code> is used.
</body></html>

0.17.2 / 0.16.5 Released

<html><head></head><body>elasticsearch version <code>0.17.1</code> has just been released. You can download it <a href="/download">here</a>. This is a quick bug fix release fixing major bugs (listed <a href="https://github.com/elasticsearch/elasticsearch/issues?sort=created&amp;labels=v0.17.1&amp;direction=desc&amp;state=closed">here</a>).
</body></html>

0.17.1 Released

<html><head></head><body><article>
			elasticsearch version <code>0.17.0</code> has just been released. You can download it <a href="/download">here</a>. This is a major release, and includes the following major features:
<h2>Improved Aliases &#x2013; Filtering and Routing</h2>
<a href="/guide/reference/api/admin-indices-aliases.html">Index aliases</a>, which are very handy in aliasing a custom name to an index (or indices) and changing it on the fly, now support the ability to associate a search filter which will automatically filter all the search requests when using the alias, and a routing value (will automatically be used when hitting the alias).
This can come in handy in several scenarios, for example, a single index holding multi user data can have a username as the alias, associate a term filter that filters the results based on the username, and possibly use the username as the routing value (thus hitting less shards when searching).
<h2>Realtime GET</h2>
The <a href="/guide/reference/api/get.html">get</a> API is now fully realtime. This means that its not affected by the scheduled refresh / visibility of changes done to the index for search, and once a document is indexed / deleted, it will be immediately visible for GET.
Also, a new <a href="/guide/reference/api/multi-get.html">multi get</a> API has been added to execute multi document get in an optimized manner.
Note, the realtime get feature does not require &#x201C;refreshing&#x201D; the Lucene index, and does not come with the overhead associated with it.
<h2>Nested objects / docs</h2>
Nested objects / documents are now supported, allowing for a document to be broken down into root document and nested document (mainly make sense for multi valued inner objects). This allows for better searchability across those nested documents, as well as better faceting.
The <a href="/guide/reference/mapping/nested-type.html">nested</a> mapping allows to control which inner objects will be nested. Searching &#x201C;within&#x201D; nested docs is done using nested <a href="/guide/reference/query-dsl/nested-query.html">query</a> and <a href="/guide/reference/query-dsl/nested-filter.html">filter</a>. Faceting is explained in the <a href="/guide/en/elasticsearch/reference/current/search-facets.html">faceting</a> section.
Note, the fact that those nested documents are &#x201C;within&#x201D; the json document indexed allows to index them within the same &#x201C;block&#x201D;, resulting in extremely fast &#x201C;join&#x201D; implementation (compared to parent child mapping).
<h2>Plugins &amp; Site Plugins</h2>
The <a href="/guide/reference/modules/plugins.html">plugins</a> module has been enhanced to support downloading plugins from github. Plugins can now also serve static sites, for example, installing two commonly used web admins for elasticsearch is as simple as:
<pre class="dontquote prettyprint">bin/plugin -install mobz/elasticsearch-head bin/plugin -install lukas-vlcek/bigdesk</pre>
<h2>Many Indices / Large Cluster</h2>
Numerous improvements to large clusters and many indices scenarios, especially around memory utilization and full cluster restart times.
<h2>Lucene 3.3</h2>
An upgrade to Lucene 3.3 version bringing bug fixes and a new <a href="/guide/reference/index-modules/merge.html">tiered</a> based segment merging.
<h2>Nodes APIs</h2>
All nodes level APIs now support executing the APIs against nodes not only based on their IDs. For example:
<pre class="dontquote prettyprint"># Address curl localhost:9200/_cluster/nodes/10.0.0.3,10.0.0.4 curl localhost:9200/_cluster/nodes/10.0.0.* # Names curl localhost:9200/_cluster/nodes/node_name_goes_here curl localhost:9200/_cluster/nodes/node_name_goes_* # Attributes (set something like node.rack: 2 in the config) curl localhost:9200/_cluster/nodes/rack:2 curl localhost:9200/_cluster/nodes/ra*:2 curl localhost:9200/_cluster/nodes/ra*:2*</pre>
<h2>EC2 specific network host settings</h2>
Specific ec2 settings for host names when the <code>cloud-aws</code> plugin is installed. For example, setting <code>network.host</code> to <code>_ec2:privateIp_</code> will automatically use the configured instance private ip address to bind and publish to.
<h2>Small(er) improvements and bug fixes</h2>
As usual, many more smaller features (more analyzers, more customization) and bug fixes are in this release.
enjoy! 
-shay.banon
			<aside id="post_tags">
							</aside>
		</article></body></html>

0.17.0 Released

<html><head></head><body>elasticsearch version <code>0.16.2</code> has just been released. You can download it <a href="/download">here</a>. The bulk of the work on this release (backported from master) revolves around improved memory usage, especially with the filter cache (there is a new default filter cache, which takes the size in bytes as a limit on works on the node level).
As usual, bug fixes and other minor improvements. Upgrade is highly recommended.
</body></html>

0.16.2 Released

<html><head></head><body>elasticsearch version <code>0.16.0</code> has just been released. You can download it <a href="/download">here</a>. This release is quite a big and important one, adding many new features, and fixes several major/critical bugs.
Features/enhancements include the ability to update many more index level settings at runtime, several new search types, new facet called term_stats, faster facets execution all around, upgrade to Lucene 3.1, improved support for many indices (memory wise), faster recover post full restart, much improved shard allocation logic to reduce load, and many more.
Many bugs have also been fix, with several critical ones revolving around possible data loss. Its highly recommended to upgrade.
All changes are listed in the <a href="/download/2011/04/23/0.16.0.html">0.16.0 download page</a>.
<h1>Upgrade Notes</h1>
The <code>_id</code> field is no longer indexed by default. No functionality is lost by it, but, if the <code>_id</code> field was used in term/terms query/filters, it should be replaced with the new ids query/filter. The issue explaining it is <a href="https://github.com/elasticsearch/elasticsearch/issues/868">here</a>.
</body></html>

0.16.0 Released

<html><head></head><body>elasticsearch has had the ability to control the <a href="/guide/reference/api/search/search-type.html">search type</a> since its inception. It includes the ability to control how to execute the &#x201C;query&#x201D; and &#x201C;fetch&#x201D; phases of a distributed search execution, as well as the ability to compute distributed frequencies across shards (the &#x201C;dfs&#x201D; phase).
Two things that many users have been asking for is the ability to easily iterate over a very large result set as fast as possible, and the ability to just count results (possibly with facets) without actually fetching any hits back. Both features are now supported as new search types.
<h1>count</h1>
The first is the <code>count</code> search type. It allows to get back the total number of hits matching a query, with the ability to have facets configured in an optimized (implementation wise and performance wise) manner. For example:
<pre class="dontquote prettyprint">curl -XGET &apos;http://localhost:9200/twitter/tweet/_search?search_type=count&apos; -d &apos;{
 &quot;query&quot;: {
 &quot;filtered&quot; : {
 &quot;query&quot; : {
 &quot;query_string&quot; : {
 &quot;query&quot; : &quot;some query string here&quot;
 }
 },
 &quot;filter&quot; : {
 &quot;term&quot; : { &quot;user&quot; : &quot;kimchy&quot; }
 }
 }
 }
}
&apos;
</pre>
The result will not include any hits, just the <code>total_hits</code> and optional facets results.
<h1>scan</h1>
The new <code>scan</code> type allows to scroll a very large result set in an optimized manner. When scrolling large result set using one of the other search types, there is an overhead (that increases the more we scroll &#x201C;into&#x201D; the result set) that comes from the fact that sorting needs to be computed (either by score, or custom). The <code>scan</code> type does no sorting, allowing it to optimize the scrolling process. It uses the same scroll mechanism when using other search types.
The initial search execution bootstraps the scanning process:
<pre class="dontquote prettyprint">curl -XGET &apos;localhost:9200/_search?search_type=scan&amp;scroll=10m&amp;size=50&apos; -d &apos;
{
 &quot;query&quot; : {
 &quot;match_all&quot; : {}
 }
}
&apos;
</pre>
The <code>scroll</code> parameter indicates two things, the fact that we wish to scroll further into the result set, and the timeout value for maintaing the scroll &#x201C;open&#x201D; (the timeout value applies per request, not globally across the scroll process).
The <code>size</code> parameter controls how many hits we want to get back. Note, the actual number of hits will be the the size provided times the number of shards, which is done in order to further optimize the scrolling process.
The result of the search request is a <code>scroll_id</code>. The <code>scroll_id</code> should then be used as a parameter to the next search request:
<pre class="dontquote prettyprint">curl -XGET &apos;localhost:9200/_search/scroll?scroll=10m&apos; -d &apos;c2NhbjsxOjBLMzdpWEtqU2IyZHlmVURPeFJOZnc7MzowSzM3aVhLalNiMmR5ZlVET3hSTmZ3OzU6MEszN2lYS2pTYjJkeWZVRE94Uk5mdzsyOjBLMzdpWEtqU2IyZHlmVURPeFJOZnc7NDowSzM3aVhLalNiMmR5ZlVET3hSTmZ3Ow==&apos;
</pre>
The results now will include hits, and another <code>scroll_id</code>, which should then be used as the parameter to the next scroll request. Also, the <code>scroll</code> parameter needs to be provided again in order to indicate we wish to continue the scrolling process.
The &#x201C;exit&#x201D; point from the scrolling process is when no hits are returned back.
-shay.banon</body></html>

New Search Types

<html><head></head><body>When using elasticsearch, many times the initial usage of an index in the cluster is bulk indexing data into it, and then moving into a more &#x201C;streamlined&#x201D; operations that are applied in realtime against it. When doing something like bulk indexing, changing the default settings of the index can improve the speed at which documents are indexed, but, those settings then do not really apply for the case where we want to do real time indexing of data.
Some of those settings include <code>index.refresh_interval</code>, which defaults to <code>1s</code>. Setting this value to be higher can improve indexing speed. Other settings include low level Lucene settings such as <code>index.merge.policy.merge_factor</code>.
In the upcoming 0.16 version (and already in master), there has been a lot of work going into being able to update a subset of index level settings in real time to improve that. It uses the same update settings API already exposed that allowed to change only the <code>index.number_of_replicas</code>.
For example, lets say we are going to do some bulk indexing into an index, we can simply change the relevant index settings to the following:
<pre class="dontquote prettyprint">curl -XPUT localhost:9200/test/_settings -d &apos;{
 &quot;index&quot; : {
 &quot;refresh_interval&quot; : &quot;-1&quot;,
 &quot;merge.policy.merge_factor&quot; : 30
 }
}&apos;
</pre>
The above will disable refreshing of the index completely, and change the merge factor to be 30. Once the bulk loading is done, we can get back to the default settings:
<pre class="dontquote prettyprint">curl -XPUT localhost:9200/test/_settings -d &apos;{
 &quot;index&quot; : {
 &quot;refresh_interval&quot; : &quot;1s&quot;,
 &quot;merge.policy.merge_factor&quot; : 10
 }
}&apos;
</pre>
This makes elasticsearch even more elastic, allowing to munge it to adapt to what is currently required from it. The following are a list of issues listing the currently updateable settings:
<ul>
<li><a href="https://github.com/elasticsearch/elasticsearch/issues/762">762</a></li>
<li><a href="https://github.com/elasticsearch/elasticsearch/issues/765">765</a></li>
<li><a href="https://github.com/elasticsearch/elasticsearch/issues/758">778</a></li>
<li><a href="https://github.com/elasticsearch/elasticsearch/issues/799">799</a></li>
</ul>
If you are missing some settings, ping the mailing list and we can see if they can also be updated dynamically.</body></html>

Update Settings

<html><head></head><body>elasticsearch version <code>0.15.0</code> has just been released. You can download it <a href="/download">here</a>. This is another major release that includes several major features and of course, the typical set of bug fixes and enhancements. Some of the major features include:
<h1>Versioning</h1>
Versioning got its own <a href="/blog/versioning/">blog post</a> detailing how to use the feature. It now allows for a more interesting interaction model where we can use optimistic concurrency control when creating and updating documents.
<h1>Percolator</h1>
<a href="/blog/percolator/">Percolator</a> is an interesting feature, basically turning search upside down: Instead of indexing docs and searching for them, during the indexing process docs are &#x201C;percolated&#x201D; to find out which queries match them. I can&#x2019;t wait to see all of the cool things that will be built on top of elasticsearch utilizing this feature!.
<h1>Date Histogram</h1>
One of the more powerful features of elasticsearch are <a href="/guide/en/elasticsearch/reference/current/search-facets.html">facets</a>, and one of my favorite facets is the <a href="/guide/reference/api/search/facets/histogram-facet.html">histogram</a> facet (think &quot;number of comments per month). When using it on a <code>date</code> field though, it lacks some important features.
The <a href="/guide/reference/api/search/facets/date-histogram-facet.html">date_histogram</a> facet comes to solve and enhance the regular histogram by allowing you to &#x201C;bucket&#x201D; results by uneven intervals, like months and years.
On top of that, you can provide a time zone to control exactly how the &#x201C;buckets&#x201D; of hits are defined &#x2013; something you can&#x2019;t really do on the client side.
<h1>Search Filter</h1>
The new <a href="/guide/reference/api/search/filter.html">filter</a> element to the search API will now allow you to do facet based navigation in a much simpler way. The new element filters out returned hits, but will not be applied to facet calculations. (although each facet can still have its own filters as well using the <code>facet_filter</code> element).
This allows things like navigation-by-facets to be done in a much simpler manner, while controlling which facets are affected by it and which ones still apply only to the original user-supplied query.
<h1>Many More</h1>
There are many more performance improvements, enhancements, features, and bug fixes, all listed in the <a href="/download/2011/02/18/0.15.0.html">0.15 download page</a>.
-shay.banon
</body></html>

0.15.0 Released

<html><head></head><body><article>
			Upcoming elasticsearch 0.15 has some very cool features baked into it, one of those is versioning. Each document indexed in elasticsearch is now versioned and this allow for some cool operations done on it. But first, a simple example, starting with an index request:
<pre class="dontquote prettyprint">curl -XPUT &apos;localhost:9200/twitter/tweet/1?pretty=true&apos; -d &apos;{ &#xA0; &#xA0; &quot;message&quot; : &quot;elasticsearch now has versioning support!&quot; }&apos;</pre>
This is all business as usual, the interesting bit is in the response:
<pre class="dontquote prettyprint">{ &#xA0; &quot;ok&quot; : true, &#xA0; &quot;_index&quot; : &quot;twitter&quot;, &#xA0; &quot;_type&quot; : &quot;tweet&quot;, &#xA0; &quot;_id&quot; : &quot;1&quot;, &#xA0; &quot;_version&quot; : 1 }</pre>
Notice the <code>_version</code> returned when performing an index operation. This is now the version associated with this <code>tweet</code> registered with id <code>1</code>. If we do a get, we will get the version as well:
<pre class="dontquote prettyprint">curl -XGET &apos;localhost:9200/twitter/tweet/1?pretty=true&apos;</pre>
<pre class="dontquote prettyprint">{ &#xA0; &quot;_index&quot; : &quot;twitter&quot;, &#xA0; &quot;_type&quot; : &quot;tweet&quot;, &#xA0; &quot;_id&quot; : &quot;1&quot;, &#xA0; &quot;_version&quot; : 1, &#xA0; &quot;_source&quot; : { &#xA0; &#xA0; &#xA0; &quot;message&quot; : &quot;elasticsearch now has versioning support!&quot; &#xA0; &#xA0;} }</pre>
Same applies when doing a search, we will get a <code>_version</code> per search hit.
<h1>Optimistic Concurrency Control</h1>
Now, the interesting bits can start, as we can use the versioning feature to perform <a href="http://en.wikipedia.org/wiki/Optimistic_concurrency_control">optimistic concurrency control</a>. For example, lets say we update the first tweet we indexed:
<pre class="dontquote prettyprint">curl -XPUT &apos;localhost:9200/twitter/tweet/1?pretty=true&apos; -d &apos;{ &#xA0; &#xA0; &quot;message&quot; : &quot;elasticsearch now has versioning support, cool!&quot; }&apos;</pre>
What we will get back is:
<pre class="dontquote prettyprint">{ &#xA0; &quot;ok&quot; : true, &#xA0; &quot;_index&quot; : &quot;twitter&quot;, &#xA0; &quot;_type&quot; : &quot;tweet&quot;, &#xA0; &quot;_id&quot; : &quot;1&quot;, &#xA0; &quot;_version&quot; : 2 }</pre>
Note the <code>_version</code> value has been incremented to <code>2</code>. But, we can also provide the version that we would like the update to be performed:
<pre class="dontquote prettyprint">curl -XPUT &apos;localhost:9200/twitter/tweet/1?version=2&amp;pretty=true&apos; -d &apos;{ &#xA0; &#xA0; &quot;message&quot; : &quot;elasticsearch now has versioning support, double cool!&quot; }&apos;</pre>
This will update the tweet again, and increment its version to <code>3</code>. Now, lets say someone wanted to do another update on a stale data, and execute the above again:
<pre class="dontquote prettyprint">curl -XPUT &apos;localhost:9200/twitter/tweet/1?version=2&amp;pretty=true&apos; -d &apos;{ &#xA0; &#xA0; &quot;message&quot; : &quot;elasticsearch now has versioning support, stale cool!&quot; }&apos;</pre>
What we would get now is a conflict, with the HTTP error code of <code>409</code> and a response:
<pre class="dontquote prettyprint">{ &#xA0; &quot;error&quot; : &quot;VersionConflictEngineException[[twitter][2] [tweet][1]: version conflict, current [3], required [2]]&quot; }</pre>
This notion is very powerful, especially with the near real time aspect of elasticsearch. Version information is completely real time, thus, a get and an index can be repeated until the index has been refreshed without the need to call explicit refresh each time.
<h1>Put if Absent</h1>
The versioning feature now allow to perform a &#x201C;put if absent&#x201D; logic by using the <code>create</code> flag when indexing, for example, lets say we want to create the first tweet (its already there in the index):
<pre class="dontquote prettyprint">curl -XPUT &apos;localhost:9200/twitter/tweet/1?op_type=create&amp;pretty=true&apos; -d &apos;{ &#xA0; &#xA0; &quot;message&quot; : &quot;elasticsearch now has versioning support, but I am not absent...!&quot; }&apos;</pre>
Now, we would get an error that the document already exists, with an HTTP response code of <code>409</code>.
<pre class="dontquote prettyprint">{ &#xA0; &quot;error&quot; : &quot;DocumentAlreadyExistsEngineException[[twitter][2] [tweet][1]: document already exists]&quot; }</pre>
<h1>Other Nice Side Effects</h1>
<ul>
<li>When deleting, and the document does not exists, a flag indicating that it was not found will be returned with an HTTP return code of <code>404</code>.</li>
<li>There is no longer a need to use the <code>create</code> operation type to improve indexing speed, a doc is automatically &#x201C;created&#x201D; (and not &#x201C;updated&#x201D;) if it does not exists.</li>
</ul>
			<aside id="post_tags">
							</aside>
		</article></body></html>

Versioning

<article>
Percolator is another feature that will be part of upcoming elasticsearch 0.15. The percolator allows you to register queries against an index, and then send <code>percolate</code> requests which include a doc, and getting back the queries that match on that doc out of the set of registered queries.

Think of it as the reverse operation of what elasticsearch does by nature: instead of sending docs, indexing them, and then running queries, one sends queries, registers them, and then sends docs and finds out which queries match that doc.

As an example, a user can register an interest (a query) on all tweets that contain the word “elasticsearch”. For every tweet, one can percolate the tweet against all registered user queries, and find out which ones matched.

<h2>Sample Usage</h2>
First, create an index we will work with:

<pre class="prettyprint">curl -XPUT 'localhost:9200/twitter'
</pre>
Next, we will register a percolator query with a specific name called <code>elasticsearch</code> against the <code>twitter</code> index:

<pre class="prettyprint">curl -XPUT localhost:9200/_percolator/twitter/elasticsearch -d '{
 "query" : {
 "field" : {
 "message" : "elasticsearch"
 }
 }
}'
</pre>
And now, we can percolate a document and see which queries match on it (note, its not really indexed!):

<pre class="prettyprint">curl -XGET localhost:9200/twitter/tweet/_percolate -d '{
 "doc" : {
 "message" : "this new elasticsearch percolator feature is nice, borat style"
 }
}'
</pre>
And the matched percolated queries are part of the response:

<pre class="prettyprint">{"ok":true,"matches":["elasticsearch"]}
</pre>
<h2>Percolate on Index / Bulk</h2>
Percolation can also be done when executing an index request. Lets index the tweet from before, and also specify that it needs to be percolated by adding the <code>percolate</code> parameter:

<pre class="prettyprint">curl -XPUT 'localhost:9200/twitter/tweet/1?percolate=*' -d '{
 "message" : "this new elasticsearch percolator feature is nice, borat style"
}'
</pre>
And in the response, we will also get the percolated queries it matched on:

<pre class="prettyprint">{
 "ok":true,
 "_index":"twitter",
 "_type":"tweet",
 "_id":"1",
 "_version":1,
 "matches":["elasticsearch"]
}
</pre>
<h2>Filtering Executed Queries</h2>
Since the registered percolator queries are just docs in an index, one can filter the queries that will be used to percolate a doc. For example, we can add a color field to the registered query:

<pre class="prettyprint">curl -XPUT localhost:9200/_percolator/twitter/elasticsearch -d '{
 "color" : "green",
 "query" : {
 "field" : {
 "message" : "elasticsearch"
 }
 }
}'
</pre>
And then, we can percolate a doc that only matches on <code>green</code> percolated queries:

<pre class="prettyprint">curl -XGET localhost:9200/twitter/tweet/_percolate -d '{
 "doc" : {
 "message" : "this new elasticsearch percolator feature is nice, borat style"
 },
 "query" : {
 "term" : {
 "color" : "green"
 }
 }
}'
</pre>
With no matches received for <code>blue</code> percolated queries:

<pre class="prettyprint">curl -XGET localhost:9200/twitter/tweet/_percolate -d '{
 "doc" : {
 "message" : "this new elasticsearch percolator feature is nice, borat style"
 },
 "query" : {
 "term" : {
 "color" : "blue"
 }
 }
}'
</pre>
The filtering can also be passed as a query string to the percolated parameter when indexing a doc, for example:

<pre class="prettyprint"># matches
curl -XPUT 'localhost:9200/twitter/tweet/1?percolate=color:green' -d '{
 "message" : "this new elasticsearch percolator feature is nice, borat style"
}'
# no match
curl -XPUT 'localhost:9200/twitter/tweet/1?percolate=color:blue' -d '{
 "message" : "this new elasticsearch percolator feature is nice, borat style"
}'
</pre>
<h2>How it Works</h2>
The <code>_percolator</code> which holds the repository of registered queries is just a another index in ES. The query is registered under a concrete index that exists (or will exist) in ES. That index name is represented as the type in the _percolator index (a bit confusing, I know…).

The fact that the queries are stored as docs in another index (<code>_percolator</code>) gives us both the persistency nature of it, and the ability to filter out queries to execute using another query 😊.

The <code>_percolator</code> index uses the <code>index.auto_expand_replica</code> setting to make sure that each data node will have access locally to the registered queries, allowing for fast query executing to filter out queries to run against a percolated doc.

The percolate API uses the whole number of shards as percolating processing “engines”, both primaries and replicas. In our above case, if the test index has 2 shards with 1 replica, 4 shards will round robing in handing percolate requests. (dynamically) increasing the number of replicas will increase the number of percolation power.

Note, percolate request will prefer to be executed locally, and will not try and round robin across shards if a shard exists locally on a node that received a request (for example, from HTTP). Its important to do some roundrobin in the client code among ES nodes (in any case its recommended).

<aside id="post_tags">
</aside>
</article>

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

<

2011

<

may

>

Data Visualization with ElasticSearch and Protovis

0.16.1 Released

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS