<html><head></head><body>ElasticSearch version <code>0.8.0</code> has just been released. You can download it <a href="/downloads/past-releases/0-8-0">here</a>. This release includes several bug fixes and memory footprint improvements, and one major feature, Hadoop integration. This allows to use Hadoop HDFS as elasticsearch gateway storage, and enabling it is as simple as:Installing the hadoop plugin using <code>bin/plugin -install hadoop</code>.Changing the configuration to include:<pre class="dontquote prettyprint">gateway:
 type: hdfs
 hdfs:
 uri: hdfs://host:port
 path: path/to/folder
</pre>-shay.banon</body></html>

0.8.0 Released

<html><head></head><body>ElasticSearch version <code>0.7.1</code> has just been released. You can download it <a href="/download">here</a>. This release fixed a major bug when indexing large documents resulting in storing additional null bytes (and returning them).
Version <code>0.7.1</code> also brings a major feature to elasticsearch, recovery throttling. In elasticsearch, there are two types of recovery. The first, is recovery from the gateway. This happens only when the first shard is allocated in the cluster. The second recovery happens when nodes move or allocate shards around. The recovery process in both cases include recovering both each shard index files, and the transaction log.
Up until version <code>0.7.1</code>, elasticsearch would basically go full force in performing the recovery. If a new node would join the cluster, all the possible shards would be allocated to it, and all will perform recovery in parallel. More over, each single shard index file recovery will happen in parallel as well.
This can lead to a heavy load on the nodes, making them less responsive for on going operations performed on them. From version <code>0.7.1</code>, recovery throttling is enabled, basically allowing only for a controlled number of concurrent recovery operations, and concurrent stream (single shard index file) recovery operation. Both counts are maintained on the node level, regardless of the number of indices or shards.
The <code>indices.recovery.throttler.concurrent_recoveries</code> setting controls the number of concurrent recoveries allowed (shard recoveries). It defaults to the number of cores. The <code>indices.recovery.throttler.concurrent_streams</code> control the concurrent shard index file recoveries, and defaults to the number of cores as well.
-shay.banon
</body></html>

0.7.1 Released

<html><head></head><body>ElasticSearch version <code>0.7.0</code> has just been released. You can download it <a href="/downloads/past-releases/0-7-0">here</a>. This release brings much improved stability, and several features:<h2>Zen Discovery</h2>A discovery module called <a href="/guide/reference/modules/discovery/zen.html">zen</a> built from the ground up to work well, and fast with elasticsearch. This is now the default discovery module, with the jgroups discovery module moving to be provided as a plugin.<h2>Groovy Client</h2>A native groovy client providing a Groovyfied API build on top of the native Java API. More details provided in the <a href="/blog/2010/04/19/elasticseach_just_got_groovy.html">ElasticSearch Just Got Groovy</a> blog post. As a side note, anybody up for building a Scala/JRbuy client?<h2>Cloud</h2>First and foremost, native cloud support, providing zero conf cloud discovery ( No Special Node&#x2122; ) and the ability to persist long term index storage on different cloud providers blob stores. More information can be found in the <a href="/blog/2010/05/11/here-comes-the-cloud.html">Here Comes the Cloud</a> blog post.<h2>Memcached Transport</h2>For that extra oomph when HTTP is not fast enough (mainly from other languages), elasticsearch supports a subset of the memcached protocol. Basically, the implementation implements REST on top of memcached (as much as possible). More info can be found <a href="/guide/reference/modules/memcached.html">here</a>.<h2>Simpler Plugin Management</h2>Many things in elasticsearch are implemented as a plugin. For example, the cloud support or memcached support are implemented as plugins. Now, installing a plugin is as simple as issuing the following command:<pre class="dontquote prettyprint">bin/plugin -install cloud
bin/plugin -install transport-memcached
</pre><h2>Analysis ICU</h2>Better support when working with unicode through the ICU analysis plugin. More info <a href="/guide/reference/index-modules/analysis/icu-plugin.html">here</a>.<h2>More APIs</h2>More information on nodes using the new Node stats API, as well as the ability to restart a node.<h2>JVM Clients</h2>Simpler dependency management, requiring only lucene as a dependency.<h2>XContent</h2>Though currently mainly for internal use, an abstraction on top of <code>JSON</code> has been created, inspired by <code>JSON</code> called <code>XContent</code>. There is support a <code>JSON</code> implementation for it, but also support for <code>XSON</code>, which is a binary <code>JSON</code> format for faster and smaller (message footprint) messages. The Java API already uses it automatically (not for indexed documents), and both the REST API and the indexed documents can be either in <code>JSON</code> or <code>XSON</code> format. <code>XSON</code> format will be documented in the near future to allow for non JVM based clients to use it.-shay.banon</body></html>

0.7.0 Released

<html><head></head><body>From the get go, elasticsearch has been designed and built for the cloud. From its internal architecture, to how it works in its distributed nature. In the upcoming 0.7 version, the cloud vision has been fully realized.
The Cloud integration revolves around two major components in ElasticSearch: Discovery and Gateway.
<h2>Cloud Discovery</h2>
One of the main problems with running distributed systems on the cloud is discovery. Products that can do &#x201C;zero conf&#x201D; discovery use multicast for it (elasticsearch among them), and in most cloud providers (Amazon AWS or Rackspace) multicast is disabled. The typical way to work around it is to use unicast discovery, which requires setting up a specific list of IPs/Hosts (routers or gossip servers).
Unicast discovery is problematic when used on the cloud. Machines can come and go, and their IP is not static. Cloud providers work around that by providing the ability to have a set of &#x201C;elastic IPs&#x201D;. But, at the end, the management of the cloud installation becomes a pain. At least two servers must be associated with an elastic IP and become a special exception case which needs to be managed. This goes completely against &#x201C;zero conf&#x201D; discovery and heavily complicates the cloud installation.
ElasticSearch has a new discovery module called &#x201C;Zen&#x201D; which was built from the ground up to work well in cloud environments (and integrate well with other elasticsearch modules). The cloud extension to it provides &#x201C;zero conf&#x201D; discovery in cloud environments.
In a nutshell, when running on the cloud, the list of machines that are already running on the cloud is available through cloud APIs. This information can be used to perform &#x201C;zero conf&#x201D; discovery. This follows the motto that the should be embraced by any system running on the cloud: All Machines are Created Equal.
So, how do you enabled cloud discovery on the cloud? With a few lines of configuration:
<pre class="prettyprint lang-js">cloud:
 account: &amp;lt;Your Amazon AWS Account Here&amp;gt;
 key: &amp;lt;Your Amazon AWS Secret Key Here&amp;gt;
 compute:
 type: amazon
discovery:
 type: cloud
</pre>
The above configuration enables auto discovery in Amazon AWS. Simply replace <code>amazon</code> with <code>rackspace</code> to work on the Rackspace cloud. There is a long list of compute cloud providers supported, including GoGrid, and Terremark.
<h2>Gateway</h2>
ElasticSearch has been designed to do reliable asynchronous long term persistency. This enables several features including the ability for fast local &#x201C;runtime&#x201D; storage (including in-memory) while having a long term storage that can be slower. The Gateway concept is described in the <a href="/blog/searchengine-time-machine">Search Engine Time Machine</a> post.
But first, a step back. When designing a system that would be deployed on the cloud, lets take a search engine for example <img src="https://api.contentstack.io/v2/assets/575e4c919985d58976376b18/download?uid=blt7b9c4317749b1d53?uid=blt7b9c4317749b1d53" data-sys-asset-uid="bltc733e47382b122d9" alt=";)" class="wp-smiley"> , things come and go. One of those things that come and go are disks. So, local storage, in cloud environments, is considered transient. In Amazon AWS for example, EBS (Elastic Block Store) was introduced to provide a mountable disk that survives restarts. So, we could configure our search engine to store the index on EBS. But, EBS requires periodic snapshotting to S3 (amazon blob store) for &#x201C;safe&#x201D; persistency, since EBS can certainly suffer from failures as well. Of course, this means more money spent on your cloud deployment since now one pays for both EBS and S3.
One way to work around this is to persist directly from the local store to S3 by writing some sort of synchronization script / code. But, if the machines fails we will loose all the data up to the point when the script last ran. The next step is to add replication (and sharding for performance) and so on. All of this is provided by elasticsearch out of the box.
Here is how elasticsearch can be configured to store both its cluster metadata (to survive full cluster failure) and indices in the cloud:
<pre class="prettyprint lang-js">cloud:
 account: &amp;lt;Your Amazon AWS Account Here&amp;gt;
 key: &amp;lt;Your Amazon AWS Secret Key Here&amp;gt;
 blobstore:
 type: amazon
gateway:
 type: cloud
 cloud:
 container: mycontainerhere
</pre>
The above simple configuration will store things in Amazon S3. Simply change <code>amazon</code> to <code>rackspace</code> to use Rackspace CloudFiles. There is a long list of blobstore providers supported, including Azureblob.
<h2>Final Words</h2>
As you can see, elasticsearch is now a first class citizen when running on the cloud. I believe that it has actually created a new level of intimate integration of products with the cloud. Both the Discovery and Gateway means that managing an elasticsearch deployment on the cloud is a breeze.
As a side note, I would like to note that cross cloud support is done using <a href="http://jclouds.googlecode.com">jclouds</a>. Highly recommended.
-shay.banon</body></html>

Here Comes the Cloud

Articles By

Register to Attend

Sign In to Attend

Register to Watch

You'll also receive an email with related content

Watch Now

Learn More

Contact Info

See All Posts

What to explore next...

Load more news

Load more press releases

Load More

View more learning opportunities

More

Date

Location

Agenda

Hosted by

Related workshops

See when this webinar starts in my time zone

Highlights

Featured webinar

On-demand webinar

Upcoming webinar

View next

Register now

Translate Content

<html><head></head><body><article>
			ElasticSearch version <code>0.14.0</code> has just been released. You can download it <a href="/download">here</a>. This release includes bug fixes as well as several features:
<h2>Index Templates</h2>
In ElasticSearch, creating several indices is a very simple, more over, searching over more than one index gives a lot of flexibility when it comes to designing your search system. Index templates allow to define both settings and mapping template that will be applied when an index is created based on simple matching on the index name, for example:
<pre class="dontquote prettyprint">curl -XPUT localhost:9200/_template/template_1 -d &apos; { &#xA0; &#xA0; &quot;template&quot; : &quot;te*&quot;, &#xA0; &#xA0; &quot;settings&quot; : { &#xA0; &#xA0; &#xA0; &#xA0; &quot;number_of_shards&quot; : 1 &#xA0; &#xA0; }, &#xA0; &#xA0; &quot;mappings&quot; : { &#xA0; &#xA0; &#xA0; &#xA0; &quot;type1&quot; : { &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &quot;_source&quot; : { &quot;enabled&quot; : false } &#xA0; &#xA0; &#xA0; &#xA0; } &#xA0; &#xA0; } } &apos;</pre>
<h2>Highlighting Based on _source</h2>
Up until now, highlighting required storing each separate field (usually in addition to storing to full source document). In 0.14, highlighting can now be performed directly on the _source document, causing considerable reduction in the index size when using highlighting.
<h2>Parent / Child Support</h2>
In ElasticSearch, changing a specific field in a document required full reindexing. With the new parent child support, certain use cases can use it in order to index child document to an existing parent document, and then issue queries that will be executed on the child docs, but resulting in the parent docs.
This include simple mapping definition in the child doc specifying the parent doc type, as well as reusing the routing feature added in 0.13 to make sure each child doc will end up in the same shard as the parent doc (thus reducing the cost of the &#x201C;join&#x201D; process to be in shard).
Two queries have been added to the Query DSL, <code>has_child</code> and <code>top_children</code> that allow to query child docs and return parent docs.
This feature is by no means complete, there is still work left in making it more usable (for example, returning the child docs as part of the search request), but the gist is there.
<h2>Network Optimizations</h2>
ElasticSearch is being used heavily on cloud environments, such as ec2, where network resources are constrained. In 0.14, there has been several network optimizations to handle it better on top of the already existing feature of compressing all network traffic (by setting <code>transport.tcp.compress</code> to <code>true</code>).
<h2>Analyze API</h2>
Even wondered how text gets broken down into tokens that end up being indexed? The analyze API allows to provide a text that will go through the analysis process and output the tokens (with the meta information) generated from it. Its as simple as calling:
<pre class="dontquote prettyprint">curl -XGET localhost:9200/[index_name]/_analyze?text=this is a test</pre>
<h2>Many more</h2>
There are many more small features and enhancements, as well as important bug fixes. Make sure to read the release notes on the wiki.
-shay.banon
			<aside id="post_tags">
							</aside>
		</article></body></html>

0.14.0 Released

<html><head></head><body>ElasticSearch version <code>0.13.0</code> has just been released. You can download it <a href="/download">here</a>. This release includes important bug fixes (most important one is a possible corruption in the index data) and some new features, including:<h2>More Mapping Features</h2>Support for a new IP (ipv4) type (that is automatically detected) allowing to do range queries (for example) on IP ranges. More enhancements to dynamic templates matching, a compression threshold on the source field, and an analyzer mapping field.<h2>Explicit Shard Routing Control</h2>Up until 0.13, documents were routed to the relevant shard based on their respective type and id values. In this version, they are not routed only by the id value by default (check the upgrade notes below), and they can be routed based on a custom value. This means that doing searches can be much faster when knowing in advance the list of routing values the search should executed on.For example, when indexing a blog post as a single document, and a blog comment as another document, the comment document can be routed to the same shard as the blog post using the blog post id. This means that when wanting to search only on things relating to a specific blog post, only a single shard needs to be searched on and not all shards of that index.This is very similar to the multi indices feature (and the ability to search on several of them), but provides a much higher control of routing within an index.<h2>Improved Support for Large Number of Indices</h2>Large number of indices now consume much less resources on the cluster as a whole, and a new API allowing to close and open an index has been added.<h2>Many More Smaller Features</h2>There are many more smaller features, with the list available <a href="https://github.com/elasticsearch/elasticsearch/issues/closed">here</a> under the 0.13 tag.<h2>Upgrade Notes</h2>Two important nodes when upgrading to 0.13:The first, due to the change in the default routing mechanism of routing only based on the id and not the type, existing systems upgrading to the new version must set the <code>cluster.routing.operation.use_type</code> to <code>true</code> in order to maintain the same hashing mechanism as in previous releases.The second, the location of the index files have changed. It used to be stored under the &#x201C;work&#x201D; directory, but it has been moved to a new &#x201C;data&#x201D; directory. This means two things, if you are using the default, then rename the work directory under the installation to data. If <code>path.work</code> was being explicitly set, change the setting to <code>path.data</code>.This was done in order to simplify the management of files elasticsearch created. Now, there are three places where data is created: <code>path.data</code> for index related data information, <code>path.logs</code> for the location of the logs, and <code>path.work</code> for temporal data.-shay.banon</body></html>

0.13.0 Released

<html><head></head><body>ElasticSearch version <code>0.12.0</code> has just been released. You can download it <a href="/download">here</a>. This release is a bit early due to a major bug found with full cluster recovery when using more than one index and the local gateway.
Along the way, some features were added as well, those include:
<h2>More Script Language Support</h2>
On top of mvel, there is now support for JavaScript, Python, and Groovy for executing scripts. Scripts can be used in different components within elasticsearch, including custom scoring, facets, and others. This support is going to be utilized in future features as well.
<h2>Dynamic Templates</h2>
Dynamic schema introduction is already a part of elasticsearch, but sometimes, the defaults are just not good enough. Dynamic templates allow to control different mappings aspects while still retaining the dynamic nature of a document. A dynamic template, for example, can turn a single field automatically into a `multi_field` or have fields matching a template be stored.
<h2>Geo Overhaul</h2>
The geo support aim was to support multiple location per document. The current design was problematic in supporting that, so a new design was implemented. This sadly requires a full reindexing if using geo features, as well as requiring to explicitly define `geo_point` mapping type.
<h2>Facets Improvements</h2>
Both the `term` and the `statistical` facets now can be executed on more than one field and aggregating the results across all fields values.
<h2>Query Filters Improvements</h2>
Fine grained control over which filter results are cached or not, with sensible defaults per filter (as filters behave differently).
<h2>Other</h2>
Several small bug fixes and improvements. Note, if you use the thrift client, the thrift protocol has changed not to include some keywords in certain langs. Make sure to regenerate the client code.
-shay.banon
</body></html>

0.12.0 Released

<html><head></head><body>ElasticSearch version <code>0.11.0</code> has just been released. You can download it <a href="/download">here</a>. This is a major release for elasticsearch, both in terms of feature set as well as stability.
Major list of features include:
<h2>Zero Conf Persistency</h2>
Out of the box long term persistency using a &#x201C;local&#x201D; gateway dubbed <a href="/blog/2010/09/27/zero_conf_persistency.html">zero conf persistency</a>.
<h2>River</h2>
Generic support for <a href="/blog/2010/09/28/the_river.html" title="s">river</a> which are services running within the elasticsearch cluster indexing streams of data. Rivers exists for <a href="/blog/2010/09/28/the_river.html">twitter</a>, <a href="/blog/2010/09/28/the_river_searchable_couchdb.html">couchdb</a>, and <a href="/blog/2010/09/28/the_river_rabbitmq.html">rabbitmq</a>.
<h2>Bulk API</h2>
A new API to do bulk indexing of data, greatly increasing the throughput of the indexing process. Details <a href="/guide/reference/api/bulk.html">here</a>.
<h2>Thrift REST Transport</h2>
A Thrift based REST transport (mimicking HTTP) for faster execution compared with plain HTTP.
<h2>Minor Features</h2>
Minor features include <a href="/guide/reference/mapping/source-field.html">_source fields</a> and the ability to use them in plain fields, <a href="/guide/en/elasticsearch/reference/current/search-request-named-queries-and-filters.html">named_filters</a>, improved geo support, faster query_string and field queries execution, and of course, bug fixes.
-shay.banon
</body></html>

0.11.0 Released

<html><head></head><body><a href="/blog/2010/09/28/the_river.html">The River</a> allows to easily define data sources and have elasticsearch index them. The example provided in the post is twitter, but a river is an open API and can have different implementations. One of those coming out at 0.11 is <a href="http://couchdb.apache.org/">CouchDB</a>.
The <a href="http://github.com/elasticsearch/elasticsearch/issues/382">CouchDB River</a> allows to automatically index couchdb and make it searchable using the excellent <a href="http://guide.couchdb.org/draft/notifications.html">_changes</a> stream couchdb provides. Setting it up is as simple as executing the following against elasticsearch:
<pre class="prettyprint lang-java">curl -XPUT &apos;localhost:9200/_river/my_db/_meta&apos; -d &apos;{
 &quot;type&quot; : &quot;couchdb&quot;,
 &quot;couchdb&quot; : {
 &quot;host&quot; : &quot;localhost&quot;,
 &quot;port&quot; : 5984,
 &quot;db&quot; : &quot;my_db&quot;,
 &quot;filter&quot; : null
 }
}&apos;</pre>
This call will create a river that uses the <code>_changes</code> stream to index all data within couchdb. Moreover, any &#x201C;future&#x201D; changes will automatically be indexed as well, making your search index and couchdb synchronized at all times.
On top of that, in case of a failover, the couchdb river will automatically be started on another elasticsearch node, and continue indexing from the last indexed seq.
As you can see, elasticsearch can easily have several couchdb rivers (and other types of rivers) running at the same time, all pointing to different databases and indexing them into different indices (or the same index, you choose) using the same elasticsearch cluster.
This means that being able to search couchdb has just become really really simple. The couchdb river is provided as a plugin (in upcoming 0.11) and can be installed using <code>plugin -install river-couchdb</code>.
-shay.banon
</body></html>

Searchable CouchDB

<html><head></head><body><article>
			Another <a href="/blog/2010/09/28/the_river.html">River</a> implementation, <a href="http://github.com/elasticsearch/elasticsearch/issues/380">RabbitMQ River</a> allowing to automatically index a rabbitmq queue. The format of the messages follows the new <a href="http://github.com/elasticsearch/elasticsearch/issues/371">bulk api</a> format:
<pre class="prettyprint lang-js">{ &quot;index&quot; : { &quot;_index&quot; : &quot;twitter&quot;, &quot;_type&quot; : &quot;tweet&quot;, &quot;_id&quot; : &quot;1&quot; } { &quot;tweet&quot; : { &quot;text&quot; : &quot;this is a tweet&quot; } } { &quot;delete&quot; : { &quot;_index&quot; : &quot;twitter&quot;, &quot;_type&quot; : &quot;tweet&quot;, &quot;_id&quot; : &quot;2&quot; } } { &quot;create&quot; : { &quot;_index&quot; : &quot;twitter&quot;, &quot;_type&quot; : &quot;tweet&quot;, &quot;_id&quot; : &quot;1&quot; } { &quot;tweet&quot; : { &quot;text&quot; : &quot;another tweet&quot; } }</pre>
Creating the rabbitmq river is as simple as:
<pre class="dontquote prettyprint">curl -XPUT &apos;localhost:9200/_river/my_river/_meta&apos; -d &apos;{ &#xA0; &#xA0; &quot;type&quot; : &quot;rabbitmq&quot;, }&apos;</pre>
One of the nice features of this river is automatically bulking queue messages if the queue is overloaded, allowing for faster catchup with the messages streamed into the queue.
-shay.banon
			<aside id="post_tags">
							</aside>
		</article></body></html>

RabbitMQ River

<html><head></head><body>One of the problems elasticsearch aims at solving is &#x201C;the river&#x201D; problem. The River is the stream of constant data and somehow finding a way to waddle through it and make something meaningful out of it.
That constant data stream can come in different forms and from different sources. It can come directly from a user in an application that uses elasticsearch directly. For example, publishing a new status message, a new blog comment, or a review of a restaurant on apps that automatically apply that change to elasticsearch.
Another option is for the data to be pushed to elasticsearch. For example, <a href="http://github.com/cloudera/flume">flume</a>, cloudera log aggregator, can use an <a href="http://github.com/tallpsmith/elasticflume">elasticsearch sink</a> to push log changes to elasticsearch.
The last option, and the one discussed here is where data is pulled from one source and applied to elasticsearch. As an example, someone can write a twitter component that listens on twitter stream updates, and apply them to elasticsearch.
Those type of components, aside from writing the core code that does it, require additional services to be provided for them. For example, the twitter component will require failover support (if it fails, start it on another node), and possibly state storage (what was the last tweet indexed).
<a href="http://github.com/elasticsearch/elasticsearch/issues/377">Rivers</a> in elasticsearch provides just that. A river is a service running within elasticsearch cluster and tries and solve the third type of integration point mentioned above. Rivers are allocated to nodes within the cluster. Are provided with automatic failover in case of node failure, and allow to store state associated with them.
The River implementation is a bit of a cheat <img src="https://api.contentstack.io/v2/assets/575e4da03dc542cb38c0b21d/download?uid=blt27dedd15ec38629f?uid=blt27dedd15ec38629f"> , rivers are simply represented as different types within an index called <code>_river</code>. Creating them is as simple as creating a document named <code>_meta</code> within the river (type). Deleting them is just a matter of deleting the river (type). And last, they can easily store state as addition document(s) within the index type.
ElasticSearch has a framework support for rivers, and upcoming 0.11 version will come with several different implementations of rivers. The one covered here is the <a href="http://github.com/elasticsearch/elasticsearch/issues/378">twitter river</a>. Here is how it can be created:
<pre class="prettyprint lang-java">curl -XPUT localhost:9200/_river/my_twitter_river/_meta -d &apos;
{
 &quot;type&quot; : &quot;twitter&quot;,
 &quot;twitter&quot; : {
 &quot;user&quot; : &quot;twitter_user&quot;,
 &quot;password&quot; : &quot;twitter_passowrd&quot;
 }
}
&apos;</pre>
Once created, the <a href="http://dev.twitter.com/pages/streaming_api">global stream of changes, a.k.a the hose</a> will start to be indexed into elasticsearch (including all the relevant metadata, like geo location, places, replies, and so on). Think about the power and capabilities you get with all that data indexed in elasticsearch <img src="https://api.contentstack.io/v2/assets/575e4da03dc542cb38c0b21d/download?uid=blt27dedd15ec38629f?uid=blt27dedd15ec38629f"> .
Deleting the twitter river is as simple as:
<pre class="prettyprint lang-java">curl -XDELETE localhost:9200/_river/my_twitter_river</pre>
The twitter river will be provided as a plugin in 0.11, and can be easily installed using <code>plugin -install river-twitter</code>.</body></html>

The River

<html><head></head><body>With the new &#x201C;local gateway&#x201D;, upcoming elasticsearch version 0.11 will provide zero conf long term persistency out of the box.
In the <a href="/blog/searchengine-time-machine/">Search Engine Time Machine</a> post, elasticsearch support for long term persistency is explained. The idea is built around providing long term persistency using a shared storage solution. A common storage option between all nodes (shared file system, S3, HDFS) is used to asynchronously write changes in both the cluster meta data (indices created, mappings) and the actual indices to it.
The shared storage option has several benefits. One obvious one is the ability to store (parts) the index in memory, and still maintain long term persistency in case of full cluster shutdown. It also provides a native solution if backup is required of the indices (to s3 for example).
It does come with an overhead though, the first is the actual need for a shared storage solution for long term persistency, which does complicate things for simpler and get it started scenarios. The other is the fact that potentially very large data set will be stored, where simply using a shared storage is an overhead that is unacceptable (mainly due to cost).
In upcoming 0.11 version, another gateway option is provided, called local gateway. The local gateway option allows the cluster to restore both its state and the indices from each node local storage (local file system). And, in order to provide the best out of the box experience, this gateway is now the default one set.
The cluster state, which includes the indices created, mappings, and other meta information is versioned and stored on the nodes. In order to recover it, the <code>gateway.recovery_after_nodes</code> (or <code>gateway.recovery_after_master_nodes</code>) should be set to high enough value out of the total expected cluster size in order to ensure latest state recovery.
Shards are recovered once a quorum (by default) of the shard with its replicas is found. For this reason (and others) it is recommended to have at least 2 replicas per shard set.
Indices (and the transaction log) must be file system based, to provide recoverability in case of full shutdown.
The local gateway allows for simple to use long term persistency with elasticsearch and should simplify greatly using elasticsearch with full persistency support.
-shay.banon
</body></html>

Zero Conf Persistency

<html><head></head><body>ElasticSearch version <code>0.10.0</code> has just been released. You can download it <a href="/download">here</a>. This is a major release for elasticsearch, both in terms of feature set as well as stability.
Major partial list of features include:
<h2>Geo Support</h2>
Geo location support has been added, allowing to have geo query based capabilities (distance, bounding box, polygon) as well as facet support (distance based). More info can be found <a href="/blog/2010/08/16/geo_location_and_search.html">here</a>.
<h2>Update Number of Replicas Dynamically</h2>
Allow to change the number of replicas an index has using a simple API.
<h2>More Facets</h2>
Range, filter, and more term facet options.
<h2>More Mapping Options</h2>
Ability to compress the <code>_source</code> field with extensive optimization at decompression only when needed (for example, decompressing directly down into the REST stream).
<h2>New Gateway Structure</h2>
A new gateway structure reducing the chances of gateway corruption as well as building the basis for future options such as saving versions of the gateway and allowing to recover from them. Here is the <a href="http://gist.github.com/546494">upgrade script</a>.
<h2>Transport Compression</h2>
The ability to configure the communication between nodes to work in a compressed mode, as well as different components using it by default (for example, peer recovery fetches the index in compressed mode).
<h2>Minor Enhancements, Bugs Squashing</h2>
A lot of work has going into improved stability of elasticsearch, better memory management, and major bugs squashing. ElasticSearch is being used by several companies to index very large amount of data with large cluster size successfully with snapshot versions of 0.10.
-shay.banon
</body></html>

0.10.0 Released

<html><head></head><body><img alt="" src="https://api.contentstack.io/v2/assets/575e4c910342dfd7382631ce/download?uid=bltbb70923e471eb9f0?uid=bltbb70923e471eb9f0" data-sys-asset-uid="blt5dc0ed94dd0e9c27" height="120px" style="float:left; margin:0 20px 20px 0;">One of the coolest search technology combinations out there are the ability to combine geo and search. Queries such as give me all the restaurants that serves meat ([insert your query here]) within 20 miles from me, or create a distance heat map of them, is slowly becoming a must have for any content website. This is becoming even more relevant with new browsers supporting <a href="http://html5demos.com/geo">Geolocation API</a>.
Already in <a href="http://github.com/elasticsearch/elasticsearch">master</a> (and in the upcoming <code>0.9.1</code> release), elasticsearch comes with rich support for geo location. Lets take a drive down the geo support path:
<h2>Indexing Location Aware Documents</h2>
In general, documents indexed are not required to define any predefined mapping in order to use geo location features, but they should conform to a convention if none is defined. For example, lets take an example of a &#x201C;pin&#x201D; that we want to index its location and maybe some tags its associated with:
<pre class="prettyprint lang-js">{
 &quot;pin&quot; : {
 &quot;location&quot; : {
 &quot;lat&quot; : 40.12,
 &quot;lon&quot; : -71.34
 },
 &quot;tag&quot; : [&quot;food&quot;, &quot;family&quot;],
 &quot;text&quot; : &quot;my favorite family restaurant&quot;
 }
}</pre>
The <code>location</code> element is a &#x201C;geo enabled&#x201D; location since it has <code>lat</code> and <code>lon</code> properties. Once one follows the above conventions, all geo location features are enabled for <code>pin.location</code>.
If explicit setting is still required, then its easy to define a mapping that defines a certain property as a <code>geo_point</code>. Here is an example:
<pre class="prettyprint lang-js">{
 &quot;pin&quot; : {
 &quot;properties&quot; : {
 &quot;location&quot; : {
 &quot;type&quot; : &quot;geo_point&quot;
 }
 }
 }
}</pre>
By defining the <code>location</code> property as <code>geo_point</code>, this means that now we can index location data in many different formats, starting from the lat/lon example above, up to <code>geohash</code>. For information on all the available formats, check out <a href="http://github.com/elasticsearch/elasticsearch/issues/278">278</a>.
<div class="rounded" style="color: #f5f5f5; background: #333; padding: 1em; margin: 1em 0;">Update: The automatic mapping of &#x201C;geo enabled&#x201D; properties has been disabled since publishing this article. You have to provide the correct mapping for geo properties. Please see the <a href="/guide/reference/mapping/geo-point-type.html">documentation</a>.</div>
<h2>Find By Location</h2>
The first thing after indexing location aware documents, is being able to query them. There are several ways to be able to query such information, the simplest one is by <a href="http://github.com/elasticsearch/elasticsearch/issues/279">distance</a>. Here is an example:
<pre class="prettyprint lang-js">{
 &quot;filtered&quot; : {
 &quot;query&quot; : {
 &quot;field&quot; : { &quot;text&quot; : &quot;restaurant&quot; }
 },
 &quot;filter&quot; : {
 &quot;geo_distance&quot; : {
 &quot;distance&quot; : &quot;12km&quot;,
 &quot;pin.location&quot; : {
 &quot;lat&quot; : 40,
 &quot;lon&quot; : -70
 }
 }
 }
 }
}</pre>
The above will search for all documents with <code>text</code> of <code>restaurant</code> that exists within <code>12km</code> of the provided location. The location point can accept several different formats as well, detailed at <a href="http://github.com/elasticsearch/elasticsearch/issues/279">279</a>.
The next query supported is a <a href="http://github.com/elasticsearch/elasticsearch/issues/290">bounding box query</a>, allowing to restrict the results into a geo box defined by the top left, and bottom right coordinates. Here is an example:
<pre class="prettyprint lang-js">{
 &quot;filtered&quot; : {
 &quot;query&quot; : {
 &quot;field&quot; : { &quot;text&quot; : &quot;restaurant&quot; }
 },
 &quot;filter&quot; : {
 &quot;geo_bounding_box&quot; : {
 &quot;pin.location&quot; : {
 &quot;top_left&quot; : {
 &quot;lat&quot; : 40.73,
 &quot;lon&quot; : -74.1
 },
 &quot;bottom_right&quot; : {
 &quot;lat&quot; : 40.717,
 &quot;lon&quot; : -73.99
 }
 }
 }
 }
 }
}</pre>
The last, and the most advance form of geo query is a <a href="http://github.com/elasticsearch/elasticsearch/issues/294">polygon based search</a>, here is an example:
<pre class="prettyprint lang-js">{
 &quot;filtered&quot; : {
 &quot;query&quot; : {
 &quot;field&quot; : { &quot;text&quot; : &quot;restaurant&quot; }
 },
 &quot;filter&quot; : {
 &quot;geo_polygon&quot; : {
 &quot;pin.location&quot; : {
 &quot;points&quot; : [
 {&quot;lat&quot; : 40, &quot;lon&quot; : -70},
 {&quot;lat&quot; : 30, &quot;lon&quot; : -80},
 {&quot;lat&quot; : 20, &quot;lon&quot; : -90}
 ]
 }
 }
 }
 }
}</pre>
<h2>Sorting</h2>
The ability to sort results not just by ranking (how relevant is the document to the query), but also by distance allows for much greater geo usability. There is now a new <code>_geo_distance</code> <a href="http://github.com/elasticsearch/elasticsearch/issues/306">sort type</a> allowing to sort based on a distance from a specific location:
<pre class="prettyprint lang-js">{
 &quot;sort&quot; : [
 {
 &quot;_geo_distance&quot; : {
 &quot;pin.location&quot; : [-40, 70],
 &quot;order&quot; : &quot;asc&quot;,
 &quot;unit&quot; : &quot;km&quot;
 }
 }
 ],
 &quot;query&quot; : {
 &quot;field&quot; : { &quot;text&quot; : &quot;restaurant&quot; }
 }
}</pre>
On top of that, elasticsearch will now return all the values per hit of fields sorted on, allowing to easily display this important information.
<h2>Faceting</h2>
Faceting, the ability to show an aggregated views on top of the search results go hand in hand with geo. For example, one would like to get the number of hits matching the search query within 10 miles, 20 miles, and above from his location. The <a href="http://github.com/elasticsearch/elasticsearch/issues/286">geo distance</a> facet provides just that:
<pre class="prettyprint lang-js">{
 &quot;query&quot; : {
 &quot;field&quot; : { &quot;text&quot; : &quot;restaurant&quot; }
 },
 &quot;facets&quot; : {
 &quot;geo1&quot; : {
 &quot;geo_distance&quot; : {
 &quot;pin.location&quot; : {
 &quot;lat&quot; : 40,
 &quot;lon&quot; : -70
 },
 &quot;ranges&quot; : [
 { &quot;to&quot; : 10 },
 { &quot;from&quot; : 10, &quot;to&quot; : 20 },
 { &quot;from&quot; : 20, &quot;to&quot; : 100 },
 { &quot;from&quot; : 100 }
 ]
 }
 }
 }
}</pre>
<h2>Summary</h2>
The combination of search with geo is a natural one, and slowly becoming critical to any (web) application, especially with HTML 5 and mobile devices becoming more and more widespread. elasticsearch upcoming geo support brings this integration into a whole new level, and enables application to provide rich geo and search functionality easily (ohh, and scale <img src="https://api.contentstack.io/v2/assets/575e4c919985d58976376b18/download?uid=blt7b9c4317749b1d53?uid=blt7b9c4317749b1d53" data-sys-asset-uid="bltc733e47382b122d9" alt=";)" class="wp-smiley"> ).
-shay.banon</body></html>

Geo Location and Search

<html><head></head><body>ElasticSearch version <code>0.9.0</code> has just been released. You can download it <a href="/download">here</a>. This is a major release for elasticsearch, both in terms of feature set as well as stability.
Major partial list of features include:
<h2>Facets Support</h2>
<a href="/guide/en/elasticsearch/reference/current/search-facets.html">Facets</a> allow to provide aggregated data view correlating to the search query executed. ElasticSearch now comes with several facets implementations, including the typical &#x201C;terms&#x201D; facets (allowing to get the most popular terms, and how often they occur), statistical facets providing statistical information on numeric fields including count, total, mean, min, max, variance, sum of squares, and standard deviation. And, the coolest facet type, histogram facets, which based on a field, break it into buckets and provide data on the relevant buckets derived from the same field, another field, or a script.
<h2>Scripting Support</h2>
Added as a general feature within elasticsearch, <a href="/guide/reference/modules/scripting.html">scripting</a> allows to define scripts that are evaluated at runtime and can be used in different elasticsearch features, such as facets, script search fields, script filter, and so on.
<h2>More Queries and Filters</h2>
Additional queries and filters have been added. Thanks to the Query DSL of elasticsearch, adding queries is a snap. Queries include fuzzy query, custom score query (based on scripts), script filter, and/or/not filters, and more.
<h2>Improved Gateway Recovery</h2>
A major feature in elasticsearch, allowing to reuse existing index files when recovering from the gateway after a full cluster restart significantly reducing the time it takes to recover from the gateway. This include additions to the gateway behavior including the ability to control when the initial recovery will happen as a factor of the number of nodes in the cluster and time. Also, the shutdown API has been enhanced to better handle full cluster shutdown.
<h2>Script Search Fields</h2>
The ability to load custom data (based on non stored fields) as part of the search request.
<h2>Improves Fluent Java / Groovy API</h2>
The Java / Groovy API has been greatly enhanced to provide more fluent API execution.
<h2>AWS Cloud Specific Plugin</h2>
The cloud API has been rewritten to use directly the amazon AWS API, providing better stability and features when using AWS. The cloud plugin now only works with Amazon AWS.
<h2>Stability, Bug Squashing, and Memory Usage Improvements</h2>
A lot of work has going into improved stability of elasticsearch, better memory management, and major bugs squashing. ElasticSearch is being used by several companies to index very large amount of data with large cluster size successfully with snapshot versions of 0.9.
-shay.banon
</body></html>

0.9.0 Released

<html><head></head><body><div class="main">
					<img class="left-img" alt="" src="http://media.xircles.codehaus.org/_projects/groovy/_logos/medium.png" height="60px">
Just pushed into master (upcoming 0.7 release) a <a href="http://groovy.codehaus.org/">Groovy</a> client wrapper on-top of the <a href="/guide/reference/java-api/">Java API</a> elasticsearch provides.
Using elasticsearch with dynamic languages makes a lot of sense, especially thanks to its <a href="/blog/your-data-your-search/">domain driven approach</a>, and thanks to the fact that Groovy runs on the JVM, it can make use of the native elasticsearch Java API. Here are some examples:
Creating a node (that acts as a client) within the cluster is simple using the <code>GNodeBuilder</code>:
<pre class="dontquote prettyprint">&lt;span class=&quot;kwd&quot;&gt;def&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; nodeBuilder &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; &lt;/span&gt;&lt;span class=&quot;kwd&quot;&gt;new&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; org&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;elasticsearch&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;groovy&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;typ&quot;&gt;GNodeBuilder&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;()&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
nodeBuilder&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;settings &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 node &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 client &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; &lt;/span&gt;&lt;span class=&quot;kwd&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
&lt;/span&gt;&lt;span class=&quot;kwd&quot;&gt;def&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; gNode &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; nodeBuilder&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;node&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;()&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
&lt;/span&gt;&lt;span class=&quot;kwd&quot;&gt;def&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; client &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; gNode&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;client&lt;/span&gt;</pre>
Note, right from the start, the domain driven settings applied. Settings in elasticsearch can be defined using <code>JSON</code>, and, by utilizing Grails <code>JsonBuilder</code>, they can be expressed as a Groovy <code>Closure</code>.
Next, lets index some data:
<pre class="dontquote prettyprint">&lt;span class=&quot;kwd&quot;&gt;def&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; future &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; client&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;index &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 index &lt;/span&gt;&lt;span class=&quot;str&quot;&gt;&quot;twitter&quot;&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 type &lt;/span&gt;&lt;span class=&quot;str&quot;&gt;&quot;tweet&quot;&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 id &lt;/span&gt;&lt;span class=&quot;str&quot;&gt;&quot;1&quot;&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 source &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 user &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; &lt;/span&gt;&lt;span class=&quot;str&quot;&gt;&quot;kimchy&quot;&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 message &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; &lt;/span&gt;&lt;span class=&quot;str&quot;&gt;&quot;elasticsearch is groovy&quot;&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;

&lt;/span&gt;&lt;span class=&quot;com&quot;&gt;// a listener can be added to the future&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
future&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;successs &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;typ&quot;&gt;IndexResponse&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; response &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;-&gt;&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 println &lt;/span&gt;&lt;span class=&quot;str&quot;&gt;&quot;Indexed $response.index/$response.type/$response.id&quot;&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;

&lt;/span&gt;&lt;span class=&quot;com&quot;&gt;// or, we can wait for the response&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
println &lt;/span&gt;&lt;span class=&quot;str&quot;&gt;&quot;Indexed $future.response.index/$future.response.type/$future.response.id&quot;&lt;/span&gt;</pre>
Here, we indexed a tweet into an index called <code>twitter</code>, the type is a <code>tweet</code> and under id <code>1</code>. Note that the indexed <code>JSON</code> is expressed using the same <code>JsonBuilder</code>.
Also, all operations in elasticsearch are asynchronous allowing to either register a listener (on success/failure) or work with an <code>ActionFuture</code>. The <code>future</code> in the Groovy case is an enhanced Groovy future called <code>GActionFuture</code>. It allows to wait for the <code>response</code>, or register <code>Closure</code> that will be called on a successful index, failed index, or both.
All APIs are used exactly the same as the above index one. Let me finish with an example of the Search API, which shows the power of the search query DSL:
<pre class="dontquote prettyprint">&lt;span class=&quot;kwd&quot;&gt;def&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; search &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; client&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;search &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 indices &lt;/span&gt;&lt;span class=&quot;str&quot;&gt;&quot;twitter&quot;&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 types &lt;/span&gt;&lt;span class=&quot;str&quot;&gt;&quot;tweet&quot;&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 source &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 query &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 term&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;user&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt; &lt;/span&gt;&lt;span class=&quot;str&quot;&gt;&quot;kimchy&quot;&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
 &lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
&lt;/span&gt;&lt;span class=&quot;pun&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;

println &lt;/span&gt;&lt;span class=&quot;str&quot;&gt;&quot;Search returned $search.response.hits.totalHits total hits&quot;&lt;/span&gt;&lt;span class=&quot;pln&quot;&gt;
println &lt;/span&gt;&lt;span class=&quot;str&quot;&gt;&quot;First hit tweet message is $search.response.hits[0].source.message&quot;&lt;/span&gt;</pre>
As you can see, using elasticsearch from Groovy is groovy ;). Someone up for building a grails plugin utilizing this?
-shay.banon
	
				</div></body></html>

ElasticSearch Just Got Groovy

<html><head></head><body>ElasticSearch version <code>0.6.0</code> has just been released. You can download it <a href="/download">here</a>. This release brings much improved stability, and several features:
First, a big rename has occurred. All the JSON API now uses &#x201C;underscore casing&#x201D; instead of &#x201C;CamelCase casing&#x201D;. This makes elasticsearch more streamlined with other JSON based REST APIs out there.
The JSON API is much more flexible now, supporting numbers provided as strings, and boolean values provided as either numbers or strings. This makes using elasticsearch from dynamic languages more easy.
<code>_all</code> field support has been added, automatically creating a field that includes all the different fields in the JSON document for simpler searching (no need to explicitly specify the field name to search on). One of the nice things about the <code>_all</code> field is that it takes boost level setting of different fields into account. More information on the <code>_all</code> field can be found <a href="/guide/reference/mapping/all-field.html">here</a>.
Highlighting is now supported as part of the search request.
Simpler Query DSL including support for <code>fuzzy_like_this</code> queries and <code>gt/lt/gte/lte</code> on <code>range</code> queries.
<a href="/guide/reference/api/admin-indices-aliases.html">Index Aliases</a> API allows to create aliases associated with a single index or more and executing other APIs using it instead of the actual index names.
A new plugin system has been develop allowing to easily extend elasticsearch with the first plugin being the <code>attachments</code> plugin allowing to index &#x201C;attachments&#x201D; such as documents, images, mails, and so on.
Internal changes to how communication is handled between nodes resulting in much smaller messages passing around over the low level transport layer and a lower latency/overhead for each API.
Many bug fixes and performance enhancements slowly making elasticsearch as rock solid as it should be!
Last but not least, elasticsearch is now on Maven repository, with a <a href="http://oss.sonatype.org/content/repositories/releases/">releases repo</a> and a <a href="http://oss.sonatype.org/content/repositories/snapshots">snapshots repo</a>.
-shay.banon
</body></html>

0.6.0 Released

<html><head></head><body>ElasticSearch version <code>0.5.0</code> has just been released. You can download it <a href="/download">here</a>. This release brings much improved stability, better handling of mapping definitions, and several features:
Several new queries have been added, including <code>moreLikeThis</code> , <code>moreLikeThisField</code>, <code>fieldQuery</code>, <code>queryString</code> with multiple fields.
<a href="/guide/reference/api/search/facets/terms-facet.html">terms</a> API allowing to get terms (from one or more indices) of one or more fields and their respective document frequencies (how often they exists in documents). This can be very handy to implement things like tag clouds or simple auto suggest.
<a href="/guide/en/elasticsearch/reference/current/cluster-health.html">cluster_health</a> API for simple indication on the health of the cluster, as well as the ability to wait for the cluster to reach a health status.
<a href="/guide/reference/api/more-like-this.html">moreLikeThis</a> API to search for documents that are like a certain document.
<a href="/guide/reference/java-api/">Java API</a> exposing all of elasticsearch operations/actions using simple, transport based, async API to use with any JVM based language.
There are many more minor features and bug fixes, all listed <a href="http://github.com/elasticsearch/elasticsearch/issues/closed">here</a> under the <code>v0.5.0</code> tag.
-shay.banon
</body></html>

0.5.0 Released

<html><head></head><body><a href="http://en.wikipedia.org/wiki/NoSQL">NoSQL</a> seems to be all the hype, helping people get away from databases into more (horizontally) scalable solutions while sacrificing certain database characteristics (and gaining others, such as being schema free). And NoSQL products seems to start showing up everywhere such as <a href="http://incubator.apache.org/cassandra/">Cassandra</a>, <a href="http://hadoop.apache.org/hbase/">HBase</a>, <a href="http://riak.basho.com/">riak</a>, <a href="http://www.mongodb.org/">mongodb</a>, <a href="http://project-voldemort.com/">Project Voldemort</a>, and <a href="http://code.google.com/p/terrastore/">Terrastore</a> to name a few.
	
	To this list, I would also add long time (data) grid providers such as <a href="http://www.gigaspaces.com">GigaSpaces</a>, <a href="http://www.oracle.com/technology/products/coherence/index.html">Oracle Coherence</a>, and <a href="http://publib.boulder.ibm.com/infocenter/wxdinfo/v6r0/topic/com.ibm.websphere.xd.doc/info/prodovr/cobgojbectgrid.html">IBM ObjectGrid</a>. Products that are playing at a much higher playground than the previous ones (think NoSQL++).
	
	As a side note, its great that the NoSQL solution is gaining steam. If in the past (3-4 years ago) I had to explain and educate why such concepts as collocation, master worker, map reduce, and in memory storage are very important when designing scalable systems, now they have become almost common knowledge.
	
	All products mentioned provide, in one form or another, the ability to query data. Some use SQL like queries, others use map reduce. At one point though, regardless of where or how you store your data, you would want to add search to it. What do I mean when I say search? I mean search in what you expect search engines to provide you with. Think of twitter solving their data storage problems, but not being able to search on all the tweets (preferably in real time).
	
	But now we find ourself in a problematic situation. We finally managed to solve the scaling requirements of our data storage, only to find that we need to add search to our system (hopefully, you are adding search from the get go, as its a must have feature in any application). And the search solution needs to scale. More over, since the dream scenario is that our search solution would allow us to search on all our data, it needs to scale as much as our NoSQL solution does.
	
	For this reason, a true, distributed, scalable search solution is required. And guess what, I think I know of one :). So how to we pull this off, the integration between elasticsearch and our chosen NoSQL solution?
	
	Well, we can start with the simplest solution. Anytime we update the NoSQL solution, we can go ahead and execute the same or a similar update to elasticsearch. Since elasticsearch has a REST API, it can basically integrate with any language that interacts with our chosen NoSQL. This is the simplest way to solve our problem.
	
	A more interesting solution would be to integrate search into NoSQL. Think of an indexing stored procedure running within our NoSQL cluster and every time something changes, the same change is applied to elasticsearch as well. Moreover, converting from either pure JSON, column based storage, or even Object based is just a matter of converting them into an indexable JSON that elasticsearch provides. And, thanks to the fact that elasticsearch data model is flexible (as explained in the <a href="/blog/your-data-your-search/">Your Data, Your Search</a> blog post), the conversion is extremely simple.
	
	As an example, with <a href="http://www.gigaspaces.com">GigaSpaces</a>, we can register an event container (against a customizable matching query/template) that will automatically apply any changes done in the data grid into elasticsearch, the code would generally look something like this (including a sneak peak at the upcoming elasticsearch Java API):
	
	<pre class="prettyprint lang-java">void onEvent(Object event, EntryArrivedRemoteEvent event) {
 String index = &quot;myIndex&quot;;
 String type = event.getClass().getSimpleName();
 String id = extractId(event);
 switch (remoteEvent.getNotifyActionType()) {
 case NOTIFY_LEASE_EXPIRATION:
 case NOTIFY_TAKE:
 esClient.delete(deleteRequest(&quot;myIndex&quot;)
 .type(type)
 .id(id));
 break;
 case NOTIFY_UPDATE:
 case NOTIFY_WRITE:
 esClient.index(indexRequest(&quot;myIndex&quot;)
 .type(type)
 .id(id)
 .source(toJson(event)));
 break;
 }
}</pre>
	When doing so, we can actually get all the features <a href="http://www.gigaspaces.com">GigaSpaces</a> provides as a high throughput, low latency, scalable data grid (among many other features), with the option to perform full text search on all/part of the data stored.
	
	Another example, which is already there to play with (proper documentation coming soon) is the integration done between <a href="http://code.google.com/p/terrastore/">Terrastore</a> and elasticsearch. Since terrastore handles data in JSON format, there is no conversion needed and applying changes in terrastore is just a matter of registering an <code>EventListener</code> (simplified version of <a href="http://code.google.com/p/terrastore/source/browse/src/main/java/terrastore/search/ElasticSearchListener.java?repo=search">this code</a>):
	
	<pre class="prettyprint lang-java">void onValueChanged(final String bucket, final String key, final byte[] value) {
 esClient.index(indexRequest(&quot;myIndex&quot;)
 .type(bucket)
 .id(key)
 .source(value));
}

void onValueRemoved(final String bucket, final String key) {
 esClient.delete(deleteRequest(&quot;myIndex&quot;)
 .type(bucket)
 .id(key));
}</pre>
	At the end, your system of record can be a database, a NoSQL product, or something else entirely. With search becoming a requirement in any application built this days (real time search is all the buzz ;) ), a proper thought into how to solve this problem will help you in the long run. And remember, if your data needs scaling, you search solution has to be able to scale as well.
	
	-shay.banon
	
	If someone is interested (I sure as hell am) at getting the same integration into other NoSQL solutions, such as <a href="http://incubator.apache.org/cassandra/">Cassandra</a>, <a href="http://hadoop.apache.org/hbase/">HBase</a>, <a href="http://riak.basho.com/">riak</a>, <a href="http://www.mongodb.org/">mongodb</a>, and <a href="http://project-voldemort.com/">Project Voldemort</a>, drop a line at <a href="/community">elasticsearch</a> and lets go for it.
	</body></html>

NoSQL, Yes Search

<div class="main">
					Building a highly available product requires some creative thinking. High availability is measured in two aspects when talking about distributed products. The first is the ability to survive partial cluster failure gracefully (and scale to new nodes when added), and the second is handling complete cluster failure or shutdown (and bringing it back up again).
The following covers how a distributed system can handle these two problems, especially in cloud environments, and what ElasticSearch does in such cases.
<h2>Partial Cluster Failure</h2>
Partial cluster failure means that one or more nodes failed within the cluster. Support in a distributed system for partial cluster failure means (surprise surprise) not losing data when such events happen. It gets even more important when working in the cloud, since the control one has over the servers (for good or bad) is out of our hands. Moreover, with cloud providers innovating with cool features like Amazon’s <a href="http://aws.amazon.com/ec2/spot-instances/">spot instances</a> (the ability to bid for instances), we cannot control when our nodes operate. They basically come and go as they wish.
This is a question of state, and where the state is stored. I will cover the case when the state is stored locally to the node, which means that if you lose your node, you lose the data stored with the node. It is important to understand that almost any system will work much faster with local state (local file system, â€œlocalâ€ memory) than with a shared storage solution (like NFS). Also, when more than one node needs to access shared state (like NFS), usually locking needs to take place, which hinders the scalability of the system.
The most common solution to partial cluster failure is to replicate data between nodes. The ability to control the number of replicas in the cluster allows one to provide a higher degree of availability (this also relates to total cluster failure, see below). The idea of replication is to have one node that perform the operation on itself, and then to replicate the data or operation to its replicas.
In the case of ElasticSearch, an index is broken down into shards, and each shard can have zero or more replicas. Both are configurable when creating an index. Here is an example of creating an index with 3 shards, each with 2 replicas (note, this is configurable on a per index basis):
<pre class="dontquote prettyprint">&lt;span class="pln"&gt;$ curl &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="pln"&gt;XPUT http&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="com"&gt;//localhost:9200/twitter/ -d '&lt;/span&gt;&lt;span class="pln"&gt;
index &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt;
 number_of_shards &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; &lt;/span&gt;&lt;span class="lit"&gt;3&lt;/span&gt;&lt;span class="pln"&gt;
 number_of_replicas &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; &lt;/span&gt;&lt;span class="lit"&gt;2&lt;/span&gt;&lt;span class="pln"&gt;
&lt;/span&gt;&lt;span class="str"&gt;'&lt;/span&gt;</pre>
When an operation is presented to a node, it is routed automatically to the primary shard within its replication group, which performs the operation and then replicates it to all its backups. Replication to the backup is done in parallel using non-blocking IO, which basically means that more replicas mainly just means more traffic on the network.
Replicas accept read operations as well, meaning that the more replicas you have, the more highly available your index is, but also the more scalable it is in terms of search and get operations!
Once we work under the assumption that node level state is volatile, we can store the index itself in a fast medium, perhaps a local file system (even SSD), JVM heap memory, or even non-JVM (or native) memory.
The choice of where to store an index for ElasticSearch is configurable on the index level. Here is an example that indicates that the â€œtwitterâ€ index be stored in main memory:
<pre class="dontquote prettyprint">&lt;span class="pln"&gt;$ curl &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="pln"&gt;XPUT http&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="com"&gt;//localhost:9200/twitter/ -d '&lt;/span&gt;&lt;span class="pln"&gt;
index &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt;
 store&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt;
 type &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; memory
&lt;/span&gt;&lt;span class="str"&gt;'&lt;/span&gt;</pre>
But if node storage is considered transient, what do we do regarding long term persistency of the index? The answer is the next section.
<h2>Full Cluster Failure / Long Term Persistency</h2>
Handling full cluster failure or shutdown means that both the state of the cluster (i.e. which indices exist, what are the their settings, what are their mapping definitions, etc.) and the content of each index must be stored persistently elsewhere.
A solution for this is similar to <a href="http://www.apple.com/macosx/what-is-macosx/time-machine.html">Time Machine</a>, we need to persist the state into a long term storage. This solution is similar to what data grids have called <a href="http://www.infoq.com/articles/write-behind-caching">Write Behind</a>.
The idea is that short term high availability is maintained by replication, while long term persistency is done by asynchronously writing deltas of the state into long term (persistent) storage. The benefit is that real time operations are not affected by this process.
So, what happens when we restart a cluster? When it first starts up, it queries the long term storage for the cluster state (indices created, settings, mappings, etc.) and established them locally (creating indices, creating mapping). Each time a shard is first instantiated within its shard replication group, it will also recover its state from the long term persistency.
ElasticSearch provides this capability with a module called <a href="/guide/reference/modules/gateway/">Gateway</a>. For example, to have the state stored in a shared file system, each node needs to start with a configuration of:
<pre class="dontquote prettyprint">&lt;span class="pln"&gt;gateway &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt;
 type &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; fs
 fs &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt;
 location &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; &lt;/span&gt;&lt;span class="str"&gt;/shared/&lt;/span&gt;&lt;span class="pln"&gt;fs&lt;/span&gt;&lt;span class="pun"&gt;/&lt;/span&gt;</pre>
This can also be started right from the command line using (only on unix):
<pre class="dontquote prettyprint">&lt;span class="pln"&gt;$ bin&lt;/span&gt;&lt;span class="pun"&gt;/&lt;/span&gt;&lt;span class="pln"&gt;elasticsearch &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="pln"&gt;f 
 &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="typ"&gt;Des&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;gateway&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;type&lt;/span&gt;&lt;span class="pun"&gt;=&lt;/span&gt;&lt;span class="pln"&gt;fs &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="typ"&gt;Des&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;gateway&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;fs&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;location&lt;/span&gt;&lt;span class="pun"&gt;=&lt;/span&gt;&lt;span class="str"&gt;/shared/&lt;/span&gt;&lt;span class="pln"&gt;fs&lt;/span&gt;&lt;span class="pun"&gt;/&lt;/span&gt;</pre>
The above means that the cluster state will be stored on a shared file system, and each index created will automatically store its state on a shared file system as well. But, since each index has its own <a href="http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/">index gateway</a>, we can get clever and decide on a per index level if it should be stored using a gateway or not. For example, the following example disables long term storage of an index:
<pre class="dontquote prettyprint">&lt;span class="pln"&gt;$ curl &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="pln"&gt;XPUT http&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="com"&gt;//localhost:9200/my_special_volatile_index/ -d '&lt;/span&gt;&lt;span class="pln"&gt;
index &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt;
 gateway&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt;
 type &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; none
&lt;/span&gt;&lt;span class="str"&gt;'&lt;/span&gt;</pre>
As you can see, this is really powerful. More over, this is the perfect solution for cloud environments. In the cloud, storing long term state is usually done using what the cloud provider provides. For example, with Amazon, its <a href="http://aws.amazon.com/ebs/">EBS</a> or <a href="http://aws.amazon.com/s3/">S3</a>. Right out of the (current version) box, you can use Amazon EBS for long term storage in ElasticSearch (since its a mounted shared file system). In the future, there will also be a gateway module to store the state directly on Amazon S3 and others.
<h2>Final Words</h2>
This is only the tip of the iceberg of how ElasticSearch replication, recovery and gateway works, and the future will just increase the iceberg (right back at you, global warming!). For example, what I am playing with now is with the idea of treating indices state as a version control, being able to tag them, and recover back specific tags into new indices or current ones. I hope that at least in terms of handling cluster failures (both partial and total), ElasticSearch makes more sense now.
-shay.banon
	
				</div>

Search Engine Time Machine

One of the main aspects when working with business data is to try and have all different components in an ever evolving system to understand the same data structure/format (or as close as possible). This was the main drive for me when developing the data model ElasticSearch supports and the different search and interaction with the data model once indexed.
	For example, lets assume we want build our own Amazon store, and want to provide search for it. The first thing we have are of course books:
	<pre class="prettyprint">{
 "book" : {
 "isbn" : "0812504321"
 "name" : "Call of the Wild",
 "author" : {
 "first_name" : "Jack",
 "last_name" : "London"
 },
 "pages" : 128,
 "tag" : ["fiction", "children"]
 }
} </pre>The above is a very simple representation of a book in JSON, but already you can see that the book itself has an <code>author</code>, which is a complex object that has fields, and <code>tag</code>, which is an array of tag names. What you hope is that each component in your Amazon system would be able to interact with the same representation of a book, and ElasticSearch was built just for that. It slurps up any valid JSON, and more over, it understand the elements within the JSON.
	In our case, we can first throw this book into ElasticSearch to index it by simple HTTP PUT it into <code>localhost:9200/amazon/book/0812504321</code>. We index it into an index called <code>amazon</code>, and a type called <code>book</code>. Once we did that, we can simple search for it, for example using GET on <code>localhost:9200/amazon/book/_search?q=tag:fiction</code>.
	But, as mentioned before, ElasticSearch understands the structure of the JSON book, which means we can execute the following search to find all the books that were authored by someone with first name of Jack: <code>localhost:9200/amazon/book/_search?q=author.first_name:Jack</code>.
	Now, after our Amazon store has great search support for books and we are successful, we decide to start selling music as well. For example:
	<pre class="prettyprint">{
 "cd" : {
 "asin" : "B00192IV0O"
 "name" : "THE E.N.D. (Energy Never Dies)",
 "artist" : "Black Eyed Peas",
 "label" : "Interscope",
 "release_date": "2009-06-09",
 "tag" : ["hip-hop", "pop-rap"]
 }
} </pre>And, we want to index it as well. Here, comes <code>types</code> in ElasticSearch into play. We can simply index the mentioned JSON into another type (we already have <code>book</code>) called <code>cd</code>, and PUT this JSON into <code>localhost:9200/amazon/cd/B00192IV0O</code>. Now, we can easily search for all the cds by the <code>Interscope</code> label: <code>localhost:9200/amazon/cd/_search?q=label:Interscope</code>.
	The search capabilities do not end there though, lets say we want to find everything within the Amazon index that has <code>energy</code> in its name. Thats a simple search request that is not restricted to the type: <code>localhost:9200/amazon/_search?q=name:energy</code>. Now we are starting to get into really nice search capabilities, both the ability to control types within an index, and be able to search across them. Of course, if we want to search just across the <code>cd</code> and <code>book</code> types (assuming we have more types later one), we can simple execute: <code>localhost:9200/amazon/book,cd/_search?q=name:energy</code>.
	Sometimes though, we want to narrow down a certain field in the search query to be executed against a specific type. The query itself it executed against the whole index, while just an element in it (for example, in a boolean query) is executed within the specified type. This can certainly be done, and here is an example: <code>localhost:9200/amazon/_search?cd.name:energy</code>. In this case we still execute the search against the <code>amazon</code> index, but our field is <code>cd.name</code>, which ElasticSearch automatically identifies as a typed field, and execute the part of the query that relates to the typed field just within&nbsp;the <code>cd</code> type.
	This has been a very short example of ElasticSearch support for any JSON structure, with different JSON types supported through multi-types in an index. A final word, and when things gets really interesting (and really scalable), is the fact that ElasticSearch support multi indices as well. In our example, the <code>book</code> and <code>cd</code> can be two completely different indices (with different settings), and ElasticSearch allows to search across them in a similar manner that you search across types.
	-shay.banon

Your Data, Your Search

<html><head></head><body>ElasticSearch is an open source, distributed, RESTful, search engine which is built on top of <a href="http://lucene.apache.org">Lucene</a> internally and enjoys all the features it provides.
All the features it has are listed in the <a>home page</a>, so no need to list them here again (with the extra splash, if I might add). ElasticSearch itself was born out of my frustration with the fact that there isn&#x2019;t really a good, open source, solution for distributed search engine out there, which also combines what I expect of search engines after building <a href="http://www.compass-project.org">Compass</a> (and on that, I will blog later&#x2026;).
I have been working on this for the past several months, pouring my search and distributed knowledge into this (and portions of my heart and time ;) ), Enjoy!.
</body></html>

0.8.0 Released

0.7.1 Released

0.7.0 Released

Here Comes the Cloud

Products & Solutions

Company

Resources

Featured topics

News

<

2010

<

may

>

0.8.0 Released

0.7.1 Released

0.7.0 Released

Here Comes the Cloud