The Elasticsearch Query Language (ES|QL) has been added to the Elasticsearch repository. ES|QL is a powerful declarative language that's native to Elasticsearch and designed for composability, expressiveness, and speed.

Why another language?

avg_salary

lang

Dedicated query engine

image1.png

Where can I get it?

illustration-stack-modernize-solutions-1689x980.png

New to the Elasticsearch repository: ES|QL

We are pleased to announce the release of  Elasticsearch for Apache Hadoop (ES-Hadoop) 2.3.0 and 2.2.1

Elasticsearch for Apache Hadoop 2.3.0 and 2.2.1 released

We are pleased to announce that Elasticsearch for Apache Hadoop (ES-Hadoop) is joining the release bonanza through the GA release of ES-Hadoop 2.2.0 and 2.1.3

amsterdam-tour-de-hallen-10-29-2015 (1010).jpg

Elasticsearch for Apache Hadoop 2.2.0 and 2.1.3 released

Elasticsearch for Apache Hadoop (ES-Hadoop) 2.2 rc has been released, packing a significant number of bug fixes and enhancements.

<html><head></head><body>Celebrating the start of 2016, Elasticsearch for Apache Hadoop (ES-Hadoop) 2.2 rc1 has been <a href="/downloads/past-releases/elasticsearch-apache-hadoop-2-2-0-rc1">released</a>.
Packing a significant number of bug fixes and enhancements, this release candidate is the last step towards a full general availability release for the current development branch. As always, the artifacts are available at the <a href="/downloads/past-releases/elasticsearch-apache-hadoop-2-2-0-rc1">download page</a> or <a href="https://www.elastic.co/guide/en/elasticsearch/hadoop/current/install.html">Maven</a>.
<h2>Spark-ling updates</h2>ES-Hadoop 2.2 RC1 introduced support for the just-released Spark 1.6, in particular skipping <a href="https://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html#spark-data-sources">pushed down filters</a> that otherwise would be processed again in Spark (despite being already handled by the connector). For large result sets, this results in an important optimization.
<img src="https://api.contentstack.io/v2/assets/575e4d1543e9adc538715166/download?uid=blt8da2cef59ae066b8" data-sys-asset-uid="blt8da2cef59ae066b8" alt="spark-sql.png" style="float:left; max-width:100%;">
The push-down translation has been improved, in particular when dealing with <code>IN</code> filters by providing better matching when dealing with raw terms vs. values (such as dates or timestamps).
Speaking of Spark SQL, the schema declaration has been <a href="https://www.elastic.co/guide/en/elasticsearch/hadoop/master/mapping.html#mapping-multi-values">improved</a> to handle multi-valued/array fields in a simple and elegant fashion (whether the fields are nested or not).
In addition, the connector configuration is now sanitized and safely passed throughout a Spark job; this addressed a subtle bug caused by command&#xA0;line-only properties being discarded during a job stage and causing abnormal behavior.
<h2>YARN Enhancements</h2>A batch of updates were done to the YARN module by upgrading to Elasticsearch 2.1.x and allowing JVM system properties to be passed directly in the children container.
<h2>Repository HDFS </h2> 
<img src="https://api.contentstack.io/v2/assets/575e4d14d8edd48f769374e8/download?uid=bltae58b099e26dc9fb" data-sys-asset-uid="bltae58b099e26dc9fb" alt="repository.png" style="float:right; max-width:100%;" height="183" width="283">
The repository HDFS plugin has seen a lot of activity. While currently for Elasticsearch 2.0 and 2.1, it requires the JVM security manager to be disabled (as Hadoop is significantly greedier than Elasticsearch itself in terms of permissions), starting from Elasticsearch 2.2, due to the security improvements&#xA0;the plugin can customize its own code base grants.
Please&#xA0;note that the migration of the plugin to Elasticsearch core has <a href="https://github.com/elastic/elasticsearch/issues/15191">already started</a> and is currently scheduled for Elasticsearch 2.3.
More about that&#xA0;in a future blog post!
<h2>Network improvements</h2>The wan/cloud <a href="https://www.elastic.co/guide/en/elasticsearch/hadoop/master/cloud.html">feature</a> has seen a lot of uptake which exposed the connector to more varied network topologies and configuration. This led to a number of fixes in the way ES-Hadoop handles Elasticsearch clusters with hostnames and IPs (typically with network publishing enabled) and the translation between the two.
Overall, the connector picks up more information about its environments, reducing the amount of extra configuration on the user&apos;s behalf.
<h2>Feedback</h2>Please let us know what you think about RC1! We love to hear from you on <a href="https://github.com/elastic/elasticsearch-hadoop">GitHub</a>, <a href="http://www.twitter.com/elastic">Twitter</a> or the <a href="http://discuss.elastic.co/">forums</a>. (<a href="https://www.elastic.co/community">IRC</a> works too).
Looking forward to 2016!
</body></html>

hold-sparkling-banner.jpg

hold-sparkling.jpg

Elasticsearch for Apache Hadoop 2.2 RC1 is out

Announcing the GA release of ES-Hadoop 2.1, which embraces the emerging, real-time components in the Hadoop ecosystem such as Apache Spark and Storm.

<html><head></head><body>Elasticsearch for Apache Hadoop, affectionately known as ES-Hadoop, enables Hadoop users and data-hungry businesses to enhance their work-flows with a full-blown search and analytics engine, in real-time. We are thrilled to announce the GA&#xA0;release of <a href="/downloads/past-releases/elasticsearch-apache-hadoop-2-1-0">ES-Hadoop 2.1</a>.

This&#xA0;GA release has been the result of 4 successful Betas: <a href="/blog/es-hadoop-2-0-1-and-2-1-beta1">Beta1</a>, <a href="/blog/elasticsearch-hadoop-2-0-2-and-2-1-beta2">Beta2</a>, <a href="/blog/elasticsearch-yarn-and-ssl">Beta3</a>, <a href="/blog/elasticsearch-hadoop-2-1-0-beta4-released">Beta4</a>, plus <a href="/blog/es-hadoop-2-1-rc1-released">RC1</a>, over the last 10 months. We would like to thank the community and all our users for their valuable feedback during the Beta process.

Version 2.1 embraces the emerging, real-time components in the Hadoop ecosystem, (in particular <a href="http://spark.apache.org/">Apache Spark</a> and <a href="http://storm.apache.org/">Storm</a>), beefs up security support by adding SSL/TLS transport encryption with both HTTP and PKI authentication, and introduces a YARN module. All while preserving backwards compatibility, extending the number of supported Hadoop <a href="/guide/en/elasticsearch/hadoop/2.1/requirements.html">runtimes</a>, and gaining new certifications. Plus, we&apos;re still bringing you all this functionality with only a single JAR file to download, no dependencies required.

In fact, upgrading to 2.1 is simply a matter of updating the ES-Hadoop JAR. However, to help those with longer release cycles, we have also released <a href="/downloads/past-releases/elasticsearch-apache-hadoop-2-0-3">ES-Hadoop 2.0.3</a>, the last maintenance release for the 2.0.x branch. It contains only important bug-fixes without introducing any new features.

<h2>Native Integration with Spark and Spark SQL</h2>
Since its announcement, Apache Spark has taken the &apos;Big Data&apos; world by storm. While ES-Hadoop 2.0 provides support for Spark through its Map/Reduce functionality, 2.1 goes beyond that, providing native Java and Scala APIs tightly integrated with Spark Core and Spark SQL. This means one can easily index and analyze data through Elasticsearch in real-time directly from Spark. Simply put,&#xA0;through ES-Hadoop, Spark treats Elasticsearch indexes as <code>RDD</code>s (Resilient Distributed Dataset) and <code>DataFrame</code>s.

Furthermore, the connector can take any Spark <code>RDD</code> and index it or run SQL queries on top Elasticsearch indexes.

In good ol&apos; ES-Hadoop tradition, the developer never has to leave her environment; the same Spark API, conventions and data types work transparently on top of Elasticsearch.

<pre class="prettyprint">import org.elasticsearch.spark._ 
val sc = new SparkContext(conf) 
// collection of data
val numbers = Map(&quot;one&quot; -&gt; 1, &quot;two&quot; -&gt; 2, &quot;three&quot; -&gt; 3)
val airports = Map(&quot;arrival&quot; -&gt; &quot;Otopeni&quot;, &quot;SFO&quot; -&gt; &quot;San Fran&quot;)
sc.makeRDD(Seq(numbers, airports)).saveToEs(&quot;spark/docs&quot;)
// case classes
case class Trip(departure: String, arrival: String, days: long) 
val upcomingTrip = Trip(&quot;OTP&quot;, &quot;SFO&quot;, 10)
val lastWeekTrip = Trip(&quot;MUC&quot;, &quot;OTP&quot;, 3)
val rdd = sc.makeRDD(Seq(upcomingTrip, lastWeekTrip)) 
sc.saveToEs(rdd, &quot;spark/trips&quot;)
</pre>
Also available in Spark SQL as <code>DataFrame</code> (or &lt;code&gt;SchemaRDD for those on Spark 1.1-1.2):

<pre class="prettyprint">// as a DataFrame
val df = sqlContext.read().format(&quot;org.elasticsearch.spark.sql&quot;).load(&quot;spark/trips&quot;)
df.printSchema()
// root
//|-- departure: string (nullable = true)
//|-- arrival: string (nullable = true)
//|-- days: long (nullable = true)
val filter = df.filter(df(&quot;arrival&quot;).equalTo(&quot;OTP&quot;).and(df(&quot;days&quot;).gt(3))
</pre>
And also in pure SQL:

<pre class="prettyprint">CREATE TEMPORARY TABLE trips USING org.elasticsearch.spark.sql OPTIONS (path &quot;spark/trips&quot;)
SELECT departure FROM trips WHERE arrival = &quot;OTP&quot; and days &gt; 3
</pre>
The integration is not only elegant and out of the developer&apos;s way, but also quite efficient; ES-Hadoop connector pushed down all of Spark SQL queries translating them into <a href="/guide/en/elasticsearch/reference/current/query-dsl.html">Query DSL</a>, thus effectively executing the queries directly on Elasticsearch and leveraging its lightning fast search capabilities; both of the queries above are translated by the connector at runtime into:

<pre class="prettyprint">{
 &quot;query&quot; : {
 &quot;filtered&quot; : {
 &quot;query&quot; : {
 &quot;match_all&quot; : {}
 },
 &quot;filter&quot; : {
 &quot;and&quot; : [{
 &quot;query&quot; : {
 &quot;match&quot; : {
 &quot;arrival&quot; : &quot;OTP&quot;
 }
 }
 }, {
 &quot;days&quot; : {
 &quot;gt&quot; : 3
 }
 }
 ]
 }
 }
 }
}
</pre>
Note that as with all the other integrations, through ES-Hadoop all the reads and writes in the example above are executed in parallel across the index shards, for what we call partition-to-partition architecture.

2.1 supports Spark 1.0 through 1.4, in other words all stable releases currently known, so whatever Spark version is targeted, ES-Hadoop can be used right away.

<h2>Storm Integration</h2>
In addition to Apache Spark, ES-Hadoop 2.1 also provides native integration with Apache Storm, exposing Elasticsearch as <code>Spout</code> stream search results or as a `Bolt&apos; indexing Tuples flowing through your topology and making them accessible for analysis immediately.

<pre class="prettyprint">TopologyBuilder builder = new TopologyBuilder();
builder.setBolt(&quot;esBolt&quot;, new EsBolt(&quot;twitter/tweets&quot;));
</pre>
Executing queries in Elasticsearch for Storm is yet another one-liner:

<pre class="prettyprint">TopologyBuilder builder = new TopologyBuilder();
builder.setSpout(&quot;es-spout&quot;, new EsSpout(&quot;twitter/tweets&quot;, &quot;?q=nfl*), 5);
builder.setBolt(&quot;bolt&quot;, new PrinterBolt()).shuffleGrouping(&quot;es-spout&quot;);
</pre>
Under the covers, ES-Hadoop uses its parallelized infrastructure to map the Spout and Bolt instances across the index shards.

Low-latency/high-performance patterns like micro-batching and tick-tuples are supported to provide excellent throughput out of the box and closely integrate the real-time capabilities of Storm and Elasticsearch.

<h2>Security enhancements</h2>
2.1 introduces official support for cryptographic connections between Elasticsearch and Hadoop clusters through SSL/TLS, enabling data-sensitive environments to transparently encrypt data at transport level, thus prevent snooping and preserving data confidentiality. Furthermore, in addition to HTTP authentication, 2.1 introduces Public Key Infrastructure (PKI) for encrypting the authentication process as well.

<h2>Elasticsearch on YARN</h2>
Another major addition in 2.1 is the introduction of the <a href="/guide/en/elasticsearch/hadoop/current/es-yarn.html">Elasticsearch-on-YARN</a> (aka ES-Yarn) project for running an Elasticsearch cluster within a YARN environment. Similar to the <a href="https://github.com/elastic/elasticsearch-hadoop/tree/master/repository-hdfs">repository-hdfs</a> plugin, ES-Yarn is distributed as part of the ES-Hadoop&#xA0;project, but is independent and has no dependencies outside YARN itself.

With ES-Yarn, one can now provision, start and stop Elasticsearch directly on a YARN cluster. In YARN lingo, es-yarn bootstraps a client that deploys a dedicated <code>ApplicationManager</code> in YARN which, on its behalf, creates one container for each Elasticsearch node required.

For the user, es-yarn is a straight-forward CLI (Command-Line Interface) for deploying and managing the life cycle of the Elasticsearch cluster within YARN.

Simply download <code>elasticsearch-yarn-2.1.jar</code> and run:

<pre class="prettyprint">$ hadoop jar elasticsearch-yarn-2.1.jar
No command specified
Usage:
 -download-es : Downloads Elasticsearch.zip
 -install : Installs/Provisions Elasticsearch-YARN into HDFS
 -install-es : Installs/Provisions Elasticsearch into HDFS
 -start : Starts provisioned Elasticsearch in YARN
 -status : Reports status of Elasticsearch in YARN
 -stop : Stops Elasticsearch in YARN
 -help : Prints this help
Configuration options can be specified _after_ each command; see the documentation for more information.
</pre>
<h2>Enhanced Functionality</h2>
2.1 introduces a number of enhancements to existing features such as:

<ul>
	<li>JSON results - Documents from Elasticsearch are returned in raw format, as JSON documents. (As an implementation note the data from Elasticsearch is passed to the client in verbatim form, without any processing).</li>
	<li>Document metadata - In addition to its content, a document&apos;s metadata can be now returned without any extra network cost.</li>
	<li>Inclusion/Exclusion of fields - On the mapping front, it is now possible to specify what fields to be included or excluded for data about to be written to&#xA0;Elasticsearch. This makes it quite handy not only for doing quick transformation of the data but also specifying document metadata without storing it.</li>
	<li>Client-node routing&#xA0;-&#xA0;For clusters in restrained environments, it is now possible to use the connector through client nodes only. That is, rather than accessing the cluster data nodes directly, the connector will use the client nodes instead (which do need to have the HTTP(S) port opened) and ask those to do the work on its behalf. This change will impact parallelism as the connector will not communicate directly with the nodes. However, the performance penalty is insignificant unless a lot of data is read/written and when locality is not of importance.</li>
</ul>
<h2>Certifications</h2>
While certifications are not as cool as talking about new features, we know customers love peace of mind as much as we do. We increased the number of supported platforms in 2.1, reaching out to our partners to make sure ES-Hadoop works properly out of the box. We are happy to report that 2.1 is certified with CDH 5.x, HDP 2.x, and MapR 4.x, as well as being Databricks Spark certified.

We look forward to your feedback on Elasticsearch Hadoop 2.1 GA! You can find the binaries available on the <a href="/downloads/hadoop">download page</a>, the sources on <a href="https://github.com/elastic/elasticsearch-hadoop">GitHub</a> and the new features are&#xA0;explained in the <a href="/guide/en/elasticsearch/hadoop/current/index.html">reference documentation</a>. As always, you can file bugs or feature requests on our <a href="https://github.com/elastic/elasticsearch-hadoop/issues">issue tracker</a> and ask questions on the <a href="https://discuss.elastic.co/c/elasticsearch-and-hadoop">forum</a> or <a href="/community">IRC</a>.
</body></html>

elephant-hadoop-elasticsearch-spark-storm-full-bleed.jpg

elephant-hadoop-elasticsearch-spark-storm-thumbnail.jpeg

Elasticsearch for Apache Hadoop 2.1 GA: Spark, Storm and More

<html><head></head><body>I am happy to announce that <a href="http://www.elasticsearch.org/blog/es-hadoop-2-0-g/">Elasticsearch for Apache Hadoop 2.0 GA</a> has been <a href="http://www.cloudera.com/content/cloudera/en/partners/partners-listing.html?category-21=cloudera-partners%3Acertified%2Fcertified&amp;q=elasticsearch">certified for CDH 5</a>.
<img src="https://api.contentstack.io/v2/assets/575e4c723dc542cb38c07da7/download?uid=bltbdcaec231e418988?uid=bltbdcaec231e418988" data-sys-asset-uid="blt01b51c447d60a326">
Since the beginning, <a href="http://www.elasticsearch.org/overview/hadoop/">Elasticsearch for Apache Hadoop 2.0</a> (es-hadoop) has been tested against the popular CDH distribution and, during the few last months, we have been working closely with Cloudera to complete the process. With this certification in place, we can offer users complete peace of mind that Elasticsearch is thoroughly tested and validated against the CDH environment.
Adding to our existing partnerships with <a href="http://www.elasticsearch.com/blog/elasticsearch-and-hortonworks-partner-what-you-need-to-know/">Hortonworks/HDP</a> and <a href="http://www.elasticsearch.com/blog/elasticsearch-mapr-hadoop-best-worlds/">MapR</a>, es-hadoop users can rest assured that whatever <a href="http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/requirements.html#requirements-hadoop">Hadoop distribution</a> they use, we at Elasticsearch are fully committed to supporting it.
We are delighted to see es-hadoop used in production by businesses in a variety of fields, from social recommendations to financial services, multi-national media companies to global telecoms. Organizations in each of these industries leverage es-hadoop to gather insight and perform analytics on massive volumes of data, and we&#x2019;re proud we can help them achieve their analysis goals in real-time.
We&#x2019;ll continue to expand on es-hadoop&#x2019;s feature set to bring Elasticsearch&#x2019;s rich experience to users of CDH 5 and all other flavors of Hadoop. We know your business is hungry for data, and we&#x2019;re proud to serve up the best possible search and analytics experience. But don&#x2019;t take our word for it - <a href="http://www.elasticsearch.org/downloads/elasticsearch-for-apache-hadoop-2-0-0/">give it a spin</a> and let us know what you think!
If you are interested in data exploration, search and identifying anomalies in real-time on your Hadoop data, you are warmly invited to an upcoming webinar by yours truly on Wednesday August 20th, which will showcase the above through Elasticsearch for Apache Hadoop. <a href="http://www.elasticsearch.org/webinars/elasticsearch-and-apache-hadoop/">Registration is now open.</a>
</body></html>

Elasticsearch Hadoop certified for Cloudera CDH5

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Software Engineer

Costin Leau

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

O conteúdo desta página não está disponível no idioma selecionado. A Elastic está trabalhando para garantir que o conteúdo esteja disponível em vários idiomas. Agradecemos a compreensão.

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/elastic-cloud-account-terms" target="_self">Termos de Serviço</a> e também concorda em receber <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">comunicações especiais</a> por meio dos dados de contato que forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/serverless-preview-terms" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Ao enviar, você concorda com os <a href="/pt/agreements/global/cloud-services-monthly" rel="nofollow">Termos de Serviço do Elastic Cloud</a>. Seus dados pessoais serão tratados de acordo com a <a href="/pt/legal/privacy-statement">Declaração de Privacidade da Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/terms-of-use" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

Explore similar demos

Explore outras demonstrações semelhantes

Overview

Load More

Speakers

Close

See more insights

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Consultar a hora de início do webinar no meu fuso horário

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Não vai conseguir participar? Inscreva-se e nós lhe enviaremos a gravação. Você também receberá um email com conteúdo relacionado.

Author

Articles by Costin Leau

Software Engineer, Elastic

New to the Elasticsearch repository: ES|QL

Elasticsearch for Apache Hadoop 2.3.0 and 2.2.1 released

Elasticsearch for Apache Hadoop 2.2.0 and 2.1.3 released

Elasticsearch for Apache Hadoop 2.2 RC1 is out

Elasticsearch for Apache Hadoop 2.1 GA: Spark, Storm and More

Elasticsearch Hadoop certified for Cloudera CDH5