Update November 2, 2015: Please note that facets are not supported in <a href="https://www.elastic.co/blog/elasticsearch-2-0-0-released">Elasticsearch 2.0+</a>. 
For those of you familiar with 
	<a href="/products/elasticsearch">Elasticsearch</a>, you know that its an amazing modern, scalable, full-text search engine with <a href="http://lucene.apache.org/" target="_blank">Apache Lucene</a> and the <a href="http://en.wikipedia.org/wiki/Inverted_index" target="_blank">inverted index</a> at its core. Elasticsearch allows users to query their data and provides efficient and blazingly fast look up of documents that make it perfect for creating real-time analytics dashboards.

	Currently, Elasticsearch includes 
	<a href="http://en.wikipedia.org/wiki/Faceted_search" target="_blank">faceted search</a>, a functionality that allows users to compute aggregations of their data. For example, a user with twitter data could create buckets for the number of tweets per year, quarter, month, day, week, hour, or minute using the <a href="/guide/reference/api/search/facets/date-histogram-facet.html">date histogram facet</a>, making it quite simple to create histograms.

	Faceted search is a powerful tool for data visualization. 
	<a href="/products/kibana">Kibana</a> is a great example of a front-end interface that makes good use of facets. However, there are some major restrictions to faceting. Facets do not retain information about which documents fall into which buckets, making complex querying difficult. Which is why, Elasticsearch is pleased to introduce the <a href="/guide/en/elasticsearch/reference/master/search-aggregations.html">aggregations</a> framework with the 1.0 release. Aggregations rips apart its faceting restraints and provides developers the potential to do much more with visualizations.
<h2>Aggregations (=Awesomeness!)</h2>
	Aggregations is "faceting reborn". Aggregations incorporate all of the faceting functionality while also providing much more powerful capabilities. Aggregations is a generic but extremely powerful framework for building any type of aggregation. There are several different types of aggregations, but they fall into two main categories: bucketing and metric. Bucketing aggregations produce a list of buckets, each one with a set of documents that belong to it (e.g., 
	<a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-terms-aggregation.html">terms</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-range-aggregation.html">range</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-daterange-aggregation.html">date range</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-histogram-aggregation.html">histogram</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-datehistogram-aggregation.html">date histogram</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-geodistance-aggregation.html">geo distance</a>). Metric aggregations keep track and compute metrics over a set of documents (e.g., <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-min-aggregation.html">min</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-max-aggregation.html">max</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-sum-aggregation.html">sum</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-avg-aggregation.html">avg</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-stats-aggregation.html">stats</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-extendedstats-aggregation.html">extended stats</a>).
<h2>Using Aggregations for Data Visualization (with D3)</h2>
	Lets dive right in and see the power that aggregations give us for data visualization. We will create a donut chart and a dendrogram using the Elasticsearch aggregations framework, the 
	<a href="/guide/en/elasticsearch/client/javascript-api/current/index.html">Elasticsearch javascript client</a>, and <a href="http://d3js.org/" target="_blank">D3</a>.

	If you are new to Elasticsearch, it is very easy to get started. Visit the Elasticsearch 
	<a href="/products/elasticsearch" target="_blank">overview</a> page to learn how to download, install, and run Elasticsearch version 1.0.
<h3>Requirements:</h3><ol>
	<li><a href="https://download.elasticsearch.org/elasticsearch/elasticsearch-js/elasticsearch-js-master.zip">elasticsearch.js</a></li>
	<li><a href="https://github.com/mbostock/d3/releases/download/v3.4.1/d3.v3.zip" target="_blank">d3.v3.js</a>, or include the D3 source in a script tag in the html</li>
	<li><a href="http://requirejs.org/docs/release/2.1.10/comments/require.js" target="_blank">require.js</a> (save file as require.js)</li>
	<li>nfl_2013.json and&nbsp;nfl_mapping.json (<a href="https://github.com/stormpython/Elasticsearch-datasets/archive/master.zip" target="_blank">download</a>)</li>
	<li>a text editor</li>
	<li>a&nbsp;<a href="http://www.linuxjournal.com/content/tech-tip-really-simple-http-server-python" target="_blank">webserver</a></li>
</ol>
	* a link to the index.html and main javascript file can be found 
	<a href="https://gist.github.com/stormpython/8796112" target="_blank">here</a>. Alternatively, you can clone the <a href="https://github.com/stormpython/nfl-elastic-aggregations" target="_blank">project</a> on github.
<h2>Uploading Data to our Elasticsearch Index</h2>
	We will need to create an index and add some data.

	Since the (American) football season just ended, let's send it off by exploring some NFL data. For this tutorial, we will only be concerned with 
	<a href="http://en.wikipedia.org/wiki/Touchdown" target="_blank">touchdowns</a>. Download the dataset (nfl_2013.json) and its mappings (nfl_mapping.json) using the link above.

	Fire up Elasticsearch from your localhost. Now, let's add an index and some data. From the terminal:
<pre class="prettyprint" lang="bsh">curl -XPOST localhost:9200/nfl?pretty
curl -XPUT localhost:9200/nfl/2013/_mapping?pretty -d @nfl_mapping.json
curl -XPOST localhost:9200/nfl/2013/_bulk?pretty --data-binary @nfl_2013.json
</pre>
	The NFL data file consists of play-by-play data with 18 fields.
<pre class="prettyprint" lang="json">{
 def: "HOU", // defensive team
 defscore: "13", // score for defensive team
 description: "(11:34) S.Ridley right guard for 8 yards TOUCHDOWN.", // play description
 down: "2", // down for current play (up to 4)
 gameid: "20130113_HOU@NE", // game id with date and teams
 min: "26", // minutes remaining in game
 nextscore: "7", // ???
 off: "NE", // offensive team
 offscore: "17", // score for offensive team
 qtr: "3", // quarter
 scorechange: "0", // amount offensive score changed
 scorediff: "4", // difference in score
 season: "2012", // nfl season (year)
 sec: "34", // seconds remaining within minute of play
 series1stdn: "1", // ???
 teamwin: "1", // either 1 or 0, demarks the team winning
 togo: "6", // number of yards to go to reach a first down
 ydline: "8" // yard line ball is on to start play
}
</pre><h2>Setting up our HTML and Javascript Files</h2>
	Lets begin by creating a directory for our project. Lets call it nfl. Now add index.html and a scripts subdirectory. Within scripts, place our downloaded elasticsearch.js, d3.js, and require.js files. We will also create our main.js file within scripts.

	<img src="https://api.contentstack.io/v2/assets/575e4d909e7a83165490ebef/download?uid=blte7d029a25ada9ab9?uid=blte7d029a25ada9ab9" height="118" width="300">

	Let's add some markup to index.html.
<pre class="dontquote prettyprint" lang="html">&lt;!DOCTYPE html&gt;
&lt;html&gt;
 &lt;head&gt;
 &lt;title&gt;Elastic Aggregations&lt;/title&gt;
 &lt;script src="scripts/require.js"&gt;&lt;/script&gt;
 &lt;script&gt;require(["scripts/main"], function () {})&lt;/script&gt;
 &lt;style&gt;&lt;/style&gt;
 &lt;/head&gt;
 &lt;body&gt;
 &lt;div id="donut-chart"&gt;&lt;/div&gt;
 &lt;/body&gt;
&lt;/html&gt;
</pre>
	Now, let's add some javascript to main.js.
<pre class="prettyprint" lang="js">define(['scripts/d3.v3', 'scripts/elasticsearch'], function (d3, elasticsearch) {
 "use strict";
 var client = new elasticsearch.Client();
});
</pre><h2 dir="ltr">Donut Chart with Terms Aggregation</h2>
	Let's start by making a donut chart of the number of touchdowns scored in each quarter. We will use the 
	<a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-terms-aggregation.html">terms aggregation</a> to accomplish this. The same thing can be done using the <a href="/guide/en/elasticsearch/reference/current/search-facets-terms-facet.html#_ordering">terms facet</a>, but this will demonstrate how aggregations can accomplish the same things as facets.

	Since there are typically only 4 quarters in an NFL game, we should end up with a donut chart with 4 slices. To ensure that we only return 4 quarters we will filter out overtime quarters. We will also need to filter for touchdowns.

	So lets create our query using the Elasticsearch javascript client and add it to main.js.
<pre class="prettyprint" lang="js">define(['scripts/d3.v3', 'scripts/elasticsearch'], function (d3, elasticsearch) {
 "use strict";
 var client = new elasticsearch.Client();
 client.search({
 index: 'nfl',
 size: 5,
 body: {
 // Begin query.
 query: {
 // Boolean query for matching and excluding items.
 bool: {
 must: { match: { "description": "TOUCHDOWN" }},
 must_not: { match: { "qtr": 5 }}
 }
 },
 // Aggregate on the results
 aggs: {
 touchdowns: {
 terms: {
 field: "qtr",
 // order by quarter, ascending
 order: { "_term" : "asc" }
 }
 }
 }
 // End query.
 }
 }).then(function (resp) {
 console.log(resp);
 // D3 code goes here.
 });
});
</pre>
	If we fire up a webserver, open up our browser, and navigate to the console under developer tools, you should see the output. Voila, data!

	<img src="https://api.contentstack.io/v2/assets/575e4d900342dfd738265e8f/download?uid=bltf88fc86d2a67c783?uid=bltf88fc86d2a67c783" height="356" width="595">

	Now, let's create a donut chart. Add your donut chart d3 code to main.js.
<pre class="prettyprint" lang="js">...
 }).then(function (resp) {
 console.log(resp);
 // D3 code goes here.
 var touchdowns = resp.aggregations.touchdowns.buckets;
 // d3 donut chart
 var width = 600,
 height = 300,
 radius = Math.min(width, height) / 2;
 var color = ['#ff7f0e', '#d62728', '#2ca02c', '#1f77b4'];
 var arc = d3.svg.arc()
 .outerRadius(radius - 60)
 .innerRadius(120);
 var pie = d3.layout.pie()
 .sort(null)
 .value(function (d) { return d.doc_count; });
 var svg = d3.select("#donut-chart").append("svg")
 .attr("width", width)
 .attr("height", height)
 .append("g")
 .attr("transform", "translate(" + width/1.4 + "," + height/2 + ")");
 var g = svg.selectAll(".arc")
 .data(pie(touchdowns))
 .enter()
 .append("g")
 .attr("class", "arc");
 g.append("path")
 .attr("d", arc)
 .style("fill", function (d, i) { return color[i]; });
 g.append("text")
 .attr("transform", function (d) { return "translate(" + arc.centroid(d) + ")"; })
 .attr("dy", ".35em")
 .style("text-anchor", "middle")
 .style("fill", "white")
 .text(function (d) { return d.data.key; });
 });
});
</pre>
	Let's add some styling to index.html.
<pre class="dontquote prettyprint" lang="html">&lt;style&gt;
 body {
 font: 14px sans-serif;
 }
 .arc path {
 stroke: #fff; 
 stroke-width: 3px;
 }
&lt;/style&gt;
</pre>
	Refresh your page in the browser, and you should see a nice donut chart.

	<img src="https://api.contentstack.io/v2/assets/575e4d90d8edd48f76938a08/download?uid=blt61674effbc538cb9?uid=blt61674effbc538cb9" height="220" width="278">

	Of course, you can create the same donut chart using a terms facet, no surprises here. So, let's try something a bit more complex.
<h2>Dendrogram with Nested Terms Aggregations</h2>
	Let's say we want to know for the 2013 season (through the 12th week), for each team, the name of the player(s) who scored touchdowns and the total number of touchdowns they scored in each quarter. For example, for the Denver Broncos (an NFL team), Peyton Manning (a player) was responsible for 36 touchdowns with:
<ol>
	<li>9 touchdowns in the 1st qtr</li>
	<li>6 touchdowns in the 2nd qtr</li>
	<li>12 touchdowns in the 3rd qtr</li>
	<li>9 touchdowns in the 4th qtr</li>
</ol>
	Let's write this query. We will need to filter out incomplete passes, interceptions, fumbles, and over-turned plays to get a more accurate number.
<pre class="prettyprint" lang="js">define(['scripts/d3.v3', 'scripts/elasticsearch'], function (d3, elasticsearch) {
 "use strict";
 var client = new elasticsearch.Client();
 client.search({
 index: 'nfl',
 size: 5,
 body: {
 query: {
 bool: {
 must: { match: { "description": "TOUCHDOWN"}},
 must_not: [
 { match: { "description": "intercepted"}},
 { match: { "description": "incomplete"}},
 { match: { "description": "FUMBLES"}},
 { match: { "description": "NULLIFIED"}}
 ]
 }
 },
 aggs: {
 teams: {
 terms: {
 field: "off",
 exclude: "", // exclude empty strings.
 size: 5 // limit to top 5 teams (out of 32).
 },
 aggs: {
 players: {
 terms: {
 field: "description",
 include: "([a-z]?[.][a-z]+)", // regex to pull out player names.
 size: 20 // limit to top 20 players per team. 
 },
 aggs: {
 qtrs: {
 terms: {
 field: "qtr"
 }
 }
 }
 }
 }
 }
 }
 }
 }).then(function (resp) {
 console.log(resp);
 // D3 code goes here.
 });
});
</pre>
	You can check the results in the console.

	<img src="https://api.contentstack.io/v2/assets/575e4d91d8edd48f76938a21/download?uid=blt0c061456d9582b5f?uid=blt0c061456d9582b5f" height="298" width="300">

	* Note that the team nodes have a touchdown count that is less than the sum of its leaf nodes. This is because touchdowns that resulted from passes are counted twice, once for the player catching the pass and once for the&nbsp;
	<a href="http://en.wikipedia.org/wiki/Quarterback">quarterback</a>&nbsp;(e.g. Peyton Manning) throwing the pass.

	Let's create our dendrogram. We will modify index.html.
<pre class="dontquote prettyprint" lang="html">...
&lt;body&gt;
 &lt;div id="donut-chart"&gt;&lt;/div&gt;
 &lt;div id="dendrogram"&gt;&lt;/div&gt;
&lt;/body&gt;
...
</pre>
	Add your d3 code to main.js.
<pre class="dontquote prettyprint" lang="js">...
}).then(function (resp) {
 console.log(resp);
 // D3 code goes here.
 var root = createChildNodes(resp);
 // d3 dendrogram
 var width = 600,
 height = 2000;
 var color = ['#ff7f0e', '#d62728', '#2ca02c', '#1f77b4'];
 var cluster = d3.layout.cluster()
 .size([height, width - 200]);
 var diagonal = d3.svg.diagonal()
 .projection(function(d) { return [d.y, d.x]; });
 var svg = d3.select("#dendrogram").append("svg")
 .attr("width", width)
 .attr("height", height)
 .append("g")
 .attr("transform", "translate(120,0)");
 var nodes = cluster.nodes(root),
 links = cluster.links(nodes);
 var link = svg.selectAll(".link")
 .data(links)
 .enter().append("path")
 .attr("class", "link")
 .attr("d", diagonal);
 var node = svg.selectAll(".node")
 .data(nodes)
 .enter().append("g")
 .attr("class", "node")
 .attr("transform", function(d) { return "translate(" + d.y + "," + d.x + ")"; });
 node.append("circle")
 .attr("r", 4.5)
 .style("fill", function (d) {
 return d.children ? "#ffffff" : color[d.key - 1];
 })
 .style("stroke", function (d) {
 return d.children ? "#4682B4" : color[d.key - 1];
 });
 node.append("text")
 .attr("dx", function(d) { return d.children ? -8 : 8; })
 .attr("dy", 3)
 .style("text-anchor", function(d) { return d.children ? "end" : "start"; })
 .text(function(d) { return d.children? d.key : d.key + ": " + d.doc_count; });
 d3.select(self.frameElement).style("height", height + "px");
 function createChildNodes(dataObj) {
 var root = {};
 root.key = "NFL";
 root.children = dataObj.aggregations.teams.buckets;
 root.children.forEach(function (d) { d.children = d.players.buckets; });
 root.children.forEach(function (d) { 
 d.children.forEach(function (d) { 
 d.children = d.qtrs.buckets; 
 });
 });
 return root;
 }
});
...
</pre>
	And finally, let's add the styling.
<pre class="dontquote prettyprint" lang="html">&lt;style&gt;
 ...
 .node circle {
 fill: #fff;
 stroke: steelblue;
 stroke-width: 1.5px;
 }
 .node {
 font: 10px sans-serif;
 }
 .link {
 fill: none;
 stroke: #ccc;
 stroke-width: 1.5px;
 }
&lt;/style&gt;
</pre>
	Refresh your browser and voila, dendrogram goodness!
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf38746e3027e48c5/5bbda559192fad64364a4d79/dendrogram1.png" data-sys-asset-uid="bltf38746e3027e48c5" alt="dendrogram1.png" style="background-color: initial;">
As you can see, aggregations are a powerful tool for creating visualizations. Of course, you can create much more rich and interactive visualizations than what we have created here, but hopefully this will get you started. Happy searching!

data-visualisation-elasticsearch.png

Data visualization with Elasticsearch aggregations and D3

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Shelby Sturgis

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

El contenido de esta página no está disponible en el idioma seleccionado. Elastic está trabajando para garantizar que el contenido esté disponible en varios idiomas. Gracias por tu paciencia mientras trabajamos.

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Al enviar, confirma que ha leído y acepta nuestras <a href="/es/legal/elastic-cloud-account-terms" target="_self">Condiciones de servicio</a>, y que acepta recibir las <a href="/es/legal/privacy-statement#how-we-use-the-information" target="_self">comunicaciones seleccionadas</a> en los detalles de contacto que proporcionó anteriormente. Consulte la <a href="/es/legal/privacy-statement/" target="_self">Declaración de privacidad de Elastic</a> para obtener más información o para cancelar su suscripción en cualquier momento.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Al enviar, confirma que ha leído y acepta nuestras <a href="/es/legal/serverless-preview-terms" target="_self">Condiciones de servicio</a>, y que Elastic puede <a href="/es/legal/privacy-statement#how-we-use-the-information" target="_self">ponerse en contacto con usted</a> acerca de nuestros productos y servicios relacionados. Consulte la <a href="/es/legal/privacy-statement/" target="_self">Declaración de privacidad de Elastic</a> para obtener más información o para cancelar su suscripción en cualquier momento.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Al enviar esta información, aceptas los <a href="/es/agreements/global/cloud-services-monthly" rel="nofollow">Términos de servicio de Elastic</a>. Los datos personales se procesarán de conformidad con la <a href="/es/legal/privacy-statement">declaración de privacidad de Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Al enviar, confirma que ha leído y acepta nuestras <a href="/es/legal/terms-of-use" target="_self">Condiciones de servicio</a>, y que Elastic puede <a href="/es/legal/privacy-statement#how-we-use-the-information" target="_self">ponerse en contacto con usted</a> acerca de nuestros productos y servicios relacionados. Consulte la <a href="/es/legal/privacy-statement/" target="_self">Declaración de privacidad de Elastic</a> para obtener más información o para cancelar su suscripción en cualquier momento.

Hosted by

All Solutions

Follow us on Youtube

Watch now (no PT)

Articles by

Share by Email

Filters

All (no PT translation)

Contact information

See when this webinar starts in my time zone

Ver cuándo comienza este webinar en mi zona horaria

Thank you for your interest!

Register to Attend

Small image for

Register to Watch

Continue reading

Featured webinar

What to explore next...

Reset all

Load more press releases

Sign In to Attend

Share

Upcoming webinar

Learn more

Search Integrations

Share on Reddit

Share on Twitter

Share on Facebook

Load more news

View more posts

On-demand webinar

Follow us on Facebook

Author

Press Release

You'll also receive an email with related content.

You'll also receive an email with related content

Además, te enviaremos contenidos relevantes

Print

Contact Info

Overview

Video for

Read less

Register now

Share this story

See All Posts

See all top stories

View next

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

¿No puedes asistir? Regístrate y te enviaremos la grabación. También recibirás un correo electrónico con contenido relacionado.

Follow us on LinkedIn

Highlights

Explore similar demos

Follow us on Twitter

Share on LinkedIn

Load More

See more insights

Speakers

Close

Thank you for registering. We will send you a confirmation email soon.

Gracias por registrarte. Te enviaremos un correo electrónico de confirmación pronto.

Author

Articles by Shelby Sturgis

Data visualization with Elasticsearch aggregations and D3

Sign up for Elastic Cloud free trial