Update November 2, 2015: Please note that facets are not supported in <a href="https://www.elastic.co/blog/elasticsearch-2-0-0-released">Elasticsearch 2.0+</a>. 
For those of you familiar with 
	<a href="/products/elasticsearch">Elasticsearch</a>, you know that its an amazing modern, scalable, full-text search engine with <a href="http://lucene.apache.org/" target="_blank">Apache Lucene</a> and the <a href="http://en.wikipedia.org/wiki/Inverted_index" target="_blank">inverted index</a> at its core. Elasticsearch allows users to query their data and provides efficient and blazingly fast look up of documents that make it perfect for creating real-time analytics dashboards.

	Currently, Elasticsearch includes 
	<a href="http://en.wikipedia.org/wiki/Faceted_search" target="_blank">faceted search</a>, a functionality that allows users to compute aggregations of their data. For example, a user with twitter data could create buckets for the number of tweets per year, quarter, month, day, week, hour, or minute using the <a href="/guide/reference/api/search/facets/date-histogram-facet.html">date histogram facet</a>, making it quite simple to create histograms.

	Faceted search is a powerful tool for data visualization. 
	<a href="/products/kibana">Kibana</a> is a great example of a front-end interface that makes good use of facets. However, there are some major restrictions to faceting. Facets do not retain information about which documents fall into which buckets, making complex querying difficult. Which is why, Elasticsearch is pleased to introduce the <a href="/guide/en/elasticsearch/reference/master/search-aggregations.html">aggregations</a> framework with the 1.0 release. Aggregations rips apart its faceting restraints and provides developers the potential to do much more with visualizations.
<h2>Aggregations (=Awesomeness!)</h2>
	Aggregations is "faceting reborn". Aggregations incorporate all of the faceting functionality while also providing much more powerful capabilities. Aggregations is a generic but extremely powerful framework for building any type of aggregation. There are several different types of aggregations, but they fall into two main categories: bucketing and metric. Bucketing aggregations produce a list of buckets, each one with a set of documents that belong to it (e.g., 
	<a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-terms-aggregation.html">terms</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-range-aggregation.html">range</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-daterange-aggregation.html">date range</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-histogram-aggregation.html">histogram</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-datehistogram-aggregation.html">date histogram</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-geodistance-aggregation.html">geo distance</a>). Metric aggregations keep track and compute metrics over a set of documents (e.g., <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-min-aggregation.html">min</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-max-aggregation.html">max</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-sum-aggregation.html">sum</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-avg-aggregation.html">avg</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-stats-aggregation.html">stats</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-extendedstats-aggregation.html">extended stats</a>).
<h2>Using Aggregations for Data Visualization (with D3)</h2>
	Lets dive right in and see the power that aggregations give us for data visualization. We will create a donut chart and a dendrogram using the Elasticsearch aggregations framework, the 
	<a href="/guide/en/elasticsearch/client/javascript-api/current/index.html">Elasticsearch javascript client</a>, and <a href="http://d3js.org/" target="_blank">D3</a>.

	If you are new to Elasticsearch, it is very easy to get started. Visit the Elasticsearch 
	<a href="/products/elasticsearch" target="_blank">overview</a> page to learn how to download, install, and run Elasticsearch version 1.0.
<h3>Requirements:</h3><ol>
	<li><a href="https://download.elasticsearch.org/elasticsearch/elasticsearch-js/elasticsearch-js-master.zip">elasticsearch.js</a></li>
	<li><a href="https://github.com/mbostock/d3/releases/download/v3.4.1/d3.v3.zip" target="_blank">d3.v3.js</a>, or include the D3 source in a script tag in the html</li>
	<li><a href="http://requirejs.org/docs/release/2.1.10/comments/require.js" target="_blank">require.js</a> (save file as require.js)</li>
	<li>nfl_2013.json and&nbsp;nfl_mapping.json (<a href="https://github.com/stormpython/Elasticsearch-datasets/archive/master.zip" target="_blank">download</a>)</li>
	<li>a text editor</li>
	<li>a&nbsp;<a href="http://www.linuxjournal.com/content/tech-tip-really-simple-http-server-python" target="_blank">webserver</a></li>
</ol>
	* a link to the index.html and main javascript file can be found 
	<a href="https://gist.github.com/stormpython/8796112" target="_blank">here</a>. Alternatively, you can clone the <a href="https://github.com/stormpython/nfl-elastic-aggregations" target="_blank">project</a> on github.
<h2>Uploading Data to our Elasticsearch Index</h2>
	We will need to create an index and add some data.

	Since the (American) football season just ended, let's send it off by exploring some NFL data. For this tutorial, we will only be concerned with 
	<a href="http://en.wikipedia.org/wiki/Touchdown" target="_blank">touchdowns</a>. Download the dataset (nfl_2013.json) and its mappings (nfl_mapping.json) using the link above.

	Fire up Elasticsearch from your localhost. Now, let's add an index and some data. From the terminal:
<pre class="prettyprint" lang="bsh">curl -XPOST localhost:9200/nfl?pretty
curl -XPUT localhost:9200/nfl/2013/_mapping?pretty -d @nfl_mapping.json
curl -XPOST localhost:9200/nfl/2013/_bulk?pretty --data-binary @nfl_2013.json
</pre>
	The NFL data file consists of play-by-play data with 18 fields.
<pre class="prettyprint" lang="json">{
 def: "HOU", // defensive team
 defscore: "13", // score for defensive team
 description: "(11:34) S.Ridley right guard for 8 yards TOUCHDOWN.", // play description
 down: "2", // down for current play (up to 4)
 gameid: "20130113_HOU@NE", // game id with date and teams
 min: "26", // minutes remaining in game
 nextscore: "7", // ???
 off: "NE", // offensive team
 offscore: "17", // score for offensive team
 qtr: "3", // quarter
 scorechange: "0", // amount offensive score changed
 scorediff: "4", // difference in score
 season: "2012", // nfl season (year)
 sec: "34", // seconds remaining within minute of play
 series1stdn: "1", // ???
 teamwin: "1", // either 1 or 0, demarks the team winning
 togo: "6", // number of yards to go to reach a first down
 ydline: "8" // yard line ball is on to start play
}
</pre><h2>Setting up our HTML and Javascript Files</h2>
	Lets begin by creating a directory for our project. Lets call it nfl. Now add index.html and a scripts subdirectory. Within scripts, place our downloaded elasticsearch.js, d3.js, and require.js files. We will also create our main.js file within scripts.

	<img src="https://api.contentstack.io/v2/assets/575e4d909e7a83165490ebef/download?uid=blte7d029a25ada9ab9?uid=blte7d029a25ada9ab9" height="118" width="300">

	Let's add some markup to index.html.
<pre class="dontquote prettyprint" lang="html">&lt;!DOCTYPE html&gt;
&lt;html&gt;
 &lt;head&gt;
 &lt;title&gt;Elastic Aggregations&lt;/title&gt;
 &lt;script src="scripts/require.js"&gt;&lt;/script&gt;
 &lt;script&gt;require(["scripts/main"], function () {})&lt;/script&gt;
 &lt;style&gt;&lt;/style&gt;
 &lt;/head&gt;
 &lt;body&gt;
 &lt;div id="donut-chart"&gt;&lt;/div&gt;
 &lt;/body&gt;
&lt;/html&gt;
</pre>
	Now, let's add some javascript to main.js.
<pre class="prettyprint" lang="js">define(['scripts/d3.v3', 'scripts/elasticsearch'], function (d3, elasticsearch) {
 "use strict";
 var client = new elasticsearch.Client();
});
</pre><h2 dir="ltr">Donut Chart with Terms Aggregation</h2>
	Let's start by making a donut chart of the number of touchdowns scored in each quarter. We will use the 
	<a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-terms-aggregation.html">terms aggregation</a> to accomplish this. The same thing can be done using the <a href="/guide/en/elasticsearch/reference/current/search-facets-terms-facet.html#_ordering">terms facet</a>, but this will demonstrate how aggregations can accomplish the same things as facets.

	Since there are typically only 4 quarters in an NFL game, we should end up with a donut chart with 4 slices. To ensure that we only return 4 quarters we will filter out overtime quarters. We will also need to filter for touchdowns.

	So lets create our query using the Elasticsearch javascript client and add it to main.js.
<pre class="prettyprint" lang="js">define(['scripts/d3.v3', 'scripts/elasticsearch'], function (d3, elasticsearch) {
 "use strict";
 var client = new elasticsearch.Client();
 client.search({
 index: 'nfl',
 size: 5,
 body: {
 // Begin query.
 query: {
 // Boolean query for matching and excluding items.
 bool: {
 must: { match: { "description": "TOUCHDOWN" }},
 must_not: { match: { "qtr": 5 }}
 }
 },
 // Aggregate on the results
 aggs: {
 touchdowns: {
 terms: {
 field: "qtr",
 // order by quarter, ascending
 order: { "_term" : "asc" }
 }
 }
 }
 // End query.
 }
 }).then(function (resp) {
 console.log(resp);
 // D3 code goes here.
 });
});
</pre>
	If we fire up a webserver, open up our browser, and navigate to the console under developer tools, you should see the output. Voila, data!

	<img src="https://api.contentstack.io/v2/assets/575e4d900342dfd738265e8f/download?uid=bltf88fc86d2a67c783?uid=bltf88fc86d2a67c783" height="356" width="595">

	Now, let's create a donut chart. Add your donut chart d3 code to main.js.
<pre class="prettyprint" lang="js">...
 }).then(function (resp) {
 console.log(resp);
 // D3 code goes here.
 var touchdowns = resp.aggregations.touchdowns.buckets;
 // d3 donut chart
 var width = 600,
 height = 300,
 radius = Math.min(width, height) / 2;
 var color = ['#ff7f0e', '#d62728', '#2ca02c', '#1f77b4'];
 var arc = d3.svg.arc()
 .outerRadius(radius - 60)
 .innerRadius(120);
 var pie = d3.layout.pie()
 .sort(null)
 .value(function (d) { return d.doc_count; });
 var svg = d3.select("#donut-chart").append("svg")
 .attr("width", width)
 .attr("height", height)
 .append("g")
 .attr("transform", "translate(" + width/1.4 + "," + height/2 + ")");
 var g = svg.selectAll(".arc")
 .data(pie(touchdowns))
 .enter()
 .append("g")
 .attr("class", "arc");
 g.append("path")
 .attr("d", arc)
 .style("fill", function (d, i) { return color[i]; });
 g.append("text")
 .attr("transform", function (d) { return "translate(" + arc.centroid(d) + ")"; })
 .attr("dy", ".35em")
 .style("text-anchor", "middle")
 .style("fill", "white")
 .text(function (d) { return d.data.key; });
 });
});
</pre>
	Let's add some styling to index.html.
<pre class="dontquote prettyprint" lang="html">&lt;style&gt;
 body {
 font: 14px sans-serif;
 }
 .arc path {
 stroke: #fff; 
 stroke-width: 3px;
 }
&lt;/style&gt;
</pre>
	Refresh your page in the browser, and you should see a nice donut chart.

	<img src="https://api.contentstack.io/v2/assets/575e4d90d8edd48f76938a08/download?uid=blt61674effbc538cb9?uid=blt61674effbc538cb9" height="220" width="278">

	Of course, you can create the same donut chart using a terms facet, no surprises here. So, let's try something a bit more complex.
<h2>Dendrogram with Nested Terms Aggregations</h2>
	Let's say we want to know for the 2013 season (through the 12th week), for each team, the name of the player(s) who scored touchdowns and the total number of touchdowns they scored in each quarter. For example, for the Denver Broncos (an NFL team), Peyton Manning (a player) was responsible for 36 touchdowns with:
<ol>
	<li>9 touchdowns in the 1st qtr</li>
	<li>6 touchdowns in the 2nd qtr</li>
	<li>12 touchdowns in the 3rd qtr</li>
	<li>9 touchdowns in the 4th qtr</li>
</ol>
	Let's write this query. We will need to filter out incomplete passes, interceptions, fumbles, and over-turned plays to get a more accurate number.
<pre class="prettyprint" lang="js">define(['scripts/d3.v3', 'scripts/elasticsearch'], function (d3, elasticsearch) {
 "use strict";
 var client = new elasticsearch.Client();
 client.search({
 index: 'nfl',
 size: 5,
 body: {
 query: {
 bool: {
 must: { match: { "description": "TOUCHDOWN"}},
 must_not: [
 { match: { "description": "intercepted"}},
 { match: { "description": "incomplete"}},
 { match: { "description": "FUMBLES"}},
 { match: { "description": "NULLIFIED"}}
 ]
 }
 },
 aggs: {
 teams: {
 terms: {
 field: "off",
 exclude: "", // exclude empty strings.
 size: 5 // limit to top 5 teams (out of 32).
 },
 aggs: {
 players: {
 terms: {
 field: "description",
 include: "([a-z]?[.][a-z]+)", // regex to pull out player names.
 size: 20 // limit to top 20 players per team. 
 },
 aggs: {
 qtrs: {
 terms: {
 field: "qtr"
 }
 }
 }
 }
 }
 }
 }
 }
 }).then(function (resp) {
 console.log(resp);
 // D3 code goes here.
 });
});
</pre>
	You can check the results in the console.

	<img src="https://api.contentstack.io/v2/assets/575e4d91d8edd48f76938a21/download?uid=blt0c061456d9582b5f?uid=blt0c061456d9582b5f" height="298" width="300">

	* Note that the team nodes have a touchdown count that is less than the sum of its leaf nodes. This is because touchdowns that resulted from passes are counted twice, once for the player catching the pass and once for the&nbsp;
	<a href="http://en.wikipedia.org/wiki/Quarterback">quarterback</a>&nbsp;(e.g. Peyton Manning) throwing the pass.

	Let's create our dendrogram. We will modify index.html.
<pre class="dontquote prettyprint" lang="html">...
&lt;body&gt;
 &lt;div id="donut-chart"&gt;&lt;/div&gt;
 &lt;div id="dendrogram"&gt;&lt;/div&gt;
&lt;/body&gt;
...
</pre>
	Add your d3 code to main.js.
<pre class="dontquote prettyprint" lang="js">...
}).then(function (resp) {
 console.log(resp);
 // D3 code goes here.
 var root = createChildNodes(resp);
 // d3 dendrogram
 var width = 600,
 height = 2000;
 var color = ['#ff7f0e', '#d62728', '#2ca02c', '#1f77b4'];
 var cluster = d3.layout.cluster()
 .size([height, width - 200]);
 var diagonal = d3.svg.diagonal()
 .projection(function(d) { return [d.y, d.x]; });
 var svg = d3.select("#dendrogram").append("svg")
 .attr("width", width)
 .attr("height", height)
 .append("g")
 .attr("transform", "translate(120,0)");
 var nodes = cluster.nodes(root),
 links = cluster.links(nodes);
 var link = svg.selectAll(".link")
 .data(links)
 .enter().append("path")
 .attr("class", "link")
 .attr("d", diagonal);
 var node = svg.selectAll(".node")
 .data(nodes)
 .enter().append("g")
 .attr("class", "node")
 .attr("transform", function(d) { return "translate(" + d.y + "," + d.x + ")"; });
 node.append("circle")
 .attr("r", 4.5)
 .style("fill", function (d) {
 return d.children ? "#ffffff" : color[d.key - 1];
 })
 .style("stroke", function (d) {
 return d.children ? "#4682B4" : color[d.key - 1];
 });
 node.append("text")
 .attr("dx", function(d) { return d.children ? -8 : 8; })
 .attr("dy", 3)
 .style("text-anchor", function(d) { return d.children ? "end" : "start"; })
 .text(function(d) { return d.children? d.key : d.key + ": " + d.doc_count; });
 d3.select(self.frameElement).style("height", height + "px");
 function createChildNodes(dataObj) {
 var root = {};
 root.key = "NFL";
 root.children = dataObj.aggregations.teams.buckets;
 root.children.forEach(function (d) { d.children = d.players.buckets; });
 root.children.forEach(function (d) { 
 d.children.forEach(function (d) { 
 d.children = d.qtrs.buckets; 
 });
 });
 return root;
 }
});
...
</pre>
	And finally, let's add the styling.
<pre class="dontquote prettyprint" lang="html">&lt;style&gt;
 ...
 .node circle {
 fill: #fff;
 stroke: steelblue;
 stroke-width: 1.5px;
 }
 .node {
 font: 10px sans-serif;
 }
 .link {
 fill: none;
 stroke: #ccc;
 stroke-width: 1.5px;
 }
&lt;/style&gt;
</pre>
	Refresh your browser and voila, dendrogram goodness!
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf38746e3027e48c5/5bbda559192fad64364a4d79/dendrogram1.png" data-sys-asset-uid="bltf38746e3027e48c5" alt="dendrogram1.png" style="background-color: initial;">
As you can see, aggregations are a powerful tool for creating visualizations. Of course, you can create much more rich and interactive visualizations than what we have created here, but hopefully this will get you started. Happy searching!

data-visualisation-elasticsearch.png

Data visualization with Elasticsearch aggregations and D3

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Shelby Sturgis

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

O conteúdo desta página não está disponível no idioma selecionado. A Elastic está trabalhando para garantir que o conteúdo esteja disponível em vários idiomas. Agradecemos a compreensão.

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/elastic-cloud-account-terms" target="_self">Termos de Serviço</a> e também concorda em receber <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">comunicações especiais</a> por meio dos dados de contato que forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/serverless-preview-terms" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Ao enviar, você concorda com os <a href="/pt/agreements/global/cloud-services-monthly" rel="nofollow">Termos de Serviço do Elastic Cloud</a>. Seus dados pessoais serão tratados de acordo com a <a href="/pt/legal/privacy-statement">Declaração de Privacidade da Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/terms-of-use" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

Explore similar demos

Explore outras demonstrações semelhantes

Overview

Load More

Speakers

Close

See more insights

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Consultar a hora de início do webinar no meu fuso horário

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Não vai conseguir participar? Inscreva-se e nós lhe enviaremos a gravação. Você também receberá um email com conteúdo relacionado.

Author

Articles by Shelby Sturgis

Data visualization with Elasticsearch aggregations and D3

Sign up for Elastic Cloud free trial