Update November 2, 2015: Please note that facets are not supported in <a href="https://www.elastic.co/blog/elasticsearch-2-0-0-released">Elasticsearch 2.0+</a>. 
For those of you familiar with 
	<a href="/products/elasticsearch">Elasticsearch</a>, you know that its an amazing modern, scalable, full-text search engine with <a href="http://lucene.apache.org/" target="_blank">Apache Lucene</a> and the <a href="http://en.wikipedia.org/wiki/Inverted_index" target="_blank">inverted index</a> at its core. Elasticsearch allows users to query their data and provides efficient and blazingly fast look up of documents that make it perfect for creating real-time analytics dashboards.

	Currently, Elasticsearch includes 
	<a href="http://en.wikipedia.org/wiki/Faceted_search" target="_blank">faceted search</a>, a functionality that allows users to compute aggregations of their data. For example, a user with twitter data could create buckets for the number of tweets per year, quarter, month, day, week, hour, or minute using the <a href="/guide/reference/api/search/facets/date-histogram-facet.html">date histogram facet</a>, making it quite simple to create histograms.

	Faceted search is a powerful tool for data visualization. 
	<a href="/products/kibana">Kibana</a> is a great example of a front-end interface that makes good use of facets. However, there are some major restrictions to faceting. Facets do not retain information about which documents fall into which buckets, making complex querying difficult. Which is why, Elasticsearch is pleased to introduce the <a href="/guide/en/elasticsearch/reference/master/search-aggregations.html">aggregations</a> framework with the 1.0 release. Aggregations rips apart its faceting restraints and provides developers the potential to do much more with visualizations.
<h2>Aggregations (=Awesomeness!)</h2>
	Aggregations is "faceting reborn". Aggregations incorporate all of the faceting functionality while also providing much more powerful capabilities. Aggregations is a generic but extremely powerful framework for building any type of aggregation. There are several different types of aggregations, but they fall into two main categories: bucketing and metric. Bucketing aggregations produce a list of buckets, each one with a set of documents that belong to it (e.g., 
	<a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-terms-aggregation.html">terms</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-range-aggregation.html">range</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-daterange-aggregation.html">date range</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-histogram-aggregation.html">histogram</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-datehistogram-aggregation.html">date histogram</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-geodistance-aggregation.html">geo distance</a>). Metric aggregations keep track and compute metrics over a set of documents (e.g., <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-min-aggregation.html">min</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-max-aggregation.html">max</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-sum-aggregation.html">sum</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-avg-aggregation.html">avg</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-stats-aggregation.html">stats</a>, <a href="/guide/en/elasticsearch/reference/master/search-aggregations-metrics-extendedstats-aggregation.html">extended stats</a>).
<h2>Using Aggregations for Data Visualization (with D3)</h2>
	Lets dive right in and see the power that aggregations give us for data visualization. We will create a donut chart and a dendrogram using the Elasticsearch aggregations framework, the 
	<a href="/guide/en/elasticsearch/client/javascript-api/current/index.html">Elasticsearch javascript client</a>, and <a href="http://d3js.org/" target="_blank">D3</a>.

	If you are new to Elasticsearch, it is very easy to get started. Visit the Elasticsearch 
	<a href="/products/elasticsearch" target="_blank">overview</a> page to learn how to download, install, and run Elasticsearch version 1.0.
<h3>Requirements:</h3><ol>
	<li><a href="https://download.elasticsearch.org/elasticsearch/elasticsearch-js/elasticsearch-js-master.zip">elasticsearch.js</a></li>
	<li><a href="https://github.com/mbostock/d3/releases/download/v3.4.1/d3.v3.zip" target="_blank">d3.v3.js</a>, or include the D3 source in a script tag in the html</li>
	<li><a href="http://requirejs.org/docs/release/2.1.10/comments/require.js" target="_blank">require.js</a> (save file as require.js)</li>
	<li>nfl_2013.json and&nbsp;nfl_mapping.json (<a href="https://github.com/stormpython/Elasticsearch-datasets/archive/master.zip" target="_blank">download</a>)</li>
	<li>a text editor</li>
	<li>a&nbsp;<a href="http://www.linuxjournal.com/content/tech-tip-really-simple-http-server-python" target="_blank">webserver</a></li>
</ol>
	* a link to the index.html and main javascript file can be found 
	<a href="https://gist.github.com/stormpython/8796112" target="_blank">here</a>. Alternatively, you can clone the <a href="https://github.com/stormpython/nfl-elastic-aggregations" target="_blank">project</a> on github.
<h2>Uploading Data to our Elasticsearch Index</h2>
	We will need to create an index and add some data.

	Since the (American) football season just ended, let's send it off by exploring some NFL data. For this tutorial, we will only be concerned with 
	<a href="http://en.wikipedia.org/wiki/Touchdown" target="_blank">touchdowns</a>. Download the dataset (nfl_2013.json) and its mappings (nfl_mapping.json) using the link above.

	Fire up Elasticsearch from your localhost. Now, let's add an index and some data. From the terminal:
<pre class="prettyprint" lang="bsh">curl -XPOST localhost:9200/nfl?pretty
curl -XPUT localhost:9200/nfl/2013/_mapping?pretty -d @nfl_mapping.json
curl -XPOST localhost:9200/nfl/2013/_bulk?pretty --data-binary @nfl_2013.json
</pre>
	The NFL data file consists of play-by-play data with 18 fields.
<pre class="prettyprint" lang="json">{
 def: "HOU", // defensive team
 defscore: "13", // score for defensive team
 description: "(11:34) S.Ridley right guard for 8 yards TOUCHDOWN.", // play description
 down: "2", // down for current play (up to 4)
 gameid: "20130113_HOU@NE", // game id with date and teams
 min: "26", // minutes remaining in game
 nextscore: "7", // ???
 off: "NE", // offensive team
 offscore: "17", // score for offensive team
 qtr: "3", // quarter
 scorechange: "0", // amount offensive score changed
 scorediff: "4", // difference in score
 season: "2012", // nfl season (year)
 sec: "34", // seconds remaining within minute of play
 series1stdn: "1", // ???
 teamwin: "1", // either 1 or 0, demarks the team winning
 togo: "6", // number of yards to go to reach a first down
 ydline: "8" // yard line ball is on to start play
}
</pre><h2>Setting up our HTML and Javascript Files</h2>
	Lets begin by creating a directory for our project. Lets call it nfl. Now add index.html and a scripts subdirectory. Within scripts, place our downloaded elasticsearch.js, d3.js, and require.js files. We will also create our main.js file within scripts.

	<img src="https://api.contentstack.io/v2/assets/575e4d909e7a83165490ebef/download?uid=blte7d029a25ada9ab9?uid=blte7d029a25ada9ab9" height="118" width="300">

	Let's add some markup to index.html.
<pre class="dontquote prettyprint" lang="html">&lt;!DOCTYPE html&gt;
&lt;html&gt;
 &lt;head&gt;
 &lt;title&gt;Elastic Aggregations&lt;/title&gt;
 &lt;script src="scripts/require.js"&gt;&lt;/script&gt;
 &lt;script&gt;require(["scripts/main"], function () {})&lt;/script&gt;
 &lt;style&gt;&lt;/style&gt;
 &lt;/head&gt;
 &lt;body&gt;
 &lt;div id="donut-chart"&gt;&lt;/div&gt;
 &lt;/body&gt;
&lt;/html&gt;
</pre>
	Now, let's add some javascript to main.js.
<pre class="prettyprint" lang="js">define(['scripts/d3.v3', 'scripts/elasticsearch'], function (d3, elasticsearch) {
 "use strict";
 var client = new elasticsearch.Client();
});
</pre><h2 dir="ltr">Donut Chart with Terms Aggregation</h2>
	Let's start by making a donut chart of the number of touchdowns scored in each quarter. We will use the 
	<a href="/guide/en/elasticsearch/reference/master/search-aggregations-bucket-terms-aggregation.html">terms aggregation</a> to accomplish this. The same thing can be done using the <a href="/guide/en/elasticsearch/reference/current/search-facets-terms-facet.html#_ordering">terms facet</a>, but this will demonstrate how aggregations can accomplish the same things as facets.

	Since there are typically only 4 quarters in an NFL game, we should end up with a donut chart with 4 slices. To ensure that we only return 4 quarters we will filter out overtime quarters. We will also need to filter for touchdowns.

	So lets create our query using the Elasticsearch javascript client and add it to main.js.
<pre class="prettyprint" lang="js">define(['scripts/d3.v3', 'scripts/elasticsearch'], function (d3, elasticsearch) {
 "use strict";
 var client = new elasticsearch.Client();
 client.search({
 index: 'nfl',
 size: 5,
 body: {
 // Begin query.
 query: {
 // Boolean query for matching and excluding items.
 bool: {
 must: { match: { "description": "TOUCHDOWN" }},
 must_not: { match: { "qtr": 5 }}
 }
 },
 // Aggregate on the results
 aggs: {
 touchdowns: {
 terms: {
 field: "qtr",
 // order by quarter, ascending
 order: { "_term" : "asc" }
 }
 }
 }
 // End query.
 }
 }).then(function (resp) {
 console.log(resp);
 // D3 code goes here.
 });
});
</pre>
	If we fire up a webserver, open up our browser, and navigate to the console under developer tools, you should see the output. Voila, data!

	<img src="https://api.contentstack.io/v2/assets/575e4d900342dfd738265e8f/download?uid=bltf88fc86d2a67c783?uid=bltf88fc86d2a67c783" height="356" width="595">

	Now, let's create a donut chart. Add your donut chart d3 code to main.js.
<pre class="prettyprint" lang="js">...
 }).then(function (resp) {
 console.log(resp);
 // D3 code goes here.
 var touchdowns = resp.aggregations.touchdowns.buckets;
 // d3 donut chart
 var width = 600,
 height = 300,
 radius = Math.min(width, height) / 2;
 var color = ['#ff7f0e', '#d62728', '#2ca02c', '#1f77b4'];
 var arc = d3.svg.arc()
 .outerRadius(radius - 60)
 .innerRadius(120);
 var pie = d3.layout.pie()
 .sort(null)
 .value(function (d) { return d.doc_count; });
 var svg = d3.select("#donut-chart").append("svg")
 .attr("width", width)
 .attr("height", height)
 .append("g")
 .attr("transform", "translate(" + width/1.4 + "," + height/2 + ")");
 var g = svg.selectAll(".arc")
 .data(pie(touchdowns))
 .enter()
 .append("g")
 .attr("class", "arc");
 g.append("path")
 .attr("d", arc)
 .style("fill", function (d, i) { return color[i]; });
 g.append("text")
 .attr("transform", function (d) { return "translate(" + arc.centroid(d) + ")"; })
 .attr("dy", ".35em")
 .style("text-anchor", "middle")
 .style("fill", "white")
 .text(function (d) { return d.data.key; });
 });
});
</pre>
	Let's add some styling to index.html.
<pre class="dontquote prettyprint" lang="html">&lt;style&gt;
 body {
 font: 14px sans-serif;
 }
 .arc path {
 stroke: #fff; 
 stroke-width: 3px;
 }
&lt;/style&gt;
</pre>
	Refresh your page in the browser, and you should see a nice donut chart.

	<img src="https://api.contentstack.io/v2/assets/575e4d90d8edd48f76938a08/download?uid=blt61674effbc538cb9?uid=blt61674effbc538cb9" height="220" width="278">

	Of course, you can create the same donut chart using a terms facet, no surprises here. So, let's try something a bit more complex.
<h2>Dendrogram with Nested Terms Aggregations</h2>
	Let's say we want to know for the 2013 season (through the 12th week), for each team, the name of the player(s) who scored touchdowns and the total number of touchdowns they scored in each quarter. For example, for the Denver Broncos (an NFL team), Peyton Manning (a player) was responsible for 36 touchdowns with:
<ol>
	<li>9 touchdowns in the 1st qtr</li>
	<li>6 touchdowns in the 2nd qtr</li>
	<li>12 touchdowns in the 3rd qtr</li>
	<li>9 touchdowns in the 4th qtr</li>
</ol>
	Let's write this query. We will need to filter out incomplete passes, interceptions, fumbles, and over-turned plays to get a more accurate number.
<pre class="prettyprint" lang="js">define(['scripts/d3.v3', 'scripts/elasticsearch'], function (d3, elasticsearch) {
 "use strict";
 var client = new elasticsearch.Client();
 client.search({
 index: 'nfl',
 size: 5,
 body: {
 query: {
 bool: {
 must: { match: { "description": "TOUCHDOWN"}},
 must_not: [
 { match: { "description": "intercepted"}},
 { match: { "description": "incomplete"}},
 { match: { "description": "FUMBLES"}},
 { match: { "description": "NULLIFIED"}}
 ]
 }
 },
 aggs: {
 teams: {
 terms: {
 field: "off",
 exclude: "", // exclude empty strings.
 size: 5 // limit to top 5 teams (out of 32).
 },
 aggs: {
 players: {
 terms: {
 field: "description",
 include: "([a-z]?[.][a-z]+)", // regex to pull out player names.
 size: 20 // limit to top 20 players per team. 
 },
 aggs: {
 qtrs: {
 terms: {
 field: "qtr"
 }
 }
 }
 }
 }
 }
 }
 }
 }).then(function (resp) {
 console.log(resp);
 // D3 code goes here.
 });
});
</pre>
	You can check the results in the console.

	<img src="https://api.contentstack.io/v2/assets/575e4d91d8edd48f76938a21/download?uid=blt0c061456d9582b5f?uid=blt0c061456d9582b5f" height="298" width="300">

	* Note that the team nodes have a touchdown count that is less than the sum of its leaf nodes. This is because touchdowns that resulted from passes are counted twice, once for the player catching the pass and once for the&nbsp;
	<a href="http://en.wikipedia.org/wiki/Quarterback">quarterback</a>&nbsp;(e.g. Peyton Manning) throwing the pass.

	Let's create our dendrogram. We will modify index.html.
<pre class="dontquote prettyprint" lang="html">...
&lt;body&gt;
 &lt;div id="donut-chart"&gt;&lt;/div&gt;
 &lt;div id="dendrogram"&gt;&lt;/div&gt;
&lt;/body&gt;
...
</pre>
	Add your d3 code to main.js.
<pre class="dontquote prettyprint" lang="js">...
}).then(function (resp) {
 console.log(resp);
 // D3 code goes here.
 var root = createChildNodes(resp);
 // d3 dendrogram
 var width = 600,
 height = 2000;
 var color = ['#ff7f0e', '#d62728', '#2ca02c', '#1f77b4'];
 var cluster = d3.layout.cluster()
 .size([height, width - 200]);
 var diagonal = d3.svg.diagonal()
 .projection(function(d) { return [d.y, d.x]; });
 var svg = d3.select("#dendrogram").append("svg")
 .attr("width", width)
 .attr("height", height)
 .append("g")
 .attr("transform", "translate(120,0)");
 var nodes = cluster.nodes(root),
 links = cluster.links(nodes);
 var link = svg.selectAll(".link")
 .data(links)
 .enter().append("path")
 .attr("class", "link")
 .attr("d", diagonal);
 var node = svg.selectAll(".node")
 .data(nodes)
 .enter().append("g")
 .attr("class", "node")
 .attr("transform", function(d) { return "translate(" + d.y + "," + d.x + ")"; });
 node.append("circle")
 .attr("r", 4.5)
 .style("fill", function (d) {
 return d.children ? "#ffffff" : color[d.key - 1];
 })
 .style("stroke", function (d) {
 return d.children ? "#4682B4" : color[d.key - 1];
 });
 node.append("text")
 .attr("dx", function(d) { return d.children ? -8 : 8; })
 .attr("dy", 3)
 .style("text-anchor", function(d) { return d.children ? "end" : "start"; })
 .text(function(d) { return d.children? d.key : d.key + ": " + d.doc_count; });
 d3.select(self.frameElement).style("height", height + "px");
 function createChildNodes(dataObj) {
 var root = {};
 root.key = "NFL";
 root.children = dataObj.aggregations.teams.buckets;
 root.children.forEach(function (d) { d.children = d.players.buckets; });
 root.children.forEach(function (d) { 
 d.children.forEach(function (d) { 
 d.children = d.qtrs.buckets; 
 });
 });
 return root;
 }
});
...
</pre>
	And finally, let's add the styling.
<pre class="dontquote prettyprint" lang="html">&lt;style&gt;
 ...
 .node circle {
 fill: #fff;
 stroke: steelblue;
 stroke-width: 1.5px;
 }
 .node {
 font: 10px sans-serif;
 }
 .link {
 fill: none;
 stroke: #ccc;
 stroke-width: 1.5px;
 }
&lt;/style&gt;
</pre>
	Refresh your browser and voila, dendrogram goodness!
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf38746e3027e48c5/5bbda559192fad64364a4d79/dendrogram1.png" data-sys-asset-uid="bltf38746e3027e48c5" alt="dendrogram1.png" style="background-color: initial;">
As you can see, aggregations are a powerful tool for creating visualizations. Of course, you can create much more rich and interactive visualizations than what we have created here, but hopefully this will get you started. Happy searching!

data-visualisation-elasticsearch.png

Data visualization with Elasticsearch aggregations and D3

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Shelby Sturgis

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/elastic-cloud-account-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">ausgewählte Nachrichten und Informationen</a> an die oben von Ihnen bereitgestellten Kontaktangaben zu senden. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/serverless-preview-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Beim Absenden akzeptieren Sie die <a href="/de/agreements/global/cloud-services-monthly" rel="nofollow">Nutzungsbedingungen von Elastic Cloud</a>. Die Verarbeitung Ihrer personenbezogenen Daten unterliegt der <a href="/de/legal/privacy-statement">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/terms-of-use" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Der Inhalt dieser Seite ist in der ausgewählten Sprache nicht verfügbar. Wir bei Elastic arbeiten daran, die bereitgestellten Inhalte in verschiedenen Sprachen anzubieten. Bis dahin bitten wir Sie um etwas Geduld und hoffen auf Ihr Verständnis!

Author

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Articles by

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Sie können nicht dabei sein? Registrieren Sie sich, damit wir Ihnen die Aufzeichnung zukommen lassen können. Außerdem erhalten Sie eine E‑Mail mit Begleitmaterial.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Learn More

Read less

You'll also receive an email with related content

Wir schicken Ihnen zudem relevante Informationen, die von Interesse sein könnten.

Thank you for registering. We will send you a confirmation email soon.

Vielen Dank für Ihre Registrierung. Sie erhalten in Kürze eine Bestätigungs-E‑Mail von uns.

Author

Articles by Shelby Sturgis

Data visualization with Elasticsearch aggregations and D3

Sign up for Elastic Cloud free trial