elasticsearch-weekly-blog-banner.png

elasticsearch-weekly-blog-thumb.png

This Week in Elasticsearch - 2020-02-21

overview.png

A Profile a Day Keeps the Doctor Away: The Elasticsearch Search Profiler

timelion-dashboard-anomaly-detector.jpg

graph-watcher.svg

Implementing a Statistical Anomaly Detector in Elasticsearch - Part 3

Last week, we <a href="/blog/implementing-a-statistical-anomaly-detector-part-1">built a pipeline aggregation</a> which distills thousands of data-points into a handful of representative metrics. &nbsp;This forms the basis of Atlas, and does all the heavy lifting required to implement the anomaly detector. This week, we'll finish the implementation and generate some fun graphs.

The aggregation we built is designed to be run over a specific window of time: given a date range, it will emit a 90th percentile surprise value for each metric. To fully implement Atlas, we need to plot those 90th percentile values themselves over time. This functionality is not currently possible&nbsp;using just&nbsp;Pipeline aggs (although a "sliding histogram" functionality has been proposed which would fill the gap). &nbsp;

Instead, we are going to move the responsibility to <a href="https://github.com/elastic/timelion">TimeLion</a>, which is well suited for this type of post-processing (Timelion is a new {Re}search project to put fluent time-series manipulation inside Kibana; you <a href="https://www.elastic.co/blog/timelion-timeline">can read more about it here</a>).

<div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/k2yLxWQHuQcjHDCsBaBeYV.jpg" data-uuid="k2yLxWQHuQcjHDCsBaBeYV" data-v="4" data-type="inline">
</div>
If you revisit the simulator code, you'll see that <a href="https://github.com/polyfractal/playground/blob/master/hotcloud/src/query.rs#L206-L231">we run a series of queries</a> after the data has been generated. &nbsp;We slide our Pipeline agg across the data in one-hour increments (with a window size of 24 hours). &nbsp;We also use <code>filter_path</code> to minimize the response output: we don't actually care about the 60,000 buckets… we just want the "ninetieth_surprise" from each metric. &nbsp;Filtering the response cuts down network transfer considerably. &nbsp;The values are then indexed back into&nbsp;Elasticsearch so we can chart them later.

We pre-processed these values ahead of time in the simulator&nbsp;to simplify the demonstration, but in a real system you'd likely have a Watcher or cronjob executing the query every hour and saving the results.

<h2>Plotting 90th percentile surprise</h2>
With the heavy lifting done last week, we can turn to TimeLion to finish the implementation. The first order of business is to pull down the 90th values for a particular metric.&nbsp; We can do that with the following TimeLion syntax:

<pre class="prettyprint">.es('metric:0', metric='avg:value').label("#0 90th surprise")
</pre>
Which will generate a graph that looks something like this:

<img src="https://api.contentstack.io/v2/assets/575e4d5b58208ba076e2a399/download?uid=blt409736e0f2bc01bf?uid=blt409736e0f2bc01bf" style="line-height: 1.6em; background-color: initial;" data-sys-asset-uid="bltfcaa0498c1735021" alt="plotting-90th-percentile-surprise">

Well that looks fun! &nbsp;There is definitely&nbsp;something&nbsp;happening. &nbsp;Let's walk through what this chart means, since it is fundamental to how Atlas works:

<ul>
	<li>Last week, we calculated the "surprise" of each time-series: the deviation from it's own moving average</li>
	<li>We then collected the top 90th percentile of these "surprise" values, and are now plotting them over time</li>
	<li>Effectively, this chart is showing us the variability of the top "surprise" (deviations).</li>
	<li>A large spike in this chart means the data has become more surprising&nbsp;... the top 90th percentiles has changed drastically (up or down, since we used an absolute value to calculate surprise)&nbsp;</li>
</ul>
Effectively, if we see a spike we can conclude the underlying data has changed enough to shift our normal variance, likely due to a disruption. &nbsp;This is the heart of Atlas: don't watch your data because there is just too much. &nbsp;Instead, watch the variance of the&nbsp;90th percentile of&nbsp;deviations from the mean.

If you compare the above graph to the actual data for metric #0, you’ll see the stark difference:

<img src="https://api.contentstack.io/v2/assets/575e4d5b58208ba076e2a394/download?uid=bltd57ebe4279e3f0cd?uid=bltd57ebe4279e3f0cd" style="line-height: 1.6em; background-color: initial;" data-sys-asset-uid="blte6b217ca9a981a9a" alt="plotting-90th-percentile-surprise-difference">

<h2>Building the Atlas Dashboard</h2>
Of course, the trick is to now automatically identify those spikes and graph/alert on them. &nbsp;Let's start building that logic. &nbsp;Atlas alerts when the 90th percentile surprise is 3 standard deviations above the moving average. If you decompose that problem, you'll see several necessary&nbsp;components:

<ul>
	<li>A rolling, three standard deviation</li>
	<li>A rolling average of the data</li>
	<li>Adding the rolling std on top of the rolling data. &nbsp;This represents the "threshold" the data must stay below</li>
	<li>Alerting when the data breaches the "threshold"</li>
</ul>
First, we construct the rolling three standard deviations. &nbsp;We do this with a custom <code>movingstd()</code> function (see footnote for source, it is essentially identical to a <code>movingavg()</code> function), then multiply it by three to get the third sigma:

Note: I'm indenting all the queries to make them easier to read

<pre class="prettyprint">.es('metric:0', metric='avg:value')
 .movingstd(6)
 .multiply(3)
</pre>
Next we write a snippet to&nbsp;calculate a rolling average of the data itself:

<pre class="prettyprint">.es('metric:0', metric='avg:value')
 .movingaverage(6)
</pre>
And finally, we combine those two snippets by adding them together&nbsp;to create the "threshold". &nbsp;This will create a line that is three standard deviations above the moving average of the data:

<pre class="prettyprint">.es('metric:0', metric='avg:value')
 .movingaverage(6)
 .sum(
 .es('metric:0', metric='avg:value')
 .movingstd(6)
 .multiply(3)
 )
</pre>
Now that we have a "threshold", we can plot this with the original data and see how they compare:

<img src="https://api.contentstack.io/v2/assets/575e4d5c3dc542cb38c0a5ff/download?uid=blt7799a370525294f2?uid=blt7799a370525294f2" data-sys-asset-uid="blt7d789fb327782f94" alt="plotting-90th-percentile-surprise-threshold">

Hmm, ok. &nbsp;It's not really clear right now if the threshold is working or not. &nbsp;The chart is difficult to read;&nbsp;as soon as the surprise value spikes&nbsp;it causes a subsequent spike in the threshold. &nbsp;This happens because the spike causes a large change in variance, which the rolling stddev picks up, causing the threshold itself&nbsp;to shoot up. &nbsp;

If we zoom in on the first spike, we can see that the 90th percentile briefly passes the threshold before the rolling stddev "catches up":

<img src="https://api.contentstack.io/v2/assets/575e4d5b9e7a83165490e319/download?uid=bltbec0e6a60ea66445?uid=bltbec0e6a60ea66445" data-sys-asset-uid="blta53dda2af3d8d64c" alt="plotting-90th-percentile-surprise-catches-up">

(Apologies, this chart is mislabeled: "metric: 0" should read "#0 Threshold")

It's now clear: we&nbsp;want to show is the moment in time when the surprise crosses the threshold, and ignore the threshold otherwise (since it is only useful in that first instant). &nbsp;Let’s display a single bar when it crosses the threshold instead of a continuous line.

To do that, we add a custom <code>showifgreater()</code> function. &nbsp;This will only show data-points in the first series if they are greater than the data-points in the second series (see footnote for source):

<pre class="prettyprint">.es('metric:0', metric='avg:value').showifgreater(...)
</pre>
And to complete our query, we only want to show the data if it is greater than three standard deviations (aka if it breaches than the threshold), and then we want to show it as bars instead of lines. &nbsp;That gives us our final query:

<pre class="prettyprint">.es('metric:0', metric='avg:value')
 .showifgreater(
 .es('metric:0', metric='avg:value')
 .movingaverage(6)
 .sum(
 .es('metric:0', metric='avg:value')
 .movingstd(6)
 .multiply(3)
 )
 ).bars()
 .yaxis(2)
 .label("#0 anomalies")
</pre>
Which generates a much nicer looking chart:

<img src="https://api.contentstack.io/v2/assets/575e4d5bd8edd48f7693810c/download?uid=bltd86b71a6eadf7129?uid=bltd86b71a6eadf7129" data-sys-asset-uid="bltaf24dbf798dcc689" alt="plotting-90th-percentile-surprise-nicer-chart">

Finally, let's add back the data itself so we have something to compare against:

<pre class="prettyprint">.es('metric:0', metric='avg:value')
 .label("#0 90th surprise"),
.es('metric:0', metric='avg:value')
 .showifgreater(
 .es('metric:0', metric='avg:value')
 .movingaverage(6)
 .sum(
 .es('metric:0', metric='avg:value')
 .movingstd(6)
 .multiply(3)
 )
 ).bars()
 .yaxis(2)
 .label("#0 anomalies")
</pre>
<img src="https://api.contentstack.io/v2/assets/575e4d5bd8edd48f76938107/download?uid=bltd6370a6b304071b8?uid=bltd6370a6b304071b8" data-sys-asset-uid="bltd79b2eefd3287e43" alt="plotting-90th-percentile-surprise-final-compare">

Voila! &nbsp;We've implemented Atlas! &nbsp;The complete dashboard includes a chart for each metric, as well as a chart showing when disruptions were created (which you obviously would not have in a production environment, but is useful for verifying our simulation):

<img src="https://api.contentstack.io/v2/assets/575e4d5b43e9adc538715d88/download?uid=bltcb7f0c54a52464d1?uid=bltcb7f0c54a52464d1" style="line-height: 1.6em; background-color: initial;" data-sys-asset-uid="blt4a7dadcedb841631" alt="plotting-90th-percentile-surprise-final">

<h2>Analysis of anomalies
</h2>
If you work through the disruption chart (top-left), you'll find an associated anomaly in at least one of the metric charts,&nbsp;often in several simultaneously. &nbsp;Encouragingly, anomalies are tagged for all types of disruptions (node, query, metric). &nbsp;The footnote contains a list of the disruptions and their magnitude to give you an idea of impact. &nbsp;For example, one "Query Disruption"&nbsp;lasted three hours and only affected 12 of the 500 total queries (2.4%) 

One phenomenon you see in the charts are spikes that stay elevated for a while. &nbsp;This is in part due to the duration of the disruption, some which last multiple hours. &nbsp;But it is also due to the limitation in our pipeline agg mentioned last week: namely, we are selecting the "largest" surprise from each timeseries, not the "last" surprise. &nbsp;That means disruptions are drawn out for an extra 24 hours in the worst case, because the surprise only resets once the disruption falls out of the window. &nbsp;This is entirely dependent on the chosen window sizes, and the sensitivity can be tuned by increasing/decreasing the window. 

This phenomenon doesn't affect anomaly detection much, although it becomes more apparent if you try to use longer windows of time. &nbsp;Once pipeline aggs have the ability to select "last", this should be resolved.

<h2>Conclusion
</h2>
So, that's Atlas. &nbsp;A fairly simple -- but very effective -- statistical anomaly detection system built at eBay, and now implemented in Elasticsearch + Timelion. Prior to Pipeline aggs, this could have been implemented by a lot of client-side logic. &nbsp;But the prospect of streaming 60k buckets back to the client every hour for processing is not enticing... pipeline aggs have moved the heavy lifting to the server for more efficient processing. 

Pipeline aggs are still very young, so expect more functionality to be added over time. &nbsp;If you have a use-case that is difficult to express in pipelines, please let us know! 

<h2>The end! Or is it...
</h2>
"But wait!"&nbsp;&nbsp;you say, "this is just plotting the anomalies. &nbsp;How do I get alerts?"&nbsp;For that answer, <a href="/blog/implementing-a-statistical-anomaly-detector-part-3">you'll have to wait until next week</a>, when we implement the TimeLion syntax as a Watcher watch so that you can get automated alerts to email, Slack, etc. See you next week!

<h3>Footnotes</h3>
<ul>
	<li>The custom <code>movingstd()</code> and <code>showifgreater()</code> Timelion functions <a href="https://gist.github.com/polyfractal/5c584f515c4dbd2d5a8f">can be found here.</a>&nbsp; Shut down Kibana, add the functions&nbsp;to your Timelion source (<code>kibana/installedPlugins/timelion/series_functions/&lt;function_name&gt;.js</code>), delete the optimized bundle (<code>rm kibana/optimize/bundles/timelion.bundle.js</code>) and restart Kibana. &nbsp;The functions should be available for use in Timelion now. &nbsp;Note: Javascript is not my forte, so these are not elegant examples of code :)</li>
	<li>The disruptions that were simulated are attached below. &nbsp;The format is in the format: &nbsp;Disruption type: hour_start-hour_end [affected nodes/queries/metrics] 
	<ul>
		<li>Metric Disruption: 505-521 [0, 3, 4]</li>
		<li>Metric Disruption: 145-151 [3, 4]</li>
		<li>Node Disruption: 279-298 [0]</li>
		<li>Metric Disruption: 240-243 [1]</li>
		<li>Query Disruption: 352-355 [5, 23, 27, 51, 56, 64, 65, 70, 72, 83, 86, 95, 97, 116, 135, 139, 181, 185, 195, 200, 206, 231, 240, 263, 274, 291, 295, 307, 311, 315, 322, 328, 337, 347, 355, 375, 385, 426, 468]</li>
		<li>Metric Disruption: 172-181 [0, 2]</li>
		<li>Node Disruption: 334-337 [0]</li>
		<li>Query Disruption: 272-275 [63, 64, 168, 179, 193, 204, 230, 295, 308, 343, 395, 458]</li>
	</ul></li>
	<li>After last week's article, there were some questions regarding the nature of the data: namely, the data was generated using normal (gaussian) curves. &nbsp;Would Atlas work with skewed data, since much data in real life doesn't follow a nice normal distribution? &nbsp;I ran <a href="https://gist.github.com/polyfractal/f635d9fd8c6857c90dcd">a quick test using LogNormal curves</a> which skew heavily to the left, and it appear Atlas continues to function well. &nbsp;The Atlas paper corroborates this empirical evidence. &nbsp;Atlas relies on the 90th surprise over time to be somewhat normal, which seems to hold true even if the underlying data is heavily skewed. &nbsp;There may be followup articles about Atlas behavior under different data distributions as I find time to experiment.</li>
</ul>

Screen Shot 2015-11-19 at 12.02.56 PM.png

Implementing a statistical anomaly detector in Elasticsearch - Part 2

In this series, we implement a statistical anomaly detector to detect when correlated changes in variance occur, providing forewarning of a disruption.

This graph shows the min/max/avg of 45 million data points (75,000 individual time series over 600 hours). There are eight large-scale, simulated disruptions in this graph...can you spot them?
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0c03825bd0e0db61/5bbc7bf763ed239936a7da68/all-data-points-min-max-avg.png" data-sys-asset-uid="blt0c03825bd0e0db61" alt="all-data-points-min-max-avg.png"> 
No? It’s ok, I couldn’t either. When you aggregate all your data together into a single graph, the variability of your data tends to smooth out all but the most obvious changes. In contrast, here are a random selection from the 75,000 series that make up the first graph:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt34a46d6fe45700b1/5bbc7bf169d3fcaf5dbc0167/four-random-charts-total-set.png" data-sys-asset-uid="blt34a46d6fe45700b1" alt="four-random-charts-total-set.png"> 
These individual charts make it obvious where disruptions might be occurring. But now we have a new problem: there are 75,000 such charts to keep an eye on! And since every chart has its own dynamic range and noise profile, how do you set a threshold for all of them?
This conundrum is faced by many companies. As data collection increases, we need new ways to automatically sift through the data and find anomalies. This is exactly what eBay has done with their new <a href="http://www.ebaytechblog.com/2015/08/19/statistical-anomaly-detection/">Atlas anomaly detection algorithm</a>.
The authors of the algorithm realized that any individual series may look anomalous simply due to chance so simple thresholds won’t work, while at the same time aggregating all the data together smooths the data out too much.
Instead, Atlas aggregates together correlated changes in the variance of the underlying data. This means it is robust to noise, but remains sensitive enough to alert when there is a real, correlated shift in the underlying distribution due to a disruption.
After reading the paper, I wanted to see if it could be implemented in Elasticsearch. Over the next three parts of this article, we’ll be exploring how Atlas works and how to build it with a combination of Pipeline aggregations, TimeLion graphs and Watcher alerts. To give you a sneak peak, our final chart will look like this:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt30955f10b6a83ceb/5bbc7bf320bcc78f36235bff/final-atlas-chart.png" data-sys-asset-uid="blt30955f10b6a83ceb" alt="final-atlas-chart.png"> 
Which accurately places an “anomaly bar” on top of each simulated disruption that was generated for this subset of data. Let’s get to it!
<h2>Simulating a production cluster</h2>The first order of business is to generate some data. Since we aren't eBay, we'll have to settle for simulated data. I wrote a <a href="https://github.com/polyfractal/playground/tree/master/hotcloud">quick "Production Cluster Simulator" in Rust</a> which generates our dataset. At a high-level, the simulation is very simple:
<ul>
	<li>There are <code>n</code> nodes in the cluster</li>
	<li>There are <code>q</code> queries which are executed against this cluster (imagine they are
search queries for products, etc)
	</li>
	<li>There are <code>m</code> metrics which are computed against the result of each query</li>
	<li>A data-point is recorded for each <code>(node, query, metric)</code> tuple every hour</li>
</ul>To make the simulation realistic, each <code>(node, query, metric)</code> has a unique gaussian distribution which it uses to generate values. For example, the tuple <code>(1,1,1)</code> may pull values from a normal distribution centered on <code>58</code> and a width of 2 std, while the tuple <code>(2,1,1)</code> may pull values from a distribution centered on 70 and 1 std.
This setup is important for several reasons. First, the same query and metric combination will have a different distribution on each node, introducing a wide amount of natural variability. Second, because these distributions are normal Gaussian, they are not limited to a hard range of values but could theoretically throw outliers (just statistically less likely). Finally, even though there is a large amount of variability between the tuples...the individual <code>(node, query, metric)</code> tuples are very consistent: they always pull from their own distribution.
The final piece in the simulator is simulating disruptions. Each <code>(node, query, metric)</code> tuple also has a "disrupted" distribution. At random points in the timeline, the simulator will toggle one of three disruptions:
<ul>
	<li>Node Disruptions: a single node in the cluster will become disrupted. That means all query/metric pairs will change over to their "disrupted" distribution on that node. E.g. if node <code>1</code> becomes disrupted, all <code>(1, query, metric)</code> tuples are affected</li>
	<li>Query Disruptions: up to one-tenth of all queries in the cluster will changeover to their "disrupted" distribution, regardless of the metric or node. E.g. if query <code>1</code> becomes disrupted, all <code>(node, 1, metric)</code> tuples are affected</li>
	<li>Metric Disruptions: one or more metrics in the cluster will changeover to their "disrupted" distribution, regardless of query or node. E.g. if metric <code>1</code> becomes disrupted, all <code>(node, query, 1)</code> tuples are affected</li>
</ul>The disruptions randomly last between 2 and 24 hours each, and may overlap. The goal of these disruptions are to model a wide variety of errors, some which may be subtle (handful of queries which show abnormal results) while others are more obvious (an entire node showing abnormal results). Importantly, these disruptions pull from the same dynamic range as the "good" distributions: they aren't an order of magnitude larger, or have drastically wider spread (which would both be fairly easy to spot). Instead, the distributions are very similar...just different from the steady-state "good" distributions, making them rather subtle to spot.
<h2>Find the disruptions!</h2>Ok, enough words. Let's look at some graphs of our data. We saw this graph earlier; it is the min/max/avg of all data across a month of simulated time. I simulated 30 nodes, 500 queries and 5 metrics, polled every hour for 600 hours. That comes out to 75,000 points of data per hour and 45m total.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0c03825bd0e0db61/5bbc7bf763ed239936a7da68/all-data-points-min-max-avg.png" data-sys-asset-uid="blt0c03825bd0e0db61" alt="all-data-points-min-max-avg.png"> 
We already know this graph isn’t super useful, so let’s try to partition the data a bit to gain more insight. The next graph shows Metric #0, which experienced two direct disruptions (e.g. all node/query pairs involving Metric #0 were affected, totaling 15k "disrupted" data-points per hour, for several hours each disruption):
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt183a9151d22ed7e2/5bbc7bf93bd14fb77f741bcc/metric-0-min-max-avg.png" data-sys-asset-uid="blt183a9151d22ed7e2" alt="metric-0-min-max-avg.png"> 
If you squint closely, you can see two minor bumps in the average. Both bumps are indeed caused by simulated disruptions...but only the second bump is actually due to a direct "outage" of Metric #0. The first bump is caused by a different disruption. The other direct outage of Metric #0 we are looking for isn't visible here.
Ok, let's try one last time. What if I told you that Query #204 had a direct outage where all node/metric pairs involving #204 were affected? Can you see it on this graph:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltde4e216962140479/5bbc7c00cca23e4e0cc6b117/query-204-min-max-avg.png" data-sys-asset-uid="bltde4e216962140479" alt="query-204-min-max-avg.png"> Well, you can see some disruptions to the average...but none of those are the direct disruption (they are related to other disruptions to individual nodes or metrics).
Ok, so at this point I think you can appreciate the subtlety of these disruptions. They are very small, and even when you have foreknowledge about which chart to look at it can be very hard to pin down the error. If you don't know a disruption is occurring, you have to monitor all the individual time-series (which is not feasible at eBay’s scale).
<h2>Implementing Atlas in Elasticsearch</h2>The first step is to design a set of aggregations. Here is the final query, which we'll walk through step-by-step:
<pre class="prettyprint">{
 "query": {
 "filtered": {
 "filter": {
 "range": {
 "hour": {
 "gte": "{{start}}",
 "lte": "{{end}}"
 }
 }
 }
 }
 },
 "size": 0,
 "aggs": {
 "metrics": {
 "terms": {
 "field": "metric",
 “size”: 5
 },
 "aggs": {
 "queries": {
 "terms": {
 "field": "query",
 "size": 500
 },
 "aggs": {
 "series": {
 "date_histogram": {
 "field": "hour",
 "interval": "hour"
 },
 "aggs": {
 "avg": {
 "avg": {
 "field": "value"
 }
 },
 "movavg": {
 "moving_avg": {
 "buckets_path": "avg",
 "window": 24,
 "model": "simple"
 }
 },
 "surprise": {
 "bucket_script": {
 "buckets_path": {
 "avg": "avg",
 "movavg": "movavg"
 },
 "script": "(avg - movavg).abs()"
 }
 }
 }
 },
 "largest_surprise": {
 "max_bucket": {
 "buckets_path": "series.surprise"
 }
 }
 }
 },
 "ninetieth_surprise": {
 "percentiles_bucket": {
 "buckets_path": "queries&gt;largest_surprise",
 "percents": [
 90.0
 ]
 }
 }
 }
 }
 }
}
</pre>It's long, but not super complicated. Let's start looking at the components individually. First, we set up our query scope with a simple filtered query (or bool + filter if you are on 2.0)&nbsp;over a time range. We'll use this to generate data points for each 24 hour window in our data:
<pre class="prettyprint">"query": {
 "filtered": {
 "filter": {
 "range": {
 "hour": {
 "gte": "{{start}}",
 "lte": "{{end}}"
 }
 }
 }
 }
},
</pre>Next is the aggregation, which is where all the heavy lifting takes place. This aggregation uses a two-tiered set of <code>terms</code> aggregations on <code>metric</code> and <code>query</code>:
<pre class="prettyprint">"aggs": {
 "metrics": {
 "terms": {
 "field": "metric"
 },
 "aggs": {
 "queries": {
 "terms": {
 "field": "query",
 "size": 500
 },
 "aggs": {
 "series": {
 "date_histogram": {
 "field": "hour",
 "interval": "hour"
 },
</pre>We want a single “surprise value” per metric, so our first aggregation is a <code>terms</code> on “metric”. Next, we need to partition each term by query (and ask for all 500 queries). Finally, for each query, we need to generate a window of data over the last 24 hours, which means we need an hourly date histogram.
At this point, we have a time-series per query per metric, for any given date range defined in our query. This will generate 60,000 buckets (5 metrics *&nbsp;500 queries&nbsp;*&nbsp;24 hours) which will then be used to calculate our actual statistics.
With the bucketing done, we can move onto the statistical calculations embedded inside the date histogram:
<pre class="prettyprint">"avg": {
 "avg": {
 "field": "value"
 }
},
"movavg": {
 "moving_avg": {
 "buckets_path": "avg",
 "window": 24,
 "model": "simple"
 }
},
"surprise": {
 "bucket_script": {
 "buckets_path": {
 "avg": "avg",
 "movavg": "movavg"
 },
 "script": "(avg - movavg).abs()"
 }
}
</pre>Here we calculate the average of each bucket, which acts to “collapse” all the values in that hour to a single value we can work with. Next, we define a Moving Average Pipeline aggregation. The moving average takes average bucket values (since the <code>buckets_path</code> points to "avg") and uses a simple average over the 24 hour period.
In the Atlas paper, eBay actually uses a moving median, instead of an average. We don't have a moving median (yet!), but I found that the simple average works just fine. The <code>simple</code> average is used here, instead of the weighted variations, because we don't want any time-based weighting...we actually just want the average of the window.
Next, we define the "surprise" aggregation, which calculates each bucket's deviation from the average. It does this by using a Bucket Script pipeline agg which subtracts the "avg" from the "moving average", then takes the absolute value (which allows us to detect both positive and negative surprises).
This surprise metric is end-goal for each time-series. Since surprise is the deviation from the average, it is a good indicator when something changes. &nbsp;If a series is averaging &nbsp;10 +/- 5, and the values suddenly start reporting as 15 +/- 5 ... the surprise will spike because it deviates from the previous averages.&nbsp; In transient outliers, this is smoothed out quickly. &nbsp;But when an actual disruption occurs and the trend changes, the surprise will alert us.
<h3>Collecting the relevant data</h3>The Atlas paper doesn't really care about all the surprise values, it just wants the top 90th percentile. &nbsp;The top 90th represents the "most surprising" surprise.&nbsp;So we need to extract that from our giant list of calculated surprise.
The Atlas design takes the last surprise value from each series, finds the 90th percentile and records that. Unfortunately, that isn't quite possible with pipeline aggregations yet. We don't have a way to specify the "last" bucket in a series (although I've opened a ticket to include this functionality in the future).
Instead, we can "cheat" and record the largest surprise from the series, which works as an acceptable proxy. We can then accumulate all the largest values and find the 90th percentile:
<pre class="prettyprint"> "largest_surprise": {
 "max_bucket": {
 "buckets_path": "series.surprise"
 }
 }
 }
 },
 "ninetieth_surprise": {
 "percentiles_bucket": {
 "buckets_path": "queries&gt;largest_surprise",
 "percents": [
 90.0
 ]
 }
 }
 }
 }
 }
}
</pre>Which concludes the aggregation. The end result is that we will have a single "ninetieth_surprise" value per metric, which represents the 90th percentile of the largest surprise values from each query in each metric. This aggregation crunches through 60,000 buckets to calculate 5 aggregated values.
Those 5 aggregated values are the pieces of data that we will use in the next step.
<h2>To be continued</h2>This article is already getting quite long, <a href="https://www.elastic.co/blog/implementing-a-statistical-anomaly-detector-part-2">so we are going to resume next week.</a> This aggregation gets us about 80% of the way towards implementing Atlas. The next step is to take these calculated 90th percentile metrics and graph them, as well as alert when they go "out of bounds".
Footnotes
<ul>
	<li><a href="https://github.com/polyfractal/playground/tree/master/hotcloud">Rust Simulator code</a> (a very fast'n'dirty Rust program, please don't judge it too harshly!)</li>
	<li><a href="http://www.ebaytechblog.com/2015/08/19/statistical-anomaly-detection">Statistical Anomaly Detection — eBay's Atlas Anomaly Detector</a></li>
	<li>Goldberg, David, and Yinan Shan. "<a href="https://www.usenix.org/system/files/conference/hotcloud15/hotcloud15-goldberg.pdf">The importance of features for statistical anomaly detection.</a>" Proceedings of the 7th USENIX Conference on Hot Topics in Cloud Computing. USENIX Association, 2015.</li>
</ul>

Implementing a Statistical Anomaly Detector in Elasticsearch - Part 1

Last week, <a href="https://www.elastic.co/blog/staying-in-control-with-moving-averages-part-1">we introduced how to build a simple Control Chart</a> using the new moving_avg pipeline aggregations. &nbsp;The demonstration used a very simple dataset, where the trend was very flat, and the spike was very obvious. &nbsp;In fact, it was so obvious you could probably catch it with a simple threshold.This week, we'll show how the same control chart can be used in more tricky scenarios, such as constantly increasing linear trends, or cyclic/seasonal data<h2 dir="ltr" rel="line-height:1.38;margin-top:10pt;margin-bottom:0pt;">Linear Trends</h2>The example from last week was very simple, and a threshold set by eye would have been sufficient. &nbsp;For example, you could easily determine the ideal mean, calculate three standard deviations yourself, and alert when it goes above that point. &nbsp;This works well for flat trends, but what if your data happens to have a constant linear trend? Just&nbsp;as a refresher, here is the aggregation we built last week: <pre class="prettyprint" style="font-size: 12.6000003814697px;">{ "size": 0, "aggs": { "date_histo": { "histogram": { "field": "tick", "interval": "hour" }, "aggs": { "stats": { "extended_stats": { "field": "value" } }, "movavg_mean": { "moving_avg": { "buckets_path": "stats.avg", "window": 24, "model": "ewma", "settings": { "alpha": 0.1 } } }, "movavg_std": { "moving_avg": { "buckets_path": "stats.std_deviation", "window": 24, "model": "ewma" } }, "shewhart_ucl": { "bucket_script": { "buckets_path": { "mean": "movavg_mean.value", "std": "movavg_std.value" }, "script": "mean + (3 * std)" } } } } } } </pre> Let's re-use that same aggregation on some data with a constant linear trend, which includes the same spike on Thursday. &nbsp;Without changing anything, we'll see: <a href="https://api.contentstack.io/v2/assets/575e4c9c3dc542cb38c084f9/download?uid=blt1e5f788806ad9d40?uid=blt1e5f788806ad9d40"><img src="https://api.contentstack.io/v2/assets/575e4c9c3dc542cb38c084f9/download?uid=blt1e5f788806ad9d40?uid=blt1e5f788806ad9d40" data-sys-asset-uid="blt59216816fa960ad9" width="689" height="149" style="width: 689;height: 149;"/></a> Smoothed average: purpleMax value: yellowUpper control limit: green As you can see, a&nbsp;simple threshold would no longer work; it would be triggered due to the natural growth of the values. &nbsp;There are several ways you could work around it (plot a linear threshold trigger, diff against yesterday, etc). &nbsp;But the control chart handles this scenario in stride without modification Because the threshold is generated dynamically based on the "local" data in the moving averages, the constant linear trend is no problem and everything just works. <h2 dir="ltr">Cyclic Trends</h2>Cyclic trends are even more fun. &nbsp;Imagine your data has some seasonality. &nbsp;In this case, I just plotted a random sine wave, but you'll see this cyclic behavior everywhere in real data: &nbsp;sales numbers, server utilization, queue lengths, etc. &nbsp;Cycles can be very tricky for simpler spike detection algorithms. &nbsp;The algorithm needs to differentiate between the natural peaks and real spikes. If we apply the exact same aggregation as before, we get a decent chart:<a href="https://api.contentstack.io/v2/assets/575e4c9cd8edd48f7693603f/download?uid=blte63ebbbcb7a64d8e?uid=blte63ebbbcb7a64d8e"><img src="https://api.contentstack.io/v2/assets/575e4c9cd8edd48f7693603f/download?uid=blte63ebbbcb7a64d8e?uid=blte63ebbbcb7a64d8e" data-sys-asset-uid="bltdd73a9e465e66522" width="696" height="150" style="width: 696;height: 150;"/></a>Smoothed average: purpleMax value: yellowUpper control limit: green You'll notice a problem though. &nbsp;The maximum values in yellow consistently "trip" the threshold (in green) on the leading edge. &nbsp;It looks like the green threshold lags behind the data, and never quite anticipates the upcoming cycle. &nbsp; The problem is the moving average model. &nbsp;Simpler models like linear and ewma always display a certain amount of lag, and in particular struggle with cyclic data. &nbsp;The lag was present in all the previous examples (go look), it just usually isn’t a problem with non-seasonal data. Instead, we should use holt_winters, a moving average model that includes terms that can account for seasonality. &nbsp;Let's replace the two previous moving averages with this: <pre class="prettyprint">"movavg_mean": { "moving_avg": { "buckets_path": "stats.avg", "window": 200, "model": "holt_winters", "settings": { "period": 69 } } }, "movavg_std": { "moving_avg": { "buckets_path": "stats.std_deviation", "window": 150, "model": "holt_winters", "settings": { "period": 69 } } }, </pre> You'll notice a few changes. &nbsp;Obviously, we swapped ewma for holt_winters. &nbsp;Next, we changed the window size. &nbsp;Holt-Winters requires a larger window so that it can more accurately model seasonal behavior. &nbsp;Finally, we specified how large the "period" of the data is. &nbsp;In this case, it is roughly 62 hours from peak to peak. &nbsp;Holt-Winters has more parameters that are tunable, but we are going to rely on the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline-movavg-aggregation.html#movavg-minimizer">automatic minimization</a> to choose those for us. The graph that we get out looks much better: <a href="https://api.contentstack.io/v2/assets/575e4c9cd8edd48f7693603a/download?uid=blt0c6c71d33f680ac4?uid=blt0c6c71d33f680ac4"><img src="https://api.contentstack.io/v2/assets/575e4c9cd8edd48f7693603a/download?uid=blt0c6c71d33f680ac4?uid=blt0c6c71d33f680ac4" data-sys-asset-uid="bltc28ae47c3a2fd1f6" width="724" height="151" style="width: 724;height: 151;"/></a>Smoothed average: purpleMax value: yellowUpper control limit: green The threshold now lines up with the data perfectly, and we correctly detect the spike (and nothing else). &nbsp;You will notice a new anomaly though. &nbsp;Exactly one period after the first spike a new spike exists where there wasn’t one previously. &nbsp;And if you look closely, you'll see a tiny spike two periods afterwards which is also new.This is an artifact from Holt-Winters. &nbsp;Forecasts are built based on past seasonal data, and since the past data&nbsp;had a spike, you'll see traces of that in future forecasts. &nbsp;This artifact can be diminished slightly by increasing the window length, and in general isn't usually large enough to trigger a "threshold breach". <h2 dir="ltr">Extra Credit: Configuring a Watcher alert</h2>If you have Watcher installed -- <a href="/what-is/elasticsearch-alerting">an alerting and notification plugin for Elasticsearch</a> -- it is trivial to add a watch which will alert you when a spike has been detected. &nbsp;We will define a watch that checks every hour (finer granularity is unnecessary, since the data is only logged at 1hr intervals). &nbsp; Then we plop in our aggregation, and setup some email and logging notifications and define the condition. &nbsp;The condition is simply checking to see if the maximum value is greater than the upper control limit. <pre class="prettyprint">curl -XPUT 'http://localhost:9200/_watcher/watch/log_error_watch' -d '{ "trigger": { "schedule": { "interval": "1hr" } }, "input": { "search": { "request": { "indices": ["reactor_logs"], "body": { "size": 0, "aggs": { "histo": { "date_histogram": { "field": "tick", "interval": "hour" }, "aggs": { "stats": { "extended_stats": { "field": "value" } }, "movavg_mean": { "moving_avg": { "buckets_path": "stats.avg", "window": 24, "model": "ewma", "settings": { "alpha": 0.1 } } }, "movavg_std": { "moving_avg": { "buckets_path": "stats.std_deviation", "window": 24, "model": "ewma" } }, "shewhart_ucl": { "bucket_script": { "buckets_path": { "mean": "movavg_mean.value", "std": "movavg_std.value" }, "script": "mean + (3 * std)" } } } } } } } } }, "condition": { "script": { "inline": "def lastBucket = ctx.payload.aggregations.histo.buckets.last(); return lastBucket.stats.max &gt; lastBucket.shewhart_ucl.value" } }, "actions": { "log_error": { "logging": { "text": "Reactor Meltdown!" } }, "send_email": { "email": { "to": "user@example.com", "subject": "Watcher Notification - Reactor Meltdown!", "body": "Reactor is melting down, please investigate. :)" } } } }' </pre> With this in place, Watcher will email you as soon as the upper control limit has been reached. &nbsp;It is fairly trivial to extend this to log/alert on “warnings”, such as when values exceed two standard deviations instead of three, or has remained above the mean for more than 10 consecutive hours. &nbsp;The sky is the limit! <h2 dir="ltr">Conclusion</h2>I hope this article was interesting. &nbsp;Most folks are acquainted with the smoothing capabilities of moving averages;&nbsp; they are great for smoothing out noise so you can see the more general trend. &nbsp;But they can also be the building blocks for much richer functionality, such as finding anomalous data points in a dynamic dataset. &nbsp;It's fairly remarkable how powerful simple, statistical techniques can be in practice.Because of the new functionality in pipeline aggregations, all of this functionality can now be expressed in Elasticsearch itself. &nbsp;And when coupled with Watcher, you can build robust alerting and notifications directly from your data, without having to pipe it to an external system first. In the future, we'll be looking at how you can forecast into the future with moving averages, other methods for anomaly detection and more. &nbsp;Stay tuned!<div> </div>

staying-in-control-with-moving-averages-full-bleed.jpg

Staying in Control with Moving Averages - Part 2

<html><head></head><body>In manufacturing and business processes, there is a common tool called a control chart. Created in 1920 by Dr. Walter Shewhart, a control chart is used to determine if a process is &quot;in control&quot; or &quot;out of control&quot;. &#xA0;
At the time, Dr. Shewhart was working at Bell Labs trying to improve the signal quality of telephone lines. &#xA0;Poorly-machined components was a leading cause of signal degradation, so improving manufacturing processes to produce more uniform components was a critical step in improving signal quality.
 
Dr. Shewhart realized that all processes, manufacturing or otherwise, have some amount of natural variation. &#xA0;The key was to identify when the variation was behaving normally (&quot;in control&quot;), and when it suddenly began to change (&quot;out of control&quot;). &#xA0;A process that has gone out of control needs to be halted so the problem can be fixed, instead of churning out sloppy manufactured components.
 
Control charts work by triggering an alert when the value diverges sufficiently from the mean by a certain amount. &#xA0;In practice, they are very simple and intuitive to read, and often act as front-line anomaly detectors due to their simplicity and robustness.
 
<h1 dir="ltr" style="line-height:1.38;margin-top:10pt;margin-bottom:0pt;">Smoothing with Moving Averages</h1> 
Control charts can be built fairly easily in Elasticsearch using a combination of aggregations, including the <a href="https://www.elastic.co/blog/out-of-this-world-aggregations">new pipeline aggregations</a>. &#xA0;To get started, let&#x2019;s look at some synthetic data that I generated for this post. For fun, we can imagine it is coolant temperature (in celsius) for a nuclear reactor.
 
Let&#x2019;s take a look at the data first, using a <code>histogram</code> bucket and an <code>extended_stats</code>&#xA0;metric:
 
<pre class="prettyprint">{
 &quot;size&quot;: 0,
 &quot;aggs&quot;: {
 &quot;histo&quot;: {
 &quot;date_histogram&quot;: {
 &quot;field&quot;: &quot;tick&quot;,
 &quot;interval&quot;: &quot;hour&quot;
 },
 &quot;aggs&quot;: {
 &quot;stats&quot;: {
 &quot;extended_stats&quot;: {
 &quot;field&quot;: &quot;value&quot;
 }
 }
 }
 }
 }
}
</pre> 
 
In the graph, we are plotting the `avg` for each bucket:
 
<a href="https://api.contentstack.io/v2/assets/575e4c93d8edd48f76935eaa/download?uid=blt8f7aabf0ab90819b?uid=blt8f7aabf0ab90819b"><img style="width: 675px; height: 148px;" src="https://api.contentstack.io/v2/assets/575e4c93d8edd48f76935eaa/download?uid=blt8f7aabf0ab90819b?uid=blt8f7aabf0ab90819b" data-sys-asset-uid="blt505f2e098625939e" height="148" width="675"></a>
Click for full size.
 As you can see, the data is basically a flat trend, with a random distribution around ~30. &#xA0;The data is noisy, so the first thing you might like to do is smooth it out so you can see the general trend better. &#xA0;Moving averages are great for this. 
A moving average basically takes a window of values, computes the average, then moves the window forward one step. &#xA0;There are <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline-movavg-aggregation.html">several different types of moving averages</a> that you can choose from. &#xA0;We are&#xA0;going to use an Exponentially-Weighted Moving Average (EWMA). &#xA0;This type of moving average reduces the &quot;importance&quot; of a datapoint exponentially as it becomes &quot;older&quot; in the window. &#xA0;This helps keep the moving average centered on the data instead of lagging behind.
 
In the following query, we add a <code>movavg_mean</code>&#xA0;moving average pipeline aggregation that computes the moving average of each bucket&apos;s avg (i.e. a sliding mean of means):
 
<pre class="prettyprint">{
 &quot;size&quot;: 0,
 &quot;aggs&quot;: {
 &quot;histo&quot;: {
 &quot;date_histogram&quot;: {
 &quot;field&quot;: &quot;tick&quot;,
 &quot;interval&quot;: &quot;hour&quot;
 },
 &quot;aggs&quot;: {
 &quot;stats&quot;: {
 &quot;extended_stats&quot;: {
 &quot;field&quot;: &quot;value&quot;
 }
 },
 &quot;movavg_mean&quot;: {
 &quot;moving_avg&quot;: {
 &quot;buckets_path&quot;: &quot;stats.avg&quot;,
 &quot;window&quot;: 24,
 &quot;model&quot;: &quot;ewma&quot;,
 &quot;settings&quot;: {
 &quot;alpha&quot;: 0.1 
 }
 }
 }
 }
 }
 }
}
</pre> 
There are a few interesting bits here:
 
<ul>
	<li dir="ltr"><code>buckets_path</code>&#xA0;points to the&#xA0;<code>avg</code>&#xA0;value calculated inside our <code>extended_stats</code>&#xA0;metric</li>
	<li dir="ltr"><code>window</code>&#xA0;is set to 24, meaning we want to average the last 24 hours together</li>
	<li dir="ltr"><code>model</code>&#xA0;is set to <code>ewma</code></li>
	<li dir="ltr">And finally, we configure some <code>settings</code> for this particular model. &#xA0;The setting <code>alpha</code>&#xA0;controls how &quot;smooth&quot; the generated moving average is. &#xA0;The default (0.3) is usually pretty good, but I liked the look of 0.1 better for this demo. &#xA0;Check out the docs <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-pipeline-movavg-aggregation.html#_ewma_exponentially_weighted">for more info on how <code>alpha</code> functions</a>.</li>
</ul>And the resulting graph now includes a nicely smoothed line (purple):
 <a href="https://api.contentstack.io/v2/assets/575e4c939985d58976376b54/download?uid=bltfc30b6158212fb3f?uid=bltfc30b6158212fb3f"><img style="width: 615px; height: 148px;" src="https://api.contentstack.io/v2/assets/575e4c939985d58976376b54/download?uid=bltfc30b6158212fb3f?uid=bltfc30b6158212fb3f" data-sys-asset-uid="bltaef12ba1fa1e993a" height="148" width="615"></a> 
<h1 dir="ltr" style="line-height:1.38;margin-top:10pt;margin-bottom:0pt;">In control?</h1> 
So, the question is... does this chart look &quot;in control&quot;? &#xA0;Is there a reason you should shut down the reactor, or is everything operating smoothly?
I admit, I was being sneaky in the previous graph: &#xA0;I plotted the average. &#xA0;As <a href="https://www.elastic.co/blog/averages-can-dangerous-use-percentile">discussed previously</a>, the average is a pretty poor metric in most cases. &#xA0;In this dataset, it is hiding a big spike that I placed on Thursday.
If we plot the maximum value in each bucket (yellow line) the spike is immediately clear:
 <a href="https://api.contentstack.io/v2/assets/575e4c933dc542cb38c0838d/download?uid=bltc51d81ed9940ab84?uid=bltc51d81ed9940ab84"><img style="width: 611px; height: 149px;" src="https://api.contentstack.io/v2/assets/575e4c933dc542cb38c0838d/download?uid=bltc51d81ed9940ab84?uid=bltc51d81ed9940ab84" data-sys-asset-uid="bltc49c13ad62b5c41d" height="149" width="611"></a> 
I hope you turned the reactor off on Thursday! ;)
 
How might we have detected this spike? &#xA0;In this chart, the anomaly is absurdly clear. &#xA0;You could use a simple threshold. &#xA0;But as we&apos;ll see later, thresholds often fail under more complex patterns.
Instead, let&apos;s build a control chart. &#xA0;Control charts consider a process &quot;out of control&quot; if datapoints start falling three standard deviations away from the mean. &#xA0;With that in mind, we can modify our aggregation to turn it into a bona fide control chart. &#xA0;To do so, we need to add two new aggregations: &#xA0;a moving average on the standard deviation, and a script that calculates the upper limit:
 
<pre class="prettyprint">{
 &quot;size&quot;: 0,
 &quot;aggs&quot;: {
 &quot;date_histo&quot;: {
 &quot;histogram&quot;: {
 &quot;field&quot;: &quot;tick&quot;,
 &quot;interval&quot;: &quot;hour&quot;
 },
 &quot;aggs&quot;: {
 &quot;stats&quot;: {
 &quot;extended_stats&quot;: {
 &quot;field&quot;: &quot;value&quot;
 }
 },
 &quot;movavg_mean&quot;: {
 &quot;moving_avg&quot;: {
 &quot;buckets_path&quot;: &quot;stats.avg&quot;,
 &quot;window&quot;: 24,
 &quot;model&quot;: &quot;ewma&quot;,
 &quot;settings&quot;: {
 &quot;alpha&quot;: 0.1 
 }
 }
 },
 &quot;movavg_std&quot;: {
 &quot;moving_avg&quot;: {
 &quot;buckets_path&quot;: &quot;stats.std_deviation&quot;,
 &quot;window&quot;: 24,
 &quot;model&quot;: &quot;ewma&quot;
 }
 },
 &quot;shewhart_ucl&quot;: {
 &quot;bucket_script&quot;: {
 &quot;buckets_path&quot;: {
 &quot;mean&quot;: &quot;movavg_mean.value&quot;,
 &quot;std&quot;: &quot;movavg_std.value&quot;
 },
 &quot;script&quot;: &quot;mean + (3 * std)&quot;
 }
 }
 }
 }
 }
}
</pre> 
The new <code>&quot;movavg_std&quot;</code>&#xA0;pipeline agg is very simple: &#xA0;it is simply an EWMA (with default settings) that averages the <code>stats.std_deviation</code>&#xA0;metric over the last 24 hours. &#xA0; 
The <code>&quot;shewhart_ucl&quot;</code>&#xA0;pipeline agg is a <code>bucket_script</code>&#xA0;which computes the &quot;upper control limit&quot;; aka, the point in time when you start worrying because the process has gone out of control. &#xA0;Think of it as a dynamic threshold. &#xA0;The threshold is calculated by multiplying the rolling standard deviation by three, then adding it to the rolling mean. 
I omitted it for brevity, but most control charts also include a &quot;lower control limit&quot;. &#xA0;To add that, you would simply copy <code>&quot;shewhart_ucl&quot;</code>&#xA0;, subtract three standard deviations instead of adding, and rename it to &quot;shewhart_lcl&quot;.
Note: I&#x2019;m using an inline script for convenience. You can substitute it for a static script if dynamic, inline scripting is disabled on your cluster. 
 <a href="https://api.contentstack.io/v2/assets/575e4c939985d58976376b66/download?uid=blt289da18209131391?uid=blt289da18209131391"><img style="width: 622px; height: 149px;" src="https://api.contentstack.io/v2/assets/575e4c939985d58976376b66/download?uid=blt289da18209131391?uid=blt289da18209131391" data-sys-asset-uid="blt06f300d1a30f0358" height="149" width="622"></a> 
Smoothed average: purple
Max value: yellow
Upper control limit: green
 We can graph this and see that the spike (yellow) shoots up past the control limit (green). &#xA0;In a real system, this is when you send out an alert or email. &#xA0;Or maybe something more drastic, since this is a nuclear reactor we are modeling ;)<h1 dir="ltr" style="margin-top: 10pt; margin-bottom: 0pt; line-height: 1.38; background-color: rgb(255, 255, 255);">Conclusion</h1> 
That&apos;s all for this week. &#xA0;To recap, we used the new pipeline aggregations to smooth out our data with a moving average. &#xA0;We then constructed a control chart to dynamically find outliers by calculating an &quot;upper control limit&quot; based on the moving average and a moving standard deviation.
In <a href="https://www.elastic.co/blog/staying-in-control-with-moving-averages-part-2">part two</a>, we&apos;ll look at how the same control chart can be used for more interesting data patterns, such as linear trends and cyclic behavior. We&apos;ll also see how to integrate it with Watcher so that we can receive email notifications automatically.&#xA0; <a href="https://www.elastic.co/blog/staying-in-control-with-moving-averages-part-2">Check it out</a>!</body></html>

Staying in Control with Moving Averages - Part 1

<article>
 Welcome Back! Last time on <a href="https://www.elastic.co/blog/intro-to-aggregations">Intro to Aggregations</a>, we explored the basic concepts of aggregations and how to start using them. We toyed with simple bucket and metric aggregations, which gave us simple analytics.
 Today, we are going to learn about sub-aggregations. The most powerful feature of aggregations in Elasticsearch is the ability to embed aggregations (both buckets and metrics) inside…wait for it…other aggregations. Sub-aggregations allow you to continuously refine and separate groups of criteria of interest, then apply metrics at various levels in the aggregation hierarchy to generate your report.
 The first approach we will look at involves embedding buckets inside of other buckets. If we take our first example from last week about the Five Boroughs, we can embed an additional bucket underneath to determine the top three incident contributing factors in each borough:<pre class="prettyprint">GET /nyc_visionzero/_search?search_type=count{ "aggs" : { "all_boroughs": { "terms": { "field": "borough" }, "aggs": { "cause": { "terms": { "field": "contributing_factor_vehicle", "size": 3 } } } } } }</pre>
 In this aggregation, the original “all_boroughs” aggregation remains unchanged. There is a new “aggs” element that has been added as a JSON sibling to “terms” (i.e., on the same level in the JSON tree). The new element tells Elasticsearch to begin a sub-aggregation. Inside the new “aggs” element, we name our new sub-aggregation (“cause”) and use another terms bucket. This bucket’s criteria is the contributing factor field, limited to 3 results.
 The response looks like this (truncated):<pre class="prettyprint">{ "took": 29, "aggregations": { "all_boroughs": { "buckets": [ { "key": "BROOKLYN", "doc_count": 86549, "cause": { "buckets": [ { "key": "Unspecified", "doc_count": 57203 }, { "key": "Driver Inattention/Distraction", "doc_count": 6954 }, { "key": "Failure to Yield Right-of-Way", "doc_count": 4209 } ] } }, ...</pre>
 Now you can see that we have our original aggregation data. We see again that Brooklyn has over 86,000 traffic incidents. But now it’s been enriched with the top three causes (“Unspecified”, “Inattention”, and “Failure to yield”) and the results of these newly added bucket are embedded under the results of the old bucket. If you were to run this example, you would see that similar data has enriched the rest of the boroughs as well, since the sub-aggregation runs for every bucket generated by the parent.
 And this being Elasticsearch, we can embed as much as we want. You could, for example, find out how many incidents occurred per-month, per-cause, per-borough:<pre class="prettyprint">GET /nyc_visionzero/_search?search_type=count { "aggs" : { "all_boroughs": { "terms": { "field": "borough" }, "aggs": { "cause": { "terms": { "field": "contributing_factor_vehicle", "size": 3 }, "aggs": { "incidents_per_month": { "date_histogram": { "field": "@timestamp", "interval": "month" } } } } } } } }</pre>
 Pretty cool, huh?
 <h2>Mixing Buckets and Metrics</h2>
 On their own, metrics are a bit pedestrian. When looking at the examples last week, you may have been a bit underwhelmed. On their own, metrics can only calculate values against the global dataset.
 But when metrics are embedded under a bucket as a sub-aggregation, they become infinitely more useful. Metrics will calculate their value based on the documents residing inside the bucket. This allows you to set up a range of criteria and sub-criteria with buckets, then place metrics to calculate values for your reports about each criteria.
 As an example, let’s calculate the total number of cyclists injured in each borough:<pre class="prettyprint">GET /nyc_visionzero/_search?search_type=count { "aggs" : { "all_boroughs": { "terms": { "field": "borough" }, "aggs": { "total_cyclists_injured": { "sum" : { "field": "number_of_cyclist_injured" } } } } } }</pre>
 We are using the same terms bucket to provide a criteria (“Which borough did this incident happen in?”), and then embedding a sum metric underneath it to calculate the total number of injuries. The response shows us that, although there were many incidents in each borough, relatively few of those involved cyclist injuries:<pre class="prettyprint">{ "took": 19, "aggregations": { "all_boroughs": { "buckets": [ { "key": "BROOKLYN", "doc_count": 86549, "total_cyclists_injured": { "value": 2691 } }, { "key": "MANHATTAN", "doc_count": 76122, "total_cyclists_injured": { "value": 2000 } }, { "key": "QUEENS", "doc_count": 73000, "total_cyclists_injured": { "value": 1271 } }, { "key": "BRONX", "doc_count": 36239, "total_cyclists_injured": { "value": 503 } }, { "key": "STATEN ISLAND", "doc_count": 15763, "total_cyclists_injured": { "value": 69 } } ] } } }</pre>
 You’ll notice here that the metric is extracting the value of the `total_cyclists_injured` field and summing that for each borough, rather than just counting documents. Once you start mixing buckets and metrics, the sky’s the limit. Try running this example:<pre class="prettyprint">GET /nyc_visionzero/_search?search_type=count { "aggs" : { "all_boroughs": { "terms": { "field": "borough" }, "aggs": { "causes": { "terms": { "field": "contributing_factor_vehicle", "size": 3 }, "aggs": { "incidents_per_month": { "date_histogram": { "field": "@timestamp", "interval": "month" }, "aggs": { "cyclist_injury_avg": { "avg": { "field": "number_of_cyclist_injured" } } } } } }, "total_cyclists_injured": { "sum" : { "field": "number_of_cyclist_injured" } } } }, "top_causes": { "terms": { "field": "contributing_factor_vehicle", "size": 3 } }, "outside_borough": { "missing": { "field": "borough" }, "aggs": { "total_cyclists_injured": { "sum" : { "field": "number_of_cyclist_injured" } } } } } }</pre>
 The aggregation is a bit large. But it is simple once you break it down into components:
 <ul>
 <li>all_boroughs.causes.incidents_per_month.cyclist_injury_avg: reports average number of cyclists injured per-month, per contributing cause factor, per borough</li>
 <li>all_boroughs.total_cyclists_injured: reports total cyclists injured per borough, irrespective of cause</li>
 <li>top_causes: reports the top three contributing incident causes, irrespective of borough</li>
 <li>outside_borough.total_cyclists_injured: reports total cyclists injured in areas outside of a borough</li>
 </ul>
 <h2>Conclusion</h2>
 That last aggregation was pretty nifty, no? This entire report runs in a single API call and takes only milliseconds to execute. It operates on fresh, near-real-time data. As new incidents stream into your system, your aggregations will immediately update to reflect the new data. This is what makes Kibana so powerful: near-real-time rollups and summaries of your data, sliced and diced as you wish.
 These first two introductory articles have barely scratched the surface of aggregations in Elasticsearch. In the following articles, we will begin to dissect datasets and show how real insights can be extracted using aggregations. Stay tuned!
 <aside id="post_tags"></aside>
</article>

Intro to Aggregations pt. 2: Sub-Aggregations

<html><head></head><body><section id="post_content">
<article>
Many people are familiar with Elasticsearch for its search functionality. And while it is excellent at search, many organizations are using Elasticsearch for an entirely different purpose: analytics.

Beneath the surface of Elasticsearch is a powerful analytics engine, waiting to be unleashed on your data. This article series will show you a developer-centric approach to using aggregations, the analytical functionality in Elasticsearch, that can help you to analyze your data, build custom dashboards, and reap the benefits of near real-time responses.

<h2>Search &amp; Analytics: Two Sides of the Same Coin</h2>
When searching for a document, you have a goal in mind. For instance, &#x201C;Find all transactions for UserXYZ.&#x201D; You don&#x2019;t know which documents are related to the query, but you know that a collection of documents will match the query.

Search is all about retrieving a subset of documents from the entire dataset. In contrast, analytic workloads don&#x2019;t care about individual documents. A dashboard won&#x2019;t show individual transactions, but rather the aggregate summary of those transactions. When building analytics, you roll up your data into useful summaries like, &#x201C;What was the total revenue from the transactions by UserXYZ?&quot; Search is concerned with retrieving documents, and analytics is busy calculating summaries about those documents.

Because aggregations operate over the same Lucene indices that power search, they gain the near real-time nature of Lucene data. This means that reports generated by aggregations will update as soon as the data changes, rather than waiting for the next nightly batch cron-job to finish in other systems.

<h2>Getting Started with Aggregations</h2>
Since this is a developer-focused series, we suggest you follow along at home, so that you can play with the data and the various code examples that we show. For this article, we are going to use the New York City Traffic Incident dataset. We used this dataset <a href="http://www.elasticsearch.org/blog/byodemos-new-york-city-traffic-incidents/">a while ago to demonstrate Kibana and Logstash</a> for non-developers.

This series of blogs will utilize a range of aggregation features, some of which have been introduced in later versions of Elasticsearch. To follow the entire series from start to finish, we recommend using Elasticsearch version 1.4.2. But this introductory blog only deals with the basics, so any version &gt;= 1.0 will work.


 To follow along, you&apos;ll need to restore a Snapshot into your local cluster. The snapshot is about 200MB, and it may take some time depending on your connection (NOTE: the snapshot URL only works through the API, it won&apos;t work in your browser):

<pre class="programlisting prettyprint lang-js">// Register the NYC Traffic Repository
PUT /_snapshot/demo_nyc_accidents
{
 &quot;type&quot;: &quot;url&quot;,
 &quot;settings&quot;: {
 &quot;url&quot;: &quot;http://download.elasticsearch.org/demos/nycopendata/snapshot/&quot;
 }
}
// (Optional) Inspect the repository to view available snapshots
GET /_snapshot/demo_nyc_accidents/_all
// Restore the snapshot into your cluster
POST /_snapshot/demo_nyc_accidents/demo_nyc_accidents/_restore
// Watch the download progress.
GET /nyc_visionzero/_recovery
</pre>
Once your cluster has finished restoring the Snapshot, let&#x2019;s perform a simple search to see what the data holds:

<pre class="prettyprint ">GET /nyc_visionzero/_search
{
 &quot;_shards&quot;: {...},
 &quot;hits&quot;: {
 &quot;total&quot;: 332871,
 &quot;max_score&quot;: 1,
 &quot;hits&quot;: [
 {
 &quot;_index&quot;: &quot;nyc_visionzero&quot;,
 &quot;_type&quot;: &quot;logs&quot;,
 &quot;_id&quot;: &quot;97vKYDDLSZuDmj-B2RLzXg&quot;,
 &quot;_score&quot;: 1,
 &quot;_source&quot;: {
 &quot;message&quot;: [
 &quot;04/02/2014,23:15,QUEENS,11377,40.7372519,-73.9179831,\&quot;(40.7372519, -73.9179831)\&quot;,48 STREET ,50 AVENUE ,,0,0,0,0,0,0,0,0,Unspecified,,,,,316762&quot;
 ],
 &quot;@version&quot;: &quot;1&quot;,
 &quot;@timestamp&quot;: &quot;2014-04-03T06:15:00.000Z&quot;,
 &quot;host&quot;: &quot;localhost&quot;,
 &quot;date&quot;: &quot;04/02/2014&quot;,
 &quot;time&quot;: &quot;23:15&quot;,
 &quot;borough&quot;: &quot;QUEENS&quot;,
 &quot;zip_code&quot;: &quot;11377&quot;,
 &quot;latitude&quot;: &quot;40.7372519&quot;,
 &quot;longitude&quot;: &quot;-73.9179831&quot;,
 &quot;location&quot;: &quot;(40.7372519, -73.9179831)&quot;,
 &quot;on_street_name&quot;: &quot;48 STREET&quot;,
 &quot;cross_street_name&quot;: &quot;50 AVENUE&quot;,
 &quot;off_street_name&quot;: null,
 &quot;number_of_persons_injured&quot;: 0,
 &quot;number_of_persons_killed&quot;: 0,
 &quot;number_of_pedestrians_injured&quot;: 0,
 &quot;number_of_pedestrians_killed&quot;: 0,
 &quot;number_of_cyclist_injured&quot;: 0,
 &quot;number_of_cyclist_killed&quot;: 0,
 &quot;number_of_motorist_injured&quot;: 0,
 &quot;number_of_motorist_killed&quot;: 0,
 &quot;unique_key&quot;: &quot;316762&quot;,
 &quot;coords&quot;: [
 -73.9179831,
 40.7372519
 ],
 &quot;contributing_factor_vehicle&quot;: &quot;Unspecified&quot;
 }
 },
 ...
</pre>
Each document represents a single traffic &#x201C;incident&#x201D; in NYC. These incidents contain a variety of metadata, including the time, the street name, the latitude and longitude, the borough, the statistics about injuries, and a summary of the contributing factors. There are 300,000+ documents; plenty to play with.

Let&#x2019;s start building a few simple analytics. In Elasticsearch, all analytics are built using aggregations. Aggregations are constructed similar to queries, via a JSON-based DSL. The aggregation is appended to a search request, and both the search and aggregation are executed simultaneously.

Here is a simple aggregation:

<pre class="prettyprint ">GET /nyc_visionzero/_search?search_type=count
{
 &quot;aggs&quot; : {
 &quot;all_boroughs&quot;: {
 &quot;terms&quot;: {
 &quot;field&quot;: &quot;borough&quot;
 }
 }
 }
}
</pre>
Several things to note in this example:

<ul>
	<li>It is being executed against the _search endpoint. Aggregations are just another feature of search, and use the same API endpoint.</li>
	<li>Aggregations are specified under the &#x201C;aggs&#x201D; parameter. You can still specify a &#x201C;query&#x201D; for normal search if you also want to execute a search query.</li>
	<li>We are executing the search with a count search_type. This omits the fetch phase in search, and is a performance trick if you don&#x2019;t care about the search results (just the aggregation results)</li>
</ul>
So, what is this aggregation doing? It&apos;s building a list of all the boroughs in NYC, based on the boroughs included in the documents in the dataset. Here is the response:

<pre class="prettyprint ">{
 &quot;took&quot;: 11,
 &quot;hits&quot;: {
 &quot;total&quot;: 332871,
 &quot;max_score&quot;: 0,
 &quot;hits&quot;: []
 },
 &quot;aggregations&quot;: {
 &quot;all_boroughs&quot;: {
 &quot;buckets&quot;: [
 {
 &quot;key&quot;: &quot;BROOKLYN&quot;,
 &quot;doc_count&quot;: 86549
 },
 {
 &quot;key&quot;: &quot;MANHATTAN&quot;,
 &quot;doc_count&quot;: 76122
 },
 {
 &quot;key&quot;: &quot;QUEENS&quot;,
 &quot;doc_count&quot;: 73000
 },
 {
 &quot;key&quot;: &quot;BRONX&quot;,
 &quot;doc_count&quot;: 36239
 },
 {
 &quot;key&quot;: &quot;STATEN ISLAND&quot;,
 &quot;doc_count&quot;: 15763
 }
 ]
 }
 }
}
</pre>
Notice that the aggregation results are named &#x201C;all_boroughs&#x201D;, which was defined in the aggregation.

As you can see, there are five boroughs in NYC. Under each borough is a document count: Brooklyn had 86,000 traffic incidents, while Staten Island only had 15,000. Congrats, you ran your first aggregation!

<h2>Bucketing Based on Criteria</h2>
The &#x201C;all_boroughs&#x201D; aggregation we ran was an example of a bucket aggregation. Bucketing aggregations define a criteria. If documents match that criteria, they are added to the bucket. Documents can be added to multiple buckets, or to no buckets at all. When the aggregation finishes, you are left with a collection of documents matching various criteria.

In the example above, we used a Terms Bucket. It&apos;s called a Terms Bucket because this aggregation dynamically builds buckets based on the terms in your data. There were five boroughs in our data, so we got five corresponding buckets. There are also many other bucketing aggregations at your disposal. For example, you could bucket by time using a date_histogram. This will generate one bucket per month (giving you effectively a line-graph over time):

<pre class="prettyprint ">GET nyc_visionzero/_search?search_type=count
{
 &quot;aggs&quot; : {
 &quot;months&quot;: {
 &quot;date_histogram&quot;: {
 &quot;field&quot;: &quot;@timestamp&quot;,
 &quot;interval&quot;: &quot;month&quot;
 }
 }
 }
}
</pre>
Or you could bucket by how many cyclists were injured in each incident, using a histogram bucket:

<pre class="prettyprint ">GET nyc_visionzero/_search?search_type=count
{
 &quot;aggs&quot; : {
 &quot;injuries&quot;: {
 &quot;histogram&quot;: {
 &quot;field&quot;: &quot;number_of_cyclist_injured&quot;,
 &quot;interval&quot;: 1
 }
 }
 }
}
</pre>
You could even bucket documents based on the criteria of missing a field, using the missing bucket. This will collect all documents that don&#x2019;t have a value for that particular field:

<pre class="prettyprint ">GET nyc_visionzero/_search?search_type=count
{
 &quot;aggs&quot; : {
 &quot;missing_borough&quot;: {
 &quot;missing&quot;: {
 &quot;field&quot;: &quot;borough&quot;
 }
 }
 }
}
</pre>
There are <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations.html">many bucketing aggregations available</a>. Skim the reference documentation at some point to familiarize yourself with the various buckets.

<h2>Metrics: More Than Just Doc Counts</h2>
You may have noticed something interesting about the bucket aggregation responses: they only list document counts. Buckets simply collect documents based on a criteria, which means the only statistic they possess is a document count.

But what if you want to calculate a value based on fields in the document, like the average price or the total revenue? These operations are performed by the second type of Aggregation in Elasticsearch called metrics.

Metrics are simple mathematical operations, like min, max, avg, sum, percentiles etc. Metrics extract values out of the documents to use for the calculation.

Metrics can be used to calculate a simple number. For example, we can calculate the total number of cyclists injured in NYC:

<pre class="prettyprint ">GET /nyc_visionzero/_search?search_type=count
{
 &quot;aggs&quot; : {
 &quot;cyclist_injuries&quot;: {
 &quot;sum&quot; : {
 &quot;field&quot;: &quot;number_of_cyclist_injured&quot;
 }
 }
 }
}
</pre>
Which returns a result of over 7,000 injuries:

<pre class="prettyprint ">{
 &quot;took&quot;: 8,
 &quot;aggregations&quot;: {
 &quot;cyclist_injuries&quot;: {
 &quot;value&quot;: 7129
 }
 }
}
</pre>
<h2>Naming aggregations</h2>
If you look back at all the examples so far &#x2014;buckets and metrics &#x2014; you&apos;ll see that all aggregations must be named. This is important because it allows you to use several aggregations simultaneously. For instance, we could specify a bucket and a metric:

<pre class="prettyprint ">GET /nyc_visionzero/_search?search_type=count
{
 &quot;aggs&quot; : {
 &quot;all_boroughs&quot;: {
 &quot;terms&quot;: {
 &quot;field&quot;: &quot;borough&quot;
 }
 },
 &quot;cyclist_injuries&quot;: {
 &quot;sum&quot; : {
 &quot;field&quot;: &quot;number_of_cyclist_injured&quot;
 }
 }
 }
}
</pre>
Because we gave each aggregation its own name, the responses that comes back from Elasticsearch will have two named sections with the respective data. Furthermore, these two separate aggregations will be executed simultaneously in a single pass over the data. By merging multiple aggregations into a single API call, you can build many reports with just one pass over your data.

<h2>Conclusion</h2>
In this blog, we learned that aggregations serve a purpose different than search, but operate on the same near-realtime Lucene indices under the covers. We explored how to use buckets and metrics, the building blocks of the aggregation DSL. Next week we&apos;ll learn how to use sub-aggregations to create sophisticated, multi-level reports. Stay tuned!

<aside id="post_tags">
</aside>
</article>
</section>
</body></html>

Intro to Aggregations

<blockquote>When you help people build applications with Elasticsearch every day, you run into a lot of unique requirements and scenarios. All of us here at Elasticsearch have an ever-growing “bag of tricks”, and it only seems fair to share those tricks with you. We hope you find them interesting, and perhaps useful to your application.</blockquote><hr style="width:100%">A while ago on Twitter, someone was asking if aggregations could be used to categorize data based on irregular product codes. They were combining data from several legacy systems, so the product codes were erratic and not internally consistent.There were two classes of product codes:<ul><li>AB123: Two letters followed by three digits</li><li>A99999: One letter followed by five digits</li></ul>The goal was to determine how many of each type existed, and how many were missing certain meta-data tags. Many people are stumped by situations like this, since it is initially unclear how to categorize two groups of product codes that are only related to each other by the structural format of their ID.<h3>Option 1: Pre-parse with Grok and Logstash</h3>The best solution is to deal with inconsistencies like this at the input level. For example, you can build a <a href="http://logstash.net/docs/1.4.2/filters/grok">Grok filter in Logstash</a> to identify the two different patterns and tag them appropriately. Once tagged, it is trivial to do aggregations on the structured document data. If a new pattern occurs in your data that doesn’t match the pattern, Logstash will emit a _grokparsefailure tag.This has the benefit of improving search results, since only properly tagged documents will be visible to search. You can go back and fix/reindex the “broken” data at your lesiure and iteratively improve your search results through better-tagged data.<h3>Option 2: Regex to the rescue</h3>But things are not always this simple. What if you just indexed 10TB of data and realized you forgot to include appropriate pre-parsing? Re-indexing would be a pain, or potentially impossible. We need a solution that operates on your existing data.The key is to use a regular expression and filter buckets. A <a href="/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filter-aggregation.html">filter bucket</a> will hold all documents matching its filtering criteria. If we place a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html" target="_self">regexp filter</a> inside the bucket, we can find all product IDs matching a certain pattern.Once documents have been sorted into one of the filter buckets, we can apply other bucketing and metrics to derive statistics.Let’s take a look at a very simple example. First we index some data with mixed product codes:<pre class="prettyprint lang-js">PUT /test/data/_bulk {"index":{}} {"product_code" : "AB123"} {"index":{}} {"product_code" : "XY345"} {"index":{}} {"product_code" : "AZ987"} {"index":{}} {"product_code" : "ZZ192"} {"index":{}} {"product_code" : "A99999"} {"index":{}} {"product_code" : "A12345"} {"index":{}} {"product_code" : "A98765"} {"index":{}} {"some_other_field" : "xyz"} {"index":{}} {"some_other_field" : "123"} </pre>Then we can run a very simple aggregation to sort out the various codes:<pre class="prettyprint lang-js">GET /test/data/_search?search_type=count { "aggs" :{ "total_count" : { "global" : {} }, "XX999" : { "filter" : { "regexp":{ "product_code" : "[a-z]{2}[0-9]{3}" } } }, "X99999" : { "filter" : { "regexp":{ "product_code" : "[a-z]{1}[0-9]{5}" } } }, "no_format" : { "missing" : { "field" : "product_code" } } } } </pre>This query will give a document count for each product code matching the two regex patterns. The missing bucket, which will get <a href="https://github.com/elasticsearch/elasticsearch/issues/5659">a nice performance boost in Elasticsearch v1.3.0</a>, will show all documents that don’t have any product code at all. From this base, it is easy to add extra metrics, such as the average price of each product, or the average number sold each day for the last month, etc.<h3>The takeaway tip</h3>The key to this tip is the filter bucket. This bucket accepts any Elasticsearch filter, which means you can construct the same kind of complex filtering operations which you already use in search requests. And because these are filters, they enjoy all the performance benefits inherent to filters.Starting in Elasticsearch version 1.3.0, you will also have access to the <a href="https://github.com/elasticsearch/elasticsearch/issues/6118">filters bucket</a> (note the plural). Although functionally equivalent to using multiple filter buckets, the filters simplifies the syntax for applying multiple filters at the same time.So next time you need to aggregate some statistics, but are stumped by the irregular or inconsistent nature of your data, remember the filter bucket (and the upcoming filters bucket).

Quick Tips: regex filter buckets

<html><head></head><body>With the release of Elasticsearch 1.1.0, there is a new metric aggregation available to users: the <a href="/guide/en/elasticsearch/reference/current/search-aggregations-metrics-percentile-aggregation.html">Percentile metric</a>. Percentiles tell you the value at which a certain percentage of your data is included. So a 95th percentile tells you the value which is greater than or equal to 95% of your data. 
Ok&#x2026;but why is that useful?
Imagine you are the administrator for a large website. One of your goals is to guarantee fast response times to all website visitors, no matter where in the world they live. How do you analyze your data to guarantee that the latency is small?
Most people reach for basic statistics like mean, median or max. Each have their place, but for populations of data they often hide the truth. Mean and median tend to hide outliers, since the majority of your data is &#x201C;normal&#x201D;. In contrast, the max is a hypercondriac and easily distorted by a single outlier.
Let&#x2019;s look at a graph. If you rely on simple metrics like mean or median, you might see a graph that looks like this:
<img src="https://api.contentstack.io/v2/assets/575e4c719e7a83165490bc9b/download?uid=blt86780645c2e698a4?uid=blt86780645c2e698a4" data-sys-asset-uid="bltaf8769c89900971b" alt="Mean + Median" style="max-width:100%;">
That doesn&#x2019;t look so bad, does it? Average and median response time is around 50ms, and creeps up to 100ms for a little while. A different truth is apparent when you include the 99th percentile:
<img src="https://api.contentstack.io/v2/assets/575e4c719985d58976376589/download?uid=blte0e63d25a4006d9c?uid=blte0e63d25a4006d9c" data-sys-asset-uid="blt2a9f08e2a9d169a0" alt="99th Percentile" style="max-width:100%;">
Wow! That certainly doesn&#x2019;t look good at all! At 9:30am, the mean is telling you &#x201C;Don&#x2019;t worry, the average latency is only 75ms&#x201D;. In contrast, the 99th percentile says &#x201C;99% of your values are less than 850ms&#x201D;, which is a very different picture. One percent of all your customers are experiencing 800+ ms latencies, which could be very bad for business.
<h3>Using the percentile</h3>
The new percentile metric works just like the simpler stats metrics like <code>min</code> and <code>avg</code>. It is a metric that can be applied to any aggregation bucket. The percentile metric will then calculate a set of percentiles based on the documents that fall within the bucket. Let&#x2019;s look at a simple example:
<pre class="prettyprint noescape">curl -XGET localhost:9200/website/logs/_search -d &apos;
{
 &quot;aggs&quot; : {
 &quot;load_time_outlier&quot; : {
 &quot;percentiles&quot; : {
 &quot;field&quot; : &quot;load_time&quot; 
 }
 }
 }
}&apos;
</pre>
By default, the <code>percentiles</code> metric will calculate a set of default percentiles (<code>[ 1, 5, 25, 50, 75, 95, 99 ]</code>) and return you the value for each one:
<pre class="prettyprint noescape">{
 ...

 &quot;aggregations&quot;: {
 &quot;load_time_outlier&quot;: {
 &quot;1.0&quot;: 15,
 &quot;5.0&quot;: 20,
 &quot;25.0&quot;: 33,
 &quot;50.0&quot;: 38,
 &quot;75.0&quot;: 45,
 &quot;95.0&quot;: 60,
 &quot;99.0&quot;: 867
 }
 }
}
</pre>
Often, only the extreme percentiles are important to you, such as the 95th and 99.9th percentile. In this case, you can specify just the percentile you are interested in:
<pre class="prettyprint noescape">curl -XGET localhost:9200/website/logs/_search -d &apos;
{
 &quot;aggs&quot; : {
 &quot;load_time_outlier&quot; : {
 &quot;percentiles&quot; : {
 &quot;field&quot; : &quot;load_time&quot; ,
 &quot;percents&quot; : [ 95, 99.9 ]
 }
 }
 }
}&apos;
</pre>
Being a metric, we can nest it inside of buckets to get more sophisticated analysis. Going back to our original goal &#x2014; detecting high latency based on geographical location &#x2014; we can build an aggregation that buckets users by their country and then computes a percentile on <code>load_time</code>
<pre class="prettyprint noescape">curl -XGET localhost:9200/website/logs/_search -d &apos;
{
 &quot;aggs&quot; : {
 &quot;countries&quot; : {
 &quot;terms&quot; : {
 &quot;field&quot; : &quot;country_code&quot; 
 },
 &quot;aggs&quot; : {
 &quot;load_time_outlier&quot; : {
 &quot;percentiles&quot; : {
 &quot;field&quot; : &quot;load_time&quot; ,
 &quot;percents&quot; : [ 95 ]
 }
 } 
 }
 }
 }
}&apos;
</pre>
And now we can see that Antarctica has a particularly slow 95th percentile (for some strange reason):
<pre class="prettyprint noescape">{
 ...

 &quot;aggregations&quot;: {
 &quot;country&quot; : {
 &quot;buckets&quot;: [
 {
 &quot;key&quot; : &quot;AY&quot;,
 &quot;doc_count&quot; : 20391,
 &quot;load_time_outlier&quot;: {
 &quot;95.0&quot;: 1205
 }
 },
 ...
</pre>
<h3>percentiles are (usually) approximate</h3>
All good things come at a price, and with percentiles it usually boils down to approximations. Fundamentally, percentiles are very expensive to calculate. If you want to calculate the 95th percentile, you need to sort all your values from least to greatest, then find the value at <code>myArray[ count(myArray) * 0.95]</code>
This works fine for small data that fits in memory, but simply fails when you have terrabytes of data spread over a cluster of servers (which is common for Elasticsearch users). The exact method just won&apos;t work for Elasticsearch.
Instead, we use an algorithm called T-Digest (<a href="https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf">you can read more about it here</a>). Without getting bogged down in technical details, it is sufficient to make the following claims about T-Digest:
<ul>
<li>For small datasets, your percentiles will be highly accurate (potentially 100% exact if the data is small enough)</li>
<li>For larger datasets, T-Digest will begin to trade accuracy for memory savings so that your node doesn&apos;t explode</li>
<li>Extreme percentiles (e.g. 95th) tend to be more accurate than interior percentiles (e.g. 50th)</li>
</ul>
The following chart shows the relative error on a uniform distribution depending on the number of collected values and the requested percentile:
<img src="https://api.contentstack.io/v2/assets/575e4c710342dfd738262c1d/download?uid=blt8effc77cdc9fc798?uid=blt8effc77cdc9fc798" data-sys-asset-uid="blt79b53c69a6579e34" alt="Percentiles error" style="max-width:100%;">
The absolute error of a percentile is the actual value minus the approximate value. It is often useful to express that as a relative percentage rather than in absolute difference. In the chart, we can see that at 1000 values, the 50th percentile is 0.26% off the true 50th percentile. In absolute terms, if the true 50th was <code>100ms</code>, T-Digest might have told us <code>100.26ms</code>. Practically speaking, the error is often negligible, especially when you are looking at the more extreme percentiles
The chart also shows how precision is as you add more data. The reason why error diminishes for large number of values is that the law of large numbers makes the distribution of values more and more uniform and the t-digest tree can do a better job at summarizing it. It would not be the case on more skewed distributions.
The memory-vs-accuracy tradeoff is configurable via a <code>compression</code> parameter, which you can <a href="/guide/en/elasticsearch/reference/current/search-aggregations-metrics-percentile-aggregation.html#_compression">find more details about in the documentation</a>.
<h3>Conclusion</h3>
Now armed with some basic knowledge about percentiles, hopefully you are beginning to see applications all over your data. These approximate algorithms are exciting new territory for Elasticsearch. We look forward to your feedback on the <a href="https://groups.google.com/forum/?fromgroups#!forum/elasticsearch">mailing list</a> or <a href="https://twitter.com/elasticsearch">Twitter</a>!</body></html>

Averages Can Be Misleading: Try a Percentile

<html><head></head><body>Stop words are a fundamental part of Information Retrieval lore. Common wisdom dictates that you should identify and remove stop words from your index. This increases both performance (fewer terms in your dictionary) and more relevant search results.
Elasticsearch supports stop word removal through the <a href="/guide/en/elasticsearch/reference/current/analysis-stop-tokenfilter.html">Stop token filter</a>, but a new query was recently added which makes this filter unnecessary: <a href="/guide/en/elasticsearch/reference/current/query-dsl-common-terms-query.html">Common Terms Query</a>.
<h2>Background</h2>
Before we go much farther, Why would you want to remove stop words anyway? In most cases, stop words add little semantic value to a sentence. They are filler words that help sentences flow better, but provide very little context on their own. Stop words are usually words like &#x201C;to&#x201D;, &#x201C;I&#x201D;, &#x201C;has&#x201D;, &#x201C;the&#x201D;, &#x201C;be&#x201D;, &#x201C;or&#x201D;, etc etc.
Even worse than being void of useful information&#x2026;they are everywhere! Stop words are the most frequent words in the English language. Because of this, most sentences share a similar percentage of stop words. Stop words bloat your index without providing any extra value.
If they are both common and lacking in much useful information, why not remove them?
Removing stop words helps decrease the size of your index as well as the size of your query. Fewer terms is always a win with regards to performance. And since stop words are semantically empty, relevance scores are unaffected.
<h2>Stop words are useless&#x2026;</h2>
&#x2026;unless you actually need them. The classic example is a phrase from Shakespeare:
<blockquote>&#x201C;To be or not to be.&#x201D;</blockquote>
This sentence is composed entirely of stop words, and would therefore not be represented in your index. That&#x2019;s a problem. At this point, people usually start maintaining multiple mappings of their data: one with stop words removed, and one that has stop words intact. They are boosted differently, to favor phrases with no stop words.
This procedure, however, eliminates the benefits of stop word removal. Instead of decreasing the number of terms being searched (and indexed), we just doubled it by indexing the field in two different ways!
<h2>Enter Common Terms</h2>
The new Common Terms Query is designed to fix this situations, and it does so through a very clever mechanism. At a high level, Common Terms analyzes your query, identifies which words are &#x201C;important&#x201D; and performs a search using just those words. Only after documents are matched with important words are the &#x201C;unimportant&#x201D; words considered
The motivation behind Common Terms is to leverage the power of stop word removal (faster searches) without eliminating stop words entirely (because they can contribute to score sometimes). You can have your cake and eat it too!
<h3>How Common Terms works</h3>
Let&#x2019;s first look at how you construct a Common query:
<pre class="prettyprint lang-json">{
 &quot;common&quot;: {
 &quot;body&quot;: {
 &quot;query&quot;: &quot;this is bonsai cool&quot;,
 &quot;cutoff_frequency&quot;: 0.001
 }
 }
}
</pre>
Note: Common Terms has also been incorporated into the Match query and can be enabled by setting `cutoff_frequency` to a value like 0.001
Several things happen when you execute this query:
<ul>
<li>The query is sent to each shard in the search</li>
<li>At each shard, Elasticsearch looks at the document frequencies for each term</li>
<li>If a term has a document frequency less than 0.1% (0.001) then it is considered &#x201C;low frequency&#x201D;. Otherwise, it is moved to a secondary &#x201C;high frequency&#x201D; list</li>
<li>The &#x201C;low frequency&#x201D; list is rewritten into a Boolean Should clause (logical OR). In this example, it would contain &#x201C;bonsai&#x201D;, &#x201C;cool&#x201D;</li>
<li>Any documents that match the low frequency list are then scored against the remaining high frequency list (&#x201C;this&#x201D;, &#x201C;is&#x201D;)</li>
</ul>
Internally, the query is rewritten into roughly this representation:
<pre class="prettyprint lang-json">{
 &quot;bool&quot;: {
 &quot;must&quot;: [
 { &quot;term&quot;: { &quot;body&quot;: &quot;bonsai&quot;}},
 { &quot;term&quot;: { &quot;body&quot;: &quot;cool&quot;}}
 ],
 &quot;should&quot;: [
 { &quot;term&quot;: { &quot;body&quot;: &quot;this&quot;}}
 { &quot;term&quot;: { &quot;body&quot;: &quot;is&quot;}}
 ]
 }
}
</pre>
It should be easy to see how this can greatly affect performance. The low frequency words act as a pre-filter, drastically reducing the number of terms that need to be evaluated/scored against the full index.
<h3>Adaptive Stop Word Evaluation</h3>
With traditional stop word schemes, you must first create a list of stop words. Every domain is unique when it comes to stop words: there are no &#x201C;pre-made&#x201D; stop word lists on the internet.
As an example, consider the word &#x201C;video&#x201C;. For most businesses, &#x201C;video&#x201D; is an important word &#x2013; it shouldn&#x2019;t be removed. But if you are Youtube, &#x201C;video&#x201D; is probably mentioned in thousands of places&#x2026;it is definitely a stop word in this context. Traditional stop word removal would need a human to sit down, compile a list of domain-specific stop words, add it to Elasticsearch and then routinely maintain the list with additions/deletions.
In contrast, Common Terms adapts to your domain. Words with high frequency will automatically be recognized as stop words. Different companies, using the same query, will get different stop word behavior due to the unique set of documents in their index.
<h3>Customizing Common Terms</h3>
Common Terms allows considerable customization. Since it is effectively creating two Bool queries, there are a number of options you can twiddle. Let&#x2019;s look at an example that uses all the options:
<pre class="prettyprint lang-json">{
 &quot;common&quot;: {
 &quot;body&quot;: {
 &quot;query&quot;: &quot;nelly the elephant not as a cartoon&quot;,
 &quot;cutoff_frequency&quot;: 0.001,
 &quot;low_freq_operator&quot;: &quot;or&quot;,
 &quot;high_freq_operator&quot;: &quot;or&quot;,
 &quot;minimum_should_match&quot;: {
 &quot;low_freq&quot; : &quot;60%&quot;,
 &quot;high_freq&quot; : &quot;20%&quot;
 }
 }
 }
}
</pre>
In this query, we are specifying that both the high- and low-frequency term booleans should use an &#x201C;Or&#x201D; operator (e.g. go into a Should clause). Alternatively, we could make one or both of these options an &#x201C;And&#x201D; (go into a Must clause).
We are also specifying how many of the terms we want to match, for both high- and low-frequencies. This can be either a percentage or an exact value. By fiddling with the cuttoff and minimum_should_match values, you can very easily hone in on the behavior that you want.
<h3>Revisiting Shakespeare</h3>
Let&#x2019;s revisit that Shakespearean phrase we saw earlier. What happens when we query for &#x201C;To be or not to be&#x201D; with Common Terms?
<pre class="prettyprint lang-json">{
 &quot;common&quot;: {
 &quot;body&quot;: {
 &quot;query&quot;: &quot;To be or not to be&quot;,
 &quot;cutoff_frequency&quot;: 0.001
 }
 }
}
</pre>
In most corpuses, all those terms are going to be well over 1%, so they are all going to be categorized as highly frequent terms. What does Common Terms do in this situation?
Since all terms are highly frequent and there are no low frequency terms, Elasticsearch converts the high-frequency list into a Must clause:
<pre class="prettyprint lang-json">{
 &quot;bool&quot;: {
 &quot;must&quot;: [
 { &quot;term&quot;: { &quot;body&quot;: &quot;to&quot;}},
 { &quot;term&quot;: { &quot;body&quot;: &quot;be&quot;}},
 { &quot;term&quot;: { &quot;body&quot;: &quot;or&quot;}},
 { &quot;term&quot;: { &quot;body&quot;: &quot;not&quot;}}
 ]
 }
}
</pre>
The theory is that, individually, these terms carry very little semantic information and will match too many documents. But when they are all required, they become much more exclusive and match a limited subset of documents.
Sometimes this approach can be too strict, so it may make sense to relax the requirements with a minimum_should_match:
<pre class="prettyprint lang-json">{
 &quot;common&quot;: {
 &quot;body&quot;: {
 &quot;query&quot;: &quot;To be or not to be&quot;,
 &quot;cutoff_frequency&quot;: 0.001,
 &quot;high_freq_operator&quot;: &quot;or&quot;,
 &quot;minimum_should_match&quot;: {
 &quot;high_freq&quot; : &quot;70%&quot;
 }
 }
 }
}
</pre>
Which will convert the Must clause into a Should that must match 70% of the terms. This is more restrictive than a simple OR, but less restrictive than an AND.
<h2>Conclusion</h2>
The Common Terms Query is exciting, since it is truly a win-win. You gain the speed of stop word removal, but maintain the precision of leaving stop words in the index. Stop words will still have their uses, but we envision the majority of queries converting to a Common query. 
It is a powerful optimization that doesn&#x2019;t sacrifice relevance for speed. Give it a shot with your data and see if it helps boost performance!</body></html>

Stop Stopping Stop Words: a Look at Common Terms Query

<html><head></head><body><table style="background: #FFFFD2;">
<tbody>
<tr>
<td>NOTE: This article now contains outdated information. Please reference our <a href="https://www.elastic.co/guide/index.html">docs</a>, peruse our <a href="https://www.elastic.co/blog">latest blogs</a>, and visit our <a href="https://discuss.elastic.co/">forums</a> for the latest and greatest. Thank you.</td>
</tr>
</tbody>
</table>There is a new feature in the 0.90 branch that is pretty awesome: the <a href="/guide/reference/query-dsl/terms-filter/">Terms Filter</a> now supports document lookups.
In a normal Terms Filter, you provide a list of Terms that you want to filter against. This is fine for small lists, but what if you have 100 terms? A thousand terms? That is a lot of data to pass over the wire. If that list of terms is stored in your index somewhere, you also have to retrieve it first&#x2026;just so you can pass it back to Elasticsearch.
The new lookup feature tells Elasticsearch to use another document as your terms array. Instead of passing 1000 terms, you simply tell the Terms Filter &#x201C;Hey, all the terms I want are in this document&#x201D;. Elasticsearch will fetch that document internally, extract the terms and perform your query.
Let&#x2019;s work our way through a concrete example of how it all works.
<h2>A concrete example</h2>The <a href="/guide/reference/query-dsl/terms-filter/">Terms Filter page has a good Twitter example.</a> I encourage you to read over it. But I come from a biology background and you can only read so many Twitter examples&#x2026;so let&#x2019;s try something a bit different and bioinformatics-y.
Imagine you run a bioinformatics search engine and database. You have two indices holding two different types of data. First, you have an index that stores academic research articles.
This document represents a single scientific paper. It has two fields, the title of the paper and a list of all proteins relevant to the topic:
<pre class="prettyprint lang-bash">curl -XPUT localhost:9200/papers/paper/paper789 -d &apos;{
 &quot;title&quot; : &quot;Ahi1, whose human ortholog is mutated in Joubert syndrome, is required for Rab8a localization, ciliogenesis and vesicle trafficking.&quot;
 &quot;proteins&quot; : [ &quot;Ahi1&quot;, &quot;Rab8a&quot; ]
}&apos;
</pre>Next, you have an index which holds data from a <a href="http://en.wikipedia.org/wiki/DNA_microarray">microarray experiment</a>. Microarrays are a method to determine whether a gene&#x2019;s activity is more (&#x201C;up-regulated&#x201D;) or less (&#x201C;down-regulated&#x201D;) than its normal rate. Microarrays are about the size of a postage stamp and test thousands of genes at once.
The resulting &#x201C;up-regulated&#x201D; field could potentially be thousands long: 
<pre class="prettyprint lang-bash">curl -XPUT localhost:9200/microarrays/experiment/experiment1234 -d &apos;{
 &quot;upregulated_proteins&quot; : [ &quot;Ahi1&quot;, &quot;Wnt&quot;, &quot;SHH&quot;, &quot;CDC42&quot;, &quot;GSK-3B&quot;, [.......] ]
}&apos;
</pre>Given this data, a very common question might be: &#x201C;Show me all papers about the proteins up-regulated in this experiment&#x201D;.
<h2>Filtering the papers</h2>Before lookups, the way to accomplish this is a GET plus a Filtered Search. First you have to GET the microarray experiment and extract the array of terms:
<pre class="prettyprint lang-bash">curl -XGET localhost:9200/microarrays/experiment/experiment1234
# ...Extract array in your app... #
</pre>Then perform a search query using the filter you need:
<pre class="prettyprint lang-bash">curl -XGET localhost:9200/papers/paper/_search -d &apos;{
 &quot;query&quot;:{
 &quot;filtered&quot;:{
 &quot;filter&quot;:{
 &quot;terms&quot; : {
 &quot;proteins&quot;:[ &quot;Ahi1&quot;, &quot;Wnt&quot;, &quot;SHH&quot;, &quot;CDC42&quot;, &quot;GSK-3B&quot;, [.......] ]
 }
 }
 }
 }
}
</pre>This works, but you can see why it isn&#x2019;t ideal. Not only do we need to perform two requests &#x2013; a GET and a Search &#x2013; but we have to shuffle a potentially large term array across the wire twice. The lookup feature allows you to bypass this inefficiency.
<h2>Filtering, this time with Lookups</h2>Lookups use documents themselves as the list of Terms, which means you can avoid unnecessary requests. Let&#x2019;s try again, but this time with lookups.
The data is organized the same as before, but when we search we skip the extraneous GET phase and go straight to the Terms Filter with the new Lookup syntax: 
<pre class="prettyprint lang-bash">curl -XGET localhost:9200/papers/paper/_search -d &apos;{
 &quot;query&quot; : {
 &quot;filtered&quot; : {
 &quot;filter&quot; : {
 &quot;terms&quot; : {
 &quot;proteins&quot; : {
 &quot;index&quot; : &quot;microarrays&quot;,
 &quot;type&quot; : &quot;experiment&quot;,
 &quot;id&quot; : &quot;experiment1234&quot;,
 &quot;path&quot; : &quot;upregulated_proteins&quot;
 },
 &quot;_cache_key&quot; : &quot;experiment_1234&quot;
 }
 }
 }
 }
}&apos;
</pre>Neat! How does it work? Let&#x2019;s just look at the new Terms Filter syntax line by line:
<blockquote>
	<pre class="prettyprint lang-json">{
 &quot;terms&quot;:{
 &quot;proteins&quot;:{
 </pre>Filter terms in the <code>&quot;proteins&quot;</code> field&#x2026;.
	<pre class="prettyprint lang-json"> &quot;index&quot;:&quot;microarrays&quot;,
 &quot;type&quot;:&quot;experiment&quot;,
 &quot;id&quot;:&quot;experiment1234&quot;,
 &quot;path&quot;:&quot;upregulated_proteins&quot;
 </pre>&#x2026;matching the array of terms located in the <code>&quot;upregulated_proteins&quot;</code> field, inside the document located at <code>/microarrays/experiment/experiment1234</code>&#x2026;
	<pre class="prettyprint lang-json"> },
 &quot;_cache_key&quot;:&quot;experiment_1234&quot;
 }
}
 </pre>&#x2026;and save (cache) the result of this filter under the name &#x201C;experiment_1234&#x2033; so we can use it again later without loading the document.
</blockquote><h2>More than just convenience</h2>The new Lookup feature is certainly useful. But it offers more than just convenience: there are tangible performance benefits.
<ul>
	<li>Eliminates the need for extraneous round-trips
	<ul>
		<li>Removes network latency</li>
	</ul>
	</li>
	<li>Caches the result
	<ul>
		<li>Removes the need to even load the term lookup document on subsequent requests</li>
	</ul>
	</li>
	<li>Custom cache key name
	<ul>
		<li>Reduces cache memory usage. The Terms Filters usually construct cache-key names by concatenating together the list of Terms. Obviously, a 1000 term cache-key is going to have a very long string for a name, using a lot of memory.</li>
	</ul>
	</li>
</ul>Performance can be boosted even more if the Lookup index (<code>/microarray/</code> in this example) is fully replicated to each node. The Lookup will prefer shards that are local, removing the need to query other shards to get the lookup document. While inter-node latency is usually pretty low, zero latency is always faster.
<h2>Conclusion</h2>This is just one example of using the new Lookup feature. Lookups are predominantly used to boost performance when filtering large Term lists. Check out the <a href="/guide/reference/query-dsl/terms-filter/">Terms Filter documentation</a> for more details about settings (adjusting cache memory, etc) as well as the standard Twitter example.
</body></html>

Terms Filter Lookup

<html><head></head><body><section id="post_content">
<article>
In this article, we are going to take a look at Elasticsearch&#x2019;s <a href="http://www.elasticsearch.org/guide/reference/mapping/routing-field/">Routing</a> feature. We&#x2019;ll discuss what routing is and why you might consider using a non-default routing scheme.

<h2>What is Routing?</h2>
All of your data lives in a primary shard, somewhere in the cluster. You may have five shards or five hundred, but any particular document is only located in one of them. Routing is the process of determining which shard that document will reside in.

Because Elasticsearch tries hard to make defaults work for 90% of users, routing is handled automatically. For most users, it doesn&#x2019;t matter where a document is stored.

The default routing scheme hashes the ID of a document and uses that to find a shard. This includes both user-provided IDs and randomly generated IDs picked by Elasticsearch. Default routing gives an even distribution of documents across the entire set of shards &#x2013; you won&#x2019;t have any &#x201C;hotspots&#x201D; where documents tend to cluster on one shard or another.

<h2>The need for custom routing</h2>
Random routing works well most of the time, but there are scenarios where domain knowledge of your data can lead to better performance with a custom routing solution. Imagine a single index with 20 shards (overallocated to support future growth).

What happens when a search request is executed on the cluster?

<ol>
	<li>Search request hits a node</li>
	<li>The node broadcasts this request to every shard in the index (either a primary or replica shard)</li>
	<li>Each shard performs the search query and responds with results</li>
	<li>Results are merged on the gateway node, sorted and returned to the user</li>
</ol>
Elasticsearch has no idea where to look for your document. All the docs were randomly distributed around your cluster&#x2026;so Elasticsearch has no choice but to broadcasts the request to all 20 shards. This is a non-negligible overhead and can potentially impact performance.

Wouldn&#x2019;t it be nice if we could tell Elasticsearch which shard the document lived in? Then you would only have to search one shard to find the document(s) that you need.

This is exactly what custom routing does.

Instead of blindly broadcasting to all shards, you tell Elasticsearch &#x201C;Hey! Search for the data on this shard! It&#x2019;s all there, I promise!&#x201D;. For example, you could route documents based on their user_id. Or their zip or postcode. Or whatever is commonly searched/filtered in your application.

Routing ensures that all documents with the same routing value will locate to the same shard, eliminating the need to broadcast searches. This design pattern is often called the &#x201C;User Data Flow&#x201D;, popularized from <a href="http://vimeo.com/44716955#t=825">Shay&#x2019;s talk at Berlin Buzzwords</a>.

<h2>Custom Routing is Fast</h2>
This point is so important, so central to custom routing, that it deserves its own section. Custom routing ensures that only one shard is queried.

This has the potentially to increase performance noticeably, if your problem fits into the niche that custom routing serves.
It doesn&#x2019;t matter if you have 20 or 100 shards in your index, custom routing ensures that only the shards holding your data are queried. Under the right data organization, this can be the difference between a cluster that is struggling and one that doesn&#x2019;t break a sweat.

<h2>Setting up your custom routing</h2>
To begin using custom routing, you need to tell Elasticsearch that you want certain types to use your routing instead of default. This is done by setting the mapping for a particular type:

<pre class="prettyprint lang-bash">$ curl -XPUT &apos;http://localhost:9200/store/order/_mapping&apos; -d &apos;
{
 &quot;order&quot;:{
 &quot;_routing&quot;:{
 &quot;required&quot;:true
 }
 }
}&apos;
</pre>
<blockquote>Note: setting &#x201C;required&#x201D; to true is not explicitly required to use routing, but it is a good practice. If set to true, Elasticsearch will raise errors when you attempt certain operations without specifying a routing value. This will save you a lot of heartache.</blockquote>
This mapping says that every document in the &#x201C;order&#x201D; type must specify a custom routing. If you try to index a document without specify a routing, the operation will fail with an error. Indexing a document with custom routing is very easy &#x2013; you just specify the routing in the URI:

<pre class="prettyprint lang-bash">$ curl -XPOST &apos;http://localhost:9200/store/order?routing=user123&apos; -d &apos;
{
 &quot;productName&quot;:&quot;sample&quot;,
 &quot;customerID&quot;:&quot;user123&quot;
}
</pre><h3 style="margin-top: 30px;">Extracting routing from the document</h3>
<del>Sometimes it is necessary to derive the routing value from the document itself. Elasticsearch can automatically extract the routing information if you tweak the mapping just slightly and add a <code>path</code> parameter:
</del>
<pre class="prettyprint lang-bash">&lt;del&gt;$ curl -XPUT &apos;http://localhost:9200/store/order/_mapping&apos; -d &apos;
{
 &quot;order&quot;:{
 &quot;_routing&quot;:{
 &quot;required&quot;:true,
 &quot;path&quot;:&quot;customerID&quot;
 }
 }
}&apos;
&lt;/del&gt;</pre>
<del>Now when you index, you can omit the routing parameter on the URI:
</del>
<pre class="prettyprint lang-bash">&lt;del&gt;$ curl -XPOST &apos;http://localhost:9200/store/order&apos; -d &apos;
{
 &quot;productName&quot;:&quot;sample&quot;,
 &quot;customerID&quot;:&quot;user123&quot;
}&apos;
&lt;/del&gt;</pre>
<del>Note: even though extraction is convenient, it is less performant than specifying the routing in the URI. The gateway node must load the document, extract the routing value, then serialize the document to the final shard/node. If you specify the routing in the URI, Elasticsearch can immediately serialize the doc to the appropriate shard without any overhead of loading the doc.</del>
WARNING: extracting custom routing from the document is no longer supported; it was <a href="https://github.com/elastic/elasticsearch/pull/11074">removed in Elasticsearch v2.0</a>. &#xA0;The ability to extract routing from the document itself can lead to strange edge-cases and potential trouble (e.g. if the field is multi-valued, which value do you use for routing?) &#xA0;If you are &lt;v2.0, it is not recommended to use extraction. 
<h2>Searching with Routing</h2>
Great! You have routing setup! So how do you start searching with routing? It is easy. Simply append the appropriate routing parameter to the URI:

<pre class="prettyprint lang-bash">$ curl -XGET &apos;http://localhost:9200/store/order/_search?routing=user123&apos; -d &apos;
{
 &quot;query&quot;:{
 &quot;filtered&quot;:{
 &quot;query&quot;:{
 &quot;match_all&quot;:{}
 },
 &quot;filter&quot;:{
 &quot;term&quot;:{
 &quot;userID&quot;:&quot;user123&quot;
 }
 }
 }
 }
}&apos;
</pre>
This query will find all the orders of user &#x201C;user123&#x2033;. Notice that in this particular example, a filtered query is essential.

If we instead just used the <code>match_all</code>:

<pre class="prettyprint lang-bash">$ curl -XGET &apos;http://localhost:9200/store/order/_search?routing=user123&apos; -d &apos;
{
 &quot;query&quot;:{
 &quot;match_all&quot;:{}
 }
}&apos;
</pre>
What you are actually asking Elasticsearch is &#x201C;Match all documents that reside on the shard which &#x2018;user123&#x2032; routes to&#x201D;.

This shard almost certainly has documents other than those belonging to user123. Routing ensures that documents of a particular routing value all go to the same shard&#x2026;but that doesn&#x2019;t mean that other documents aren&#x2019;t routed to the shard too.

So make sure you include filters where appropriate. :)
<h3>Searching multiple routing values</h3>
When a document is indexed, you may only specify a single routing value. It doesn&#x2019;t make sense for a document to be routed to multiple shards!

But when searching, you can specify more than one routing value to search on. Imagine an ACL system where routing is based on a UserGroup. A user can be a member of several groups, so when searching you want to search all documents that the user has access to.

Each UserGroup is sandboxed to a single shard and with multiple routing values you can search just the shards that are relevant:

<pre class="prettyprint lang-bash">$ curl -XGET &apos;http://localhost:9200/forum/posts?routing=Admin,Moderator&apos; -d &apos;{...}&apos;
</pre>
That will search both the &#x201C;Admin&#x201D; and the &#x201C;Moderator&#x201D; shard, while ignoring every other UserGroup.

<h2>Considerations and complications</h2>
There are a several&#xA0;gotchas that you should be aware of when implementing a custom routing solution.&#xA0;
<h3>Parent/Child/Grandchild schemes</h3>
Parent/Child mappings ensure that all children are routed to the same shard as the parent for performance reasons. Internally, Elasticsearch is setting the child&#x2019;s routing value equal to the ID of the parent, ensuring that everyone colocates to the same shard.

However, if you add a third tier &#x2013; grandchildren &#x2013; parent/child mapping fails. Look at these sample documents:

<pre class="prettyprint lang-bash">curl -XPUT localhost:9200/parentchild/product/Product001 -d &apos;{...}&apos;
curl -XPUT localhost:9200/parentchild/vendors/VendorABC?parent=Product001 -d &apos;{...}&apos;
curl -XPUT localhost:9200/parentchild/vendordetails/LocationXYZ?parent=VendorABC -d &apos;{...}&apos;
</pre>
Fairly straightforward nested relationships. Products have children called &#x201C;Vendors&#x201D;, while those have their own children called &#x201C;VendorDetails&#x201D;. So why doesn&#x2019;t this work? Let&#x2019;s look at the derived routing values:

<table style="width: 600px; margin-left: auto; margin-right: auto; text-align: center;">
<thead style="font-weight: bold; border-bottom: solid 1px black; margin-bottom: 10px;">
<tr>
	<th>Doc ID
	</th>
	<th>Parent
	</th>
	<th>Routing Value
	</th>
</tr>
</thead>
<tbody>
<tr>
	<td style="padding-top: 10px;">Product001
	</td>
	<td style="padding-top: 10px;">-
	</td>
	<td style="padding-top: 10px;">Product001
	</td>
</tr>
<tr>
	<td>VendorABC
	</td>
	<td>Product001
	</td>
	<td>Product001
	</td>
</tr>
<tr>
	<td>LocationXYZ
	</td>
	<td>VendorABC
	</td>
	<td>VendorABC
	</td>
</tr>
</tbody>
</table>
As you can see, docs will automatically use the ID of their parent for routing. VendorABC uses Product001 (fine so far), but LocationXYZ uses VendorABC (bad). This means data is not being colocated correctly.

The solution is simple: tell the grandchild (and any subsequent great-grandchildren) that it&#x2019;s routing value should be the ID of the grandparent. Internally, Elasticsearch will give preference to the routing parameter over the parent parameter:

<pre class="prettyprint lang-bash">curl -XPUT localhost:9200/parentchild/product/Product001 -d &apos;{...}&apos;
curl -XPUT localhost:9200/parentchild/vendors/VendorABC?parent=Product001 -d &apos;{...}&apos;
curl -XPUT localhost:9200/parentchild/vendordetails/LocationXYZ?parent=VendorABC&amp;routing=Product001 -d &apos;{...}&apos;
</pre>
<table style="width: 600px; margin-left: auto; margin-right: auto; text-align: center;">
<thead style="font-weight: bold; border-bottom: solid 1px black; margin-bottom: 10px;">
<tr>
	<th>Doc ID
	</th>
	<th>Parent
	</th>
	<th>Routing Value
	</th>
</tr>
</thead>
<tbody>
<tr>
	<td style="padding-top: 10px;">Product001
	</td>
	<td style="padding-top: 10px;">-
	</td>
	<td style="padding-top: 10px;">Product001
	</td>
</tr>
<tr>
	<td>VendorABC
	</td>
	<td>Product001
	</td>
	<td>Product001
	</td>
</tr>
<tr>
	<td>LocationXYZ
	</td>
	<td>VendorABC
	</td>
	<td>Product001
	</td>
</tr>
</tbody>
</table>
<h3>Hotspots</h3>
When Elasticsearch manages the routing, it ensures that distribution is fairly uniform across all your shards. However, once you start implementing your own custom schemes, it is entirely possible that this uniformity is lost. Say you are routing by userID. Most of your users are small and have a handful of documents&#x2026;but occasionally you get a monster user that has millions.

Instead of being distributed uniformly around the cluster, custom routing will ensure that all the documents of MonsterUser&#xA0;go to a single shard. That&#x2019;s good for performance &#x2013; searches will execute quickly.

But what happens if MonsterUser2 and MonsterUser3 also get allocated to the same shard? Now you have three big users located on a single shard while the rest of your shards are only lightly loaded.

Not a great situation. These types of scenarios can and do happen. The best defense against these &#x201C;data hotspots&#x201D; is to manually identify large (or otherwise performance-hungry) users and split them into their own index. You can then setup an <a href="http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases/">alias</a> to make the separation transparent to your application.

Now you have the best of both worlds: custom routing to keep most users sandboxed to a single shard, while big users are split into their own index and set of shards. Cool!

<h3>You can check-out any time you like, but you can never leave!</h3>
Once you specify&#xA0;custom routing when indexing&#xA0;a document, you must keep specifying it whenever&#xA0;you want to update it or when you want to delete it. You&apos;ve taken some control from Elasticsearch and there&apos;s no easy way to give this&#xA0;control back if you decide you want to stop using custom routing at some point. The only way is to reindex your data (without specifying custom routing).

<h2>Conclusion</h2>
Custom routing is a powerful feature that can boost performance in specific situations. While default routing will suffice for most people, sometimes you just need more control over the placement of documents.

<aside id="post_tags">
</aside>
</article>
</section></body></html>

Customizing Your Document Routing

<html><head></head><body>WARNING: This article contains outdated information. We no longer recommend taking its advice.<section id="post_content">
<article>
When building queries in Elasticsearch, you&#x2019;ll often find yourself composing sets of filters. Say you need to filter users that are:

<ul>
	<li>Gender: Male</li>
	<li>Age: 23-26</li>
	<li>Language: English</li>
</ul>
Generally you should place these filters inside of a <a href="http://www.elasticsearch.org/guide/reference/query-dsl/bool-filter/">Boolean Filter</a>.

But wait&#x2026;isn&#x2019;t there an <a href="http://www.elasticsearch.org/guide/reference/query-dsl/and-filter/">And Filter</a>? What about the <a href="http://www.elasticsearch.org/guide/reference/query-dsl/or-filter/">Or Filter</a> and <a href="http://www.elasticsearch.org/guide/reference/query-dsl/not-filter/">Not Filters</a>? Are they simply an alternative syntax to the Bool filter?

The short answer is: No, they are very distinct. More importantly, they can have a big impact on your query performance.

<h2>BitSets</h2>
The key to understanding the difference will require a detour into BitSets. Fundamentally, a <a href="http://en.wikipedia.org/wiki/Bit_array">BitSet</a> is just an array of bits that represent state. A position in the bit array is either a 1 or a 0.

Filters don&#x2019;t score documents &#x2013; they simply include or exclude. If a document matches a filter, it is represented with a one in the BitSet; otherwise a zero. This means that Elasticsearch can store an entire segment&#x2019;s filter state (&#x201C;who matches this particular filter?&#x201D;) in a single, compact BitSet.

The first time Elasticsearch executes a filter, it parses Lucene segment data structures to determine what matches your filter. Instead of throwing away this information, it caches it inside a BitSet.

The next time the same filter is executed, Elasticsearch can reference the compact BitSet instead of the Lucene segments. This has huge performance benefits.

<h2>How do I love thee BitSets? Let me count the ways&#x2026;</h2>
It is hard to overstate how fast bitwise operations are.

Bitwise operations are such a basic unit of computation that a CPU has dedicated bitwise circuitry. Performing an in-memory bitwise AND is orders of magnitude faster than parsing Lucene data structures and performing the intersection manually.

BitSets play nicely with other filter BitSets too. Have three filters? Elasticsearch will do a bitwise AND on all three bitsets to derive your final matching set of documents.

Even better, BitSets are cached independent of the query itself. A complex query could have a dozen filters, but each of those filter BitSets are independant and entirely reusable in other contexts. This allows Elasticsearch to be very efficient with filter reuse.

Furthermore, since BitSets are stored on a per-segment basis, Elasticsearch can do some cool performance tricks. Lucene segments are immutable &#x2013; once written to disk, they will never change.

If a particular filter doesn&#x2019;t match any documents in a segment (the BitSet is all zeros), in the future Elasticsearch can ignore that entire BitSet when performing it&#x2019;s filtering operation.

Similarly, as new segments are added, cached filter BitSets do not need to be invalidated. If you are indexing new documents into a MySQL table, for example, the B-Tree index is constantly updating due to the nature of B-Trees (they are sorted data structures).

With Elasticsearch filter caches, only newly created segments need to build filter BitSet; old BitSets can be reused without modification.

<h2>Boolean vs. And/Or/Not</h2>
&#x201C;Ok, we get it. BitSets are cool! Why does this concern me?&#x201D; you may be asking yourself.

It matters because the Bool filter utilizes BitSets while the And/Or/Not filters do not. If you put a Terms Filter inside of an And&#x2026;no BitSet will be used, even though it exists.

Why?

And/Or/Not work on a doc-by-doc basis. They first load whatever data is needed into the Field Data memory pool then iterate over the docs, excluding as they go. BitSets are not used and there is no composability from cached filters. Elasticsearch is simply scanning down the list of documents and checking each one individually

If you have multiple Filters, And/Or/Not will &#x201C;short-circuit&#x201D; the operation, only passing the matching docs to the next filter.

This reduces the amount of work that each subsequent filter needs to perform. For this reason, your &#x201C;heaviest&#x201D; filter should always be placed last &#x2013; typically Geo filters since they can perform some heavy computations to determine distance.

<h2>When to use And/Or/Not</h2>
It may seem that Bool filters are superior in every regard. Is there a time to use And/Or/Not?

And/Or/Not are more performant when you are working with filters that do not return a bitset. These are operations that must iterate over every document anyway. For example, a custom script is not BitSet-able because it performs computation on every document.

In these cases, And/Or/Not is a better choice than the Bool. The list of Non-Bitset filters is very small, so it is easy to remember:

<ul>
	<li>Geo* filters</li>
	<li>Scripts</li>
	<li>Numeric_range</li>
</ul>
That&#x2019;s it! Every other filter should be placed inside of a Bool.

<h2>Combining Bool with And/Or/Not</h2>
You&#x2019;ll often run into situations where you need several BitSet filters and several Non-BitSet filters. In these situations, combine Bool with And/Or/Not where appropriate. Always wrap the entire composition with an And/Or/Not. So for example, if you have:

<ul>
	<li>Gender: Male</li>
	<li>Age: 23-26</li>
	<li>Language: English</li>
	<li>Custom Script</li>
	<li>Geo</li>
</ul>
Your filter list may look something like this:

<pre class="prettyprint lang-json">{
 &quot;and&quot; : [
 {
 &quot;bool&quot; : {
 &quot;must&quot; : [
 { &quot;term&quot; : {} },
 { &quot;range&quot; : {} },
 { &quot;term&quot; : {} }
 ]
 }
 },
 { &quot;custom_script&quot; : {} },
 { &quot;geo_distance&quot; : {} }
 ]
}
</pre>
<h2>Conclusion</h2>
Filters enable you to find documents that you want, but also boost query performance by offloading simple exclusion tasks to fast BitSet operations. When composing filters, make sure you spend a few extra minutes organizing them to use the appropriate class of aggregating filter:

<ul>
	<li>Geo, Script or Numeric_range filter: Use And/Or/Not Filters</li>
	<li>Everything else: Use Bool Filter</li>
</ul>
<h3>Some additional sources:</h3>
<ul>
	<li><a href="https://groups.google.com/d/msg/elasticsearch/2SusmSoShlo/qlOAbWmDRqwJ">Filter execution order</a></li>
	<li><a href="https://groups.google.com/d/msg/elasticsearch/PS12RcyNSWc/I1PX1r0RfFcJ">Bool vs And/Or/Not</a></li>
</ul>
<aside id="post_tags">
</aside>
</article>
</section>
</body></html>

All About Elasticsearch Filter BitSets

Data in the real world is rarely simple - often times it is a jumble of interlocking relations.How do you represent relational data in Elasticsearch? There are a few mechanisms that can be used to provide relation support. Each has their pros and cons, making them useful for different situations.<h3>Inner Objects</h3>The simplest mechanism are named "inner objects". These are JSON objects embedded inside your parent document:<pre class="prettyprint noescape">{ "name":"Zach", "car":{ "make":"Saturn", "model":"SL" } } </pre>Simple, right? The "car" field is just another JSON object, with the inner object having two properties ("make" and "model"). This inner object mapping will work as long as you have a one-to-one relationship between the root object and the inner object. E.g. every person has at most one "car".But what if Zach owns two cars, and we add another person (Bob) who owns just one car?<pre class="prettyprint noescape">{ "name" : "Zach", "car" : [ { "make" : "Saturn", "model" : "SL" }, { "make" : "Subaru", "model" : "Imprezza" } ] } { "name" : "Bob", "car" : [ { "make" : "Saturn", "model" : "Imprezza" } ] } </pre>Ignoring the fact that Saturn never made an Imprezza car, what happens when we try to search for it? Logically, only Bob has a "Saturn Imprezza", so we should be able to do a query like:`query: car.make=Saturn AND car.model=Imprezza`Right? Well, no, that doesn't work. If you perform that query, you'll receive both documents as the result. What happens is that Elasticsearch internally flattens inner objects into a single object. So Zach's entry actually looks like this:<pre class="prettyprint noescape">{ "name" : "Zach", "car.make" : ["Saturn", "Subaru"] "car.model" : ["SL", "Imprezza"] } </pre>Which explains why it was returned as a result. Elasticsearch is fundamentally flat, so internally the documents are represented as flattened fields. Hmm.<h3>Nested</h3>As an alternative to inner objects, Elasticsearch provides the concept of " <a href="https://www.elastic.co/guide/reference/mapping/nested-type.html">nested types</a>". Nested documents look identical to inner objects at the document level, but provide the functionality we were missing above (as well as some limitations).Example nested document: <pre class="prettyprint noescape">{ "name" : "Zach", "car" : [ { "make" : "Saturn", "model" : "SL" }, { "make" : "Subaru", "model" : "Imprezza" } ] } </pre>At the mapping level, nested types must be explicitly declared (unlike inner objects, which are automatically detected):<pre class="prettyprint noescape">{ "person":{ "properties":{ "name" : { "type" : "string" }, "car":{ "type" : "nested" } } } } </pre>The problem with inner objects was that each nested JSON object is not treated as an individual component of the document. Instead they were merged with other inner objects sharing the same property names.This is not the case with nested documents. Each nested doc remains independent, and you can perform a query like `car.make=Saturn AND car.model=Imprezza` without a problem.Elasticsearch is still fundamentally flat, but it manages the nested relation internally to give the appearance of nested hierarchy. When you create a nested document, Elasticsearch actually indexes two separate documents (root object and nested object), then relates the two internally. Both docs are stored in the same Lucene block on the same Shard, so read performance is still very fast.This arrangement does come with some disadvantages. Most obvious, you can only access these nested documents using a special ` <a href="https://www.elastic.co/guide/reference/query-dsl/nested-query.html">nested query</a>`. Another big disadvantage comes when you need to update the document, either the root or any of the objects.Since the docs are all stored in the same Lucene block, and Lucene never allows random write access to it's segments, updating one field in the nested doc will force a reindex of the entire document.This includes the root and any other nested objects, even if they were not modified. Internally, ES will mark the old document as deleted, update the field and then reindex everything into a new Lucene block. If your data changes often, nested documents can have a non-negligible overhead associated with reindexing.Lastly, it is not possible to "cross reference" between nested documents. One nested doc cannot "see" another nested doc's properties. For example, you are not able to filter on "A.name" but facet on "B.age". You can get around this by using `include_in_root`, which effectively copies the nested docs into the root, but this get's you back to the problems of inner objects.<h3>Parent/Child</h3>The last method that Elasticsearch provides are <a href="https://www.elastic.co/guide/reference/mapping/parent-field.html">Parent/Child types</a>. This scheme is a looser coupling than nested, and gives you a set of slightly more powerful queries. Let's look at an example where a single person has multiple homes (in different states). Your parent has a mapping as usual, perhaps:<pre class="prettyprint noescape">{ "mappings":{ "person":{ "name":{ "type":"string" } } } } </pre>While your children have their own mapping outside the parent, with a special `_parent` property set: <pre class="prettyprint noescape">{ "homes":{ "_parent":{ "type" : "person" }, "state" : { "type" : "string" } } } </pre>The `_parent` field tells Elasticsearch that the "Employers" type is a child of the "Person" type. Adding documents to this scheme is very easy. The parent doc is indexed as normal:<pre class="prettyprint noescape">$ curl -XPUT localhost:9200/test/person/zach/ -d' { "name" : "Zach" } </pre>And indexing children documents is almost like normal, except you need to specify which parent this child belongs to in the query parameter ('zach' in this case, which is the ID that we used in the above document): <pre class="prettyprint noescape">$ curl -XPOST localhost:9200/homes?parent=zach -d' { "state" : "Ohio" } $ curl -XPOST localhost:9200/test/homes?parent=zach -d' { "state" : "South Carolina" } </pre>Both of these documents are now associated with the 'zach' parent document, which allows you to use special queries such as:<ul><li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-has-parent-query.html" target="_self">Has Parent Query</a>, which works on parent documents and return children.</li><li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-has-child-query.html" target="_self">Has Child Query</a>, which works on children documents and returns parents</li></ul>You can also query the parents or children types individually, since they are first-class types and will respond to normal queries as usual (you just can't use the relationship values).The big problem with Nested was their storage: everything is stored in the same Lucene block. Parent/Child removes this limitation by separating the two documents and only loosely coupling them. There are some pros and cons to this. The loose coupling means you are more free to update/delete children docs, since they have no effect on the parent or other children.The downside is that Parent/Child are slightly less performant than Nested. The children docs are routed to the same shard as the parent, so they will still benefit from shard-level caches and memory filtering. But they aren't quite as fast as nested since they are not colocated in the same Lucene block. There is also a bit more memory overhead, since ElasticSearch needs to keep an in-memory "join table", which manages the relations.Lastly, you'll run into situations where sorting or scoring are, frankly, very difficult. For example, it is impossible to tell which child documents matched your `Has_Child` filter, just that one of the docs of the returned parent matched the criteria. This can be frustrating depending on your use-case.<h3>Denormalization</h3>Sometimes the best option is to simply denormalize your data where appropriate. The relational facilities that Elasticsearch provides are great for certain scenarios...but were never meant to provide the robust relational features that you expect from an RDBM.At it's heart, Elasticsearch is a flat hierarchy and trying to force relational data into it can be very challenging. Sometimes the best solution is to judiciously choose which data to denormalize, and where a second query to retrieve children is acceptable. Denormalization gives you arguably the most power and flexibility.Of course, this comes with the burden of administrative overhead. You get to manage relations, and you get to perform the required queries/filters to associate the various types. Yay!<h3>Conclusion and Recap</h3>This turned into a long, wordy article, so here is a bulletted recap:Inner Object<ul><li>Easy, fast, performant</li><li>Only applicable when one-to-one relationships are maintained</li><li>No need for special queries</li></ul>Nested<ul><li>Nested docs are stored in the same Lucene block as each other, which helps read/query performance. Reading a nested doc is faster than the equivalent parent/child.</li><li>Updating a single field in a nested document (parent or nested children) forces ES to reindex the entire nested document. This can be very expensive for large nested docs</li><li>"Cross referencing" nested documents is impossible</li><li>Best suited for data that does not change frequently</li></ul>Parent/Child<ul><li>Children are stored separately from the parent, but are routed to the same shard. So parent/children are slightly less performance on read/query than nested</li><li>Parent/child mappings have a bit extra memory overhead, since ES maintains a "join" list in memory</li><li>Updating a child doc does not affect the parent or any other children, which can potentially save a lot of indexing on large docs</li><li>Sorting/scoring can be difficult with Parent/Child since the Has Child/Has Parent operations can be opaque at times</li></ul>Denormalization<ul><li>You get to manage all the relations yourself!</li><li>Most flexible, most administrative overhead</li><li>May be more or less performant depending on your setup</li></ul>

Managing Relations Inside Elasticsearch

<html><head></head><body>WARNING: This article contains outdated information. We no longer recommend taking its advice.<section id="post_content">
<article>

	A popular use of Elasticsearch is to index and store log files, often through a <a href="http://logstash.net">Logstash</a> server. Logstash takes log files, parses them into fields and stores them in Elasticsearch for easy searching/sorting/faceting. A popular front-end for Logstash is <a href="http://kibana.org/">Kibana</a>, which lets you easily dig through your mountains of log data.


	Hmm&#x2026;doesn&apos;t Elasticsearch log a variety of events itself? Could you configure Elasticsearch to dump it&apos;s logs to Logstash&#x2026;which would then insert them back into Elasticsearch for searching? The answer is yes, and it is surprisingly easy.


	At first glance, it seems like the sort of thing that will tear a hole in the fabric of space and time. Or at the very least destroy your cluster.


	The fact is, this could be a downright bad idea. But hey, it&apos;s possible, simple and fun! Perhaps someone will find this useful and applicable for their setup.

<h2>Configuring Elasticsearch</h2>

	The first step is to configure Elasticsearch so that logs can be piped into Logstash. There are several ways to do this in Log4J, but the method chosen was to pipe the logs to Logstash over a socket.


	The advantage of this method is that you can have several ES nodes all sending data over a socket to a single Logstash server. If you chose something else, like monitoring files, the Logstash server would need access to each server to access the file.


	There are two modifications to the default `logging.yml` file that you must make(<a href="https://gist.github.com/polyfractal/4968387">the final logging.yml can be found in this gist</a>). First, tell ES that we are adding a new output logger by appending a &#x201C;socketappender&quot; label the `rootlogger` line:

<pre class="prettyprint lang-yaml">rootLogger: INFO, console, file, socketappender
</pre>

	Next, place this configuration code at the bottom of your file:

<pre class="prettyprint lang-yaml">socketappender:
 type: org.apache.log4j.net.SocketAppender
 port: 9500
 remoteHost: localhost
 layout:
 type: pattern
 conversionPattern: &quot;[%d{ISO8601}][%-5p][%-25c] %m%n&quot;
</pre>

	This config block says that the label &#x201C;socketappender&quot; is an `org.apache.log4j.net.SocketAppender` class, connects to `remoteHost` on `port` 9500. You can, of course, change the host and port to suit your fancy. The layout section is copied from prior logging examples in the file, and basically controls how the log output should be formatted.


	Save the file and restart Elasticsearch, and you&apos;re done with configuring ES. Simple!

<h2>Configuring Logstash</h2>

	I&apos;m going to assume that you have Logstash installed and know how to use it. What we need to do is tell Logstash to act as a Log4J server, so that it can accept socket connections from Elasticsearch. Create a new configuration file and place this inside:

<pre class="prettyprint lang-default-markup">input {
 log4j {
 mode =&gt; &quot;server&quot;
 port =&gt; 9500
 type =&gt; eslogs
 }
}
output {
 stdout { debug =&gt; true debug_format =&gt; &quot;json&quot;}
 elasticsearch {
 type =&gt; eslogs
 cluster =&gt; &quot;elasticsearch&quot;
 }
}
</pre>

	This config file makes Logstash act as a Log4J server on port 9500. Any log events accepted over the Log4J server will be set as the &#x201C;eslogs&quot; type. The output section is fairly typical of an ES-logstash setup &#x2013; adjust as necessary to connect to your ES cluster. Make sure the &#x201C;type&quot; is set to whatever was set in the input. Fire up Logstash with this configuration and you are all done!

<h2>Conclusion</h2>

	If everything is setup correctly, ES will now be piping log data into Logstash, which will then pipe it back into your ES cluster. If you have Kibana installed, you can easily start searching your ES log data.



	The usefulness of this entire setup is a bit suspicious. On one hand, it gives you the power and flexibility of ES for digging into your ES logs, probably after a crisis has been resolved and you are looking for the root cause. It centralizes and collects all the logs of your cluster into one location for debugging, instead of hunting through a dozen individual log files and trying to piece together the sequence of events.


	On the other hand, if nodes are dropping in and out of your cluster due to problems, their log data is by default not going to be available&#x2026;so this setup would not be overly useful during the actual crisis itself. It also adds more overhead to your cluster in terms of indexing, which could be the last thing you want if your cluster is melting down. It could also generate a significant amount of traffic depending on how you have your logging set (e.g. you probably don&apos;t want Debug set on all your nodes).


	Regardless, it&apos;s a fun little experiment that makes you appreciate the power and flexibility of all the tools described above.

<aside id="post_tags">
</aside>
</article>
</section>
</body></html>

Logging Elasticsearch Events with Logstash (and Elasticsearch)

In our <a href="/2013/02/04/starts-with-phrase-matching/" title="Starts-with Phrase Matching">last article on starts-with phrase matching</a>, we ran into a situation where the returned scores were suspicious. As a refresher, here is the query in question:<pre class="prettyprint lang-bash">$ curl -XGET localhost:9200/startswith/test/_search?pretty -d '{
 "query": {
 "match_phrase_prefix": {
 "title": {
 "query": "d",
 "max_expansions": 5
 }
 }
 }
 }' | grep title

 "_score" : 1.0, "_source" : {"title":"drunk"}
 "_score" : 0.30685282, "_source" : {"title":"dzone"}
 "_score" : 0.30685282, "_source" : {"title":"data"}
 "_score" : 0.30685282, "_source" : {"title":"drive"}
</pre>See how the document “drunk” receives a score of 1.0, while the rest have a score of 0.3? Shouldn’t these docs all have the same score, since they match the query for “d” equally the same? The answer is yes, but there is a very good reason for this scoring discrepancy.<h2>Relevancy Scoring</h2>Part of the scoring algorithm used by Elasticsearch (and Lucene underneath) includes “<a href="http://en.wikipedia.org/wiki/Term-frequency">Term Frequency – Inverse Document Frequency</a>” statistics to help calculate relevancy of documents in the index.A lot has been written on the subject of TF-IDF, but it basically says “the more a term appears in a document, the more relevant this document is. But the relevancy is dampened by how often the term appears in the entire index”. Rare terms are only present in a few documents, which means any query matching a rare term becomes highly relevant. Conversely, common terms are found everywhere, so their relevancy to the query is low.Elasticsearch faces an interesting dilemma when you execute a search. Your query needs to find all the relevant documents…but these documents are scattered around any number of shards in your cluster. Each shard is basically a Lucene index, which maintains its own TF and DF statistics. A shard only knows how many times “pineapple” appears within the shard, not the entire cluster.But the relevancy algorithm uses TF-IDF…doesn’t it need to know how the TF and DF for the entire index, not for each shard?<h2>Default search type: Query Then Fetch</h2>The answer is yes and no. By default, Elasticsearch will use a search type called “<a href="/guide/reference/api/search/search-type.html">Query Then Fetch</a>“. The way it works is as follows:<ol>
<li>Send the query to each shard</li>
<li>Find all matching documents and calculate scores using local Term/Document Frequencies</li>
<li>Build a priority queue of results (sort, pagination with from/to, etc)</li>
<li>Return metadata about the results to requesting node. Note, the actual document is not sent yet, just the scores</li>
<li>Scores from all the shards are merged and sorted on the requesting node, docs are selected according to query criteria</li>
<li>Finally, the actual docs are retrieved from individual shards where they reside.</li>
<li>Results are returned to the client</li>
</ol>This system usually works fine. In most cases, your index has “enough” documents to smooth out the Term/Document frequency statistics. So while each shard may not have complete knowledge of frequencies across the cluster, results are “good enough” because the frequencies are fairly similar everywhere.But in the case of our query mentioned at the beginning of this article, the default search type sometimes fails. <h2>DFS Query Then Fetch</h2>In the last article, we built an index without specifying shard count – ElasticSearch used the default of 5 shards. We then inserted a measly five documents into the index and demanded ES return relevant results and accurate scores. Not quite fair, is it?The scoring discrepancies were caused by the Query Then Fetch search type. Each shard only contained 1 or 2 documents (the hashing algorithm used by ES ensures a relatively random distribution). When we asked Elastic to compute scores, each shard only had a tiny view of the five-doc index…so scores were inaccurate.Luckily, Elasticsearch doesn’t leave you out to dry. If you have a situation where this scoring discrepancy is problematic, ES provides a search type called “DFS Query Then Fetch”. The procedures is almost identical to Query then Fetch, except it performs a pre-query to calculate global document frequencies.<ol>
<li>Prequery each shard asking about Term and Document frequencies</li>
<li>Send the query to each shard</li>
<li>Find all matching documents and calculate scores using global Term/Document Frequencies calculated from the prequery.</li>
<li>Build a priority queue of results (sort, pagination with from/to, etc)</li>
<li>Return metadata about the results to requesting node. Note, the actual document is not sent yet, just the scores</li>
<li>Scores from all the shards are merged and sorted on the requesting node, docs are selected according to query criteria</li>
<li>Finally, the actual docs are retrieved from individual shards where they reside.</li>
<li>Results are returned to the client</li>
</ol>If we apply this new search type to our previous query, we get scoring results that make sense (e.g. they are all identical):<pre class="prettyprint lang-bash">$ curl -XGET 'localhost:9200/startswith/test/_search?pretty=true&search_type=dfs_query_then_fetch' -d '{
 "query": {
 "match_phrase_prefix": {
 "title": {
 "query": "d",
 "max_expansions": 5
 }
 }
 }
 }' | grep title

 "_score" : 1.9162908, "_source" : {"title":"dzone"}
 "_score" : 1.9162908, "_source" : {"title":"data"}
 "_score" : 1.9162908, "_source" : {"title":"drunk"}
 "_score" : 1.9162908, "_source" : {"title":"drive"}
</pre><h2>Conclusion</h2>Of course, better accuracy doesn’t come for free. The prequery causes an extra round-trip between the shards, which could cause a performance hit depending on size of the index, number of shards, query rate, etc etc. And in most cases, it is totally unnecessary…having “enough” data solves the problem for you.But sometimes you’ll run into strange scoring situations, and in those cases, it’s useful to know how to tweak the search execution plan with DFS Query then Fetch.

Understanding "Query Then Fetch" vs "DFS Query Then Fetch"

WARNING: This article contains outdated information. We no longer recommend taking its advice.“Starts-with” functionality is common in search applications. Either you want to return results as the user types (ala Google Instant) or you simply want to search partial phrases and return any matches. This can be accomplished in Elasticsearch several ways.In this article, we are going to explore phrase matching at query time, instead of building the functionality directly into the index using shingles or nGrams. In particular, we are going to focus on `<a href="/guide/en/elasticsearch/reference/current/query-dsl-match-query.html#_match_phrase_prefix">match_phrase_prefix</a>` query to do the heavy lifting.This query take the normal `match` query and adds phrase support + fuzzy prefix capability. The phrase matching comes from the ability to look at token offsets, allowing the query to know when tokens follow each other in a phrase. The prefix capability will take the last portion of your query and expand it into new query tokens.For example, if your query is “dog f” then match_phrase_prefix will expand this into new queries:<ul><li>dog fight</li><li>dog food</li><li>dog freight</li><li>dog farce</li><li>[...]</li></ul>Handy, right?<h2>Getting Started</h2>Let’s go ahead and insert some data into a newly created index. <a href="https://gist.github.com/4704772">All the commands necessary to follow along are located in a Gist</a>. Our data looks like this:<pre class="prettyprint lang-bash">$ curl -XGET localhost:9200/startswith/test/_search?pretty -d '{ "query": { "match_all" : {} } }' | grep title {"title":"data"} {"title":"drive"} {"title":"drunk"} {"title":"river dog"} {"title":"dzone"} </pre>Our goal is to do a do a phrase match on the single character “d”. Ideally, we would retrieve all the titles that start with “d”, but not return the [river dog] entry because it starts with an “r”.Let’s go ahead and try the match_phrase_prefix query and see what happens:<pre class="prettyprint lang-bash">$ curl -XGET localhost:9200/startswith/test/_search?pretty -d '{ "query": { "match_phrase_prefix": { "title": { "query": "d", "max_expansions": 5 } } } }' | grep title "_score" : 0.70710677, "_source" : {"title":"data"} "_score" : 0.70710677, "_source" : {"title":"drive"} "_score" : 0.70710677, "_source" : {"title":"dzone"} "_score" : 0.44194174, "_source" : {"title":"river dog"} "_score" : 0.30685282, "_source" : {"title":"drunk"} </pre>Hmm…well that didn’t work. Why not? Because we didn’t specify a mapping for this new index, Elasticsearch simply used the default Standard Analyzer. And if you use Standard on [river dog], you’ll end up with this<pre class="prettyprint lang-bash">curl -XGET 'localhost:9200/startswith/_analyze?analyzer=standard&amp;pretty' -d 'river dog' | grep token "token" : "river", "token" : "dog", </pre>And that’s where the problem comes from. Standard tokenizes on whitespace, so [river dog] becomes [river] and [dog]. The match_phrase_prefix query doesn’t match [river], but does match [dog]. You can see this reflected in the score of “river dog”, about half as much as all the other entries because only one token matched.The solution, as is so often the case, comes in the form of a mapping. We want Elasticsearch to look at the entire field as a single token, not parse it into individual tokens. We could simply set the field to not_analyzed, but that will make it case sensitive.Instead, we are going to use the &lt;a href="/guide/en/elasticsearch/reference/current/analysis-keyword-tokenizer.html"&gt;keyword tokenizer&lt;/a&gt; plus the &lt;a href="/guide/en/elasticsearch/reference/current/analysis-lowercase-tokenfilter.html"&gt;lowercase filter&lt;/a&gt;.The keyword tokenizer is simple, and difficult to appreciate until you run into a situation like this one. It simply takes the entire field and emits it as a single token. It doesn’t really tokenize at all in fact, just returns what you give it.`Lowercase` filter is pretty obvious, I should think.Go ahead and delete the old index, recreate it with the following mapping and re-insert all the data.<pre class="prettyprint lang-bash">{ "settings":{ "index":{ "analysis":{ "analyzer":{ "analyzer_startswith":{ "tokenizer":"keyword", "filter":"lowercase" } } } } }, "mappings":{ "test":{ "properties":{ "title":{ "search_analyzer":"analyzer_startswith", "index_analyzer":"analyzer_startswith", "type":"string" } } } } } </pre>Now, when we execute the same match_phrase_prefix query from above, we get much better results:<pre class="prettyprint lang-bash">$ curl -XGET localhost:9200/startswith/test/_search?pretty -d '{ "query": { "match_phrase_prefix": { "title": { "query": "d", "max_expansions": 5 } } } }' | grep title "_score" : 1.0, "_source" : {"title":"drunk"} "_score" : 0.30685282, "_source" : {"title":"dzone"} "_score" : 0.30685282, "_source" : {"title":"data"} "_score" : 0.30685282, "_source" : {"title":"drive"} </pre>Huzzah! The incorrect [river dog] entry is no longer present in our search results. Let’s add another character to our query, making the search for “dr” instead of just “d”. Does it still work?<pre class="prettyprint lang-bash">curl -XGET localhost:9200/startswith/test/_search?pretty -d '{ "query": { "match_phrase_prefix": { "title": { "query": "dr", "max_expansions": 5 } } } }' | grep title "_score" : 1.0, "_source" : {"title":"drunk"} "_score" : 0.30685282, "_source" : {"title":"drive"} </pre>Yep! Works perfectly, only returning the two documents that start with “dr”.You may notice that one document (“drunk”) receives a score of 1 while the rest have scores of 0.306…there is a good reason for this, but it is outside the scope of this article. Suffice to say that it is an artifact of having a low document count in the index. You can ignore this peculiarity for now.<h2>Conclusion</h2>So, what have we learned? If you want to build a “starts with” functionality at query time, you still need to have appropriate mapping at index time. In this article, we looked at how a simple keyword tokenizer (plus lowercase filter) can ensure that the match_phrase_prefix query returns the right results, by keeping our fields from being broken into multiple terms.

Starts-With Phrase Matching

<html><head></head><body><section id="post_content">
		<article>
			In this article, I want to introduce Shingles. Shingles are effectively word-nGrams. Given a stream of tokens, the shingle filter will create new tokens by concatenating adjacent terms. 
For example, given the phrase &#x201C;Shingles is a viral disease&#x201D;, a shingle filter might produce:
<ul>
<li>Shingles is</li>
<li>is a</li>
<li>a viral</li>
<li>viral disease</li>
</ul>
The shingle filter allows you to adjust min_shingle_size and max_shingle_size, so you can create new shingle tokens of any size.
Do you see why these are awesome? Shingles effectively give you the ability to pre-bake phrase matching. By building phrases into the index, you can avoid creating phrases at query time and save some processing time/speed. 
The downside is that you have larger indices and potentially more memory usage if you try to facet/sort on the shingled field.
You can get similar functionality from nGrams, but the overhead is even higher since a single phrase can generate a large number of nGrams. In many cases, search relevance is perfectly fine using shingles over nGrams. Subjectively, I&#x2019;d argue that shingles give better results. 
The only feature you lose is the ability to tolerate spelling mistakes&#x2026;but there are better ways to handle that behavior (which I&#x2019;ll cover in a later article).
<h2>Using Shingles</h2>
Ok, let&#x2019;s dive into how shingles are actually used. First, <a href="https://gist.github.com/4542494">check out this gist</a> &#x2013; I&#x2019;ve written out a sample mapping that you can create in your Elasticsearch cluster.
The mapping is pretty simple. A single field named `title`, an analyzer and two filters. Let&#x2019;s work through this piece by piece.
<pre class="prettyprint lang-json">&quot;mappings&quot;:{
 &quot;product&quot;:{
 &quot;properties&quot;:{
 &quot;title&quot;:{
 &quot;search_analyzer&quot;:&quot;analyzer_shingle&quot;,
 &quot;index_analyzer&quot;:&quot;analyzer_shingle&quot;,
 &quot;type&quot;:&quot;string&quot;
 }
 }
 }
 }
</pre>
The mapping section is simple. We define a type (`product`) and a field (`title`). The field specifies the `analyzer_shingle` analyzer for both search and index.
The fun starts to happen in the analyzer:
<pre class="prettyprint lang-json">&quot;analyzer_shingle&quot;:{
 &quot;tokenizer&quot;:&quot;standard&quot;,
 &quot;filter&quot;:[&quot;standard&quot;, &quot;lowercase&quot;, &quot;filter_stop&quot;, &quot;filter_shingle&quot;]
}
</pre>
ES first tokenizes an input text, then filters. So this analyzer first applies the <a href="http://www.elasticsearch.org/guide/reference/index-modules/analysis/standard-tokenizer.html">standard tokenizer</a>, then walks through the <a href="http://www.elasticsearch.org/guide/reference/index-modules/analysis/standard-tokenfilter.html">standard filter</a>, <a href="http://www.elasticsearch.org/guide/reference/index-modules/analysis/lowercase-tokenfilter.html">lowercase filter</a> and our two custom filters: `filter_stop` and `filter_shingle`.
First, `filter_stop`:
<pre class="prettyprint lang-json">&quot;stop&quot;:{
 &quot;type&quot;:&quot;stop&quot;,
 &quot;enable_position_increments&quot;:&quot;false&quot;
}
</pre>
`Filter_stop` isn&#x2019;t actually a custom filter &#x2013; we just need to adjust a default setting of the `stop` filter. As the stop filter walks through a token stream and removes stopwords (such as &#x201C;I&#x201D;, &#x201C;to&#x201D;, &#x201C;are&#x201D;, etc), it records these locations in the token stream. 
In the case of shingles, this is bad. The shingle filter will see these &#x201C;skipped&#x201D; tokens and insert an underscore in it&#x2019;s place. Resulting tokens will look like this:
<pre class="prettyprint lang-bsh">$ curl -XGET &apos;localhost:9200/test/_analyze?analyzer=analyzer_shingle&amp;#038;pretty&apos; -d &apos;Test text to see shingles&apos; | grep &quot;token&quot;

&quot;tokens&quot; : [ {
 &quot;token&quot; : &quot;test&quot;,
 &quot;token&quot; : &quot;test text&quot;,
 &quot;token&quot; : &quot;test text _&quot;,
 &quot;token&quot; : &quot;test text _ see&quot;,
 &quot;token&quot; : &quot;test text _ see shingles&quot;,
 &quot;token&quot; : &quot;text&quot;,
 &quot;token&quot; : &quot;text _&quot;,
 &quot;token&quot; : &quot;text _ see&quot;,
 &quot;token&quot; : &quot;text _ see shingles&quot;,
 &quot;token&quot; : &quot;_ see&quot;,
 &quot;token&quot; : &quot;_ see shingles&quot;,
 &quot;token&quot; : &quot;see&quot;,
 &quot;token&quot; : &quot;see shingles&quot;,
 &quot;token&quot; : &quot;shingles&quot;,
</pre>
To get rid of this behavior, we set the `enable_position_increments` property equal to false. Our new token stream will look like this:
<pre class="prettyprint lang-bsh">$ curl -XGET &apos;localhost:9200/test/_analyze?analyzer=analyzer_shingle&amp;#038;pretty&apos; -d &apos;Test text to see shingles&apos; | grep &quot;token&quot;

&quot;tokens&quot; : [ {
 &quot;token&quot; : &quot;test&quot;,
 &quot;token&quot; : &quot;test text&quot;,
 &quot;token&quot; : &quot;test text see&quot;,
 &quot;token&quot; : &quot;test text see shingles&quot;,
 &quot;token&quot; : &quot;text&quot;,
 &quot;token&quot; : &quot;text see&quot;,
 &quot;token&quot; : &quot;text see shingles&quot;,
 &quot;token&quot; : &quot;see&quot;,
 &quot;token&quot; : &quot;see shingles&quot;,
 &quot;token&quot; : &quot;shingles&quot;,
</pre>
Much better! The next, and last, filter is our shingle filter:
<pre class="prettyprint lang-json">&quot;filter_shingle&quot;:{
 &quot;type&quot;:&quot;shingle&quot;,
 &quot;max_shingle_size&quot;:5,
 &quot;min_shingle_size&quot;:2,
 &quot;output_unigrams&quot;:&quot;true&quot;
}
</pre>
Fairly self-explanatory. Min_shingle_size and max_shingle_size control the length of the new shingle tokens. Min_shingle_size has to be at least 2, but you can enable the output of unigrams (aka single word tokens) with a separate setting.
I&apos;m a fan of shingles because they give you both exact-match and phrase matching. An exact match will hit all the shingled tokens and boost the score appropriately, while other queries can still hit parts of the phrase. And since shingles are stored as tokens in the index, their TF-ID is calculated and rarer phrase matches enjoy a bigger score boost than more common phrases.
I used a stopword filter in my analyzer to get rid of pesky stopwords, but there&apos;s no reason you need to. Some applications need stopwords. Similarly, I tokenized with Standard, but a Whitespace or Word-Delimited tokenizer may be more appropriate for you.
			<aside id="post_tags">
							</aside>
		</article>
			</section></body></html>

Searching with Shingles

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Software Engineer

Zachary Tong

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/elastic-cloud-account-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">ausgewählte Nachrichten und Informationen</a> an die oben von Ihnen bereitgestellten Kontaktangaben zu senden. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/serverless-preview-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Beim Absenden akzeptieren Sie die <a href="/de/agreements/global/cloud-services-monthly" rel="nofollow">Nutzungsbedingungen von Elastic Cloud</a>. Die Verarbeitung Ihrer personenbezogenen Daten unterliegt der <a href="/de/legal/privacy-statement">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/terms-of-use" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Der Inhalt dieser Seite ist in der ausgewählten Sprache nicht verfügbar. Wir bei Elastic arbeiten daran, die bereitgestellten Inhalte in verschiedenen Sprachen anzubieten. Bis dahin bitten wir Sie um etwas Geduld und hoffen auf Ihr Verständnis!

Author

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Articles by

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Sie können nicht dabei sein? Registrieren Sie sich, damit wir Ihnen die Aufzeichnung zukommen lassen können. Außerdem erhalten Sie eine E‑Mail mit Begleitmaterial.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Learn More

Read less

You'll also receive an email with related content

Wir schicken Ihnen zudem relevante Informationen, die von Interesse sein könnten.

Thank you for registering. We will send you a confirmation email soon.

Vielen Dank für Ihre Registrierung. Sie erhalten in Kürze eine Bestätigungs-E‑Mail von uns.

Author

Articles by Zachary Tong

Software Engineer, Elastic

This Week in Elasticsearch - 2020-02-21

A Profile a Day Keeps the Doctor Away: The Elasticsearch Search Profiler

Implementing a Statistical Anomaly Detector in Elasticsearch - Part 3

Implementing a statistical anomaly detector in Elasticsearch - Part 2

Implementing a Statistical Anomaly Detector in Elasticsearch - Part 1

Staying in Control with Moving Averages - Part 2

Staying in Control with Moving Averages - Part 1

Intro to Aggregations pt. 2: Sub-Aggregations

Intro to Aggregations