Elastic machine learning anomaly scoring has changed in 6.5. Learn how the new scoring relates to the normalization of partitions and multi-bucket anomalies.

banner-1183445_1920.jpg

banner-1183445_1280.jpg

Changes to Elastic Machine Learning Anomaly Scoring in 6.5

Learn the differences between these two types of analysis via a practical use case involving document access and potential information stealing.

outliers.jpg

temporal-distance-917364_640.jpg

Temporal vs. Population Analysis in Elastic Machine Learning

Optimize your results of your Machine Learning jobs by taking control of which data gets analyzed. Customize the datafeed with filters to get focused results.

pexels-photo-712786.jpeg

adult-blur-bokeh-712786.jpg

Filtering Input Data to Refine Machine Learning Jobs

Leverage the power of complex elasticsearch aggregation queries for your ML jobs. Follow this example of using a derivative aggregation to see how it works.

Recently I had a question that came up asking how to manage a derivative aggregation in a Machine Learning job. Custom aggregations are supported as part of the Machine Learning job's data feed config, and <a href="https://www.elastic.co/guide/en/x-pack/current/ml-configuring-aggregation.html">there is documentation</a> on how to do this. There are a few specific rules that should be followed so let's go through an example from start to finish so that you can see how it is done.
For this particular example, we'll use the following data set.&nbsp;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt814a8cedaadd92d3/5e346ace0aff5474f7624f90/blog-custom-elasticsearch-aggregations-01.png">
 
Under normal circumstances, just analyzing the data using ML's <code>low_sum</code> function is good enough to detect the anomaly of the "brown-out" on the last day of the data viewed above. But, for the sake of argument, let's imagine that what we're really interested is in the rate of change (the derivative) of the data set. In other words, the "recovery" coming out of the brown-out is a sharp, positive, rate of change that should have a high derivative at that moment. So, using the derivative aggregation in an Elasticsearch query, we can calculate the instantaneous rate of change and then use ML to see if the derivative is extreme at any point in time.&nbsp;
<h2>Defining The Job</h2>Let's create a job config that will do this for us. I'll use the <a href="https://www.elastic.co/guide/en/x-pack/current/ml-api-quickref.html">ML API</a> (invoked via the dev Console in Kibana) to do the creation:
<pre>PUT _xpack/ml/anomaly_detectors/orders_deriv
{
 "description": "Derivative of Order Volume",
 "analysis_config": {
 "bucket_span": "5m",
 "detectors": [
 {
 "detector_description": "sum(orders_deriv)",
 "function": "sum",
 "field_name": "orders_deriv"
 }
 ],
 "influencers": [],
 "summary_count_field_name": "doc_count"
 },
 "model_plot_config": {
 "enabled": "true"
 },
 "data_description": {
 "time_field": "@timestamp"
 }
}
</pre>Notice the usage of <code>summary_count_field_name</code>, this is required when using aggregated queries in Machine Learning. Also note that the <code>field_name</code> of "orders_deriv" in the detector doesn’t exist in the raw data, it will instead come from Elasticsearch's derivative aggregation in the datafeed’s query, outlined below.
<h3>Defining the Datafeed</h3>Let's create the datafeed which defines the&nbsp;source of the data to feed to the ML job:
<pre>PUT _xpack/ml/datafeeds/datafeed-orders_deriv/
{
 "job_id": "orders_deriv",
 "indices": [
 "it_ops_kpi-2017"
 ],
 "types": [
 "doc"
 ],
 "aggregations": {
 "buckets": {
 "date_histogram": {
 "field": "@timestamp",
 "interval": "5m",
 "time_zone": "UTC"
 },
 "aggregations": {
 "@timestamp": {
 "max": {
 "field": "@timestamp"
 }
 },
 "orders": {
 "sum": {
 "field": "events_per_min"
 }
 },
 "orders_deriv": {
 "derivative": {
 "buckets_path": "orders"
 }
 }
 }
 }
 }
}
</pre>
	As shown in the&nbsp;<a href="https://www.elastic.co/guide/en/x-pack/current/ml-configuring-aggregation.html">documentation</a>, there first must be a&nbsp;buckets aggregation, which in turn must contain a date histogram aggregation. This requirement ensures that the aggregated data is a time series. After that, however, we do a sum aggregation (to mimic what we were doing before with ML's <code>low_sum</code> function) and then a derivative aggregation on that summed value. When all is said and done, Elasticsearch is creating and returning a "new" field called "orders_deriv" to the ML job. 
One can test the datafeed's query and see if the outcome is as expected - just use the <code>_preview</code> endpoint:
<pre>GET _xpack/ml/datafeeds/datafeed-orders_deriv/_preview
</pre>The return may look something like:
<pre>[
 {
 "@timestamp": 1485643149000,
 "doc_count": 4
 },
 {
 "@timestamp": 1485643449000,
 "orders_deriv": 66,
 "doc_count": 5
 },
 {
 "@timestamp": 1485643749000,
 "orders_deriv": 20,
 "doc_count": 5
 },
 {
 "@timestamp": 1485644049000,
 "orders_deriv": -3,
 "doc_count": 5
 },
 {
 "@timestamp": 1485644349000,
 "orders_deriv": -42,
 "doc_count": 5
 },
...
</pre> 
<h3>The Outcome</h3>The result of running the job (either via the UI or the API) indeed shows the large, positive, rate of change at 11:10AM:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt12d926435b5ffefd/5e346ace7e155302ca0c8f9e/blog-custom-elasticsearch-aggregations-02.png"> 
 
We can also "overlay" this job with a straight-up analysis of the raw data and see them together and confirm that the high derivative is when the orders sharply recover after the brown-out:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0a4b697d1c40c2c2/5e346ace7ca70442647a6221/blog-custom-elasticsearch-aggregations-03.png"> 
However, notice in this case, the little thumbnail graph on the left (for the derivative analysis) does not show the plot of the derivative over time like the Single Metric View does. Why is that, you might ask?
Well, it is because in the Single Metric View,&nbsp; the chart drawn shows the user the "actual" values from the ML model if <code>model_plot_config</code> is enabled. If <code>model_plot_config</code> is not enabled, then the "actual" values will come from a query to ES for the raw data that is executed when the view is loaded. As such, if <code>model_plot_config</code> is not enabled, then the Single Metric Viewer will look the same as the thumbnail chart in the Anomaly Explorer:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt68d1a92c0e16a341/5e346acde147ae4537d946d1/blog-custom-elasticsearch-aggregations-04.png">
This is currently a limitation of the UI - it is unable to, on-the-fly, reverse engineer the complex Elasticsearch query that was used as part of the datafeed in order to query the raw data to draw the line. We hope to be able to support this in the future though.
You can find more about X-Pack and Machine Learning <a href="https://www.elastic.co/products/x-pack/">here</a>. If you are already using Machine Learning then make sure you check out our <a href="https://www.elastic.co/products/x-pack/machine-learning/recipes">prebuilt recipes</a> to help bootstrap your efforts.

custom-aggregations-elasticsearch-fullbleed.jpg

pea-1205673_640.jpg

Custom Elasticsearch Aggregations for Machine Learning Jobs

A complete breakdown of how machine learning in X-Pack scores anomalies and ranks them automatically on a severity scale from zero to one hundred.

anomaly_scoring_full_bleed.jpg

anomaly_scoring_thumb.jpg

Machine Learning Anomaly Scoring and Elasticsearch - How it Works

Connecting Machine Learning to X-Pack Alerting for notification of anomalies is now easy in version 5.5

Articles by Rich Collier

Changes to Elastic Machine Learning Anomaly Scoring in 6.5

Temporal vs. Population Analysis in Elastic Machine Learning

Filtering Input Data to Refine Machine Learning Jobs

Custom Elasticsearch Aggregations for Machine Learning Jobs

Machine Learning Anomaly Scoring and Elasticsearch - How it Works

Alerting on Machine Learning Jobs in Elasticsearch v5.5

Products & Solutions

Company

Resources

Featured topics

News

Articles by Rich Collier

Changes to Elastic Machine Learning Anomaly Scoring in 6.5

Temporal vs. Population Analysis in Elastic Machine Learning

Filtering Input Data to Refine Machine Learning Jobs

Custom Elasticsearch Aggregations for Machine Learning Jobs

Machine Learning Anomaly Scoring and Elasticsearch - How it Works

Alerting on Machine Learning Jobs in Elasticsearch v5.5