Learn how to use custom rules to customize your Elastic machine learning jobs and capture domain expertise for you use case.

Elastic machine learning anomaly detection operates fully unsupervised. When a machine learning job is running, a dynamic, ever-learning model of the data is built and updated in real-time. This model enables our users to detect statistically anomalous events in a completely automated fashion. But there is one thing the model is not aware of: the data domain.
Whilst evaluating a statistical anomaly, questions might be raised. Is this a real point of interest? Is this actionable? The answers require expert knowledge about the specific use case and the domain of the analyzed data. To further understand this, let us run through a couple of examples that demonstrate the importance of domain knowledge. 
<h2>Numerical Conditions</h2>For the first use case, let us take the role of a DevOps engineer who is monitoring the machines running a service. We are interested in anomalies in CPU utilization. For this, we create a job using the Advanced Job Creation wizard. We create a detector looking at median CPU, partitioned by resource, using a 30 min bucket span interval. By using resource as the partitioning field, this will detect anomalies in CPU utilization for each resource independently:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2ba89d4bb3282039/5c57ded1122247d10be8ea9b/add_detector.png" data-sys-asset-uid="blt2ba89d4bb3282039" width="650" alt="add_detector.png" "="">
Now let us imagine that over a long period of time the power of our infrastructure surpasses the demand of our users, which means that many of our resources will be mostly idle. What will happen if there is any light momentary activity on any of these idle resources? The model will flag any small deviations as anomalous because it has learnt to expect CPU utilization to be close to zero. This is illustrated in the example below, where a cluster of anomalies has been detected for the resource ap_409, as the CPU utilization briefly rises to values between 1% and 8%.
<img src="https://lh5.googleusercontent.com/xXRXbW8BRFUQ88wwAYhycOLRkJX-jcF7sw-IbazmDCSLVIrhYkAG-uihwEA_rk-F-00jGfVZIE4Ivkcl69V3K1FsRguThFJuSFq7f9vlmBXpJgwRG5KXq_XhjgcvoYH0zCOWM-xC" width="650" alt="">
Whilst the model is right in treating this behavior as statistically anomalous, these short periods of time when the CPU utilization rises to values that we consider relatively low are of no particular interest to us.
To solve this problem, starting with version 6.4.0 of the Elastic Stack, machine learning provides <a href="/guide/en/elastic-stack-overview/6.4/ml-rules.html">custom rules</a> functionality which enables us to change the behavior of anomaly detectors by providing domain-specific knowledge. When we create a rule, we specify conditions, scope, and actions. When the conditions of a rule are satisfied, its actions are triggered.
So for this first example, let’s create a rule to instruct our CPU utilization detector to refrain from generating machine learning results when there are anomalous events related to low CPU usage with a value below a threshold of 10%. Rules on a detector are configured through a new Configure rules item that has been added to the actions menu in the Anomalies table of the <a href="/guide/en/elastic-stack-overview/6.4/ml-gs-job2-analyze.html">Anomaly Explorer</a> and <a href="/guide/en/elastic-stack-overview/6.4/ml-gs-job1-analyze.html">Single Metric Viewer</a>:
<img src="https://lh3.googleusercontent.com/g0CwFpegEzsHEgXvu3zEYXo6OU8-HmJ39_WQAO-D8CLCPCVJp-u1euV56T2gprG39DMsdWt5ZvKddJmDPPNpMOMLeECQ0kQOgPxjRE5sCd0JIf76JD6CtmCmch57MnlLCr23tcE8" width="650" alt="" "="">
Selecting this ‘Configure rules’ item will open a panel which we can use for creating a rule to tell the detector to stop generating results when the value of the anomaly is less than 10%:
<img src="https://lh5.googleusercontent.com/qTOKMTpARTHN9GkRdoKGRHtSkX-nLLpWKZTS8Q5XvUyXmxV5HCuFnTfj_JVXzGOr_m6QwQcovLXg1kIftDLqmWZSQcKgV7BJTYPZN3BDxONtSHGcPbjrdDSgFcfpiKcb_mLPLsBt" width="650" alt="" "="">
The first step in configuring a rule is to define the actions that should be triggered when the rule applies. We can choose to not create a result and to not update the model. In this case we will select just to skip the result, which is the recommended action, leaving the model to be updated as usual with the corresponding series value.
<img src="https://lh6.googleusercontent.com/Sm_c3YfAG5MKy_oPi9M59U6uAUDFhY1_oz-JJQ6SuQr1GKtIIWll6hvgCB3WYGSju0KMkZ9sLbhuES69J4ySIcTgwWtzxQuIASCjrQ0ClUwAfU3Xa1t2f3x5RhdJmKj9NIRLznkb" width="450" "="">
The second step is to add the numeric condition, which defines when the rule applies. Conditions can be set against the actual or typical properties of the results, or against how much the actual value differs from typical. Here we apply the condition to the actual field of the result, setting the operator to ‘less than’, and entering a value of 10.
<img src="https://lh5.googleusercontent.com/OjTp6X-OUErLr9tSuVVJqHugxS1SLyn2jkLTmwBlxpvFo8vK3fF3OrkamZTumT3vAp8IYaPCvefU8wGMKgD51NFqPmwOVGoqkp2YGBORxhUokuyxfovs8RMlu5HXYG4udvFGhWjI" width="650">
We can now save the new rule, instructing our detector to stop creating anomaly results when the value of the CPU utilization is less than 10%.
For jobs that are currently running, changes to the rules on a detector will take effect immediately. However, to apply these changes to existing results you must clone and rerun the job from the <a href="/guide/en/elastic-stack-overview/6.4/ml-gs-job1-manage.html#ml-gs-job1-manage">Job Management</a> page. Note that rerunning a job may take some time and should only be done once you have completed all your changes to the rules for this job.
Let’s take a look at how this rule affects the results for our job. As shown in the screenshot below, we no longer see the anomalies for the ap_409 resource highlighted earlier, with results only created when the CPU utilization is above 10% as being worthy of further investigation.
<img src="https://lh3.googleusercontent.com/by_yPqUyR1_hYgxi2885VmOvQpSObJ7WTty3_-Meku5IFlo4j9gJqZJ4WhfYURZpT3girH5jxC5QgQATPL0KV_p7VVwyCKc9HyQDTUxvH97VuDwhRVyjgY0HD4ZWMLA3QXh6REqk" width="650" alt="" "="">
<h2>Other Conditions</h2>In the example above we used a custom rule to skip results where the actual value was less than 10. There are other conditions available which can be applied to the typical value and the difference between the typical and actual values.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbd9745b6df886039/5c305206ff9c993a6eb5a446/condition_field.png" data-sys-asset-uid="bltbd9745b6df886039" alt="condition_field.png" style="width: 434px; height: 208px;" width="434" height="208"><h2>JSON Configuration for Numeric Conditions</h2>Before adding a custom rule containing a numeric condition, the analysis configuration would have originally looked like this:
<pre class="prettyprint">{
 "analysis_config": {
 "bucket_span": "30m",
 "detectors": [
 {
 "detector_description": "CPU utilization",
 "function": "median",
 "field_name": "utilization",
 "partition_field_name": "resource"
 }
 ],
 "influencers": [
 "resource"
 ]
 },
 …
}
</pre>After adding a custom rule, it becomes:
<pre class="prettyprint">{
 "analysis_config": {
 "bucket_span": "30m",
 "detectors": [
 {
 "detector_description": "CPU utilization",
 "function": "median",
 "field_name": "utilization",
 "partition_field_name": "resource",
 "custom_rules": [
 {
 "actions": [ "skip_result" ],
 "conditions": [
 {
 "applies_to": "actual",
 "operator": "lt",
 "value": 10.0
 }
 ]
 }
 ]
 }
 ],
 "influencers": [
 "resource"
 ]
 },
 …
}
</pre><h2>Adding Scope</h2>For the second use case, we will take the role of a security analyst who tries to detect whether data is being exfiltrated over the DNS protocol. We are going to use a machine learning recipe for detecting DNS tunneling that is discussed in this <a href="/blog/using-machine-learning-and-elasticsearch-for-security-analytics-deep-dive">security analytics blog</a>. Using <a href="/products/beats/packetbeat">Packetbeat</a> to collect DNS data, we can split URLs between the subdomain and the highest registered domain.
Our job contains a detector using the <a href="/guide/en/elastic-stack-overview/6.4/ml-info-functions.html">high_info_content</a> function to monitor for unusually high amounts of information passed in the subdomain field compared to other highest_registered_domain values.
<pre class="prettyprint">{
 "analysis_config": {
 "detectors": [
 {
 "function": "high_info_content",
 "field_name": "subdomain",
 "over_field_name": "highest_registered_domain"
 }
 ]
 },
 ...
}
</pre>Let’s take a look at the results for this job on our data set.
<img src="https://lh6.googleusercontent.com/QjJOIKUFqf3ETGb0tyvpDnNYndEjFSte9FX7NfQyvOIrYNyxsn7Ivx-uNVyPeVTrHpOwvf0DVE244BpugENNjhRE6hx6VeqjWPx5XYHVy18GhX6b_8k2ouOwr-wuh9B1zj_TVuLH" width="650" alt="" "="">
The job has succeeded in identifying anomalies with regard to the amount of information sent over DNS requests among the various domains. However, many of the results are from widely used domains which we know do not relate to malicious activity. We may also see results from our own organization, such as from test infrastructure, which again we know we can trust. As with the first use case we looked at, these anomalies are of no interest to us given our knowledge of the domain.
To address this issue, let’s use another rule, this time one using a scope property to skip results when the highest registered domain is one we know we can trust. The scope of a rule makes use of machine learning <a href="/guide/en/elastic-stack-overview/6.4/ml-rules.html">filters</a>, which are lists of values that we can use to include or exclude events from the rule's application. You can configure filter lists on a new screen in the Machine Learning Settings page in Kibana which has been added in version 6.4 of the Elastic Stack. Shown below is a safe_domains filter list we’ve created containing a list of domains we consider to be safe.
<img src="https://lh3.googleusercontent.com/p2FsDmCm4KpElm2V3Zn31mgZgdTAAHkE0N6RCO7s-AaWgIpNUiuiwk0cTks14qgA8OS9MpZHlv4RWYYfls1K0gL-IECQ4wHWhfcRLxv4JqDhppe2SBNdMGaDwFbzxVgcvHWv2Eyy" width="650" alt="" "="">
To create the rule, we open the rule editor by using the ‘Configure rules’ item from the anomalies table actions menu:
<img src="https://lh3.googleusercontent.com/1uyvREIsuTbxLHt2evLXnMqj2umjYZjJfaV6ALe3mBG7kp9MXTW3UXVBHrq8L9q0xXFDdrNc5GCp-2H7vir5wWfjq0OPJHIT_ofDT_HIxY4QjlXpRzDYuC_J2mkgUoAUbxgMgcxr" width="650" alt="" "="">
Again we choose to skip results when the rule applies, but this time rather than using a numeric condition, we use the Scope section to add a scope for the rule, such that it applies only to certain highest registered domains. We pick the in&nbsp;option from the scope controls, and select the safe_domains&nbsp;filter list which we created in the Filter Lists settings page:
<img src="https://lh4.googleusercontent.com/T64PopxR97glEDMYUTh1PoVfLbiBXJyvCdvOaw4lPnENm1Q0fdeR0EUxnKXqKfnNmLyLX8JZhE_PrcifpyPRAq58_arp8IJqDOszDsQrqHKiEo7YjDuGePvZmxD_jIApqpgSn_n1" width="650" alt="" "="">
Saving these edits creates a rule which will tell the detector to stop creating results when the highest_registered_domain&nbsp;is in our&nbsp;safe_domains&nbsp;filter list.
The rule will take effect for new results on saving, but let’s see how the results look when we rerun the job with this rule applied. Comparing the screen below with the original job shown earlier, we no longer see anomalies for any of the domains added to our&nbsp;safe_domains&nbsp;filter list:
<img src="https://lh3.googleusercontent.com/WKo64TA3_FMbAR6tGFuwKD_ABDg4tqjmzuvCsZuaMKrHXjX1HPSvKcU-i-e4ZkcBr0IjYgDec801sFkXKrZl-UIMfPrkn-8zb9OmxYPYjiem1x4rEC6nQs0oZoSLI0-iv0tDgPUh" width="650" alt="" "="">
So applying our knowledge of which domains we consider to be safe has removed the uninteresting results from our view, focussing our attention on potentially suspicious activity to domains which we haven’t classified as ‘safe’.
<h2>JSON Configuration for Filter Lists</h2><pre class="prettyprint">{
 "analysis_config": {
 "bucket_span": "15m",
 "detectors": [
 {
 "function": "high_info_content",
 "field_name": "subdomain",
 "over_field_name": "highest_registered_domain",
 "custom_rules": [
 {
 "actions": [ "skip_result" ],
 "scope": {
 "highest_registered_domain": {
 "filter_id": "safe_domains",
 "filter_type": "include"
 }
 }
 }
 ]
 }
 ]
 },
…
}
</pre><h2>Diving into the Different Rule Actions</h2>In the above examples we saw the usage of the <code>skip_result</code> action but another available choice is the <code>skip_model_update</code> action. A rule can use either of the actions or combine them both at the same time. In this section, we discuss in more detail what the available actions do and how they differ.
The <code>skip_result</code> action instructs the anomaly detector not to create an anomaly record when the rule applies. This means you will not see anomalies that you are not interested in. The remaining interesting anomalies will then be scored relative to each other giving you more meaningful ranking and severity categorization. (The scoring mechanism is part of the normalization process that is used by machine learning and you can learn more about it in this <a href="/blog/machine-learning-anomaly-scoring-elasticsearch-how-it-works">blog about anomaly scoring</a>).
The <code>skip_model_update</code> action instructs the anomaly detector to not use a particular value when the model is updated. In other words, the model remains unaffected by that value. This is useful when we expect certain values (e.g. from a test agent) that are highly unusual and we do not want the model to be altered by these, yet we do want anomalous results for them.
The default and recommended action is <code>skip_result</code>. This is the action to use for instructing the detector to ignore certain results without affecting the model itself. Interfering with the way the model updates may have a negative impact on the quality of the results, and should only be used in advanced use cases.
For most use cases we recommend selecting to skip results only as this ensures the modeled data reflects the true behavior of the system. For example, in the case of underutilized resources where the CPU would be low for the majority of time, selecting to not update the model would mean most data would not excluded from the model and therefore we would not have a true model of the low usage machine.
<h2>Rules vs. Datafeed Queries</h2>At first sight, it might appear that using rules is just another way of filtering the data that feeds into a job. For example, a rule that skips results when the partition field value is in a filter sounds equivalent to having a query that filters out such documents. But it is not. There is a fundamental difference. When the data is filtered before reaching a job it is as if they never existed for the job. With rules, the data still reaches the job and affects its behavior (depending on the rule actions).
For example, a rule with the <code>skip_result</code> action means all data will still be modeled. On the other hand, a rule with the <code>skip_model_update</code> action means results will still be created even though the model will not be updated by data matched by a rule.
If you wish to completely exclude data from an analysis, we would still recommend using a filter in the <a href="/guide/en/elasticsearch/reference/6.4/ml-put-datafeed.html#_request_body_6">datafeed query</a>. For example, if you wanted to analyze error messages and were not concerned with info and debug messages, then using a filter in the datafeed query would be preferable. This filters the data before reaching the machine learning analysis engine and is more efficient as it&nbsp;will use less resource.
It is important to understand the difference in relation to your use case and choose between simply using a datafeed query or using a custom rule with a suitable set of actions.
<h2>Summary</h2>Using custom rules allows you to apply your domain expertise to provide more meaningful scoring and ranking of anomalies that are important to you.
Custom rules are flexible and can be applied to numerical or categorical values. When applied to a job that is running in real-time, a custom rule will affect results from this point forwards. To apply them to historical data, a job can be cloned and re-run.
The <a href="/guide/en/elastic-stack-overview/6.4/ml-rules.html">custom rules</a> functionality has been added to machine learning in version 6.4 of the Elastic stack. Enjoy using it and let us know what you think of it on our <a href="https://discuss.elastic.co">Discuss forum</a>.

custom_job_banner.jpg

sound-console-thumbnail.jpg

How to Capture Domain Knowledge in Elastic Machine Learning Jobs with Custom Rules

Walk through new features for making it easier to create useful machine learning jobs in 6.1 with the new Data Visualizer and modules.

The release of Elasticsearch and X-Pack 6.1 introduced new features to make it easier for users to get started creating machine learning jobs. &nbsp;The features for data visualizer and machine learning modules are now available when selecting&nbsp;to create a new machine learning (ML)&nbsp;job.

<img src="https://assets.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbc41057108ca8067/5c307451c599e16966241230/ML_6.1_Create_Job_-_new_options" width="100%" data-sys-asset-uid="bltbc41057108ca8067">

The data visualizer will allow users to view their indexes and get an understanding of the metrics and fields that are in their data. It provides a contextual view into the indexes to show which fields are getting used, how many documents contain those fields, and the number of unique values in the field.

Here is a walk through of the data visualizer from one of its creators:

<div style="height: 478px; margin-bottom:20px;">
	<iframe width="100%" height="478" src="//www.youtube.com/embed/uIR1b1hRrLU?rel=0" class="video-iframe" frameborder="0" allowfullscreen="">
	</iframe>
</div>
The other new feature is ML modules. It&nbsp;provides a set of machine learning jobs, visualizations, and linked dashboards when the system identifies known field types. If indexes contain fields that are recognized by ML modules then there will be an additional button during the ML job creation step or within the data visualizer to automatically add a package for those fields that includes the ML jobs, visualizations, and linked dashboards.

Here is a walk through of the machine learning modules:

<div style="height: 478px; margin-bottom:20px;">
	<iframe width="100%" height="478" src="//www.youtube.com/embed/YOE1lvKtqC8?rel=0" class="video-iframe" frameborder="0" allowfullscreen="">
	</iframe>
</div>
We're excited to add these features to make it easier to create more effective machine learning jobs. If you have feedback or questions on data visualizer or ML modules we invite you to join the conversation in the <a href="https://discuss.elastic.co">Discuss forums</a>.

X-Pack machine learning capabilities come&nbsp;with an Elastic Platinum subscription, but you can try it out for free by downloading and installing <a href="https://www.elastic.co/downloads/x-pack">X-Pack</a>.

default-blog-banner.png

default-blog-thumb.png

Introducing data visualization and modules in machine learning

A complete breakdown of how machine learning in X-Pack scores anomalies and ranks them automatically on a severity scale from zero to one hundred.

Articles by Pete Harverson

How to Capture Domain Knowledge in Elastic Machine Learning Jobs with Custom Rules

Introducing data visualization and modules in machine learning

Machine Learning Anomaly Scoring and Elasticsearch - How it Works

Products & Solutions

Company

Resources

SaaS

ORCHESTRATION

Articles by Pete Harverson

How to Capture Domain Knowledge in Elastic Machine Learning Jobs with Custom Rules

Introducing data visualization and modules in machine learning

Machine Learning Anomaly Scoring and Elasticsearch - How it Works