Learn how to use custom rules to customize your Elastic machine learning jobs and capture domain expertise for you use case.

Elastic machine learning anomaly detection operates fully unsupervised. When a machine learning job is running, a dynamic, ever-learning model of the data is built and updated in real-time. This model enables our users to detect statistically anomalous events in a completely automated fashion. But there is one thing the model is not aware of: the data domain.
Whilst evaluating a statistical anomaly, questions might be raised. Is this a real point of interest? Is this actionable? The answers require expert knowledge about the specific use case and the domain of the analyzed data. To further understand this, let us run through a couple of examples that demonstrate the importance of domain knowledge. 
<h2>Numerical Conditions</h2>For the first use case, let us take the role of a DevOps engineer who is monitoring the machines running a service. We are interested in anomalies in CPU utilization. For this, we create a job using the Advanced Job Creation wizard. We create a detector looking at median CPU, partitioned by resource, using a 30 min bucket span interval. By using resource as the partitioning field, this will detect anomalies in CPU utilization for each resource independently:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2ba89d4bb3282039/5c57ded1122247d10be8ea9b/add_detector.png" data-sys-asset-uid="blt2ba89d4bb3282039" width="650" alt="add_detector.png" "="">
Now let us imagine that over a long period of time the power of our infrastructure surpasses the demand of our users, which means that many of our resources will be mostly idle. What will happen if there is any light momentary activity on any of these idle resources? The model will flag any small deviations as anomalous because it has learnt to expect CPU utilization to be close to zero. This is illustrated in the example below, where a cluster of anomalies has been detected for the resource ap_409, as the CPU utilization briefly rises to values between 1% and 8%.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt823d1827d70bf8d7/5fa95cfb42256d5ffdf4178b/blog-domain-knowledge-1.png" width="650" alt="">
Whilst the model is right in treating this behavior as statistically anomalous, these short periods of time when the CPU utilization rises to values that we consider relatively low are of no particular interest to us.
To solve this problem, starting with version 6.4.0 of the Elastic Stack, machine learning provides <a href="/guide/en/elastic-stack-overview/6.4/ml-rules.html">custom rules</a> functionality which enables us to change the behavior of anomaly detectors by providing domain-specific knowledge. When we create a rule, we specify conditions, scope, and actions. When the conditions of a rule are satisfied, its actions are triggered.
So for this first example, let’s create a rule to instruct our CPU utilization detector to refrain from generating machine learning results when there are anomalous events related to low CPU usage with a value below a threshold of 10%. Rules on a detector are configured through a new Configure rules item that has been added to the actions menu in the Anomalies table of the <a href="/guide/en/elastic-stack-overview/6.4/ml-gs-job2-analyze.html">Anomaly Explorer</a> and <a href="/guide/en/elastic-stack-overview/6.4/ml-gs-job1-analyze.html">Single Metric Viewer</a>:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd1a7b0cb0a7861db/5fa95cfb97f9f35d942e99fa/blog-domain-knowledge-2.png" width="650" alt="" "="">
Selecting this ‘Configure rules’ item will open a panel which we can use for creating a rule to tell the detector to stop generating results when the value of the anomaly is less than 10%:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt09220c9f8350e6e8/5fa95cf572a3526f28dba27e/blog-domain-knowledge-3.png" width="650" alt="" "="">
The first step in configuring a rule is to define the actions that should be triggered when the rule applies. We can choose to not create a result and to not update the model. In this case we will select just to skip the result, which is the recommended action, leaving the model to be updated as usual with the corresponding series value.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0e4540d4b13ccf1b/5fa95cf34a4abb73ff79c4fe/blog-domain-knowledge-4.png" width="450" "="">
The second step is to add the numeric condition, which defines when the rule applies. Conditions can be set against the actual or typical properties of the results, or against how much the actual value differs from typical. Here we apply the condition to the actual field of the result, setting the operator to ‘less than’, and entering a value of 10.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt160156fbef318e40/5fa95cfbffb61e50a0faabaf/blog-domain-knowledge-5.png" width="650">
We can now save the new rule, instructing our detector to stop creating anomaly results when the value of the CPU utilization is less than 10%.
For jobs that are currently running, changes to the rules on a detector will take effect immediately. However, to apply these changes to existing results you must clone and rerun the job from the <a href="/guide/en/elastic-stack-overview/6.4/ml-gs-job1-manage.html#ml-gs-job1-manage">Job Management</a> page. Note that rerunning a job may take some time and should only be done once you have completed all your changes to the rules for this job.
Let’s take a look at how this rule affects the results for our job. As shown in the screenshot below, we no longer see the anomalies for the ap_409 resource highlighted earlier, with results only created when the CPU utilization is above 10% as being worthy of further investigation.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1a62b2121a0a2219/5fa95cf424612d7400da3838/blog-domain-knowledge-6.png" width="650" alt="" "="">
<h2>Other Conditions</h2>In the example above we used a custom rule to skip results where the actual value was less than 10. There are other conditions available which can be applied to the typical value and the difference between the typical and actual values.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbd9745b6df886039/5c305206ff9c993a6eb5a446/condition_field.png" data-sys-asset-uid="bltbd9745b6df886039" alt="condition_field.png" style="width: 434px; height: 208px;" width="434" height="208">
<h2>JSON Configuration for Numeric Conditions</h2>Before adding a custom rule containing a numeric condition, the analysis configuration would have originally looked like this:
<pre class="prettyprint">{
 "analysis_config": {
 "bucket_span": "30m",
 "detectors": [
 {
 "detector_description": "CPU utilization",
 "function": "median",
 "field_name": "utilization",
 "partition_field_name": "resource"
 }
 ],
 "influencers": [
 "resource"
 ]
 },
 …
}
</pre>After adding a custom rule, it becomes:
<pre class="prettyprint">{
 "analysis_config": {
 "bucket_span": "30m",
 "detectors": [
 {
 "detector_description": "CPU utilization",
 "function": "median",
 "field_name": "utilization",
 "partition_field_name": "resource",
 "custom_rules": [
 {
 "actions": [ "skip_result" ],
 "conditions": [
 {
 "applies_to": "actual",
 "operator": "lt",
 "value": 10.0
 }
 ]
 }
 ]
 }
 ],
 "influencers": [
 "resource"
 ]
 },
 …
}
</pre><h2>Adding Scope</h2>For the second use case, we will take the role of a security analyst who tries to detect whether data is being exfiltrated over the DNS protocol. We are going to use a machine learning recipe for detecting DNS tunneling that is discussed in this <a href="/blog/using-machine-learning-and-elasticsearch-for-security-analytics-deep-dive">security analytics blog</a>. Using <a href="/products/beats/packetbeat">Packetbeat</a> to collect DNS data, we can split URLs between the subdomain and the highest registered domain.
Our job contains a detector using the <a href="/guide/en/elastic-stack-overview/6.4/ml-info-functions.html">high_info_content</a> function to monitor for unusually high amounts of information passed in the subdomain field compared to other highest_registered_domain values.
<pre class="prettyprint">{
 "analysis_config": {
 "detectors": [
 {
 "function": "high_info_content",
 "field_name": "subdomain",
 "over_field_name": "highest_registered_domain"
 }
 ]
 },
 ...
}
</pre>Let’s take a look at the results for this job on our data set.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9e87fea3a44c63f5/5fa95cf3c1502b76a169aa83/blog-domain-knowledge-7.png" width="650" alt="" "="">
The job has succeeded in identifying anomalies with regard to the amount of information sent over DNS requests among the various domains. However, many of the results are from widely used domains which we know do not relate to malicious activity. We may also see results from our own organization, such as from test infrastructure, which again we know we can trust. As with the first use case we looked at, these anomalies are of no interest to us given our knowledge of the domain.
To address this issue, let’s use another rule, this time one using a scope property to skip results when the highest registered domain is one we know we can trust. The scope of a rule makes use of machine learning <a href="/guide/en/elastic-stack-overview/6.4/ml-rules.html">filters</a>, which are lists of values that we can use to include or exclude events from the rule's application. You can configure filter lists on a new screen in the Machine Learning Settings page in Kibana which has been added in version 6.4 of the Elastic Stack. Shown below is a safe_domains filter list we’ve created containing a list of domains we consider to be safe.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt504061874ae396c1/5fa95cf509ed4c6f277eb4b0/blog-domain-knowledge-8.png" width="650" alt="" "="">
To create the rule, we open the rule editor by using the ‘Configure rules’ item from the anomalies table actions menu:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf7faf0b4d1a8fb61/5fa95cfb4e40cf53001f4f73/blog-domain-knowledge-9.png" width="650" alt="" "="">
Again we choose to skip results when the rule applies, but this time rather than using a numeric condition, we use the Scope section to add a scope for the rule, such that it applies only to certain highest registered domains. We pick the in&nbsp;option from the scope controls, and select the safe_domains&nbsp;filter list which we created in the Filter Lists settings page:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc5de80c12100be5c/5fa95cfc6f82405d9a4aa7ce/blog-domain-knowledge-10.png" width="650" alt="" "="">
Saving these edits creates a rule which will tell the detector to stop creating results when the highest_registered_domain&nbsp;is in our&nbsp;safe_domains&nbsp;filter list.
The rule will take effect for new results on saving, but let’s see how the results look when we rerun the job with this rule applied. Comparing the screen below with the original job shown earlier, we no longer see anomalies for any of the domains added to our&nbsp;safe_domains&nbsp;filter list:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd6c1095959dc7368/5fa95d0309ed4c6f277eb4b4/blog-domain-knowledge-11.png" width="650" alt="" "="">
So applying our knowledge of which domains we consider to be safe has removed the uninteresting results from our view, focussing our attention on potentially suspicious activity to domains which we haven’t classified as ‘safe’.
<h2>JSON Configuration for Filter Lists</h2><pre class="prettyprint">{
 "analysis_config": {
 "bucket_span": "15m",
 "detectors": [
 {
 "function": "high_info_content",
 "field_name": "subdomain",
 "over_field_name": "highest_registered_domain",
 "custom_rules": [
 {
 "actions": [ "skip_result" ],
 "scope": {
 "highest_registered_domain": {
 "filter_id": "safe_domains",
 "filter_type": "include"
 }
 }
 }
 ]
 }
 ]
 },
…
}
</pre><h2>Diving into the Different Rule Actions</h2>In the above examples we saw the usage of the <code>skip_result</code> action but another available choice is the <code>skip_model_update</code> action. A rule can use either of the actions or combine them both at the same time. In this section, we discuss in more detail what the available actions do and how they differ.
The <code>skip_result</code> action instructs the anomaly detector not to create an anomaly record when the rule applies. This means you will not see anomalies that you are not interested in. The remaining interesting anomalies will then be scored relative to each other giving you more meaningful ranking and severity categorization. (The scoring mechanism is part of the normalization process that is used by machine learning and you can learn more about it in this <a href="/blog/machine-learning-anomaly-scoring-elasticsearch-how-it-works">blog about anomaly scoring</a>).
The <code>skip_model_update</code> action instructs the anomaly detector to not use a particular value when the model is updated. In other words, the model remains unaffected by that value. This is useful when we expect certain values (e.g. from a test agent) that are highly unusual and we do not want the model to be altered by these, yet we do want anomalous results for them.
The default and recommended action is <code>skip_result</code>. This is the action to use for instructing the detector to ignore certain results without affecting the model itself. Interfering with the way the model updates may have a negative impact on the quality of the results, and should only be used in advanced use cases.
For most use cases we recommend selecting to skip results only as this ensures the modeled data reflects the true behavior of the system. For example, in the case of underutilized resources where the CPU would be low for the majority of time, selecting to not update the model would mean most data would not excluded from the model and therefore we would not have a true model of the low usage machine.
<h2>Rules vs. Datafeed Queries</h2>At first sight, it might appear that using rules is just another way of filtering the data that feeds into a job. For example, a rule that skips results when the partition field value is in a filter sounds equivalent to having a query that filters out such documents. But it is not. There is a fundamental difference. When the data is filtered before reaching a job it is as if they never existed for the job. With rules, the data still reaches the job and affects its behavior (depending on the rule actions).
For example, a rule with the <code>skip_result</code> action means all data will still be modeled. On the other hand, a rule with the <code>skip_model_update</code> action means results will still be created even though the model will not be updated by data matched by a rule.
If you wish to completely exclude data from an analysis, we would still recommend using a filter in the <a href="/guide/en/elasticsearch/reference/6.4/ml-put-datafeed.html#_request_body_6">datafeed query</a>. For example, if you wanted to analyze error messages and were not concerned with info and debug messages, then using a filter in the datafeed query would be preferable. This filters the data before reaching the machine learning analysis engine and is more efficient as it&nbsp;will use less resource.
It is important to understand the difference in relation to your use case and choose between simply using a datafeed query or using a custom rule with a suitable set of actions.
<h2>Summary</h2>Using custom rules allows you to apply your domain expertise to provide more meaningful scoring and ranking of anomalies that are important to you.
Custom rules are flexible and can be applied to numerical or categorical values. When applied to a job that is running in real-time, a custom rule will affect results from this point forwards. To apply them to historical data, a job can be cloned and re-run.
The <a href="/guide/en/elastic-stack-overview/6.4/ml-rules.html">custom rules</a> functionality has been added to machine learning in version 6.4 of the Elastic stack. Enjoy using it and let us know what you think of it on our <a href="https://discuss.elastic.co">Discuss forum</a>.

custom_job_banner.jpg

sound-console-thumbnail.jpg

How to Capture Domain Knowledge in Elastic Machine Learning Jobs with Custom Rules

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Dimitrios Athanasiou

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/elastic-cloud-account-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">ausgewählte Nachrichten und Informationen</a> an die oben von Ihnen bereitgestellten Kontaktangaben zu senden. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/serverless-preview-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Beim Absenden akzeptieren Sie die <a href="/de/agreements/global/cloud-services-monthly" rel="nofollow">Nutzungsbedingungen von Elastic Cloud</a>. Die Verarbeitung Ihrer personenbezogenen Daten unterliegt der <a href="/de/legal/privacy-statement">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/terms-of-use" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Der Inhalt dieser Seite ist in der ausgewählten Sprache nicht verfügbar. Wir bei Elastic arbeiten daran, die bereitgestellten Inhalte in verschiedenen Sprachen anzubieten. Bis dahin bitten wir Sie um etwas Geduld und hoffen auf Ihr Verständnis!

Author

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Articles by

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Sie können nicht dabei sein? Registrieren Sie sich, damit wir Ihnen die Aufzeichnung zukommen lassen können. Außerdem erhalten Sie eine E‑Mail mit Begleitmaterial.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Learn More

Read less

You'll also receive an email with related content

Wir schicken Ihnen zudem relevante Informationen, die von Interesse sein könnten.

Thank you for registering. We will send you a confirmation email soon.

Vielen Dank für Ihre Registrierung. Sie erhalten in Kürze eine Bestätigungs-E‑Mail von uns.

Author

Articles by Dimitrios Athanasiou

How to Capture Domain Knowledge in Elastic Machine Learning Jobs with Custom Rules

Sign up for Elastic Cloud free trial