Generating alerts for anomaly detection jobs

Elastic Stack Serverless

This guide explains how to create alerts that notify you automatically when an anomaly is detected in a anomaly detection job, or when issues occur that affect job performance.

Kibana's alerting features support two types of machine learning rules, which run scheduled checks on your anomaly detection jobs:

Anomaly detection alert

Checks job results for anomalies that match your defined conditions and raises an alert when found.

Anomaly detection jobs health

Monitors the operational status of a job and alerts you if issues occur (such as a stopped datafeed or memory limit errors).

If the conditions of a rule are met, an alert is created, and any associated actions (such as sending an email or Slack message) are triggered. For example, you can configure a rule that checks a job every 15 minutes for anomalies with a high score and sends a notification when one is found.

In Stack Management > Rules, you can create both types of machine learning rules. In the Machine Learning app, you can create only anomaly detection alert rules; create them from the anomaly detection job wizard after you start the job or from the anomaly detection job list.

Before you begin, make sure that:

• You have at least one running anomaly detection job.
• You have appropriate user permissions to create and manage alert rules.
• If you would like to send notifications about alerts (such as Slack messages, emails, or webhooks), make sure you have configured the necessary connectors.

Anomaly detection alert rules monitor if the anomaly detection job results contain anomalies that match the rule conditions.

To set up an anomaly detection alert rule:

1. Open Rules: find Stack Management > Rules in the main menu or use the global search field.
2. Select the Anomaly detection rule type.

×

3. Select the anomaly detection job that the rule applies to.
4. Select a type of machine learning result. You can create rules based on bucket, record, or influencer results.
5. (Optional) Configure the anomaly_score that triggers the action. The anomaly_score indicates the significance of a given anomaly compared to previous anomalies. The default severity threshold is 75 which means every anomaly with an anomaly_score of 75 or higher triggers the associated action.
6. Select whether you want to include interim results. Interim results are created before a bucket is finalized and might disappear after full processing.
   • Include interim results if you want to be notified earlier about a potential anomaly even if it might be a false positive.
   • Don't include interim results if you want to get notified only about anomalies of fully processed buckets.

×

7. (Optional) Configure Advanced settings:
   • Configure the Lookback interval to define how far back to query previous anomalies during each condition check. Its value is derived from the bucket span of the job and the query delay of the datafeed by default. It is not recommended to set the lookback interval lower than the default value, as it might result in missed anomalies.
   • Configure the Number of latest buckets to specify how many buckets to check to obtain the highest anomaly score found during the Lookback interval. The alert is created based on the highest scoring anomaly from the most anomalous bucket.

×

8. Set how often to check the rule conditions by selecting a time value and unit under Rule schedule.
9. (Optional) Configure Advanced options:
   • Define the number of consecutive matches required before an alert is triggered under Alert delay.
   • Enable or disable Flapping Detection to reduce noise from frequently changing alerts. You can customize the flapping detection settings if you need different thresholds for detecting flapping behavior.

×

Next, define the actions that occur when the rule conditions are met.

Anomaly detection jobs health rules monitor job health and alerts if an operational issue occurred that may prevent the job from detecting anomalies.

To set up an anomaly detection jobs alert rule:

1. Open Rules: find Stack Management > Rules in the main menu or use the global search field.
2. Select the Anomaly detection jobs rule type.

×

3. Include jobs and groups:

   • Select the job or group that the rule applies to. If you add more jobs to the selected group later, they are automatically included the next time the rule conditions are checked. To apply the rule to all your jobs, you can use a special character (*). This ensures that any jobs created after the rule is saved are automatically included.
   • (Optional) To exclude jobs that are not critically important, use the Exclude field.

4. Enable the health check types you want to apply. All checks are enabled by default. At least one check needs to be enabled to create the rule. The following health checks are available:

   • Datafeed is not started: Notifies if the corresponding datafeed of the job is not started but the job is in an opened state. The notification message recommends the necessary actions to solve the error.

   • Model memory limit reached: Notifies if the model memory status of the job reaches the soft or hard model memory limit. Optimize your job by following these guidelines or consider amending the model memory limit.

   • Data delay has occurred: Notifies when the job missed some data. You can define the threshold for the amount of missing documents you get alerted on by setting Number of documents. You can control the lookback interval for checking delayed data with Time interval. Refer to the Handling delayed data page to see what to do about delayed data.

   • Errors in job messages: Notifies when the job messages contain error messages. Review the notification; it contains the error messages, the corresponding job IDs and recommendations on how to fix the issue. This check looks for job errors that occur after the rule is created; it does not look at historic behavior.

×

5. Set how often to check the rule conditions by selecting a time value and unit under Rule schedule. It is recommended to select an interval that is close to the bucket span of the job.

6. (Optional) Configure Advanced options:

   • Define the number of consecutive matches required before an alert is triggered under Alert delay.
   • Enable or disable Flapping Detection to reduce noise from frequently changing alerts. You can customize the flapping detection settings if you need different thresholds for detecting flapping behavior.

×

Next, define the actions that occur when the rule conditions are met.

You can send notifications when the rule conditions are met and when they are no longer met. These rules support:

• Alert summaries: Combine multiple alerts into a single notification, sent at regular intervals.
• Per-alert actions for anomaly detection: Trigger an action when an anomaly score meets the defined condition.
• Per-alert actions for job health: Trigger an action when an issue is detected in a job’s health status (for example, a stopped datafeed or memory issue).
• Recovery actions: Notify when a previously triggered alert returns to a normal state.

To set up an action:

1. Select a connector.

2. Set the action frequency. Choose whether you want to send:

   • Summary of alerts: Groups multiple alerts into a single notification at each check interval or on a custom schedule.
   • A notification for each alert: Sends individual alerts as they are triggered, recovered, or change state.

Example: Summary of alerts

You can choose to create a summary of alerts on:

• Each check interval: Sends a summary every time the rule runs (for example, every 5 minutes).
• Custom interval: Sends a summary less often, on a schedule you define (for example, every hour), which helps reduce notification noise. A custom action interval cannot be shorter than the rule's check interval.

For example, send slack notifications that summarize the new, ongoing, and recovered alerts:

×

Example: For each alert

Choose how often the action runs:

• at each check interval,
• only when the alert status changes, or
• at a custom action interval.

For anomaly detection alert rules, you must also choose whether the action runs when the anomaly score matches the condition or when the alert recovers:

×

For anomaly detection jobs health rules, choose whether the action runs when the issue is detected or when it is recovered:

×

3. Specify that actions run only when they match a KQL query or occur within a specific time frame.

4. Use variables to customize the notification message. Click the icon above the message field to view available variables, or refer to action variables. For example:

×

After you save the configurations, the rule appears in the Stack Management > Rules list; you can check its status and see the overview of its configuration information.

When an alert occurs for an anomaly detection alert rule, it is always the same name as the job ID of the associated anomaly detection job that triggered it. You can review how the alerts that are occured correlate with the anomaly detection results in the Anomaly explorer by using the Anomaly timeline swimlane and the Alerts panel.

If necessary, you can snooze rules to prevent them from generating actions. For more details, refer to Snooze and disable rules.

The following variables are specific to the machine learning rule types. An asterisk (*) marks the variables that you can use in actions related to recovered alerts.

You can also specify variables common to all rules.

Every anomaly detection alert has the following action variables:

context.anomalyExplorerUrl^*

URL to open in the Anomaly Explorer.

context.isInterim

Indicates if top hits contain interim results.

context.jobIds^*

List of job IDs that triggered the alert.

context.message^*

A preconstructed message for the alert.

context.score

Anomaly score at the time of the notification action.

context.timestamp

The bucket timestamp of the anomaly.

context.timestampIso8601

The bucket timestamp of the anomaly in ISO8601 format.

context.topInfluencers

The list of top influencers. Limited to a maximum of 3 documents.

context.topRecords

The list of top records. Limited to a maximum of 3 documents.

Every health check has two main variables: context.message and context.results. The properties of context.results may vary based on the type of check. You can find the possible properties for all the checks below.

context.message^*

A preconstructed message for the alert.

context.results

Contains the following properties:

context.message^*

A preconstructed message for the rule.

context.results

Contains the following properties:

context.message^*

A preconstructed message for the rule.

context.results

For recovered alerts, context.results is either empty (when there is no delayed data) or the same as for an active alert (when the number of missing documents is less than the Number of documents threshold set by the user).
Contains the following properties:

context.message^*

A preconstructed message for the rule.

context.results

Contains the following properties: