Create rule | SIEM Guide [7.8]

The SIEM app is now a part of the Elastic Security solution. Click here to view the current documentation.
IMPORTANT: No additional bug fixes or documentation updates will be released for this version.

› › ›

« Detections API Get rule »

Create ruleedit

Creates a new signal detection rule.

You can create two types of rules:

Query-based rules, which search the defined indices and creates a signal when a document matches the rule’s query.
Machine learning rules, which create a signal when a machine learning job discovers an anomaly above the defined threshold (see Anomaly Detection with Machine Learning).

To create machine learning rules, you must have the appropriate license or use a cloud deployment. Additionally, for the machine learning rule to function correctly, the associated machine learning job must be running.

To retrieve machine learning job IDs, which are required to create machine learning jobs, call the Elasticsearch Get jobs API. Machine learning jobs that contain siem in the groups field can be used to create rules:

...
"job_id": "linux_anomalous_network_activity_ecs",
"job_type": "anomaly_detector",
"job_version": "7.7.0",
"groups": [
  "auditbeat",
  "process",
  "siem"
],
...

Additionally, you can set up notifications for when rules create signals. The notifications use the Kibana Alerting and Actions framework. Each action type requires a connector. Connectors store the information required to send notifications via external systems. These action types are supported for rule notifications:

Slack
email
PagerDuty
Webhook

For more information on PagerDuty fields, see PagerDuty Send a v2 Event API.

To retrieve connector IDs, which are required to configure rule notifications, call GET <kibana host>:<port>/api/action/_find.

For detailed information on Kibana actions and alerting, and additional API calls, see:

Request URLedit

POST <kibana host>:<port>/api/detection_engine/rules

Request bodyedit

A JSON object with these fields:

Name	Type	Description	Required
actions	actions[]	Array defining the automated actions (notifications) taken when signals are produced.	No
anomaly_threshold	Integer	Anomaly score threshold above which the rule creates signals. Valid values are from `0` to `100`.	Yes, for machine learning rules. Not allowed in `query` rule types.
description	String	The rule’s description.	Yes
enabled	Boolean	Determines whether the rule is enabled.	No, defaults to `true`.
false_positives	String[]	String array used to describe common reasons why the rule may issue false-positive signals.	No, defaults to an empty array.
filters	Object[]	The query and filter context array used to define the conditions for when signals are created from events.	No, defaults to an empty array.
from	String	Time from which data is analyzed each time the rule executes, using a date math range. For example, `now-4200s` means the rule analyzes data from 70 minutes before its start time.	No, defaults to `now-6m` (analyzes data from 6 minutes before the start time).
rule_id	String	Unique ID used to identify rules. For example, when a rule is converted from a third-party security solution.	No, automatically created when it is not provided.
index	String[]	Indices on which the rule functions.	No. For query rules, defaults to the SIEM indices defined on the Kibana Advanced Settings page (Kibana → Management → Advanced Settings → `siem:defaultIndex`).
interval	String	Frequency of rule execution, using a date math range. For example, `"1h"` means the rule runs every hour.	No, defaults to `5m` (5 minutes).
machine_learning_job_id	String	Machine learning job ID the rule monitors for anomaly scores.	Yes, for machine learning rules. Not allowed in `query` rules types.
query	String	Query used by the rule to create a signal.	No. For `query` rules types, defaults to an empty string. Not allowed in `machine-learning` rule types.
language	String	Determines the query language, which must be `kuery` or `lucene`.	No. For `query` rule types, defaults to `kuery`. Not allowed in `machine-learning` rule types.
output_index	String	Index to which signals detected by the rule are saved.	No, if unspecified signals are saved to `.siem-signals-<space_name>` index, where `<space_name>` is the name of the Kibana space in which the rule exists.
saved_id	String	Kibana saved search used by the rule to create signals.	Yes, for `saved_query` rules only.
meta	Object	Placeholder for metadata about the rule.	No
risk_score	Integer	A numerical representation of the signal’s severity from 0 to 100, where: `0` - `21` represents low severity `22` - `47` represents medium severity `48` - `73` represents high severity `74` - `100` represents critical severity	Yes
max_signals	Integer	Maximum number of signals the rule can create during a single execution.	No, defaults to `100`.
name	String	The rule’s name.	Yes
note	String	Notes to help investigate signals produced by the rule.	No
severity	String	Severity level of signals produced by the rule, which must be one of the following: `low`: Signals that are of interest but generally not considered to be security incidents `medium`: Signals that require investigation `high`: Signals that require immediate investigation `critical`: Signals that indicate it is highly likely a security incident has occurred	Yes
tags	String[]	String array containing words and phrases to help categorize, filter, and search rules.	No, defaults to an empty array.
throttle	String	Determines how often actions are taken: `no_actions`: Never `rule`: Every time new signals are detected `1h`: Every hour `1d`: Every day `7d`: Every week	Yes, when actions are used to send notifications.
type	String	Data type on which the rule is based: `query`: query-based conditions with or without additional filters. `saved_query`: saved search, identified in the `saved_id` field. `machine_learning`: rule based on a machine learning job’s anomaly scores.	Yes
threat	threat[]	Object containing attack information about the type of threat the rule monitors, see ECS threat fields.	No, defaults to an empty array.
references	String[]	String array containing notes about or references to relevant information about the rule.	No, defaults to an empty array.
version	Integer	The rule’s version number.	No, defaults to `1`.

`actions` schemaedit

Name	Type	Description	Required
action_type_id	String	The action type used for sending notifications, can be: `.slack` `.email` `.pagerduty` `.webhook`	Yes
group	String	Optionally groups actions by use cases. Use `default` for signal notifications.	Yes
id	String	The connector ID.	Yes
params	Object	Object containing the allowed connector fields, which varies according to the connector type: For Slack: `message` (string, required): The notification message. For email: `to`, `cc`, `bcc` (string): Email addresses to which the notifications are sent. At least one field must have a value. `subject` (string, optional): Email subject line. `message` (string, required): Email body text. For Webhook: `body` (string, required): JSON payload. For PagerDuty: `severity` (string, required): Severity of on the signal notification, can be: `Critical`, `Error`, `Warning` or `Info`. `eventAction` (string, required): Event action type, which can be `trigger`, `resolve`, or `acknowledge`. `dedupKey` (string, optional): Groups signal notifications with the same PagerDuty alert. `timestamp` (DateTime, optional): ISO-8601 format timestamp. `component` (string, optional): Source machine component responsible for the event, for example `siem`. `group` (string, optional): Enables logical grouping of service components. `source` (string, optional): The affected system. Defaults to the Kibana saved object ID of the action. `summary` (string, options): Summary of the event. Defaults to `No summary provided`. Maximum length is 1024 characters. `class` (string, optional): Value indicating the class/type of the event.	Yes

All text fields (such as message fields) can contain placeholders for rule and signal details:

{{state.signals_count}}: Number of signals detected
{{{context.results_link}}}: URL to the signals in Kibana
{{context.rule.anomaly_threshold}}: Anomaly threshold score above which signals are generated (machine learning rules only)
{{context.rule.description}}: Rule description
{{context.rule.false_positives}}: Rule false positives
{{context.rule.filters}}: Rule filters (query-based rules only)
{{context.rule.id}}: Unique rule ID returned after creating the rule
{{context.rule.index}}: Indices rule runs on (query-based rules only)
{{context.rule.language}}: Rule query language (query-based rules only)
{{context.rule.machine_learning_job_id}}: ID of associated machine learning job (machine learning rules only)
{{context.rule.max_signals}}: Maximum allowed number of signals per rule execution
{{context.rule.name}}: Rule name
{{context.rule.output_index}}: Index to which signals are written
{{context.rule.query}}: Rule query (query-based rules only)
{{context.rule.references}}: Rule references
{{context.rule.risk_score}}: Rule risk score
{{context.rule.rule_id}}: Generated or user-defined rule ID that can be used as an identifier across systems
{{context.rule.saved_id}}: Saved search ID
{{context.rule.severity}}: Rule severity
{{context.rule.threat}}: Rule threat framework
{{context.rule.timeline_id}}: Associated timeline ID
{{context.rule.timeline_title}}: Associated timeline name
{{context.rule.type}}: Rule type
{{context.rule.version}}: Rule version

`threat` schemaedit

Only threats described using the MITRE ATT&CK^TM framework are displayed in the UI (SIEM → Detections → Manage signal detection rules → <rule name>).

Name	Type	Description	Required
framework	String	Relevant attack framework.	Yes
tactic	Object	Object containing information on the attack type: `id` - string, required `name` - string, required `reference` - string, required	Yes
technique	Object	Object containing information on the attack technique: `id` - string, required `name` - string, required `reference` - string, required	Yes

Example requestsedit

Example 1

Query-based rule that searches for processes started by MS Office:

POST api/detection_engine/rules
{
  "rule_id": "process_started_by_ms_office_program",
  "risk_score": 50,
  "description": "Process started by MS Office program - possible payload",
  "interval": "1h", 
  "name": "MS Office child process",
  "severity": "low",
  "tags": [
   "child process",
   "ms office"
   ],
  "type": "query",
  "from": "now-70m", 
  "query": "process.parent.name:EXCEL.EXE or process.parent.name:MSPUB.EXE or process.parent.name:OUTLOOK.EXE or process.parent.name:POWERPNT.EXE or process.parent.name:VISIO.EXE or process.parent.name:WINWORD.EXE",
  "language": "kuery",
  "filters": [
     {
      "query": {
         "match": {
            "event.action": {
               "query": "Process Create (rule: ProcessCreate)",
               "type": "phrase"
            }
         }
      }
     }
  ],
  "enabled": false
}

	The rule runs every hour.
	When the rule runs it analyzes data from 70 minutes before its start time.

If the rule starts to run at 15:00, it analyzes data from 13:50 until 15:00. When it runs next, at 16:00, it will analyze data from 14:50 until 16:00.

Example 2

Machine learning rule that creates signals, and sends Slack notifications, when the linux_anomalous_network_activity_ecs machine learning job discovers anomalies with a threshold of 70 or above:

POST api/detection_engine/rules
{
  "anomaly_threshold": 70,
  "rule_id": "ml_linux_network_high_threshold",
  "risk_score": 70,
  "machine_learning_job_id": "linux_anomalous_network_activity_ecs",
  "description": "Generates signals when the job discovers anomalies over 70",
  "interval": "5m",
  "name": "Anomalous Linux network activity",
  "note": "Shut down the internet.",
  "severity": "high",
  "tags": [
   "machine learning",
   "Linux"
   ],
  "type": "machine_learning",
  "from": "now-6m",
  "enabled": true,
  "throttle": "rule",
  "actions": [
    {
      "action_type_id": ".slack",
      "group": "default",
      "id": "5ad22cd5-5e6e-4c6c-a81a-54b626a4cec5",
      "params": {
        "message": "Urgent: {{context.rule.description}}"
      }
    }
  ]
}

Response codeedit

200: Indicates a successful call.

Response payloadedit

A JSON object that includes a unique ID, the time the rule was created, and its version number. If the request payload did not include a rule_id field, a unique rule ID is also generated.

Example response for a query-based rule:

{
  "created_at": "2020-04-07T14:51:09.755Z",
  "updated_at": "2020-04-07T14:51:09.970Z",
  "created_by": "LiverpoolFC",
  "description": "Process started by MS Office program - possible payload",
  "enabled": false,
  "false_positives": [],
  "from": "now-70m",
  "id": "6541b99a-dee9-4f6d-a86d-dbd1869d73b1",
  "immutable": false,
  "interval": "1h",
  "rule_id": "process_started_by_ms_office_program",
  "output_index": ".siem-signals-default",
  "max_signals": 100,
  "risk_score": 50,
  "name": "MS Office child process",
  "references": [],
  "severity": "low",
  "updated_by": "LiverpoolFC",
  "tags": [
    "child process",
    "ms office"
  ],
  "to": "now",
  "type": "query",
  "threat": [],
  "version": 1,
  "actions": [],
  "filters": [
    {
      "query": {
        "match": {
          "event.action": {
            "query": "Process Create (rule: ProcessCreate)",
            "type": "phrase"
          }
        }
      }
    }
  ],
  "throttle": "no_actions",
  "query": "process.parent.name:EXCEL.EXE or process.parent.name:MSPUB.EXE or process.parent.name:OUTLOOK.EXE or process.parent.name:POWERPNT.EXE or process.parent.name:VISIO.EXE or process.parent.name:WINWORD.EXE",
  "language": "kuery"
}

Example response for a machine learning job rule:

{
  "created_at": "2020-04-07T14:45:15.679Z",
  "updated_at": "2020-04-07T14:45:15.892Z",
  "created_by": "LiverpoolFC",
  "description": "Generates signals when the job discovers anomalies over 70",
  "enabled": true,
  "false_positives": [],
  "from": "now-6m",
  "id": "83876f66-3a57-4a99-bf37-416494c80f3b",
  "immutable": false,
  "interval": "5m",
  "rule_id": "ml_linux_network_high_threshold",
  "output_index": ".siem-signals-default",
  "max_signals": 100,
  "risk_score": 70,
  "name": "Anomalous Linux network activity",
  "references": [],
  "severity": "high",
  "updated_by": "LiverpoolFC",
  "tags": [
    "machine learning",
    "Linux"
  ],
  "to": "now",
  "type": "machine_learning",
  "threat": [],
  "version": 1,
  "actions": [
    {
      "action_type_id": ".slack",
      "group": "default",
      "id": "5ad22cd5-5e6e-4c6c-a81a-54b626a4cec5",
      "params": {
        "message": "Urgent: {{context.rule.description}}"
      }
    }
  ],
  "throttle": "rule",
  "note": "Shut down the internet.",
  "status": "going to run",
  "status_date": "2020-04-07T14:45:21.685Z",
  "anomaly_threshold": 70,
  "machine_learning_job_id": "linux_anomalous_network_activity_ecs"
}