Anomaly Detection with Machine Learning
editAnomaly Detection with Machine Learning
editFor Free Trial
and Platinum License deployments,
Machine Learning functionality is available throughout the SIEM app. You can
view the details of detected anomalies within the Anomalies
table widget
shown on the Hosts, Network and associated Details pages, or even narrow to
the specific daterange of an anomaly from the Max Anomaly Score
details in
the overview of the Host and IP Details pages. Each of these interfaces also
offer the ability to drag and drop details of the anomaly to Timeline, such
as the Entity
itself, or any of the associated Influencers
.
Manage machine learning jobs
editFor users with the ml_admin
role, the Anomaly Detection
interface within
the main navigation header can be used for for viewing, starting, and stopping
SIEM machine learning jobs.
To add a custom job to the Anomaly Detection
interface, add a SIEM
tag to
the job’s Group
field (Kibana → Machine learning → Create/Edit job → Job details).
Prebuilt Jobs
editThe SIEM app ships with prebuilt Machine Learning Jobs for detecting anomalies.
If you ship data using Beats and your environment
is configured with the appropriate indices (auditbeat-*
, filebeat-*
, packetbeat-*
,
or winlogbeat-*
) via Kibana → Management → Index Patterns, the jobs will be installed
on page load, and will be displayed within the Anomaly Detection
interface.
Machine learning jobs look back and analyse two weeks of historical data prior to the time they are enabled. After jobs are enabled, they continuously analyse incoming data. When jobs are stopped and restarted within the two week timeframe, previously analysed data is not processed again.
The following prebuilt jobs are available:
rare_process_by_host_windows_ecs rare_process_by_host_linux_ecs |
Identifies rare processes that do not usually run on individual Windows/Linux hosts, which can indicate execution of unauthorized services, malware, or persistence mechanisms. Processes are considered rare when they only run occasionally as compared with other processes running on the host. Beats required on hosts:
|
windows_anomalous_network_activity_ecs linux_anomalous_network_activity_ecs |
Identifies Windows/Linux processes that do not usually use the network but have unexpected network activity, which could indicate command-and-control, lateral movement, persistence, or data exfiltration activity. A process with unusual network activity can denote process exploitation or injection, where the process is used to run persistence mechanisms that allow a malicious actor remote access or control of the host, data exfiltration, and execution of unauthorized network applications. Beats required on hosts:
|
windows_anomalous_path_activity_ecs |
Identifies processes started from atypical folders in the file system, which might indicate malware execution or persistence mechanisms. In corporate Windows environments, software installation is centrally managed and it is unusual for programs to be executed from user or temporary directories. Processes executed from these locations can denote that a user downloaded software directly from the internet or a malicious script/macro executed malware. Beats required on hosts:
|
windows_anomalous_process_all_hosts_ecs linux_anomalous_process_all_hosts_ecs |
Searches for rare processes running on multiple Windows/Linux hosts in an entire fleet or network. This reduces the detection of false positives since automated maintenance processes often only occasionally run on a single machine but are common to all or many hosts in a fleet. Beats required on hosts:
|
windows_anomalous_process_creation |
Identifies unusual parent/child process relationships that could indicate malware execution or persistence mechanisms. Malicious scripts often call on other applications and processes as part of their exploit payload. For example, when a malicious Office document runs scripts as part of an exploit payload, Excel or Word may start a script interpreter process, which, in turn, runs a script that downloads and executes malware. Another common scenario is Outlook running an unusual process when malware is downloaded in an email. Monitoring and identifying anomalous process relationships is an excellent way of detecting new and emerging malware that is not yet recognized by anti-virus scanners. Beats required on hosts:
|
windows_anomalous_script |
Searches for PowerShell scripts with unusual data characteristics, such as obfuscation, that may be a characteristic of malicious PowerShell script text blocks. Beats required on hosts:
|
windows_anomalous_service |
Searches for unusual Windows services that could indicate execution of unauthorized services, malware, or persistence mechanisms. In corporate Windows environments, hosts do not generally run many rare or unique services. This job helps detect malware and persistence mechanisms that have been installed and run as a service. Beats required on hosts:
|
windows_anomalous_user_name_ecs linux_anomalous_user_name_ecs |
Searches for activity from users who are not normally active, which could indicate unauthorized changes, activity by unauthorized users, lateral movement, and compromised credentials. In organizations, new usernames are not often created apart from specific types of system activities, such as creating new accounts for new employees. These user accounts quickly become active and routine. Events from rarely used usernames can point to suspicious activity. Additionally, automated Linux fleets tent to see activity from rarely used usernames only when personnel log in to make authorized or unauthorized changes, or threat actors have acquired credentials and log in for malicious purposes. Unusual usernames can also indicate pivoting, where compromised credentials are used to try and move laterally from one host to another. Beats required on hosts:
|
linux_anomalous_network_port_activity_ecs |
Identifies unusual destination port activity that could indicate command-and-control, persistence mechanism, or data exfiltration activity. Rarely used destination port activity is generally unusual in Linux fleets and can indicate unauthorized access or threat actor activity. Beats required on hosts:
|
linux_anomalous_network_service |
Searches for unusual listening ports that could indicate execution of unauthorized services, backdoors, or persistence mechanisms. Beats required on hosts:
|
linux_anomalous_network_url_activity_ecs |
Searches for unusual web URL requests from hosts, which could indicate malware delivery and execution. Wget and cURL are commonly used by Linux programs to download code and data. Most of the time, their usage is entirely normal. Generally, because they use a list of URLs, they repeatedly download from the same locations. However, Wget and cURL are sometimes used to deliver Linux exploit payloads, and threat actors use these tools to download additional software and code. For these reasons, unusual URLs can indicate unauthorized downloads or threat activity. Beats required on hosts:
|
suspicious_login_activity_ecs |
Identifies an unusually high number of authentication attempts. Beats required on hosts:
|
View detected anomalies
editTo view the Anomalies
table widget and Max Anomaly Score By Job
details,
the user must have the ml_admin
or ml_user
role.
To adjust the score
threshold for which anomalies
are shown, you can modify Kibana → Management → Advanced Settings → siem:defaultAnomalyScore
.