Web Server Suspicious User Agent Requests

edit

IMPORTANT: This documentation is no longer updated. Refer to Elastic's version policy and the latest documentation.

Web Server Suspicious User Agent Requests

edit

This rule detects unusual spikes in web server requests with uncommon or suspicious user-agent strings. Such activity may indicate reconnaissance attempts by attackers trying to identify vulnerabilities in web applications or servers. These user-agents are often associated with automated tools used for scanning, vulnerability assessment, or brute-force attacks.

Rule type: esql

Rule indices: None

Severity: low

Risk score: 21

Runs every: 10m

Searches indices from: now-9m (Date Math format, see also Additional look-back time)

Maximum alerts per execution: 100

References: None

Tags:

Domain: Web
Domain: Network
Use Case: Threat Detection
Tactic: Reconnaissance
Tactic: Credential Access
Data Source: Network Packet Capture
Data Source: Nginx
Data Source: Apache
Data Source: Apache Tomcat
Data Source: IIS
Resources: Investigation Guide

Version: 2

Rule authors:

Elastic

Rule license: Elastic License v2

Investigation guide

edit

## Triage and analysis

Disclaimer: This investigation guide was created using generative AI technology and has been reviewed to improve its accuracy and relevance. While every effort has been made to ensure its quality, we recommend validating the content and adapting it to suit your specific environment and operational needs.

Investigating Web Server Suspicious User Agent Requests

This rule flags surges of web requests that advertise scanner or brute-force tool user agents, signaling active reconnaissance against your web servers and applications. A common pattern is dirsearch or gobuster sweeping for hidden paths, firing hundreds of rapid GETs across diverse URLs from one host and probing admin panels, backup folders, and robots.txt.

Possible investigation steps

Verify whether the activity aligns with approved scanners or uptime checks by cross-referencing inventories, allowlists, change windows, and egress ranges; otherwise enrich the originating IP with ASN, geolocation, and threat reputation to gauge risk.
Sample representative requests to identify targeted paths and payloads (e.g., admin panels, .git/.env, backups, traversal, SQLi/XSS markers) and note any successful responses or downloads that indicate information exposure.
Analyze request rate, methods, and status-code distribution to separate noisy recon from successful discovery or brute-force patterns, highlighting any POST/PUT with nontrivial bodies.
Correlate the same client across hosts and security layers (application/auth logs, WAF/CDN, IDS) to determine whether it is scanning multiple services, triggering signatures, or attempting credential stuffing.
Assess user-agent authenticity and evasiveness by comparing HTTP header order/values and TLS fingerprints (JA3/JA4) to expected clients, and verify true client identity via forwarded-for headers if behind a proxy or CDN.

False positive analysis

Legitimate, scheduled vulnerability assessments by internal teams (e.g., Nessus, Nikto, or OpenVAS) can generate large volumes of requests with those user-agent strings across many paths.
Developer or QA testing using discovery/fuzzing or intercept-proxy tools (Dirsearch, Gobuster, Ffuf, Burp, or OWASP ZAP) may unintentionally target production hosts, producing a short-lived spike with diverse URLs.

Response and remediation

Immediately contain by blocking or rate-limiting the originating IPs at the WAF/CDN and edge firewall, and add temporary rules to drop or challenge requests that advertise tool user agents such as "nikto", "sqlmap", "dirsearch", "wpscan", "gobuster", or "burp".
If traffic is proxied (CDN/reverse proxy), identify the true client via forwarded headers and extend blocks at both layers, enabling bot management or JS challenges on swept paths like /admin, /.git, /.env, /backup, and common discovery endpoints.
Eradicate exposure by removing or restricting access to sensitive files and directories uncovered by the scans, rotating any credentials or API keys found, invalidating active sessions, and disabling public access to administrative panels until hardened.
Recover by verifying no unauthorized changes or data exfiltration occurred, tuning per-IP and per-path rate limits to prevent path-sweeps while preserving legitimate traffic, and reintroducing normal rules only after fixes are deployed and stability is confirmed.
Escalate to incident response if sensitive files are successfully downloaded (HTTP 200/206 on /.git, /.env, or backups), any login or account creation succeeds, multiple hosts or environments are targeted, or activity persists after blocking via UA spoofing or rapid IP rotation.
Harden long term by enforcing WAF signatures for known scanner UAs and path patterns, denying directory listing and direct access to /.git, /.env, /backup and similar artifacts, requiring MFA/VPN for /admin and management APIs, and deploying auto-ban controls like fail2ban or mod_security.

Rule query

edit

from logs-network_traffic.http-*, logs-network_traffic.tls-*, logs-nginx.access-*, logs-apache.access-*, logs-apache_tomcat.access-*, logs-iis.access-*

| eval Esql.user_agent_original_lower = to_lower(user_agent.original)

| where
    (url.original is not null or url.full is not null) and
  (
		Esql.user_agent_original_lower like "mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/74.0.3729.169 safari/537.36" or // Nikto
		Esql.user_agent_original_lower like "nikto*" or // Nikto
		Esql.user_agent_original_lower like "mozilla/4.0 (compatible; msie 8.0; windows nt 5.1; trident/4.0)" or // Nessus Vulnerability Scanner
        Esql.user_agent_original_lower like "*nessus*" or // Nessus Vulnerability Scanner
		Esql.user_agent_original_lower like "sqlmap/*" or // SQLMap
		Esql.user_agent_original_lower like "wpscan*" or // WPScan
		Esql.user_agent_original_lower like "feroxbuster/*" or // Feroxbuster
		Esql.user_agent_original_lower like "masscan*" or // Masscan & masscan-ng
		Esql.user_agent_original_lower like "fuzz*" or // Ffuf
		Esql.user_agent_original_lower like "mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/user_agent.original like~ 87.0.4280.88 safari/537.36" or // Dirsearch
		Esql.user_agent_original_lower like "mozilla/4.0 (compatible; msie 6.0; windows nt 5.1)" or // Dirb
		Esql.user_agent_original_lower like "dirbuster*" or // Dirbuster
		Esql.user_agent_original_lower like "gobuster/*" or // Gobuster
        Esql.user_agent_original_lower like "*dirsearch*" or // dirsearch
		Esql.user_agent_original_lower like "*nmap*" or // Nmap Scripting Engine
        Esql.user_agent_original_lower like "*hydra*" or // Hydra Brute Forcer
        Esql.user_agent_original_lower like "*w3af*" or // w3af Web Application Attack and Audit Framework
        Esql.user_agent_original_lower like "*arachni*" or // Arachni Web Application Security Scanner
        Esql.user_agent_original_lower like "*skipfish*" or // Skipfish Web Application Security Scanner
        Esql.user_agent_original_lower like "*openvas*" or // OpenVAS Vulnerability Scanner
        Esql.user_agent_original_lower like "*acunetix*" or // Acunetix Vulnerability Scanner
        Esql.user_agent_original_lower like "*zap*" or // OWASP ZAP
        Esql.user_agent_original_lower like "*burp*" // Burp Suite
  )

| eval Esql.url_text  = case(url.original is not null, url.original, url.full)
| eval Esql.url_lower = to_lower(Esql.url_text)

| keep
    @timestamp,
    event.dataset,
    user_agent.original,
    source.ip,
    agent.id,
    host.name,
	Esql.url_lower,
    Esql.user_agent_original_lower
| stats
    Esql.event_count = count(),
    Esql.url_path_count_distinct = count_distinct(Esql.url_lower),
    Esql.host_name_values = values(host.name),
    Esql.agent_id_values = values(agent.id),
    Esql.url_path_values = values(Esql.url_lower),
    Esql.user_agent_original_values = values(Esql.user_agent_original_lower),
    Esql.event_dataset_values = values(event.dataset)
    by source.ip, agent.id
| where
    Esql.event_count > 50 and Esql.url_path_count_distinct > 10

Framework: MITRE ATT&CK^TM

Tactic:
- Name: Reconnaissance
- ID: TA0043
- Reference URL: https://attack.mitre.org/tactics/TA0043/
Technique:
- Name: Active Scanning
- ID: T1595
- Reference URL: https://attack.mitre.org/techniques/T1595/
Sub-technique:
- Name: Scanning IP Blocks
- ID: T1595.001
- Reference URL: https://attack.mitre.org/techniques/T1595/001/
Sub-technique:
- Name: Vulnerability Scanning
- ID: T1595.002
- Reference URL: https://attack.mitre.org/techniques/T1595/002/
Sub-technique:
- Name: Wordlist Scanning
- ID: T1595.003
- Reference URL: https://attack.mitre.org/techniques/T1595/003/
Tactic:
- Name: Credential Access
- ID: TA0006
- Reference URL: https://attack.mitre.org/tactics/TA0006/
Technique:
- Name: Brute Force
- ID: T1110
- Reference URL: https://attack.mitre.org/techniques/T1110/

« Web Server Potential Spike in Error Response Codes Azure Compute Snapshot Deletion by Unusual User and Resource Group »