Which Microsoft Sentinel rule types can be migrated automatically?

Automatic Migration focuses on the rules that carry your detection logic. In 9.5, it translates Scheduled and Near Real Time (NRT) analytics rules from Microsoft Sentinel, exported from your Sentinel workspace, and handles the translation for you.

It uses the same mapping and translation as our existing rule migrations, now extended to Microsoft Sentinel. The following are supported:

• Integration identification with just rule export
• Support for the following rule types:
• Near-real-time (NRT) detection analytics rules
• Scheduled Analytic Rules
• Support for Watchlists to ES|QL Lookups
• Severity Mapping

How to migrate Microsoft Sentinel detection rules to Elastic

The migration runs in a few steps, from exporting your rules in Sentinel to reviewing the translated versions in Elastic. Once you've decided which rules and data to migrate, follow these steps:

1. On the Security Launchpad, open Manage Automatic Migrations, select your AI provider, and expand Migrate your existing SIEM rules to Elastic.

1. Select the drop-down on the top right for Microsoft Sentinel. Let Elastic guide you through exporting your rules from Sentinel and uploading them into Elastic Security. Elastic handles the finer details by scanning for watchlists and then prompts you to upload them when found.

1. Once the rules are uploaded, you can view their status.

• Installed: Already added to Elastic SIEM. Click View to manage and enable it.
• Translated: Ready to install. This rule was mapped to an Elastic-authored rule, or translated by Automatic Import. Click Install to install it.
• Partially translated: Part of the query could not be translated. You may need to specify an index pattern for the rule query, upload missing files, or fix broken rule syntax.
• Not translated: None of the original query could be translated.
• Failed: Translation failed. Refer to the error for details.

For more information, refer to the technical documentation.

1. After clicking View Rules, you will have the ability to edit and install rules.

Should you migrate rules first or data first?

One of the first decisions in a migration is sequencing: data or rules first. Elastic supports both paths, so you can start wherever makes sense for your team.

| Path | When to use | What happens | | :---- | :---- | :---- | | Rules first | You do not know exactly which data sources to prioritise before moving any logs. | Translate your Sentinel rules first. Elastic identifies which integrations those rules need, so you can plan data onboarding around what your detections actually require. | | Data first | Your log sources are already being onboarded, or you want detections to work the moment they're installed. Onboarding data beforehand improves the translation quality. | Onboard your log sources into Elastic, then migrate your Sentinel rules to match. Rules can be installed and enabled immediately against data that's already flowing. | | Custom data | You have proprietary or non-standard log sources that don't map to a prebuilt Elastic integration. | Use Automatic Import to ingest custom data sources in minutes, then migrate or write rules against them. |

By identifying exactly which integrations are needed before moving a single log, teams can build a precise, risk-aware roadmap for their migration project. This transparency eliminates the guesswork and helps ensure that critical visibility gaps are addressed long before you fully decommission your environment.

What happens after your Sentinel rules are running in Elastic

Once your rules are running in Elastic, Workflows lets you build automation around them. The moment a rule fires, a workflow can kick off multi-step remediation, enrichment, and notification automatically. And building these automations in Agent Builder lowers the barrier, so you can create a workflow in natural language.

How Elastic AI fits into a Sentinel-to-Elastic migration

Elastic Security brings generative AI into the SOC with retrieval augmented generation (RAG) and open agentic frameworks. Automatic Migration joins the lineup of Elastic Security’s AI features, helping SOC teams strengthen defenses across the IT environment:

• Automatic Migration for Detection Rules complements Elastic’s deep library of prebuilt rules to broaden detection use case coverage.
• Automatic Import extends visibility and powers detection rules by onboarding custom data sources in minutes.
• Agent Skills assist in the response process and less time context switching.

Elastic’s SIEM and XDR solution helps analysts detect earlier and respond faster.

Try automatic detection rule migration

Migrating a SIEM has always meant rebuilding your detection rules by hand, and that cost is what keeps teams on a platform long after they've decided to leave. Automatic Migration simplifies that process, providing mapping to existing Elastic rules and helping to translate the rest. Your watchlists and severity levels carry over as well, and you move on your own terms, with your data and your tooling under your control. For further details check out our documentation.

Try it free, or get in touch. Have feedback? Tell us what you think in the Elastic Community Slack channel or on the Elastic Security forum.

How threat intelligence works in Elastic Security

In modern security operations, threat intelligence must work across detection, investigation, and response, not sit in a reference table.

Elastic Security supports this in two ways. Ingested intelligence via integrations drives continuous detection and historical hunting. Agentic workflows, built on Elastic Workflows and Agent Builder, provide on-demand enrichment and investigative reasoning during an active investigation. This post focuses on how Elastic's Google Threat Intelligence (GTI) integration powers ingestion-based detection and hunting, and how it fits into a broader, more dynamic SOC model where AI-driven workflows use that intelligence at alert time.

What Google Threat Intelligence provides

The Google Threat Intelligence integration brings curated threat intelligence directly into Elastic Security, making it actionable across detection and investigation. GTI combines intelligence from Google's global security visibility with VirusTotal data to deliver enriched context on indicators of compromise, with coverage across malware, ransomware, phishing, infostealers, malicious infrastructure, threat actors, and other adversary activity.

Each indicator is returned with: a verdict (Malicious, Suspicious, or Undetected), a severity, and a composite threat score from 0–100. Because that score is derived from multiple signals, security teams can prioritize indicators based on confidence rather than their presence alone.

How the Google Threat Intelligence integration works in Elastic Security

Setup takes only a few minutes. You provide your GTI API key in the Elastic integration, and ingestion begins on a scheduled polling interval, with no additional infrastructure or collectors required. The integration ingests two primary data streams.

| Purpose | Threat List | IOC Stream | |--------|-------------|------------| | Purpose | High-confidence detection | Threat hunting + early visibility | | Volume | Curated, lower volume | Broader, higher volume | | Best for | Precision-critical alerting | Emerging and exploratory activity |

As data is ingested, indicators are standardized using the Elastic Common Schema (ECS), along with GTI context, such as verdict, severity, score, malware families, threat actor associations, and campaign metadata (where available). This enables GTI to be searched and correlated consistently alongside other ECS-compliant intelligence sources (including TAXII feeds), custom intelligence, and the broader security telemetry already present in Elastic Security. Elastic also manages indicator lifecycle automatically, including expiration and revocation, which reduces matches against stale intelligence. Once ingested, GTI indicators become part of the same searchable dataset as logs, endpoint, and cloud telemetry, enabling unified correlation across the environment.

Using Google Threat Intelligence for indicator match detections

Elastic's indicator match rules use GTI data to detect when known malicious IPs, domains, URLs, or file hashes appear in security telemetry, continuously correlating intelligence against observed activity and surfacing matches for investigation. Because GTI provides structured fields such as score, verdict, and severity, teams can tune detections by confidence: high-confidence indicators can trigger immediate escalation, while lower-confidence indicators can be routed for review or further validation.

Threat hunting with GTI indicators in Elastic Security

With GTI metadata, analysts can pivot from a single IOC to all associated infrastructure and search historical telemetry; not just check if an indicator appeared, but understand what campaign it belongs to.

GTI enriches indicators with metadata such as threat actor associations and malware family context, allowing analysts to move beyond single-IOC searches. Hunters can pivot from an adversary or campaign to all associated indicators (IPs, domains, and file hashes) and search across historical telemetry using ES|QL. This makes it straightforward to determine whether known malicious infrastructure has ever interacted with the environment.

Monitoring threat intelligence activity with GTI dashboards

The integration includes prebuilt dashboards that provide visibility into threat intelligence activity and the detections GTI drives. Using saved searches and aggregated metrics, these dashboards summarize observed threats across malware families, campaigns, threat actors, toolkits, and vulnerabilities, helping SOC teams understand which threat types are most active in their environment and how intelligence is being operationalized.

Google Threat Intelligence feed categories and coverage

GTI includes 14 categorized feed categories, so organizations can tailor coverage to their needs and subscription level. Supported categories include:

• Cryptominers
• Trending threats
• Initial access and delivery vectors
• Infostealers
• IoT threats
• Linux malware
• Malicious infrastructure
• General malware
• Mobile threats
• macOS threats
• Phishing
• Ransomware
• Threat actors
• Vulnerability exploitation and weaponization

Availability depends on your Google Threat Intelligence subscription tier, and additional feeds can be enabled without changes to the Elastic configuration.

Agentic enrichment and real-time triage with Elastic Workflows

For ambiguous or emerging indicators not yet in an indexed feed, Elastic Security supports AI-driven investigation through Agent Builder and Elastic Workflows, which complement intelligence ingestion by enabling real-time enrichment and reasoning during an investigation.

With workflows, an analyst is no longer limited to the intelligence already in the index. During alert triage, a workflow can query external intelligence and reputation services such as VirusTotal in real time, enrich an alert with fresh context about the IPs, domains, or file hashes involved, correlate that live intelligence against Elastic telemetry, and summarize the findings into a structured investigation context that the analyst can act on. Agent Builder extends this further: teams can compose reusable, task-specific capabilities, such as agent skills for alert triage, enrichment, or case handling, so the assistant executes multi-step investigative tasks with the consistency of traditional automation, through a natural-language interface.


This introduces a complementary model. Ingested intelligence (GTI, TAXII, and custom feeds) provides continuous detection and historical hunting against indicators you already hold. Agentic workflows provide on-demand enrichment and investigative reasoning at alert time, reaching out to live sources and assembling context on the fly. Together, they enable teams to detect known threats at scale and provide context to investigations.

Getting started with Google Threat Intelligence in Elastic Security

To use the Google Threat Intelligence integration in Elastic Security, you need an active GTI license and API key.

1. Install: open Integrations catalog in Kibana → search "Google Threat Intelligence" → add integration → enter your API key
2. Configure the data streams: enable Threat List (high-confidence detections) and IOC Stream (hunting coverage) → set polling frequency to match API limits and operational needs
3. Tune: prebuilt indicator match rules activate automatically; if alert volume is high, start by filtering on confidence threshold

All indicators are stored in Elasticsearch and accessible through the GTI threat intelligence data view, enabling search, correlation, and custom detection logic. Full configuration details and troubleshooting guidance are available in the official documentation.

Tying it all together

Threat intelligence only matters if a team can act on it. By bringing Google Threat Intelligence into Elastic Security, SOC teams get ingestion-based detection running continuously across their telemetry and agent-driven investigation reasoning over that intelligence in real time. The combination lets threat intelligence operate continuously and contextually, helping analysts move from indicators to confident decisions faster.

This post walks through how Elastic's InfoSec team built a monitoring pipeline for Claude Code and Claude Cowork using their native OpenTelemetry (OTel) export capabilities and Elastic's own OTel ingestion infrastructure. We cover the telemetry schema, the gateway deployment, custom Elasticsearch mappings and ingest pipelines, managed configuration delivery, and the security use cases enabled by this data.

Why Elastic's InfoSec team monitors AI agents

At Elastic, we practice what we call "Customer Zero." The InfoSec team is the first and most demanding user of Elastic's products, always running the newest versions in production. Our goal is to use our own products to improve our security posture whenever we can.

Claude Code and Cowork are now in active use across Elastic's engineering organization. Claude Code runs locally on developer machines as a CLI-based AI coding assistant. Cowork is part of the Claude Desktop app and also runs locally. It can read files, execute code in a sandbox, search the web, and interact with connected services like Slack, GitHub, Jira, and Google Calendar through MCP connectors. Both tools support connecting to internal systems, which means they operate in a trust boundary that security teams need to monitor.

What Claude Code and Cowork export via OpenTelemetry

Both products export telemetry through standard OpenTelemetry protocols, emitting the same five event types:

• api_request — model, cost, token counts, latency
• tool_result — tool name, MCP server and tool name, success/failure, duration
• tool_decision — auto-approved vs user-approved
• user_prompt — what the user asked the agent to do
• api_error — error message, status code

Every event includes user identity (user.email, organization.id), session context (session.id, prompt.id, event.sequence), and cost/token fields on API request events. Claude Code telemetry is opt-in and redacts prompts and tool arguments by default; enable them with OTEL_LOG_USER_PROMPTS=1 and OTEL_LOG_TOOL_DETAILS=1. Cowork is configured centrally in the Anthropic admin portal and includes full details automatically.

For the full telemetry schema, see the Claude Code Monitoring documentation and the Claude Cowork Monitoring documentation.

Architecture: Getting the data to Elasticsearch

There are two ways to get Claude Code and Cowork OTel data into Elasticsearch. We deployed the self-managed gateway approach first, but Elastic Cloud users have a simpler option.

Option 1: EDOT OTel Gateway (self-managed)

This is the approach we used internally. Since Elastic's InfoSec team runs self-managed ECK (Elastic Cloud on Kubernetes) clusters, we deployed an Elastic Distribution of the OpenTelemetry Collector (EDOT) as a gateway. Both Claude Code and Cowork run locally on user machines and send OTLP/HTTP to the gateway, which authenticates the request and writes to Elasticsearch.

We used the opentelemetry-collector Helm chart with the EDOT collector image (docker.elastic.co/elastic-agent/elastic-otel-collector). The EDOT image provides native Elastic data stream routing, which is important for getting logs into the right data streams without extra configuration.

The gateway runs in deployment mode and uses bearer token authentication via the bearertokenauth extension.

Here is the core collector configuration:

config:
  extensions:
    bearertokenauth:
      scheme: "Bearer"
      token: "${env:OTEL_GATEWAY_TOKEN}"
  receivers:
    otlp:
      protocols:
        http:
          endpoint: "0.0.0.0:4318"
          auth:
            authenticator: bearertokenauth
  processors:
    transform/route:
      log_statements:
        - context: log
          conditions:
            - resource.attributes["service.name"] == "claude-code"
          statements:
            - set(resource.attributes["data_stream.dataset"], "claude_code")
        - context: log
          conditions:
            - resource.attributes["service.name"] == "cowork"
          statements:
            - set(resource.attributes["data_stream.dataset"], "claude_cowork")
  exporters:
    elasticsearch:
      endpoints: ["https://your-elasticsearch:9200"]
      user: "${env:ES_USERNAME}"
      password: "${env:ES_PASSWORD}"
  service:
    extensions: [bearertokenauth]
    pipelines:
      logs:
        receivers: [otlp]
        processors: [transform/route]
        exporters: [elasticsearch]

Option 2: Elastic Cloud Managed OTLP Endpoint (no gateway needed)

If you are running Elastic Cloud (Serverless or Hosted), you can skip the gateway entirely. Elastic's Managed OTLP (mOTLP) endpoint provides a resilient, auto-scaling ingestion layer that accepts OTLP data directly — no collector infrastructure to deploy or maintain.

To use it, point Claude Code's OTLP exporter directly at your Elastic Cloud mOTLP endpoint:

export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_LOGS_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<your-motlp-endpoint>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey <your-api-key>"
export OTEL_RESOURCE_ATTRIBUTES="data_stream.dataset=claude_code"

The data_stream.dataset resource attribute is important here; it controls which data stream receives the logs. Without it, data lands in a generic OTel data stream where your custom index templates and ingest pipelines will not apply. Set it to claude_code or claude_cowork so the data routes to the dedicated logs-claude_code.otel-* or logs-claude_cowork.otel-* streams with the correct field mappings.

With mOTLP, you get native OTLP ingestion with automatic data stream routing, a built-in failure store to protect data during indexing issues, and no APM Server requirement.

The managed endpoint supports all the same custom index templates and ingest pipelines described below, you just don't need to operate the gateway.

For full setup details, see the Elastic Cloud Managed OTLP documentation. Note that it is not currently possible to control index permissions for mOTLP.

Custom Elasticsearch mappings and ingest pipelines

By default, OTel attributes are indexed as keywords in Elasticsearch. That works for filtering and grouping, but it breaks numeric aggregations. You cannot SUM or AVG a keyword field. We created custom mappings to fix the field types and an ingest pipeline to parse JSON string fields into structured objects.

Component template

The component template overrides the default keyword mappings for numeric and boolean fields, and adds flattened type mappings for the JSON-encoded tool parameters:

PUT _component_template/logs-claude_code.otel@custom
{
  "template": {
    "mappings": {
      "properties": {
        "cost_usd":                  { "type": "float" },
        "duration_ms":               { "type": "long" },
        "input_tokens":              { "type": "long" },
        "output_tokens":             { "type": "long" },
        "cache_creation_tokens":     { "type": "long" },
        "cache_read_tokens":         { "type": "long" },
        "prompt_length":             { "type": "long" },
        "tool_result_size_bytes":    { "type": "long" },
        "success":                   { "type": "boolean" },
        "tool_parameters_flattened": { "type": "flattened" },
        "tool_input_flattened":      { "type": "flattened" }
      }
    }
  }
}

The flattened type is important here. tool_parameters and tool_input arrive as JSON strings containing nested keys like mcp_server_name, mcp_tool_name, bash_command, or command. By parsing them into flattened fields, you can query individual keys without creating an unbounded number of mapped fields.

A future enhancement will be to extract high-value fields from these JSON payloads into dedicated mapped fields — things like MCP server names, tool names, and bash commands — to drive richer analytics, aggregations, and detection rules directly on those values.

Index template

The index template composes in all the standard OTel component templates plus our custom one. It matches both logs-claude_code.otel-* and logs-claude_cowork.otel-* so both data streams share the same field mappings:

PUT _index_template/logs-claude_code.otel
{
  "index_patterns": [
    "logs-claude_code.otel-*",
    "logs-claude_cowork.otel-*"
  ],
  "composed_of": [
    "logs@mappings",
    "logs@settings",
    "otel@mappings",
    "otel@settings",
    "logs-otel@mappings",
    "semconv-resource-to-ecs@mappings",
    "logs@custom",
    "logs-otel@custom",
    "logs-claude_code.otel@custom",
    "ecs@mappings"
  ],
  "priority": 150,
  "data_stream": {},
  "allow_auto_create": true,
  "ignore_missing_component_templates": [
    "logs@custom",
    "logs-otel@custom"
  ]
}

Ingest pipeline

The ingest pipeline parses tool_parameters and tool_input from JSON strings into objects, writing to separate *_flattened target fields to avoid conflicts with the original keyword-mapped attributes:

PUT _ingest/pipeline/logs-claude_code.otel@custom
{
  "description": "Parse JSON string fields in Claude Code/Cowork OTel telemetry",
  "processors": [
    {
      "json": {
        "field": "attributes.tool_parameters",
        "target_field": "tool_parameters_flattened",
        "if": "ctx.attributes?.tool_parameters != null && ctx.attributes.tool_parameters.startsWith('{')",
        "ignore_failure": true
      }
    },
    {
      "json": {
        "field": "attributes.tool_input",
        "target_field": "tool_input_flattened",
        "if": "ctx.attributes?.tool_input != null && ctx.attributes.tool_input.startsWith('{')",
        "ignore_failure": true
      }
    }
  ]
}

After creating all three resources, new data flowing into the logs-claude_code.otel-* and logs-claude_cowork.otel-* data streams will have correct numeric field types and searchable structured tool parameters.

Configuring telemetry export

Claude Code and Cowork are configured differently. Claude Code uses standard OpenTelemetry environment variables. Cowork OTel export is configured centrally by administrators in the Anthropic admin portal.

Claude Code supports managed settings that are deployed by IT and cannot be overridden by users. The configuration is a JSON file containing an env block:

{
  "env": {
    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
    "OTEL_METRICS_EXPORTER": "otlp",
    "OTEL_LOGS_EXPORTER": "otlp",
    "OTEL_LOG_TOOL_DETAILS": "1",
    "OTEL_LOG_USER_PROMPTS": "1",
    "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
    "OTEL_EXPORTER_OTLP_ENDPOINT": "https://your-otel-gateway:443",
    "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer your-token"
  }
}

This managed settings file can be delivered via MDM (Jamf, Intune), server-managed settings through the Claude.ai Admin Console, or file-based deployment. See the Claude Code managed settings documentation for the full list of delivery mechanisms and their security properties.

For local testing, you can put the same configuration in ~/.claude/settings.json on your own machine before rolling it out organization-wide.

Cowork

Cowork OTel export is configured centrally by administrators in the Anthropic admin portal. Administrators set the OTLP endpoint and authentication headers in the admin console, and Cowork instances automatically pick up the configuration. Prompt content and tool details are included by default without requiring additional flags.

Because Cowork runs in a sandbox, the OTel gateway endpoint must be allowlisted for outbound network access from the sandbox environment. Without this, telemetry export will fail silently.

Security use cases

The combination of event types, identity fields, and tool parameters creates a rich dataset for security operations. Here are the use cases we are building detection and investigation capabilities around.

Tool invocation auditing. Every tool call is logged with the tool name and input parameters. For MCP tools, this includes the MCP server name and tool name (e.g., slack_send_message, github/search_issues). You can detect unauthorized data access, unusual shell commands, or unexpected MCP server interactions. Use the attributes.tool_name + attributes.tool_parameters fields.

Session reconstruction. The session.id field combined with event.sequence provides a monotonically increasing counter within each session. You can reconstruct the complete sequence of a Claude session: what the user asked, what tools ran, what data was accessed, and what APIs were called. This is valuable for incident response — if you detect a suspicious tool call, you can pull the full session context.

Permission decision analysis. The attributes.event.name: tool_decision events provide insight into how each tool use was approved. This lets you detect users auto-approving risky tool categories, or identify unusual permission patterns across the fleet.

| Decision Source | Meaning | | ----- | ----- | | config | Auto-allowed by settings or policy | | hook | Decided by a configured hook script | | user_temporary | User clicked accept for this invocation | | user_permanent | User clicked "always allow" for this tool | | user_abort | User aborted the session | | user_reject | User explicitly rejected the tool use |

Cost anomaly detection. The cost_usd field on every api_request event enables per-request, per-session, and per-user cost tracking. You can alert on unusually expensive sessions or identify users with outsized consumption patterns.

Correlating with EDR data. If you are running Elastic Defend on your endpoints, you can correlate Claude's OTel telemetry with EDR process and file events to understand the full picture. When Claude Code executes a Bash command, the OTel tool_result event tells you what the agent decided to run and why (via the preceding user_prompt). The corresponding Elastic Defend process event tells you exactly what happened on the host — child processes spawned, files written, network connections made. Joining these two data sources by timestamp and host gives you both the intent (from the AI agent telemetry) and the impact (from endpoint telemetry) in a single investigation.

MCP server access monitoring. As organizations connect AI agents to internal systems through MCP, monitoring which servers are accessed and with what tools becomes critical. The tool_parameters_flattened.mcp_server_name and tool_parameters_flattened.mcp_tool_name fields provide this visibility.

For example, to see tool invocations for Slack, you could query tool_name: "mcp_tool" AND tool_parameters_flattened.mcp_tool_name:slack*.

Beyond OTel: Claude enterprise audit logs

Telemetry from Claude Code and Cowork covers agent activity on endpoints, but it doesn't capture everything. For full visibility, organizations should also collect Claude enterprise audit logs from the Compliance API. This is the only source of activity on the web interface (claude.ai) and of traditional security audit events, such as login activity, permission changes, and organization-level administration. Combining both data sources gives security teams a complete picture across all Claude products.

Conclusion

AI coding assistants and autonomous agents are becoming part of the standard enterprise toolkit. If your security team doesn't have visibility into what these tools are doing, you have a gap. Claude Code and Cowork ship with OpenTelemetry support that provides exactly the kind of telemetry security teams need; identity, session context, tool invocation details, cost data, and permission decisions. Elastic's native OTel ingestion capabilities, whether through the Managed OTLP endpoint on Elastic Cloud or the EDOT Collector in a self-managed environment, make it straightforward to get this data into Elasticsearch, where you can search it, build dashboards, and write detection rules.

If you want to get started, sign up for a free trial of Elastic Cloud and try the Managed OTLP endpoint, or install the EDOT OTel Collector in your existing environment.

References

• Claude Code Monitoring & Telemetry
• Claude Code Settings — Managed settings
• Claude Code Server-managed settings
• Claude Cowork Monitoring
• Elastic Cloud Managed OTLP Endpoint
• EDOT OTel Collector Documentation
• EDOT Collector Authentication Methods
• OpenTelemetry Protocol Exporter Configuration
• OpenTelemetry Collector Helm Chart
• Elastic Defend

Security teams can only protect what they can see. Gaps in coverage, like a macOS fleet generating logs that never reach your SIEM, an email gateway running in isolation, or a cloud environment producing findings that stay siloed in the vendor console, are easily exploited by attackers.

Elastic’s answer to this is continuous and open investment in third-party integrations, built on the belief that a strong security ecosystem requires deep integrations that make data from every corner of the stack searchable and contextualized. Today, we’re announcing nine new integrations for Elastic Security spanning cloud security, endpoint visibility, email threat detection, identity and SIEM.

Each integration ships with ingest pipelines that normalize and structure data out of the box, along with prebuilt dashboards that serve as an immediate starting point for visualization and analysis, so teams can search, correlate and investigate across new data sources from day one without writing or maintaining parsers.

macOS Security Events

Elastic Defend, the native integration that delivers Elastic Endpoint Security, collects rich security telemetry on macOS, and it is intentionally focused on high-value detection signals rather than full system auditing. Login and logout events, account creation and deletion, service registration changes and application diagnostic logs all live outside that scope, leaving threat hunters and IR teams without complete macOS context. The macOS Security Events integration complements Elastic Defend, providing the same depth of OS-level visibility offered to Windows devices via the Windows Event Logs integration.

MacOS endpoints generate tens of thousands of unified log entries per endpoint. Left unfiltered, that volume creates noise rather than signals. This integration ships with predicate-based filters that scope ingestion to security-relevant events: authentication activity, process execution, network connections, file system changes, and system configuration modifications.

These predicate-based filters enable comprehensive macOS coverage without the cost or complexity of ingesting everything. Once ingested, these events are immediately available to Elastic Security’s AI Assistant. Analysts can ask natural-language questions like "Show me all privilege escalation attempts on macOS endpoints in the last 24 hours" or "Summarize login failures for this host”, turning raw unified log entries into actionable investigation context without writing a single query.

Check out the macOS Security Events integration.

IBM QRadar

For teams running IBM QRadar in parallel with Elastic Security, alert ingestion into Elastic has become easier. The QRadar integration collects offense records from QRadar’s offense and rules endpoints, enriching each alert with the triggering rule’s name, ID, type and ownership, so analysts can triage in Elastic without switching back to QRadar.

This integration is the foundation of Elastic’s SIEM migration workflow for QRadar, which mirrors the capability already available for Splunk. Teams can also use Automatic Migration for migrating their QRadar rules into Elastic. It uses semantic search and generative AI to map existing rules to Elastic’s 1,300+ prebuilt detections, and translates anything that doesn’t map directly into ES|QL, allowing you to consolidate your SIEM footprint without manually rebuilding your entire detection library.

Check out the IBM QRadar integration.

Proofpoint Essentials

For Enterprise customers, Proofpoint’s TAP (Targeted Attack Protection) has been available in Elastic. To provide the same email threat visibility to SMB environments and the MSP and MSSPs who serve them, Proofpoint Essentials is now available.

The Proofpoint Essentials integration streams four event types into Elastic Security:

• Clicks on malicious URLs that were blocked
• Clicks that were permitted
• Messages blocked for containing threats recognized by URL Defense or Attachment Defense
• Messages delivered despite containing those threats

To easily surface this data, two prebuilt dashboards are available:

For an SMB SOC team, this means phishing attempts, malware detections and policy violations land in the same platform as the rest of your security telemetry, removing the need to switch platforms to understand the full context of a threat.

Check out the Proofpoint Essentials integration.

AWS Security Hub

AWS Security Hub aggregates findings across your AWS environment, but investigating those findings means staying inside the AWS console, separate from the rest of your team’s security data. The Elastic integration changes this by pulling Security Hub findings into Elastic in Open Cybersecurity Schema Framework (OCSF) format and normalizing them to ECS, offering schema-consistent data that’s immediately searchable via ES|QL.

Findings land in the Elastic Vulnerability Findings page, integrating AWS cloud security posture directly into the workflows already in place. From there, you can correlate Security Hub data with signals from other sources - endpoint alerts, identity events, network telemetry - to build a fuller picture of risk across your AWS environment and investigate faster than the native console allows.

Check out the AWS Security Hub integration.

More new Elastic Security integrations

In addition to the featured integrations above, the following integrations are now available, each shipping with prebuilt dashboards for immediate value:

• JupiterOne: Asset intelligence and cloud attack surface monitoring, ingesting cross-tool alerts, CVE findings, and threat detections enriched with MITRE ATT\&CK mappings and CVSS scores, and host context for unified risk visibility.
• Airlock Digital: Application allowlisting and execution control telemetry, capturing blocked process executions with command lines, file hashes and publisher context, so unauthorized execution attempts are visible and correlatable alongside the rest of your endpoint detections.
• Island Browser: Enterprise browser security events spanning user navigation, device posture, compromised credential detection and admin activity, extending Elastic’s visibility to BYOD and unmanaged devices where traditional endpoint agents can’t be deployed.
• Ironscales: AI-powered phishing detection events capturing email metadata, sender reputation, affected mailbox counts and suspicious links, correlatable with endpoint and identity data for faster investigation and response.
• Cyera: Data security posture management events, surfacing sensitive data risks including exposure severity, affected record counts, compliance framework violations, and datastore ownership across cloud environments, so sensitive data exposure doesn’t stay siloed in a separate DSPM console.

Get started

These integrations Elastic’s open approach to security. All nine integrations in this roundup ship with prebuilt dashboards and native ECS mappings, giving your team immediate visibility with no additional setup or custom visualization work required.

From there, findings, alerts and logs are immediately available to Elastic’s broader detection and investigation capabilities: Attack Discovery for surfacing multi-stage threats, AI Assistant for natural-language investigation and guided response, and to ES|QL and EQL for custom detection and hunting queries.

• Browse available integrations
• Learn about migrating to Elastic Security from other SIEMs

Have questions or feedback? Join #security-siem in the Elastic Stack Community Slack.

This article highlights how Elastic's new Terraform resources for security detection rules and exceptions expand practitioners' capabilities for detection-as-code deployment. Below you will find examples of defining and deploying your detection artifacts in Elastic Security with Terraform. We will also show how you can use Elastic's AI Agent to help quickly create the Terraform configuration for your custom rules. Finally, it also provides guidance on when to use the Elastic Stack Terraform provider versus tools from the detection-rules repository.

Managing Elastic with Terraform

Terraform is a tool created by HashiCorp (now IBM) to manage infrastructure in the cloud, or in self-managed environments, as code. With a simple stroke of HCL (HashiCorp configuration language), users can define the desired state of their cloud provider infrastructure, application, configuration, and in Elastic’s case, cluster settings, configuration, indices or streams, and now also detection rules and alerts, as fully configurable, traceable, and reviewable code in your favorite source management tool.

The Elastic Stack Terraform provider helps search, observability, and security professionals, as well as DevOps and SREs, configure their Elastic clusters with the right indices and mappings for their search use cases, SLOs or Fleet policies for their observability use case, and now, detection rules, alerts, and exceptions for their security use case. It can easily configure those, and many more objects and settings in the Elastic Stack.

Security Detection rules - now as code with Terraform

With V0.12.0 and V0.13.0 of the Elastic Stack Terraform provider, users can now manage their Detection rules and rule exceptions using Terraform. This is especially useful for users who have already been managing their Elastic deployments with Terraform and want to extend it to Detection rules.

Using the Elastic Stack Terraform Provider to deploy Rules and Exceptions

Let's look at an example of using the Elastic Stack Terraform Provider to deploy an Elastic Security Rule. In this example, we want to detect Windows Service Accounts that are performing an interactive logon on a host.

Service accounts typically have elevated privileges and rarely-rotated passwords, making them high-value targets for attackers. Since these accounts should only perform automated service logons, an interactive logon can indicate credential theft or misuse.

The first thing we need to think of is what telemetry we need to see which logons are happening on our host. Logon events are logged by the Windows Local Security Authority Subsystem Service (LSASS) whenever a logon session is successfully created on the machine. We can pick this up via an Elastic Agent with the Windows Integration installed.

The data will be written by the Elastic Agent into the system.security Data Stream, we can match it with this index pattern: logs-system.security-*. We also know that Logon events generate event code 4624 and that, in our example, the service account name starts with svc or ends with $. In addition, an interactive login will have a logon type of interactive.

So, we can match these events with an ES|QL rule like:

FROM logs-system.security-\*  
| WHERE event.code \== "4624" AND (user.name LIKE "svc\_\*" OR user.name LIKE "svc-\*"  
     OR user.name LIKE "\*\_svc" OR user.name LIKE "\*$")  
     AND winlog.logon.type IN ("Interactive", "RemoteInteractive",  
         "CachedInteractive", "CachedRemoteInteractive")  

There may be situations where we don't want this rule to run, for example, if there is a legacy application that we want to permit interactive logons from. So, we can create an Exception Item, like: user.name IS svc\_sqlbackup.

Now that we know what we want the Rule and its Exceptions to look like, we can use the Terraform provider's elasticstack_kibana_security_detection_rule, elasticstack_kibana_security_exception_list, and elasticstack_kibana_security_exception_item resources to define them in code.

Turning ES|QL rules into Terraform's configuration syntax, HCL, is a great use case for Elastic's AI Agent.
Elastic AI Agent capabilities help accelerate security operations across a wide range of tasks - from alerts triage and incident response to helping with detection lifecycle tasks.

Simply open AI Agent, and ask it to create Terraform configurations based on your query and exceptions.

You should end up with something like this:

Here's a closer look at the code.

There are a few elements to call out specifically:

• type: The type of exception list. For example: detection, endpoint, or endpoint_trusted_apps
• namespace_type: Determines whether the exception list is available in all Kibana spaces or just the single space in which it was created.

resource "elasticstack_kibana_security_exception_list" "svc_account_interactive_login" {
  list_id        = "svc-account-interactive-login-exceptions"
  name           = "Service Account Interactive Login Exceptions"
  description    = "Documented exceptions for service accounts that legitimately require interactive logon"
  type           = "detection"
  namespace_type = "single"
  tags           = ["service-accounts","windows","authentication"]
}  

This creates a new exception list.

Of note, the entries array contains the conditions under which the exception applies.

resource "elasticstack_kibana_security_exception_item" "svc_sqlbackup" {
  list_id        = elasticstack_kibana_security_exception_list.svc_account_interactive_login.list_id
  item_id        = "svc-sqlbackup-exception"
  name           = "svc_sqlbackup - Legacy SQL Backup Agent"
  description    = "Approved exception: Legacy SQL backup agent requires interactive logon per vendor documentation."
  type           = "simple"
  namespace_type = "single"
  tags           = ["sql","backup","approved"]
entries = [
    {
      field    = "user.name"
      type     = "match"
      operator = "included"
      value    = "svc_sqlbackup"
    }
  ]
} 

This adds our exception: that we don't want the rule to run if the username is svc\_sqlbackup.

Of note, the elements from enabled to the technique array are examples of the other properties that can be set on a rule.

resource "elasticstack_kibana_security_detection_rule" "svc_account_interactive_login" {
  name        = "Service Account Interactive Login"
  description = <<-EOT
    Detects interactive logins by service accounts. Service accounts should authenticate
    via service (Type 5) or batch (Type 4) logon types, not interactively. Interactive
    logins by service accounts may indicate credential theft or misuse.

    This rule identifies service accounts by common naming conventions (svc_*, svc-*,
    *_svc) and managed service accounts (*$).
  EOT

  type     = "esql"
  language = "esql"
  query    = <<-EOT
    FROM logs-system.security-* metadata _id, _version, _index
    | WHERE event.code == "4624"
      AND (user.name LIKE "svc_*" OR user.name LIKE "svc-*" OR user.name LIKE "*_svc" OR user.name LIKE "*$")
      AND winlog.logon.type IN ("Interactive", "RemoteInteractive", "CachedInteractive", "CachedRemoteInteractive")
    | KEEP @timestamp, host.name, user.name, user.domain, winlog.logon.type, source.ip, _id, _version, _index
  EOT

  enabled    = true 
  severity   = "high"
  risk_score = 73

  from     = "now-6m"
  to       = "now"
  interval = "5m"

  author  = ["Security Team"]
  license = "Elastic License v2"
  tags    = [
    "Domain: Endpoint",
    "OS: Windows",
    "Use Case: Identity and Access Audit",
    "Tactic: Initial Access",
    "Data Source: Windows Security Event Log"
  ]

  false_positives = [
    "Service accounts with documented exceptions that require interactive logon",
    "Break-glass procedures during incident response",
    "Initial service account configuration or troubleshooting"
  ]

  references = [
    "https://learn.microsoft.com/en-us/entra/architecture/service-accounts-on-premises",
    "https://blog.quest.com/10-microsoft-service-account-best-practices/",
    "https://attack.mitre.org/techniques/T1078/002/"
  ]

  threat = [
    {
      framework = "MITRE ATT&CK"
      tactic = {
        id        = "TA0001"
        name      = "Initial Access"
        reference = "https://attack.mitre.org/tactics/TA0001/"
      }
      technique = [
        {
          id        = "T1078"
          name      = "Valid Accounts"
          reference = "https://attack.mitre.org/techniques/T1078/"
          subtechnique = [
            {
              id        = "T1078.002"
              name      = "Domain Accounts"
              reference = "https://attack.mitre.org/techniques/T1078/002/"
            }
          ]
        }
      ]
    }
  ]

  exceptions_list = [
    {
      id             = elasticstack_kibana_security_exception_list.svc_account_interactive_login.id
      list_id        = elasticstack_kibana_security_exception_list.svc_account_interactive_login.list_id
      namespace_type = elasticstack_kibana_security_exception_list.svc_account_interactive_login.namespace_type
      type           = elasticstack_kibana_security_exception_list.svc_account_interactive_login.type
    }
  ]
}

Finally, we define the rule, including the ES|QL query we provided earlier and MITRE ATT\&CK classification.

You can add these resource definitions into one configuration file (perhaps security-rules.tf), add it to your configured Elastic Stack terraform directory, and then run the `terraform apply` command and parameter to deploy the Rule.

terraform apply --auto-approve

Since terraform apply runs a plan before making changes, it will automatically detect if anyone has edited a rule directly in Kibana and show you exactly what drifted: no manual exports or diffs needed.

After Terraform has made the changes, we can see the Rule in Kibana:

We can also see the Exception List:

This way, you can define your detections in Terraform and benefit from automatic deployment along with other objects you manage with Terraform.

Terraform workspaces for multi-space Elastic deployments

Terraform uses a concept called “workspaces” allowing you to reuse the same infrastructure code for multiple deployments, for example, a dev, testing, and production environment. This concept is useful for managing rules across multiple deployments and/or Kibana spaces.

Managing detections with Terraform and Detections as code

Elastic also has Detections as Code functionality available via our open detection-rules repository.

The two tools have complementary strengths and are aligned with different user profiles and workflow stages for implementing Detections as Code.

Detection as Code features in detection-rules

• Best fit user profile: Detection engineers
• Intended workflow phase: Rule authoring and validation

With dual-sync between your GitHub repo and Kibana, linting, schema validation, and unit-testing, detection-rules functionality is well-suited to experienced Detection Engineers comfortable with Git-based version control.

Elastic Stack Terraform Provider

• Best fit user profile: DevOps engineers / Platform teams
• Intended workflow phase: Deployment and operations

For users already using Terraform to manage their Elastic clusters, the Terraform Provider is a great fit, bringing consistency to all "x-as-code" operations and familiar state management and parameterization.

The key differences and optimal use cases for each tool are detailed in the comparison table below:

| Workflow Stage | detection-rules | Terraform Provider | Best Fit | | :---- | :---- | :---- | :---- | | Rule Authoring | Purpose-built tooling: create-rule wizard, TOML schema, KQL/EQL validation, field checks against ECS, Kibana-to-code export. | Standard HCL definitions; teams integrate their preferred validation tooling into existing pipelines. | detection-rules: Detection engineers authoring and refining rules daily. Teams wanting to automatically convert rules from Kibana into code. Terraform: Teams already using Terraform in their workflows, or teams wanting to automate and deploy detection rules as code, but without an established CI/CD platform. | | Testing & Validation | Built-in unit testing framework, schema validation, query validation, configurable test suites. | Terraform tests for optional unit testing. No built-in query validation: the provider relies on the Kibana API to accept or reject rule definitions at apply time. | detection-rules: Teams wanting out-of-the-box detection testing. Terraform: Platform teams managing rules as part of broader IaC with existing validation pipelines. Teams happy to write custom tests in Terraform. | | Exception Management | Native exception list handling; export/import with rules, TOML storage, and rule linking. | Exception lists can be referenced via rule attributes. | detection-rules: Teams managing exceptions as part of detection content. Terraform: Teams managing exceptions as separate infrastructure resources. | | Governance & Drift Management | VCS-based with dual sync: push rules from repo to Kibana and export from Kibana back to repo, allowing either to serve as the source of truth. Drift detection is achievable with custom export-and-diff tooling. | VCS-authoritative: state file enforces declared configuration. Native drift detection: Terraform plan surfaces any out-of-band changes made in Kibana. | detection-rules: Teams comfortable with Git-based workflows and flexible sync models. Terraform: Organisations requiring formal state reconciliation and audit trails. | | Rollback | Git history provides version control; re-import previous versions from the repo. | Revert HCL configuration in Git and re-apply to restore the previous state. | detection-rules: Teams using Git-centric recovery workflows. Terraform: Organisations with standardised rollback mechanisms across infrastructure and rulesets. | | Parameterisation & Templating | Achievable with external preprocessing (Jinja2, etc.) before import. | Native HCL features: variables, locals, for_each, dynamic blocks, and modules. | detection-rules: Teams not requiring parameterisation or with existing templating solutions. Terraform: Teams wanting native IaC parameterisation. | | Operational Integration | Focused tooling optimised for detection engineering workflows. | Unified control plane managing detection rules alongside cloud infrastructure, network policies, and other security tooling. Integrates with other resources that may be required by detections such as external connectors. | detection-rules: Specialist detection teams. More flexible if dual-sync (Kibana and repo are both sources of truth). Terraform: Platform teams managing Elastic as part of broader infrastructure. |

In short, Detection Engineers are better served by the specialized creation and testing tools provided in the detection-rules repository, while DevOps/Platform Teams should use the Terraform provider to manage detection rules as part of their broader infrastructure-as-code strategy for deployment and operations.

Try it out

To experience the full benefits of what Elastic has to offer for detection engineers, upgrade to 9.3 or start your Elastic Security free trial. Visit elastic.co/security to learn more and get started.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

This brings your detection logic and ML jobs into the same versioned, peer-reviewed workflow as your core clusters. It ensures your security posture and AI connectors are no longer manual outliers in an otherwise automated environment.

The challenge: Security and observability configuration at scale

As Elastic deployments grow, so does the complexity of managing them. Security teams maintain hundreds of detection rules. SREs configure monitoring across dozens of clusters. ML engineers tune anomaly detection jobs across multiple environments. All of these configurations must be consistent, auditable, and reproducible.

Without infrastructure as code, teams face two problems:

1. Configuration drift. Rules, policies, and monitors are created manually through the Kibana UI. Over time, production and staging diverge. No one is sure which version of a detection rule is running where.

2. Buried audit trail. When a detection rule changes or an exception is added, there's no pull request to review, no commit history to trace, and no rollback path if something breaks. Users need to put in extra effort to access such history.

Elastic Stack Terraform provider solves this by bringing these configurations into the same version-controlled, peer-reviewed workflow that teams already use for infrastructure.

Security artifacts as code: Detection rules, exceptions, and prebuilt rules

You can now manage the full lifecycle of Elastic Security detection rules through Terraform.

Detection rules

The elasticstack_kibana_security_detection_rule resource lets you define, version, and deploy detection rules in the HashiCorp Configuration Language (HCL) format:

resource "elasticstack_kibana_security_detection_rule" "suspicious_admin_logon" {
  name        = "Suspicious Admin Logon Activity"
  type        = "query"
  query       = "event.action:logon AND user.name:admin"
  language    = "kuery"
  enabled     = true
  description = "Detects suspicious admin logon activities"
  severity    = "high"
  risk_score  = 75
  from        = "now-6m"
  to          = "now"
  interval    = "5m"
  tags        = ["security", "authentication", "admin"]
}

This means your detection rules live in Git, undergo code review, and are deployed consistently across environments. No more clicking through the Kibana UI to replicate rules from staging to production.

Detection rule resource docs

Exception lists and items

The security-as-code story extends to a full suite of exception management resources:

• elasticstack_kibana_security_exception_list - Create and manage exception lists
• elasticstack_kibana_security_exception_item - Define individual exception items within a list
• elasticstack_kibana_security_list and elasticstack_kibana_security_list_item - Manage value lists for IP allowlists, file hashes, and other indicators
• elasticstack_kibana_security_list_data_streams - Associate lists with specific data streams

Here's an example that ties them together - an exception list with items that suppress known false positives for a detection rule:

resource "elasticstack_kibana_security_exception_list" "vuln_scanner_exceptions" {
  list_id        = "vuln-scanner-exceptions"
  name           = "Vulnerability Scanner Exceptions"
  description    = "Suppress alerts from authorized vulnerability scanners"
  type           = "detection"
  namespace_type = "single"
  tags           = ["security", "vulnerability-scanning"]
}

resource "elasticstack_kibana_security_exception_item" "nessus_scanner" {
  list_id        = elasticstack_kibana_security_exception_list.vuln_scanner_exceptions.list_id
  item_id        = "nessus-scanner"
  name           = "Nessus Scanner - Authorized"
  description    = "Suppress alerts from authorized Nessus scanner hosts"
  type           = "simple"
  namespace_type = "single"

  entries = [
    {
      type     = "match"
      field    = "source.ip"
      operator = "included"
      value    = "10.0.50.10"
    },
    {
      type     = "match_any"
      field    = "process.name"
      operator = "included"
      values   = ["nessus", "nessusd"]
    }
  ]

  tags = ["nessus", "authorized-scanner"]
}

resource "elasticstack_kibana_security_exception_item" "qualys_scanner" {
  list_id        = elasticstack_kibana_security_exception_list.vuln_scanner_exceptions.list_id
  item_id        = "qualys-scanner"
  name           = "Qualys Scanner - Authorized"
  description    = "Suppress alerts from authorized Qualys scanner subnet"
  type           = "simple"
  namespace_type = "single"

  entries = [
    {
      type     = "match"
      field    = "source.ip"
      operator = "included"
      value    = "10.0.51.0/24"
    }
  ]

  tags = ["qualys", "authorized-scanner"]
}

The exception list and its items are linked by list_id, so Terraform manages the dependency graph automatically. Adding a new authorized scanner is a one-line PR - no clicking through the Kibana UI, no risk of forgetting which environment got the update.

Prebuilt security rules

The elasticstack_kibana_prebuilt_rule resource lets you manage Elastic's prebuilt detection rules via Terraform. This is particularly valuable for organizations that need to track which prebuilt rules are enabled, customize their parameters, and ensure consistent deployment across environments.

ML anomaly detection as code

Machine learning anomaly detection is one of Elasticsearch's most powerful capabilities - but managing ML jobs across environments has traditionally been a manual process. You create a job in the Kibana UI, tune the detectors, configure the datafeed, and hope someone documents the settings so they can be replicated in the next environment.

The elasticstack_elasticsearch_ml_anomaly_detection_job resource changes that. You can now define the full configuration of an anomaly detection job in HCL - detectors, bucket spans, influencers, data feeds, and analysis limits - and deploy it consistently across dev, staging, and production.

resource "elasticstack_elasticsearch_ml_anomaly_detection_job" "cpu_anomalies" {
  job_id      = "high-cpu-by-host"
  description = "Detect unusual CPU usage patterns"

  analysis_config = {
    bucket_span = "15m"
    detectors   = [{
      function   = "high_mean"
      field_name = "system.cpu.user_pct"
    }]
    influencers = ["host.name"]
  }

  data_description = {
    time_field = "@timestamp"
  }
}

This matters for teams that rely on ML to catch infrastructure anomalies, unusual user behavior, or security threats. Instead of manually recreating jobs when spinning up new clusters or recovering from failures, the entire ML configuration lives in version control - reviewable, repeatable, and recoverable.

Cross-cluster automation with API keys

For organizations running multiple Elasticsearch clusters, the provider now supports cluster API keys for cross-cluster search (CCS) and cross-cluster replication (CCR). You can create API keys specifically designed for secure cross-cluster communication, enabling end-to-end automation of multi-cluster architectures.

This means you can provision two clusters, configure CCS/CCR between them, and set up the necessary security credentials - all in a single Terraform configuration.

resource "elasticstack_elasticsearch_security_api_key" "ccs_key" {
  name = "cross-cluster-search-key"
  type = "cross_cluster"

  access = {
    search = [{
      names = ["logs-*", "metrics-*"]
    }]
    replication = [{
      names = ["archive-*"]
    }]
  }

  expiration = "90d"

  metadata = jsonencode({
    environment = "production"
    purpose     = "ccs-ccr-between-prod-clusters"
    team        = "platform"
  })
}

When the type is set to cross_cluster, the API key is scoped to CCS/CCR operations. You define which index patterns are accessible for search and replication, set an expiration policy, and tag the key with metadata - all reviewable in a pull request.

Learn more about API key resources in the documentation.

AI connectors as code

The provider now supports .bedrock and .gen-ai connectors, bringing AI infrastructure into your Terraform workflows. As teams increasingly integrate large language models into their Elastic workflows - for AI assistants, attack discovery, and automated investigations - managing these connector configurations as code becomes essential.

resource "elasticstack_kibana_action_connector" "bedrock" {
  name              = "aws-bedrock"
  connector_type_id = ".bedrock"
  config = jsonencode({
    apiUrl       = "https://bedrock-runtime.us-east-1.amazonaws.com"
    defaultModel = "anthropic.claude-v2"
  })
  secrets = jsonencode({
    accessKey = var.aws_access_key
    secret    = var.aws_secret_key
  })
}

resource "elasticstack_kibana_action_connector" "openai" {
  name              = "openai"
  connector_type_id = ".gen-ai"
  config = jsonencode({
    apiProvider  = "OpenAI"
    apiUrl       = "https://api.openai.com/v1/chat/completions"
    defaultModel = "gpt-4"
  })
  secrets = jsonencode({
    apiKey = var.openai_api_key
  })
}

With these connectors defined in Terraform, you can version your AI integration configuration alongside the rest of your Elastic infrastructure - and swap models or providers through a simple PR.

Observability enhancements

Synthetics monitors

The elasticstack_kibana_synthetics_monitor resource now includes a labels field, enabling better organization and filtering of synthetic checks. Labels let you tag monitors by team, environment, or service, making it easier to manage synthetic monitoring at scale.

Additional platform improvements

Recent releases also included several resources and attributes that round out the provider's coverage:

• elasticstack_elasticsearch_alias - Manage Elasticsearch aliases as a dedicated resource
• elasticstack_kibana_default_data_view - Set the default data view for a Kibana space
• solution attribute on elasticstack_kibana_space - Configure the solution type for Kibana spaces (available from 8.16)
• Fleet agent policy enhancements - host_name_format for configuring hostname vs. FQDN, and required_versions for version pinning

Getting started

If you're already using the Elastic Stack Terraform provider, upgrade to the latest provider version to get all of these capabilities:

terraform {
  required_providers {
    elasticstack = {
      source  = "elastic/elasticstack"
      version = "~> 0.14"
    }
  }
}

If you're new to managing your Elastic Stack with Terraform, start with the provider documentation on the Terraform registry.

To start using Elastic Cloud today, log in to the Elastic Cloud console or sign up for a free trial.
For the full set of changes, check out the release notes on GitHub.

In modern security operations, detection engineering is no longer a “set it and forget it” discipline. The central challenge for any security team – and the question that underpins the entire purple-team approach is simple: how do you know whether your detection rules genuinely work? Continually validating detection logic against an ever-shifting adversary toolkit is now a fundamental requirement.

Arguably, the largest hurdle for this exercise has always been setting up the lab. Manually provisioning a multi-domain Active Directory forest, configuring it with specific vulnerabilities, and deploying a separate, contained malware analysis environment is a complex and time-consuming process. This repetitive setup work is a significant drain on an organization's most valuable resource: the time of its senior security analysts. Community discussions echo this frustration, highlighting the hours lost to manual setup before a single test can be run.

This blog details a modern solution that eliminates this bottleneck by combining rapid infrastructure automation with a unified security analytics platform. The solution leverages two key components:

1. Ludus: An open-source automation overlay that deploys and configures complex, multi-VM cyber ranges from a single command.
2. Elastic Security: The platform that unifies Security Information Event Management (SIEM), eXtended Detection and Response (XDR), and cloud security, providing a consolidated solution to ingest, detect, and respond to threats. It offers the "limitless visibility" required to observe every action within the simulated environment.

The goal of this guide is to provide a definitive, step-by-step blueprint for building this integrated system. It will show how to move from slow, manual, and inconsistent lab testing to a continuous, automated, and scalable detection-engineering workflow beyond what Elastic Cortado provides.

The Solution Architecture: Ludus + Elastic

This architecture represents a high-fidelity simulation of a modern hybrid enterprise. The Ludus range acts as the "on-prem" or IaaS data center, while the Elastic Cloud deployment represents the "SaaS" security stack. This model perfectly mirrors the hybrid and multi-cloud environments that Elastic Security is designed to protect, making the architecture of the test as valuable as the attacks themselves.

The build consists of the following core components.

| Component | | Technology | Function | | :---- | ----- | :---- | :---- | | Foundation (Infrastructure) | | Ludus (Proxmox/Ansible) | Deploys VM ranges from a single YAML config. | | Targets | | Identity - GOAD (Windows Server) Supply Chain - XZbot (Debian) | Multi-domain AD forest with intentional vulnerabilities (Kerberoasting, Print Nightmare). Linux host infected with CVE-2024-3094 for supply chain simulation. | | | | | | | The Sensor Grid (Visibility) | | Elastic Agent | Unified telemetry collection (EDR + Logs). | | The Brain (Analysis) | | Elastic Security | SIEM/XDR platform for correlation and AI-driven investigation. |

Component 1: The Foundation (Ludus)

Ludus serves as the Infrastructure-as-a-Service (IaaS) layer. Built to run on Proxmox 8/9 or Debian 12/13, it uses YAML configuration files to define complex virtual networks, supporting up to 255 distinct VLANs. Behind the scenes, Ludus easily leverages Packer and Ansible to build, configure, and deploy the virtual machine templates from that single file.
Review and follow the installation steps and hardware requirements in the Ludus quick-start.

Component 2: The Targets (The Labs)

This guide merges two distinct Ludus environments into a single, comprehensive range to test a wider spectrum of threats:

• Game of Active Directory (GOAD): A purpose-built Active Directory lab designed by security researchers at Orange Cyberdefense. It is pre-configured with the specific misconfigurations and vulnerabilities needed to simulate common identity-based attack paths, such as Kerberoasting, NTLM Relay, and Active Directory Certificate Services (ADCS) abuse.
• XZbot Malware Lab: A high-risk, high-fidelity malware environment. This lab contains the actual, functional CVE-2024-3094 backdoor. This provides a perfect, modern test case for a sophisticated software supply-chain attack.

Important Disclaimer

Handling live malware, even for research, can violate Acceptable Use Policies (AUPs) of ISPs or cloud providers. Ensure you own the infrastructure (Ludus is on-prem) and ensure your upstream ISP allows for such research, or route traffic through a VPN.

Component 3: The Sensor Grid (Elastic Agent & Defend)

To gain visibility, every virtual machine in the Ludus range across both GOAD and XZbot labs will be instrumented with Elastic Agent, a single, unified agent for data collection and protection (via Elastic Defend).

This instrumentation is automated via the badsectorlabs/ludus_elastic_agent Ansible role. This role is the critical lynchpin that programmatically bridges the infrastructure provisioning phase (Ludus/Ansible) with the security instrumentation phase (Elastic), enabling a true "infrastructure-as-code" workflow.

Crucially, the Elastic Agent policy will be configured with the Elastic Defend integration. This elevates the agent from a simple log collector to a full-powered Endpoint Detection & Response (EDR)/eXtended Detection & Response (XDR) solution, providing host-based detections (including Machine Learning (ML) driven malware and ransomware detection) and the deep, kernel-level telemetry essential for detection.

Note: For the purple team approach outlined in this blog, set policies to Detect mode.

Component 4: The Brain (Elastic Cloud Hosted / Elastic Serverless)

All security telemetry and alerts from the Elastic Agents in the Ludus range are streamed to a centralized Elastic Cloud Hosted (ECH) or Elastic Serverless deployment. This is where the unified platform's analytical power comes to life. Using a cloud-native platform is not just for hosting; it is what unlocks Elastic's most advanced, force-multiplying features, including Attack Discovery and the AI Assistant. Click here to start a trial on Elastic Cloud.

The diagram below provides an overview of the build, which is based on the GOAD lab.

Phase 1: Building and Instrumenting the Range

This section provides a technical, step-by-step guide to configuring and deploying the automated range. The process follows a clear "infrastructure-as-code" (IaC) model, where the security instrumentation is defined alongside the infrastructure itself, ensuring a consistent and repeatable monitoring posture for every deployment. The Elastic Cloud instance and its configurations can be managed with the Elastic Cloud and Elastic Stack Terraform provider for a full IaC model of the range and the SIEM.

3.1 Configuring the Elastic Agent Policy (in Kibana)

Before running the Ludus range deployment, the agent policy must be created in the Elastic Cloud instance. This policy is what enables the powerful EDR/XDR telemetry.

The operational flow is as follows:

1. Log in to the Elastic Cloud (ECH) or Elastic Serverless Kibana instance.
2. Navigate to Management > Fleet.
3. Create a new Agent policy (e.g., "ludus-range-policy"). The ludus_elastic_agent role will enroll agents into the policy you specify in your VM-level customization or into the default policy linked to the global variable.
4. Add the Elastic Defend integration to this policy.
5. Configure the Elastic Defend integration to run in Detect mode. This activates the full suite of EDR telemetries.
6. Save the policy and click "Add agent." This will provide the Enrollment token (for ludus_elastic_enrollment_token) and Fleet server URL (for ludus_elastic_fleet_server) needed for the ludus.yml file.
7. (Optional) Repeat steps 3-6 to create customized policies to align with the host’s functions and capabilities for VM-level customization of policies.

Once this policy is created and the token is pasted into the ludus.yml file, running Ludus range deploy will execute the full, automated workflow. Ludus provisions the VMs, and Ansible installs the Elastic Agent, which then enrolls in Fleet and automatically pulls down the policy containing the Elastic Defend integration. This provides the rich EDR telemetry - kernel-level process, file, network, and registry events - from the moment the lab is born.

3.2 The Ludus YAML Configuration (ludus.yml)

Ludus provides the steps to deploy the GOAD range here. The configuration for the range is stored in the ludus.yml configuration file. For the GOAD range, it is located in ad/GOAD/providers/ludus/config.yml.
The full configuration in the appendix is an example based on a sample running configuration that merges a full GOAD lab (on VLAN 10) with the XZbot lab (on VLAN 20).

To deploy a customized version during installation, update the ad/GOAD/providers/ludus/config.yml file before running the goad.sh script in step 2.

git clone https://github.com/Orange-Cyberdefense/GOAD.git
cd GOAD
sudo apt install python3.11-venv
export LUDUS_API_KEY='myapikey'  # put your Ludus admin api key here nano ad/GOAD/providers/ludus/config.yml # customize the configuration here
./goad.sh -p ludus
GOAD/ludus/local > check
GOAD/ludus/local > set_lab GOAD # GOAD/GOAD-Light/NHA/SCCM
GOAD/ludus/local > install

Two key configuration options can be used to customize the range:

1. Global Variables: To simplify the config and avoid repetition, the Elastic Agent variables are defined once at the top level in a global Ansible.vars block and are inherited by all VMs.

   The enrollment token determines the Elastic Agent policy used.

# ludus.yml
---
# --- GLOBAL ANSIBLE VARS (Simplification) ---
# Define Elastic agent vars once and apply globally
global_role_vars:
  ludus_elastic_fleet_server: "<your-fleet.example.com:443>" # Use 443 for cloud
  ludus_elastic_enrollment_token: "<your_enrollment_token>"
  ludus_elastic_agent_version: "9.2.1"

1. VM-level Variables: The Elastic Agent variables can be configured at the VM-level to customize the policy applied. These can be combined with the global variable, for example, where the agent version and fleet_server are set via global variables, and the enrollment tokens are set at the VM-level to apply different policies to VMs.

# --- VM DEFINITIONS ---
vms:
  # --- GOAD LAB (VLAN 10) ---
  - name: "{{ range_id }}-GOAD-DC01"
    hostname: "{{ range_id }}-DC01"
    template: win2019-server-x64-template
    vlan: 10
    ip_last_octet: 10
    ram_gb: 4
    cpus: 2
    windows: { sysprep: true }
    ansible:
      roles:
        - badsectorlabs.ludus_elastic_agent
      role_vars:
        ludus_elastic_enrollment_token: "<your_enrollment_token>" # different token for different policies
  # (Definitions for GOAD-DC02, GOAD-DC03, GOAD-SRV02, GOAD-SRV03 
  #  would follow, all inheriting the global ansible vars)

Automating Elastic Agent Deployment

The ludus.yml snippet above demonstrates the automation. By adding the badsectorlabs.ludus_elastic_agent role to the ansible.roles section of each VM definition, Ludus will automatically install and configure the agent during deployment.

This single Ansible role is compatible with all operating systems in our heterogeneous lab, including Windows (for GOAD), Kali, and Debian (for XZbot).

As shown in the simplified YAML, the ansible.vars block at the top level passes the critical parameters to the role:

• ludus_elastic_fleet_server: The Fleet server URL and port for your Elastic Cloud deployment (e.g., your-fleet.example.com:443).
• ludus_elastic_enrollment_token: The token that enrolls the agent.
  The full example sets the ludus_elastic_enrollment_token at the VM level to demonstrate the ability to use different policies.
• ludus_elastic_agent_version: The specific agent version to install (e.g., 9.2.1).

Note: The Kali host will have Elastic Defend also deployed to monitor attacker behavior, this won’t be possible in a real-world scenario.

Safety First: Isolation, OPSEC, and Live Malware

This section contains a critical safety and operational security (OPSEC) warning. This configuration involves a significant, non-trivial risk that must be professionally managed.

4.1 The Threat: This is Not a Simulation

It must be stated unequivocally: The Ludus XZbot lab guide and its associated Ansible role install the actual, functional CVE-2024-3094 backdoor. This is not benign, simulated code. The lab's own documentation states: "Danger: This role contains malware (on purpose)."

While described as a "passive backdoor" (meaning it requires an attacker to actively trigger it), any virtual machine running this code with an open internet connection is a catastrophic liability. It could be scanned, exploited by unknown actors, or used as a pivot point to attack other networks.

4.2 The Contradiction: Isolation vs. Cloud Connectivity

This architecture creates a direct and critical operational conflict:

1. Requirement 1 (Safety): The malware lab must be isolated from the public internet to prevent compromise or breakout.
2. Requirement 2 (Function): The Elastic Agent must have outbound internet connectivity to reach the Elastic Cloud Hosted / Elastic Serverless endpoints for enrollment and data streaming.

A novice user would fail here, either by exposing their infected lab to the world or by isolating it so completely that no security telemetry can be collected.

4.3 The Solution: Pinhole Egress via Ludus Testing mode

The conflict is resolved using Ludus's built-in "testing" mode, which provides granular control over network egress. This feature is used for the pinhole egress, which enables agent control, telemetry, and log output.

# 1. Start the isolated testing session
ludus testing start # Note external DNS resolvers may also need to be added # ludus testing allow -i 1.1.1.1,8.8.8.8

# 2. Allow Elastic Fleet Server (Control Plane)
# Replace <id> with your specific deployment ID # Note the endpoint will differ based on the cloud providers
ludus testing allow -d <your-deployment-id>.fleet.us-central1.gcp.cloud.es.io

# 3. Allow Elasticsearch Ingest (Data Plane) # Note the endpoint will differ based on the cloud providers
ludus testing allow -d <your-deployment-id>.es.us-central1.gcp.cloud.es.io

This configuration delivers an expert-level solution: the malware is safely contained, while the Elastic Agent is granted only the minimal connectivity required to make policy updates (via communication with the fleet endpoint) and to ingest data (via communication with the ES endpoint).

4.4 Accessing the Range in Testing Mode (WireGuard)

Once Testing Mode is active, standard routing fails. You cannot simply SSH into your Kali VM from your local LAN because the router drops the traffic. Ludus provides an out-of-band management channel using WireGuard.

Ludus configures a WireGuard interface (wg0) on the router VM (198.51.100.1) and assigns you a static client IP (e.g., 198.51.100.2).

• Persistent Allow Rules: The router's firewall configuration includes specific rules in the LUDUS_DEFAULTS chain. These rules explicitly ACCEPT traffic sourced from or destined to the WireGuard subnet (198.51.100.0/24).
• Priority: Because these rules exist in the LUDUS_DEFAULTS chain, they override the DROP rules applied by Testing Mode.

How to connect:

1. Generate your config: ludus user wireguard > ludus.conf
2. Import this into your local WireGuard client and activate the tunnel.
3. Connect directly to the private IPs of your VMs (e.g., 10.10.10.11) over the tunnel.

Phase 2: Executing the Attacks

With the high-fidelity, fully instrumented range deployed, the "Red Team" phase can begin. This involves logging into a dedicated attacker VM (like the included Kali VM or a remnux-analyzer VM) and executing the attacks. This activity generates the rich, malicious telemetry that Elastic Defend will capture.

This combined range allows for testing defenses against the two dominant, macro-level threat vectors: identity-based "living-off-the-land" (LotL) attacks and vulnerability-based supply-chain intrusions.

5.1 Active Directory Simulation (GOAD)

• Initial Access (Credential Stuffing)

1. The attacker targets the external perimeter. Using a list of breached credentials, you execute a password stuffing attack against the Essos.local domain. You successfully validate the credentials for the user khal.drogo.
2. Sample Tool: kerbrute or smartbrute
3. Result: Valid credentials for a low-privilege domain user.

• Privilege Escalation (PrintNightmare)

1. khal.drogo has limited rights. To gain a foothold on the CastelBlack server, you exploit PrintNightmare (CVE-2021-34527). This vulnerability in the Windows Print Spooler service allows any authenticated user to install a malicious print driver. You upload a driver that adds a new local admin user to the box.
2. Sample Tool: CVE-2021-34527.py exploit script
3. Result: Local SYSTEM access on CastelBlack.

• Credential Dump (DCSync Preparation)

1. Now running as SYSTEM/Admin on CastelBlack, you inspect the machine for cached credentials. You run Impacket's secretsdump to pull hashes from the SAM database and LSASS memory. You discover the NTLM hash for the built-in Administrator account, which was left in memory from a previous support session.
2. Sample Tool: impacket-secretsdump
3. Result: NTLM Hash of a Domain Admin or high-privilege account.

• Kerberoasting

1. With valid domain credentials, you pivot to the internal network. You request Kerberos Service Tickets (TGS) for Service Principal Names (SPNs) in the environment. You target the MSSQLSvc account. You take the encrypted ticket offline and crack it to reveal the plaintext password for the SQL service account.
2. Sample Tool: Rubeus or GetUserSPNs.py
3. Result: Plaintext password for the MSSQL service account.

• MSSQL Attacks

1. You use the cracked SQL credentials to authenticate directly to the Braavos SQL Server. Since the service account has sysadmin rights, you abuse the xp_cmdshell stored procedure. This feature allows you to spawn a Windows command shell directly from a SQL query, effectively giving you Remote Code Execution (RCE) on the database server.
2. Sample Tool: mssqlclient.py
3. Result: RCE on the Database Server.

• Persistence (Scheduled Task)

1. To ensure you don't lose access if the SQL password changes, you establish persistence. You create a Windows Scheduled Task on the compromised SQL server. This task is configured to execute a beacon binary every day, running as SYSTEM.
2. Sample Tool: schtasks.exe or PowerShell
3. Result: Long-term persistence.

5.2 Malware Lab Simulation (XZbot)

• Step 7: Supply Chain Pivot (XZ Backdoor)
• Simultaneously, you target the Linux infrastructure in the DMZ. You trigger the pre-implanted XZ Backdoor (CVE-2024-3094) on the xz-backdoor-dect VM. By manipulating the SSH handshake with a specific cryptographic key, you bypass authentication entirely and execute commands as root without leaving standard SSH logs.
• Tool: xzbot
• Result: Root access on Linux infrastructure via supply chain compromise.
• The attacker uses the xzbot client provided in the Ludus lab.
• From the attacker VM, the following command is run to trigger the backdoor on the vulnerable Debian host:
  xzbot --ssh-addr '10.X.X.X:22' -cmd 'setsid sh -c "echo test"' 2>&1
• This action causes the sshd process on the target to anomalously spawn a shell and execute the command as root, creating definitive proof of execution.

Phase 3: Unified Detection & Investigation with Elastic Security

This is the "Blue Team" payoff. The telemetry and alerts generated in Phase 2 are now available for analysis within the unified Elastic Security platform.

6.1 The "Powerful SIEM": Centralized Visibility & Prebuilt Detections

The power of the Elastic SIEM is not just in its ability to passively collect logs. Its power comes from the active analysis it performs on the deep, contextual data provided by Elastic Defend. The "Complete Endpoint Visibility" from Defend provides not just basic logs, but kernel-level telemetry - process creations, file modifications, network connections, and registry changes.

This rich data, all normalized to the Elastic Common Schema (ECS), feeds Elastic's extensive library of ~1500+ prebuilt, MITRE-mapped detection rules. These rules are researched, developed, and maintained by the Elastic Security Labs team, providing out-of-the-box detection value.

The Ludus range serves as the perfect validation platform for this value. The attacks executed in Phase 2 are not theoretical; they are mapped directly to specific expected artifacts ("smoking gun"). A combination of prebuilt rules and custom rules is intentionally used together in the example to alert on specific behaviors.

| Attack Step | MITRE ATT&CK | Elastic Detection Rule | Expected Artifact ("smoking gun") | | :---- | :---- | :---- | :---- | | 1. Credential Stuffing | T1110 (Brute Force) | Potential Account Brute Force (Custom) | Abnormal Auth Success (Event 4624 and ssh login) across hosts. | | 2. PrintNightmare | T1068 (Exploitation) | Unusual Print Spooler Child Process | Unusual Print Spooler service (spoolsv.exe) child processes. | | 3. Credential Dump | T1003.006 (OS Credential Dumping) | Potential Remote Credential Access via Registry | Abnormal access to the Security Account Manager (SAM) registry hive. | | 4. Kerberoasting | T1558.003 (Kerberoasting) | Suspicious Kerberos Authentication Ticket Request (Custom) | Event ID 4769 with 0x17 (RC4) encryption requested. | | 5. MSSQL Attacks | T1505.001 (SQL Stored Procedures) | Execution via MSSQL xp_cmdshell Stored Procedure | Execution via MSSQL xp_cmdshell stored procedure | | 6. Persistence | T1053.005 (Scheduled Task) | A scheduled task was created | Event ID 4698 or schtasks.exe /create. | | 7. XZ Backdoor | T1210 (Exploitation of Remote Services) | Potential Execution via SSH Backdoor | sshd spawns unusual child processes like sh or bash. |

Note: Elastic detection rules are open and transparent. You can view the logic, contribute, or raise issues directly on the(https://github.com/elastic/detection-rules).

6.2 Deep Dive: Tracing Process Chains with Event Analyzer

The two labs (GOAD and XZbot) provide a perfect opportunity to use Elastic's specialized investigation tools. The user interface of the Event Analyzer is designed to abstract the complexity of JSON logs into a cognitive model that aligns with how security analysts think: Process Chains. The interface is comprised of three primary interaction zones: the Graphical Canvas, the Detail Panel, and the Timeline integration.

What are we seeing?

The Graphical Canvas (The Process Tree)

The central view is a directed acyclic graph where:

• Nodes (Cubes): Each cube represents a distinct process execution. The visualization distinguishes between the "Anchor" event (highlighted with a blue halo) and the surrounding context.
• Edges (Lines): Lines represent the parent-child relationship. The directionality is implicit (top-down or left-right), showing the flow of execution.
• Visual Badging: Nodes are not static icons; they are dynamic indicators.
• Alert Badges: If a specific process triggered a detection rule (e.g., "Malware Detected"), a colored badge appears on the cube. This allows an analyst to instantly identify which step in the chain was flagged by the detection engine.
• User Context: Visual cues may indicate if a process changed user context (e.g., from a local user to SYSTEM), signaling privilege escalation.

The Detail Panel (Forensic Metadata)

Clicking on any node triggers the Detail Panel, typically sliding in from the right. This panel is the primary source of "What you can see" at a granular level. It exposes fields critical for verification:

• Command Line Arguments: This is arguably the single most valuable forensic artifact. The Analyzer displays the full string, exposing flags, scripts, and encoded payloads (e.g., powershell.exe -w hidden -enc Base64).
• Process Path and Hash: The full file path helps identify masquerading (e.g., svchost.exe running from C\:Temp instead of C:\Windows\System32). File hashes (MD5, SHA-1, SHA-256) are presented for cross-referencing with threat intelligence.
• Signer Information: Information about the binary's digital signature helps distinguish between trusted Microsoft binaries and unsigned malware.
• Related Event Counts: Instead of cluttering the graph with thousands of file modifications, the node displays summary statistics (e.g., "15 File Events," "3 Network Connections"). Clicking these stats usually drills down into a list view or timeline of those specific actions.

The Temporal Dimension (Time Filter)

A critical, often overlooked aspect of the Analyzer is its handling of time. Attacks can have long "dwell times." A parent process might have started weeks ago (e.g., a legitimate service), while the malicious child spawned today. The Analyzer includes a time slider that allows the analyst to expand the query window. By default, it might look at a narrow window around the alert, but expanding this allows the graph to "reach back" into the Warm or Cold data tiers to find the long-running parent process.

How does it work?

The operational capability of the Event Analyzer leverage the Elastic Common Schema (ECS). In a heterogeneous security environment, logs originate from diverse sources—Windows endpoints, Linux servers, network firewalls, and cloud service providers—each with a unique taxonomy. A CrowdStrike agent might label a process ID as TargetProcessId, while a Sysmon event uses ProcessId. Without normalization, correlating these events into a single chain is algorithmically impossible.
ECS solves this by enforcing a strict field hierarchy. The Event Analyzer relies on specific, high-fidelity ECS fields to construct the visual graph:

• process.entity_id: This is the cornerstone of the Analyzer's logic. Operating systems recycle Process IDs (PIDs). A PID of 1234 might belong to svchost.exe at 09:00 and malware.exe at 14:00. Relying on PID for long-term historical analysis introduces collisions that would corrupt the visual graph, linking unrelated events. The process.entity_id is a unique string generated by the Elastic Agent (or ECS-compliant beats) that persists uniquely in the index, ensuring that the graph represents a distinct execution instance, regardless of PID reuse.
• process.parent.entity_id: This field establishes the directed edge between nodes. By recursively querying for events where the process.entity_id of one event matches the process.parent.entity_id of another, the Analyzer reconstructs the lineage.

event.sequence: In high-velocity environments, the order of events (e.g., did the file modification happen before or after the network connection?) is critical. ECS timestamps and sequence numbers allow the Analyzer to order events chronologically within the visual node details.

6.3 Deep Dive: Reconstructing User Activity with Session Viewer

For the XZbot (Linux) attack, the Session Viewer is the superior tool. It is specifically designed for "monitoring and investigating session activity on Linux infrastructure".

When the Potential Execution via XZBackdoor alert fires, the analyst investigates the associated sshd process. The Session Viewer presents a "highly readable format inspired by the terminal". It reconstructs the attacker's session, showing the sshd process and its anomalous child process (sh).

Furthermore, it will show the exact command that was executed (sh -c setsid sh -c "usermod -aG sudo sysadmin_backup") and can even display the output of that command. This is the definitive "smoking gun", presented to the analyst in plain, human-readable text, effectively allowing them to watch the attacker's TTY session after the fact.

What are we seeing?

The user interface of the Session Viewer is explicitly designed to bridge the gap between abstract log analysis and the native terminal experience of a Linux administrator. Unlike the Event Analyzer, which focuses on malware process chains, the Session Viewer presents a time-ordered, tree-based visualization that reconstructs the linear narrative of a shell session.

The Process Tree and Timeline

The central component of the view is a Directed Acyclic Graph (DAG) displayed as a hierarchical list.

• Vertical Flow: The Session Viewer arranges processes vertically, mimicking the flow of a terminal history file but preserving hierarchy. Child processes are indented relative to their parents. This allows an analyst to immediately distinguish between a command run directly by the user (e.g., curl) and a process spawned by a script execution (e.g., curl executing inside a setup.sh script).
• Verbose Mode: A toggle allows analysts to switch between a filtered view (showing significant user activity) and "Verbose Mode." When enabled, this mode reveals typically noisy events like shell startup scripts (.bashrc execution), shell completion helpers, and forks caused by built-in commands. This is crucial for detecting persistence mechanisms hidden in profile scripts.

Visual Badging and Indicators

The UI employs a sophisticated system of badges and icons to provide immediate context without requiring the analyst to drill down into every node. These visual cues are essential for rapid triage.

Visual Indicators in Elastic Session Viewer

| Badge/Icon | Visual Appearance | Meaning | Forensic Implication | | :---- | :---- | :---- | :---- | | Exec User Change | Explicit Text Badge | The user context changed (e.g., su, sudo). | Critical for identifying privilege escalation. Shows exactly when a standard user became root. | | Process Alert | Gear Icon | A process event triggered a detection rule. | Indicates execution of malicious binaries or suspicious arguments (e.g., whoami). | | File Alert | Page Icon | A file modification triggered a rule. | Indicates tampering, persistence creation (cron/systemd), or exfiltration staging. | | Network Alert | Page Icon (Secondary) | A network event triggered a rule. | Indicates C2 communication, lateral movement, or exfiltration. | | Multiple Alerts | Combined Badge | Single event triggered multiple rule types. | High-confidence indicator of malicious activity (e.g., a process dropped a file and executed it). | | Alert Count | Numeric (e.g., (2)) | Total alerts associated with a node. | Helps prioritize which steps in the chain were most "noisy" to detection logic. |

Terminal Output View

Hovering over the Terminal Output button on a process node reveals a badge indicating the size of the captured output. Clicking this button opens the Terminal Output view, which renders the process.io.text data. This is the "Smoking Gun" feature for Linux investigations.

• Replay Capability: It allows the analyst to see exactly what the user saw. If an attacker ran cat /etc/passwd, the process tree shows the execution; the Terminal Output view shows the content of the passwd file as it was displayed to the attacker.
• Input Reconstruction: Because the viewer captures TTY I/O, it captures not just the command execution, but the typing. This can reveal backspaces, typos, and corrections (e.g., typing sdo [backspace] sudo), which are strong behavioral indicators of a human adversary rather than an automated script.

The Elastic Advantage: AI-Powered Automated Hunting

The process described in Phase 3 demonstrates a powerful, analyst-driven investigation. However, the primary advantage of using Elastic Cloud Hosted (ECH) or Elastic Serverless is the programmatic access to an integrated Generative AI stack. This stack elevates the process from manual correlation to AI-driven automated hunting.

Note: Elastic's AI features work with the out-of-the-box Elastic Managed LLMs or with third-party LLMs configured using one of the available connectors.

7.1 From Alerts to Attacks: Automated Correlation with Attack Discovery

The GOAD + XZbot labs will generate multiple discrete alerts, as shown in the table above. A junior analyst would be faced with a queue of alerts: Potential Kerberoasting, Suspicious Certificate Request, and Potential XZBackdoor and have to manually "stitch together" this complex, cross-domain attack.

This is the problem solved by Attack Discovery. This GenAI feature, available in Enterprise and Serverless tiers, "delivers fully automated threat hunting at scale". It "AI analyzes every alert to uncover hidden threats", automatically correlating the disparate signals from the Ludus lab into a single, high-fidelity "Attack" investigation.

The primary value of Attack Discovery for a forensic analyst is the compression of time. It automates the "mental stitching" that defines tier-one and tier-two analysis.

Deconstructing the "Mental Stitching"

Consider an example investigation without Attack Discovery.

1. Trigger: You see an alert: "Suspicious PowerShell Execution."
2. Query: You pivot to the host timeline.
3. Scan: You scroll back 15 minutes. You see a "File Download" event.
4. Hypothesis: "Maybe the user downloaded a bad file, which launched PowerShell."
5. Verification: You check the file name. It is invoice.js.
6. Conclusion: "Confirmed malware download."

This process takes between 10 and 30 minutes, dependingon the analyst's skill and familiarity with the environment. Attack Discovery performs this entire sequence in seconds. It looks at the PowerShell alert, sees the file download event in the related context, and presents a Discovery stating: "User executed suspicious PowerShell script likely originating from downloaded file 'invoice.js'."

This feature includes Data Persistence (results are saved for historical tracking) and Scheduling & Actions (it runs automatically and can trigger responses or subsequent Elastic Workflows), moving the SOC from a reactive to a proactive posture.

Example

In our example, as the Attack occurs, we start to see alerts. Instead of triaging the alerts individually, we leverage Attack Discovery for triage.
Compressing the mean-time-to-triage down to seconds and quickly identifying the 2 attacks.

7.2 Accelerating Triage with the AI Assistant

The Elastic Security Assistant uses generative AI to help you find, fix and understand security threats. It works directly inside Elastic Security. You interact with it through a chat interface to investigate alerts and write code.

In our example, once Attack Discovery identifies a correlated attack, we then use the AI Assistant to investigate. The assistant provides two key capabilities:

1. Natural Language Investigations: The analyst can ask plain-English questions like, "Summarize this attack", "What is the MITRE Tactic for this process?", "What is print spooler?" or “Provide some remediation suggestions.”

1. Agentic Query Validation workflow: This advanced feature allows the AI to "generate bespoke, validated ES|QL queries". An analyst can ask, "Find all network connections from the host involved in the XZbot alert", and the assistant will write, validate, and self-correct the query before presenting it, drastically lowering the skill barrier to high-end threat hunting.
   

How It Works

The Assistant connects your Elastic Stack to an LLM of your choice (e.g., GPT-5, Claude, Gemini). It uses Retrieval Augmented Generation (RAG) to fetch relevant data—logs, alerts, and internal documentation—from your environment. You can configure it to anonymize sensitive fields (PII or host/IP metadata) before sending the prompt to the model, ensuring your data remains private while the model reasons the behavioral patterns.

7.3 Intelligent Automation with Elastic Workflows

The attacks described above generate complex, multi-stage alerts. Handling these manually is slow. Elastic has addressed this by acquiring Keep, an open-source AIOps and alert management platform. In Elastic 9.3, this technology is integrated directly into Kibana in Technical Preview as Elastic Workflows.

What are Workflows?

Elastic Workflows is an automation engine built into the Elasticsearch platform. You define Workflows in YAML - what triggers them, what steps they take, what actions they perform - and the platform handles execution. A Workflow can query your environment, transform and enrich security data, branch based on conditions, call external APIs, and integrate with services like Slack, Jira, PagerDuty and more through connectors you've already configured. Workflows can also call AI agents to reason through complex investigations, then continue with response actions based on what the agent discovers. Elastic Workflows combines scripted automation with AI reasoning natively in your SIEM, where your security data already lives.

How It Works: The "Alert Aggregator & Workflow Engine"

Workflows become the middleware layer between detection and remediation, working through three primary mechanisms:

• Multi-Source Ingestion: Workflows extend beyond Elastic. Pulling in additional data for enrichment, analysis or initial triage.
• Workflow-as-Code (YAML): Workflows are defined in YAML files. This allows teams to version control their incident response procedures as code.
• The Workflow Engine: When an alert triggers in Elastic (or an external tool), the Workflow Engine executes a series of steps:

1. Enrichment: Querying an API (like VirusTotal or Active Directory) to add context.
2. Logic: Using if/else statements to determine severity.
3. Action: Sending a Slack message, creating a Jira ticket, or triggering an Elastic Defend response action.

Consider an example Alert and Action flow.

• Trigger: You connect the workflow to a specific rule, such as "Malicious Detection Alert".
• Steps: You define a sequence of actions.

1. Triage (Agentic): Pass the alert to the AI Assistant. Ask the questions: "How would we remediate and respond to the alert below?”
2. Enrich: Attach the AI Assistant's response as a note to the alert.
3. Respond: Create a case with a link to the alert note.

Example

In our example, we have alerts that trigger our Workflow - Alert Enrichment & Case Creation.
We will also directly trigger it from the Workflows UI to demonstrate the various steps.

• The Alert context is provided as an input to the Security AI Assistant
• The response is added as a note to the Security alerts
• A case is created with metadata from the Alert (timestamp, severity, rule name and alert reason).
• A link to the case is added to the case as a comment. Note: this is not shown in the GIF.

Conclusion: From Manual Setup to Continuous Emulation

This blog has provided a complete blueprint for an advanced, scalable, and most importantly, a safe simulation range.

1. We built: A complex, multi-lab range (GOAD + XZbot) was deployed with a single command using Ludus.
2. We instrumented: The entire range was seamlessly instrumented with Elastic Agent and Defend as part of the automated deployment, using the ludus_elastic_agent Ansible role.
3. We secured: The critical conflict between malware isolation and cloud-agent connectivity was solved using Ludus's granular "OPSEC" networking controls.
4. We validated: The platform's powerful SIEM capabilities were proven by validating Elastic's prebuilt, out-of-the-box detection rules against live, known-bad attacks.
5. We investigated: The specialized investigation tools, Event Analyzer and Session Viewer, were used to trace the exact attack paths on both Windows and Linux hosts.
6. We automated: The "force-multiplier" of Elastic's GenAI stack was demonstrated, with Attack Discovery automatically correlating disparate alerts into a single attack and the AI Assistant accelerating the final investigation.
7. We responded: The power of Elastic Workflows provide the brains and automation for complex response actions and remediation flows.

This architecture is not a one-off build. It is a blueprint for a continuous detection engineering pipeline. It "modernizes security operations" by empowering purple teams to tear down, rebuild, and re-test their defenses on demand, ensuring their detection posture evolves as fast as the threats do.

Take the Next Step: Enable Your Security Team

The architecture in this blog is more than a technical exercise; it's a blueprint for continuous security validation. By pairing this automated range with Elastic’s unified SIEM and XDR platform, you can move from periodic testing to a state of constant readiness.

We invite you to start your own trial, leverage this guide to test and evaluate the platform against real-world threats, and enable your security team with the tools to stay one step ahead of the adversary.

Using another SIEM?

No problem. You can leverage Elastic Serverless and augment your existing SIEM, then gain all of the insights above while using your native SIEM's underlying data. Get started with an Elastic Serverless deployment today. The Elastic AI SOC Engine (EASE) package delivers these AI-driven capabilities, enabling organizations to rapidly add powerful analytics and an AI layer on top of their existing tools before the full migration.

Appendix

Example Full Range

Note: The Kali VM VLAN is outside of the GOAD and XZ backdoor hosts to simulate a segmented network or a remote attacker. The Kali VM VLAN can be changed to 10/20 to simulate “assumed breach” or internal attack scenarios.

global_role_vars:
  ludus_elastic_fleet_server: "https://<fleet_domain>:<fleet_port>" #443 by default for cloud   ## Note on prem fleet server defaults to 8220
  ludus_elastic_agent_version: "9.2.1"
ludus:
  - vm_name: "{{ range_id }}-GOAD-DC01"
    hostname: "{{ range_id }}-DC01"
    template: win2019-server-x64-template
    vlan: 10
    ip_last_octet: 10
    ram_gb: 4
    cpus: 2
    windows:
      sysprep: true
    dns_rewrites:           # Any values in this array will be added to DNS for the range and return an A record for this VM's IP
      - sevenkingdoms.local
      - kingslanding.sevenkingdoms.local
      - kingslanding
    roles:
      - badsectorlabs.ludus_elastic_agent
    role_vars:
      ludus_elastic_enrollment_token: "<goad_policy_enrollment_token>"
  - vm_name: "{{ range_id }}-GOAD-DC02"
    hostname: "{{ range_id }}-DC02"
    template: win2019-server-x64-template
    vlan: 10
    ip_last_octet: 11
    ram_gb: 4
    cpus: 2
    windows:
      sysprep: true
    dns_rewrites:
      - winterfell.north.sevenkingdoms.local
      - north.sevenkingdoms.local
      - winterfell
    roles:
      - badsectorlabs.ludus_elastic_agent
    role_vars:
      ludus_elastic_enrollment_token: "<goad_policy_enrollment_token>"
  - vm_name: "{{ range_id }}-GOAD-DC03"
    hostname: "{{ range_id }}-DC03"
    template: win2016-server-x64-template
    vlan: 10
    ip_last_octet: 12
    ram_gb: 4
    cpus: 2
    windows:
      sysprep: true
    dns_rewrites:
      - essos.local
      - meereen.essos.local
      - meereen
    roles:
      - badsectorlabs.ludus_elastic_agent
    role_vars:
      ludus_elastic_enrollment_token: "<goad_policy_enrollment_token>"
  - vm_name: "{{ range_id }}-GOAD-SRV02"
    hostname: "{{ range_id }}-SRV02"
    template: win2019-server-x64-template
    vlan: 10
    ip_last_octet: 22
    ram_gb: 4
    cpus: 2
    windows:
      sysprep: true
    dns_rewrites:
      - castelblack.north.sevenkingdoms.local
      - castelblack
    roles:
      - badsectorlabs.ludus_elastic_agent
    role_vars:
      ludus_elastic_enrollment_token: "<goad_policy_enrollment_token>"
  - vm_name: "{{ range_id }}-GOAD-SRV03"
    hostname: "{{ range_id }}-SRV03"
    template: win2019-server-x64-template
    vlan: 10
    ip_last_octet: 23
    ram_gb: 4
    cpus: 2
    windows:
      sysprep: true
    dns_rewrites:
      - braavos.essos.local
      - braavos
    roles:
      - badsectorlabs.ludus_elastic_agent
    role_vars:
      ludus_elastic_enrollment_token: "<your_enrollment>"
  - vm_name: "{{ range_id }}-xz-backdoor-dect"
    hostname: "{{ range_id }}-xz-backdoor-dect"
    template: debian-12-x64-server-template
    vlan: 20
    ip_last_octet: 1
    ram_gb: 2
    cpus: 2
    linux:
      packages: # You can define packages to install on Linux hosts
        - ca-certificates
        - netcat-openbsd
        - net-tools
    roles:
      - badsectorlabs.ludus_xz_backdoor
      - badsectorlabs.ludus_elastic_agent
    role_vars:
      ludus_xz_backdoor_install_xzbot: true
      ludus_xz_backdoor_install_backdoor: true
      ludus_elastic_enrollment_token: "<linux_policy_enrollment_token>"
  - vm_name: "{{ range_id }}-kali"
    hostname: "{{ range_id }}-kali"
    template: kali-x64-desktop-template
    vlan: 50
    ip_last_octet: 99
    ram_gb: 8
    cpus: 4
    linux: true
    testing:
      snapshot: false # Snapshot this VM going into testing, and revert it coming out of testing. Default: true
      block_internet: false # Allow internet access for Kali, default is true
    roles:
      - badsectorlabs.ludus_xz_backdoor
      - badsectorlabs.ludus_elastic_agent
    role_vars:
      ludus_xz_backdoor_install_xzbot: true
      ludus_elastic_enrollment_token: "<linux_policy_enrollment_token>"

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Today, we are excited to announce a major expansion to our Automatic Migration feature that changes that narrative. In Elastic Security 9.3, we are introducing Automatic Migration support for QRadar detection rules (now in Tech Preview), joining our existing Splunk translation capabilities to further expedite your journey to Elastic Security. Let's take a closer look at what's supported.

Why SIEM Migration is Changing

Traditionally, organizations had to manually rewrite every rule when switching platforms. This created a significant bottleneck where security coverage was either delayed or lost during the transition. With the latest updates to Automatic Migration, MSSPs and large organizations running multiple SIEMs can now translate both Splunk and QRadar rules into Elastic-native logic automatically.

What’s Supported for Automatic Migration for QRadar

The same mapping and translation is applied as prior rule types but now with support for XML exported QRadar rules. The following rule types are supported:

• Event - focus on log and event data.
• Flow - typically related to network detection scenarios.
• Common - a combination of event and flow rules

We aren't just moving text; we are preserving the intelligence of your security operations. Reference sets are considered as part of the translation logic. We automatically put this information into lookup indexes where applicable. For more information on ES|QL lookup join syntax check out our docs. MITRE mappings are also preserved,so that upon rule install this is preserved in the migrated rule in elastic. Behind the scenes we take into account all building block rules as well. These building blocks help to contribute to the translation logic as seen in the summary tab for individual rules.

Streamlining the onboarding process

A common "chicken and egg" problem in SIEM migrations is whether to move data or rules first. Our framework provides flexibility for both:

1. Rule-First Insight: You can translate rules before onboarding data. Elastic will identify which integrations are required for those rules to work, allowing you to prioritize your data onboarding.
2. Data-First Traditionalism: If you prefer, you can onboard your log sources first and then migrate the rules to match.
3. Custom Data: For unique sources, use Automatic Import to ingest custom data in minutes.

By identifying exactly which integrations are needed before moving a single log, teams can build a precise, risk-aware roadmap for their migration project. This transparency eliminates the guesswork and helps ensure that critical visibility gaps are addressed long before you fully decommission your legacy environment.

Getting started with Automatic Migration for Detection Rules

To get started with Automatic Migration for Detection Rules, after deciding on migrating your detection rules and data, follow these three simple steps:

1. Navigate to Elastic Security’s “Get started” page and configure your AI Provider.

1. Select the drop down on the top right for QRadar. Let Elastic guide you through exporting your rules from QRadar and uploading them into Elastic Security. Elastic handles the finer details by scanning for reference sets, MITRE mappings, and then prompts you to upload them when found. MITRE mappings can only be included at the time of the initial translation so make sure to include them if you have this information.

1. Once the dashboards are uploaded, you can view their status.

• Installed: Already added to Elastic SIEM. Click View to manage and enable it.
• Translated: Ready to install. This rule was mapped to an Elastic-authored rule, or translated by Automatic Import. Click Install to install it.
• Partially translated: Part of the query could not be translated. You may need to specify an index pattern for the rule query, upload missing files, or fix broken rule syntax.
• Not translated: None of the original query could be translated.
• Failed: Translation failed. Refer to the error for details.

For more information, refer to the technical documentation.

1. After clicking View Rules you will have the ability to edit and install rules.

How Elastic’s AI features aid SOC teams

Elastic Security brings generative AI into the SOC with retrieval augmented generation (RAG) and open agentic frameworks. Automatic Migration joins the lineup of Elastic Security’s powerful AI features helping SOC teams strengthen defenses across the IT environment:

• Automatic Migration for Detection Rules complements Elastic’s deep library of prebuilt rules to broaden detection use case coverage.
• Automatic Import extends visibility and powers detection rules by onboarding custom data sources in minutes.
• Attack Discovery distills the alerts generated by detection rules to pinpoint advancing threats and suggest next steps.
• Elastic AI Assistant guides analysts through investigation and response using natural language.

Elastic’s Next Gen SIEM and XDR solution helps analysts detect earlier and respond faster.

Migrate to Elastic Security today

The days of being stuck with a legacy SIEM are over. Whether you are migrating from Splunk or QRadar, Elastic is here to ensure your transition is fast, accurate, and powerful. Interested in testing Elastic Security first? Try it free, or get in touch.

Have feedback? Tell us what you think in the Elastic Community Slack channel or on the Elastic Security forum.

“We’re excited to join Elastic and bring Keep’s capabilities to Elastic users,” said Tal Borenstein, founder and CEO of Keep. “As open source companies, we think there are natural synergies between Keep and Elastic, our products, and our communities.”

Keep is currently available as open source, enterprise software, and as a cloud service. Headquartered in Tel Aviv, Israel, the company’s products are used by SREs, engineers, and operations teams worldwide.

Going forward, Keep will integrate with Elasticsearch and Kibana and remain open source. Context from across the Keep tech stack (alerts, incidents, service topology, change management, playbooks) will benefit from the speed and scale of Elasticsearch. Meanwhile, Elastic users will benefit from the world-class AIOps and workflow automation capabilities that Keep brings to the Elastic Stack. Customers of our Observability, Security, and Search solutions will all benefit from greater AI-powered automation, enabled by Keep.

I’m delighted to welcome the Keep team to join us on our journey to help our users accelerate business outcomes through Search AI, and we look forward to sharing further updates on our plans.

Disclaimer

The release and timing of any features or functionality described in this document remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In Windows 10 (version RS4), Microsoft introduced the Windows Hypervisor Platform (WHP) API. This API exposes Microsoft's built-in hypervisor functionality to user-mode Windows applications. In 2024, the author used this API to create a personal project: a 16-bit MS-DOS emulator called DOSVisor. As mentioned in the release notes, there have always been plans to take this concept further and use it to emulate Windows applications. Elastic provides a research week (ON Week) twice per year for staff to work on personal projects, providing a great opportunity to begin working on this project. This project will be (unimaginatively) named WinVisor, inspired by its DOSVisor predecessor.

Hypervisors provide hardware-level virtualization, eliminating the need to emulate the CPU via software. This ensures that instructions are executed exactly as they would be on a physical CPU, whereas software-based emulators often behave inconsistently in edge cases.

This project aims to build a virtual environment for executing Windows x64 binaries, allowing syscalls to be logged (or hooked) and enabling memory introspection. The goal of this project is not to build a comprehensive and secure sandbox - by default, all syscalls will simply be logged and forwarded directly to the host. In its initial form, it will be trivial for code running within the virtualized guest to "escape" to the host. Safely securing a sandbox is a difficult task, and is beyond the scope of this project. The limitations will be described in further detail at the end of the article.

Despite having been available for 6 years (at the time of writing), it seems that the WHP API hasn’t been used in many public projects other than complex codebases such as QEMU and VirtualBox. One other notable project is Alex Ionescu's Simpleator - a lightweight Windows user-mode emulator that also utilizes the WHP API. This project has many of the same goals as WinVisor, although the approach for implementation is quite different. The WinVisor project aims to automate as much as possible and support simple executables (e.g. ping.exe) universally out of the box.

This article will cover the general design of the project, some of the issues that were encountered, and how they were worked through. Some features will be limited due to development time constraints, but the final product will at least be a usable proof-of-concept. Links to the source code and binaries hosted on GitHub will be provided at the end of the article.

Hypervisor basics

Hypervisors are powered by VT-x (Intel) and AMD-V (AMD) extensions. These hardware-assisted frameworks enable virtualization by allowing one or more virtual machines to run on a single physical CPU. These extensions use different instruction sets and, therefore, are not inherently compatible with each other; separate code must be written for each.

Internally, Hyper-V uses hvix64.exe for Intel support and hvax64.exe for AMD support. Microsoft's WHP API abstracts these hardware differences, allowing applications to create and manage virtual partitions regardless of the underlying CPU type. For simplicity, the following explanation will focus solely on VT-x.

VT-x adds an additional set of instructions known as VMX (Virtual Machine Extensions), containing instructions such as VMLAUNCH, which begins the execution of a VM for the first time, and VMRESUME, which re-enters the VM after a VM exit. A VM exit occurs when certain conditions are triggered by the guest, such as specific instructions, I/O port access, page faults, and other exceptions.

Central to VMX is the Virtual Machine Control Structure (VMCS), a per-VM data structure that stores the state of the guest and host contexts as well as information about the execution environment. The VMCS contains fields that define processor state, control configurations, and optional conditions that trigger transitions from the guest back to the host. VMCS fields can be read or written to using the VMREAD and VMWRITE instructions.

During a VM exit, the processor saves the guest state in the VMCS and transitions back to the host state for hypervisor intervention.

WinVisor overview

This project takes advantage of the high-level nature of the WHP API. The API exposes hypervisor functionality to user-mode and allows applications to map virtual memory from the host process directly into the guest's physical memory.

The virtual CPU operates almost exclusively in CPL3 (user-mode), except for a small bootloader that runs at CPL0 (kernel-mode) to initialize the CPU state before execution. This will be described in further detail in the Virtual CPU section.

Building up the memory space for an emulated guest environment involves mapping the target executable and all DLL dependencies, followed by populating other internal data structures such as the Process Environment Block (PEB), Thread Environment Block (TEB), KUSER_SHARED_DATA, etc.

Mapping the EXE and DLL dependencies is straightforward, but accurately maintaining internal structures, such as the PEB, is a more complex task. These structures are large, mostly undocumented, and their contents can vary between Windows versions. It would be relatively simple to populate a minimalist set of fields to execute a simple "Hello World" application, but an improved approach should be taken to provide good compatibility.

Instead of manually building up a virtual environment, WinVisor launches a suspended instance of the target process and clones the entire address space into the guest. The Import Address Table (IAT) and Thread Local Storage (TLS) data directories are temporarily removed from the PE headers in memory to stop DLL dependencies from loading and to prevent TLS callbacks from executing before reaching the entry point. The process is then resumed, allowing the usual process initialization to continue (LdrpInitializeProcess) until it reaches the entry point of the target executable, at which point the hypervisor launches and takes control. This essentially means that Windows has done all of the hard work for us, and we now have a pre-populated user-mode address space for the target executable that is ready for execution.

A new thread is then created in a suspended state, with the start address pointing to the address of a custom loader function. This function populates the IAT, executes TLS callbacks, and finally executes the original entry point of the target application. This essentially simulates what the main thread would do if the process were being executed natively. The context of this thread is then "cloned" into the virtual CPU, and execution begins under the control of the hypervisor.

Memory is paged into the guest as necessary, and syscalls are intercepted, logged, and forwarded to the host OS until the virtualized target process exits.

As the WHP API only allows memory from the current process to be mapped into the guest, the main hypervisor logic is encapsulated within a DLL that gets injected into the target process.

Virtual CPU

The WHP API provides a "friendly" wrapper around the VMX functionality described earlier, meaning that the usual steps, such as manually populating the VMCS before executing VMLAUNCH, are no longer necessary. It also exposes the functionality to user-mode, meaning a custom driver is not required. However, the virtual CPU must still be initialized appropriately via WHP prior to executing the target code. The important aspects will be described below.

Control registers

Only the CR0, CR3, and CR4 control registers are relevant for this project. CR0 and CR4 are used to enable CPU configuration options such as protected mode, paging, and PAE. CR3 contains the physical address of the PML4 paging table, which will be described in further detail in the Memory Paging section.

Model-specific registers

Model-Specific Registers (MSRs) must also be initialized to ensure the correct operation of the virtual CPU. MSR_EFER contains flags for extended features, such as enabling long mode (64-bit) and SYSCALL instructions. MSR_LSTAR contains the address of the syscall handler, and MSR_STAR contains the segment selectors for transitioning to CPL0 (and back to CPL3) during syscalls. MSR_KERNEL_GS_BASE contains the shadow base address of the GS selector.

Global descriptor table

The Global Descriptor Table (GDT) defines the segment descriptors, which essentially describe memory regions and their properties for use in protected mode.

In long mode, the GDT has limited use and is mostly a relic of the past - x64 always operates in a flat memory mode, meaning all selectors are based at 0. The only exceptions to this are the FS and GS registers, which are used for thread-specific purposes. Even in those cases, their base addresses are not defined by the GDT. Instead, MSRs (such as MSR_KERNEL_GS_BASE described above) are used to store the base address.

Despite this obsolescence, the GDT is still an important part of the x64 model. For example, the current privilege level is defined by the CS (Code Segment) selector.

Task state segment

In long mode, the Task State Segment (TSS) is simply used to load the stack pointer when transitioning from a lower privilege level to a higher one. As this emulator operates almost exclusively in CPL3, except for the initial bootloader and interrupt handlers, only a single page is allocated for the CPL0 stack. The TSS is stored as a special system entry within the GDT and occupies two slots.

Interrupt descriptor table

The Interrupt Descriptor Table (IDT) contains information about each type of interrupt, such as the handler addresses. This will be described in further detail in the Interrupt Handling section.

Bootloader

Most of the CPU fields mentioned above can be initialized using WHP wrapper functions, but support for certain fields (e.g. XCR0) only arrived in later versions of the WHP API (Windows 10 RS5). For completeness, the project includes a small “bootloader”, which runs at CPL0 upon startup and manually initializes the final parts of the CPU prior to executing the target code. Unlike a physical CPU, which would start in 16-bit real mode, the virtual CPU has already been initialized to run in long-mode (64-bit), making the boot process slightly more straightforward.

The following steps are performed by the bootloader:

1. Load the GDT using the LGDT instruction. The source operand for this instruction specifies a 10-byte memory block which contains the base address and limit (size) of the table that was populated earlier.

2. Load the IDT using the LIDT instruction. The source operand for this instruction uses the same format as LGDT described above.

3. Set the TSS selector index into the task register using the LTR instruction. As mentioned above, the TSS descriptor exists as a special entry within the GDT (at 0x40 in this case).

4. The XCR0 register can be set using the XSETBV instruction. This is an additional control register which is used for optional features such as AVX. The native process executes XGETBV to get the host value, which is then copied into the guest via XSETBV in the bootloader.

This is an important step because DLL dependencies that have already been loaded may have set global flags during their initialization process. For example, ucrtbase.dll checks if the CPU supports AVX via the CPUID instruction on startup and, if so, sets a global flag to allow the CRT to use AVX instructions for optimization reasons. If the virtual CPU attempts to execute these AVX instructions without explicitly enabling them in XCR0 first, an undefined instruction exception will be raised.

1. Manually update DS, ES, and GS data segment selectors to their CPL3 equivalents (0x2B). Execute the SWAPGS instruction to load the TEB base address from MSR_KERNEL_GS_BASE.

2. Finally, use the SYSRET instruction to transition into CPL3. Prior to the SYSRET instruction, RCX is set to a placeholder address (CPL3 entry point), and R11 is set to the initial CPL3 RFLAGS value (0x202). The SYSRET instruction automatically switches the CS and SS segment selectors to their CPL3 equivalents from MSR_STAR.

When the SYSRET instruction executes, a page fault will be raised due to the invalid placeholder address in RIP. The emulator will catch this page fault and recognize it as a “special” address. The initial CPL3 register values will then be copied into the virtual CPU, RIP is updated to point to a custom user-mode loader function, and execution resumes. This function loads all DLL dependencies for the target executable, populates the IAT table, executes TLS callbacks, and then executes the original entry point. The import table and TLS callbacks are handled at this stage, rather than earlier on, to ensure their code is executed within the virtualized environment.

Memory paging

All memory management for the guest must be handled manually. This means a paging table must be populated and maintained, allowing the virtual CPU to translate a virtual address to a physical address.

Virtual address translation

For those who are not familiar with paging in x64, the paging table has four levels: PML4, PDPT, PD, and PT. For any given virtual address, the CPU walks through each layer of the table, eventually reaching the target physical address. Modern CPUs also support 5-level paging (in case the 256TB of addressable memory offered by 4-level paging isn't enough!), but this is irrelevant for the purposes of this project.

The following image illustrates the format of a sample virtual address:

Using the example above, the CPU would calculate the physical page corresponding to the virtual address 0x7FFB7D030D10 via the following table entries: PML4[0xFF] -> PDPT[0x1ED] -> PD[0x1E8] -> PT[0x30]. Finally, the offset (0xD10) will be added to this physical page to calculate the exact address.

Bits 48 - 63 within a virtual address are unused in 4-level paging and are essentially sign-extended to match bit 47.

The CR3 control register contains the physical address of the base PML4 table. When paging is enabled (mandatory in long-mode), all other addresses within the context of the CPU refer to virtual addresses.

Page faults

When the guest attempts to access memory, the virtual CPU will raise a page fault exception if the requested page isn't already present in the paging table. This will trigger a VM Exit event and pass control back to the host. When this occurs, the CR2 control register contains the requested virtual address, although the WHP API already provides this value within the VM Exit context data. The host can then map the requested page into memory (if possible) and resume execution or throw an error if the target address is invalid.

Host/guest memory mirroring

As mentioned earlier, the emulator creates a child process, and all virtual memory within that process will be mapped directly into the guest using the same address layout. The Hypervisor Platform API allows us to map virtual memory from the host user-mode process directly into the physical memory of the guest. The paging table will then map virtual addresses to the corresponding physical pages.

Instead of mapping the entire address space of the process upfront, a fixed number of physical pages are allocated for the guest. The emulator contains a very basic memory manager, and pages are mapped "on demand." When a page fault occurs, the requested page will be paged in, and execution resumes. If all page "slots" are full, the oldest entry is swapped out to make room for the new one.

In addition to using a fixed number of currently mapped pages, the emulator also uses a fixed-size page table. The size of the page table is determined by calculating the maximum possible number of tables for the amount of mapped page entries. This model results in a simple and consistent physical memory layout but comes at the cost of efficiency. In fact, the paging tables take up more space than the actual page entries.

There is a single PML4 table, and in the worst-case scenario, each mapped page entry will reference unique PDPT/PD/PT tables. As each table is 4096 bytes, the total page table size can be calculated using the following formula:

PAGE_TABLE_SIZE = 4096 + (MAXIMUM_MAPPED_PAGES * 4096 * 3)

By default, the emulator allows for 256 pages to be mapped at any one time (1024KB in total). Using the formula above, we can calculate that this will require 3076KB for the paging table, as illustrated below:

In practice, many of the page table entries will be shared, and a lot of the space allocated for the paging tables will remain unused. However, as this emulator functions well even with a small number of pages, this level of overhead is not a major concern.

The CPU maintains a hardware-level cache for the paging table known as the Translation Lookaside Buffer (TLB). When translating a virtual address to a physical address, the CPU will first check the TLB. If a matching entry is not found in the cache (known as a “TLB miss”), the paging tables will be read instead. For this reason, it is important to flush the TLB cache whenever the paging tables have been rebuilt to prevent it from falling out of sync. The simplest way to flush the entire TLB is to reset the CR3 register value.

Syscall handling

As the target program executes, any system calls that occur within the guest must be handled by the host. This emulator handles both SYSCALL instructions and legacy (interrupt-based) syscalls. SYSENTER is not used in long-mode and, therefore, is not supported by WinVisor.

Fast syscall (SYSCALL)

When a SYSCALL instruction executes, the CPU transitions to CPL0 and loads RIP from MSR_LSTAR. In the Windows kernel, this would point to KiSystemCall64. SYSCALL instructions won't inherently trigger a VM Exit event, but the emulator sets MSR_LSTAR to a reserved placeholder address — 0xFFFF800000000000 in this case. When a SYSCALL instruction is executed, a page fault will be raised when RIP is set to this address, and the call can be intercepted. This placeholder is a kernel address in Windows and won't cause any conflicts with the user-mode address space.

Unlike legacy syscalls, the SYSCALL instruction doesn't swap the RSP value during the transition to CPL0, so the user-mode stack pointer can be retrieved directly from RSP.

Legacy syscalls (INT 2E)

Legacy interrupt-based syscalls are slower and have more overhead than the SYSCALL instruction, but despite this, they are still supported by Windows. As the emulator already contains a framework for handling interrupts, adding support for legacy syscalls is very simple. When a legacy syscall interrupt is caught, it can be forwarded to the “common” syscall handler after some minor translations — specifically, retrieving the stored user-mode RSP value from the CPL0 stack.

Syscall forwarding

After the emulator creates the "main thread" whose context gets cloned into the virtual CPU, this native thread is reused as a proxy to forward syscalls to the host. Reusing the same thread maintains consistency for the TEB and any kernel state between the guest and the host. Win32k, in particular, relies on many thread-specific states, which should be reflected in the emulator.

When a syscall occurs, either by a SYSCALL instruction or a legacy interrupt, the emulator intercepts it and transfers it to a universal handler function. The syscall number is stored in the RAX register, and the first four parameter values are stored in R10, RDX, R8, and R9, respectively. R10 is used for the first parameter instead of the usual RCX register because the SYSCALL instruction overwrites RCX with the return address. The legacy syscall handler in Windows (KiSystemService) also uses R10 for compatibility, so it doesn’t need to be handled differently in the emulator. The remaining parameters are retrieved from the stack.

We don’t know the exact number of parameters expected for any given syscall number, but luckily, this doesn’t matter. We can simply use a fixed amount, and as long as the number of supplied parameters is greater than or equal to the actual number, the syscall will function correctly. A simple assembly stub will be dynamically created, populating all of the parameters, executing the target syscall, and returning cleanly.

Testing showed that the maximum number of parameters currently used by Windows syscalls is 17 (NtAccessCheckByTypeResultListAndAuditAlarmByHandle, NtCreateTokenEx, and NtUserCreateWindowEx). WinVisor uses 32 as the maximum number of parameters to allow for potential future expansion.

After executing the syscall on the host, the return value is copied to RAX in the guest. RIP is then transferred to a SYSRET instruction (or IRETQ for legacy syscalls) before resuming the virtual CPU for a seamless transition back to user-mode.

Syscall logging

By default, the emulator simply forwards guest syscalls to the host and logs them to the console. However, some additional steps are necessary to convert the raw syscalls into a readable format.

The first step is to convert the syscall number to a name. Syscall numbers are made up of multiple parts: bits 12 - 13 contain the system service table index (0 for ntoskrnl, 1 for win32k), and bits 0 - 11 contain the syscall index within the table. This information allows us to perform a reverse-lookup within the corresponding user-mode module (ntdll / win32u) to resolve the original syscall name.

The next step is to determine the number of parameter values to display for each syscall. As mentioned above, the emulator passes 32 parameter values to each syscall, even if most of them are not used. However, logging all 32 values for each syscall wouldn't be ideal for readability reasons. For example, a simple NtClose(0x100) call would be printed as NtClose(0x100, xxx, xxx, xxx, xxx, xxx, xxx, xxx, xxx, ...). As mentioned earlier, there is no simple way to automatically determine the exact number of parameters for each syscall, but there is a trick that we can use to estimate it with high accuracy.

This trick relies on the 32-bit system libraries used by WoW64. These libraries use the stdcall calling convention, which means the caller pushes all parameters onto the stack, and they are cleaned internally by the callee before returning. In contrast, native x64 code places the first 4 parameters into registers, and the caller is responsible for managing the stack.

For example, the NtClose function in the WoW64 version of ntdll.dll ends with the RET 4 instruction. This pops an additional 4-bytes off the stack after the return address, which implies that the function takes one parameter. If the function used RET 8, this would suggest that it takes 2 parameters, and so on.

Even though the emulator runs as a 64-bit process, we can still load the 32-bit copies of ntdll.dll and win32u.dll into memory - either manually or mapped using SEC_IMAGE. A custom version of GetProcAddress must be written to resolve the WoW64 export addresses, but this is a trivial task. From here, we can automatically find the corresponding WoW64 export for each syscall, scan for the RET instruction to calculate the number of parameters, and store the value in a lookup table.

This method is not perfect, and there are a number of ways that this could fail:

• A small number of native syscalls don't exist in WoW64, such as NtUserSetWindowLongPtr.
• If a 32-bit function contains a 64-bit parameter, it will be split into 2x 32-bit parameters internally, whereas the corresponding 64-bit function would only require a single parameter for the same value.
• The WoW64 syscall stub functions within Windows could change in such a way that causes the existing RET instruction search to fail.

Despite these pitfalls, the results will be accurate for the vast majority of syscalls without having to rely on hardcoded values. In addition, these values are only used for logging purposes and won't affect anything else, so minor inaccuracies are acceptable in this context. If a failure is detected, it will revert back to displaying the maximum number of parameter values.

Syscall hooking

If this project were being used for sandboxing purposes, blindly forwarding all syscalls to the host would be undesirable for obvious reasons. The emulator contains a framework that allows specific syscalls to be easily hooked if necessary.

By default, only NtTerminateThread and NtTerminateProcess are hooked to catch the guest process exiting.

Interrupt handling

Interrupts are defined by the IDT, which is populated before the virtual CPU execution begins. When an interrupt occurs, the current CPU state is pushed onto the CPL0 stack (SS, RSP, RFLAGS, CS, RIP), and RIP is set to the target handler function.

As with MSR_LSTAR for the SYSCALL handler, the emulator populates all interrupt handler addresses with placeholder values (0xFFFFA00000000000 - 0xFFFFA000000000FF). When an interrupt occurs, a page fault will occur within this range, which we can catch. The interrupt index can be extracted from the lowest 8-bits of the target address (e.g., 0xFFFFA00000000003 is INT 3), and the host can handle it as necessary.

At present, the emulator only handles INT 1 (single-step), INT 3 (breakpoint), and INT 2E (legacy syscall). If any other interrupt is caught, the emulator will exit with an error.

When an interrupt has been handled, RIP is transferred to an IRETQ instruction, which returns to user-mode cleanly. Some types of interrupts push an additional "error code" value onto the stack - if this is the case, it must be popped prior to the IRETQ instruction to avoid stack corruption. The interrupt handler framework within this emulator contains an optional flag to handle this transparently.

Hypervisor shared page bug

Windows 10 introduced a new type of shared page which is located close to KUSER_SHARED_DATA. This page is used by timing-related functions such as RtlQueryPerformanceCounter and RtlGetMultiTimePrecise.

The exact address of this page can be retrieved with NtQuerySystemInformation, using the SystemHypervisorSharedPageInformation information class. The LdrpInitializeProcess function stores the address of this page in a global variable (RtlpHypervisorSharedUserVa) during process startup.

The WHP API seems to contain a bug that causes the WHvRunVirtualProcessor function to get stuck in an infinite loop if this shared page is mapped into the guest and the virtual CPU attempts to read from it.

Time constraints limited the ability to fully investigate this; however, a simple workaround was implemented. The emulator patches the NtQuerySystemInformation function within the target process and forces it to return STATUS_INVALID_INFO_CLASS for SystemHypervisorSharedPageInformation requests. This causes the ntdll code to fall back to traditional methods.

Demos

Some examples of common Windows executables being emulated under this virtualized environment below:

Limitations

The emulator has several limitations that make it unsafe to use as a secure sandbox in its current form.

Safety issues

There are several ways to "escape" the VM, such as simply creating a new process/thread, scheduling asynchronous procedure calls (APCs), etc.

Windows GUI-related syscalls can also make nested calls directly back into user-mode from the kernel, which would currently bypass the hypervisor layer. For this reason, GUI executables such as notepad.exe are only partially virtualized when run under WinVisor.

To demonstrate this, WinVisor includes an -nx command-line switch to the emulator. This forces the entire target EXE image to be marked as non-executable in memory prior to starting the virtual CPU, causing the process to crash if the host process attempts to execute any of the code natively. However, this is still unsafe to rely on — the target application could make the region executable again or simply allocate executable memory elsewhere.

As the WinVisor DLL is injected into the target process, it exists within the same virtual address space as the target executable. This means the code running under the virtual CPU is able to directly access the memory within the host hypervisor module, which could potentially corrupt it.

Non-executable guest memory

While the virtual CPU is set up to support NX, all memory regions are currently mirrored into the guest with full RWX access.

Single-thread only

The emulator currently only supports virtualizing a single thread. If the target executable creates additional threads, they will be executed natively. To support multiple threads, a pseudo-scheduler could be developed to handle this in the future.

The Windows parallel loader is disabled to ensure all module dependencies are loaded by a single thread.

Software exceptions

Virtualized software exceptions are not currently supported. If an exception occurs, the system will call the KiUserExceptionDispatcher function natively as usual.

Conclusion

As seen above, the emulator performs well with a wide range of executables in its current form. While it is currently effective for logging syscalls and interrupts, a lot of further work would be required to make it safe to use for malware analysis purposes. Despite this, the project provides an effective framework for future development.

Project links

https://github.com/x86matthew/WinVisor

The author can be found on X at @x86matthew.

In the ever-evolving landscape of cloud computing, maintaining robust security while ensuring compliance is a critical challenge for organizations of all sizes. As businesses increasingly adopt the cloud, the complexity of managing and securing data across various platforms grows exponentially.

Amazon Bedrock, with its powerful foundation of machine learning and AI services, offers a scalable, secure environment for organizations to develop and deploy intelligent applications. However, to fully harness the potential of these innovations, it’s essential to implement a streamlined approach to security and compliance.

Integrating Elastic with Amazon Bedrock can significantly enhance security monitoring and compliance management within your cloud environment. This integration leverages Elastic’s search, observability, and security capabilities to optimize how you manage and secure applications and data hosted on Amazon Bedrock.

Elastic’s security information and event management (SIEM) capabilities can be used to analyze logs and monitor events generated by applications running on Amazon Bedrock. This allows for the detection of potential security threats in real-time and automated response actions to mitigate risks.

This article will guide you through the process of setting up Amazon Bedrock integration and enabling our prebuilt detection rules to streamline your security operations. We will cover the following key aspects:

1. Prerequisites for Elastic Amazon Bedrock Integration: Understanding the core requirements for setting up Elastic Amazon Bedrock integration for cloud security.
2. Setting Up Amazon Bedrock Integration: Step-by-step instructions to set up Amazon Bedrock in your existing AWS infrastructure.
3. Enabling Prebuilt Security Rules: How to leverage prebuilt rules to detect high-confidence policy violations and other security threats.
4. Exploring High-Confidence Misconduct Blocks Detection: An in-depth look at a specific prebuilt rule designed to detect high-confidence misconduct blocks within Amazon Bedrocklogs.
5. Demonstrate an Exploit Case Scenario for Amazon Bedrock: Using a sample python script to simulate interactions with an Amazon Bedrock model for testing exploit scenarios that could trigger Elastic prebuilt detection rules.

Prerequisites for Elastic Amazon Bedrock Integration

Elastic Integration for Amazon Bedrock

The Amazon Bedrock integration collects Amazon Bedrock model invocation logs and runtime metrics with Elastic Agent. For a deeper dive on the integration, documentation can be found in our documentation.

Below are the list of prerequisites to have a complete and successful configuration of Amazon Bedrock Elastic Integration:

• AWS Account Setup
• Elastic Cloud Requirements
• Terraform (Optional)

AWS Account Setup

• Active AWS Account: Ensure you have an active AWS account with the appropriate permissions to deploy and manage resources on Amazon Bedrock.
• Amazon Bedrock Setup: Confirm that Amazon Bedrock is correctly configured and operational within your AWS environment. This includes setting up AI models, datasets, and other resources necessary for your applications. Refer to Getting started with Amazon Bedrock for additional information on the setup.
• IAM Roles and Permissions: Create or configure Identity and Access Management (IAM) roles with the necessary permissions to allow Elastic to access Amazon Bedrock resources. These roles should have sufficient privileges to read logs, metrics, and traces from AWS services. Additional details of the requirements can be found in our AWS documentation.

Elastic Cloud Requirements

| Version | 0.7.0 (Beta) | | :---- | :---- | | Compatible Kibana version(s) | 8.13.0 or higher for integration version 0.2.0 and above. Minimum Kibana Version 8.12.0 | | Supported Serverless project types | Security Observability | | Subscription level | Basic | | Level of support | Elastic |

Note: Since the integration is in Beta Release Stage, please enable Display Beta Integrations in the browse integration section of the Management pane in your Elastic stack.

Terraform

Terraform is an open source infrastructure-as-code (IaC) tool created by HashiCorp that allows you to define, provision, and manage cloud and on-premises infrastructure in a consistent and repeatable way.

This is an optional step, but good to have as the next sections of the article we use this tool to set up the required AWS Infrastructure. Deep dive on installation and docs can be found here.

Setting Up Amazon Bedrock Integration

In this section of the article, we will walk through the steps to set up Amazon Bedrock integration with Elastic in two parts:

1. Setting Up AWS Infrastructure with Terraform: In this section, we'll walk through the steps to set up an AWS infrastructure using Terraform. We'll create an S3 bucket, an EC2 instance with the necessary IAM roles and policies to access the S3 bucket, and configure security groups to allow SSH access. This setup is ideal for scenarios where you need an EC2 instance to interact with S3, such as for data processing or storage.
2. Elastic Agent and Integration Setup: In this section, we'll walk through the steps to install Elastic Agent on the AWS EC2 instance and Configure the Amazon Bedrock Integration.

Setting Up AWS Infrastructure with Terraform

The high-level configuration process will involve the following steps:

1. Configuring providers.tf
2. Configuring variables.tf
3. Configuring outputs.tf
4. Configuring main.tf

The providers.tf file typically contains the configuration for any Terraform providers you are using in your project. In our example, it includes the configuration for the AWS provider. Here is the sample content of our providers.tf file. The profile mentioned in the providers.tf should be configured in the user’s space of the AWS credentials file (~/.aws/credentials). Refer to Configuration and credential file settings - AWS Command Line Interface, which is also highlighted in the credential section of Elastic’s AWS documentation.

The variables.tf file contains the variable definitions used throughout your Terraform configuration. For our scenario, it includes the definition for the aws_region and resource_labels. Here is the sample content of our variables.tf file.

The outputs.tf file typically contains the output definitions for your Terraform configuration. These outputs can be used to display useful information after your infrastructure is provisioned. Here is the sample content of our outputs.tf file

The main.tf file typically contains the collection of all of these resources such as data sources, S3 bucket and bucket policy, Amazon Bedrock Model Invocation Log configuration, SQS Queue configuration, IAM Role and Policies required by the EC2 instance that would install Elastic Agent and stream logs and Amazon Bedrock Guardrail configuration. Here is the sample content of our main.tf file.

Once the main.tf is configured according to the requirements we can then initialize, plan and apply the terraform configuration.

terraform init // initializes the directory and sets up state files in backend
terraform plan // command creates an execution plan
terraform apply // command applies the configuration aka execution step

To tear down the infrastructure that terraform has previously created one can use the terraform destroy command.

Once the infrastructure setup is completed, necessary resource identifiers are provided via outputs.tf. We can conduct a basic verification of the infrastructure created using the following steps:

1. Verify the S3 Bucket created from the Terraform, one can either use aws cli command reference list-buckets — AWS CLI 1.34.10 Command Reference or navigate via AWS console to verify the same. 2. Verify the SQS Queue created from the terraform, one can either use aws cli command reference list-queues — AWS CLI 1.34.10 Command Reference or navigate via AWS console to verify the same.
2. Verify the EC2 Instance created from the AWS console and connect to the ec2-instance via Connect using EC2 Instance Connect - Amazon Elastic Compute Cloud and run aws s3 ls example-bucket-name to check if the instance has access to the created S3 bucket.
3. Verify the Amazon Bedrock Guardrail created from the Terraform, once can either use Amazon Bedrock API ListGuardrails - Amazon Bedrock or navigate via AWS console to verify the same.

Setting Up Elastic Agent and Integration Setup

To install Elastic Agent on the AWS EC2 instance and configure the Amazon Bedrock integration, create an agent policy using the guided steps in Elastic Agent policies | Fleet and Elastic Agent Guide [8.15]. Then log into to the ec2-instance created in the infrastructure setup steps via Connect using EC2 Instance Connect - Amazon Elastic Compute Cloud, and install the elastic agent using the guided steps in Install Elastic Agents | Fleet and Elastic Agent Guide [8.15]. During the agent installation, remember to select the agent policy created at the beginning of this setup process and use the relevant agent installation method depending on the instance created. Finally, ensure the agent is properly configured and there is incoming data from the agent.

To configure the Amazon Bedrock integration in the newly-created policy, add the Amazon Bedrock integration using the guided steps: Add an Elastic Agent integration to a policy. Enable Beta Integrations to use Amazon Bedrock integration as displayed in the image below.

Configure the Integration with AWS Access Keys to access the AWS account where Amazon Bedrock is configured. Use the Collect Logs from S3 bucket and specify the Bucket ARN created in the setup step. Please note to use either the S3 Bucket or the SQS Queue URL during the setup and not both. Add this integration to the existing policy where the ec2-instance is configured.

Verify Amazon Bedrock Model Invocation Log Ingestions

Once the Elastic Agent and integration setup is completed, we can conduct a basic verification of the integration to determine if the logs are being ingested as expected by using the following example API call:

aws bedrock-runtime converse \
--model-id "anthropic.claude-3-5-sonnet-20240620-v1:0" \
--messages '[{"role":"user","content":[{"text":"Hello "}]}]' \
--inference-config '{"maxTokens":2000,"stopSequences":[],"temperature":1,"topP":0.999}' \
--additional-model-request-fields '{"top_k":250}' \
--region us-east-1

The example API call assumes a working setup with aws cli and there is access for the foundational model Anthropic Claude Messages API - Amazon Bedrock. If the user does not have access to the model one can simply request access for models from the model-access page as suggested in Access Amazon Bedrock foundation models, or we can optionally change the API call to any existing model the user can access.

On successful execution of the above API call, the Amazon Bedrock Model invocation logs are populated and in Kibana logs-aws_bedrock.invocation-default should be populated with those invocation logs. We can use the following simple ES|QL query to return recently ingested events.

from logs-aws_bedrock.invocation-* | LIMIT 10

Enable Prebuilt Detection Rules

To enable prebuilt detection rules, first login to the elastic instance and from the left pane navigation navigate to Security → Rules → Detection rules (SIEM). Filter for “Data Source: Amazon Bedrock” from the tags section.

Enable the available prebuilt rules. For prebuilt rules, the Setup information contains a helper guide to setup AWS Guardrails for Amazon Bedrock, which is accomplished in the Setting Up AWS Infrastructure with Terraform step if the example is followed correctly and the terraform has the Amazon Bedrock Guardrail configuration. Please note this setup is vital for some of the rules to generate alerts–we need to ensure the guardrail is set up accordingly if skipped in the infrastructure setup stage.

Exploring High-Confidence Misconduct Blocks Detection

Let’s simulate a real world scenario in which a user queries a topic denied to the Amazon Bedrock model. Navigate to the Amazon Bedrock section in the Amazon UI Console, and use the left navigation pane to navigate to the Guardrails subsection under Safeguards. Use the sample guardrail created during our setup instructions for this exercise, and use the test option to run a model invocation with the guardrails and query the denied topic configured.

Repeat the query at least 6 times as the prebuilt rule is designed to alert on greater than 5 high confidence blocks. When the Alert schedule runs, we can see an alert populate for Unusual High Confidence Misconduct Blocks Detected.

Demonstrate an Exploit Case Scenario for Amazon Bedrock

To simulate an Amazon Bedrock Security bypass, we need an exploit simulation script to interact with Amazon Bedrock models. The exploit script example we provide simulates the following attack pattern:

• Attempts multiple successive requests to use denied model resources within AWS Bedrock
• Generates multiple successive validation exception errors within Amazon Bedrock
• User consistently generates high input token counts, submits numerous requests, and receives large responses that mimic patterns of resource exhaustion
• Combines repeated high-confidence 'BLOCKED' actions coupled with specific violation codes such as 'MISCONDUCT', indicating persistent misuse or attempts to probe the model's ethical boundaries

class BedrockModelSimulator:
   def __init__(self, profile_name, region_name):
       // Create a Boto3 Session Client for Ineration 
   def generate_args_invoke_model(self, model_id, user_message, tokens):     // Generate Model Invocation parameters
       guardrail_id = <<GUARDRAIL_ID>>
       guardrail_version = <<GUARDRAIL_VERSION>>

       guardrail_config = {
           "guardrailIdentifier": guardrail_id,
           "guardrailVersion": guardrail_version,
           "trace": "enabled"
       }
       conversation = [
           {
               "role": "user",
               "content": [{"text": user_message}],
           }
       ]
       inference_config = {"maxTokens": tokens, "temperature": 0.7, "topP": 1}
       additional_model_request_fields = {}

       kwargs = {
           "modelId": model_id,
           "messages": conversation,
           "inferenceConfig": inference_config,
           "additionalModelRequestFields": additional_model_request_fields
        "guardrailConfig" : guardrail_config
       }
       return kwargs

   def invoke_model(self, invocation_arguments):
       for  in range(count):
           try:
               // Invoke Model With right invocation_arguments
           except ClientError as e:
               // Error meesage

def main():
   profile_name = <<AWS Profile>>
   region_name = 'us-east-1'
   denied_model_id = // Use a denied model   
   denied_model_user_message = // Sample Message 
   available_model_id = // Use an available model  
   validation_exception_user_message = // Sample Message 
   resource_exploit_user_message = // A very big message for resource exhuastion
   denied_topic_user_message = // Sample Message that can query denied topic configured
   simulator = BedrockModelSimulator(profile_name, region_name)
   denied_model_invocation_arguments = simulator.generate_args_invoke_model(denied_model_id, denied_model_user_message, 200)
   simulator.invoke_model(denied_model_invocation_arguments)
   validation_exception_invocation_arguments = simulator.generate_args_invoke_model(available_model_id, validation_exception_user_message, 6000)
   simulator.invoke_model(validation_exception_invocation_arguments)
   resource_exhaustion_invocation_arguments = simulator.generate_args_invoke_available_model(available_model_id, resource_exploit_user_message, 4096)
   simulator.invoke_model(resource_exhaustion_invocation_arguments)
   denied_topic_invocation_arguments = simulator.generate_args_invoke_available_model_guardrail(available_model_id, denied_topic_user_message, 4096)
   simulator.invoke_model(denied_topic_invocation_arguments)

if __name__ == "__main__":
   main()

Note: The GUARDRAIL_ID and GUARDRAIL_VERSION can be found in outputs.tf

When executed in a controlled environment, the provided script simulates an exploit scenario that would generate detection alerts in Elastic Security. When analyzing these alerts using the Elastic Attack Discovery feature, the script creates attack chains that show the relationships between various alerts, giving analysts a clear understanding of how multiple alerts might be part of a larger attack.

Conclusion

Integrating Elastic with Amazon Bedrock empowers organizations to maintain a secure and compliant cloud environment while maximizing the benefits of AI and machine learning. By leveraging Elastic’s advanced security and observability tools, businesses can proactively detect threats, automate compliance reporting, and gain deeper insights into their cloud operations. Increasingly, enterprises rely on opaque data sources and technologies to reveal the most serious threats-- our commitment to transparent security is evident in our open artifacts, integrations, and source code.

Organizations that use threat indicators or observables consume, create, and/or (ideally) publish threat data. This data can be used internally or externally as information or intelligence to inform decision-making and event prioritization.

While there are several formats for this information to be structured into, the de facto industry standard is Structured Threat Information Expression (STIX). STIX is managed by the OASIS Cyber Threat Intelligence Technical Committee and enables organizations to share threat data in a standard and machine-readable format.

At Elastic, we developed the Elastic Common Schema (ECS) as a data normalization capability. “[ECS] is an open source specification, developed with support from the Elastic user community. ECS defines a common set of fields for storing event data in Elasticsearch, such as logs and metrics.” In April of 2023, Elastic contributed ECS to the OpenTelemetry Semantic Conventions (OTel) as a commitment to the joint development of an open schema.

The security community shares threat data in the STIX format, so to store that data in Elasticsearch for analysis and threat detection [1] [2] [3] [4], we created a tool that converts STIX documents into ECS and outputs the threat data either as a file or directly into Elasticsearch indices. If this was a challenge for us, it was a challenge for others - therefore, we decided to release a version of the tool.

This tool uses the Elastic License 2.0 and is available for download here.

Getting started

This project will take a STIX 2.x formatted JSON document and create an ECS version. There are three output options: STDOUT as JSON, an NDJSON file, and/or directly to an Elasticsearch cluster.

Prerequisites

The STIX 2 ECS project requires Python 3.10+ and the stix2, Elasticsearch, and getpass modules.

If exporting to Elasticsearch, you will need the host information and authentication credentials. API authentication is not yet implemented.

Setup

Create a virtual environment and install the required prerequisites.

git clone https://github.com/elastic/labs-releases.git
cd tools/stix2ecs
python -m venv /path/to/virtual/environments/stix2ecs
source /path/to/virtual/environments/stix2ecs/bin/activate
python -m pip install -r requirements.txt

Operation

The input is a STIX 2.x JSON document (or a folder of JSON documents); the output defaults to STDOUT, with an option to create an NDJSON file and/or send to an Elasticsearch cluster.

stix_to_ecs.py [-h] -i INPUT [-o OUTPUT] [-e] [--index INDEX] [--url URL] \
[--user USER] [-p PROVIDER] [-r]

By default, the ECS file is named the same as the STIX file input but with .ecs.ndjson appended.

Arguments

The script has several arguments, the only mandatory field is -i for the input. By default, the script will output the NDJSON document to STDOUT.

| Option | Description | | - | - | | -h | displays the help menu | | -i | specifies the input STIX document (mandatory) | | -o | specifies the output ECS document (optional) | | -p | defines the ECS provider field (optional) | | -r | recursive mode to convert multiple STIX documents (optional) | | -e | specifies the Elasticsearch output mode (optional) | | --index | defines the Elasticsearch Index, requires -e (optional) | | --url | defines the Elasticsearch URL, requires -e (optional) | | --user | defines the Elasticsearch username, requires -e (optional) |

Examples

There are two sample files located in the test-inputs/ directory. One is from CISA (Cybersecurity & Infrastructure Security Agency), and one is from OpenCTI (an open source threat intelligence platform).

STIX file input to STDOUT

This will output the STIX document to STDOUT in ECS format.

python stix_to_ecs.py -i test-inputs/cisa_sample_stix.json | jq

[
  {
    "threat": {
      "indicator": {
        "file": {
          "name": "123.ps1",
          "hash": {
            "sha256": "ED5D694D561C97B4D70EFE934936286FE562ADDF7D6836F795B336D9791A5C44"
          }
        },
        "type": "file",
        "description": "Simple indicator of observable {ED5D694D561C97B4D70EFE934936286FE562ADDF7D6836F795B336D9791A5C44}",
        "first_seen": "2023-11-21T18:57:25.000Z",
        "provider": "identity--b3bca3c2-1f3d-4b54-b44f-dac42c3a8f01",
        "modified_at": "2023-11-21T18:57:25.000Z",
        "marking": {
          "tlp": "clear"
        }
      }
    }
  },
...

STIX file input to ECS file output

This will create a folder called ecs in the present directory and write the ECS file there.

python python stix_to_ecs.py -i test-inputs/cisa_sample_stix.json -o ecs

cat ecs/cisa_sample_stix.ecs.ndjson | jq
{
  "threat": {
    "indicator": {
      "file": {
        "name": "123.ps1",
        "hash": {
          "sha256": "ED5D694D561C97B4D70EFE934936286FE562ADDF7D6836F795B336D9791A5C44"
        }
      },
      "type": "file",
      "description": "Simple indicator of observable {ED5D694D561C97B4D70EFE934936286FE562ADDF7D6836F795B336D9791A5C44}",
      "first_seen": "2023-11-21T18:57:25.000Z",
      "provider": "identity--b3bca3c2-1f3d-4b54-b44f-dac42c3a8f01",
      "modified_at": "2023-11-21T18:57:25.000Z",
      "marking": {
        "tlp": "clear"
      }
    }
  }
}
...

STIX file input to ECS file output, defining the Provider field

The provider field is commonly a GUID in the STIX document. To make it more user-friendly, you can use the -p argument to define the threat.indicator.provider field.

python stix_to_ecs.py -i test-inputs/cisa_sample_stix.json -o ecs -p "Elastic Security Labs"

cat ecs/cisa_sample_stix.ecs.ndjson | jq
{
  "threat": {
    "indicator": {
      "file": {
        "name": "123.ps1",
        "hash": {
          "sha256": "ED5D694D561C97B4D70EFE934936286FE562ADDF7D6836F795B336D9791A5C44"
        }
      },
      "type": "file",
      "description": "Simple indicator of observable {ED5D694D561C97B4D70EFE934936286FE562ADDF7D6836F795B336D9791A5C44}",
      "first_seen": "2023-11-21T18:57:25.000Z",
      "provider": "Elastic Security Labs",
      "modified_at": "2023-11-21T18:57:25.000Z",
      "marking": {
        "tlp": "clear"
      }
    }
  }
}
...

STIX directory input to ECS file outputs

If you have a directory of STIX documents, you can use the -r argument to recursively search through the directory and write the ECS documents to the output directory.

python stix_to_ecs.py -ri test-inputs -o ecs

STIX file input to Elasticsearch output

To output to Elasticsearch, you can use either Elastic Cloud or a local instance. Local Elasticsearch will use port 9200 and Elastic Cloud will use port 443. By default, a valid TLS session to Elasticsearch is required.

First, create an index if you don't already have one. In this example, we’re creating an index called stix2ecs, but the index name isn’t relevant.

curl -u {username} -X PUT "https://elasticsearch:port/stix2ecs?pretty"

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "stix2ecs"
}

Next, define the Elasticsearch output options.

python stix_to_ecs.py -i test-inputs/cisa_sample_stix.json -e --url https://elasticsearch:port --user username --index stix2ecs

If you’re storing the data in Elasticsearch for use in another platform, you can view the indicators using cURL.

curl -u {username} https://elasticsearch:port/stix2ecs/_search?pretty

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "stix2ecs",
        "_id" : "n2lt8IwBahlUtp0hzm9i",
        "_score" : 1.0,
        "_source" : {
          "threat" : {
            "indicator" : {
              "file" : {
                "name" : "123.ps1",
                "hash" : {
                  "sha256" : "ED5D694D561C97B4D70EFE934936286FE562ADDF7D6836F795B336D9791A5C44"
                }
              },
              "type" : "file",
              "description" : "Simple indicator of observable {ED5D694D561C97B4D70EFE934936286FE562ADDF7D6836F795B336D9791A5C44}",
              "first_seen" : "2023-11-21T18:57:25.000Z",
              "provider" : "identity--b3bca3c2-1f3d-4b54-b44f-dac42c3a8f01",
              "modified_at" : "2023-11-21T18:57:25.000Z",
              "marking" : {
                "tlp" : "clear"
              }
            }
          }
        }
      }
...

If you’re using Kibana, you can create a Data View for your stix2ecs index to view the ingested indicators.

Finally, you can use this as an indicator source for Indicator Match rules.

Summary

We hope this project helps your organization analyze and operationalize your threat data. If you’re new to the Elastic Common Schema, you can learn more about that here.

As always, please feel free to open an issue with any questions, comments, concerns, or complaints.

About Elastic Security Labs

Elastic Security Labs is the threat intelligence branch of Elastic Security dedicated to creating positive change in the threat landscape. Elastic Security Labs provides publicly available research on emerging threats with an analysis of strategic, operational, and tactical adversary objectives, then integrates that research with the built-in detection and response capabilities of Elastic Security.

Follow Elastic Security Labs on Twitter @elasticseclabs and check out our research at www.elastic.co/security-labs/.

ES|QL validation basics ft. Kibana & Elasticsearch

If you want to quickly validate a single query, or feel comfortable manually testing queries one-by-one, the Elastic Stack UI is all you need. After navigating to the Discover tab in Kibana, click on the "Try ES|QL" Technical Preview button in the Data View dropdown to load the query pane. You can also grab sample queries from the ES|QL Examples to get up and running. Introducing non-ECS fields will immediately highlight errors prioritizing syntax errors, then unknown column errors.

In this example, there are two syntax errors that are highlighted:

• the invalid syntax error on the input wheres which should be where and
• the unknown column process.worsking_directory, which should be process.working_directory.

After resolving the syntax error in this example, you’ll observe the Unknown column errors. Here are a couple reasons this error may appear:

• Fix Field Name Typos: Sometimes you simply need to fix the name as suggested in the error; consult the ECS or any integration schemas and confirm the fields are correct
• Add Missing Data: If you’re confident the fields are correct, sometimes adding data to your stack, which will populate the columns
• Update Mapping: You can configure Mappings to set explicit fields, or add new fields to an existing data stream or index using the Update Mapping API

ES|QL warnings

Not all fields will appear as errors, in which case you’re presented with warnings and a dropdown list. Hard failures (e.g. errors), imply that the rule cannot execute, whereas warnings indicate that the rule can run, but the functions may be degraded.

When utilizing broad ES|QL queries that span multiple indices, such as logs-* | limit 10, there might be instances where certain fields fail to appear in the results. This is often due to the fields being undefined in the indexed data, or not yet supported by ES|QL. In cases where the expected fields are not retrieved, it's typically a sign that the data was ingested into Elasticsearch without these fields being indexed, as per the established mappings. Instead of causing the query to fail, ES|QL handles this by returning "null" for the unavailable fields, serving as a warning that something in the query did not execute as expected. This approach ensures the query still runs, distinguishing it from a hard failure, which occurs when the query cannot execute at all, such as when a non-existent field is referenced.

There are also helpful performance warnings that may appear. Providing a LIMIT parameter to the query will help address performance warnings. Note this example highlights that there is a default limit of 500 events returned. This limit may significantly increase once this feature is generally available.

Security

In an investigative workflow, security practitioners prefer to iteratively hunt for threats, which may encompass manually testing, refining, and tuning a query in the UI. Conveniently, security analysts and engineers can natively leverage ES|QL in timelines, with no need to interrupt workflows by pivoting back and forth to a different view in Kibana. You’ll receive the same errors and warnings in the same security component, which shows Elasticsearch feedback under the hood.

In some components, you will receive additional feedback based on the context of where ES|QL is implemented. One scenario is when you create an ES|QL rule using the create new rule feature under the Detection Rules (SIEM) tab.

For example, this query could easily be converted to an EQL or KQL query as it does not leverage powerful features of ES|QL like statistics, frequency analysis, or parsing unstructured data. If you want to learn more about the benefits of queries using ES|QL check out this blog by Costin, which covers performance boosts. In this case, we must add [metadata _id, _version, _index] to the query, which informs the UI which components to return in the results.

API calls? Of course!

Prior to this section, all of the examples referenced creating ES|QL queries and receiving feedback directly from the UI. For illustrative purposes, the following examples leverage Dev Tools, but these calls are easily migratable to cURL bash commands or the language / tool of your choice that can send an HTTP request.

Here is the same query as previously shown throughout other examples, sent via a POST request to the query API with a valid query.

As expected, if you supply an invalid query, you’ll receive similar feedback observed in the UI. In this example, we’ve also supplied the ?error_trace flag which can provide the stack trace if you need additional context for why the query failed validation.

As you can imagine, we can use the API to programmatically validate ES|QL queries. You can also still use the Create rule Kibana API, which requires a bit more metadata associated with a security rule. However, if you want to only validate a query, the _query API comes in handy. From here you can use the Elasticsearch Python Client to connect to your stack and validate queries.

from elasticsearch import Elasticsearch
client = Elasticsearch(...)
data = {
"query": """
    from logs-endpoint.events.*
    | keep host.os.type, process.name, process.working_directory, event.type, event.action
    | where host.os.type == "linux" and process.name == "unshadow" and event.type == "start"     and event.action in ("exec", "exec_event")
"""
}

# Execute the query
headers = {"Content-Type": "application/json", "Accept": "application/json"}
response = client.perform_request(
"POST", "/_query", params={"pretty": True}, headers=headers, body=data
)

Leverage the grammar

One of the best parts of Elastic developing in the open is the antlr ES|QL grammar is also available.

If you’re comfortable with ANTLR, you can also download the latest JAR to build a lexer and parser.

pip install antlr4-tools # for antlr4
git clone git@github.com:elastic/elasticsearch.git # large repo
cd elasticsearch/x-pack/plugin/esql/src/main/antlr # navigate to grammar
antlr4 -Dlanguage=Python3 -o build EsqlBaseLexer.g4 # generate lexer
antlr4 -Dlanguage=Python3 -o build EsqlBaseParser.g4 # generate parser

This process will require more lifting to get ES|QL validation started, but you’ll at least have a tree object built, that provides more granular control and access to the parsed fields.

However, as you can see the listeners are stubs, which means you’ll need to build in semantics manually if you want to go this route.

The security rule GitHub CI use case

For our internal Elastic EQL and KQL query rule validation, we utilize the parsed abstract syntax tree (AST) objects of our queries to perform nuanced semantic validation across multiple stack versions. For example, having the AST allows us to validate proper field usage, verify new features are not used in older stack versions before being introduced, or even more, ensure related integrations are built based on datastreams used in the query. Fundamentally, local validation allows us to streamline a broader range of support for many stack features and versions. If you’re interested in seeing more of the design and rigorous validation that we can do with the AST, check out our detection-rules repo.

If you do not need granular access to the specific parsed tree objects and do not need to control the semantics of ES|QL validation, then out-of-the-box APIs may be all you need to validate queries. In this use case, we want to validate security detection rules using continuous integration. Managing detection rules through systems like GitHub helps garner all the benefits of using a version-controlled like tracking rule changes, receiving feedback via pull requests, and more. Conceptually, rule authors should be able to create these rules (which contain ES|QL queries) locally and exercise the git rule development lifecycle.

CI checks help to ensure queries still pass ES|QL validation without having to manually check the query in the UI. Based on the examples shown thus far, you have to either stand up a persistent stack and validate queries against the API, or build a parser implementation based on the available grammar outside of the Elastic stack.

One approach to using a short-lived Elastic stack versus leveraging a managed persistent stack is to use the Elastic Container Project (ECP). As advertised, this project will:

Stand up a 100% containerized Elastic stack, TLS secured, with Elasticsearch, Kibana, Fleet, and the Detection Engine all pre-configured, enabled, and ready to use, within minutes.

With a combination of:

• Elastic Containers (e.g. ECP)
• CI (e.g. Github Action Workflow)
• ES|QL rules
• Automation Foo (e.g. python & bash scripts)

You can validate ES|QL rules via CI against the latest stack version relatively easily, but there are some nuances involved in this approach.

Feel free to check out the sample GitHub action workflow if you’re interested in a high-level overview of how it can be implemented.

Note: if you're interested in using the GitHub action workflow, check out their documentation on using GitHub secrets in Actions and setting up Action workflows.

CI nuances

1. Any custom configuration needs to be scripted away (e.g. setting up additional policies, enrichments, etc.) In our POC, we created a step and bash script that executed a series of POST requests to our temporary CI Elastic Stack, which created the new enrichments used in our detection rules.

- name: Add Enrich Policy
  env:
    ELASTICSEARCH_SERVER: "https://localhost:9200"
    ELASTICSEARCH_USERNAME: "elastic"
    ELASTICSEARCH_PASSWORD: "${{ secrets.PASSWORD }}"
  run: |
    set -x
    chmod +x ./add_enrich.sh
    bash ./add_enrich.sh

1. Without data in our freshly deployed CI Elastic stack, there will be many Unknown Column issues as previously mentioned. One approach to address this is to build indices with the proper mappings for the queries to match. For example, if you have a query that searches the index logs-endpoint.events.*, then create an index called logs-endpoint.events.ci, with the proper mappings from the integration used in the query.

2. Once the temporary stack is configured, you’ll need extra logic to iterate over all the rules and validate using the _query API. For example, you can create a unit test that iterates over all the rules. We do this today by leveraging our default RuleCollection.default() that loads all rules, in our detection-rules repo, but here is a snippet that quickly loads only ES|QL rules.

# tests/test_all_rules.py
class TestESQLRules:
    """Test ESQL Rules."""

    @unittest.skipIf(not os.environ.get("DR_VALIDATE_ESQL"),
         "Test only run when DR_VALIDATE_ESQL environment variable set.")
    def test_environment_variables_set(self):
        collection = RuleCollection()

        # Iterate over all .toml files in the given directory recursively
        for rule in Path(DEFAULT_RULES_DIR).rglob('*.toml'):
            # Read file content
            content = rule.read_text(encoding='utf-8')
            # Search for the pattern
            if re.search(r'language = "esql"', content):
                print(f"Validating {str(rule)}")
                collection.load_file(rule)

Each rule would run through a validator method once the file is loaded with load_file.

# detection_rules/rule_validator.py
class ESQLValidator(QueryValidator):
    """Specific fields for ESQL query event types."""

    def validate(self, data: 'QueryRuleData', meta: RuleMeta) -> None:
        """Validate an ESQL query while checking TOMLRule."""
        if not os.environ.get("DR_VALIDATE_ESQL"):
            return

        if Version.parse(meta.min_stack_version) < Version.parse("8.11.0"):
            raise ValidationError(f"Rule minstack must be greater than 8.10.0 {data.rule_id}")

        client = Elasticsearch(...)
        client.info()
        client.perform_request("POST", "/_query", params={"pretty": True},
                               headers={"accept": "application/json", 
                                        "content-type": "application/json"},
                               body={"query": f"{self.query} | LIMIT 0"})

As highlighted earlier, we can POST to the query API and validate given the credentials that were set as GitHub action secrets and passed to the validation as environment variables. Note, the LIMIT 0 is so the query does not return data intentionally. It’s meant to only perform validation. Finally the single CI step would be a bash call to run the unit tests (e.g. pytest tests/test_all_rules.py::TestESQLRules).

1. Finally, CI leveraging containers may not scale well when validating many rules against multiple Elastic stack versions and configurations. Especially if you would like to test on a commit-basis. The time to deploy one stack took slightly over five minutes to complete. This measurement could greatly increase or decrease depending on your CI setup.

Conclusion

Elasticsearch's new feature, Elasticsearch Query Language (ES|QL), is a specialized query and compute engine for Elasticsearch, now in technical preview. It offers seamless integration across various Elastic services like Kibana and Timelines, with validation options for ES|QL queries. Users can validate queries through the Elastic Stack UI or API calls, receiving immediate feedback on syntax or column errors.

Additionally, ES|QL's ANTLR grammar is available for those who prefer a more hands-on approach to building lexers and parsers. We’re exploring ways to validate ES|QL queries in an automated fashion and now it’s your turn. Just know that we’re not done exploring, so check out ES|QL and let us know if you have ideas! We’d love to hear how you plan to use it within the stack natively or in CI.

We’re always interested in hearing use cases and workflows like these, so as always, reach out to us via GitHub issues, chat with us in our community Slack, and ask questions in our Discuss forums.

Check out these additional resources to learn more about how we’re bringing the latest AI capabilities to the hands of the analyst: Learn everything ES|QL Checkout the 8.11.0 release blog introducing ES|QL

ICEDID is a malware family discoveredin 2017 by IBM X-force researchers and is associated with the theft of login credentials, banking information, and other personal information. ICEDID has always been a prevalent family but achieved even more growth since EMOTET’s temporary disruption in early 2021. ICEDID has been linked to the distribution of several distinct malware families including DarkVNC and COBALT STRIKE. Regular industry reporting, including research publications like this one, help mitigate this threat.

ICEDID is known to pack its payloads using custom file formats and a custom encryption scheme. Following our latest ICEDID research that covers the GZip variant execution chain.

In this tutorial, we will introduce these tools by unpacking a recent ICEDID sample starting with downloading a copy of the fake GZip binary:

Analyzing malware can be dangerous to systems and should only be attempted by experienced professionals in a controlled environment, like an isolated virtual machine or analysis sandbox. Malware can be designed to evade detection and infect other systems, so it's important to take all necessary precautions and use specialized tools to protect yourself and your systems.

54d064799115f302a66220b3d0920c1158608a5ba76277666c4ac532b53e855f

Environment setup

For this tutorial, we’re using Windows 10 and Python 3.10.

Elastic Security Labs is releasing a set of tools to automate the unpacking process and help analysts and the community respond to ICEDID.

| Script | Description | Compatibility | | ----------------------------------------- | ---------------------------------------------------------------- | ------------------------------- | | decrypt_file.py | Decrypt ICEDID encrypted file | Windows and others (not tested) | | gzip_variant/extract_gzip.py | Extract payloads from ICEDID fake GZip file | Windows and others (not tested) | | gzip_variant/extract_payload_from_core.py | Extract and decrypt payloads from the rebuilt ICEDID core binary | Windows and others (not tested) | | gzip_variant/load_core.py | Load and execute core custom PE binary | Windows only | | gzip_variant/read_configuration.py | Read ICEDID configuration file contained in the fake GZip | Windows and others (not tested) | | rebuild_pe.py | Rebuild a PE from ICEDID custom PE file | Windows and others (not tested) |

In order to use the tools, clone the Elastic Security Lab release repository and install the nightMARE module.

git clone https://github.com/elastic/labs-releases
cd labs-release
pip install .\nightMARE\

All tools in this tutorial use the nightMARE module, this library implements different algorithms we need for unpacking the various payloads embedded within ICEDID. We’re releasing nightMARE because it is required for this ICEDID analysis, but stay tuned - more to come as we continue to develop and mature this framework.

Unpacking the fake GZip

The ICEDID fake GZip is a file that masquerades as a valid GZip file formatted by encapsulating the real data with a GZip header and footer.

GZip magic bytes appear in red.
The GZip header is rendered in green.
The dummy filename value is blue.

After the GZip header is the true data structure, which we describe below.

We will use the labs-releases\tools\icedid\gzip-variant\extract_gzip.py script to unpack this fraudulent GZip.

usage: extract_gzip.py [--help] input output

positional arguments:
  input       Input file
  output      Output directory

options:
  -h, --help  show this help message and exit

We'll use extract_gzip.py on the ICEDID sample linked above and store the contents into a folder we created called “ extract ” (you can use any existing output folder).

python extract_gzip.py 54d064799115f302a66220b3d0920c1158608a5ba76277666c4ac532b53e855f extract

============================================================
Fake Gzip
============================================================
is_dll: True
core: UponBetter/license.dat (354282 bytes)
stage_2: lake_x32.tmp (292352 bytes)

extract\configuration.bin
extract\license.dat
extract\lake_x32.tmp

This script returns three individual files consisting of:

• The encrypted configuration file: configuration.bin
• The encrypted core binary: license.dat
• The persistence loader: lake_x32.tmp

Decrypting the core binary and configuration files

The configuration and the core binary we extracted are encrypted using ICEDID’s custom encryption scheme. We can decrypt them with the labs-releases\tools\icedid\decrypt_file.py script.

usage: decompress_file.py [--help] input output

positional arguments:
  input       Input file
  output      Output file

options:
  -h, --help  show this help message and exit

As depicted here (note that decrypted files can be written to any valid destination):

python .\decrypt_file.py .\extract\license.dat .\extract\license.dat.decrypted

python .\decrypt_file.py .\extract\configuration.bin .\extract\configuration.bin.decrypted

The core binary and the configuration are now ready to be processed by additional tools. See the data from the decrypted configuration presented in the following screenshot:

Reading the configuration

The configuration file format is presented below.

The configuration can be read using the labs-releases\tools\icedid\gzip-variant\read_configuration.py script.

usage: read_configuration.py [--help] input

positional arguments:
  input       Input file

options:
  -h, --help  show this help message and exit

We’ll use the read_configuration.py script to read the configuration.bin.decrypted file we collected in the previous step.

python .\gzip-variant\read_configuration.py .\extract\configuration.bin.decrypted

============================================================
Configuration
============================================================
botnet_id: 0x3B7D6BA4
auth_var: 0x00000038
uri: /news/
domains:
        alishaskainz.com
        villageskaier.com

This configuration contains two C2 domains:

• alishaskainz[.]com
• villageskaier[.]com

For this sample, the beaconing URI that ICEDID uses is “ /news/ ”.

Rebuilding the core binary for static analysis

ICEDID uses a custom PE format to obfuscate its payloads thus defeating static or dynamic analysis tools that expect to deal with a normal Windows executable. The custom PE file format is described below.

If we want to analyze the core binary, for example with IDA Pro, we need to rebuild it into a valid PE. We use the labs-releases\tools\icedid\rebuild_pe.py script.

usage: rebuild_pe.py [--help] [-o OFFSET] input output

positional arguments:
  input                 Input file
  output                Output reconstructed PE

options:
  -h, --help            show this help message and exit
  -o OFFSET, --offset OFFSET
                        Offset to real data, skip possible garbage

However, when attempting to use rebuild_pe.py on the decrypted core binary, license.dat.decrypted , we receive the following error message:

python .\rebuild_pe.py .\extract\license.dat.decrypted .\extract\core.bin
Traceback (most recent call last):
  File "rebuild_pe.py", line 32, in <module>
    main()
  File "rebuild_pe.py", line 28, in main
    custom_pe.CustomPE(data).to_pe().write(args.output)
  File "nightmare\malware\icedid\custom_pe.py", line 86, in __init__
    raise RuntimeError("Failed to parse custom pe")
RuntimeError: Failed to parse custom pe

The subtlety here is that the custom PE data doesn’t always start at the beginning of the file. In this case, for example, if we open the file in a hexadecimal editor like HxD we can observe a certain amount of garbage bytes before the actual data.

We know from our research that the size of the garbage is 129 bytes.

With that in mind, we can skip over the garbage bytes and rebuild the core binary using the rebuild_pe.py script using the “-o 129” parameter. This time we, fortunately, receive no error message. core.bin will be saved to the output directory, extract in our example.

python .\rebuild_pe.py .\extract\license.dat.decrypted .\extract\core.bin -o 129

The rebuilt PE object is not directly executable but you can statically analyze it using your disassembler of choice.

We assigned custom names to the rebuilt binary sections ( .mare{0,1,2,…} ).

We want to credit and thank Hasherezade’s work from which we took inspiration to build this tool.

Executing the core binary (Windows only)

The core binary can’t be executed without a custom loader that understands ICEDID’s custom PE format as well as the entry point function prototype.

From our research, we know that the entry point expects a structure we refer to as the context structure, which contains ICEDID core and persistence loader paths with its encrypted configuration. The context structure is described below.

To natively execute the core binary we use the labs-releases\tools\icedid\gzip-variant\load_core.py script, but before using it we need to create the context.json file that’ll contain all the information needed by this script to build this structure.

For this sample, we copy the information contained in the fake gzip and we use the path to the encrypted configuration file. We’ve included an example at gzip_variant/context.json.example.

Please note that “field_0” and “stage_2_export” values have to be found while reversing the sample.

Here we use values from our previous research as placeholders but we have no guarantee that the sample will work 100%. For example, in this sample, we don’t know if the #1 ordinal export is the actual entry point of the persistence loader.

We also reproduce the first stage behavior by creating the UponBetter directory and moving the license.dat file into it.

We execute the labs-releases\tools\icedid\gzip_variant\load_core.py script using the decrypted core binary: license.dat.decrypted , the context.json file.

WARNING: The binary is going to be loaded/executed natively by this script, Elastic Security Labs does not take responsibility for any damage to your system. Please execute only within a safe environment.

usage: load_core.py [--help] [-o OFFSET] core_path ctx_path

positional arguments:
  core_path             Core custom PE
  ctx_path              Path to json file defining core's context

options:
  -h, --help            show this help message and exit
  -o OFFSET, --offset OFFSET
                        Offset to real data, skip possible garbage

Because we have the same garbage bytes problem as stated in the previous section, we use the “-o 129” parameter to skip over the garbage bytes.

python .\gzip-variant\load_core.py .\extract\license.dat.decrypted .\gzip-variant\context.example.json -o 129

============================================================
Core Loader
============================================================
Base address: 0x180000000
Entrypoint: 0x180001390

Press a key to call entrypoint...

When launched, the script will wait for user input before calling the entry point. We can easily attach a debugger to the Python process and set a breakpoint on the ICEDID core entry point (in this example 0x180001390 ).

Once the key is pressed, we reach the entry point.

If we let the binary execute, we see ICEDID threads being created (indicated in the following screenshot).

Unpacking and rebuilding payloads from the rebuilt core binary

For extracting any of the payloads that are embedded inside the core binary, we will use the labs-releases\tools\icedid\gzip-variant\extract_payloads_from_core.py script

usage: extract_payloads_from_core.py [--help] input output

positional arguments:
  input       Input file
  output      Output directory

options:
  -h, --help  show this help message and exit

We’ll use this script on the rebuilt core binary.

python .\gzip-variant\extract_payloads_from_core.py .\extract\core.bin core_extract

core_extract\browser_hook_payload_0.cpe
core_extract\browser_hook_payload_1.cpe

From here, we output two binaries corresponding to ICEDID’s payloads for web browser hooking capabilities, however, they are still in their custom PE format.

Based on our research, we know that browser_hook_payload_0.cpe is the x64 version of the browser hook payload and browser_hook_payload_1.cpe is the x86 version.

In order to rebuild them, we use the rebuild_pe.py script again, this time there are no garbage bytes to skip over.

python .\rebuild_pe.py .\core_extract\browser_hook_payload_0.cpe .\core_extract\browser_hook_payload_0.bin

python .\rebuild_pe.py .\core_extract\browser_hook_payload_1.cpe .\core_extract\browser_hook_payload_1.bin

Now we have two PE binaries ( browser_hook_payload_0.bin and browser_hook_payload_1.bin ) we can further analyze.

Attentive readers may observe that we have skipped the VNC server unpacking from the core binary, a decision we made intentionally. We will release it along with other tools in upcoming research, so stay tuned!

Conclusion

In this tutorial we covered ICEDID GZip variant unpacking, starting with the extraction of the fake GZip binary, followed by the reconstruction of the core binary and unpacking its payloads.

ICEDID is constantly evolving, and we are going to continue to monitor major changes and update our tooling along with our research. Feel free to open an issue or send us a message if something is broken or doesn’t work as expected.

Elastic Security Labs is a team of dedicated researchers and security engineers focused on disrupting adversaries through the publication of detailed detection logic, protections, and applied threat research.

Follow us on @elasticseclabsand visit our research portal for more resources and research.

References

The following were referenced throughout the above research:

• https://www.elastic.co/pdf/elastic-security-labs-thawing-the-permafrost-of-icedid.pdf
• https://securityintelligence.com/new-banking-trojan-icedid-discovered-by-ibm-x-force-research/
• https://www.justice.gov/opa/pr/emotet-botnet-disrupted-international-cyber-operation
• https://malpedia.caad.fkie.fraunhofer.de/details/win.darkvnc
• https://www.cybereason.com/blog/threat-analysis-report-all-paths-lead-to-cobalt-strike-icedid-emotet-and-qbot
• https://github.com/elastic/labs-releases
• https://github.com/hasherezade/funky_malware_formats/blob/f1cacba4ee347601dceacda04e4de8c699971d29/iced_id_parser/iceid_to_pe.cpp
• https://mh-nexus.de/en/hxd/
• https://hex-rays.com/IDA-pro/

Imagine you are an Endpoint artifact developer. After you put in the work to ensure protection against conventional shellcode injections or ransomware innovations, how do you know it actually works before you send it out into the world?

First, you set up your end-to-end system, which involves setting up several services, the infrastructure, network configuration, and more. Then, you run some malware; the data you collect answers questions about performance and efficacy, and may be an important research resource in the future. After you spend a day testing and gathering your results, you may want to run several hundred hashes over multiple kinds of operating systems and machine types, a daunting task if done entirely manually.

To automate this process and test our protections at scale, we built Detonate, a system that is used by security research engineers to measure the efficacy of our Elastic Security solution in an automated fashion. Our goal is to have it take security researchers only a couple of clicks to test our protections against malware. (Thus: click, click… boom!)

In this series of posts, we’ll: - Introduce Detonate and why we built it - Explore how Detonate works and the technical implementation details - Describe case studies on how our teams use it at Elastic - Discuss opening our efficacy testing to the community to help the world protect their data from attack

Interested in other posts on Detonate? Check out Part 2 - Into The Weeds: How We Run Detonate where we break down how Detonate works and dive deeper into the technical implementation.

What is Detonate?

At a high level, Detonate runs malware and other potentially malicious software in a controlled (i.e., sandboxed) environment where the full suite of Elastic Security capabilities are enabled. Detonate accepts a file hash (usually a SHA256) and performs the following actions:

• Prepares all files needed for detonation, including the malicious file
• Provisions a virtual machine (VM) instance in a sandboxed environment, with limited connectivity to the outside world
• Waits until file execution completes; this happens when, for example, an execution result file is found or the VM instance is stopped or older than a task timeout
• Stops the running VM instance (if necessary) and cleans up the sandboxed environment
• Generates an event summary based on telemetry and alerts produced during detonation

The results of these detonations are made available to the team for research and development purposes. By post-processing the logs, events, and alerts collected during detonation, we can enrich them with third-party intelligence and other sources to evaluate the efficacy of new and existing Elastic Security protection features.

What does it help us with?

Measuring Efficacy

To build the best EPP on the market, we have to continuously measure the effectiveness of our product against the latest threats. Detonate is used to execute many tens of thousands of samples every month from our data feeds. Gaps in coverage are automatically identified and used to prioritize improvements to our protections.

Supporting existing protections

Many of our protections have associated artifacts (such as machine learning models and rule definitions) which receive regular updates. These updates need testing to ensure we identify and remediate regressions before they end up in a user’s environment.

Detonate provides a framework and suite of tools to automate the analysis involved in this testing process. By leveraging a corpus of hashes with known good and bad software, we can validate our protections before they are deployed to users.

Threat research

Some of our security researchers scour the internet daily for new and emerging threats. By giving them an easy-to-use platform to test malicious software they find in the wild, we better understand how Elastic Security defends against those threats or if we need to update our protections.

Evaluating new protections

In addition to testing existing protections, new protections run the risk of adverse interactions with our existing suite of layered capabilities. A new protection may be easily tested on its own, but tests may hide unintended interactions or conflicts with existing protections. Detonate provides a way for us to customize the configuration of the Elastic Stack and individual protections to more easily find and identify such conflicts earlier in development.

What’s next?

In this publication, we introduced Detonate & what we use it for at Elastic. We discussed the benefits it provides our team when assessing the performance of our security artifacts.

Now that you know what it is, we will break down how Detonate works. In our next post, we’ll dive deeper into the technical implementation of Detonate and how we’re able to create this sandboxed environment in practice.

The Elastic Stack is a modular data analysis ecosystem. While this allows for engineering flexibility, it can be cumbersome to stand up a development instance for testing. The easiest way to stand up the Elastic Stack, is to use Elastic Cloud - it’s completely turnkey. However, there could be situations where Elastic Cloud won’t work for your testing environment. To help with this, this blog will provide you with the necessary information required in order to quickly and painlessly stand up a local, fully containerized, TLS-secured, Elastic Stack with Fleet and the Detection Engine enabled. You will be able to create a Fleet policy, install an Elastic Agent on a local host or VM, and send the data into your stack for monitoring or analysis.

This blog will cover the following:

• The Elastic Stack
• The Elastic Container project
• How to use the Elastic Container project
• How to navigate Kibana and use its related features for security research

The Elastic Container Project is not sponsored or maintained by the company, Elastic. Design and implementation considerations for the project may not reflect Elastic’s guidance on deploying a production-ready stack.

The Elastic Stack

The Elastic Stack is made up of several different components, each of which provide a distinct capability that can be utilized across a wide variety of use cases.

Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine. As the heart of the Elastic Stack, it centrally stores your data for lightning-fast search, fine-tuned relevancy, and powerful analytics that scale with ease.

Kibana

Kibana is the user interface that lets you visualize your Elasticsearch data and manage the Elastic Stack.

The Elastic Agent

The Elastic Agent is the modular agent that allows you to collect data from an endpoint or act as a vehicle to ship data from 3rd party sources, like threat feeds. The Elastic Security integration for endpoints prevents ransomware and malware, detects advanced threats, and arms responders with vital investigative context.

The Elastic Container Project

As mentioned above, the Elastic Stack is modular which makes it very flexible for a wide variety of use cases but this can add complexity to the implementation.

The Elastic Container project is an open source project that uses Docker Compose as a way to stand up a fully-functional Elastic Stack for use in non-production environments. This project is not sponsored or maintained by the Elastic company.

Introduction

The Elastic Container Project includes three main components:

• Elasticsearch
• Kibana
• the Elastic Agent

The project leverages Docker Compose, which is a tool to build, integrate, and manage multiple Docker containers.

To simplify the management of the containers, the project includes a shell script that allows for the staging, starting, stopping, and destroying of the containers.

Additionally, the project makes use of self-signed TLS certificates between Elasticsearch and Kibana, Kibana and your web browser, the Elastic Agent and Elasticsearch, and the Elastic Agent and Kibana.

Prerequisites

The project was built and tested on Linux and macOS operating systems. If you are using Windows, you’ll not be able to use the included shell script, but you can still run native Docker Compose commands and manually perform post-deployment steps.

While not thoroughly tested, it is recommended that you contribute 4 cores and 8 GB of RAM to Docker.

There are only a few packages you need to install:

• Docker
• Docker Compose
• jq
• Git
• cURL

macOS

If you’re running on macOS, you can install the prerequisites using Homebrew, which is an open-source package management system for macOS. Check out the Homebrew site for information on installing it if needed.

**brew install jq git**
**brew install --cask docker**

Linux

If you’re running on Linux, you can install the prerequisites using your package management system ( DNF , Yum , or APT ).

RPM-based distributions

**dnf install jq git curl**

Ubuntu

**apt-get install jq git curl**

You'll also need the Docker suite (including the docker-compose-plugin ). Check out Docker's installation instructions for your OS'

Cloning the project repository

The Elastic Container project is stored on Github. As long as you have Git installed, you can collect it from your CLI of choice.

**git clone https://github.com/peasead/elastic-container.git**
**cd elastic-container**

This repository includes everything needed to stand up the Elastic Stack containers using a single shell script.

Setting credentials

Before proceeding, ensure you update the credentials for the Elastic and Kibana accounts in the .env file located in the root directory of the repository from their defaults of changeme.

The shell script

As mentioned above, the project includes a shell script that will simplify the management of the containers.

**usage: ./elastic-container.sh [-v] (stage|start|stop|restart|status|help)**
**actions:**
 **stage downloads all necessary images to local storage**
 **start creates network and starts containers**
 **stop stops running containers without removing them**
 **destroy stops and removes the containers, the network and volumes created**
 **restart simply restarts all the stack containers**
 **status check the status of the stack containers**
 **help print this message**
 **flags:**
 **-v enable verbose output**

Stage

This option downloads all of the containers from the Elastic Docker hub. This is useful if you are going to be building the project on a system that does not always have Internet access. This is not required, you can skip this option and move directly to the start option, which will download the containers.

**$ ./elastic-container.sh stage**
**8.3.0: Pulling from elasticsearch/elasticsearch**
**7aabcb84784a: Already exists**
**e3f44495617d: Downloading [====\\>] 916.5kB/11.26MB**
**52008db3f842: Download complete**
**551b59c59fdc: Downloading [\\>] 527.4kB/366.9MB**
**25ee26aa662e: Download complete**
**7a85d02d9264: Download complete**
**…**

Start

This opinion will create the container network, download all of the required containers, set up the TLS certificates, and start and connect Elasticsearch, Kibana, and the Fleet server containers together. This option is a “quick start” to get the Elastic Stack up and running. If you have not changed your credentials in the .env file from the defaults, the script will exit.

**$ ./elastic-container.sh start**

**Starting Elastic Stack network and containers**
**[+] Running 7/8**
 **⠿ Network elastic-container\_default Created 0.0s**
 **⠿ Volume "elastic-container\_certs" Created 0.0s**
 **⠿ Volume "elastic-container\_esdata01" Created 0.0s**
 **⠿ Volume "elastic-container\_kibanadata" Created 0.0s**
 **⠿ Container elasticsearch-security-setup Waiting 2.0s**
 **⠿ Container elasticsearch Created 0.0s**
**…**

Stop

This option will stop all running containers in the project, but will not remove them.

**$ ./elastic-container.sh stop**

**Stopping running containers.**
**[+] Running 4/4**
 **⠿ Container elastic-agent Stopped 0.0s**
 **⠿ Container kibana Stopped 0.0s**
 **⠿ Container elasticsearch Stopped 0.0s**
 **⠿ Container elasticsearch-security-setup Stopped**
**…**

Destroy

This option will stop all running containers in the project, remove the container network, remove all data volumes, and remove all containers.

**$ ./elastic-container.sh destroy**

**#####**
**Stopping and removing the containers, network, and volumes created.**
**#####**
**[+] Running 8/4**
 **⠿ Container elastic-agent Removed 0.0s**
 **⠿ Container kibana Removed 0.0s**
 **⠿ Container elasticsearch Removed 0.0s**
 **⠿ Container elasticsearch-security-setup Removed 0.3s**
 **⠿ Volume elastic-container\_esdata01 Removed 0.0s**
 **⠿ Network elastic-container\_default Removed 0.1s**
**…**

Restart

This option restarts all of the project containers.

**$ ./elastic-container.sh restart

#####
Restarting all Elastic Stack components.
#####
Name Command State Ports
---------------------------
elasticsearch /bin/tini -- /usr/local/bi ... Up (healthy) 0.0.0.0:9200-\\>9200/tcp, 9300/tcp
fleet-server /usr/bin/tini -- /usr/loca ... Up 0.0.0.0:8220-\\>8220/tcp
kibana /bin/tini -- /usr/local/bi ... Up (healthy) 0.0.0.0:5601-\\>5601/tcp**

Status

This option returns the status of the project containers.

**$ ./elastic-container.sh status**
**Name Command State Ports**
**---------------------------**
**elasticsearch /bin/tini -- /usr/local/bi ... Up (healthy) 0.0.0.0:9200-\\>9200/tcp, 9300/tcp**
**fleet-server /usr/bin/tini -- /usr/loca ... Up 0.0.0.0:8220-\\>8220/tcp**
**kibana /bin/tini -- /usr/local/bi ... Up (healthy) 0.0.0.0:5601-\\>5601/tcp**

Clear

This option clears all documents in the logs and metrics indices.

**$ ./elastic-container.sh clear**

**Successfully cleared logs data stream**
**Successfully cleared metrics data stream**

Help

This option provides instructions on using the shell script.

**$ ./elastic-container.sh help**

**usage: ./elastic-container.sh [-v] (stage|start|stop|restart|status|help)**
**actions:**
 **stage downloads all necessary images to local storage**
 **start creates a container network and starts containers**
 **stop stops running containers without removing them**
 **destroy stops and removes the containers, the network and volumes created**
 **restart simply restarts all the stack containers**
 **status check the status of the stack containers**
**clear all documents in logs and metrics indexes**
 **help print this message**
**flags:**
 **-v enable verbose output**

Getting Started

Now that we’ve walked through the project overview and the shell script, let’s go through the process of standing up your own stack.

Updating variables

All of the variables are controlled in an environment file ( .env ) that is at the root of the repository. The only things that you must change are the default usernames and passwords for elastic and kibana.

Open the .env file with whatever text editor you’re most comfortable with and update the ELASTIC_PASSWORD and KIBANA_PASSWORD variables from changeme to something secure. If you do not update the credentials from the defaults in the .env file, the script will exit.

If you want to change the other variables (such as the stack version), you can do so in this file.

Starting the Elastic Stack

Starting the project containers is as simple as running the elastic-container.sh shell script with the start option.

**$ ./elastic-container.sh start**

**Starting Elastic Stack network and containers
[+] Running 7/8
⠿ Network elastic-container\_default Created 0.0s
⠿ Volume "elastic-container\_certs" Created 0.0s
⠿ Volume "elastic-container\_esdata01" Created 0.0s
⠿ Volume "elastic-container\_kibanadata" Created 0.0s
⠿ Container elasticsearch-security-setup Waiting 2.0s
⠿ Container elasticsearch Created 0.0s
⠿ Container kibana Created 0.1s
⠿ Container fleet-server Created 0.2s

Attempting to enable the Detection Engine and Prebuilt-Detection Rules
Kibana is up. Proceeding
Detection engine enabled. Installing prepackaged rules.
Prepackaged rules installed!
Waiting 40 seconds for Fleet Server setup
Populating Fleet Settings
READY SET GO!

Browse to https://localhost:5601
Username: elastic
Passphrase: you-changed-me-from-the-default-right?**

Accessing the Elastic Stack

Once the containers have all downloaded and started, you’ll get an output that tells you to browse to https://localhost:5601.

Note: You’ll need to accept the self-signed TLS certificate.

Enabling the Platinum Features

Enabling the Platinum license features are completely optional. Security features, like anti-malware, EDR, EPP, etc. are included in the Basic license. Memory, behavior, and ransomware protections are Platinum license features. If you want to change your license, we can do that with the .env file or from within Kibana. You can update to Elastic Platinum for 30-days.

If you want to use the .env file so that the features are enabled when the stack is built, change LICENSE=basic to LICENSE=trial and then start the project as normal.

If you prefer to use Kibana, click on the hamburger menu, and then click on Stack Management.

Click on License Management and then “Start a 30-day trial”.

Creating a Fleet policy

Now that we have the entire Elastic Stack up and running, we can make a Fleet policy. Fleet is a subroutine of an Elastic Agent (which was built when we ran the start option in the shell script) that enables you to manage other Elastic Agents, policies, and integrations.

Fleet is managed in Kibana, the UI that allows you to interact with data stored in Elasticsearch and manage your Elastic stack. If you’re interested in learning more about Kibana, check out the free training videos.

Log into your Kibana instance and click on the “hamburger” menu on the top left, and navigate down to “Fleet”, under the “Management” section.

Next, click on the “Agent policies” tab and then the “Create agent policy” button.

Give your new policy a name and a description (optional). Normally, we uncheck the “Collect agent logs” and “Collect agent metrics” options because it’s additional data going to the stack that we generally don’t need for our specific use-case. If you’re doing troubleshooting or interested in what’s happening behind the scenes, this data can help you understand that.

Next, click on your new policy and the blue “Add integration” button.

There are hundreds of integrations, but the ones that we’re most interested in for this blog are for Elastic Security.

To install Elastic Security, simply click on the tile on the main integrations page or search for “security”.

Next, click the “Add Endpoint and Cloud Security” button to install this integration into the policy we just created.

Name the integration and click the blue “Save and continue” button.

While the Endpoint and Cloud Security and System integrations will collect security related logs, if you’re using Sysmon on a Windows host, you may want to add the “Windows” integration to collect those logs.

Once the integration is installed, you’ll be prompted to add more Agents or to do that later. Select the “Add Elastic Agent later” option so we can make a few more changes to our policy.

Now we’ll be dropped back to our policy page.

We should have two integrations for our policy: security and system-1.

Before we add any agents, we’ll want to set our Elastic Agent to Detect (so that it allows the malware to completely execute), register the Elastic Agent as a trusted AV solution (Windows only), and instruct the Endpoint and Cloud Security integration to collect memory samples from security events. This is tremendously helpful for “fileless” malware that injects directly into memory, like Cobalt Strike.

If you want to learn more about extracting malware beacons from events generated by the Elastic Agent, check out our other publications and repositories.

To allow the malware to continue to execute, on your “Windows” policy page, click on the name of the integration (“security” in our example), set the Protection level to “Detect”.

Repeat these steps for the Ransomware, Memory threat protections, and Malicious behavior sections.

We’re setting the Elastic Agent to Detect so that the malware we’re detonating will run completely so that we can analyze the entire execution chain. If you want the malware to be stopped, you can leave this in Prevent mode.

Next, scroll to the bottom and select the “Register as antivirus” toggle and click on the “Show advanced settings” hyperlink.

Scroll down to windows.advanced.memory_protection.shellcode_collect_sample , windows.advanced.memory_protection.memory_scan_collect_sample , and windows.advanced.memory_protection.shellcode_enhanced_pe_parsing options and set the value to true.

As mentioned above, these steps are for labs, sandboxes, testing, etc. These settings can generate a lot of data, so setting these for production will need resourcing and sizing considerations.

If you’re making a policy for Linux or macOS, repeat these for the proper OS.

Once we’re done with all of the post-installation configurations, we can click the blue Save integration button.

Enabling Elastic’s Prebuilt Detection Rules

Now that we have created our Fleet agent policy we need to enable the set of pre-built detection rules associated with the OS or platform we will be deploying on (e.g Windows). To do this you will need to go to the Alerts page within the security app.

Click on the hamburger menu and select Alerts, under the Security solution.

Next, click on the blue Manage Rules button.

Once on the Rules page you can update all of the prebuilt rules provided by Elastic by clicking on the “Update Elastic prebuilt rules” button. The update framework is enabled when you go into the “Manage rules” section for the first time, if the “Update Elastic prebuilt rules” button isn’t present, refresh the screen.

Once the rules have been updated, you can browse the available detection rules, search them by a number of different patterns or simply filter by tag, which is what we will do here by searching for Windows rules.

Now we can select all of the Windows rules.

Once all of the rules have been selected, we can bulk enable them.

As the Elastic Container Project runs completely inside single Docker containers, performance impacts could be noticed if you enable all of the rules available. Explore the different rules and enable or disable them based on your infrastructure and use cases.

After we have enabled these rules they will be live and will be run against the data your endpoint agent sends into your stack. When the Detection Engine rules are triggered, they will be raised in the Alerts page in the Security Solution.

Enrolling an Elastic Agent

Still in Fleet, we have several ways to add an Elastic Agent. The most straightforward is from within the policy that we want to enroll an Elastic Agent into (otherwise you have to specify which policy you want to use). It doesn’t really matter which approach you use, but clicking on the Actions button and then Add agent works from just about anywhere in Fleet.

Scroll down and click on the OS that you’re going to be installing the Elastic Agent on, and copy/paste the instructions directly into a terminal window on the host you’re going to be installing the agent onto. Note, if you’re using Windows, use a Powershell CLI that is running as (or elevated to) an account with administrative entitlements.

Of note, because all of our TLS certificates are self-signed, we need to append the –insecure flag. This is unnecessary if you are using trusted certificates.

**.\elastic-agent.exe install --url=https://[stack-ip]:8220 --enrollment-token=[token] --insecure**

Back in Kibana, we can see confirmation that the Elastic Agent installed on the host and that data is being recorded into Elasticsearch.

We can see that the Elastic Agent is reporting into Fleet and is healthy.

If we go into the Discover tab, we can see various event types reporting into Elasticsearch. We can generate some test data by opening notepad.exe , calc.exe , and ping.exe -t www.elastic.co on the host. From Discover, we can make a simple query to validate that we’re seeing the data:

**process.name.caseless : (notepad.exe or ping.exe or calc.exe)**

Now that we’ve validated that we’re seeing data. Let's fire some malware!

Test fire some malware

There are a lot of places you can download malware from, but for this test, we’ll simply use the industry standard EICAR anti malware test file to check the functionality.

The EICAR test is a file that is universally identified by security vendors and is used to test the operation of anti malware software and platforms. It contains a single string and is non-malicious.

From within the Windows host, we’ll use Powershell to download the EICAR file.

**Invoke-WebRequest -Uri "https://secure.eicar.org/eicar.com.txt" -OutFile "eicar.txt"**

As expected, the event was immediately identified by the Elastic Agent’s security integration.

After a few minutes, the events are recorded into the Security Solution within Kibana. You can get there by clicking on the hamburger menu and then clicking on the Alerts section.

Here we can see the alert populated.

If we click on the Analyzer button, we can dig into the event to identify the process that generated the event.

In our example, we can see powershell.exe generated the event and this includes the correlated network events - secure.eicar.org , which is where the EICAR test file was downloaded from.

Summary

In this publication, we introduced you to the Elastic Stack and an open source project that can be used to quickly and securely stand up the entire stack for testing, labs, and security research.

Kibana and the Security Solution are powerful tools that are built by incident responders, threat hunters, and intelligence analysts with security practitioners in mind. To learn more about how to use these tools, Elastic has some great (free and paid) training that can help learn how to use Kibana for threat hunting.

Elastic Filebeat modules

Elastic Filebeat modules simplify the collection, parsing, and visualization of data stored in common log formats. Elastic publishes a variety of Filebeat modules that are focused on collecting the data you want for use within Elasticsearch. These modules provide a standardized and “turnkey” method to ingest specific data sources into the Elastic Stack.

Using these capabilities, the Threat Intel Filebeat module:

• Consumes threat data from six open source feeds
• Loads threat data into Elasticsearch
• Normalizes threat data into the Threat ECS fieldset
• Enables threat analysis through dashboards and visualizations

Analysts and threat hunters can use this data for raw threat hunting, enrichment, intelligence analysis and production, and detection logic.

![](/assets/images/ingesting-threat-data-with-the-threat-intel-filebeat-module/overview.jpg

The six feeds included with the 7.13 Filebeat Threat Intel module are as follows (additional feeds may be added in the future):

• Abuse.ch Malware
• Abuse.ch URL
• AlienVault Open Threat Exchange (OTX)
• Anomali Limo
• Malware Bazaar
• Malware Information Sharing Platform (MISP)

Using the Threat Intel Filebeat module, you can choose from several open source threat feeds, store the data in Elasticsearch, and leverage the Kibana Security App to aid in security operations and intelligence analysis.

Threat Intel Filebeat module

Generally, the Filebeat Threat Intel module can be started without any configuration to collect logs from Abuse.ch feeds, Anomali Limo, and Malware Bazaar. However, the optional AlienVault OTX and MISP datasets require tokens to authenticate to their feed sources. Thankfully, obtaining a token is a simple process.

AlienVault OTX

The team over at Alien Labs® has created the Open Threat Exchange (OTX)® as an open threat intelligence community. This environment provides access to a diverse community of researchers and practitioners. OTX allows anyone in the community to discuss, research, validate, and share threat data. Additionally, OTX has an Application Programming Interface (API) endpoint that provides a read-only feed; which is how the Filebeat module consumes the OTX threat data.

To access the OTX API, you simply need to create an account. Once you have an account, you can subscribe to specific OTX community reports and threat data feeds called “Pulses.” These Pulses are retrieved by the Filebeat module and stored in Elasticsearch.

Pulses are updated at various cadences, but many are daily or even hourly. The Pulse has a summary of the threat, indicators, and various other enrichments that can help you contextually assess the threat in your environment.

To subscribe to Pulses, select Browse → Pulses, and then subscribe to any Pulses that you’d like. You can sort by the most recently modified to identify the most active Pulses.

Now that you’ve subscribed to Pulses of interest, we’ll need to collect your API key.

Retrieving Your API Key

The API key is used to securely authenticate to OTX and obtain the indicators from Pulses.

To retrieve your API key, select your userID → Settings, and then copy your OTX Key.

Now that we have your OTX Key, let’s set up MISP.

MISP

The Malware Information Sharing Platform (MISP) is an open source project for collecting, storing, distributing, and sharing indicators about threats.

While MISP is extremely powerful and has a tremendous variety of features, it can be a bit cumbersome to set up. If you are planning on setting up MISP for production, check out the official documentation for installing MISP on Kali, RHEL (incl. CentOS and Fedora), or Ubuntu.

If your organization doesn’t have a MISP instance, you can use one of the many projects that use Docker to get MISP up and running. There’s a great and maintained project by Jason Kendall (@coolacid) that is about as turnkey as you could ask for.

Standing up CoolAcid’s MISP Docker Containers

As a caveat, this will cover a default development deployment of MISP. It should not be used in production. Please see the official MISP documentation for properly deploying a secure MISP instance.

As a few prerequisites, you’ll need to have Docker Compose and Git installed:

• Docker Compose is used to automate the deployment and configuration of the containers. You can check out Docker’s documentation on getting Compose installed.
• Git is a version-control framework used to coordinate software development throughout contributors and community members. You can check out the Git documentation on getting Git installed.

Next, we need to clone CoolAcid’s repository and fire up the containers.

• git clone: Copies the remote repository to your local machine into a file called “docker-misp”
• cd docker-misp: Changes into the “docker-misp” directory
• docker-compose up -d: Uses the docker-compose file in the “docker-misp” directory to download, build, and start all of the relevant containers in “detached mode” (in the background)

Code Block 1 - Starting MISP Containers

$ git clone https://github.com/coolacid/docker-misp.git
$ cd docker-misp
$ docker-compose up -d

Pulling misp (coolacid/misp-docker:core-latest)...
core-latest: Pulling from coolacid/misp-docker
a54cbf64e415: Pull complete
84e78d2508ee: Pull complete
433476aac54e: Pull complete
780a2dfa04f6: Pull complete
Digest: sha256:7f380ad0d858bdec2c4e220f612d80431b1a0b0cb591311ade38da53b50a4cc1
Status: Downloaded newer image for coolacid/misp-docker:core-latest
Pulling misp-modules (coolacid/misp-docker:modules-latest)...
modules-latest: Pulling from coolacid/misp-docker
cdd040608d7b: Pull complete
4e340668f524: Pull complete
a4501f203bb2: Downloading [=========================================>         ]  166.1MB/201.3MB
2cdaa3afcfca: Download complete
99a18a4e84d6: Downloading [=============================>                     ]  130.8MB/218.3MB
...

Once all of the containers are started, simply browse to https://localhost and log in with the default credentials of admin@admin.test and a passphrase of admin. You will immediately be required to change your passphrase.

Configuring default MISP feeds

Once you have started the MISP containers and changed your default credentials, hover over Sync Actions and then select List Feeds.

Highlight the available feeds, select “Enable selected” to enable the default feeds, and then “Fetch and store all feed data.”

Next, select on the “Event Actions” menu item, select “List Events” and you’ll see data begin to be populated. This will take a while.

While the data provided by the MISP threat feeds is being downloaded, let’s get your API key.

Collecting Your API Key

To collect your API key, select “Administration” and then “List Users.” You will see your account. Next to your “Authkey” will be an eye icon, select it to show your API key and copy that down.

Now that we have set up and configured MISP and retrieved our API key, we can configure the actual Filebeat module.

Installing Filebeat

Getting the Threat Intel module is no different than any other Filebeat module. Check out the Quick Start guide to install Filebeat either as a standalone binary or a package for macOS, Windows, or Linux.

Configuring the Threat Intel Filebeat module

Once you have Filebeat, we’ll simply enable the module (ensure filebeat is in your $PATH).

Code Block 2 - Enabling the Threat Intel Filebeat Module

filebeat modules enable threatintel

Next, let’s configure feeds. We’ll do this by modifying the module configuration files. Depending on your OS and installation method, the configuration files will be located in different locations:

Windows

• C:\Program Files\Filebeat\modules.d\threatintel.yml
• If installed with Chocolatey
• C:\ProgramData\chocolatey\lib\filebeat\tools\modules.d\threatintel.yml

macOS

• filebeat/modules.d/threatintel.yml
• If installed with Homebrew
• /usr/local/etc/filebeat/modules.d/threatintel.yml

Linux

• filebeat/modules.d/threatintel.yml
• If Installed with APT or YUM / dnf
• /etc/filebeat/modules.d/threatintel.yml

Using whichever text editor you’re most comfortable with, open threatintel.yml and we’ll add your OTX API key, your MISP API key, and validate Anomali’s credential pair.

Abuse URL feed configuration

By default, the Abuse URL feed is enabled and does not need modification. The feed includes domain, URI, and URL indicators with additional context for significant dates, tags, submitter, status, etc.

Code Block 3 - Configuring the Abuse URL Feed

abuseurl:
  enabled: true

  # Input used for ingesting threat intel data.
  var.input: httpjson

  # The URL used for Threat Intel API calls.
  var.url: https://urlhaus-api.abuse.ch/v1/urls/recent/

  # The interval to poll the API for updates.
  var.interval: 10m

Abuse malware feed configuration

By default, the Abuse malware feed is enabled and does not need modification. The feed includes file hashes and hosts with additional context for significant dates, tags, status, etc.

Code Block 4 - Configuring the Abuse Malware Feed

abusemalware:
    enabled: true

    # Input used for ingesting threat intel data.
    var.input: httpjson

    # The URL used for Threat Intel API calls.
    var.url: https://urlhaus-api.abuse.ch/v1/payloads/recent/

    # The interval to poll the API for updates.
    var.interval: 10m

MISP feed configuration

By default, the MISP feed is enabled but requires configuration. The feed includes various file and network data with additional context for significant dates, tags, status, submitter, etc.

The API endpoint that Filebeat will query needs to be configured. If you are running MISP on the same system as Filebeat, you can use var.url: https://localhost/event/restSearch. If you are running MISP elsewhere, you’ll need to enter that hostname or IP address in lieu of localhost.

The API token is the “Authkey” that you retrieved during the previous MISP setup steps. You’ll enter that as the value for var.api_token:

If you are using a self-signed SSL certificate for MISP, you’ll want to disable the SSL verification mode by uncommenting the var.ssl.verification_mode: none line.

Code Block 5 - Configuring the MISP Feed

misp:
    enabled: true

    # Input used for ingesting threat intel data, defaults to JSON.
    var.input: httpjson

    # The URL of the MISP instance, should end with "/events/restSearch".
    var.url: https://localhost/events/restSearch

    # The authentication token used to contact the MISP API. Found when looking at user account in the MISP UI.
    var.api_token: MISP-Authkey

    # Configures the type of SSL verification done, if MISP is running on self signed certificates
    # then the certificate would either need to be trusted, or verification_mode set to none.
    var.ssl.verification_mode: none

    # Optional filters that can be applied to the API for filtering out results. This should support the majority of
    # fields in a MISP context. For examples please reference the filebeat module documentation.
    #var.filters:
    #  - threat_level: [4, 5]
    #  - to_ids: true

    # How far back to look once the beat starts up for the first time, the value has to be in hours. Each request
    # afterwards will filter on any event newer than the last event that was already ingested.
    var.first_interval: 300h

    # The interval to poll the API for updates.
    var.interval: 5m

AlienVault OTX feed configuration

By default, the AlienVault OTX feed is enabled but requires configuration. The feed includes various file and network data with additional context for significant dates, tags, etc.

The API token is the “OTX Key” that you retrieved during the AlienVault OTX setup steps. You’ll enter that as the value for var.api_token:

Code Block 6 - Configuring the AlienVault OTX Feed

otx:
  enabled: true

  # Input used for ingesting threat intel data
  var.input: httpjson

  # The URL used for OTX Threat Intel API calls.
  var.url: https://otx.alienvault.com/api/v1/indicators/export

  # The authentication token used to contact the OTX API, can be found on the OTX UI.
  Var.api_token: OTX-Key

  # Optional filters that can be applied to retrieve only specific indicators.
  #var.types: "domain,IPv4,hostname,url,FileHash-SHA256"

  # The timeout of the HTTP client connecting to the OTX API
  #var.http_client_timeout: 120s

  # How many hours to look back for each request, should be close to the configured interval.
  # Deduplication of events is handled by the module.
  var.lookback_range: 1h

  # How far back to look once the beat starts up for the first time, the value has to be in hours.
  var.first_interval: 400h

  # The interval to poll the API for updates
  var.interval: 5m

Anomali feed configuration

By default, the Anomali feed is enabled but requires configuration. The feed includes various file and network data with additional context for significant dates, tags, etc.

The default username and passphrase for the Limo feed is guest:guest, but are commented out. If you do not have other credential pairs, you can simply uncomment var.username and var.password.

At the time of this writing, Anomali has 11 collections that they provide as part of their Limo feed. The var.url variable is where the collection is defined. To get a list of the collections, you can query the Anomali Limo collections API endpoint (while not required, jq makes the collections easier to read).

Code Block 7 - Configuring the Anomali Limo Collections

$ curl -L -u guest:guest https://limo.anomali.com/api/v1/taxii2/feeds/collections | jq

{
  "collections": [
    {
      "can_read": true,
      "can_write": false,
      "description": "",
      "id": "107",
      "title": "Phish Tank"
    },
    {
      "can_read": true,
      "can_write": false,
      "description": "",
      "id": "135",
      "title": "Abuse.ch Ransomware IPs"
    },
    {
      "can_read": true,
      "can_write": false,
      "description": "",
      "id": "136",
      "title": "Abuse.ch Ransomware Domains"
    },
...

The collection ID can be inserted into the Anomali configuration. There are a few ways to do this. You can:

• Manually change the ID
• Enter all of the IDs and comment out all but the collection you’re wanting to target
• Create a duplicate Anomali configuration section for each collection

The below example shows the approach of duplicate sections for each collection; notice the different collection ID for each section (31, 313, 33) in the var.url: field.

Code Block 8 - Configuring the Anomali Limo Feed

  anomali:
    enabled: true

    # Input used for ingesting threat intel data
    var.input: httpjson

    # The URL used for Threat Intel API calls. Limo has multiple different possibilities for URL's depending
    # on the type of threat intel source that is needed.
    var.url: https://limo.anomali.com/api/v1/taxii2/feeds/collections/31/objects

    # The Username used by anomali Limo, defaults to guest.
    var.username: guest

    # The password used by anomali Limo, defaults to guest.
    var.password: guest

    # How far back to look once the beat starts up for the first time, the value has to be in hours.
    var.first_interval: 400h

    # The interval to poll the API for updates
    var.interval: 5m

  anomali:
    enabled: true

    # Input used for ingesting threat intel data
    var.input: httpjson

    # The URL used for Threat Intel API calls. Limo has multiple different possibilities for URL's depending
    # on the type of threat intel source that is needed.
    var.url: https://limo.anomali.com/api/v1/taxii2/feeds/collections/313/objects

    # The Username used by anomali Limo, defaults to guest.
    var.username: guest

    # The password used by anomali Limo, defaults to guest.
    var.password: guest

    # How far back to look once the beat starts up for the first time, the value has to be in hours.
    var.first_interval: 400h

    # The interval to poll the API for updates
    var.interval: 5m

  anomali:
    enabled: true

    # Input used for ingesting threat intel data
    var.input: httpjson

    # The URL used for Threat Intel API calls. Limo has multiple different possibilities for URL's depending
    # on the type of threat intel source that is needed.
    var.url: https://limo.anomali.com/api/v1/taxii2/feeds/collections/33/objects
...

Now that we’ve configured the module to consume threat feed data, let’s send the data into Elasticsearch and visualize it with Kibana.

Setting up Elasticsearch and Kibana

The Filebeat Threat Intel module will send the configured threat feed data into Elasticsearch, which can be visualized with Kibana. Please see the Elastic documentation for setting up Elasticsearch and Kibana production environments. Additionally, if you’re looking for a turnkey approach, you can quickly and securely set up an Elastic Cloud account.

For this non-production example, we’ll be using one of the many projects that use Docker to get Elasticsearch and Kibana up and running quickly.

Standing up an Elasticsearch and Kibana container

As a caveat, this will cover a convenient default development deployment of Elasticsearch and Kibana. It should not be used in production. Please see the Elastic documentation for properly deploying a secure instance.

We’ll simply collect the repository and start the Docker containers.

• git clone: This copies the remote repository to your local machine into a folder called “elastic-container”
• cd elastic-container: Changes into the “elastic-container” directory
• sh elastic-container.sh start: This downloads and starts the Elasticsearch and Kibana containers

Code Block 9 - Starting Elastic Containers

$ git clone https://github.com/peasead/elastic-container.git
$ cd elastic-container
$ sh elastic-container.sh start

7.12.1: Pulling from elasticsearch/elasticsearch
ddf49b9115d7: Already exists
4df4d6995ad2: Pull complete
e180ce5d1430: Pull complete
b3801a448e4f: Downloading [====>                      ]  199.3MB/353.1MB
a3100bfb487c: Download complete
817ce7c869c7: Download complete
485f138f2280: Download complete

7.12.1: Pulling from kibana/kibana
ddf49b9115d7: Already exists
588c50b1b6af: Extracting [====================>       ]  34.93MB/40.52MB
9d32826b6fa0: Download complete
01017880c9d9: Download complete
efcedd43b7be: Download complete
0887ad2a14e0: Download complete
625b277c1f7b: Downloading [=====>                     ]  52.27MB/320.4MB
68815bc8856d: Download complete
e9e0d8f8fa8c: Download complete

Check out the repository documentation for additional usage and configuration options (if needed).

Once all of the containers are started, simply browse to http://localhost:5601 and log in with the default credentials of elastic and a passphrase of password.

Consuming threat data with Filebeat

There are multiple output options for Filebeat, so use whatever is easiest for you. We’ll use a local Elasticsearch instance in this example. Using a local instance of Elasticsearch and Kibana requires no modification to the filebeat.yml file.

To validate our configuration, let’s first test our configuration and access to Elasticsearch.

• filebeat test config: This will test to ensure your filebeat.yml configuration is correct (if you modified it to fit your environment)
• filebeat test output - this will test to ensure you can access Elasticsearch

Code Block 10 - Testing Filebeat Configuration and Connection

$ filebeat test config
Config OK

$ filebeat test output
elasticsearch: http://localhost:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: ::1, 127.0.0.1
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 7.12.0

To load the dashboards, index pattern, and ingest pipelines, let’s run the setup.

• filebeat setup: This will connect to Kibana and load the index pattern, ingest pipelines, and the saved objects (tags, visualizations, and dashboards)

Code Block 11 - Setting Up Filebeat Index Patterns and saved objects in Kibana

$ filebeat setup

Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
See more: https://www.elastic.co/guide/en/machine-learning/current/index.html
Loaded machine learning job configurations
Loaded Ingest pipelines

Finally, let’s start Filebeat to begin collecting!

Next, browse to Kibana and select the Dashboards app. To make the dashboards easier to find, they all use the “threat intel” tag.

There is a dashboard for each feed and an overview dashboard that shows the health of the module.

![](/assets/images/ingesting-threat-data-with-the-threat-intel-filebeat-module/overview.jpg

It may take several minutes for all of the data to be retrieved as the different sources are polled.

What’s next?

We’re working on converting the existing visualizations into Lens and adding drilldown capabilities to each visualization.

Additionally, as we mentioned in the beginning of this post, this is part one of a three-part series on operationalizing threat data in the Elastic Stack. The next post will cover enhancements to the Threat ECS fieldset and enriching threat data using local endpoint and network observations.

We’re working on adding additional open source and commercial feeds. If you have feeds that you’d like to see prioritized, please check out the contribution section below.

Finally, we’re looking at opportunities to add context and enrichments to observed events with third-party sources.

So stay tuned — we’re continuing to lean hard into empowering our customers to defend their environments. Being able to action threat data is a key part of that journey.

How can you contribute?

The Threat Intel Filebeat module was released with Elastic 7.12, which means that it is still in beta. Testing the feeds, configurations, visualizations, etc. is strongly encouraged. We love hearing feedback.

In addition to the Threat Intel module, there are some other repositories that are related to the collection, processing, and analysis of TI data:

• The Beats repository, where you can contribute to, and enhance, threat data feeds
• The Elastic Common Schema (ECS) repository, where you can be a part of the discussion on shaping how threat data is described in the Elastic Stack
• The Kibana repository, where analysts interact with the data stored in Elasticsearch
• The Detection Rules repository, where detection logic and rules are created and stored

The best way to contribute to the community is to explore the functionality, features, and documentation and let us know through a Github Issue if there is a problem or something you’d like to see.

If you’re new to Elastic, experience our latest version of the Elasticsearch Service on Elastic Cloud. Also be sure to take advantage of our Quick Start training to set yourself up for success.

• NETWIRE has shown an increase in prevalence over the last year
• Elastic Security Labs created an extractor to pull out configuration data from NETWIRE files and memory dumps targeting the functions the malware uses to extract its encrypted data
• The NETWIRE extractor is freely available for download

To download the NETWIRE configuration extractor, check out our post on the tool:

• NETWIRE configuration extractor

Preamble

NETWIRE is a Remote Access Tool (RAT) that has been used since at least 2014. It is a publicly available commodity malware and has been observed being used by financially motivated and nation-state actors.

In the second half of 2022, we noticed an uptick in the prevalence of NETWIRE usage in our telemetry data. This prompted the Elastic Security Labs team to develop a configuration extractor to assist the security community in collecting atomic indicators within the configurations. Using this extractor will support threat tracking and improve detection, prevention, and response times.

Extractor

The NETWIRE RAT uses the RC4 symmetric encryption algorithm to protect its configuration which is encrypted in the .data section along with the 16 bytes long RC4 decryption key.

While reversing our samples the analysts noticed that for both the crypto::rc4_init_sbox and crypto::rc4_decrypt functions the second argument (#2 in the image below) is always a memory address for the desired encrypted configuration value, and the third argument (#3) is an immediate value written to the memory stack before the call which represents the size of the encrypted string.

It was also noted that the function calls are one after the other. This is important to allow us to structure the extractor to look for these functions sequentially.

With $key (from the above image) in mind, we created YARA rules to identify the location of the key and encrypted configuration values.

![YARA rule section that identifies the key and encrypted configuration](/assets/images/netwire-dynamic-configuration-extraction/image5.jpg

With this information we can then use Capstone to:

1. Locate the function responsible for decrypting the configuration using YARA.

2. Disassemble the function using Capstone.

3. Extract the RC4 key address and the encrypted configuration field addresses.

4. Extract the size of the configuration field.

5. RC4 decrypt the encrypted fields and rebuild the configuration.

![RC4 decrypting the configuration](/assets/images/netwire-dynamic-configuration-extraction/image5.jpg

Once we have recreated the configuration, we can use the extractor to pull out several parameters used by NETWIRE, as well as a few basic file characteristics:

• Active Setup Key : Active Setup registry key to achieve persistence.
• C2 IP list : List of command and control (C2) server domains or IP addresses.
• Host ID : A unique identifier that is assigned to the infected machine.
• Installation path : The location where the malware will be installed.
• Keylogger logs directory : The location where the keylogging log file will be stored.
• Mutex : Mutex name, to create a synchronization object to ensure only one instance of the sample is running on the machine.
• Password : Static password to generate AES key used for encrypting the communication between the malware and the C2 server.
• Run registry key entry : Name of the entry in the run registry, used for persistence.
• Sleep in seconds : The amount of time the malware sleeps.

The configuration extractor accepts four parameters:

• -f : to specify a single NETWIRE sample
• -d : To specify a directory of NETWIRE samples
• -o : To write the configuration in JSON format to the specified file
• --all-config : To print the unparsed raw decrypted configuration

Analysis

We’ve used this extractor to examine a set of samples from the previous 180 days to extract indicators for further enrichment and analysis.

Our initially collected batch of samples came as a mixture of executable files and memory dumps. The extractor will only work on unmapped files, so the dumps which were already mapped were run through pe_unmapper.

When extracting a payload from memory, we are obtaining a memory-mapped version of it. This means that the "Raw Address" and "Raw Size" may not be correctly aligned with the correct section’s data. To correctly align the PE file, it is necessary to adjust the pointer to the raw address so that it matches the virtual address for every section.

Now we can run the configuration extractor with Poetry against our directory of unmapped binaries:

**poetry lock**
**poetry install**
**poetry shell**
**netwire-config-extractor -d sample-dir/ -o output.ndjson**

This file, output.ndjson , can then be uploaded to Kibana for further analysis.

Check out the Elastic Container project to quick spin up an Elastic Stack and start analyzing structured security-relevant data.

Next time you run into a NETWIRE sample, run it through our configuration extractor to pull out other indicators to help you on your analytic journey or begin remediating quicker.

Detection

YARA

These YARA rules can used to detect and identify NETWIRE RAT.

rule Windows_Trojan_Netwire_1 {
   meta:
       author = "Elastic Security"
       os = "Windows"
       arch = "x86"
       category_type = "Trojan"
       family = "Netwire"
       threat_name = "Windows.Trojan.Netwire"
   strings:
       $a = { 0F B6 74 0C 10 89 CF 29 C7 F7 C6 DF 00 00 00 74 09 41 89 F3 88 5C }
   condition:
       all of them
}
rule Windows_Trojan_Netwire_2 {
   meta:
       author = "Elastic Security"
       os = "Windows"
       arch = "x86"
       category_type = "Trojan"
       family = "Netwire"
       threat_name = "Windows.Trojan.Netwire"
   strings:
       $a1 = "[%.2d/%.2d/%d %.2d:%.2d:%.2d]" fullword
       $a2 = "\\Login Data"
       $a3 = "SOFTWARE\\NetWire" fullword
   condition:
       2 of them
}
rule Windows_Trojan_Netwire_3 {
   meta:
       author = "Elastic Security"
       os = "Windows"
       arch = "x86"
       category_type = "Trojan"
       family = "Netwire"
       threat_name = "Windows.Trojan.Netwire"
   strings:
       $a = { C9 0F 44 C8 D0 EB 8A 44 24 12 0F B7 C9 75 D1 32 C0 B3 01 8B CE 88 44 }
   condition:
       all of them
}
rule Windows_Trojan_Netwire_4 {
   meta:
       author = "Elastic Security"
       os = "Windows"
       arch = "x86"
       category_type = "Trojan"
       family = "Netwire"
       threat_name = "Windows.Trojan.Netwire"
   strings:
       $a1 = "http://%s%ComSpec" ascii fullword
       $a2 = "%c%.8x%s" ascii fullword
       $a3 = "%6\\6Z65dlNh\\YlS.dfd" ascii fullword
       $a4 = "GET %s HTTP/1.1" ascii fullword
       $a5 = "R-W65: %6:%S" ascii fullword
       $a6 = "PTLLjPq %6:%S -qq9/G.y" ascii fullword
   condition:
       4 of them
}

Indicators

All indicators are also available for download in both ECS and STIX format in a combined zip bundle.

The following indicators were discussed in this research.

| Indicator | Type | Note | | -------------------------------------- | ----------- | -------------- | | 139.28.38[.]235 | ipv4-addr | NETWIRE RAT C2 | | 149.102.132[.]253 | ipv4-addr | NETWIRE RAT C2 | | 184.75.221[.]115 | ipv4-addr | NETWIRE RAT C2 | | 185.136.165[.]182 | ipv4-addr | NETWIRE RAT C2 | | 185.140.53[.]139 | ipv4-addr | NETWIRE RAT C2 | | 185.140.53[.]144 | ipv4-addr | NETWIRE RAT C2 | | 185.140.53[.]154 | ipv4-addr | NETWIRE RAT C2 | | 185.140.53[.]61 | ipv4-addr | NETWIRE RAT C2 | | 185.216.71[.]251 | ipv4-addr | NETWIRE RAT C2 | | 194.36.111[.]59 | ipv4-addr | NETWIRE RAT C2 | | 194.5.98[.]126 | ipv4-addr | NETWIRE RAT C2 | | 194.5.98[.]178 | ipv4-addr | NETWIRE RAT C2 | | 194.5.98[.]188 | ipv4-addr | NETWIRE RAT C2 | | 194.5.98[.]65 | ipv4-addr | NETWIRE RAT C2 | | 212.193.29[.]37 | ipv4-addr | NETWIRE RAT C2 | | 212.193.30[.]230 | ipv4-addr | NETWIRE RAT C2 | | 213.152.161[.]249 | ipv4-addr | NETWIRE RAT C2 | | 217.151.98[.]163 | ipv4-addr | NETWIRE RAT C2 | | 23.105.131[.]166 | ipv4-addr | NETWIRE RAT C2 | | 37.0.14[.]199 | ipv4-addr | NETWIRE RAT C2 | | 37.0.14[.]203 | ipv4-addr | NETWIRE RAT C2 | | 37.0.14[.]206 | ipv4-addr | NETWIRE RAT C2 | | 37.0.14[.]208 | ipv4-addr | NETWIRE RAT C2 | | 37.0.14[.]214 | ipv4-addr | NETWIRE RAT C2 | | 37.120.217[.]243 | ipv4-addr | NETWIRE RAT C2 | | 51.161.104[.]138 | ipv4-addr | NETWIRE RAT C2 | | 54.145.6[.]146 | ipv4-addr | NETWIRE RAT C2 | | 80.66.64[.]136 | ipv4-addr | NETWIRE RAT C2 | | 85.209.134[.]105 | ipv4-addr | NETWIRE RAT C2 | | 85.31.46[.]78 | ipv4-addr | NETWIRE RAT C2 | | 94.156.35[.]40 | ipv4-addr | NETWIRE RAT C2 | | 20220627.duckdns[.]org | domain-name | NETWIRE RAT C2 | | admin96.hopto[.]org | domain-name | NETWIRE RAT C2 | | alice2019.myftp[.]biz | domain-name | NETWIRE RAT C2 | | asorock1111.ddns[.]net | domain-name | NETWIRE RAT C2 | | banqueislamik.ddrive[.]online | domain-name | NETWIRE RAT C2 | | betterday.duckdns[.]org | domain-name | NETWIRE RAT C2 | | bigman2021.duckdns[.]org | domain-name | NETWIRE RAT C2 | | blazeblaze.ddns[.]net | domain-name | NETWIRE RAT C2 | | chongmei33.myddns[.]rocks | domain-name | NETWIRE RAT C2 | | clients.enigmasolutions[.]xyz | domain-name | NETWIRE RAT C2 | | gracedynu.gleeze[.]com | domain-name | NETWIRE RAT C2 | | ingobea.hopto[.]org | domain-name | NETWIRE RAT C2 | | iphanyi.edns[.]biz | domain-name | NETWIRE RAT C2 | | iphy.strangled[.]net | domain-name | NETWIRE RAT C2 | | kimlee11.duckdns[.]org | domain-name | NETWIRE RAT C2 | | loffgghh.duckdns[.]org | domain-name | NETWIRE RAT C2 | | megaton.gleeze[.]com | domain-name | NETWIRE RAT C2 | | moran101.duckdns[.]org | domain-name | NETWIRE RAT C2 | | netuwaya.servecounterstrike[.]com | domain-name | NETWIRE RAT C2 | | nowancenorly.ddns[.]net | domain-name | NETWIRE RAT C2 | | podzeye.duckdns[.]org | domain-name | NETWIRE RAT C2 | | podzeye2.duckdns[.]org | domain-name | NETWIRE RAT C2 | | recoveryonpoint.duckdns[.]org | domain-name | NETWIRE RAT C2 | | redlinea[.]top | domain-name | NETWIRE RAT C2 | | roller.duckdns[.]org | domain-name | NETWIRE RAT C2 | | rozayleekimishere.duckdns[.]org | domain-name | NETWIRE RAT C2 | | sani990.duckdns[.]org | domain-name | NETWIRE RAT C2 | | saturdaylivecheckthisout.duckdns[.]org | domain-name | NETWIRE RAT C2 | | uhie.hopto[.]org | domain-name | NETWIRE RAT C2 | | uhie2020.duckdns[.]org | domain-name | NETWIRE RAT C2 | | wcbradley.duckdns[.]org | domain-name | NETWIRE RAT C2 | | xman2.duckdns[.]org | domain-name | NETWIRE RAT C2 | | zonedx.ddns[.]net | domain-name | NETWIRE RAT C2 |

Download netwire-configuration-extractor.tar.gz

For information on the NETWIRE malware check out the following resources:

• NETWIRE Dynamic Configuration Extraction

Getting started

Docker

The recommended and easiest way to get going is to use Docker. From the directory this README is in, you can build a local container.

docker build . -t netwire_loader_config_extractor

Then we run the container with the -v flag to map a host directory to the docker container directory.

docker run -ti --rm -v $(pwd)/data:/data netwire_loader_config_extractor:latest --help

Running it locally

As mentioned above, Docker is the recommended approach to running this project, however you can also run this locally. This project uses Poetry to manage dependencies, testing, and metadata. If you have Poetry installed already, from this directory, you can simply run the following commands to run the tool. This will setup a virtual environment, install the dependencies, activate the virtual environment, and run the console script.

poetry lock
poetry install
poetry shell
netwire-config-extractor --help

Usage

All samples need to be unpacked prior to execution extraction attempts.

Our extractor takes either a directory of samples with -d option or -f for a single sample and then can output parts of the configuration of note, specifically:

• -k : extract the encryption keys
• -c : extract the C2 information
• -s : extract the wide-character strings
• -a : extract the ASCII character strings

docker run -ti --rm -v $(pwd)/data:/data netwire_loader_config_extractor:latest -d "C:\tmp\samples"

You can collect the extracted configurations from the directory you set when running the extractor.

Download qbot-config-extractor.tar.gz

Getting Started

This tool provides a Python module and command line tool that will extract configurations from the QBOT malware samples and dump the results to screen.

For information on the QBOT attack pattern and malware analysis, check out our blog posts detailing this:

• Exploring the QBOT Attack Pattern
• QBOT Malware Analysis

Docker

We can easily run the extractor with Docker, first we need to build the image:

docker build . -t qbot-config-extractor

Then we run the container with the -v flag to map a host directory to the docker container directory:

docker run -ti --rm -v \
"$(pwd)/data":/data qbot-config-extractor:latest -d /data/

We can either specify a single sample with -f option or a directory of samples with -d.

$ docker run -ti --rm -v $(pwd)/data:/data qbot-config-extractor:latest -f data/c2ba065654f13612ae63bca7f972ea91c6fe97291caeaaa3a28a180fb1912b3a

=== Strings ===
# Blob address: 0x100840a0
# Key address: 0x10084040
[0x0]: ProgramData
[0xc]: /t4
[0x10]: EBBA
[0x15]: netstat -nao
[0x22]: jHxastDcds)oMc=jvh7wdUhxcsdt2
[0x40]: schtasks.exe /Create /RU "NT AUTHORITY\SYSTEM" /SC ONSTART /TN %u /TR "%s" /NP /F

...truncated...

=== RESOURCE 1 ===
Key: b'\\System32\\WindowsPowerShel1\\v1.0\\powershel1.exe'
Type: DataType.DOMAINS
41.228.22.180:443
47.23.89.62:995
176.67.56.94:443
103.107.113.120:443
148.64.96.100:443
47.180.172.159:443
181.118.183.98:443

...truncated...

Running it Locally

As mentioned above, Docker is the recommended approach to running this project, however you can also run this locally. This project uses Poetry to manage dependencies, testing, and metadata. If you have Poetry installed already, from this directory, you can simply run the following commands to run the tool. This will setup a virtual environment, install the dependencies, activate the virtual environment, and run the console script.

poetry lock
poetry install
poetry shell
qbot-config-extractor -h

Once that works, you can do the same sort of things as mentioned in the Docker instructions above.

Download icedid-configuration-extractor.tar.gz

For information on the ICEDID malware and network infrastructure, check out the following resources:

• ICEDIDs network infrastructure is alive and well
• ICEDID network infrastructure checking utility

Getting started

Docker

The recommended and easiest way to get going is to use Docker. From the directory this README is in, you can build a local container.

docker build . -t icedid_loader_config_extractor

Then we run the container with the -v flag to map a host directory to the docker container directory.

docker run -ti --rm -v $(pwd)/data:/data icedid_loader_config_extractor:latest --help

Running it locally

As mentioned above, Docker is the recommended approach to running this project, however you can also run this locally. This project uses Poetry to manage dependencies, testing, and metadata. If you have Poetry installed already, from this directory, you can simply run the following commands to run the tool. This will setup a virtual environment, install the dependencies, activate the virtual environment, and run the console script.

poetry lock
poetry install
poetry shell
poetry lock
poetry install
poetry shell
icedid_loader_config_extractor --help

Usage

All samples need to be unpacked prior to execution extraction attempts.

We can either specify a single sample with -f option or a directory of samples with -d.

docker run -ti --rm -v $(pwd)/data:/data icedid_loader_config_extractor:latest -d "C:\tmp\samples"

You can collect the extracted configurations from the directory you set when running the extractor.

Download bpfdoor-config-extractor.tar.gz

Overview

This tool provides a Python module and command line tool that will extract passwords from BPFDoor samples.

The Elastic Security Team has released an indepth analysis of the BPFDoor malware and created an additional tool that will scan for BPFDoor infected hosts.

• BPFDoor analysis
• BPFDoor scanner

Getting Started

Docker

We can easily run the extractor with Docker, first we need to build the image.

Building the BPFDoor Docker image

docker build . -t bpfdoor-extractor

Then we run the container with the -v flag to map a host directory to the Docker container directory that contains the BPFDoor samples.

Running the BPFDoor Docker container

docker run -ti --rm -v $(pwd)/binaries:/binaries \
  bpfdoor-extractor:latest -d /binaries/

We can either specify a single sample with -f option or a directory of samples with -d

BPFDoor Configuration Extractor help output

docker run -ti --rm bpfdoor-extractor:latest -h

Author: Elastic Security (MARE)

______ ______ ______ ______
| ___ \| ___ \|  ___||    \
| |_/ /| |_/ /| |_   | | | | ___    ___    __
| ___ \|  __/ |  _|  | | | |/  \  /  \ | '__|
| |_/ /| |    | |    | |/ /| (_) || (_) || |
\____/ \_|    \_|    |___/  \___/  \___/ |_|
 _____                 __            _____                             
/  __ \               / _|(_)        |  ___|     | |                    | |
| /  \/  ___    __  | |_     __   | |__ __  __| |_   __  __    ___ | |_  ___    __
| |     /  \ | '_ \ |  _|| | / _` | |  __|\ \/ /| __|| '__|/ _` | / __|| __|/  \ | '__|
| \__/\| (_) || | | || |  | || (_| | | |___ >  < | |_ | |  | (_| || (__ | |_| (_) || |
 \____/ \___/ |_| |_||_|  |_| \__, | \____//_/\_\ \__||_|   \__,_| \___| \__|\___/ |_|
                               __/ |
                              |___/


usage: bpfdoor-extractor [-h] (-f FILENAME | -d DIRNAME)

options:
  -h, --help            show this help message and exit
  -f FILENAME, --file FILENAME
                        File
  -d DIRNAME, --dir DIRNAME
                        Directory

Running it Locally

As mentioned above, Docker is the recommended approach to running this project, however you can also run this locally. This project uses Poetry to manage dependencies, testing, and metadata. If you have Poetry installed already, from this directory, you can simply run the following commands to run the tool. This will setup a virtual environment, install the dependencies, activate the virtual environment, and run the console script.

poetry lock
poetry install
poetry shell
bpfdoor-extractor --help

Once that works, you can do the same sort of things as mentioned in the Docker instructions above.

Download parallax-payload-extractor.tar.gz

For information on the PARALLAX malware loader and campaign observations, check out our blog posts detailing this intrusion set.

Getting started

Docker

The recommended and easiest way to get going is to use Docker. From the directory this README is in, you can build a local container.

docker build . -t parallax_loader_payload_extractor

Then we run the container with the -v flag to map a host directory to the docker container directory.

docker run -ti --rm -v $(pwd)/data:/data parallax_loader_payload_extractor:latest --help

Running it locally

As mentioned above, Docker is the recommended approach to running this project, however you can also run this locally. This project uses Poetry to manage dependencies, testing, and metadata. If you have Poetry installed already, from this directory, you can simply run the following commands to run the tool. This will setup a virtual environment, install the dependencies, activate the virtual environment, and run the console script.

poetry lock
poetry install
poetry shell
parallax_loader_payload_extractor --help

Usage

We can either specify a single sample with -f option or a directory of samples with -d. You can use the -o switch to set the output directory of the payloads.

docker run -ti --rm -v $(pwd)/data:/data parallax_loader_payload_extractor:latest -d /data -o /data

You can collect the extracted payloads from the directory you set when running the extractor, the data directory in the root of the extractor in the above example.

Download bpfdoor-scanner.tar.gz

Getting Started

This tool provides a Python script to identify hosts that are infected with the BPFDoor malware.

The Elastic Security Team has released an indepth analysis of the BPFDoor malware and created an additional tool that will extract configurations from BPFDoor malware samples.

• BPFDoor analysis
• BPFDoor configuration extractor

Permissions

On Linux (and thus in a container), the tool requires the following permissions:

• CAP_NET_BIND_SERVICE
• CAP_NET_RAW

On any *NIX host, running the script with sudo will get you what you need. As long as you don’t strip the privileges listed for your container and you publish the UDP port you intend to receive on, you should be set.

Docker

We can easily run the scanner with Docker, first we need to build the image:

Building the BPFDoor scanner Docker image

docker build . -t bpfdoor-scanner

Usage

Once you’be built the Docker iamge, we can run the container to get a list of the options.

Runing the BPFDoor container

docker run -ti --rm bpfdoor-scanner:latest --help

Usage: bpfdoor-scanner [OPTIONS]

  Sends a discovery packet to suspected BPFDoor endpoints.

  Example usage:

      sudo ./bpfdoor-scanner --target-ip 1.2.3.4

  Sends a packet to IP 1.2.3.4 using the default target port 68/UDP (tool
  listens on all ports) using the default interface on this host and listens
  on port 53/UDP to masquerade as traffic.

  NOTE: Elevated privileges are required for source ports < 1024.

Options:
  --target-ip TEXT       [required]
  --target-port INTEGER  [default: 68]
  --source-ip TEXT       IP for target to respond to and attempt to bind
                         locally  [default: 172.17.0.3]
  --source-port INTEGER  Local port to listen on for response  [default: 53]
  --timeout INTEGER      Number of seconds to wait for response  [default: 5]
  -v, --verbose          Show verbose output
  -d, --debug            Show debug output
  --version
  --help                 Show this message and exit.

The minimum required option is just --target-ip. The rest have defaults. For running in a container, you’ll want to publish the return port (defaults to 53) and specify --source-ip of the host interface you wish to use. In the following example, the IP 192.168.100.10 is the interface on my host that will receive the packet.

Example running the BPFDoor scanner

docker run -ti --publish 53:53/udp --rm bpfdoor-scanner:latest \
  --target-ip 192.168.32.18 --source-ip 192.168.100.10

Running Locally

As mentioned above, Docker is the recommended approach to running this project, however you can also run this locally. This project uses Poetry to manage dependencies, testing, and metadata. If you have Poetry installed already, from this directory, you can simply run the following commands to run the tool. This will setup a virtual environment, install the dependencies, activate the virtual environment, and run the console script.

Running BPFDoor scanner locally

poetry lock
poetry install
poetry shell
sudo bpfdoor-scanner --help

Once that works, you can do the same sort of things as mentioned in the Docker instructions above.

Download cobalt-strike-extractor.tar.gz

Overview

This tool provides a Python module and command line tool that will search Elastic Endpoint alert data for detections of Cobalt Strike and the extracted memory data. When present, this tool will extract the implant configuration using the cobaltstrike-config-extractor. The information is then normalized into an ECS-formatted JSON document and indexed into an Elasticsearch cluster or output to the terminal as JSON.

For help on creating Fleet policies to collect and analyze Cobalt Strike beacons in the Elastic Stack, check out our blog posts detailing this:

• Collecting Colbalt Strike beacons
• Extracting Cobalt Strike beacon configurations

Getting Started

Docker

The recommended and easiest way to get going is to use Docker. From the directory this README is in, you can build a local container.

docker build . -t cobalt-strike-extractor

Next, make a copy of config.reference.yml and name it config.local.yml and edit for your environment. A minimal config looks like the example below. The input and output could use the same values, but you can optionally push it to a different cluster for analysis.

## Using an Elastic Cloud instance (this is a randomly generated example)
input.elasticsearch:
  enabled: True
  cloud.id: security-cluster:dXMtd2VzdDEuZ2NwLmNsb3VkLmVzLmlvJGU0MWU1YTc3YmRjNzY2OTY0MDg2NjIzNDA5NzFjNjFkJDdlYjRlYTJkMzJkMTgzYTRiMmJkMjlkNTNjODhjMjQ4
  cloud.auth: elastic:<PASSWORD>

## Default output will use localhost:9092, see reference config
output.elasticsearch:
  enabled: True
  username: elastic
  password: <PASSWORD>

Now, run the container, passing in our local configuration. The -v flag here will add informational messages to the log output. Here, it tells us how many documents were successfully parsed and written.

docker run -ti --rm -v "$(pwd)/config.local.yml:/config.yml" \
    cobalt-strike-extractor:latest -c /config.yml -v

Output:

[2022-01-10T21:33:31.493][INFO] Setting up input/output
[2022-01-10T21:33:31.493][INFO] Connecting to Elasticsearch for input
[2022-01-10T21:33:31.493][INFO] Successfully connected to Elasticsearch for input
[2022-01-10T21:33:31.834][INFO] Connecting to Elasticsearch for output
[2022-01-10T21:33:31.835][INFO] Successfully connected to Elasticsearch for output
[2022-01-10T21:33:33.030][WARNING] Could not parse source as PE file (DOS Header magic not found.)
[2022-01-10T21:33:33.078][WARNING] CobaltStrike Beacon config not found:
[2022-01-10T21:33:33.093][WARNING] Could not parse source as PE file (DOS Header magic not found.)
[2022-01-10T21:33:33.096][WARNING] CobaltStrike Beacon config not found:
[2022-01-10T21:33:33.097][WARNING] Could not parse source as PE file (DOS Header magic not found.)
[2022-01-10T21:33:33.097][WARNING] CobaltStrike Beacon config not found:
[2022-01-10T21:33:33.097][WARNING] Could not parse source as PE file (DOS Header magic not found.)
[2022-01-10T21:33:33.098][WARNING] CobaltStrike Beacon config not found:
[2022-01-10T21:33:33.186][WARNING] Could not parse source as PE file (DOS Header magic not found.)
[2022-01-10T21:33:33.191][WARNING] CobaltStrike Beacon config not found:
[2022-01-10T21:33:33.461][WARNING] Could not parse source as PE file (DOS Header magic not found.)
[2022-01-10T21:33:33.516][WARNING] CobaltStrike Beacon config not found:
[2022-01-10T21:33:33.927][INFO] Wrote 2 docs to Elasticsearch

The [WARNING] messages here are to be expected. These are simply source documents that didn’t contain the configuration information.

Filter by time

To limit the search by time frame, you can add the --since argument, which takes either an ISO-formatted date time string or you can use Elastic date math. For example, to limit search to the last 30 days, you can do the following.

docker run -ti --rm -v "$(pwd)/config.local.yml:/config.yml" \
    cobalt-strike-extractor:latest --since "now-30d/d" -c config.local.yml

Pipe output to other tools

Lastly, you can pipe the output to other commands, such as jq to do local analysis. You can also override the configuration file values using environment variables.

docker run -i --rm -a stdin -a stdout -a stderr \
    -v "$(pwd)/config.local.yml:/config.yml" \
    -e "OUTPUT_ELASTICSEARCH_ENABLED=False" \
    -e "OUTPUT_CONSOLE_ENABLED=True" cobalt-strike-extractor:latest -c /config.yml -q | jq  '.cobaltstrike.server.hostname'

In the example above, we disabled the Elasticsearch output and enabled the Console output using environment variables. We made the output more quiet using the -q flag (hiding the warnings). Then, we used jq to just pull out the “hostname” value of the configuration.

Running it Locally

As mentioned above, Docker is the recommended approach to running this project, however you can also run this locally. This project uses Poetry to manage dependencies, testing, and metadata. If you have Poetry installed already, from this directory, you can simply run the following commands to run the tool. This will setup a virtual environment, install the dependencies, activate the virtual environment, and run the console script.

poetry lock
poetry install
poetry shell
cobalt-strike-extractor --help

Once that works, you can do the same sort of things as mentioned in the Docker instructions above.

Download emotet-configuration-extractor.tar.gz

For information on the EMOTET malware check out the following resources:

• EMOTET Dynamic Configuration Extraction

Getting started

Docker

The recommended and easiest way to get going is to use Docker. From the directory this README is in, you can build a local container.

docker build . -t emotet-config-extractor

Then we run the container with the -v flag to map a host directory to the docker container directory.

docker run -ti --rm -v $(pwd)/data:/data emotet-config-extractor:latest --help

Running it locally

As mentioned above, Docker is the recommended approach to running this project, however you can also run this locally. This project uses Poetry to manage dependencies, testing, and metadata. If you have Poetry installed already, from this directory, you can simply run the following commands to run the tool. This will setup a virtual environment, install the dependencies, activate the virtual environment, and run the console script.

poetry lock
poetry install
poetry shell
emotet-config-extractor --help

Usage

All samples need to be unpacked prior to execution extraction attempts.

Our extractor takes either a directory of samples with -d option or -f for a single sample and then can output parts of the configuration of note, specifically:

• -k : extract the encryption keys
• -c : extract the C2 information
• -s : extract the wide-character strings
• -a : extract the ASCII character strings

docker run -ti --rm -v $(pwd)/data:/data emotet-config-extractor:latest -d "C:\tmp\samples"

You can collect the extracted configurations from the directory you set when running the extractor.

Download blister-config-extractor.tar.gz

Getting Started

This tool provides a Python module and command line tool that will extract configurations from the BLISTER malware loader and dump the results to screen.

For information on the BLISTER malware loader and campaign observations, check out our blog posts detailing this:

• BLISTER Campaign Analysis
• BLISTER Malware Analysis

Docker

We can easily run the extractor with Docker, first we need to build the image:

docker build . -t blister-config-extractor

Then we run the container with the -v flag to map a host directory to the docker container directory:

docker run -ti --rm -v \
"$(pwd)/binaries":/binaries blister-config-extractor:latest -d /binaries/

We can either specify a single sample with -f option or a directory of samples with -d.

Running it Locally

As mentioned above, Docker is the recommended approach to running this project, however you can also run this locally. This project uses Poetry to manage dependencies, testing, and metadata. If you have Poetry installed already, from this directory, you can simply run the following commands to run the tool. This will setup a virtual environment, install the dependencies, activate the virtual environment, and run the console script.

poetry lock
poetry install
poetry shell
blister-config-extractor -h

Once that works, you can do the same sort of things as mentioned in the Docker instructions above.

References

• Customised Rabbit cipher implementation based on Rabbit-Cipher

• The EMOTET developers have changed the way they encode their configuration in the 64bit version of the malware.
• Using code emulation we can bypass multiple code obfuscation techniques.
• The use of code emulators in config extractors will become more prevalent in the future.

To download the EMOTET configuration extractor, check out our post on the tool:

• EMOTET configuration extractor

Preamble

The EMOTET family broke onto the malware scene as a modular banking trojan in 2014, focused on harvesting and exfiltrating bank account information by inspecting traffic. EMOTET has been adapted as an early-stage implant used to load other malware families, such as QAKBOT, TRICKBOT, and RYUK. While multiple EMOTET campaigns have been dismantled by international law enforcement entities, it has continued to operate as one of the most prolific cybercrime operations.

For the last several months, Elastic Security has observed the EMOTET developers transition to a 64-bit version of their malware. While this change does not seem to impact the core functionality of the samples we have witnessed, we did notice a change in how the configuration and strings are obfuscated. In earlier versions of EMOTET, the configuration was stored in an encrypted form in the .data section of the binary. In the newer versions the configuration is calculated at runtime. The information we need to extract the configuration from the binary is thus hidden within the actual code.

In the next sections, we’ll discuss the following as it relates to 64-bit EMOTET samples:

• EMOTET encryption mechanisms
• Reviewing the EMOTET C2 list
• Interesting EMOTET strings
• The EMOTET configuration extractor utility

Encryption keys

EMOTET uses embedded Elliptic Curve Cryptography (ECC) public keys to encrypt their network communication. While in previous versions, the keys would be stored in an XOR-encrypted blob, now the content is calculated at runtime.

In comparison the previous versions of EMOTET would store an encrypted version of the key data in the . text section of the binary.

In order to make it harder for security researchers to find the given code the malware uses Mixed Boolean-Arithmetic (MBA) as one of its obfuscation techniques. It transforms constants and simple expressions into expressions that contain a mix of Boolean and arithmetic operations.

In this example, an array of constants is instantiated, but looking at the assembly we see that every constant is calculated at runtime. This method makes it challenging to develop a signature to target this function.

We noticed that both the Elliptic Curve Diffie-Hellman (ECDH) and Elliptic Curve Digital Signature Algorithm (ECDSA) keys use the same function to decode the contents.

The ECDH key (which you can recognize by its magic ECK1 bytes) is used for encryption purposes while the ECDSA key (ECC1) is used for verifying the C2 server's responses.

By leveraging a YARA signature to find the location of this decode function within the EMOTET binary we can observe the following process:

1. Find the decoding algorithm within the binary.
2. Locate any Cross References (Xrefs) to the decoding function.
3. Emulate the function that calls the decoding function.
4. Read the resulting data from memory.

As we mentioned, we first find the function in the binary by using YARA. The signature is provided at the end of this article. It is worth pointing out that these yara signatures are used to identify locations in the binary but are, in their current form, not usable to identify EMOTET samples.

In order to automatically retrieve the data from multiple samples, we created a configuration extractor. In the snippets below, we will demonstrate, in a high level fashion, how we collect the configuration information from the malware samples.

In the above code snippet:

1. First load the YARA signature.
2. Try to find a match, and if a signature is found in the file.
3. Calculate the function offset based on the offset in the file.

In order to locate the Xrefs to this function, we use the excellent SMDA decompiler. After locating the Xrefs, we can start the emulation process using the CPU emulator, Unicorn.

1. Initialize the Unicorn emulator.
2. Load the executable code from the PE file into memory.
3. Disassemble the function to find the return and the end of the execution.
4. The binary will try to use the windows HeapAlloc API to allocate space for the decoded data. Since we don't want to emulate any windows API's, as this would add unnecessary complexity, we hook to code so that we can allocate space ourselves.
5. After the emulation has run the 64-bit “long size” register (RAX), it will contain a pointer to the key data in memory.
6. To present the key in a more readable way, we convert it to the standard PEM format.

By emulating the parts of the binary that we are interested in, we no longer have to statically defeat the obfuscation in order to retrieve the hidden contents. This approach adds a level of complexity to the creation of config extractors. However, since malware authors are adding ever more obfuscation, there is a need for a generic approach to defeating these techniques.

C2 server list

An important part of tracking malware families is to get new insights by identifying and discovering which C2 servers they use to operate their network.

In the 64-bit versions of EMOTET, we see that the IP and port information of the C2 servers are also dynamically calculated at runtime. Every C2 server is represented by a function that calculates and returns a value for the IP address and the port number.

These functions don’t have a direct cross reference available for searching. However, a procedure references all the C2 functions and creates the p_c2_list array of pointers.

After that, we can emulate every C2-server function individually to retrieve the IP and port combination as seen below.

Strings

The same method is applied to the use of strings in memory. Every string has its own function. In the following example, the function would return a pointer to the string %s\regsvr32.exe "%s".

All of the EMOTET strings share a common function to decode or resolve the string at runtime. In the sample that we are analyzing here, the string resolver function is referenced 29 times.

This allows us to follow the same approach as noted earlier in order to decode all of the EMOTET strings. We pinpoint the string decoding function using YARA, find the cross-references, and emulate the resulting functions.

Configuration extractor

Automating the payload extraction from EMOTET is a crucial aspect of threat hunting as it gives visibility of the campaign and the malware deployed by the threat actors, enabling practitioners to discover new unknown samples in a timely manner.

% emotet-config-extractor --help
usage: Emotet Configuration Extractor [-h] (-f FILE | -d DIRECTORY) [-k] [-c] [-s] [-a]

options:
  -h, --help            show this help message and exit
  -f FILE, --file FILE  Emotet sample path
  -d DIRECTORY, --directory DIRECTORY
                        Emotet samples folder
  -k                    Extract Encryption keys
  -c                    Extract C2 information
  -s                    Extract strings
  -a                    Extract strings (ascii)

Our extractor takes either a directory of samples with -d option or -f for a single sample and then can output parts of the configuration of note, specifically:

• -k : extract the encryption keys
• -c : extract the C2 information
• -s : extract the wide-character strings
• -a : extract the ASCII character stings

EMOTET uses a different routine for decoding wide and ASCII strings. That is why the extractor provides flags to extract them separately.

The C2 information displays a list of IP addresses found in the sample. It is worth noting that EMOTET downloads submodules to perform specific tasks. These submodules can contain their own list of C2 servers. The extractor is also able to process these submodules.

The submodules that we observed do not contain encryption keys. While processing submodules you can omit the -k flag.

[...]
[+] Key type: ECK1
[+] Key length: 32
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE2DWT12OLUMXfzeFp+bE2AJubVDsW
NqJdRC6yODDYRzYuuNL0i2rI2Ex6RUQaBvqPOL7a+wCWnIQszh42gCRQlg==
-----END PUBLIC KEY-----
[...]
[+] Key type: ECS1
[+] Key length: 32
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE9C8agzYaJ1GMJPLKqOyFrlJZUXVI
lAZwAnOq6JrEKHtWCQ+8CHuAIXqmKH6WRbnDw1wmdM/YvqKFH36nqC2VNA==
-----END PUBLIC KEY-----
[...]
[+] Found 64 c2 subs
174.138.33.49:7080
188.165.79.151:443
196.44.98.190:8080
[...]
[+] Starting emulation
[+] String BLOB address: 0x4000000
KeyDataBlob
[...]
[+] String BLOB address: 0x4000000
bcrypt.dll
[...]
[+] String BLOB address: 0x4000000
RNG

To enable the community to further defend themselves against existing and new variants of EMOTET, we are making the payload extractor open source under the Apache 2 License. Access the payload extractor documentation and binary download.

The future of EMOTET

The EMOTET developers are implementing new techniques to hide their configurations from security researchers. These techniques will slow down initial analysis, however, EMOTET will eventually have to execute to achieve its purpose, and that means that we can collect information that we can use to uncover more about the campaign and infrastructure. Using code emulators, we can still find and extract the information from the binary without having to deal with any obfuscation techniques. EMOTET is a great example where multiple obfuscation techniques make static analysis harder. But of course, we expect more malware authors to follow the same example. That is why we expect to see more emulation-based configuration extract in the future.

Detection

YARA

Elastic Security has created YARA rules to identify this activity. The YARA rules shown here are not meant to be used to solely detect EMOTET binaries, they are created to support the configuration extractor. The YARA rules for detecting EMOTET can be found in the protections-artifacts repository.

EMOTET key decryption function

rule resolve_keys
{
meta:
     author = "Elastic Security"
     description = "EMOTET - find the key decoding algorithm in the PE"
     creation_date = "2022-08-02"
     last_modified = "2022-08-11"
     os = "Windows"
     family = "EMOTET"
     threat_name = "Windows.Trojan.EMOTET"
     reference_sample = "debad0131060d5dd9c4642bd6aed186c4a57b46b0f4c69f1af16b1ff9c0a77b1"
   strings:
       $chunk_1 = {
        45 33 C9
        4C 8B D0
        48 85 C0
        74 ??
        48 8D ?? ??
        4C 8B ??
        48 8B ??
        48 2B ??
        48 83 ?? ??
        48 C1 ?? ??
        48 3B ??
        49 0F 47 ??
        48 85 ??
        74 ??
        48 2B D8
        42 8B 04 03
     }
   condition:
       any of them
}

EMOTET C2 aggregation

rule c2_list
{
     author = "Elastic Security"
     description = "EMOTET - find the C2 collection in the PE"
     creation_date = "2022-08-02"
     last_modified = "2022-08-11"
     os = "Windows"
     family = "EMOTET"
     threat_name = "Windows.Trojan.EMOTET"
     reference_sample = "debad0131060d5dd9c4642bd6aed186c4a57b46b0f4c69f1af16b1ff9c0a77b1"
  strings:
     $chunk_1 = {
        48 8D 05 ?? ?? ?? ??
        48 89 81 ?? ?? ?? ??
        48 8D 05 ?? ?? ?? ??
        48 89 81 ?? ?? ?? ??
        48 8D 05 ?? ?? ?? ??
        48 89 81 ?? ?? ?? ??
        48 8D 05 ?? ?? ?? ??
        48 89 81 ?? ?? ?? ??
        48 8D 05 ?? ?? ?? ??
        48 89 81 ?? ?? ?? ??
        48 8D 05 ?? ?? ?? ??
        48 89 81 ?? ?? ?? ??
        48 8D 05 ?? ?? ?? ??
        48 89 81 ?? ?? ?? ??
     }
  condition:
     any of them
}

EMOTET string decoder

rule string_decode
{
   meta:
     author = "Elastic Security"
     description = "EMOTET - find the string decoding algorithm in the PE"
     creation_date = "2022-08-02"
     last_modified = "2022-08-11"
     os = "Windows"
     family = "EMOTET"
     threat_name = "Windows.Trojan.EMOTET"
     reference_sample = "debad0131060d5dd9c4642bd6aed186c4a57b46b0f4c69f1af16b1ff9c0a77b1"
  strings:
     $chunk_1 = {
        8B 0B
        49 FF C3
        48 8D 5B ??
        33 CD
        0F B6 C1
        66 41 89 00
        0F B7 C1
        C1 E9 10
        66 C1 E8 08
        4D 8D 40 ??
        66 41 89 40 ??
        0F B6 C1
        66 C1 E9 08
        66 41 89 40 ??
        66 41 89 48 ??
        4D 3B D9
        72 ??
     }
     $chunk_2 = {
        8B 0B
        49 FF C3
        48 8D 5B ??
        33 CD
        0F B6 C1
        66 41 89 00
        0F B7 C1
        C1 E9 ??
        66 C1 E8 ??
        4D 8D 40 ??
        66 41 89 40 ??
        0F B6 C1
        66 C1 E9 ??
        66 41 89 40 ??
        66 41 89 48 ??
        4D 3B D9
        72 ??
     }
  condition:
     any of them
}

TRaDE is responsible for the detection and endpoint behavior security rules that power Elastic’s XDR capabilities. While our detection rules provide visibility to adversary behaviors, the endpoint behavior rules have the capability to prevent an attack. These rules provide protection logic used by Elastic Endpoint Security to stop threats on Windows, Linux, and MacOS endpoints. Collectively, Elastic Security supports a wide range of platforms and data sources (e.g., core cloud service providers, K8s, core operating systems, etc.).

The two rulesets: a) detection rules and b) endpoint behavior rules, consider different use cases and complement each other to provide robust coverage. The comparison table highlights unique differences between the two in terms of protection design goals, how data is processed, and which data is processed.

Detection Rules- Design Goals: Provide the most robust detection coverage of all threats, leveraging all data sources available. Some tuning of rules based on organization-specific environments is expected. - Data Streams: Will search across all specified indexes per rule within a Stack. - Engine Processing: Batch process.

Endpoint Behavior- Design Goals: Provide very high confidence, prevention-focused, minimal tuning at the expense of false negatives on a per-rule basis. We want every organization to be able to enable behavior protection and have a great experience out of the box, with little tuning required. - Data Streams: Agent searches data on the endpoint. - Engine Processing: Real time data streaming.

Behind the TRaDE crafting curtains, we leverage openly available tools to develop and test our rulesets. If you want a primer on writing Event Query Language (EQL) rules, want to generate suspicious activity to baseline your Elastic-powered detections, or quickly export those suspicious events from Elasticsearch, you may benefit from some of the tools we use. Section 1 introduces our security SIEM features via the EQLPlayground, section 2 discusses our rule testing capability RTA, and section 3 highlights our detection-rules CLI and a few valuable commands we use.

EQLPlayground

EQL was developed to express relationships between events, and, coupled with ECS, has the power to quickly correlate events across disparate data sources. Whether you want to perform a simple search with EQL, leverage advanced data stacking and filtering to discover anomalies, or define a complex hypothesis-based hunt query, EQL’s flexibility as a language can help improve your team’s effectiveness in many ways. The language is heavily used (in addition to several other language options, enabling users to leverage the most relevant and applicable features) throughout our detection-rules repo and endpoint behavior artifacts to detect adversary behaviors and express relationships between events.

While we strive to achieve feature parity between endpoint and elasticsearch EQL implementations to the extent possible, there are minor functional differences due to architectural implementations.

While reading about EQL can be very informative, playing with the query language is a much more fun and interactive learning experience! Thanks to Elastic’s own James Spiteri, you can immediately dive into an Elastic Cloud Stack and learn using the EQLPlaygound. The playground takes advantage of the native Security Timeline correlation capabilities, and provides notes to enable learning EQL. The playground is a publicly available Elastic Security instance, pre-populated with suspicious events generated from a Sofacy group payload. The only thing you need to access the site is a browser!

Essentially, you’re presented with a dataset representative of threat activity, similar to what we rely on to build our detection rules and endpoint artifacts. This event data can then be leveraged to generate your own detection logic. It also provides a small introduction to the Elastic Security Stack, and gives you an opportunity to play with some of the cool features available (e.g. Analyzer). The visual event Analyzer shows a graphical representation of a process tree, containing alerts and suspicious events detected by our Elastic Security Endpoint, and illustrates process lineage that can be used within a query.

We can use this information to understand how the adversary behavior works, and develop a query capable of identifying future malicious activity. For example, should Outlook spawn an explorer.exe child process? Explore the EQLPlayground, EQL syntax, and APIs. In the correlation view introduced with Elastic Security 7.12, you’ll have the opportunity to insert EQL and develop a query with your special sauce to detect the malicious behavior we’ve executed. You’ll also be able to look at each available field, and the data stream required to capture these events within your Stack.

As you can see, there is an example placeholder query, but you have full access to modify the query based on the full event captured and come up with the best detection. Is there something suspicious about the process tree? What about the sequence of events? Is there something fishy about rundll32.exe (a commonly used execution proxy) making external network calls?

sequence by process.entity_id with maxspan=10s
[process where process.name : "rundll32.exe" and event.type == "start"]
[network where process.name : "rundll32.exe" and not cidrmatch(destination.ip, "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "127.0.0.0/8")]

We’d love to see what cool and clever queries you’ve come up with, and if you have ideas for new rules, check out our CONTRIBUTING.md guide and submit a new rule. For now, we’ll use this query in creating a rule with the detection-rule CLI.

Red Team Automation (RTA)

One of the ways we automate testing Elastic’s ruleset is by launching RTA scripts that simulate threat behaviors. If you are unfamiliar with RTA, it is an open-source tool used by TRaDE to generate suspicious activity and unit test rules across multiple Stack releases. We encourage you to check out the 2018 post by Devon Kerr, which introduced the capability.

Sometimes folks ask our team for sample data, methods to generate suspicious events to baseline configurations, or a testing environment with many alerts already generated in the Elastic Stack. We also regression test rules to validate new features added to the SIEM or Endpoint agent, any modifications based on rule tuning, or for maintenance. This process can become time-consuming with hundreds of rules to test across multiple Stack versions.

In the latest 8.4 dev cycle, we spent some time generating new macOS, Linux, and Windows RTAs. Consistent with the openness theme, we migrated our endpoint behavior tests to the Detection Rules repo for the community! Current RTA development is focused on endpoint behavior, and we continue to expand the coverage of our rulesets with new RTAs, so look forward to even more RTAs in the not-too-distant future.

Once you’ve cloned the detection-rules repo, you’ll be able to list all available tests. Each RTA includes helpful metadata like the platform the RTA supports, the triggered rules that will alert, and the python code that generates suspicious activity on the target system. The common import is packed with useful functions to simplify creating new RTAs. For example, it provides helper functionality to temporarily edit the Windows registry, check the required operating system is running the RTA, or even execute terminal commands. Essentially, it abstracts a lot of the common activity needed across the RTA set in order to simplify the development of new RTAs, especially for those less familiar with python. The RTA library was designed to use only stdlib Python packages so that no external dependencies would be required. Using only core libraries is greatly beneficial when testing in segmented environments.

In the above example, the RTA generates activity to trigger the Suspicious Emond Child Process SIEM and Potential Persistence via Emond endpoint behavior rules. The RTA creates a bash shell process spawned from a parent process called emond. We aim to make repeatable yet non-destructive test cases to reuse testing infrastructure as much as possible between unit tests. There are many approaches to generating suspicious events that would trigger these rules, so if you’d like to contribute your creative ideas, feel free to submit a pull request to the detection-rules!

Detection Rules CLI

The detection-rules CLI is a development tool swiss-army-knife that we use to manage and test whether our rules pass validation, but there are useful commands that you can use to speed up rule testing in your own environment. If you’re familiar with Python3, getting started with the Detection Rules CLI commands will only take a few steps. It has useful commands like view-rule , which shows the rule as a JSON object in the format expected by Kibana. Conveniently, the command also validates while loading; if you ever want to test that your TOML file matches our schema quickly, you can use this command.

After you have installed the package dependencies and your credential configuration, you’re ready to use the CLI. One of the cool things about using the CLI is the ability to download data while testing an RTA using the collect-events command.

Once you start collecting events, the CLI command will idle until you're ready to save events. While you wait, you have an opportunity to jump onto the target machine, execute an RTA, detonate a malware sample, or launch any payloads to trigger an alert. These events can be stored offline and reused later in an automated testing process. With the collect-events command, you can apply several options that scope your exports, like specifying the index and specific host.id of the target system you want. Once the command starts, it gathers all events associated with the host until you’re ready to stop the collection.

As you can see, it’s possible to run the collect-events command, generate malicious activity on a target system (e.g., using an RTA), and download the events locally for review. Some users export and use these events as-is, but we intend to store these events to help automate and streamline our end-to-end testing process.

Apart from the es (Elasticsearch) function, we often use several other options like linting our ruleset with toml-lint , validating our rules with validate-all , or even surveying out ruleset against alerts with in-development commands buried deep within our dev CLI section like rule-survey. If you’re interested in reading more about the other fields available, see our guide on creating a rule with the CLI or the CLI.md. As always, if you have any questions or need help, feel free to submit an issue.

Tools like the EQLPlaygound, RTAs, and detection-rules CLI are great resources for getting started with EQL, threat hunting, and detection engineering respectively. Coupled with the detection-rules CLI and RTAs, these tools give security research engineers immediate feedback to begin managing their custom Elastic detection rules. Whether you’re using a cloud Elastic Stack, a local deployment, or are setting up a lab environment with our newly released Elastic Container Project, we’ve got you covered. These are just a few tools we use that you're welcome to try out for your internal workflows, they help us test and create rules every day.

In a following article of TRaDE craft, we’ll describe how we validate our rules across languages like EQL or KQL, and how we automate our end-to-end process. Additionally, if you’re interested in hearing how our partners at Tines have integrated Elastic detection logic, check out their blog on Automating Detection-as-Code, which walks through the Elastic SIEM, detection content development CI/CD, alert management, and response handling.

Update: RTAs have relocated to the Cortado centralized repository which now has the option to use the packages RTAs whl available under releases. See the README for more details.

We’re always interested in hearing use cases and workflows like these, so as always, reach out to us via GitHub issues, chat with us in our community Slack, and ask questions in our Discuss forums!

In this post, we'll walk through manually analyzing a Cobalt Strike C2 configuration from a binary beacon payload using the excellent Cobalt Strike Configuration Extractor (CSCE). We'll also cover enabling some newer features of the Elastic Stack that will allow you to do this at scale across all your monitored endpoints, by extracting the beacons from memory.

The team at Blackberry has a tremendous handbook called “Finding Beacons in the Dark” (registration required) that dives extensively into Cobalt Strike beacon configurations. We’ll discuss a few fields in the configurations here, but if you’re interested in learning about how beacons function, we strongly recommend checking that resource out.

Cobalt Strike Configuration Extractor

The Cobalt Strike Configuration Extractor (CSCE) by Stroz Friedberg is a "python library and set of scripts to extract and parse configurations from Cobalt Strike beacons".

To use the CSCE, we'll create a Python virtual environment, activate it, and install the CSCE Python package.

Setting up the Cobalt Strike Configuration Extractor

$ python3 -m venv csce

$ source csce/bin/activate

(csce) $ pip3 install libcsce

...truncated...
Collecting libcsce
  Using cached libcsce-0.1.0-py3-none-any.whl (24 kB)
Collecting pefile>=2019.4.18
...truncated...

Next, we can run the CSCE on the beacon payload we extracted from memory to see if there's any interesting information stored we can collect (we'll add the --pretty flag to make the output easier to read as a JSON document).

Viewing the atomic indicators of the CS beacon configuration

(csce) $ csce --pretty beacon.exe

{
  "beacontype": [
    "HTTPS"
  ],
  "sleeptime": 45000,
  "jitter": 37,
  "maxgetsize": 1403644,
  "spawnto": "GNEtW6h/g4dQzm0dOkL5NA==",
  "license_id": 334850267,
  "cfg_caution": false,
  "kill_date": "2021-12-24",
  "server": {
    "hostname": "clevelandclinic[.]cloud",
    "port": 443,
    "publickey": "MIGfMA0GCSqGSIb3DQEBAQUAA4G...
...truncated...

Immediately, we can see that the beacon uses HTTPS to communicate and that the domain is clevelandclinic[.]cloud. This gives us an atomic indicator that we can do some analysis on. Looking at the Malleable Command and Control documentation, we can get a description of the configuration variables.

As an example, we can see that the sleeptime is 450000 milliseconds, which changes the default beacon check in from every 60-seconds to 450-seconds, or 7 ½ minutes. Additionally, we see a jitter of 37 meaning that there is a random jitter of 37% of 450000 milliseconds (166,500 milliseconds), so the beacon check-in could be between 283,000 and 450,000 milliseconds (4.7 - 7.5 minutes).

Additionally, the publickey field is used by the Cobalt Strike Team Server to encrypt communications between the server and the beacon. This is different from normal TLS certificates used when accessing the C2 domain with a browser or data-transfer libraries, like cURL. This field is of note because the Team Server uses the same publickey for each beacon, so this field is valuable in clustering beacons with their perspective Team Server because threat actors often use the same Team Server for multiple campaigns, so this data from the configuration can be used to link threat actors to multiple campaigns and infrastructure.

Continuing to look at the configuration output, we can see another interesting section around the process-inject nested field, stub:

Viewing the process-inject.stub field

(csce) $ csce --pretty beacon.exe

...truncated...
  "process-inject": {
    "allocator": "NtMapViewOfSection",
    "execute": [
      "CreateThread 'ntdll!RtlUserThreadStart'",
      "CreateThread",
      "NtQueueApcThread-s",
      "CreateRemoteThread",
      "RtlCreateUserThread"
    ],
    "min_alloc": 17500,
    "startrwx": false,
    "stub": "IiuPJ9vfuo3dVZ7son6mSA==",
    "transform-x86": [
      "prepend '\\x90\\x90'"
    ],
...

The stub field contains the Base64 encoded MD5 file hash of the Cobalt Strike Java archive. To convert this, we can again use CyberChef, this time add the "From Base64" and "To Hex" recipes, ensure you change the "Delimiter" to "None" in the "To Hex" recipe.

Now that we have the MD5 value of the Java archive (222b8f27dbdfba8ddd559eeca27ea648), we can check that against online databases like VirusTotal to get additional information, specifically, the SHA256 hash (7af9c759ac78da920395debb443b9007fdf51fa66a48f0fbdaafb30b00a8a858).

Finally, we can verify the SHA256 hash with CobaltStrike to identify the version of the Java archive by going to https://verify.cobaltstrike.com and searching for the hash.

Now we know that this beacon was created using a licensed version of Cobalt Strike 4.4.

Another field from the configuration that is helpful in clustering activity is the license_id field.

Viewing Cobalt Strike watermark

...truncated
  "spawnto": "GNEtW6h/g4dQzm0dOkL5NA==",
  "license_id": 334850267,
  "cfg_caution": false,
...truncated...

This is commonly referred to as the Watermark and is a 9-digit value that is unique per license. While this value can be modified, it can still be used in conjunction with the process-inject.stub and publickey fields (discussed above) to cluster infrastructure and activity groups.

These are just a few fields that can be used to identify and cluster activities using configurations extracted from the Cobalt Strike beacon. If you're interested in a very in-depth analysis of the configuration, we recommend you check out the Finding Beacons in the Dark Cobalt Strike handbook by the team at Blackberry.

Putting Analysis to Action

To test out our analyst playbook for collecting Cobalt Strike beacon payloads, their configurations, and metadata contained within; we can apply those to more data to identify clusters of activity.

In the above illustration, we can cluster threat actors based on their shared uses of the beacon payload public key, which as we described above, is unique per Team Server. This would allow us to group multiple beacon payload hashes, infrastructure, and campaigns to a single Threat Actor.

As always, using the atomic indicators extracted from the beacon payload configurations (clevelandclinic[.]cloud in our example) allow you to identify additional shared infrastructure, target verticals, and threat actor capabilities.

This time at full speed

All of the steps that we've highlighted in this release, as well as the previous release, can be automated and written into Elasticsearch using the Cobalt Strike Beacon Extraction project.

Summary

In this post, we highlighted new features in the Elastic Stack that can be used to collect Cobalt Strike Malleable C2 beacon payloads. Additionally, we covered the processes to build Fleet policies to extract beacon payloads from memory and their configurations.

These Fleet policies and processes enable security analysts to collect Cobalt Strike beacon payloads and their configurations to identify threat actor controlled infrastructure and cluster activity.

Artifacts

Observable | Type | Note -------------------------------------------------------------------|-------------|------------------------------------------ 697fddfc5195828777622236f2b133c0a24a6d0dc539ae7da41798c4456a3f89 | SHA256 | Cobalt Strike Malleable C2 beacon payload 7475a6c08fa90e7af36fd7aa76be6e06b9e887bc0a6501914688a87a43ac7ac4 | SHA256 | Cobalt Strike Malleable C2 beacon payload f9b38c422a89d73ebdab7c142c8920690ee3a746fc4eea9175d745183c946fc5 | SHA256 | Cobalt Strike Malleable C2 beacon payload clevelandclinic[.]cloud | domain-name | Cobalt Strike Malleable C2 domain 104[.]197[.]142[.]19 | ipv4-addr | Cobalt Strike Malleable C2 IP address 192[.]64[.]119[.]19 | ipv4-addr | Cobalt Strike Malleable C2 IP address

Artifacts

Artifacts are also available for download in both ECS and STIX format in a combined zip bundle.