We’re excited to introduce a new chapter in our security bounty program on HackerOne that we soft launched in December 2024. Elastic is now offering a unique opportunity for researchers to test our detection rules (SIEM) and endpoint rules (EDR), helping to identify gaps, vulnerabilities, and areas for improvement. This program builds on the success of our existing collaboration with the security research community, with a fresh focus on external validation for SIEM and EDR rule protections, which are provided as prebuilt content for Elastic Security and deeply connected to the threat research published on Elastic Security Labs.

At Elastic, openness has always been at the core of our philosophy. We prioritize being transparent about how we protect our users. Our protections for SIEM and EDR are not hidden behind a curtain or paywall. Anyone can examine and provide immediate feedback on our protections. This feedback pipeline has proven to be a powerful enabler to refine and improve, while fostering collaboration with security professionals worldwide.

While we have performed various forms of testing internally over the years, some of which still exist today — such as emulations via internal automation capabilities, unit tests, evaluations, smoke tests, peer review processes, pen tests, and participating in exercises like Locked Shields, we want to take it one step further. By inviting the global security community to test our rules, we plan to push the maturity of our detection capabilities forward and ensure they remain resilient against evolving adversary techniques.

Elastic’s security bug bounty program offering

Elastic maintains a mature and proactive public bug bounty program, launched in 2017 which has paid out over $600,000 in awards since then. We value our continued partnership with the security research community to maintain the effectiveness of these artifacts, shared with the community to identify known and newly-discovered threats.

The scope of our bounty has included Elastic’s development supply chain, Elastic Cloud, the Elastic Stack, our product solutions, and our corporate infrastructure. This initiative provides researchers with additional guided challenges and bonus structures that will contribute directly to hardening our security detection solutions.

A new bounty focus: Elastic Security rule assessments

This latest offering marks an exciting shift by expanding the scope of our bounty program to specifically focus on detection rulesets for the first time. While bounties have traditionally targeted vulnerabilities in products and platforms, this program invites the community to explore new ground: testing for evasion and bypass techniques that affect our rules.

By initially targeting rules for Windows endpoints, this initiative creates an opportunity for the security community to showcase creative ways of evading our defenses. The focus areas for this period include key MITRE ATT&CK techniques.

Why this is important

Elastic has consistently collaborated with our community, particularly through our community Slack, where members regularly provide feedback on our detection rules. This new bounty program doesn’t overshadow the incredible contributions already made: it adds another layer of involvement, offering a structured way to reward those who have dedicated time and effort to help us and our community defend against threats of all kinds.

By expanding our program to include detection rulesets, we’re offering researchers the chance to engage in a way that has a direct impact on our defenses. We demonstrate our belief in continuous improvement, ensuring we stay ahead of adversaries, and lead the industry in creative, yet exciting ways.

Summary scope and rewards

For this initial offering, the bounty scope focuses on evasion techniques related to our detection (SIEM) and endpoint (EDR) rulesets, particularly for Windows. We are interested in submissions that focus on areas like:

• Privilege evasion: Techniques that bypass detection without requiring elevated privileges
• MITRE ATT&CK technique evasion: Creative bypasses of detection rules for specific techniques such as process injection, credential dumping, creative initial/execution access, lateral movement, and others

Submissions will be evaluated based on their impact and complexity. Over time, we plan the scope will evolve so watch out for future announcements and the Hackerone offering.

For a full list of techniques and detailed submission guidelines, view current offering.

Time bounds

For this bounty incubation period (Jan 28th 2025 - Sept 1 2025), the scope will be Windows Behavior Alerts.

Current offering

Behavior detections

Elastic invites the security community to contribute to the continuous improvement of our detection (SIEM) and endpoint (EDR) rulesets. Our mission is to enhance the effectiveness and coverage of these rulesets, ensuring they remain resilient against the latest threats and sophisticated techniques. We encourage hackers to identify gaps, bypasses, or vulnerabilities in specific areas of our rulesets as defined in the scope below.

What we’re looking for

We are particularly interested in submissions that focus on:

• Privileges: Priority is given to bypass and evasion techniques that do not require elevated privileges.
• Techniques Evasion: If a submission bypasses a single behavior detection but still triggers alerts, then it is not considered as a full bypass.

Submissions will be evaluated based on their impact and complexity. The reward tiers are structured as follows:

• Low: Alerts generated are only low severity
• Medium: No alerts generated (SIEM or Endpoint)
• High: —
• Critical: —

Rule definition

To ensure that submissions are aligned with our priorities, each offering under this category will be scoped to a specific domain, MITRE tactic, or area of interest. This helps us focus on the most critical areas while preventing overly broad submissions.

General examples of specific scopes offered at specific times might include:

• Endpoint Rules: Testing for bypasses or privilege escalation rules within macOS, Linux, Windows platforms.
• Cloud Rules: Assessing the detection capabilities against identity-based attacks within AWS, Azure, GCP environments.
• SaaS Platform Rules: Validating the detection of OAuth token misuse or API abuse in popular SaaS applications.

Submission guidelines

To be eligible for a bounty, submissions must:

1. Align with the Defined Scope: Submissions should strictly adhere to the specific domain, tactic, or area of interest as outlined in the bounty offering.
2. Provide Reproducible Results: Include detailed, step-by-step instructions for reproducing the issue.
3. Demonstrate Significant Impact: Show how the identified gap or bypass could lead to security risks while not triggering any SIEM or EDR rules within the scope of the Feature Details.
4. Include Comprehensive Documentation: Provide all necessary code, scripts, or configurations used in the testing process to ensure the issue can be independently validated. The submission includes logs, screenshots, or other evidence showing that the attack successfully bypassed specific rules without triggering alerts, providing clear proof of the issue.

Feature details scope

For this offering, here are additional details to further scope down submissions for this period:

• Target: Windows Behavior Alerts
• Scenario
  • Goal: Gain execution of an arbitrary attacker delivered executable on a system protected by Elastic Defend without triggering any alerts
  • Story: User downloads a single non-executable file from their web browser and opens it. They may click through any security warnings that are displayed by the operating system
  • Extensions in scope: lnk, js, jse, wsf, wsh, msc, vbs, vbe, chm, psc1, rdp
  • Entire scenario must occur within 5 minutes, but a reboot is allowed
• Relevant MITRE Techniques:
  • Process Injection, Technique T1055 - Enterprise | MITRE ATT&CK® into Windows processes
  • Lateral Movement via Remote Services, Technique T1021 - Enterprise | MITRE ATT&CK® and credentials
  • Phishing: Spearphishing Attachment, Sub-technique T1566.001 - Enterprise | MITRE ATT&CK® (macro enabled docs, script, shortcuts etc.)
  • Impair Defenses: Disable or Modify Tools, Sub-technique T1562.001 - Enterprise | MITRE ATT&CK® (tampering with agents without administrative privileges techniques or techniques related to tampering with Elastic agent, PPL bypass, BYOVD etc.)
• Additional Success Criteria:
  • Ideally the bypasses can be combined in one chain (e.g. one payload performing multiple techniques and bypassing multiple existing rules scoped for the same techniques) - to avoid bypasses based solely on our public FP exclusions.
  • For phishing-based initial access techniques, submissions must clearly specify the delivery method, including how the target receives and interacts with the payload (e.g., email attachment, direct download, or cloud file sharing).
• Additional Exclusions:

Here are some examples of non-acceptable submissions, but not limited to:

• Techniques that rely on small x-process WriteProcessMemory
• Techniques that rely on sleeps or other timing evasion methods
• Techniques that rely on kernel mode attacks and require administrative privileges
• Techniques that rely on Phishing, Technique T1566 - Enterprise | MITRE ATT&CK® that are user assisted beyond initial access (e.g. beyond 2 or more user clicks)
• Techniques that rely on well-documented information already in public repositories or widely recognized within the security community without any novel evasion or modification.
• Techniques that rely on legacy / unpatched systems
• Techniques that rely on highly specific environmental conditions or external factors that are unlikely to occur in realistic deployment scenarios
• Techniques that rely on rule exceptions
• Techniques that require local administrator.
• Code injection techniques that rely on small payload size (less than 10K bytes)
• Techniques that rely on less than 10,000 bytes written at a time through a cross process WriteProcessMemory

Questions and disclosure

Please view our Security Issues page for any questions or concerns related to this offering.

How to get involved

To participate and learn more, head over to HackerOne for complete details on the bounty program, submission guidelines, and reward tiers. We look forward to seeing the contributions from the research community and using these findings to continuously enhance the Elastic Security rulesets. Sign up for a free cloud trial to access Elastic Security!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Last week, security researcher Mika Ayenson authored a publication highlighting potential detection strategies and an LLM content auditing prototype solution via a proxy implemented during Elastic’s OnWeek event series. This post highlighted the importance of research pertaining to the safety of LLM technology implemented in different environments, and the research focus we’ve taken at Elastic Security Labs.

Given Elastic's unique vantage point leveraging LLM technology in our platform to power capabilities such as the Security AI Assistant, our desire for more formal detection rules, integrations, and research content has been growing. This publication highlights some of the recent advancements we’ve made in LLM integrations, our thoughts around detections aligned with industry standards, and ECS field mappings.

We are committed to a comprehensive security strategy that protects not just the direct user-based LLM interactions but also the broader ecosystem surrounding them. This approach involves layers of security detection engineering opportunities to address not only the LLM requests/responses but also the underlying systems and integrations used by the models.

These detection opportunities collectively help to secure the LLM ecosystem and can be broadly grouped into five categories:

1. Prompt and Response: Detection mechanisms designed to identify and mitigate threats based on the growing variety of LLM interactions to ensure that all communications are securely audited.
2. Infrastructure and Platform: Implementing detections to protect the infrastructure hosting LLMs (including wearable AI Pin devices), including detecting threats against the data stored, processing activities, and server communication.
3. API and Integrations: Detecting threats when interacting with LLM APIs and protecting integrations with other applications that ingest model output.
4. Operational Processes and Data: Monitoring operational processes (including in AI agents) and data flows while protecting data throughout its lifecycle.
5. Compliance and Ethical: Aligning detection strategies with well-adopted industry regulations and ethical standards.

Securing the LLM Ecosystem: five categories

Another important consideration for these categories expands into who can best address risks or who is responsible for each category of risk pertaining to LLM systems.

Similar to existing Shared Security Responsibility models, Elastic has assessed four broad categories, which will eventually be expanded upon further as we continue our research into detection engineering strategies and integrations. Broadly, this publication considers security protections that involve the following responsibility owners:

• LLM Creators: Organizations who are building, designing, hosting, and training LLMs, such as OpenAI, Amazon Web Services, or Google
• LLM Integrators: Organizations and individuals who integrate existing LLM technologies produced by LLM Creators into other applications
• LLM Maintainers: Individuals who monitor operational LLMs for performance, reliability, security, and integrity use-cases and remain directly involved in the maintenance of the codebase, infrastructure, and software architecture
• Security Users: People who are actively looking for vulnerabilities in systems through traditional testing mechanisms and means. This may expand beyond the traditional risks discussed in OWASP’s LLM Top 10 into risks associated with software and infrastructure surrounding these systems

This broader perspective showcases a unified approach to LLM detection engineering that begins with ingesting data using native Elastic integrations; in this example, we highlight the AWS Bedrock Model Invocation use case.

Integrating LLM logs into Elastic

Elastic integrations simplify data ingestion into Elastic from various sources, ultimately enhancing our security solution. These integrations are managed through Fleet in Kibana, allowing users to easily deploy and manage data within the Elastic Agent. Users can quickly adapt Elastic to new data sources by selecting and configuring integrations through Fleet. For more details, see Elastic’s blog on making it easier to integrate your systems with Elastic.

The initial ONWeek work undertaken by the team involved a simple proxy solution that extracted fields from interactions with the Elastic Security AI Assistant. This prototype was deployed alongside the Elastic Stack and consumed data from a vendor solution that lacked security auditing capabilities. While this initial implementation proved conceptually interesting, it prompted the team to invest time in assessing existing Elastic integrations from one of our cloud provider partners, Amazon Web Services. This methodology guarantees streamlined accessibility for our users, offering seamless, one-click integrations for data ingestion. All ingest pipelines conform to ECS/OTel normalization standards, encompassing comprehensive content, including dashboards, within a unified package. Furthermore, this strategy positions us to leverage additional existing integrations, such as Azure and GCP, for future LLM-focused integrations.

Vendor selection and API capabilities

When selecting which LLM providers to create integrations for, we looked at the types of fields we need to ingest for our security use cases. For the starting set of rules detailed here, we needed information such as timestamps and token counts; we found that vendors such as Azure OpenAI provided content moderation filtering on the prompts and generated content. LangSmith (part of the LangChain tooling) was also a top contender, as the data contains the type of vendor used (e.g., OpenAI, Bedrock, etc.) and all the respective metadata. However, this required that the user also have LangSmith set up. For this implementation, we decided to go with first-party supported logs from a vendor that provides LLMs.

As we went deeper into potential integrations, we decided to land with AWS Bedrock, for a few specific reasons. Firstly, Bedrock logging has first-party support to Amazon CloudWatch Logs and Amazon S3. Secondly, the logging is built specifically for model invocation, including data specific to LLMs (as opposed to other operations and machine learning models), including prompts and responses, and guardrail/content filtering. Thirdly, Elastic already has a robust catalog of integrations with AWS, so we were able to quickly create a new integration for AWS Bedrock model invocation logs specifically. The next section will dive into this new integration, which you can use to capture your Bedrock model invocation logs in the Elastic stack.

Elastic AWS Bedrock model integration

Overview

The new Elastic AWS Bedrock integration for model invocation logs provides a way to collect and analyze data from AWS services quickly, specifically focusing on the model. This integration provides two primary methods for log collection: Amazon S3 buckets and Amazon CloudWatch. Each method is optimized to offer robust data retrieval capabilities while considering cost-effectiveness and performance efficiency. We use these LLM-specific fields collected for detection engineering purposes.

Note: While this integration does not cover every proposed field, it does standardize existing AWS Bedrock fields into the gen_ai category. This approach makes it easier to maintain detection rules across various data sources, minimizing the need for separate rules for each LLM vendor.

Configuring integration data collection method

Collecting logs from S3 buckets

This integration allows for efficient log collection from S3 buckets using two distinct methods:

• SQS Notification: This is the preferred method for collecting. It involves reading S3 notification events from an AWS Simple Queue Service (SQS) queue. This method is less costly and provides better performance compared to direct polling.
• Direct S3 Bucket Polling: This method directly polls a list of S3 objects within an S3 bucket and is recommended only when SQS notifications cannot be configured. This approach is more resource-intensive, but it provides an alternative when SQS is not feasible.

Collecting logs from CloudWatch

Logs can also be collected directly from CloudWatch, where the integration taps into all log streams within a specified log group using the filterLogEvents AWS API. This method is an alternative to using S3 buckets altogether.

Integration installation

The integration can be set up within the Elastic Agent by following normal Elastic installation steps.

1. Navigate to the AWS Bedrock integration
2. Configure the queue_url for SQS or bucket_arn for direct S3 polling.

Configuring Bedrock Guardrails

AWS Bedrock Guardrails enable organizations to enforce security by setting policies that limit harmful or undesirable content in LLM interactions. These guardrails can be customized to include denied topics to block specific subjects and content filters to moderate the severity of content in prompts and responses. Additionally, word and sensitive information filters block profanity and mask personally identifiable information (PII), ensuring interactions comply with privacy and ethical standards. This feature helps control the content generated and consumed by LLMs and, ideally, reduces the risk associated with malicious prompts.

Note: other guardrail examples include Azure OpenAI’s content and response filters, which we aim to capture in our proposed LLM standardized fields for vendor-agnostic logging.

When LLM interaction content triggers these filters, the response objects are populated with amazon-bedrock-trace and amazon-bedrock-guardrailAction fields, providing details about the Guardrails outcome, and nested fields indicating whether the input matched the content filter. This response object enrichment with detailed filter outcomes improves the overall data quality, which becomes particularly effective when these nested fields are aligned with ECS mappings.

The importance of ECS mappings

Field mapping is a critical part of the process for integration development, primarily to improve our ability to write broadly scoped and widely compatible detection rules. By standardizing how data is ingested and analyzed, organizations can more effectively detect, investigate, and respond to potential threats or anomalies in logs ingested into Elastic, and in this specific case, LLM logs.

Our initial mapping begins by investigating fields provided by the vendor and existing gaps, leading to the establishment of a comprehensive schema tailored to the nuances of LLM operations. We then reconciled the fields to align with our OpenTelemetry semantic conventions. These mappings shown in the table cover various aspects:

• General LLM Interaction Fields: These include basic but critical information such as the content of requests and responses, token counts, timestamps, and user identifiers, which are foundational for understanding the context and scope of interactions.
• Text Quality and Relevance Metric Fields: Fields measuring text readability, complexity, and similarity scores help assess the quality and relevance of model outputs, ensuring that responses are not only accurate but also user-appropriate.
• Security Metric Fields: This class of metrics is important for identifying and quantifying potential security risks, including regex pattern matches and scores related to jailbreak attempts, prompt injections, and other security concerns such as hallucination consistency and refusal responses.
• Policy Enforcement Fields: These fields capture details about specific policy enforcement actions taken during interactions, such as blocking or modifying content, and provide insights into the confidence levels of these actions, enhancing security and compliance measures.
• Threat Analysis Fields: Focused on identifying and quantifying potential threats, these fields provide a detailed analysis of risk scores, types of detected threats, and the measures taken to mitigate these threats.
• Compliance Fields: These fields help ensure that interactions comply with various regulatory standards, detailing any compliance violations detected and the specific rules that were triggered during the interaction.
• OWASP Top Ten Specific Fields: These fields map directly to the OWASP Top 10 risks for LLM applications, helping to align security measures with recognized industry standards.
• Sentiment and Toxicity Analysis Fields: These analyses are essential to gauge the tone and detect any harmful content in the response, ensuring that outputs align with ethical guidelines and standards. This includes sentiment scores, toxicity levels, and identification of inappropriate or sensitive content.
• Performance Metric Fields: These fields measure the performance aspects of LLM interactions, including response times and sizes of requests and responses, which are critical for optimizing system performance and ensuring efficient operations.

Note: See the gist for an extended table of fields proposed.

These fields are mapped by our LLM integrations and ultimately used within our detections. As we continue to understand the threat landscape, we will continue to refine these fields to ensure additional fields populated by other LLM vendors are standardized and conceptually reflected within the mapping.

Broader Implications and Benefits of Standardization

Standardizing security fields within the LLM ecosystem (e.g., user interaction and application integration) facilitates a unified approach to the security domain. Elastic endeavors to lead the charge by defining and promoting a set of standard fields. This effort not only enhances the security posture of individual organizations but also fosters a safer industry.

Integration with Security Tools: By standardizing responses from LLM-related security tools, it enriches security analysis fields that can be shipped with the original LLM vendor content to a security solution. If operationally chained together in the LLM application’s ecosystem, security tools can audit each invocation request and response. Security teams can then leverage these fields to build complex detection mechanisms that can identify subtle signs of misuse or vulnerabilities within LLM interactions.

Consistency Across Vendors: Insisting that all LLM vendors adopt these standard fields drives a singular goal to effectively protect applications, but in a way that establishes a baseline that all industry users can adhere to. Users are encouraged to align to a common schema regardless of the platform or tool.

Enhanced Detection Engineering: With these standard fields, detection engineering becomes more robust and the change of false positives is decreased. Security engineers can create effective rules that identify potential threats across different models, interactions, and ecosystems. This consistency is especially important for organizations that rely on multiple LLMs or security tools and need to maintain a unified platform.

Sample LLM-specific fields: AWS Bedrock use case

Based on the integration’s ingestion pipeline, field mappings, and processors, the AWS Bedrock data is cleaned up, standardized, and mapped to Elastic Common Schema (ECS) fields. The core Bedrock fields are then introduced under the aws.bedrock group which includes details about the model invocation like requests, responses, and token counts. The integration populates additional fields tailored for the LLM to provide deeper insights into the model’s interactions which are later used in our detections.

LLM detection engineering examples

With the standardized fields and the Elastic AWS Bedrock integration, we can begin crafting detection engineering rules that showcase the proposed capability with varying complexity. The below examples are written using ES|QL.

Note: Check out the detection-rules hunting directory and aws_bedrock rules for more details about these queries.

Basic detection of sensitive content refusal

With current policies and standards on sensitive topics within the organization, it is important to have mechanisms in place to ensure LLMs also adhere to compliance and ethical standards. Organizations have an opportunity to monitor and capture instances where an LLM directly refuses to respond to sensitive topics.

Sample Detection:

from logs-aws_bedrock.invocation-*
 | WHERE @timestamp > NOW() - 1 DAY
   AND (
     gen_ai.completion LIKE "*I cannot provide any information about*"
     AND gen_ai.response.finish_reasons LIKE "*end_turn*"
   )
 | STATS user_request_count = count() BY gen_ai.user.id
 | WHERE user_request_count >= 3

Detection Description: This query is used to detect instances where the model explicitly refuses to provide information on potentially sensitive or restricted topics multiple times. Combined with predefined formatted outputs, the use of specific phrases like "I cannot provide any information about" within the output content indicates that the model has been triggered by a user prompt to discuss something it's programmed to treat as confidential or inappropriate.

Security Relevance: Monitoring LLM refusals helps to identify attempts to probe the model for sensitive data or to exploit it in a manner that could lead to the leakage of proprietary or restricted information. By analyzing the patterns and frequency of these refusals, security teams can investigate if there are targeted attempts to breach information security policies.

Potential denial of service or resource exhaustion attacks

Due to the engineering design of LLMs being highly computational and data-intensive, they are susceptible to resource exhaustion and denial of service (DoS) attacks. High usage patterns may indicate abuse or malicious activities designed to degrade the LLM’s availability. Due to the ambiguity of correlating prompt request size directly with token count, it is essential to consider the implications of high token counts in prompts which may not always result from larger requests bodies. Token count and character counts depend on the specific model, where each can be different and is related to how embeddings are generated.

Sample Detection:

from logs-aws_bedrock.invocation-*
 | WHERE @timestamp > NOW() - 1 DAY
   AND (
     gen_ai.usage.prompt_tokens > 8000 OR
     gen_ai.usage.completion_tokens > 8000 OR
     gen_ai.performance.request_size > 8000
   )
 | STATS max_prompt_tokens = max(gen_ai.usage.prompt_tokens),
         max_request_tokens = max(gen_ai.performance.request_size),
         max_completion_tokens = max(gen_ai.usage.completion_tokens),
         request_count = count() BY cloud.account.id
 | WHERE request_count > 1
 | SORT max_prompt_tokens, max_request_tokens, max_completion_tokens DESC

Detection Description: This query identifies high-volume token usage which could be indicative of abuse or an attempted denial of service (DoS) attack. Monitoring for unusually high token counts (input or output) helps detect patterns that could slow down or overwhelm the system, potentially leading to service disruptions. Given each application may leverage a different token volume, we’ve chosen a simple threshold based on our existing experience that should cover basic use cases.

Security Relevance: This form of monitoring helps detect potential concerns with system availability and performance. It helps in the early detection of DoS attacks or abusive behavior that could degrade service quality for legitimate users. By aggregating and analyzing token usage by account, security teams can pinpoint sources of potentially malicious traffic and take appropriate measures.

Monitoring for latency anomalies

Latency-based metrics can be a key indicator of underlying performance issues or security threats that overload the system. By monitoring processing delays, organizations can ensure that servers are operating as efficiently as expected.

Sample Detection:

from logs-aws_bedrock.invocation-*
 | WHERE @timestamp > NOW() - 1 DAY
 | EVAL response_delay_seconds = gen_ai.performance.start_response_time / 1000
 | WHERE response_delay_seconds > 5
 | STATS max_response_delay = max(response_delay_seconds),
         request_count = count() BY gen_ai.user.id
 | WHERE request_count > 3
 | SORT max_response_delay DESC

Detection Description: This updated query monitors the time it takes for an LLM to start sending a response after receiving a request, focusing on the initial response latency. It identifies significant delays by comparing the actual start of the response to typical response times, highlighting instances where these delays may be abnormally long.

Security Relevance: Anomalous latencies can be symptomatic of issues such as network attacks (e.g., DDoS) or system inefficiencies that need to be addressed. By tracking and analyzing latency metrics, organizations can ensure that their systems are running efficiently and securely, and can quickly respond to potential threats that might manifest as abnormal delays.

Advanced LLM detection engineering use cases

This section explores potential use cases that could be addressed with an Elastic Security integration. It assumes that these fields are fully populated and that necessary security auditing enrichment features (e.g., Guardrails) have been implemented, either within AWS Bedrock or via a similar approach provided by the LLM vendor. In combination with the available data source and Elastic integration, detection rules can be built on top of these Guardrail requests and responses to detect misuse of LLMs in deployment.

Malicious model uploads and cross-tenant escalation

A recent investigation into the Hugging Face Interface API revealed a significant risk where attackers could upload a maliciously crafted model to perform arbitrary code execution. This was achieved by using a Python Pickle file that, when deserialized, executed embedded malicious code. These vulnerabilities highlight the need for rigorous security measures to inspect and sanitize all inputs in AI-as-a-Service (AIAAS) platforms from the LLM, to the infrastructure that hosts the model, and the application API integration. Refer to this article for more details.

Potential Detection Opportunity: Use fields like gen_ai.request.model.id, gen_ai.request.model.version, and prompt gen_ai.completion to detect interactions with anomalous models. Monitoring unusual values or patterns in the model identifiers and version numbers along with inspecting the requested content (e.g., looking for typical Python Pickle serialization techniques) may indicate suspicious behavior. Similarly, a check prior to uploading the model using similar fields may block the upload. Cross-referencing additional fields like gen_ai.user.id can help identify malicious cross-tenant operations performing these types of activities.

Unauthorized URLs and external communication

As LLMs become more integrated into operational ecosystems, their ability to interact with external capabilities like email or webhooks can be exploited by attackers. To protect against these interactions, it’s important to implement detection rules that can identify suspicious or unauthorized activities based on the model’s outputs and subsequent integrations.

Potential Detection Opportunity: Use fields like gen_ai.completion, and gen_ai.security.regex_pattern_count to triage malicious external URLs and webhooks. These regex patterns need to be predefined based on well-known suspicious patterns.

Hierarchical instruction prioritization

LLMs are increasingly used in environments where they receive instructions from various sources (e.g., ChatGPT Custom Instructions), which may not always have benign intentions. This build-your-own model workflow can lead to a range of potential security vulnerabilities, if the model treats all instructions with equal importance, and they go unchecked. Reference here.

Potential Detection Opportunity: Monitor fields like gen_ai.model.instructions and gen_ai.completion to identify discrepancies between given instructions and the models responses which may indicate cases where models treat all instructions with equal importance. Additionally, analyze the gen_ai.similarity_score, to discern how similar the response is from the original request.

Extended detections featuring additional Elastic rule types

This section introduces additional detection engineering techniques using some of Elastic’s rule types, Threshold, Indicator Match, and New Terms to provide a more nuanced and robust security posture.

• Threshold Rules: Identify high frequency of denied requests over a short period of time grouped by gen_ai.user.id that could be indicative of abuse attempts. (e.g. OWASP’s LLM04)
• Indicator Match Rules: Match known malicious threat intel provided indicators such as the LLM user ID like the gen_ai.user.id which contain these user attributes. (e.g. arn:aws:iam::12345678912:user/thethreatactor)
• New Terms Rules: Detect new or unusual terms in user prompts that could indicate usual activity outside of the normal usage for the user’s role, potentially indicating new malicious behaviors.

Summary

Elastic is pioneering the standardization of LLM-based fields across the generative AI landscape to enable security detections across the ecosystem. This initiative not only aligns with our ongoing enhancements in LLM integration and security strategies but also supports our broad security framework that safeguards both direct user interactions and the underlying system architectures. By promoting a uniform language among LLM vendors for enhanced detection and response capabilities, we aim to protect the entire ecosystem, making it more secure and dependable. Elastic invites all stakeholders within the industry, creators, maintainers, integrators and users, to adopt these standardized practices, thereby strengthening collective security measures and advancing industry-wide protections.

As we continue to add and enhance our integrations, starting with AWS Bedrock, we are strategizing to align other LLM-based integrations to the new standards we’ve set, paving the way for a unified experience across the Elastic ecosystem. The seamless overlap with existing Elasticsearch capabilities empowers users to leverage sophisticated search and analytics directly on the LLM data, driving existing workflows back to tools users are most comfortable with.

Check out the LLM Safety Assessment, which delves deeper into these topics.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

• On March 29, 2024, Andres Freund identified malicious commits to the command-line utility XZ, impacting versions 5.6.0 and 5.6.1 for Linux, and shared the information on the oss-security mailing list.
• Andres’ discovery was made after an increase of 500ms in latency was observed with SSH login attempts initiated from a development system, amongst other anomalies.
• The backdoor identified has been designed to circumvent authentication controls within SSH to remotely execute code, potentially gaining access to other systems in the environment.
• The code commits were added and signed by JiaT75 (now suspended), who contributed to the popular open source project for several years.
• Security researchers are still undertaking an initial analysis of the payload, dissecting both the build process and the backdoor.
• Elastic has released both YARA signatures, detection rules, and osquery queries, allowing Linux system maintainers to understand the impact and block potential compromises early.

The XZ / liblzma backdoor at a glance

On March 29 2024, the widely adopted XZ package used within many Linux distributions as a library used by the system to interact with SSH client connections (and many other system utilities) was pulled into the spotlight after a 500ms delay with intermittent failures. What began as a routine investigation into that anomaly would take a surprising and unexpected twist: malicious, obfuscated code was planted in the package by a maintainer–code that was also in circulation for a few weeks via a poisoned build process.

Andres Freund, the developer who initially identified the malicious contributions, observed that the changes had been implemented in versions 5.6.0 and 5.6.1 of the XZ Utils package but had not been widely adopted across all Linux distributions, outside of select bleeding-edge variants typically used for early-stage testing.

Initial analysis has shown that the backdoor is designed to circumvent authentication controls in sshd via systemd and attempts to execute code within a pre-authentication context. Observations made so far have shown that the malicious code is not in its final target state and was perhaps caught early through haphazard mistakes the developer neglected to consider, causing impacts to legitimate SSH use cases.

Alongside the malicious package being circulated within a small number of Linux distributions, several observations have been made in the popular package management software HomeBrew, which has impacted some macOS users. The maintainers of Homebrew-- and other software packages that included this library-- are presently rolling back to prior versions that aren't impacted by these malicious changes, although mainly out of an abundance of caution, as compromised builds were only targeting deb and rpm packages.

The following notice was released on the Tukaani Project’s homepage (the project owner of the XZ Utils Git repository) shortly after the news of the backdoor broke.

The compromise itself, while high risk, is relatively minor in terms of real-world impact given the stage of discovery. This situation should remind security professionals about the importance of understanding supply-chain compromise, monitoring Linux workloads, and auditing system controls. In this situation, defenders had the advantage of time.

Backdoor analysis

XZ backdoor build process:

CVE-2024-3094 explains how the changes in the liblzma were created from the malicious additions to the library’s build scripts and directly impacts any software that links the library on an impacted system.

The maliciously modified build script is divided into three stages, starting with the additions in m4/build-to-host.m4 and progressing through the obfuscation and execution stages. At a high level, some obfuscation techniques include character substitution and selective byte processing commands via the tr and head commands to decode and execute the malicious payloads in the test files. Interestingly, many impacted tools used are standard Linux system tools typically used by administrators for legitimate purposes.

The build process runs as follows :

• Stage 0: The initial malicious code additions attempt to decode the Stage 1 script (hidden code segments) by changing byte values from specific test files, which under normal circumstances appear corrupt, to form a valid XZ stream.
• Stage 1: This stage leverages a bash file with special checks (e.g., the Linux architecture the script runs on) and Bash commands to analyze the environment (e.g. [ "$(uname)" = "Linux" ]) to ensure compatible conditions are met for the backdoor. Depending on the outcome of the checks, additional malicious scripts or payloads may be executed.
• Stage 2: This phase involves an infected.txt file, which details the altered extraction and compilation code modifications, namely:
  • Reconstruction Data: Byte manipulation and decoding techniques on obfuscated compressed data from test files to reconstruct the malicious payload using commands like sed and awk
  • Obfuscation and Extraction: Complex decryption and obfuscation techniques using the tr command to extract the binary backdoor to remain hidden from typical detection mechanisms
  • Build Process Manipulation: This changes the build and compilation steps to embed the binary backdoor into Linux system processes
  • Extension Mechanism: A design that allows for new scripts and updates to the backdoor without modifying the original payload
  • Future Stage Preparation: Sets the groundwork for malicious follow-up activities, like propagating the backdoor

Assessing impact:

Given the limited usage of the impacted beta distributions and software, this compromise should impact few systems. Maintainers of Linux systems are however encouraged to ensure systems are not running impacted versions of xzutils / liblzma by leveraging the following osquery queries:

Linux:

SELECT 'DEB Package' AS source, name, version,
  CASE
    WHEN version LIKE '5.6.0%' OR version LIKE '5.6.1%' THEN 'Potentially Vulnerable'
    ELSE 'Most likely not vulnerable'
  END AS status
FROM deb_packages
WHERE name = 'xz-utils' OR name = 'liblzma' OR name LIKE 'liblzma%'
UNION
SELECT 'RPM Package' AS source, name, version,
  CASE
    WHEN version LIKE '5.6.0%' OR version LIKE '5.6.1%' THEN 'Potentially Vulnerable'
    ELSE 'Most likely not vulnerable'
  END AS status
FROM rpm_packages
WHERE name = 'xz-utils' OR name = 'liblzma' OR name LIKE 'liblzma%';

macOS:

SELECT 'Homebrew Package' AS source, name, version,
  CASE
    WHEN version LIKE '5.6.0%' OR version LIKE '5.6.1%' THEN 'Potentially Vulnerable'
    ELSE 'Most likely not vulnerable'
  END AS status
FROM homebrew_packages
WHERE name = 'xz' OR name = 'liblzma';

The following KQL query can be used to query Elastic Defend file events:

event.category : file and host.os.type : (macos or linux) and file.name : liblzma.so.5.6.*

Alternatively, manually checking the version of XZ running on a system is as simple as running the following commands (from researcher Kostas) and checking the output version. Remember, versions 5.6.0 and 5.6.1 are impacted and should be rolled back or updated to a newer version.

for xz_p in $(type -a xz | awk '{print $NF}' | uniq); do strings "$xz_p" | grep "xz (XZ Utils)" || echo "No match found for $xz_p"; done

Malware protection

The following YARA signature (disk and in-memory) is deployed in Elastic Defend to block the XZ backdoor.

rule Linux_Trojan_XZBackdoor {
    meta:
        author = "Elastic Security"
        fingerprint = "f1982d1db5aacd2d6b0b4c879f9f75d4413e0d43e58ea7de2b7dff66ec0f93ab"
        creation_date = "2024-03-30"
        last_modified = "2024-03-31"
        threat_name = "Linux.Trojan.XZBackdoor"
        reference_sample = "5448850cdc3a7ae41ff53b433c2adbd0ff492515012412ee63a40d2685db3049"
        severity = 100
        arch_context = "x86"
        scan_context = "file, memory"
        license = "Elastic License v2"
        os = "linux"
    strings:
        /* potential backdoor kill-switch as per https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01?permalink_comment_id=5006558#file-hashes-txt-L115 */
        $a1 = "yolAbejyiejuvnup=Evjtgvsh5okmkAvj"
/* function signature in liblzma used by sshd */
        $a2 = { F3 0F 1E FA 55 48 89 F5 4C 89 CE 53 89 FB 81 E7 00 00 00 80 48 83 EC 28 48 89 54 24 18 48 89 4C 24 10 }
 /* unique byte patterns in backdoored liblzma */
        $b1 = { 48 8D 7C 24 08 F3 AB 48 8D 44 24 08 48 89 D1 4C 89 C7 48 89 C2 E8 ?? ?? ?? ?? 89 C2 }
        $b2 = { 31 C0 49 89 FF B9 16 00 00 00 4D 89 C5 48 8D 7C 24 48 4D 89 CE F3 AB 48 8D 44 24 48 }
        $b3 = { 4D 8B 6C 24 08 45 8B 3C 24 4C 8B 63 10 89 85 78 F1 FF FF 31 C0 83 BD 78 F1 FF FF 00 F3 AB 79 07 }
    condition:
        1 of ($a*) or all of ($b*)
}

Detections of this signature will appear in Elastic as follows:

Behavior Detection

Leveraging Elastic Defend’s network and process events, we published a new EQL detection rule to identify instances where the SSHD service starts, spawns a shell process and immediately terminates unexpectedly all within a very short time span:

sequence by host.id, user.id with maxspan=1s
 [process where host.os.type == "linux" and event.type == "start" and event.action == "exec" and process.name == "sshd" and
    process.args == "-D" and process.args == "-R"] by process.pid, process.entity_id
 [process where host.os.type == "linux" and event.type == "start" and event.action == "exec" and process.parent.name == "sshd" and 
  process.executable != "/usr/sbin/sshd"] by process.parent.pid, process.parent.entity_id
 [process where host.os.type == "linux" and event.action == "end" and process.name == "sshd" and process.exit_code != 0] by process.pid, process.entity_id
 [network where host.os.type == "linux" and event.type == "end" and event.action == "disconnect_received" and process.name == "sshd"] by process.pid, process.entity_id

Linux: the final frontier

While observations of supply chain-based attacks or exploitation of vulnerabilities rarely reach this level of global press coverage, Elastic’s observations described in the 2023 Global Threat Report show that Linux-based signature events continue to grow in our dataset. This growth is partially tied to growth in the systems we observe that report on threat behavior, but it strongly suggests that adversaries are becoming increasingly focused on Linux systems.

Linux is and will continue to be on the minds of threat groups, as its widespread adoption across the internet reinforces its importance. In this case, adversarial groups were trying to circumvent existing controls that would allow for future compromise through other means.

While the objectives of the person(s) behind the XZ backdoor haven’t been made clear yet, it is within the technical capabilities of many threat entities focused on espionage, extortion, destruction of data, intellectual property theft, and human rights abuses. With the ability to execute code on impacted Internet-accessible systems, it’s reasonable to assume that bad actors would further infiltrate victims. Elastic Security Labs sees that Linux visibility has been dramatically improving and enterprises have started to effectively manage their Linux populations, but many organizations reacting to this supply chain compromise are still at the start of that process.

To understand how Elastic is currently assessing internal risk of this vulnerability in our products please see the advisoryhere.

This document was updated on December 17, 2021 to reflect a revised CVSS score for CVE-2021-45046, and new findings by the community.

In recent days Log4Shell, or CVE-2021-44228, has dominated the news cycle in the world of information security and beyond. Elastic released an advisory detailing how Elastic products and users are impacted, and a blog post describing how our users can leverage Elastic Security to help defend their networks.

Many readers have further questions as to how we’re tracking this issue within Elastic Security, what our coverage is now, and what we’re expecting to do next. This post outlines a few details for our current status, and provides details regarding a new, related vulnerability: CVE-2021-45046.

Elastic Security response

As you may imagine, the team has worked tirelessly to ensure that we’re developing detections for both active exploitation of the vulnerability, as well as post-compromise indicators, and will continue active development until further notice.

We’re spending time focusing on detailed detections that better align with some of the emerging trends that adversaries are now taking advantage of as they have time to develop their attack strategies. And we’re not working in silence — those that may have had a chance to catch up on our original post a few days ago will be pleasantly surprised we’ve added further detections and hunting examples, and will continue to do so as we learn more with the community.

Alongside the threat research and signature development, we’ve noted some interesting observations:

• We noted several instances of generic crypto miners for Linux being deployed that appeared to be related to exploitation of this CVE, but determined that they are benign true positives
• We’ve stopped at least eight different families of malware being deployed using the log4j exploit, indicating widespread adoption of the exploit by threats of all kinds
• While we are observing coverage across our full protection suite (such as behavior protection), it is noteworthy that our free basic-tier malware protection is successfully preventing initial access

We will aim to keep users and readers apprised of findings, and hope to share additional observations in the wild as we see them.

A new contender: CVE-2021-45046

While we watch the CVE-2021-44228 (Log4Shell) vulnerability dominate the news cycles, a new contender, CVE-2021-45046, was accidentally introduced to Log4j2j version 2.15.0, allowing adversaries to invoke a Denial of Service, and a remote code execution condition through specially crafted payloads. Previous mitigations to avoid Information Disclosure vulnerabilities by setting the log4j2.noFormatMsgLookup state to true do not mitigate against this new finding, according to the CVE details.

While initially CVE-2021-45046 carried a lower CVSS score of 3.7 due to the impact of the initially discovered condition that can be invoked, this was re-evaluated to a 9.0 indicating limited remote code execution was possible. The finding was shared on December 16, 2021 by Alvaro Muñoz, who identified that while the default setting formatMsgNoLookups was accurately set to true, there were alternative locations for lookups to take place. Technical details are still unfolding from the community, however the Log4j2 team shared the following message within their security updates:

The reason these measures are insufficient is that, in addition to the Thread Context attack vector mentioned above, there are still code paths in Log4j where message lookups could occur: known examples are applications that use Logger.printf("%s", userInput), or applications that use a custom message factory, where the resulting messages do not implement StringBuilderFormattable. There may be other attack vectors.

The safest thing to do is to upgrade Log4j to a safe version, or remove the JndiLookup class from the log4j-core jar. Reference here

Given this new information, and readily availablePOCs available for exploitation, the Apache team has recommended those impacted upgrade to the latest, safe version of Log4j2, or alternatively remove the JndiLookup class from the log4j-core jar.

Elastic Security has observed many threat actors and benign scanners leveraging this new methodology already in some edge environments, with payloads incorporating previous attack methodologies such as key extraction attempts and base64 encoded payloads:

We anticipate adding further details as we learn them, and thank the team at lunasec specifically for providing a detailed, early summary of this emerging situation, and of course, provide kudos to Alvaro Muñoz of Github Security Lab for the findings.

Thank you (again!), from Elastic Security

We want to thank all of the security teams across the globe for your tireless work this week. As we referenced before, openness and collaboration in the security community to safeguard all users is paramount when facing such a serious and pervasive vulnerability.

Existing Elastic Security users can access these capabilities within the product. If you’re new to Elastic Security, take a look at our Quick Start guides (bite-sized training videos to get you started quickly) or our free fundamentals training courses.

Get started with a free 14-day trial of Elastic Cloud. Or download the self-managed version of the Elastic Stack for free.

References

https://logging.apache.org/log4j/2.x/security.html

https://www.lunasec.io/docs/blog/log4j-zero-day-severity-of-cve-2021-45046-increased/

• To understand how Elastic is currently assessing internal risk of this vulnerability in our products please see the advisoryhere.
• This blog has been updated (Dec. 17, 2021) with further detection and hunting improvements since its initial publish.

Overview

This blog post provides a summary of CVE-2021-44228 and provides Elastic Security users with detections to find active exploitation of the vulnerability in their environment.

Further updates will be provided to this post as we learn more. This version is accurate as of Tuesday, December 14, 2021. Updates from Apache may be investigated directly via the security page for Log4j2.

Summary of CVE-2021-44228 (Log4Shell)

Log4j2 is an open source logging framework incorporated into many Java based applications on both end-user systems and servers. In late November 2021, Chen Zhaojun of Alibaba identified a remote code execution vulnerability, ultimately being reported under the CVE ID : CVE-2021-44228, released to the public on December 10, 2021. The vulnerability is exploited through improper deserialization of user-input passed into the framework. It permits remote code execution and it can allow an attacker to leak sensitive data, such as environment variables, or execute malicious software on the target system.

The identified vulnerability impacts all versions of Log4j2 from version 2.0-beta9 to version 2.14.1. Early methods to patch the issue resulted in a number of release candidates, culminating in recommendations to upgrade the framework to Log4j2 2.15.0-rc2 at the time of this post.

Given the trivial complexity and the nature of observed widespread exploitation, mitigation should be considered critical in any environment that has identified software leveraging vulnerable versions of Log4j2.

Detecting Exploitation of Log4Shell in Elastic Security

Elastic Security users can use the following Event Correlation detection rule to identify active exploitation of the Log4j2 vulnerability. Depending on the format of the host based event data you may need to modify this detection to match your data fields.

Detection Rule when using Endpoint data

sequence by host.id with maxspan=1m
 [network where event.action == "connection_attempted" and
  process.name : "java" and
  /*
     outbound connection attempt to
     LDAP, RMI or DNS standard ports
     by JAVA process
   */
  destination.port in (1389, 389, 1099, 53, 5353)] by process.pid
 [process where event.type == "start" and

  /* Suspicious JAVA child process */
  process.parent.name : "java" and
   process.name : ("sh",
                   "bash",
                   "dash",
                   "ksh",
                   "tcsh",
                   "zsh",
                   "curl",
                   "perl*",
                   "python*",
                   "ruby*",
                   "php*",
                   "wget")] by process.parent.pid

Detection Rule when using Auditbeat data

sequence by agent.id with maxspan=1m
 [network where event.action == "connected-to" and
  process.name : "java" and
  /*
     outbound connection attempt to
     LDAP, RMI or DNS standard ports
     by JAVA process
   */
  destination.port in (1389, 389, 1099, 53, 5353)] by process.pid
 [process where event.type == "start" and

  /* Suspicious JAVA child process */
  process.parent.name : "java" and
   process.name : ("sh",
                   "bash",
                   "dash",
                   "ksh",
                   "tcsh",
                   "zsh",
                   "curl",
                   "perl*",
                   "python*",
                   "ruby*",
                   "php*",
                   "wget")] by process.parent.pid

Detection rule when using Endgame streamed events

sequence by agent.id with maxspan=1m
 [network where event.category == "network" and
  process.name : "java" and
  /*
     outbound connection attempt to
     LDAP, RMI or DNS standard ports
     by JAVA process
   */
  destination.port in (1389, 389, 1099, 53, 5353)] by process.pid
 [process where event.type == "start" and

  /* Suspicious JAVA child process */
  process.parent.name : "java" and
   process.name : ("sh",
                   "bash",
                   "dash",
                   "ksh",
                   "tcsh",
                   "zsh",
                   "curl",
                   "perl*",
                   "python*",
                   "ruby*",
                   "php*",
                   "wget")] by process.parent.pid

This detection rule looks for a sequence of an outbound connection attempt to standard ports for LDAP, RMI and DNS (often abused via recently observed JAVA/JNDI injection attacks) followed by a child process of the same Java process instance.

Now, let’s demonstrate how this rule detects exploitation of the log42j vulnerability:

The screenshot above shows an attacker exploiting the vulnerability with a base-64 encoded payload targeting an example vulnerable application created by Christophe Tafani-Dereeper.

This screenshot shows the detection of the active exploitation of CVE-2021-44228 within Elastic Security detailing both the alert and timeline view of the exploit.

The screenshot above shows in the investigation of the detection alert that Java executed a shell script to download and run a bash script.

Update: Detection & hunting improvements

Suspicious Shell Commands Execution via Java

Based on observed publicly known malicious Java classes served via log4j exploit, you can hunt for suspicious shell scripts and ingress tool transfer commands:

process where event.type == "start" and
  process.parent.name : "java*" and

  /* Ingress tools transfer via common shell command interpreters */

  /* linux or macos */
  (
   (process.name : ("sh", "bash", "python*") and
    process.command_line : ("*curl*|*sh*", "*wget*|*bash", "*curl*|*bash*", "*curl*|*bash*", "*http*|*sh*", "*python*http*")) or

  /* windows */
  (process.name : ("powershell.exe", "pwsh.exe", "cmd.exe") and
   process.command_line : ("*.downloadstring*", "*.downloadfile*", "*.downloaddata*", "*BitsTransfer*", "* -enc*", "* IEX*", "*wp-content*", "*wp-admin*", "*wp-includes*", "*$*$*$*$*$*", "*^*^*^*^*^*^*^*^*^*", "*.replace*", "*start-process*", "*http*", "*cmd*powershell*")))

Untrusted File Execution via JAVA

Identifies when a JAVA interpreter creates an executable file (PE/ELF) and the file is subsequently executed.

Detection Rule when using Endpoint data

sequence by host.id with maxspan=5m
 [ file where event.type != "deletion" and
  process.name : ("java", "java.exe", "javaw.exe") and

  (file.extension : ("exe", "com", "pif", "scr") or
      /* Match Windows PE files by header data (MZ) */
  file.Ext.header_bytes : ("4d5a*", "7f454c46*")) and

  not file.path :  ("?:\\Program Files\\*",
                    "?:\\Program Files (x86)\\*") ] by file.path
 [ process where event.type == "start" and
  not process.code_signature.trusted == true ] by process.executable

Detection rule when using Endgame streamed events

sequence by agent.id with maxspan=5m
  [ file where event.type != "deletion"
    process.name : ("java", "java.exe", "javaw.exe")] by file_path
  [ process where event.type == "start" and
  not process.code_signature.trusted == true] by process_path

Potential CoinMiner activity

Process with command line common to cryptocurrency miner (most observed campaigns leveraging log4j exploit are coinminers):

process where event.type == "start" and
 process.command_line :
       ("* pool.*", "*-u*--coin*", "*.xmr.*", "*.xmr1.*",
        "*stratum*", "*elitter.net*", "*cryptonight*",
        "*-a scrypt*", "*stratum1*", "*-userpass*", "*-max-cpu-usage*",
      "*qhor.net*", "*-wallet*pool*", "*--donate-level*", "*supportxmr.com*")

Other relevant post exploitation detections :

Attempt to Disable IPTables or Firewall

Tampering of Bash Command-Line History

System Log File Deletion

Potential Reverse Shell Activity via Terminal

Suspicious JAVA Child Process

Attempt to Disable Syslog Service

Elastic Endgame EQL Queries

Suspicious Java Netcon followed by Unusual Child Process

sequence with maxspan=5s
 [network where process_name == "java*" and destination_port in (1389, 389, 1099, 53, 5353) and
  destination_address != "127.0.0.1" and not destination_address == "::1"] by pid
 [process where opcode in (1,5) and
  /* Suspicious JAVA child process */
  parent_process_name == "java*" and
   process_name in ("sh", "bash", "dash", "ksh", "tcsh", "zsh", "curl", "perl*", "python*", "ruby*", "php*", "wget", "powershell.exe", "cmd.exe")] by ppid

Suspicious Shell Commands Execution via Java

process where opcode in (1,5) and
  parent_process_name == "java*" and
  /* Ingress tools transfer via common shell command interpreters */

  /* linux or macos */
 (
  (process_name in ("sh", "bash", "python") and
   wildcard(command_line, "*curl*|*sh*", "*wget*|*bash", "*curl*|*bash*", "*curl*|*bash*", "*http*|*sh*", "*python*http*")) or
  /* windows */
  (process_name in ("powershell.exe", "pwsh.exe", "cmd.exe") and
   wildcard(command_line,"*.downloadstring*", "*.downloadfile*", "*.downloaddata*", "*BitsTransfer*", "* -enc*", "* IEX*", "*wp-content*", "*wp-admin*", "*wp-includes*", "*$*$*$*$*$*", "*^*^*^*^*^*^*^*^*^*","*.replace*", "*start-process*", "*http*", "*cmd*powershell*")))

Common Coin Miners as a descendant of JAVA

process where opcode in (1, 3, 4, 5) and
 descendant of [process where opcode in (1, 3, 4, 5) and process_name == "java*"] and
 wildcard(command_line, "* pool.*", "*-u*--coin*", "*.xmr.*", "*.xmr1.*", "*stratum*", "*elitter.net*", "*cryptonight*", "*-a scrypt*", "*stratum1*",
"*-userpass*", "*-max-cpu-usage*", "*qhor.net*", "*-wallet*pool*",  "*--donate-level*", "*supportxmr.com*",
/* evasion commands */
"*base64*", "*history -c*", "*ld.so.preload*", "*nmi_watchdog*", "*ufw*disable*", "*.bash_history*", "*chmod*+x*",
"*tor2web*", "*kill*-9*", "*python*-c*http*")

Untrusted File Execution via JAVA

sequence with maxspan=2m
  [ file where opcode != 2 and file_name == "*.exe" and process_name == "java*"] by file_path
  [ process where opcode in (1,5)] by process_path

Community Detections

A number of community members discussing widespread exploitation of the vulnerability have provided insights into a number of early detection methods that analysts may leverage to identify if systems they are using have been exploited or are under active exploitation:

• A series of payloads have been shared by the GreyNoise team, including payloads containing both encoded and decoded variants for analysts looking to explore logs stored within their systems. This has been complemented with a list of initial tagged IPs attempting exploitation of the vulnerability.

• Florian Roth of Nextron Systems has provided a series of checks for local exploitation using grep / zgrep, alongside some initial YARA signatures in a Gist listed on his Github account. Florian also shared a method for generating Thinkst CanaryTokens to test systems you may manage for exploitability.

• Rob Fuller (Mubix) has shared a list of known file hashes for vulnerable versions of the framework, here.

Additional Mitigation Strategies

Outside of the recommended guidance from the Apache team regarding the deployment of the latest, patched versions of the Log4j2 framework to update, a number of mitigations have been widely suggested to prevent exploitation:

• Fastly have suggested checking if your version of Log4j supports executing the JVM with JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true to disable the lookup functionality to the remote server. This should apply to versions 2.10.0 through 2.15.0.

• To prevent lateral movement from a vulnerable host, or exploitation over the network, limiting connectivity from potentially vulnerable systems to external resources to trusted applications and / or services is recommended.

Thank you, from Elastic Security.

We want to thank all of the security teams across the globe for your tireless work today and through the weekend, especially those of you listed in this post. Openness and collaboration in the security community to safeguard all users is paramount when facing such a serious and pervasive vulnerability. We want you to know we are here with you every step of the way.

Existing Elastic Security can access these capabilities within the product. If you’re new to Elastic Security, take a look at our Quick Start guides (bite-sized training videos to get you started quickly) or our free fundamentals training courses. You can always get started with a free 14-day trial of Elastic Cloud. Or download the self-managed version of the Elastic Stack for free.

Reference Material

https://www.lunasec.io/docs/blog/log4j-zero-day/

https://www.tenable.com/blog/cve-2021-44228-proof-of-concept-for-critical-apache-log4j-remote-code-execution-vulnerability

https://www.crowdstrike.com/blog/log4j2-vulnerability-analysis-and-mitigation-recommendations/

https://mbechler.github.io/2021/12/10/PSA_Log4Shell_JNDI_Injection/

https://www.greynoise.io/viz/query/?gnql=CVE-2021-44228

https://logging.apache.org/log4j/2.x/security.html#

https://github.com/christophetd/log4shell-vulnerable-app

Dirty Pipe is a local privilege escalation vulnerability that is easily exploitable with a handful of working exploit POCs already available. Its broad scope (any user-readable file and affected Linux versions) along with its evolving nature (the SUID shell backdoor exploit) make CVE-2022-0847 especially dangerous for administrators of systems that are potentially vulnerable.

What is Dirty Pipe (CVE-2022-0847)?

CVE-2022-0847 is a Linux local privilege escalation vulnerability, discovered by security researcher Max Kellermann that takes advantage of the way the Linux kernel manages page files and named pipes allowing for the overwriting of data in read-only files. This vulnerability impacts Linux kernels 5.8 and later until any version before 5.16.11, 5.15.25, and 5.10.102.

What is the impact?

With many POC’s already released, this vulnerability can be easily exploited to gain root-level privileges by, for instance, rewriting sensitive files like “/etc/passwd” or hijacking a SUID root binary (like sudo) via injection of malicious code.

What is Elastic doing about it?

Elastic is releasing detection logic and Auditd rules that can be used to detect exploitation of this vulnerability.

Dirty Pipe Details

The vulnerability can be exploited due to a flaw in the new pipe buffer structure where a flag member lacked proper initialization and could then contain a stale value. This could then be used to write to pages within the page cache behind read-only files, allowing for privilege escalation. Given the specific nature of this vulnerability, detection can be quite difficult.

Linux Pipes & CVE-2022-0847

Pipes are an interprocess communication mechanism represented as a file within Linux that can receive input data and provide an output for that data. The output of one process can become the input of another using a “pipe” to forward that data between.

Pipes are managed by the CPU in memory and their data is referred to as a “page”.

The exploitation of this vulnerability utilizes a process called “page splicing”. Page splicing is used to merge data between different pipe pages in memory without having to rewrite the data.

The flag we referenced in the summary is the PIPE_BUF_FLAG_CAN_MERGE flag. This must be set in order for a page cache to be merged and is only set when the pipe page becomes full. Howerver, if the page cache is emptied completely this flag remains (lack of initialization) which is where the problem lies.

The exploit functions generally by:

1. Opening a new pipe
2. Filling the pipe’s page cache with arbitrary data in order to set the PIPE_BUF_FLAG_CAN_MERGE flag
3. Draining the page cache of data but retaining the PIPE_BUF_FLAG_CAN_MERGE flag and replacing the data with the new data they want to overwrite a read-only file with
4. The splice (“page splicing”) syscall is then used to merge the pages (the pipe page and target file page) leading to the new data being added to a target file bypassing the read-only permissions

Many of the exploit POCs observed so far target the /etc/passwd file to overwrite and provide the users with elevated root privileges. Other variants of the exploit released allow for the creation of a SUID shell backdoor by overwriting a binary that has SUID permissions (superuser capabilities) giving the user a root shell and complete control.

We anticipate that adversaries and researchers will develop a multitude of other exploitation chains with this particular vulnerability.

Proof Of Concept Code

The security community has developed a multitude of different tests that adversaries may take advantage of in future attacks against systems. POCs listed below are authored to help security researchers identify if systems are impacted by the vulnerability, and furthermore - test detection strategies.

• Original Max Kellermann write-up: https://dirtypipe.cm4all.com/
• SUID shell: ​​https://haxx.in/files/dirtypipez.c
• Passwd overwrite: https://github.com/liamg/traitor
• Passwd overwrite: ​​https://github.com/imfiver/CVE-2022-0847
• Metasploit module: https://github.com/rapid7/metasploit-framework/pull/16303

Finding systems vulnerable to Dirty Pipe

Beyond using a traditional vulnerabilty scanner, there are several ways to detect systems vulnerable to Dirty Pipe.

Using the Elastic Security Integration

If you have Auditbeat, Filebeat (with the Auditd module enabled), or the Elastic Agent (with the Security or Auditd integrations deployed) you can use the Lens visualization tool (located in Kibana) to quickly compile and save a list of vulnerable systems as evidenced in the screenshot below:

Using the Osquery Manager Integration

Additionally, you can use the Osquery Manager integration to collect the kernel information from all endpoints. To do this, you need to add the Osquery Manager integration to an Elastic Agent policy (Integrations → Osquery Manager → Add Osquery Manager). Once you’ve added the integration, you can perform a simple query: SELECT version FROM kernel_info; which will return the hostname and Linux kernel version from all endpoints with the policy.

Detecting CVE-2022-0847 exploitation using Auditd

Auditd is the userspace component of the Linux Auditing System. Auditd stands for Audit Daemon and is a background running service responsible for collecting and writing log files to disk. The Linux Audit System includes a kernel component that hooks system calls and communicates those to Auditd. Auditd is capable of logging System Calls, File Access, and certain pre-configured Audit events. You can install and enable Auditd for free with the package manager on your Linux distribution of choice.

Auditd rules

Auditd rules define what is to be captured and logged. These rules are generally defined in an audit.rules file and placed at /etc/audit/audit.rules or /etc/audit/rules.d/audit.rules. Events are written to /var/log/audit/audit.log on the local system.

Once you have installed and enabled Auditd, you can add the below lines to your audit.rules file to detect Dirty Pipe exploitation attempts.

Dirty Pipe Auditd rules

-a always,exit -F arch=b64 -S splice -F a0=0x3 -F a2=0x5 -F a3=0x0 -F key=dirtypipe
-a always,exit -F arch=b64 -S splice -F a0=0x6 -F a2=0x8 -F a3=0x0 -F key=dirtypipe
-a always,exit -F arch=b64 -S splice -F a0=0x7 -F a2=0x9 -F a3=0x0 -F key=dirtypipe

The aforementioned rules were adapted by Elastic Security from initial findings by Jonas LeJon.

Linux Auditing System event collection with Elastic

There are a few different ways to collect Linux Auditing System events using Elastic. You can either use the Elastic Agent with the Auditd integration, Auditbeat, or the Auditd module for Filebeat.

Remember, if you’re using the Auditd integrations for the Elastic Agent or Filebeat, you’ll need to create the Auditd rules described above.

The Elastic Agent w/Auditd Integration

The Elastic Agent with the Auditd Integration allows for the collection of Auditd rules. To collect these events, you need to add the Auditd integration to an Elastic Agent policy (Integrations → Auditd → Add Auditd).

Once this integration is installed to an Elastic Agent policy and deployed to endpoints, you will see Auditd events populated in Kibana.

You can verify that you are receiving Auditd events in Kibana by using the Kibana query event.dataset : "auditd.log".

Auditbeat

You can use the Auditbeat Auditd module to collect the Linux Audit Framework logs. To do this, install Auditbeat. You might encounter errors if another process besides Auditbeat, such as Auditd, is registered to receive data from the Linux Audit Framework. To prevent this conflict, you can stop and disable Auditd from running.

Stopping and disabling Auditd

sudo service auditd.service stop
sudo chkconfig auditd.service off

Edit the /etc/auditbeat/auditbeat.yml file to point to your local, remote, or cloud cluster and add the Dirty Pipe rules provided above in the Auditd rules section.

Adding Dirty Pipe detection rules to the Auditbeat configuration file

# ===== Modules configuration =====

auditbeat.modules:

* module: auditd

# Load audit rules from separate files. Same format as audit.rules(7)

  audit_rule_files: [ '${path.config}/audit.rules.d/*.conf' ]
  audit_rules: |

## Define audit rules here

## Create file watches (-w) or syscall audits (-a or -A). Uncomment these

## examples or add your own rules

    -a always,exit -F arch=b64 -S splice -F a0=0x3 -F a2=0x5 -F a3=0x0 -F key=dirtypipe
    -a always,exit -F arch=b64 -S splice -F a0=0x6 -F a2=0x8 -F a3=0x0 -F key=dirtypipe
    -a always,exit -F arch=b64 -S splice -F a0=0x7 -F a2=0x9 -F a3=0x0 -F key=dirtypipe

…truncated…

Check the configuration and connectivity of Auditbeat using the test commands.

Testing the Auditbeat configuration and output settings

sudo auditbeat test config
sudo auditbeat test output

Run the Auditbeat setup command using sudo auditbeat setup.

Start Auditbeat using sudo systemctl start auditbeat.service.

Now you should be able to verify events are being populated in the auditbeat-* Data View within Kibana.

Filebeat

You can use the Auditd module for Filebeat to collect the Auditd logs as well. To do this, install Filebeat and then enable the Auditd module

sudo filebeat modules enable auditd

Next, go into the Auditd configuration file and enable log collection, test, setup, and then start Filebeat.

Enabling Auditd log in the Filebeat configuration file

sudo vi /etc/filebeat/modules.d/auditd.yml

# Module: auditd

# Docs: <https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-auditd.html>

* module: auditd
  log:
    enabled: true

# Set custom paths for the log files. If left empty

# Filebeat will choose the paths depending on your OS

    #var.paths:

Testing the Filebeat configuration and output settings

sudo filebeat test config
sudo filebeat test output

Run the Filebeat setup command using sudo filebeat setup.

Start Filebeat using sudo systemctl start filebeat.service.

Detecting Dirty Pipe with Elastic

Now that Linux Audit Framework events are being populated by either the Elastic Agent, Auditbeat, or Filebeat, you can run queries to detect exploitation attempts using the Kibana Query Language (KQL) in Discover or the Endpoint Query Language (EQL) in Kibana’s Security → Timelines → New Timeline → Correlation query editor.

Hunt queries in Kibana

KQL query compatible with using the Elastic Agent, Auditbeat, or Filebeat:

KQL query to detect Dirty Pipe exploitation attempts

auditd.log.key : dirtypipe and process.name : *

EQL query compatible with using the Auditbeat:

EQL query to detect Dirty Pipe exploitation attempts

process where tags : "dirtypipe" and not process.name : ""

Detection Engine alerts

You can also create a Detection Engine alert to monitor for exploitation attempts.

Exploitation attempts will be recorded in the Kibana Security Solution in the Alerts section.

Respond to Observed Threats

Elastic makes it easy to quickly respond to a threat by isolating the host while still allowing it to communicate with your stack in order to continue monitoring actions taken and/or remediate the threat.

Defense in Depth Recommendations

The following steps can be leveraged to improve a network’s protective posture:

1. Review and ensure that you have deployed the latest stable and vendor-supplied kernel for your OS’
2. Review and implement the above detection logic within your environment using technology described in the post
3. Maintain backups of your critical systems to aid in quick recovery

References

The following research was referenced throughout the document:

• Exploit CVE reference: CVE-2022-0847
• Write-up using eBPF for some detections: https://sysdig.com/blog/cve-2022-0847-dirty-pipe-sysdig
• Original Max Kellermann write-up: https://dirtypipe.cm4all.com/
• SUID shell: ​​https://haxx.in/files/dirtypipez.c
• Passwd overwrite: https://github.com/liamg/traitor
• Passwd overwrite: ​​https://github.com/imfiver/CVE-2022-0847
• Metasploit module: https://github.com/rapid7/metasploit-framework/pull/16303
• Original Auditd detection logic: https://twitter.com/jonasl/status/1501840914381258756?s=20&t=MIWwwXpl5t0JiopVxX5M5Q

On February 23, 2022, the ESET threat research team disclosed a series of findings pertaining to a Data Wiper malware campaign, impacting hundreds of systems across Ukraine, named HERMETICWIPER. Elastic previously published research on Operation Bleeding Bear, a campaign targeted towards Ukrainian assets with similar destructive intentions.

Malware Wipers remain a common tactic of adversaries looking to cause havoc on systems impacted by their payloads. Typically this class of malware is designed to wipe the contents of any drives a system may have, rendering the end-users personal data lost. Many more recent examples of this class of payload incorporate tactics that also tamper with the boot process, with HERMETICWIPER being no exception.

Customers leveraging the Elastic Agent version 7.9+, and above are protected against this specific malware, with further research being undertaken to improve detection efficacy.

Malware Wipers & Ukrainian Targets

Unfortunately, this is not the first time this year that Ukranian systems have been the target of Data-wiping payloads - Microsoft published findings pertaining to similar, observed attacks that impacted systems within Ukraine, however initially impacting a far smaller number of systems. The publication outlined that the targeting of this specific earlier campaign was focused on multiple government agencies, non-profits, and information technology organizations throughout the country.

Malware Stage Analysis

HERMETICWIPER is digitally signed by Hermetica Digital Ltd., an organization registered in Cyprus, and embeds 4 legitimate driver files from EaseUS Partition Manager that are compressed using MS-DOS utility (mscompress). Hermetica Digital Ltd. has revoked the code-signing certificate.

Upon execution, HERMETICWIPER creates a kernel mode service and interacts with it via DeviceIoControl API function. The main objective is to corrupt any attached physical drive and render the system data unrecoverable.

Below is a summary of the events generated during the installation phase using, Windows events logs and Elastic Agent.

Following the installation process, HERMETICWIPER determines the dimensions of each partition by calculating the bytes in each sector and sectors in each cluster using the GetDiskFreeSpaceW Windows API function.

The malware interacts with the IOCTL interface, passing the parameter IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS with a value of 0x560000 to the device driver in order to retrieve the physical location of the root driver (\.\C). The root drive corresponds to the volume Windows uses to boot, and its identification is essential to achieve a destructive impact.

The NTFS/FAT boot sector and random file physical offsets are enumerated for each accessible physical drive, and then overwritten by the output of the CryptGenRandom API function and a series of FSCTL_GET_RETRIEVAL_POINTERS and FSCTL_MOVE_FILE IOCTLs.

Once the system crashes or restarts, the system is unable to boot and the data is corrupted.

Interesting Functionality

Similar to different ransomware families, HERMETICWIPER avoids specific critical folders and files during the wiping process. This ensures the machine is still operable and will not impact the disk wiping/file corrupting process at a later stage.

Another interesting technique observed when targeted files are queued for wiping is how they are accessed by concatenating the value ::$INDEX_ALLOCATION to a filename. This documented NTFS trick is an additional method to bypass access-control list (ACL) permissions on targeted files to provide more reliability when accessing these files.

HERMETICWIPER also modifies two registry settings during execution (ShowCompColor and ShowInfoTip), setting those key values to 0. Within Windows, when a user chooses to compress NTFS directories/files, there is a setting that allows the user to differentiate them in Windows Explorer showing them as blue representing compressed data or green for encrypted data. This is an attempt by the malware to not set off any suspicious behavior to the user with different coloring on directories/files before the disk corruption occurs on the machine.

Shredding Component Analysis

The malware wipes specific target folders/files writing pre-generated random data at specific disk addresses. It does this by setting up 4 different shredding queues in the binary.

Each queue usage and its functionality is undetermined, but are used at different points in the sample. The shredding queue is composed of a linked list of targets which contain random pre-generated data (generated at queuing) of the size of the target, the disk number and a linked list of “file” parts with disk addresses and sizes.

HERMETICWIPER Structure for ShredTarget function

struct ctf::ShredTarget
{
ctf::ShredTarget *p_next;
ctf::ShredTarget *p_prev;
ctf::FilePart *p_parts;
int disk_number;
uint8_t *p_random_filled_buffer;
int p_random_filled_buffer_size;
};

HERMETICWIPER Structure for FilePart function

struct ctf::FilePart
{
ctf::FilePart *p_next;
ctf::FilePart *p_prev;
uint64_t start_address;
uint64_t size;
};

HERMETICWIPER targeting file, folder, and disk partitions

ctf::QueueFileShred
ctf::QueueFolderShred
ctf::callback::IfPathContainNtUserQueueFileShred
ctf::callback::QueueNtfsBitmapAndLogAttributeShred
ctf::callback::QueueFileShredIfNotSymlink
ctf::callback::QueuePartitionFirstClusterShred
ctf::callback::QueuePartitionShred

The malware emphasizes the following items that are targeted for shredding.

• The dropped driver if something goes wrong or after service start:

• The malware process itself if driver launch goes wrong:

• The disk’s partition first cluster (enumerates up to 100):

• The System Volume information direct used to store Windows restore points:

Interestingly if the computer doesn’t belong to a domain controller it will target more assets:

After queuing the different targets previously described, the sample starts different synchronous/asynchronous shredding threads for each of its queues:

The thread launcher will then start a new thread for each target.

The shredding thread will then iterate through the target’s file parts and use the driver for writing at addresses on specified disk.

Driver Analysis

The driver that is loaded by the user mode component is quite similar to the driver that belongs to Eldos Rawdisk and has been leveraged previously by threat actors like Shamoon and Lazarus. The difference is that HERMETICWIPER abuses a driver (epmntdrv.sys) that belongs to EaseUS Partition Master, a legitimate disk partitioning software.

When the driver is loaded, it creates a device named \Device\EPMNTDRV and creates a symbolic link to be exposed to user mode. Then, it initializes the driver object with the following entry points.

Looking at the dispatch function that handles the IRP_MJ_CREATE requests, we can see that the driver builds the name of the symlink \Device\HarddiskX\Partition0 and saves a pointer to its file object on the driver’s file object fs context. The driver then uses the volume manager device object to obtain a pointer to the highest level device object in the disk device stack.

After that, it iterates over the stack looking for the Disk driver, that is the Microsoft storage class driver that implements functionality common to all storage devices. Once found, it saves a pointer to its device object in the FsContext2 field of the file object structure.

Moving to the function that handles the write requests, we can see that it builds an asynchronous Input Output Request Packet (IRP), which is an API used for drivers to communicate with each other, and forwards it the volume manager device. The buffer used in the IRP is described by the Memory Descriptor List (MDL) driver function. Finally, a completion routine is provided that will free the MDL and release memory used by the IRP.

The read requests are similar to the write requests in concept, in other words, the IoBuildsynchronousFsdRequest() API function uses the IRP_MJ_READ driver function instead of the IRP_MJ_WRITE driver function when sending the IRP to the driver. Finally, the routine that handles I/O control codes finds the highest device object in the stack where the volume manager is located and calls IoBuildDeviceIoControlRequest() to forward the IRP that contains the I/O control code to the appropriate driver.

All in all, the driver functionality is very simple. It acts as a proxy between user space and the low level file system drivers, allowing raw disk sector manipulation and as a result circumventing Windows operating system security features.

Prebuilt Detection Engine Alerts

The following existing public detection rules can also be used to detect some of the employed post exploitation techniques described by Symantec Threat Intelligence Team and ESET [1][2][3] :

• Suspicious Cmd Execution via WMI (Deployment of wiper via Impacket WMI)
• Direct Outbound SMB Connection (SMB spreader)
• Remotely Started Services via RPC (Remcom)
• Lateral Tool Transfer (staging PE via file shares for remote execution)
• Potential Credential Access via Windows Utilities
• Potential Credential Access via LSASS Memory Dump
• Process Execution from an Unusual Directory
• Execution from Unusual Directory - Command Line
• Scheduled Task Execution
• Scheduled Task Creation
• Suspicious MSHTA Execution

YARA Rules

rule Windows_Wiper_HERMETICWIPER {
    meta:
        Author = "Elastic Security"
        creation_date = "2022-02-24"
        last_modified = "2022-02-24"
        os = "Windows"
        arch = "x86"
        category_type = "Wiper"
        family = "HERMETICWIPER"
        threat_name = "Windows.Wiper.HERMETICWIPER"
        description = "Detects HERMETICWIPER used to target Ukrainian organization"
        reference_sample = "1bc44eef75779e3ca1eefb8ff5a64807dbc942b1e4a2672d77b9f6928d292591"

    strings:
        $a1 = "\\\\?\\C:\\Windows\\System32\\winevt\\Logs" wide fullword
        $a2 = "\\\\.\\EPMNTDRV\\%u" wide fullword
        $a3 = "tdrv.pdb" ascii fullword
        $a4 = "%s%.2s" wide fullword
        $a5 = "ccessdri" ascii fullword
        $a6 = "Hermetica Digital"
    condition:
        all of them
}

Observables

| Observable | Type | Reference | Note | | ---------------------------------------------------------------- | ------- | ------------- | ------------- | | 1bc44eef75779e3ca1eefb8ff5a64807dbc942b1e4a2672d77b9f6928d292591 | SHA-256 | Wiper malware | HERMETICWIPER | | 0385eeab00e946a302b24a91dea4187c1210597b8e17cd9e2230450f5ece21da | SHA-256 | Wiper malware | HERMETICWIPER | | 3c557727953a8f6b4788984464fb77741b821991acbf5e746aebdd02615b1767 | SHA-256 | Wiper malware | HERMETICWIPER | | 2c10b2ec0b995b88c27d141d6f7b14d6b8177c52818687e4ff8e6ecf53adf5bf | SHA-256 | Wiper malware | HERMETICWIPER |

Artifacts

Artifacts are also available for download in both ECS and STIX format in a combined zip bundle.

References

The following research was referenced throughout the document:

• https://twitter.com/ESETresearch/status/1496581903205511181
• https://twitter.com/juanandres_gs/status/1496607141888724997
• https://elastic.co/security-labs/operation-bleeding-bear
• https://therecord.media/microsoft-data-wiping-malware-disguised-as-ransomware-targets-ukraine-again/
• https://opencorporates.com/companies/cy/HE419469
• https://www.easeus.com/partition-manager
• https://docs.microsoft.com/en-us/windows/win32/devio/device-input-and-output-control-ioctl-
• https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getdiskfreespacew
• https://docs.microsoft.com/en-us/windows/win32/secauthz/access-tokens
• https://docs.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-findresourcew
• https://docs.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-loadresource

BPFDoor is a backdoor payload specifically crafted for Linux. Its purpose is for long-term persistence in order to gain re-entry into a previously or actively compromised target environment. It notably utilizes BPF along with a number of other techniques to achieve this goal, taking great care to be as efficient and stealthy as possible. PWC researchers discovered this very interesting piece of malware in 2021. PWC attributes this back door to a specific group from China, Red Menshen, and detailed a number of interesting components in a high-level threat research post released last week.

PWC’s findings indicated that ​​Red Menshen had focused their efforts on targeting specific Telecommunications, Government, Logistics, and Education groups across the Middle East and Asia. This activity has been across a Monday-to-Friday working period, between 01:00 UTC and 10:00 UTC, indicating that the operators of the malware were consistent in their attacks, and operation during a working week.

Perhaps most concerningly, the payload itself has been observed across the last 5 years in various phases of development and complexity, indicating that the threat actor responsible for operating the malware has been at it for some time, undetected in many environments.

BPFDoor Tools

The Elastic Security Team has created a few tools that will aid researchers in analyzing the BPFDoor malware.

The BPFDoor scanner will allow you to scan for hosts infected with the BPFDoor malware and the BPFDoor configuration extractor will allow you to extrapolate the malware’s configuration or hardcoded values which can lead to additional observations you can use for further analysis, developing additional signatures or connecting to the backdoor utilizing our client.

• BPFDoor scanner
• BPFDoor configuration extractor

Attack Lifecycle

This inherently passive backdoor payload is built to be a form of persistence – a method to regain access if the first or second stage payloads are lost. It is built for and intended to be installed on high-uptime servers or appliances, IoT/SCADA, or cloud systems with access to the Internet. The backdoor usually sits in temporary storage so if a server were to be rebooted or shut down, the backdoor would be lost.

It should be assumed that if this malware is found on a system the initial-access (1st stage) or post-exploitation (2nd stage) payloads are still most likely present and possibly active elsewhere in the environment. This backdoor excels at stealth, taking every opportunity to blend in and remain undetected.

In the below steps, we will break BPFDoor’s actions down according to the vast majority of the samples available.

1. When executed the binary copies itself into /dev/shm/. A temporary filesystem /dev/shm stands for shared memory and is a temporary file storage facility serving as an efficient means of inter-process communication
2. Renames its process to kdmtmpflush, a hardcoded process name
3. Initializes itself with the -init flag and forks itself. Forking in Linux means creating a new process by duplicating the calling process
4. Deletes itself by removing the original binary invoked. The forked process continues to run
5. Alters the forked processes’ creation and modification time values, also known as timestomping
6. Creates a new process environment for itself and removes the old one setting (spoofing) a new process name. It changes the way it appears on the system akin to wearing a mask. The process is still kdmtmpflush but if you were to run a ps you would see whatever value it set
7. Creates a process ID (PID) file in /var/run. PID files are text files containing the process of the associated program meant for preventing multiple starts, marking residency, and used by the program to stop itself. This file resides in /var/run, another temporary file storage facility
8. Creates a raw network socket. On Linux, a socket is an endpoint for network communication that allows you to specify in detail every section of a packet allowing a user to implement their own transport layer protocol above the internet (IP) level
9. Sets BPF filters on the raw socket. BPF allows a user-space program to attach a filter onto any socket and allow or disallow certain types of data to come through the socket
10. Observes incoming packets
11. If a packet is observed that matches the BPF filters and contains the required data it is passed to the backdoor for processing
12. It forks the current process again
13. Changes the forked processes working directory to /
14. Changes (spoofs) the name of the forked process to a hardcoded value
15. Based on the password or existence of a password sent in the “magic packet” the backdoor provides a reverse shell, establishes a bind shell, or sends back a ping

Atypical BPFDoor sample

Of note there is one sample we have come across that does not seem to exhibit steps 1 - 4. It doesn’t alter its initial name to a hardcoded value and simply executes from its placed location, otherwise, it models the same behavior.

Below you can see visual representations of the BPFDoor process tree, utilizing Elastic’s Analyzer View. The first image displays the tree prior to active use of the backdoor (i.e reverse shell, bind shell, or pingback) and the second image after a reverse shell has connected and performed post-exploitation activities.

Defense Evasion Insights

BPFDoor is interesting given the anti-forensics, and obfuscation tactics used. Astute readers will observe slight differences in the PID tree visible when running a ps ajxf on an infected host when compared to executed data within the Analyzer View inside of Elastic. This is due to the process name spoofing mentioned in step 6 (above) of the attack lifecycle above. The image below is taken from a system running BPFDoor with an active reverse shell connection established:

The difference lies in the fact that kdmtmpflush and sh are run prior to spoofing, and are captured at runtime by Elastic Endpoint. This is an accurate representation of the processes active on the host, further confirming the importance of appropriate observation software for Linux hosts - you can’t always trust what you see on the local system:

BPFDoor also holds in its repertoire the ability to subvert the traditional Linux socket client - server architecture in order to hide its malicious traffic. The methods which it utilizes to achieve this are both unusual and intriguing.

The sockets interface is almost synonmous with TCP/IP communication. This simple interface has endured for over 40 years - predating both Linux and Windows implementations.

BPFDoor uses a raw socket (as opposed to ‘cooked’ ones that handle IP/TCP/UDP headers transparently) to observe every packet arriving at the machine, ethernet frame headers and all. While this might sound like a stealthy way to intercept traffic, it’s actually not – on any machine with a significant amount of network traffic the CPU usage will be consistently high.

That’s where BPF comes in - an extremely efficient, kernel-level packet filter is the perfect tool to allow the implant to ignore 99% of network traffic and only become activated when a special pattern is encountered. This implant looks for a so-called magic packet in every TCP, UDP and ICMP packet received on the system.

Once activated, a typical reverse shell - which this back door also supports - creates an outbound connection to a listener set up by the attacker. This has the advantage of bypassing firewalls watching inbound traffic only. This method is well-understood by defenders, however. The sneakiest way to get a shell connected would be to reuse an existing packet flow, redirected to a separate process.

In this attack, the initial TCP handshake is done between the attacker and a completely legitimate process – for example nginx or sshd. These handshake packets happen to be also delivered to the backdoor (like every packet on the system) but are filtered out by BPF. Once the connection is established, however, BPFDoor sends a magic packet to the legitimate service. The implant receives it and makes a note of the originating IP and port the attacker is using, and it opens a new listening socket on an inconspicuous port (42391 - 43391).

The implant then reconfigures the firewall to temporarily redirect all traffic from the attacker’s IP/port combination to the new listening socket. The attacker initiates a second TCP handshake on the same legitimate port as before, only now iptables forwards those packets to the listening socket owned by the implant. . This establishes the communication channel between attacker and implant that will be used for command and control. The implant then covers its tracks by removing the iptables firewall rules that redirected the traffic.

Despite the firewall rule being removed, traffic on the legitimate port will continue to be forwarded to the implant due to how Linux statefully tracks connections. No visible traffic will be addressed to the implant port (although it will be delivered there).

BPF Filters

As stated in step 9 (above), BPF or Berkeley Packet Filters is a technology from the early ’90s that allows a user-space program to attach a network filter onto any socket and allow or disallow certain types of data to come through the socket. These filters are made up of bytecode that runs on an abstract virtual machine in the Linux kernel. The BPF virtual machine has functionality to inspect all parts of incoming packets and make an allow/drop decision based on what it sees. . You can see in the image example below what this looks like within the BPFDoor source code:

We took this BPF code, converted it, and wrote it up as pseudo code in an effort to aid our research and craft packets able to successfully get through these filters in order to activate the backdoor.

The above capabilities allow BPFDoor to attach a filter onto any socket and allow or disallow certain types of data to come through the socket - used carefully by the adversary to invoke a series of different functions within the payload.

Historical Analysis

We wanted to see over time, between BPFDoor payloads, what, if anything, the threat actors modified. A number of samples were detonated and analyzed ranging from the uploaded source code to a sample uploaded last month. We found that the behavior over time did not change a great deal. It maintained the same relative attack lifecycle with a few variations with the hardcoded values such as passwords, process names, and files - this is not uncommon when compared to other malware samples that look to evade detection or leverage payloads across a variety of victims.

We posture that the threat group would change passwords and update process or file names in an effort to improve operational security and remain hidden. It also makes sense that the general functionality of the backdoor would not change in any great way. As the saying goes “If it’s not broken, don’t fix it”. Our malware analysis and reverse engineering team compared the source code (uploaded to VirusTotal and found on Pastebin) to a recently uploaded sample highlighting some of the notable changes within the main function of the malware in the images below.

As we mentioned earlier, one recent sample we have come across that does not seem to exhibit some of the tactics of prior payloads has been observed - It doesn’t alter its initial name to a hardcoded value and simply executes from its placed location, otherwise, it models relatively the same behavior.

Linux Malware Sophistication

A trend we have had the privilege of observing at Elastic, is the threat landscape of Linux targeted attacks - these being focused often on cloud workloads, or systems that typically have less observational technology configured in many of the environments we see. The trend of complex, well-designed payloads is something that is often simply overlooked, and specifically in the case of BPFDoor, remained hidden for years.

It is important to consider these workloads a critical component of your security posture: A lack of visibility within cloud workloads will eventually lead to large gaps in security controls - adversarial groups are further growing to understand these trends, and act accordingly. Best practices state that endpoint defenses should be consistent across the fleet of systems under management, and conform to a least privilege architecture.

Detection of BPFDoor

After researching this malware it became apparent as to why the backdoor remained in use and hidden for so long. If you aren’t intimately familiar with Linux process abnormalities or weren’t looking for it you would generally not detect it. Even though it takes advantage of Linux capabilities in a stealthy manner to evade detection, there are still opportunities for both behavioral and signature-based detections.

The first area of opportunity we witnessed while testing was the behavior we observed during the initial execution of the malware, specifically its working directory, in a shared memory location /dev/shm. This is a native temporary filesystem location in Linux that uses RAM for storage, and a binary executing from it let alone generating network connections is fairly uncommon in practice.

During execution, BPFDoor removes existing files from /dev/shm and copies itself there prior to initialization. A detection for this would be any execution of a binary from this directory as root (you have to be root to write to and read from this directory).

This was verified by detonating the binary in a VM while our Elastic Agent was installed and observing the sequence of events. You can see an image of this detection on the Kibana Security Alerts page below. This rule is publicly available as an Elastic SIEM detection rule - Binary Executed from Shared Memory Directory:

The second opportunity we noticed, for detection, was a specific PID file being created in /var/run. We noticed the dropped PID file was completely empty while doing a quick query via the Osquery integration to the /var/run directory. While this is not inherently malicious, it is unusual for the file size of a PID to be 0 or above 10 bytes and thus we created an additional rule centered around detecting this unusual behavior.

Our Abnormal Process ID or Lock File Created rule identifies the creation of a PID file in the main directory of /var/run with no subdirectory, ignoring common PID files to be expected:

The third area we wanted to look at was the network connections tied to two of the three capabilities (reverse shell and bind shell) the backdoor possesses. We wanted to see if there were any suspicious network connections tied to process or user abnormalities we could sequence together based off of the way BPFDoor handles establishing a reverse or bind shell.

The reverse shell was the first capability focused on. Taking a deep look at the process tree in and around the reverse shell establishment allowed us to key in on what would be considered a strange or even abnormal sequence of events leading to and involving an outbound network connection.

We developed a hunt rule sequence that identifies an outbound network connection attempt followed by a session id change as the root user by the same process entity. The reason we developed these network focused hunt rules is due to possible performance issues caused if running these continually.

The bind shell was the last capability we honed in on. Identifying an abnormal sequence of events surrounding the bind shell connection was difficult due to the way it forks then accepts the connection and kills the accepting process post established connection. Therefore we had to focus on the sequence of events within the process entity id directly involving the network connection and subsequent killing of the accepting process.

After developing the 2 detection rules along with the 2 hunt rules listed below and in addition to the 6 YARA signatures deployed we were able to detect BPFDoor in a myriad of different ways and within different stages of its life cycle. As stated earlier though, if you detect this malware in your environment it should be the least of your concerns given the threat actor will most likely have already successfully compromised your network via other means.

Existing Detection Rules

The following Elastic Detection Rules will identify BPFDoor activity:

• Abnormal Process ID or Lock File Created
• Binary Executed from Shared Memory Directory

Hunting Queries

This EQL rule can be used to successfully identify BPFDoor reverse shell connections having been established within your environment:

EQL BPFDoor reverse shell hunt query

sequence by process.entity_id with maxspan=1m
[network where event.type == "start" and event.action == "connection_attempted" and user.id == "0" and not process.executable : ("/bin/ssh", "/sbin/ssh", "/usr/lib/systemd/systemd")]
[process where event.action == "session_id_change" and user.id == "0"]

The hunt rule we created here identifies a sequence of events beginning with a session id change, followed by a network connection accepted, in correlation with ptmx file creation and a deletion of the process responsible for accepting the network connection. This EQL rule can be used to successfully identify BPFDoor bind shell connections within your environment:

EQL BPFDoor bind shell hunt query

sequence by process.entity_id with maxspan=1m
[process where event.type == "change" and event.action == "session_id_change" and user.id == 0 and not process.executable : ("/bin/ssh", "/sbin/ssh", "/usr/lib/systemd/systemd")]
[network where event.type == "start" and event.action == "connection_accepted" and user.id == 0]
[file where event.action == "creation" and user.id == 0 and file.path == "/dev/ptmx"]
[process where event.action == "end" and user.id == 0 and not process.executable : ("/bin/ssh", "/sbin/ssh", "/usr/lib/systemd/systemd")]

YARA Rules

In addition to behavioral detection rules in the Elastic Endpoint, we are releasing a set of BPFDoor Yara signatures for the community.

BPFDoor YARA rule

rule Linux_Trojan_BPFDoor_1 {

    meta:
        Author = "Elastic Security"
        creation_date = "2022-05-10"
        last_modified = "2022-05-10"
        os = "Linux"
        arch = "x86"
        category_type = "Trojan"
        family = "BPFDoor"
        threat_name = "Linux.Trojan.BPFDoor"
        description = "Detects BPFDoor malware."
        reference_sample = "144526d30ae747982079d5d340d1ff116a7963aba2e3ed589e7ebc297ba0c1b3"
    strings:
        $a1 = "hald-addon-acpi: listening on acpi kernel interface /proc/acpi/event" ascii fullword
        $a2 = "/sbin/iptables -t nat -D PREROUTING -p tcp -s %s --dport %d -j REDIRECT --to-ports %d" ascii fullword
        $a3 = "avahi-daemon: chroot helper" ascii fullword
        $a4 = "/sbin/mingetty /dev/tty6" ascii fullword
        $a5 = "ttcompat" ascii fullword
    condition:
        all of them
}

rule Linux_Trojan_BPFDoor_2 {
    meta:
        Author = "Elastic Security"
        creation_date = "2022-05-10"
        last_modified = "2022-05-10"
        os = "Linux"
        arch = "x86"
        category_type = "Trojan"
        family = "BPFDoor"
        threat_name = "Linux.Trojan.BPFDoor"
        description = "Detects BPFDoor malware."
        reference_sample = "3a1b174f0c19c28f71e1babde01982c56d38d3672ea14d47c35ae3062e49b155"
    strings:
        $a1 = "hald-addon-acpi: listening on acpi kernel interface /proc/acpi/event" ascii fullword
        $a2 = "/sbin/mingetty /dev/tty7" ascii fullword
        $a3 = "pickup -l -t fifo -u" ascii fullword
        $a4 = "kdmtmpflush" ascii fullword
        $a5 = "avahi-daemon: chroot helper" ascii fullword
        $a6 = "/sbin/auditd -n" ascii fullword
    condition:
        all of them
}

rule Linux_Trojan_BPFDoor_3 {
    meta:
        Author = "Elastic Security"
        creation_date = "2022-05-10"
        last_modified = "2022-05-10"
        os = "Linux"
        arch = "x86"
        category_type = "Trojan"
        family = "BPFDoor"
        threat_name = "Linux.Trojan.BPFDoor"
        description = "Detects BPFDoor malware."
        reference_sample = "591198c234416c6ccbcea6967963ca2ca0f17050be7eed1602198308d9127c78"
    strings:
        $a1 = "[-] Spawn shell failed." ascii fullword
        $a2 = "[+] Packet Successfuly Sending %d Size." ascii fullword
        $a3 = "[+] Monitor packet send." ascii fullword
        $a4 = "[+] Using port %d"
        $a5 = "decrypt_ctx" ascii fullword
        $a6 = "getshell" ascii fullword
        $a7 = "getpassw" ascii fullword
        $a8 = "export %s=%s" ascii fullword
    condition:
        all of them
}

rule Linux_Trojan_BPFDoor_4 {
    meta:
        Author = "Elastic Security"
        creation_date = "2022-05-10"
        last_modified = "2022-05-10"
        os = "Linux"
        arch = "x86"
        category_type = "Trojan"
        family = "BPFDoor"
        threat_name = "Linux.Trojan.BPFDoor"
        description = "Detects BPFDoor malware."
        reference_sample = "591198c234416c6ccbcea6967963ca2ca0f17050be7eed1602198308d9127c78"
    strings:
        $a1 = { 45 D8 0F B6 10 0F B6 45 FF 48 03 45 F0 0F B6 00 8D 04 02 00 }
    condition:
        all of them
}

rule Linux_Trojan_BPFDoor_5 {
    meta:
        Author = "Elastic Security"
        creation_date = "2022-05-10"
        last_modified = "2022-05-10"
        os = "Linux"
        arch = "x86"
        category_type = "Trojan"
        family = "BPFDoor"
        threat_name = "Linux.Trojan.BPFDoor"
        description = "Detects BPFDoor malware."
        reference_sample = "76bf736b25d5c9aaf6a84edd4e615796fffc338a893b49c120c0b4941ce37925"
    strings:
        $a1 = "getshell" ascii fullword
        $a2 = "/sbin/agetty --noclear tty1 linux" ascii fullword
        $a3 = "packet_loop" ascii fullword
        $a4 = "godpid" ascii fullword
        $a5 = "ttcompat" ascii fullword
        $a6 = "decrypt_ctx" ascii fullword
        $a7 = "rc4_init" ascii fullword
        $b1 = { D0 48 89 45 F8 48 8B 45 F8 0F B6 40 0C C0 E8 04 0F B6 C0 C1 }
    condition:
        all of ($a*) or 1 of ($b*)
}

rule Linux_Trojan_BPFDoor_6 {
    meta:
        Author = "Elastic Security"
        creation_date = "2022-05-10"
        last_modified = "2022-05-10"
        os = "Linux"
        arch = "x86"
        category_type = "Trojan"
        family = "BPFDoor"
        threat_name = "Linux.Trojan.BPFDoor"
        description = "Detects BPFDoor malware."
        reference_sample = "dc8346bf443b7b453f062740d8ae8d8d7ce879672810f4296158f90359dcae3a"
    strings:
        $a1 = "getpassw" ascii fullword
        $a2 = "(udp[8:2]=0x7255) or (icmp[8:2]=0x7255) or (tcp[((tcp[12]&0xf0)>>2):2]=0x5293)" ascii fullword
        $a3 = "/var/run/haldrund.pid" ascii fullword
        $a4 = "Couldn't install filter %s: %s" ascii fullword
        $a5 = "godpid" ascii fullword
    condition:
        all of them
}

Interacting with BPFDoor

The Elastic Security Team has released several tools that can aid in further research regarding BPFDoor to include a network scanner used to identify infected hosts, a BPFDoor malware configuration extractor, and a BPFDoor client binary that can be used to actively interact with a sample.

BPFDoor Scanner

The Elastic Security Team has released a Python script that can identify if you have BPFDoor infected hosts.

The scanner sends a packet to a defined IP address using the default target port (68/UDP)and default interface. It listens to return traffic on port 53/UDP.

BPFDoor Configuration Extractor

This tool will allow you to extract configurations from any BPFDoor malware you may have collected. This will allow you to develop additional signatures and further analysis of the malware as well as your environment.

The BPFDoor configuration extractor can be downloaded here.

BPFDoor Client POC

Quickly after beginning our research into this malware we realized we would also need to actively interact with BPFDoor in order to observe the full extent of the capabilities that it possesses and monitor what these capabilities would look like from a host and SIEM level.

In order to do this, we had to break down the BPF filters in the BPFDoor source code so we could craft packets for the different protocols. To do this, we used Scapy, a packet manipulation program, to ensure we could pass the filters for the purpose of activating the backdoor. Once we ensured we could pass the filters, Rhys Rustad-Elliott, an engineer at Elastic built a BPFDoor client that accepts a password, IP address, and port allowing you to connect to a BPFDoor sample and interact if you possess the sample’s hardcoded passwords.

Depending on the password or lack of password provided, BPFDoor will behave exactly the same way it would in the wild. You can invoke a reverse shell, establish a bind shell, or connect to it with no supplied password to receive a ping-back confirming its installation.

Researchers looking to use BPFDoor can reach out to Elastic Security for access to the BPFDoor client POC. Please note that these tools will be shared at our discretion with those in the trusted security community looking to improve the detection of this vulnerability.

Impact

The following MITRE ATT&CK Tactic, Techniques, and Sub-techniques have been observed with the BPFDoor malware.

Tactics

Tactics represent the “why” of an ATT&CK technique or sub-technique. It is the adversary’s tactical goal: the reason for performing an action.

• Execution

Techniques (sub-techniques)

Techniques (and sub-techniques) represent ‘how’ an adversary achieves a tactical goal by performing an action.

• Native API
• External Remote Services
• Hide Artifacts
• Indicator Removal on Host
• Non-Application Layer Protocol
• Command and Scripting Interpreter: Unix Shell
• Abuse Elevation Control Mechanism: Setuid and Setgid

Source Pseudocode

To clearly articulate the details of this malware, we’ve created two diagrams that outline the specific pseudocode for BPFDoor based on the source code uploaded to VT and found on Pastebin. While this contains a lot of detail, it is simple to understand if researchers choose to further this research.

Summary

While threat groups continue to increase in maturity, we expect this kind of mature, well designed and hidden threat will continue to be found within Linux environments. These kinds of findings reiterate the importance of comprehensive security controls across the entirety of a fleet, rather than simply focusing on user endpoints.

BPFDoor demonstrates a perfect example of how important monitoring workloads within Linux environments can be. Payloads such as this are near-on impossible to observe and detect without sufficient controls, and should be considered a moving trend within the general adversarial landscape.

Observables

| Observable | Type | Reference | Note | | ---------------------------------------------------------------- | ------------ | -------------------- | -------------------------------- | | /dev/shm/kdmtmpflush | process name | BPFDoor process name | Observed process name of BPFDoor | | /var/run/haldrund.pid | file name | BPFDoor file name | Observed BPFDoor PID file | | /var/run/kdevrund.pid | file name | BPFDoor file name | Observed BPFDoor PID file | | /var/run/xinetd.lock | file name | BPFDoor file name | Observed BPFDoor lock file | | 74ef6cc38f5a1a80148752b63c117e6846984debd2af806c65887195a8eccc56 | SHA-256 | BPFDoor malware | | | 07ecb1f2d9ffbd20a46cd36cd06b022db3cc8e45b1ecab62cd11f9ca7a26ab6d | SHA-256 | BPFDoor malware | | | 76bf736b25d5c9aaf6a84edd4e615796fffc338a893b49c120c0b4941ce37925 | SHA-256 | BPFDoor malware | | | 93f4262fce8c6b4f8e239c35a0679fbbbb722141b95a5f2af53a2bcafe4edd1c | SHA-256 | BPFDoor malware | | | 96e906128095dead57fdc9ce8688bb889166b67c9a1b8fdb93d7cff7f3836bb9 | SHA-256 | BPFDoor malware | | | 599ae527f10ddb4625687748b7d3734ee51673b664f2e5d0346e64f85e185683 | SHA-256 | BPFDoor malware | | | 2e0aa3da45a0360d051359e1a038beff8551b957698f21756cfc6ed5539e4bdb | SHA-256 | BPFDoor malware | | | f47de978da1dbfc5e0f195745e3368d3ceef034e964817c66ba01396a1953d72 | SHA-256 | BPFDoor malware | | | fd1b20ee5bd429046d3c04e9c675c41e9095bea70e0329bd32d7edd17ebaf68a | SHA-256 | BPFDoor malware | | | 5faab159397964e630c4156f8852bcc6ee46df1cdd8be2a8d3f3d8e5980f3bb3 | SHA-256 | BPFDoor malware | | | f8a5e735d6e79eb587954a371515a82a15883cf2eda9d7ddb8938b86e714ea27 | SHA-256 | BPFDoor malware | | | 5b2a079690efb5f4e0944353dd883303ffd6bab4aad1f0c88b49a76ddcb28ee9 | SHA-256 | BPFDoor malware | | | 97a546c7d08ad34dfab74c9c8a96986c54768c592a8dae521ddcf612a84fb8cc | SHA-256 | BPFDoor malware | | | c80bd1c4a796b4d3944a097e96f384c85687daeedcdcf05cc885c8c9b279b09c | SHA-256 | BPFDoor malware | | | 4c5cf8f977fc7c368a8e095700a44be36c8332462c0b1e41bff03238b2bf2a2d | SHA-256 | BPFDoor malware | |

References

• https://doublepulsar.com/bpfdoor-an-active-chinese-global-surveillance-tool-54b078f1a896
• https://www.pwc.com/gx/en/issues/cybersecurity/cyber-threat-intelligence/cyber-year-in-retrospect/yir-cyber-threats-report-download.pdf
• https://www.pangulab.cn/en/post/the_bvp47_a_top-tier_backdoor_of_us_nsa_equation_group

Artifacts

Artifacts are also available for download in both ECS and STIX format in a combined zip bundle.

Readers Note:

Elastic has undergone a series of investigations internally and has not yet identified malicious actions that may pertain to this event. Okta has also released two statements relating to the incident in question that may be reviewed here and here.

The LAPSUS$ group

Financially motivated adversary groups executing ransomware attacks have rightfully gotten our attention in recent years. Similar to Lulzec, there’s a new group catching attention with different motivations, targeting larger organizations.

The LAPSUS$ group emerged onto the scene a number of months ago, targeting high-profile organizations such as Nvidia, Samsung, and Ubisoft — making various demands that in some cases, resulted in either data dumps or screenshots of internal systems shared via the group’s Telegram account. These were sometimes determined by user-voted polls within the group, suggesting that this is only the beginning of a series of attacks the group is undertaking more frequently as they gain press coverage.

Groups of this nature focus on data theft and extortion via means of social engineering — commonly, targeted spear phishing campaigns.

The latest organization under the microscope of the LAPSUS$ group is Okta, the identity provider for thousands of companies of all sizes. Surprisingly, LAPSUS$ chose to note in their release of information that their targeting of Okta was not for access to Okta’s systems, but rather that of their customers:

The latest target: Okta

After LAPSUS$ sent a notification last night to the Telegram account, Okta’s CEO responded with a series of Tweets and an official statement regarding the suspected compromise, stating that it occurred in January 2022, similar to dates visible in the screenshots from Telegram post:

While the initial notice provided some insights into potential scope and timing of the incident, many customers are still interested in identifying the scope of the access, and how to assess if there is any local impact within their organization.

The updated notice released by Okta suggested that access was limited to a specific end-user system with no ability to create or delete users or download customer information. However, it did have the ability to reset passwords and MFA tokens for users, while not obtaining access to them. Responses from LAPSUS$ are included for context, and suggest more may need to be investigated.

In the third update notification shared by David Bradbury at Okta, a correction was made indicating that a small (2.5% of customer base) were potentially impacted by the incident. Further details will be shared via a Webinar scheduled at 8AM, PDT Wednesday, March 23rd.. A link to sign up for the webinar is located within the aforementioned update post.

As more information pertaining to the breach is released by either LAPSUS$ or Okta, we will maintain the accuracy of information shared within this post.

Threat hunting Okta logs in Elastic

The good news is that customers of Okta do have access to relatively comprehensive log information regarding activity within their account. Okta has configured a default 90 day retention window for system events. Okta released an updated statement stating customers do not have to respond to the incident immediately, but for those wishing to investigate further, the following threat hunting information is still valuable.

The process to get started with ingesting Okta logs is simple — a prebuilt integration for Okta Log ingestion is available as a one-click module configurable within Kibana:

Alternatively, the Okta Filebeat Module can easily be added to Elastic to provide insights into previous account activity.

Configuring the Okta Module is simple, providing you tweak the initial_interval value to 90 days:

~ ~ ~
- module: okta
  system:
    var.url: https://yourOktaDomain/api/v1/logs
    var.api_key: XXXX-XXXX...XXXX-XXXX'
    var.initial_interval: 90d # will fetch events starting 90 days ago.
~ ~ ~

Once events are ingested, a number of Lucene queries are easily leveraged for early/initial signs of compromise. While these events are not a comprehensive set of queries, they should provide ample detail for any security team to investigate potential suspicious activity:

MFA device reset via console for any user:

event.module:"okta" AND event.action:"user.mfa.factor.reset_all"

User Account email update records updated to a new value:

event.module:"okta" AND event.action:"user.account.update_primary_email"

User Privilege granted for an account within your Okta organization:

event.module:"okta" AND event.action:"user.account.privilege.grant"

Okta Administrative staff have a series of privileges that allow for user-impersonation via their management service. Logs pertaining to this action should be inspected:

event.module:okta AND (event.action:user.session.impersonation.grant OR event.action:user.session.impersonation.initiate)

There are many other ways to look for suspicious activity in your Okta data. In addition to these queries, Elastic provides a large set of prebuilt detections for suspicious Okta activity used by other adversarial groups in our open detection-rules repo. This will be useful in generating alerts as Okta logs are coming into Elastic. You can use the query logic in those rules to drive other hunts beyond the four we mention above as well.

Not familiar with what suspicious Okta data looks like?

Read the blog from December 2020 where we discussed the subject and released an open adversary simulation tool called Dorothy to help security teams test visibility, monitoring, and detection capabilities for Okta logs.

We expect many security teams will give SSO logs extra attention in light of this incident, and this tool may help teams get up to speed on the subject.

Earlier events: Microsoft, Nvidia, Samsung, Ubisoft

As previously stated, the LAPSUS$ group has been on a serious compromise train over the past few months, targeting a number of high-profile targets. Numerous details have been shared across a number of different media outlets, and a common theme of social engineering and internal access has been observed across many of the attacks:

• 37GB of Source Code was leaked from Microsoft in an earlier dump identified this week
• Ubisoft - Company-wide password reset after unusual activity identified on systems located
• Nvidia issues notice after internal systems indicate data compromise
• Samsung confirms source-code compromise via the LAPSUS$ group

As further information is uncovered, and mechanisms for detection improve, Elastic Security will continue to provide further updates to this and provide subsequent posts relating to detections.

If you haven’t checked out the Elastic Security solution, take a look at our Quick Start guides (bite-sized training videos to get you started quickly) or our free fundamentals training courses. You can always get started with a free 14-day trial of Elastic Cloud. Or download the self-managed version of the Elastic Stack for free.

The Microsoft 365 Defender team released a post detailing several identified vulnerabilities. These vulnerabilities allow adversarial groups to easily escalate privileges on Linux systems, allowing for deployment of payloads, ransomware, or other malicious actions. Collectively known as Nimbuspwn, these vulnerabilities include a series of security issues within networkd-dispatcher, specifically directory traversal, symlink race, and TOCTU race conditions.

Details are covered in their detailed post, and further information will be available within the two requested CVEs: CVE-2022-29799 and CVE-2022-29800. At the time of publication these CVE IDs are still reserved.

While this class of vulnerability requires local shell access to exploit, it should be considered important for those that currently leverage networkd-dispatcher within their Linux workload environments. A patch by the creator has been implemented to resolve the issue under the guidance of Microsoft, and should be implemented by those that have systems impacted by this vulnerability.

Detecting Nimbuspwn activity within Elastic

Our research team at Elastic has focused on building out a series of initial detections that leverage Elastic Security, alongside OSquery.

Firstly, those wishing to understand what systems in their environment may be impacted need to determine systems that have networkd-dispatcher installed:

Writing an OSquery search that returns the installed version of Networkd-Dispatcher is relatively trivial, and understanding the systems that may be at risk are returned at a glance. In the screenshot above, we can see an example host listed with a version number of 2.1-2, specific to Ubuntu. The version installed within your environment may be slightly different depending on the distribution. An example query has been provided below.

Select version from deb_packages rpm_packages where name=’networkd-dispatcher’;

We leveraged the initial research paper from Microsoft, determining a specific malicious pattern adversaries may use to exploit this vulnerability:

The Elastic Security team wrote an EQL Detection Rule to detect suspicious child processes of Networkd-Dispatcher. Any child-process detected by this rule should be considered highly suspicious given the circumstances, and should be investigated. Further analysis will likely be provided as our security community builds more POCs for this exploit. An example query appears below:

process where event.type == "start" and process.parent.name : "networkd-dispatcher" and not process.name in ("networkctl", "networkd-dispatcher")

Given the nature of this exploit, we expect far greater diversity in POCs over the coming weeks. You can expect updates in the form of further signatures or rules accordingly.

Defensive recommendations

Organizations impacted by vulnerabilities discovered by the Microsoft team should follow guidance provided by Microsoft in their initial post, and update their instances of networkd-dispatcher. Elastic recommends investigating hosts that are found to be running vulnerable versions of network-dispatcher with the aforementioned detections for any sign of compromise.

Not already using Elastic Security? You can always get started with a free 14-day trial of Elastic Cloud.