Most vendors cannot answer it cleanly. And the answer matters more than almost anything else on the RFP.

What open actually means

The industry has gotten comfortable treating “open” as a checkbox. Support an open schema, publish an API, sponsor a standard, done. But a standard tells you how data is shaped. It tells you nothing about whether you can actually get it. Open data means three things, and you need all three:

It is yours, at no additional cost. You generated this telemetry. You paid to collect it, and you paid to store it. If your vendor charges you a second time to access it, that is not a data feature. That is a toll booth on your own driveway.

It is all of your data. Not the alerts. Not a normalized summary. Not the tables the vendor decided are ready. The full-fidelity record because you cannot predict today which field matters in next year’s investigation.

It is real time. This one used to be a nice-to-have. Not anymore. CrowdStrike’s own 2026 Global Threat Report clocked the average eCrime breakout time at 29 minutes, the fastest at 27 seconds, and one intrusion where exfiltration started four minutes after initial access. Attackers are moving faster too, using AI to shorten the gap between breaking in and doing real damage. If your telemetry arrives in batches, minutes apart, the attack can be over before your data shows up. You would not accept a smoke detector that checks for fire twice an hour.

When a vendor fails any of these three, they are not protecting your data. They are building an artificial moat with it. Ingestion is always frictionless. Egress is licensed, delayed, or degraded. That asymmetry is not an accident of engineering. It is the business model, and the industry has a name for it: lock-in.

What the documentation actually says

Don’t take our word for it. Every claim in this table links to the vendor’s own documentation. Read it yourself, and hold us to the same bar.

Vendor

How you get your data out

Cost to access your data

Freshness

Fidelity

Openness

CrowdStrike

Falcon Data Replicator: batched file export to object storage

Licensed add-on

Bulk batches, not a live stream; deleted after 7 days for CrowdStrike managed-buckets

Raw telemetry is only available via FDR; the event stream API sends detections, not telemetry

Restricted

Palo Alto Networks

XSIAM Event Forwarding: batch files to a Palo Alto-managed bucket

Two paid Event Forwarding add-ons: GB and Endpoint

Batch; up to 2 hours to appear; kept 14 days

Endpoint and log data split across the two add-ons

Restricted

Microsoft

Defender streaming API; Sentinel data export

Streaming: pay Azure infra; Sentinel export billed per GB

Real-time stream (via Azure Event Hubs)

Generally-available fields only; Azure destinations only

Partially open

Google

Bulk export to your storage, or a continuous BigQuery feed

Export capped; BigQuery billed by query

Live feed 5-10 min behind (top tier only); raw export is point-in-time

Live feed is normalized only; no continuous raw path

Limited

Splunk 

Search/export REST API

Included

On-demand query via export data API; data available near-real-time

Full events returned as structured JSON

Open

Elastic

REST APIs including Search, Point in time and ES|QL

No export license or per-query export fee; Elastic Cloud applies ordinary cloud data-transfer metering, not an egress toll

On-demand query via API; data available to query as soon as it is indexed

Full raw and parsed events, returned as structured JSON

Open

This table reflects each vendor's public documentation as of September 2026.

A note on the ratings, because we want them to be defensible, not convenient. Open means all three tests pass: no added cost, full fidelity, and no batch window between the moment data arrives and the moment you can use it. Data you can query the moment it lands clears that bar; a scheduled batch measured in minutes or hours does not. Partially open means the vendor genuinely tries but with real constraints; credit to Microsoft for a true streaming path, even if it covers only generally-available fields and only lands in Azure. Limited means the data comes back, but late or in a form only the vendor can read. Restricted means access to your own telemetry is a paid product, a delayed batch, or both.

And since Elastic is on this list too, as an endpoint agent and a SIEM, hold us to the same bar. Our rating is about live access: APIs that return JSON, with no license between you and your own telemetry. On Elastic Cloud you pay ordinary data-transfer rates like any cloud service. What you never pay is a license to reach data that was already yours.

If any vendor believes we have mischaracterized their documentation, I genuinely want to hear it, and we will correct it.

So what does this actually cost you?

Here is what that comparison means when it really counts, in the middle of an incident. Detection you cannot act on in time is not detection. When your telemetry arrives in scheduled batches, a window opens between the alert and the evidence. Sometimes minutes, sometimes longer, and in that window the attack is live while your data is not yet in front of you. You paid to collect that telemetry. You paid to store it. And at the one moment it matters, it is still in transit. Or worse, it is not there at all because exporting it requires a separate license. That’s the difference between stopping an intrusion and reading about it afterward.

You do not create that gap during an incident. You inherit it at purchase. You already run a strong endpoint agent. It works, and you trust it. Now that same vendor offers to be your SIEM as well. One console, one relationship, one invoice. There is nothing wrong with one vendor doing both. We do both at Elastic. The question is what it costs you to change your mind. So look closely at what it takes to move that endpoint telemetry somewhere other than their own platform. Every vendor makes it easy to get data in. Getting it back out is something you buy, then wait for.

It is rarely just one handoff. Most security environments are heterogeneous by design, with each layer chosen because it is good at its own job. Modern attacks move across all of them, from a stolen identity to a cloud workload to an endpoint, and you only see the full chain if the data from each can meet in one place, quickly and without a toll. When a vendor makes its telemetry expensive or slow to share, it is not just charging you. It’s fragmenting the picture, and a fragmented picture is exactly where intrusions hide. The all-in-one pitch offers to solve that. But the fragmentation was manufactured: vendors made their data hard to move, then sold you the one console where it comes back together. So before convenience makes the decision for you, ask three questions, and make the vendor answer them in writing:

• Can I get all of my telemetry out, continuously, without buying a separate license to do it?

• How long, exactly, from the moment an event happens to the moment it is usable in a system I chose?

• On the day I add a different analytics engine, a data lake, or a new AI model, what breaks?

If the honest answers are “extra cost,” “in batches,” and “quite a lot,” then you are not buying a SIEM. You are renting access to your own data. The point is not that these layers must stay separate. Plenty of teams consolidate for good reasons, and we sell both layers ourselves. The point is that the choice should stay yours: you can add the analytics platform you want, or leave the one you have, without paying a toll or waiting on a batch to get your own data. The only reason to accept less is that a vendor made leaving hard enough that staying felt like a decision. It was not a decision. It was the absence of one.

The strategic question

Your SIEM is not a tool you bought. It is where isolated alerts become an attack story, and where you go to find out what actually happened. The vendor holding it is a strategic dependency, whether you planned it that way or not.

So evaluate them like one. Put data portability on the RFP, not in the demo, and score it on two things: what it costs to move your telemetry somewhere else, and how much delay it adds.

Because you cannot build a real-time defense on a delayed copy of your own telemetry, and you should not have to buy your data back to try.

It is your data. Any vendor who makes that complicated has told you what kind of partner they intend to be.

Before we go deeper, let's set some context. Many SIEMs treat entities as flat records, a snapshot of what's true right now, rebuilt on demand from raw logs. That works until you need to know how the environment got here, what changed, what was resolved, how a risk score compounded over time. Elastic's Entity Store is architected differently. Every entity is a living record, continuously enriched by background jobs called maintainers, each responsible for one facet of the entity. Maintainers establish relationships, perform identity resolution, and calculate risk scoring. These are all composed onto the record over time, all inspectable, and all correctable. A companion post on entity record quality explains why this matters.

Going one level deeper, this piece explores how records are built and connected in the first place, along with how their risk scores are updated, relationships are tracked, and identities are resolved. The short answer is a single abstraction. Entity store v2, introduced in Elastic v9.4, is built on maintainers. Each maintainer runs on its own clock, continuously updating the record you eventually query. The entity store is the result of a set of maintainers enriching each entity based on the raw or derived signal around it, not an append-only index you simply write entities into.

Let's build that picture from the engine up.

What the entity store does and why it runs on ES|QL

The Entity Store is the layer that turns your telemetry (for example endpoint, identity provider, and cloud logs), identity and asset inventories, and threat detections into one queryable profile per user, host, and service. This means analysts can pivot on information and insights gained from an enriched entity record, instead of reconstructing it from raw logs every time.

Entity store Version 1 did that with Elasticsearch transforms.

Version 2 rebuilds the engine on Elasticsearch Query Language (ES|QL). Each entity extraction cycle runs a query that filters the relevant logs and aggregates them. In the next crucial step, the cycle performs a LOOKUP JOIN back against the existing entity index to carry forward previously observed entity attributes and behaviors.

FROM logs-*
  | WHERE ...                                          // logs for this entity type + window
  | STATS ... BY entity.id                             // collapse many events into one record
  | LOOKUP JOIN .entities.v2.latest.security_default-00001
      ON entity.id                                     // field retention against the store itself
  | EVAL ...                                            // keep-latest / keep-first retention

ES|QL provides query flexibility and coverage for the fields and field types that the schema needs. It also provides a pipeline that you can reason about and extend, with retention and merge logic expressed directly, instead of through the limited flexibility and operational choreography needed with transforms.

How an entity record is built, from raw signal to resolved identity

Here's the part that's usually glossed over. Let's take a look at the entity extraction and building process by following one host from its raw event to its finished record.

• Step 1: The raw signal ingested to Elasticsearch. Your endpoint detection and response (EDR) agent and identity provider write events into data streams that are matched by a logs-* index pattern, as do your cloud integrations. There isn’t anything entity-shaped yet, just events with fields like user.name, host.name, and event.category.

• Step 2: Extraction collapses events into a record. During its run cycle, the ES|QL query above filters the logs relevant to an entity definition, and then STATS ... BY entity.id collapses potentially thousands of events into one entity store row/record per entity, keeping the first-seen and latest entity lifecycle values that matter. The LOOKUP JOIN merges that fresh row with the entity's existing record, so nothing observed outside the window is lost. The output is a single, denormalized document in the latest index: the entity's base identity and lifecycle, in addition to its attributes.

• Step 3: Identity is keyed deliberately. Which entity a row belongs to is the highest-stakes decision in the whole system. Key it wrong and you can blend several distinct users into one entity, or split one user across many. So the store doesn't guess. It derives a deterministic identifier, the entity unique ID (EUID), from the fields that actually identify the entity.

• Step 4. Maintainers compose the rest. The entity record is there, but it doesn’t yet know who it talks to (relationships) or what behaviors it exhibits. It also doesn’t yet know which various account entities are used by the same user or how risky the entity is. Those facets are layered onto the entity record by maintainers, and that's the heart of Entity Analytics.

The key to building entity relationships - the deterministic entity ID

The Entity Store currently supports three primary, schema-backed entity types: users, hosts, and services. Alongside these sits a fourth, polymorphic category known as generic entities.
The generic entity type isn’t constrained to a fixed schema, but rather provides an extensibility layer for the entity store platform. For example, by leveraging cloud and orchestrator fields, it represents resources like EC2 instances, S3 buckets, and Kubernetes clusters as first-class entities today. This flexibility allows the entity store to encompass a broad spectrum of assets without the architectural overhead of defining a new entity type for every resource class.

Everything the entity store does, including merging today's events onto the current entity record and linking two accounts into one resolved user, hinges on one field: entity.id, the EUID.

The EUID is a deterministic, human-readable string (a type prefix and then the fields that actually identify the entity, joined with @), rather than a random universally unique identifier (UUID) or a hash. Note the addition of a user entity namespace at the end of the user EUID. We’ll explain the role that plays below.

user:jane.doe@example.com@okta
host:9f86d081-1e0c-4b3f-8a2d-2c1e7bed425e
service:api-gateway

How is the EUID derived? For each entity, the store walks an ordered list of candidate identity fields and takes the first complete one. If a field a candidate needs is missing, that candidate is skipped and the next is tried, so a partial observation never produces a malformed ID.

Why do we need a namespace for user entities? While host and service entities rely on sufficiently unique keys like host.id or service.name, a user's identity, for example an email address, can be observed in many different log sources, but is authoritative only within its issuing domain. The namespace provides the necessary disambiguation, resolving the specific collision challenges unique to user identity records.

| Entity type | ID fields (priority order) | Example EUID | Namespace | | :----------------------- | :-------------------------------------------------------------- | :----------------------------------------------------- | :------------------------------------------------------------- | | Host | host.id, then host.name, then host.hostname | host:9f86d081-1e0c-4b3f-8a2d-2c1e7bed425e | (none) | | Service | service.name | service:api-gateway | (none) | | User (identity provider) | user.email, then user.id, then user.name@domain, then user.name | user:jane.doe@elastic.com@okta | Provider name: okta, entra_id, microsoft_365, active_directory | | User (local/endpoint) | user.name scoped to host.id | user:jdoe@9f86d081-1e0c-4b3f-8a2d-2c1e7bed425e@local | local |

• Host: The first present of host.id, host.name, host.hostname, giving host:<value>.
• Service: service.name, giving service:<name>.
• User: This is the interesting one, because a user's identity depends on where their activity was observed.

Users: Identity provider versus the local host. A user's EUID always ends in a namespace, the last @-delimited segment, and that segment is what stops two accounts that merely look alike from colliding.

When a user comes from an identity provider, the namespace is that provider (okta, entra_id, microsoft_365, active_directory) and the store keys on the most authoritative identifier it has: email, then user id, then name@domain, and then name.

user:jane.doe@elastic.com@okta

The same user's Entra ID account becomes …@entra_id, a different EUID, on purpose. They’re two authoritative accounts until resolution ties them together. We are going to see how resolution achieves this later in the blog.

When a user is seen only through endpoint or host telemetry, with no authoritative directory account behind them, the store scopes them to the machine and suffix the ID as local:

user:jdoe@9f86d081-1e0c-4b3f-8a2d-2c1e7bed425e@local

The middle segment is the host's durable identifier (host.id) and not its renamable hostname, so jdoe on a laptop stays distinct from jdoe on a shared bastion host, and the identity survives a machine being renamed or reimaged. The local namespace is an explicit signal: This is activity on this box, not a verified global identity. (Note the host in the example above and this local user share the same host.id. That's how the Entity Store knows they belong together.)

What makes an entity source authoritative and how to construct one

The EUID logic leans on a word that deserves a precise definition: authoritative. When an identity-provider-backed user earns a real namespace (okta or entra_id, among others) and high-confidence treatment, it's because the incoming events cleared a specific bar. Here's the bar and how to clear it with your own integrations.

What qualifies as authoritative identity data. The store treats an event as an authoritative identity signal when either of these is true:

| Classification path | Required ECS fields | Confidence | Resulting namespace | | :-------------------------------------- | :-------------------------------------------------------- | :--------- | :----------------------------------- | | Asset inventory | event.kind: asset | High | Provider name (okta, entra_id, etc.) | | Endpoint-observed (neither bar cleared) | user.name + host.id present, but no authoritative signal | Medium | local (scoped to host) | | Unclassifiable | None of the above | N/A | No entity created |

• The event is an asset / inventory document with event.kind: asset. This is how Elastic's identity integrations (Okta, Entra ID, Active Directory, and the cloud asset sources) publish their user and account inventories.

Clear either bar and the user becomes a first-class, identity-provider-backed entity at high confidence, and the preferred canonical record when identities are later resolved. Miss both, but carry a user.name and a host.id, and the user is instead scoped to that host in the local namespace at medium confidence. If both of these conditions are false and no host exists to associate the event with, no entity is created at all. That last case is deliberate restraint, not a gap; the store would rather create nothing than manufacture a noisy identity from an ambiguous event.

Two filters apply before any of this runs: The event's event.outcome must not be failure (a failed login isn’t evidence that an account exists), and it must carry at least one of user.email, user.id, or user.name.

Which namespace you get. Once an event qualifies, the namespace is derived from its source, the first non-empty of event.module or the leading segment of data_stream.dataset.

An authoritative event from an unrecognized source still creates a real entity. It just lands in the unknown namespace and you lose provider-level disambiguation (identical usernames from two namespaces can collide) and clean grouping. So, for a custom integration, the goal isn't only to qualify as authoritative, it's to be recognized.

How to accommodate a custom integration or pipeline. If you ingest identity data through a custom Fleet integration, a Logstash pipeline, or an Elasticsearch ingest pipeline, set these Elastic Common Schema (ECS) fields so the store classifies your entities the way you intend:

1. Mark the shape. For an account inventory, such as a local directory service, or configuration management database (CMDB), set event.kind: asset.
2. Provide an identity. Populate at least one of user.email (this is best, since it's the top priority), user.id, or user.name, plus user.domain where you have it.
3. Name your source. Set event.module (or the leading segment of data_stream.dataset) to a value the store maps. If you're feeding one of the known providers, reuse its naming so you inherit the right namespace. If your source is genuinely new, expect unknown until a mapping is added.
4. Don't let real identities look local. The local classification triggers when an identity event carries both user.name and host.id but doesn't clear the authoritative bar. If you're publishing directory data, don't attach a host.id to it. That field is the signal that says "endpoint-observed, scope it to this box." Reserve it for genuinely host-local activity.
5. Skip the noise. Don't emit failure outcomes as identity evidence. Note, too, that the store already excludes common shared and service account names (root, jenkins, deploy, postgres, admin, and similar) from the local namespace, so they never become per-host user entities.

Get these right, and your custom source behaves exactly like a built-in one: Authoritative users resolve and score at full fidelity, and endpoint-observed users stay correctly host-scoped. Plus, nothing downstream has to special-case where the data came from.

How maintainers add entity resolution, relationships, and risk scoring to entities

As discussed at the start of this post, the entity store is updated by dedicated background tasks called maintainers. Each of these has a specific job: building relationships between entities and resolving identities, along with updating risk scores.

The framework gives every maintainer its lifecycle, scheduling, and health reporting for free, making the entity store extensible by design. Any new entity enrichment capability ships as a new maintainer on the same rails, without re-architecting the entity store or breaking backward compatibility. Several maintainers already run in the current version, and recent additions, such as entity relationships derived from observed entity behaviors, have been added exactly this way. Even entity risk scoring, one of the most impactful risk-centric capabilities in the product, is implemented as just another maintainer composing one more facet of the entity record.

Maintainers discover and store entity relationships

Let’s take a look at one of the relationship maintainers. Once a day, the accesses_frequently / accesses_infrequently maintainer runs an ES|QL query over relevant telemetry and, for each actor→target pair, counts successful accesses over a 30-day window. If the count is:

• greater than or equal to an Elastic-defined threshold, the relationship becomes accesses_frequently.
• less than the threshold, the relationship becomes accesses_infrequently.

Today, it reads from Elastic Defend (endpoint logins), AWS CloudTrail (StartSession / SendSSHPublicKey), system auth (SSH logins), and system security (Windows events 4624/4648). A sibling, communicates_with, builds communication links from Elastic Defend, system auth, system security, Jamf Pro, and AWS CloudTrail.

The relationships are written straight onto the entity record as arrays of target IDs:

// a user entity in .entities.v2.latest.security_default-00001
"entity": {
  "id": "user-abc…",
  "relationships": {
    "accesses_frequently": { "ids": ["host-def…", "host-ghi…"] },
    "communicates_with":   { "ids": ["service-jkl…"] }
  }
}

A maintainer performs entity resolution

The resolution maintainer links fragmented accounts, the Okta jdoe, the Entra ID jdoe, and the on-prem Active Directory jdoe, into one resolved user. The risk maintainer then scores the resolved user, so the risky service token and the benign laptop login that both belong to John Doe are scored as John Doe: one record, not three that an analyst has to reconcile in their head.

Resolution happens two ways. Most of it is automatic: a background task runs every five minutes and links user entities that share the same user.email, so accounts converge on their own as the data arrives. When you need to step in, you can link or unlink manually from the entity flyout in the UI, or call the API directly.

POST /api/security/entity_store/resolution/link
{
  "target_id": "user:jane.doe@elastic.com@okta",
  "entity_ids": ["user:jane.doe@elastic.com@entra_id"]
}

POST /api/security/entity_store/resolution/unlink
{
  "entity_ids": ["user:jane.doe@elastic.com@entra_id"]
}

The near-term plan is to expand the out-of-the-box matching logic so more identities link automatically without anyone manually creating the links.

A maintainer calculates and updates entity risk scores

The risk score maintainer runs hourly by default. On each run, it queries the detection alerts over a configurable rolling time window (past 30 days by default) and for each entity those alerts touch, aggregates the alert risk scores into a single normalized entity risk score and risk level.

Entity risk scoring also extends across resolved identities. First the risk scoring maintainer calculates an entity risk score for each user entity (one per account) based on any detection alerts related to that account. Next it calculates a distinct entity risk score for the resolved user across all of their linked accounts. For example, a malicious service token alert on one account, and an excessive laptop login rate alert from a different account, where both accounts belong to John Doe, roll up into a single user entity risk score for the resolved John Doe. That holistic view is possible because the resolution maintainer ran first, then the risk scoring maintainer reads the graph (set of linked accounts) that the resolution maintainer produced, and is able to assign a risk score to the user, not only the constituent accounts If an analyst or AI agent decides to investigate the provenance of a resolved user risk score, the answer is easily obtained.

Every entity store decision is inspectable and correctable

The store's engine and its relationship maintainers are built on ES|QL, the same query language that Elastic users already employ, so what a maintainer reads,computes, and writes is inspectable, rather than proprietary magic. Every entity lives as a document in an Elasticsearch index, so its full state is there to read, including the relationships that were drawn and the accounts that were resolved into one user, plus the signals behind a risk score.

Entity data is wrong sometimes; that's the reality of identity data. Because the state is represented as Elasticsearch documents, you can correct them through APIs. When resolution gets it wrong, two people merged or one user split across two records, you can link or unlink identities right through the UI, and the risk score maintainer rescores the corrected resolved user on its next run. When you need to pull an entity into scope or set its criticality, watchlists let you say so directly. With Elastic Agent Builder, an AI agent can walk you through an entity's data and make those corrections for you, without leaving the Elastic UI.

What entity maintainers mean for security analysts

With entity resolution, relationship mapping, and risk scoring running as automated maintainers, human analysts and AI agents gain efficient access to the entity context they need to perform investigations. The entity store is extensible, so entities continue gaining relationships and enrichments as new maintainers ship or new data sources are ingested, with no migration or re-architecture. Nothing is hidden, so every risk score traces back to the alerts and other factors (for example, asset criticality, watchlist membership) that produced it, and a finding becomes something you can explain using the Agent Builder chat with the entity analytics tools and skills rather than a black box you have to trust.

In addition, wrong data doesn't have to stay wrong; a bad resolution link takes one API call or Agent Builder action to fix. What you're left with is a single queryable record per entity, host, and service that can evolve over time.

What’s coming next in the Entity Analytics Roadmap

The science of entity analytics is ongoing. One area we’re digging into is integrating non-human identities (NHI) as entities. AI agents, service accounts, and agentic workloads are among the fastest-growing concerns across today’s attack surface. While this research continues, the core objectives involve finding an optimal way to link an AI agent's sessions to existing entities by tracking its utilized Service accounts or connected devices and services, as well as assessing the risk of AI agent behavior

Also under the hood, we're reworking log extraction to split the work by data source confidence, so authoritative identities and lower-confidence enrichment are handled separately.
Additionally, we’re exploring enhancing the agentic UEBA capabilities to reason about an entity, connect the dots, and suggest relevant leads for the analyst to explore.

Entity risk scoring is getting continued attention too. Currently, a detection alert carries one static alert risk score that affects the entity risk score of every entity it involves, so the actor and the target can come out looking equally risky. The direction is towards a dynamic, entity-centric risk score that gives each entity its own contribution based on the role it played in the observed interaction, and its own context.

For practitioners with their hands on the Elastic Security UI, the entity analytics overview experience is due for a refresh, with risky entities shown more simply and clearer actions to investigate them.

Entity analytics is available in Elastic Security. Learn more about entity analytics.

The full path from config file to dashboard looks like this:

Why developer workstations are the npm supply-chain gap

CI/CD pipelines have their own supply-chain guardrails that can be managed at the enterprise level. Workstations are where the gap usually sits. A developer running npm install on a freshly compromised package is a different surface from the build pipeline, and the package manager itself is the right place to apply a delay before a fresh release becomes installable. npm's min-release-age setting, available since npm 11.10, does exactly that: the value is a number of days, and npm excludes any version published more recently from resolution. In MITRE ATT\&CK terms, the technique it blunts is Compromise Software Dependencies and Development Tools (T1195.001).

Most other package managers have an equivalent flag; cooldowns.dev tracks which ones, what each flag is called, and recommended values. This post focuses on npm, and recent attacks have happened, but the technique applies to any of them.

We enforce the setting with Jamf: a script writes min-release-age=7 to each user's .npmrc and the machine-global npmrc, and re-applies once a day. A removal therefore self-heals within 24 hours, and the telemetry is what makes the removal visible at all: the 6-hour heartbeat catches the gap window before Jamf closes it, and shows us which hosts keep removing the setting.

What does an npm cooldown config file look like?

The target is small: npm writes the setting to a user-level .npmrc, typically 10 to 50 bytes.

~/.npmrc    →    min-release-age=7        # days

Machine-global settings live in a handful of well-known paths (/opt/homebrew/etc/npmrc and /usr/local/etc/npmrc on macOS, /etc/npmrc and /usr/lib/node_modules/npm/.npmrc on Linux), so a complete inventory has to include those paths too. While we haven’t deployed a Windows version yet, we did include all the details to make this work in the Linux and Windows variants section. These can be found in the section Linux and Windows variants.

Two ingredients made this look easy. First, as InfoSec is Customer Zero at Elastic, the Elastic Agent is already rolled out to every endpoint and runs with the privileges to read these files. Second, the schema is essentially flat: one key, one value, one config file. So in theory, you configure a file input, tail the cooldown lines, and ship them to Elasticsearch.

The complication is that auth tokens for private registries (//registry.npmjs.org/:_authToken=...) live in the same .npmrc file. None of that can leave the host. Any approach has to filter the file body before it transits anywhere.

First approach: Monitoring .npmrc with the Custom Logs Filestream integration

The Custom Logs Filestream integration in Fleet wraps Filebeat's modern filestream input and lets you specify path globs and an ingest pipeline. Note: this is a distinct Fleet integration from the older "Custom Logs" integration, which wraps the legacy log input. The two have different configuration surfaces and file-identity models, so if you're migrating between them the reference linked above is worth reading before either direction.

We configured the integration across all users on macOS, plus the system-global npm paths:

/Users/*/.npmrc
/opt/homebrew/etc/npmrc
/usr/local/etc/npmrc
/etc/npmrc

The ingest pipeline did four things:

1. Grok the message field to pull out the min-release-age=value pair.
2. Drop any line whose key wasn't min-release-age, so registry auth tokens never made it into Elasticsearch.
3. Convert the value to a long.
4. Remove the raw message and event.original fields before indexing.

It worked, with three caveats that changed our minds about whether it was the right integration for this job.

Three filestream behaviours that don't fit config file monitoring

Native file identity beats fingerprint for small files

Filebeat's default file identity strategy is "fingerprint", which hashes the first N bytes of a file to give it a stable ID across rotations. The fingerprint length defaults to 1024 bytes, and a file shorter than that minimum is silently held back from ingestion until it grows. A 22-byte .npmrc will never grow, so the data never moves. This is in line with the description of the Filestream integration and also apparent in the agent logs:

"ingestion from some files will be delayed, files need to be at least
 1024 in size for ingestion to start"

The fix is to switch the integration's advanced options to use native file identity, which uses inode plus device and has no size floor. In our deployment on Elastic Agent 9.x, disabling fingerprint alone did not fall back to native; we had to flip both toggles explicitly.

clean_inactive and ignore_older are coupled

We wanted periodic re-emission of each config file so a stale event could be distinguished from a current one. filestream's clean_inactive is the right knob for that, but it requires ignore_older to also be set. Together they cleanly trim files that haven't moved recently, which is useful for rotating log files and not useful for static config files that may sit unchanged for months. We disabled clean_inactive.

Filestream emits on append, not on removal

This third learning is structural rather than a config gotcha. filestream is designed for log tailing: when bytes arrive at the end of a file, an event is emitted. When content disappears (because npm rewrites .npmrc without the cooldown line, or because the file is deleted), filestream sees a modification, but every resulting line is dropped by our allowlist filter, and nothing reaches Elasticsearch. The last "set" event for that host stays in the index indefinitely.

For application logs that's correct behavior, since log lines aren't usually retracted. For config file state monitoring, removal is the signal the adoption campaign needs most, and it never arrives. That's why we switched to CEL.

Why nvm inflates npm version counts for cooldown adoption

Machines routinely report more than one npm version, because every Node.js version installed through nvm (Node Version Manager) ships its own npm binary. The distortion stood out as soon as we checked test pulls from our osquery manager integration package-version inventory: one machine reported 34 distinct npm versions.

The config side of this is forgiving. npm config set min-release-age 7 writes to the user-level ~/.npmrc, and every npm installation on the machine reads that file, including the per-Node copies nvm installs. Enforcement is the strict side: versions older than 11.10 ignore the key. A machine carrying npm 11.12 in one nvm environment and npm 10.9 in another is only protected when the newer npm is the one running npm install. A count of machines with any capable npm overstates who is protected, which is one more reason to track the config file directly rather than infer adoption from version counts.

Second approach: Monitoring .npmrc with CEL snapshot semantics

The CEL input in Elastic Agent was designed for HTTP API polling, but it also exposes a file() function and a dir() function for filesystem access. That enables a different model from filestream. filestream tails lines as they arrive; CEL takes a snapshot of the whole file each interval. For state monitoring, the snapshot is what we want: the unit of interest is the current contents of .npmrc as a whole.

Here is the complete script we run, trimmed to npm (our production version watches the other package managers' config files with the same pattern, and carries an extra guard for non-UTF-8 file content):

(
  (
    try(dir("/Users")).as(entries, type(entries) != type("") ?
      entries.filter(u,
        u.is_dir
        && !string(u.name).startsWith(".")
        && string(u.name) != "Shared"
        && string(u.name) != "Guest"
      ).map(u, "/Users/" + string(u.name) + "/.npmrc")
    : [])
  ) + (
    try(dir("/home")).as(entries, type(entries) != type("") ?
      entries.filter(u,
        u.is_dir
        && !string(u.name).startsWith(".")
      ).map(u, "/home/" + string(u.name) + "/.npmrc")
    : [])
  ) + (has(state.files) ? state.files : [])
).map(f,
  try(file(f)).as(content,
    type(content) == type("") ?
      {"file": f, "exists": false}
    :
      {"file": f, "body": string(content),
       "hash": content.sha256().hex(), "exists": true}
  )
).as(file_data, {
  "events": file_data.filter(fd, fd.exists).map(fd, {
    "message": fd.body,
    "file": {"path": fd.file, "hash": {"sha256": fd.hash}},
  }),
  "cursor": {"hashes": file_data.filter(fd, fd.exists)
    .map(fd, {"file": fd.file, "hash": fd.hash})},
  "url": state.url,
  "files": has(state.files) ? state.files : [],
})

The first block enumerates user home directories at runtime, with filtering that path globs can't express, and appends the machine-global paths from state.files. try(dir(...)) and try(file(...)) return an error string when the path doesn't exist, so each block checks the type of the result and collapses a missing directory or file to an empty result. A macOS host has no /home, a Linux host has no /Users, and the same script runs on both.

The initial state supplies the global paths and the poll interval is 6 hours:

files:
  - /opt/homebrew/etc/npmrc
  - /usr/local/etc/npmrc
  - /etc/npmrc
  - /usr/lib/node_modules/npm/.npmrc

An aside on Fleet: the integration to add in the UI is called "Custom API using Common Expression Language", not "CEL". It exposes the full CEL runtime including file system access. Set resource.url: file:///dev/null to satisfy the required URL field without making an HTTP request, and put the initial state in the "Custom request cursor" YAML at the bottom of the form.

How the CEL integration evolved: emit-on-change, tombstones and heartbeat

The CEL snapshot integration shown above is the third version. The path there is the useful part.

Version one emitted only on change. Each heartbeat hashed every file and emitted an event only when the hash differs from the cursor, the state the input persists between runs. Efficient, and it produced the removal signal cleanly: npm rewrites .npmrc in place when a setting changes, the hash moves, the new snapshot carries no min-release-age line, and the ingest pipeline marks the event cooldown.absent = true.

Version two added a deletion tombstone. Hash comparison can't see a file that stopped existing (npm config delete min-release-age removes .npmrc entirely when it's the only setting), because version one filtered out non-existent files before emitting. A one-block extension emitted a stub event, once, for any file that was in the cursor but no longer on disk.

Version three replaced both with a heartbeat, because the dashboard demanded it. The adoption dashboard counts hosts whose cooldown state falls inside the selected time window. Under emit-on-change, a host that sets a cooldown emits exactly one event and then goes silent. Once that single event ages past the dashboard's window (say, now-7d), the host disappears from the "adopted" count even though the cooldown is still in place. We watched adoption climb during rollout and then start to erode a few days later, purely as an artifact of one-shot events aging out of the window. The telemetry was correct; the time-windowed view of it was not.

So the final integration re-emits each existing file's current state on every 6-hour heartbeat. Every host re-reports its cooldown posture four times a day, the windowed dashboard reflects live state, and a host that stops appearing is genuinely offline rather than merely quiet. Removal of the cooldown line still surfaces as cooldown.absent = true in the next snapshot, and a deleted file drops out of subsequent snapshots, so the tombstone becomes unnecessary.

The cost is volume: a snapshot per file per host per interval instead of one event per change. At a 6-hour cadence that is a few thousand small documents a day across the fleet, which is immaterial. At a 60-second interval it stops being immaterial: always-emitting at 60s ships the same unchanged state every minute, which we measured at roughly a thousand-fold more documents for zero added signal. If you adopt the heartbeat, set the interval in hours.

The detection latency for a removal is the heartbeat interval, 6 hours in our case. For an adoption campaign measured over days and weeks against a 7-day cooldown, that is not a meaningful constraint. If you need lower-latency, discrete removal events for alerting, run the emit-on-change variant instead; the snapshot model supports both.

How to filter .npmrc auth tokens before they leave the host

A snapshot-based integration sends the whole file body across the wire to the ingest pipeline. That's fine for config keys; it is not fine for .npmrc registry auth tokens. Filtering at the ingest pipeline is too late, because by then the tokens have already transited the network.

The fix is an agent-side script processor in the integration's Advanced options → Processors field. It keeps only min-release-age lines and drops everything else before the event leaves the workstation:

- script:
    lang: javascript
    source: >
      function process(event) {
        var msg = event.Get("message");
        if (msg == null) return;
        var filtered = [];
        var lines = msg.split('\n');
        for (var i = 0; i < lines.length; i++) {
          var line = lines[i].trim();
          if (line.indexOf('min-release-age') === 0) {
            filtered.push(line);
          }
        }
        event.Put("message", filtered.join('\n'));
      }

Tokens never reach the wire. We verified by adding a test token to a .npmrc on a managed host, running a cycle, and confirming zero search hits for the token string in Elasticsearch.

The npm cooldown ingest pipeline

The pipeline below is the npm-only variant of the one we run in production, and it is short enough to show whole. Grok extracts the key and value, a Set processor turns "no key found" into the explicit removal signal, and the raw message is removed before indexing:

PUT _ingest/pipeline/npm-cooldown-workstation
{
  "description": "Parses min-release-age from .npmrc snapshots",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "(?<cooldown.key>min-release-age)\\s*=\\s*(?<cooldown.value>[^\\n\\r]*)"
        ],
        "ignore_missing": true,
        "ignore_failure": true
      }
    },
    {
      "set": {
        "if": "ctx.cooldown?.key == null",
        "field": "cooldown.absent",
        "value": true
      }
    },
    {
      "set": {
        "if": "ctx.cooldown?.key != null",
        "field": "cooldown.unit",
        "value": "days"
      }
    },
    {
      "convert": {
        "field": "cooldown.value",
        "type": "long",
        "ignore_missing": true,
        "ignore_failure": true
      }
    },
    {
      "remove": {
        "field": ["message", "event.original"],
        "ignore_missing": true
      }
    }
  ]
}

Two details earned their place the hard way. The value capture is [^\n\r]*, which stops at both LF and CRLF line endings; an earlier revision used [^ \r]*, which truncated any value containing a space and could run across newlines. And cooldown.unit is set explicitly even though npm's unit is always days, because dashboard rows that read 7 days stay unambiguous when the telemetry later grows beyond npm.

The cooldown.absent = true event is the payoff. The host was in the index six hours ago with cooldown.key = min-release-age; the next snapshot has no key; the dashboard shows the transition. That is the removal signal filestream could not produce.

CEL vs. filestream for config file monitoring

We ran both in parallel for a few weeks and then made a call.

The core reason is the shape of the data. filestream is built for append-only logs, where new lines arrive at the end of a file and the job is to harvest them. .npmrc is a state file: npm rewrites it in place when a setting changes and deletes it when the last setting is removed. The question the telemetry needs to answer, "what is the cooldown state of this host right now, and when did it change," is a question about the current contents of a file. Tail offsets can't answer it.

| | CEL | Filestream | | :---- | :---- | :---- | | Designed for | State files (snapshot + hash) | Append-only logs (tail) | | Cooldown-line removal detection | Yes (next snapshot marks cooldown.absent) | No | | File deletion detection | Yes (host drops out of snapshots) | No | | Snapshot semantics | Whole file | Per line | | Auth-token posture | Filtered agent-side before transit | Same (agent-side allowlist) | | Fleet integration | Custom API using Common Expression Language | Custom Logs Filestream | | Config complexity | ~40-line CEL integration | Declarative path globs | | State refresh / latency | 6h heartbeat | ~10s (tail) |

Running both meant every config change produced two events into two separate datasets, and we maintained two ingest pipelines with dashboards split across indices. The complexity cost outweighs any redundancy benefit, and the only thing filestream gives that CEL doesn't is faster detection latency, which is not a meaningful constraint for adoption tracking on a 7-day cooldown.

We also considered osquery and auditd before settling on CEL. osquery can read config file contents on a schedule through its file and file_lines tables, and auditd can fire on file writes, but neither produces a clean current-state signal across ingestion, and both would mean standing up a second collection path next to the agent already deployed for endpoint telemetry.

CEL won. End-to-end validation on a single macOS workstation:

| Scenario | Expected | Result | | :---- | :---- | :---- | | npm config set min-release-age 5 | cooldown.key = min-release-age, cooldown.value = 5 | Pass | | .npmrc exists with no cooldown key | cooldown.absent = true | Pass | | npm config delete min-release-age (deletes the file) | path absent from subsequent snapshots | Pass | | Auth token line in .npmrc | No token in Elasticsearch | Pass |

npm cooldown monitoring on Linux and Windows

macOS was the primary platform; the design generalizes cleanly.

Linux needed no integration change at all. The /home block in the integration above already covers it: on each platform, the directory that doesn't exist fails the type check and collapses to an empty list, and the other side fills in. The Linux variant runs the same integration through the same ingest pipeline into the same dataset, with /etc/npmrc and /usr/lib/node_modules/npm/.npmrc in the global watch list.

Windows is a separate integration on the same dataset, and it's the one variant we designed but haven't deployed: our own fleet has too few Windows machines to justify the rollout yet. We're documenting it anyway because Windows is the most common OS in corporate fleets, so for many readers this variant is the one that matters most. The integration enumerates C:/Users/* (forward slashes work on Windows under Go/CEL), excludes Public, Default, and Default User, and watches each user's .npmrc plus the system globals under C:/ProgramData. The Grok pattern's value capture, [^\n\r]*, stops at both LF and CRLF line endings, so it works unchanged on Windows without capturing the trailing \r in the value. The agent-side script processor is also unchanged.

Rollout status and extending beyond npm

The pipeline is now reporting from several hundred macOS, each re-emitting its current npm cooldown state on the 6-hour heartbeat. We're sharing the technique at this stage so other security engineering teams can build on it; once the rollout reaches the full fleet we'll follow up with an org-wide adoption-rate post. The Windows variant stays on the shelf until our Windows population justifies the rollout; if your fleet is Windows-heavy, the design in this post is ready to adapt.

In production the same integration already watches the config files for pip, uv, pnpm, yarn Berry, and bun. That expansion deserves its own post, because cooldown values are not unit-comparable across package managers, and one manager's documentation and implementation disagree about the unit by three orders of magnitude. If you're extending this design beyond npm, check cooldowns.dev for each flag's unit before you compare values across tools.

What we learned about npm cooldown monitoring with Elastic Agent

• npm cooldown adoption telemetry is state monitoring. The signal that matters most is a host removing min-release-age, and append-driven log tailing cannot see that happen.
• A snapshot-based CEL integration produces the removal signal directly: re-emit each .npmrc's current state on a 6-hour heartbeat, and a removal shows up as the key's absence in the next snapshot.
• A time-windowed adoption dashboard needs the heartbeat, not one-shot change events, which age out of the window and make adoption look like it's eroding when it isn't.
• Small config files (under 1024 bytes) need native file identity, not fingerprint, regardless of which integration you use.
• An agent-side script processor keeps .npmrc registry auth tokens on the workstation: tokens never reach the wire.
• nvm inflates capability counts: each Node.js version ships its own npm, all of them read the shared ~/.npmrc, but only npm 11.10+ enforces the key. Track the config file, count enforcement-capable versions separately.

This community-powered innovation is evident in everything we do. Elastic is a security tool built by security users, for security users. We’ve sat in the seat. We’ve worked the queue at 2 a.m. We’ve chased an alert that turned out to be nothing and missed the one that turned out to be everything. What we're continually building and improving is the security operations center (SOC) we wished we'd had back then. This means agents that carry the machine-speed work, leave critical judgment to analysts, and a platform that connects the two.

With Elastic, machine speed and human judgment work together in a single loop. Stopping more at the endpoint reduces the number of alerts. Those that remain surface the real threats, and you can validate them before they reach a queue. The work underneath is increasingly automated. Each piece makes the next one lighter, and none of it asks you to hand judgment over to a black box.

Alert Zero: From alert queue to validated threats

Every SOC is chasing a queue worked down to what actually matters, the SOC's version of “inbox zero.” When we built our suite of tools, our goal was Alert Zero, a state that always felt out of reach. It’s a goal that teams move toward, with agents and analysts working together. It doesn’t mean zero alerts or replacing the analysts.

How Attack Discovery investigates alerts like an analyst

Attack Discovery has always pulled related alerts together into a single view of an attack. Now it goes further, working through them the way a human analyst would:

1. Threat-hunts raw events beyond the initial alerts.
2. Checks entity risk for the users and hosts involved.
3. Corroborates findings across other data sources.
4. Classifies the event as a validated attack.

Your team gets a short list of validated attacks to work, instead of a wall of raw alerts to triage.

Closing detection gaps with auto-drafted rules

When Attack Discovery finds something that your rules missed, it drafts a detection rule to close the gap and hands it to an analyst to approve, helping to make the entire workflow more efficient and to reduce the source of false positives.

Security teams need the how, not just the what, and Attack Discovery shows its work, so you can see how it got to each answer and recommended action. Every step of the reasoning is visible, so an analyst knows why an alert became an attack. You can run it however fits your team, whether you kick it off yourself or set a recurring cadence. You can even trigger it from Elastic Workflows. A separate alert analysis workflow addresses the volume from the other side, differentiating between likely false and true positives, so analysts lose fewer hours to low-fidelity alerts, and leaving Attack Discovery a cleaner set to investigate.

Elastic Defend endpoint protection: vulnerable driver coverage and Windows on ARM

Fewer alerts reach the queue when more threats are stopped on the device, so prevention starts at the endpoint.

Vulnerable driver coverage that keeps pace with disclosure

Elastic Defend now gets ahead of vulnerable drivers. Attackers exploit these by bringing a signed, trusted driver with a known flaw and using it to reach the kernel, and coverage for a new one has traditionally arrived on a release cycle. Our threat research team monitors public disclosure sources, like VirusTotal, loldrivers.io, and Microsoft's blocklist. Through an always-on process, Elastic automatically generates and instantly deploys YARA rules as new drivers are disclosed, so protection keeps pace instead of waiting on a release. That speed matters when AI-driven attacks can move from one machine to the next in under a minute, faster than any response workflow can react.

Full endpoint protection for Windows on ARM

Windows on ARM is now fully covered in Defend, which brings Surface and other ARM-based laptops into the same protection as the rest of your fleet. Teams can roll this feature out across every endpoint without paying per device to do so. A new endpoint troubleshooting skill also rounds out this capability, automatically flagging policy and performance issues so your team spends less time chasing them.

Elastic Workflows: SOC automation you can describe in plain language

Build automations in plain language with version control

Elastic Workflows makes automations faster to build and shows you exactly what a workflow will do before it runs. The latest updates start with plain-language authoring, so you can describe the automation you want and have it generated for you. Versioning then tracks every change, so you can compare any two versions and roll back to a working one in a click. You always know who changed what and when. Visual Mode shows a workflow as a graph, with its triggers, steps, branches, and logic visible at a glance next to the YAML. Drag-and-drop editing is coming next.

Human-in-the-loop approvals routed to Slack

Automation you can trust starts with understanding what Workflows will do and when it will ask for help. When a workflow reaches a decision that needs a person or an approval or other input, it pauses and routes the request to a tool your team already uses, like Slack. Automation handles the routine, while your team stays on top of the decisions that need judgment. Workflows runs natively within the Elasticsearch platform extending across search, observability, and security, so it runs where your security data already lives, rather than stitched as a layer on top.

Together, our drive to Alert Zero, enhanced endpoint protection, and automation where your data lives reinforce each other. Stronger prevention keeps alerts from being raised in the first place, and the ones that remain arrive validated instead of raw. Automation underneath keeps prevention and investigation moving at machine speed, while your analysts stay on the decisions that need a human. That’s what the agentic SOC looks like when it’s built to help the people in it rather than replace them.

"Security teams are not losing because they lack tools; they're losing because the tools generate more work than the team can absorb," said Mike Nichols, general manager Security, Elastic. "Elastic Security is built by people who've sat in the SOC and worked the queue. These updates go after one of the biggest sources of analyst burnout, which are alerts that shouldn't be alerts in the first place. We know every barrier is a liability and we’re building an open, transparent platform that breaks down those barriers and makes it easier for teams to customize and manage the security stack they need to protect their organizations."

Find Elastic Security at Black Hat and DEF CON 2026

At the booth, you can see all of this working on real attacks:

• Agents helping cut false positives on the way to Alert Zero, showing their reasoning at every step.
• Security information and event management (SIEM) built for the agentic SOC.
• Extended detection and response (XDR) across cloud, Kubernetes, and endpoint.
• Native automation with Elastic Workflows.
• Endpoint defense that holds across Linux, macOS, and Windows.
• Threat research from Elastic Security Labs that ships in the platform.

Bring your hardest questions.

Elastic is showing up across the week, at Black Hat and around the community.

• Sober Speakeasy with Sober in Cyber, an alcohol-free networking evening at the Mob Museum's Underground Speakeasy. Tuesday, August 4, 2026, 7:00–9:30 p.m. PT. All infosec professionals are welcome, sober or sober-curious.
• Blitz & Defend, an executive evening with Elastic and AWS at Allegiant Stadium, a behind-the-scenes tour, and reception in the Raiders locker room. Wednesday, August 5, 2026, 6:30-8:30 p.m. PT.
• GDIT Sip & Cipher, a cybersecurity networking reception at Toca Madera, with Elastic among the sponsors. Wednesday, August 5, 2026, 6:00–9:00 p.m. PT.
• InnovatHERs: Women Shaping Tomorrow, a breakfast with Women in CyberSecurity (WiCyS) to celebrate and connect women in the field. Thursday, August 6, 2026, 7:00–9:00 a.m. PT.
• Find us at DEF CON, too: Security people go to Black Hat because they have to. They go to DEF CON because they want to. That's why we're proud to be a Blue Tier sponsor of Blue Team Village at DEF CON this year, the highest-traffic village and home to the SOC, Digital Forensics and Incident Response (DFIR), and incident response communities. It's where defenders come to sharpen their craft, run capture-the-flag, and swap real-world detection and response tactics. For us, showing up there is about being where our people already are, because Elastic Security is built by the same community that fills the room. Come find us.

Where to begin. For the fourth consecutive year, Elastic has had the privilege of serving as a trusted industry partner on Exercise Defence Cyber Marvel - the UK Ministry of Defence's flagship cyber exercise series. DCM26 was, without question, the most ambitious iteration yet, and we're chuffed to bits to finally be able to talk about what we built, how we built it, and what we learnt along the way.

What is Defence Cyber Marvel?

For those unfamiliar, Defence Cyber Marvel (DCM) is the largest UK military cyber exercise series that focuses on defending traditional IT networks, corporate environments, and complex industrial control systems in realistic, high-pressure scenarios. It showcases responsible cyber power whilst enhancing readiness, interoperability, and resilience across Defence and allied nations. Now in its fifth year, DCM has evolved from an Army Cyber Association initiative into a tri-service operation led by Cyber and Specialist Operations Command (CSOC).

The UK Government published an official press release for DCM26, which provides an excellent overview of the exercise's strategic importance. As the British High Commissioner to Singapore noted, the exercise demonstrates the deep cooperation between the UK and trusted partners, a reminder of the strength of shared strategic partnerships in an increasingly complex security landscape.

At its core, DCM is a force-on-force cyber exercise: defending Blue Teams protect their assigned networks and infrastructure from attacking Red Teams, using a range of techniques. Activities span changing default passwords and hardening firewalls through to deploying enterprise-grade, AI-powered cyber defence with Elastic Security. The activities of each team are monitored by the White Team to establish a score factoring in system availability, attack detection, incident reporting, and system restoration.. It stretches the most experienced teams whilst also facilitating a unique training mechanism for junior teams on their first exposure to a cyber range, and that dual purpose is what makes DCM such a valuable exercise.

The scale of DCM26

DCM26 brought together over 2,500 personnel from 29 participating countries and 70 organisations, coordinated from a central Exercise Control (EXCON) based out of Singapore, with EXCON hosting over 600 participants. The exercise ran across a hybrid compute environment spanning the CR14 cyber range and AWS, hosting over 5,000 virtual systems.

The exercise itself ran for five days of execution (9–13 February 2026), preceded by optional instructor-led pre-training and connectivity checks. The scenario, built on the Defence Academy Training Environment (DATE) Indo-Pacific Operating Environment, placed teams as Cyber Protection Teams defending deployed military systems during an escalating regional crisis.Blue Teams were geographically dispersed,some in their home locations across the UK and internationally, others deployed overseas, all connecting into the range via VPN.

Participants included representatives from UK Defence, cross-government departments such as the National Crime Agency, the Department for Work and Pensions, the Cabinet Office, and the Department for Business and Trade, alongside international partners forming up to 40 teams. Following the success of last year's exercise in the Republic of Korea, Singapore served as the exercise hub for the first time, reflecting the UK's commitment to deepening cooperation with Indo-Pacific partners on shared security challenges.

In short, it's a serious exercise. High-pressure, force-on-force, with real consequences for scoring and real learning outcomes for every participant.

The deployments: Our Elastic infrastructure

This year's infrastructure represented a significant architectural evolution from previous iterations. Rather than deploying individual Elastic Cloud clusters per team, we moved to a single, space-based multi-tenanted Elastic Cloud deployment for the Blue Teams. We also provided deployments for functions outside of the Blue Teams. Let me break down each deployment and why it exists.

Blue Teams: Multi-tenanted Elastic Security

The centrepiece of our contribution was a single Elastic Cloud deployment serving all 40 defending Blue Teams, separated using Kibana Spaces and datastream namespaces. Each of the 39 teams had its own isolated workspace, including dashboards, agents, and detection rules.

Here's what the Terraform resource looked like for creating each team's space:

# Create 40 Blue Team spaces
resource "elasticstack_kibana_space" "blue_team" {
  count = var.team_count

  space_id    = local.space_ids[count.index]
  name        = "Blue Team ${local.team_numbers[count.index]}"
  description = "Isolated space for BT-${local.team_numbers[count.index]} with space-aware Fleet visibility"

  disabled_features = []
  color             = "#0077CC"
}

Each team's space got a dedicated set of three Fleet agent policies: on day 1a Deployed network policy, day 2, a Host Nation network policy, and finally a PacketCapture policy for network traffic monitoring. The phased access control was elegant in its simplicity: setting enable_hostnation_network = true in our terraform.tfvars and running terraform apply expanded each team's role permissions and made their Host Nation agent policy visible in their space. The exercise went from one network to two without a single manual click in Kibana.

The data isolation relied on datastream namespaces. Each agent policy is written to team-specific namespaces like bt_01_deployed and bt_01_hostnation, producing data streams following the pattern:

logs-system.auth-bt_01_hostnation
logs-system.syslog-bt_01_hostnation
metrics-system.cpu-bt_01_hostnation
logs-endpoint.events.process-bt_01_hostnation
logs-windows.forwarded-bt_01_hostnation
logs-auditd.log-bt_01_hostnation

Each team's Kibana security role was then scoped to only those data streams using dynamic index privilege blocks:

# Deployed data streams (always granted)
indices {
  names = [
    "logs-*-${local.deployed_namespaces[count.index]}",
    "metrics-*-${local.deployed_namespaces[count.index]}",
    ".fleet-*"
  ]
  privileges = ["read", "view_index_metadata"]
}

# HostNation data streams (conditional on enable_hostnation_network)
dynamic "indices" {
  for_each = var.enable_hostnation_network ? [1] : []
  content {
    names = [
      "logs-*-${local.hostnation_namespaces[count.index]}",
      "metrics-*-${local.hostnation_namespaces[count.index]}"
    ]
    privileges = ["read", "view_index_metadata"]
  }
}

Authentication was handled via Keycloak SSO, with Elasticsearch role mappings connecting Keycloak groups to Kibana roles:

resource "elasticstack_elasticsearch_security_role_mapping" "blue_team" {
  count = var.team_count

  name    = "bt-${local.team_numbers[count.index]}-keycloak-mapping"
  enabled = true

  roles = [
    elasticstack_kibana_security_role.blue_team[count.index].name
  ]

  rules = jsonencode({
    field = {
      groups = "${local.keycloak_groups[count.index]}"
    }
  })
}

The default integration policies were simple by design. Each team received: System for core OS telemetry, Elastic Defend for Endpoint Detection and Response, Windows event forwarding, Auditd for Linux audit logging, and Network Packet Capture integrations. That's over 400 integration policies managed as code via the Elastic Stack Terraform Provider.

A note on Elastic Defend: due to the effectiveness of Elastic's endpoint protection - which is trusted in production by the US DOD and IC, read more about that here - and the fact that nobody in their right mind is burning zero-day exploits on a training exercise, we're forced tohandicap Elastic Defend by disabling Prevent mode, leaving it in Detect-only mode. Teams get alerts when something malicious happens, but with no automatic mitigation. We also completely disable Memory Threat Prevention and Detection as this discovers the majority of attacking team implants and beacons, which would rather spoil the game for the Red Teams. Toward the end of the exercise, we allowed the teams the freedom to use Elastic Defend to its full capability, but not before letting the Red Teams get a strong foothold.

We also pre-installed Elastic's prebuilt detection rules into each team space - the full set from Elastic Security Labs, continuously updated in an open repository. These rules were setup to ensure they only queried indices that the team's namespace-scoped permissions allowed, preventing any cross-team data leakage in detection rule execution.

Additionally, each team space had its Security Solution default index configured to scope detection rules to only that team's data streams, rather than the default broad pattern. This was handled by a Terraform null_resource that called the Kibana internal settings API to set securitySolution:defaultIndex for each space.

At peak, this deployment was ingesting 800,000 events per second (EPS) across all 40 teams. That's a serious amount of data, and the cluster handled it comfortably thanks to the autoscaling capabilities of Elastic Cloud. That given, back in 2018 we were doing 5 million events per second with eBay.

Data lifecycle was managed by an Index Lifecycle Management (ILM) policy that rolled indices over after one day or 50 GB (whichever came first), moved them to a warm phase after two days for read-only optimisation and force-merging, and then deleted data after ten days. As a result, the storage costs were minimized while maintaining the exercise window requirements. Below is an example of how the ILM policy was implemented.

resource "elasticstack_elasticsearch_index_lifecycle" "dcm5_10day_retention" {
  name = "dcm5-10day-retention"

  hot {
    min_age = "0ms"

    set_priority {
      priority = 100
    }

    rollover {
      max_age                = "1d"
      max_primary_shard_size = "50gb"
    }
  }

  warm {
    min_age = "2d"

    set_priority {
      priority = 50
    }

    readonly {}

    forcemerge {
      max_num_segments = 1
    }
  }

  delete {
    min_age = "${var.data_retention_days}d"

    delete {
      delete_searchable_snapshot = true
    }
  }
}

The shard stress test: Proving multi-tenancy at scale

Before committing to this architecture for a live military exercise, we needed to prove it would be able to meet our requirements and have an appropriate failover in place in the event of issues. Moving from individual deployments to a single multi-tenanted cluster introduced real risks: resource contention, ingest bottlenecks, data leakage across spaces due to misconfiguration, large TCP connection counts on the Elasticsearch nodes, and a significantly larger shard count since each team generates its own set of indices.

So we built a dedicated testing rig. The plan was straightforward: deploy 50 Kibana Spaces, create an agent policy in each space, launch 6,000 EC2 instances (120 per tenant, across six subnets in three availability zones), and load-test the lot. We monitored everything with AutoOps and Stack Monitoring.

The deployment flow worked like this: Terraform created the VPC and subnets across three availability zones, provisioned the 50 Kibana Spaces and their space-scoped Fleet policies, generated enrolment tokens, and then launched EC2 instances in batches. Each instance installed Elastic Agent on boot and enrolled against its space-specific token.

We hit some interesting challenges along the way. The standard Elastic Stack Terraform Provider didn't support space-aware Fleet operations at the time, so we forked it and added space ID handling to the Fleet resources - without that modification, every agent would have enrolled into the default space regardless of policy assignment. This wasn't the first time we'd had to extend the provider for an exercise; two years ago, for DCM2, we'd added the elasticsearch_cluster_info data source. Fortunately, the upstream provider has since added support for space_ids in version 0.12.2.

We also ran into AWS EC2 API rate limits when trying to spin up all 6,000 instances simultaneously, so we batched deployments at 500 instances with five-minute cool-off periods between batches.

The results were reassuring. All 6,000 agents were typically enrolled within 20 minutes of deployment. In our tests, space isolation worked as expected with no observed data leakage between tenants. Fleet policy updates propagated to all agents within 60 seconds. Search queries scoped to individual spaces remained fast under full load. And the multi-AZ distribution proved resilient during simulated availability zone failures.

This testing gave us the confidence to commit to the architecture for the live exercise.

Red Teams: C2 implant observability

A separate, dedicated Elastic deployment was stood up for the Red Teams, focused on Command and Control (C2) implant observability. This gave the attacking teams visibility into their own operations, including implant status, beacon callbacks, and operational progress, without any risk of cross-pollination with the Blue Team's data. The Red Teams used Tuoni as their C2, which is a framework developed by Clarified Security for red teaming. In DCM3, we worked with Clarified Security to ensure it properly supported the Elastic Common Schema, making future integration with Elastic much easier.

NSOC: Exercise Network Security Operations Centre

The core exercise, Network Security Operations Centre (NSOC), ran on its own Elastic deployment, providing the exercise control staff with an overarching view of range health, security monitoring across the entire infrastructure, and critically, audit logging for all the AI services we deployed. Every Bedrock API invocation was logged in CloudWatch and observable in this deployment, meaning the NSOC had complete visibility into what was being asked to the AI agents and by whom . More on this in the AI section below.

Infrastructure automation: Terraform and Catapult

Everything you've seen above was managed as Infrastructure as Code. Our provider.tf gives a sense of the provider ecosystem we were orchestrating:

terraform {
  required_version = ">= 1.5"

  required_providers {
    elasticstack = {
      source  = "elastic/elasticstack"
      version = "~> 0.13.1"
    }
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    vault = {
      source  = "hashicorp/vault"
      version = "~> 3.20"
    }
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 5.15.0"
    }
  }

  backend "s3" {
    bucket  = "elastic-terraform-state-dcm5"
    key     = "prod/terraform.tfstate"
    region  = "eu-west-2"
    encrypt = true
  }
}

The total resource footprint managed by Terraform was substantial: one Elastic Cloud deployment with autoscaling, 40 Kibana Spaces, 120 Fleet agent policies (three per team), 400+ integration policies, 40 Kibana security roles, 40 Keycloak role mappings, ILM policies for data retention, 41 AWS IAM users for Bedrock GenAI connectors (one per team space plus a default), 41 Kibana GenAI action connectors, AWS Bedrock guardrails, Cloudflare Zero Trust tunnels for Tines access, Tines action connectors per team space, detection service accounts stored in HashiCorp Vault, and per-space Security Solution default index configuration. All state was stored in an encrypted S3 backend.

For the agent and proxy deployment onto the actual range systems, we used Catapult, an excellent open-source tool built by the team at Clarified Security. Catapult wraps Ansible with a container-based execution model that's purpose-built for cyber range deployments. It handled the installation and enrolment of Elastic Agents across the range infrastructure. The configuration of proxy servers (each team had a dedicated Squid proxy for its deployed network, this was to simulate a single point of egress as it would be in the real world. Traffic was routing through endpoints like http://elastic-proxy.dsoc.XX.dcm.ex:3128), and the deployment of Cloudflare tunnels for Tines connectivity.

During provisioning, the following were written to HashiCorp Vault by Terraform and consumed by Catapult: Credentials, enrolment tokens, API keys, proxy configurations, Tines service account credentials.. The Vault paths followed a consistent structure like dcm/gt/elastic/prod/enrollment_tokens/BT-XX-Deployed and dcm/gt/elastic/tines-sa/tines-sa-btXX, making it straightforward for the Catapult playbooks to pull the right credentials for each team.

Training: setting teams up for success

Deploying the platform is one thing; ensuring people can actually use it is another. We provided on-range, instructor-led training to the Blue Teams during the pre-exercise phase. This covered Elastic Security fundamentals, navigating their team space in Kibana, working with the prebuilt detection rules, using Discover for log analysis and threat hunting, building custom dashboards, understanding Elastic Defend alerts, and getting familiar with the Timeline investigation tool.

The exercise instruction itself noted this training was optional but "highly recommended," and from what we saw, the teams who attended absolutely hit the ground running on Day one of execution. Training and enablement are just as important as the technology deployment itself. Handing a team enterprise-grade security tooling which they don't know how to use would'nt have been helpful for anyone.

The On-Range AI service: Compliant, audited, Guardrailed

This year marked our debut in providing AI access to the DCM range. We provided a compliant AI service directly on the range, backed by UK-tenanted AWS Bedrock models - specifically Claude 3.7 Sonnet running in the eu-west-2 (London) region. This wasn't AI for the sake of AI; it was a carefully architected service with guardrails, complete audit logging, and RBAC-aware access controls. We were trusted with running this service due to Elastic's experience in the AI space.

The AI service had multiple consumers on the range, and this is an important distinction. The compliant Bedrock connector we provisioned into each team's space wasn't just powering our custom agents - it also powered Elastic's native AI features, specifically:

Elastic AI Assistant for Security

The Elastic AI Assistant was available in every Blue Team space, connected to our on-range Bedrock connector. This gave teams a context-aware chat interface directly within Elastic Security where they could ask questions about their alerts, get help writing ES|QL queries, investigate suspicious processes, and get guided remediation steps. The AI Assistant uses Retrieval-Augmented Generation (RAG) with Elastic's Knowledge Base feature, which is pre-populated with articles from Elastic Security Labs. Teams could also add their own documents, such as range-specific SOPs, threat intel, or team notes, to the Knowledge Base to further ground the assistant's responses in their operational context.

What made this particularly valuable in the exercise context was the AI Assistant's ability to help less experienced analysts understand what they were looking at. A junior analyst facing their first live implant beacon could ask the assistant to explain the alert, suggest investigation steps, and even help draft the incident report. The data anonymisation settings ensured that sensitive field values could be obfuscated before being sent to the LLM provider.

Elastic Attack Discovery

Attack Discovery was another significant consumer of our on-range AI service. Attack Discovery uses LLMs to analyse alerts in a team's environment and identify threats by correlating alerts, behaviours, and attack paths. Each "discovery" represents a potential attack and describes relationships among multiple alerts - telling teams which users and hosts are involved, how alerts map to the MITRE ATT&CK matrix, and which threat actor might be responsible.

For a cyber exercise in which Red Teams actively launched coordinated attacks, Attack Discovery was transformative. Instead of manually triaging hundreds of individual alerts, Blue Teams could run Attack Discovery to surface the high-level attack narratives, for example, "these 15 alerts are all part of a lateral movement chain from host X to host Y, likely by threat actor Z", and focus their investigation time where it mattered most. It's the kind of capability that directly reduces mean time to respond, and fights alert fatigue, which is precisely what you need when you're under sustained attack for five days straight.

The custom AI agents: Elastic Agent Builder

Beyond the native Elastic AI features, we built three bespoke AI agents using Elastic Agent Builder. Agent Builder is Elastic's framework for building custom AI agents that combine LLM instructions with modular, reusable tools, each tool being an ES|QL query, a built-in search capability, workflow execution, or an external integration via MCP. Agents parse natural language requests, select the appropriate tools, execute them, and iterate until they can provide a complete answer, all while managing context with data inside Elasticsearch. You can read more about the framework in the Agent Builder documentation and the Elasticsearch Labs deep dive.

The three key components of Agent Builder that we leveraged were:

Agents: Custom LLM instructions and a set of assigned tools that define the agent's persona, capabilities, and behaviour boundaries. Each agent has a system prompt that controls its mission, the tools it can access, and the structure of its responses.

Tools: Modular functions that agents use to search, retrieve, and manipulate Elasticsearch data. We built custom ES|QL tools that queried specific indices containing exercise documentation, playbooks, and reports.

Agent Chat: The conversational interface - both the built-in Kibana UI and the programmatic API - that participants used to interact with the agents.

Agent and tool configurations are defined as JSON and managed via the Agent Builder APIs, making the entire agent lifecycle - from prompt engineering to tool binding - reproducible and version-controllable. We'll share the GrantPT agent configuration and tool definitions in a follow-up post for those who want to replicate this approach - watch this space.

Here's what each agent did:

1. GrantPT - The general-purpose assistant

Available to all ~2,500 exercise participants, GrantPT was our primary AI agent and the best demonstration of how straightforward Agent Builder makes it to stand up a capable, domain-specific assistant. The agent's configuration consisted of a JSON object defining its system prompt, persona, and an array of bound tool IDs - that's it. No custom application code, no bespoke API layer, just declarative configuration.

What gave GrantPT its depth was the tooling. We defined a mix of built-in platform tools and custom ES|QL tools, each registered with a description, a parameterised query, and typed parameter definitions. For example, the knowledge base tool accepted a target_index and a semantic query parameter, executing a parameterised ES|QL query against our dcm5-grantpt-* indices with semantic search ranking:

FROM dcm5-grantpt-* METADATA _score, _index
| WHERE _index == ?target_index
| WHERE content: ?query
| SORT _score DESC
| LIMIT 10

A separate index discovery tool let the agent dynamically enumerate available knowledge base indices at the start of each conversation, meaning we could add new documentation indices during the exercise without reconfiguring the agent; it would simply discover them on the next interaction.

We also built a Jira integration tool that performed semantic search across ingested helpdesk tickets, enabling GrantPT to surface relevant troubleshooting context from prior support requests. This was particularly useful for the HelpDesk Analysts, who could ask GrantPT about recurring issues and get responses grounded in actual ticket history rather than generic guidance.

The RBAC-tailored response behaviour came from a combination of the agent's system prompt, which instructed it to contextualise answers based on the user's role, and the underlying Elasticsearch security model. Because each tool's ES|QL query is executed within the user's security context, the agent can only surface documents accessible to the user's role. A Blue Team member asking about exercise procedures would get results scoped to their team's accessible indices, whilst a HelpDesk Analyst would see results from helpdesk-specific indices. The agent didn't need explicit role-switching logic; Elasticsearch's native document-level security handled scoping, and the agent simply worked with whatever results were returned. This is one of the things that makes Agent Builder genuinely elegant - by inheriting Elasticsearch's security model, you get RBAC-aware AI without writing a single line of authorisation code.

2. REDRock - The adversary's companion

This agent was exclusively available to Red Teams. REDRock followed the same Agent Builder pattern, a dedicated system prompt defining its adversarial persona, bound to its own set of custom ES|QL tools querying Red Team-specific indices. These indices contained the Red Team playbooks, Tuoni C2 documentation, known system vulnerabilities within the range environment, and information about deployed services. The tool definitions mirrored the same parameterised semantic search pattern used by GrantPT, but were scoped to indices accessible only to Red Team roles. Red Team operators could query attack vectors, check for known weaknesses in target systems, and get contextual guidance on their operational plans. It was, quite frankly, like giving the attackers an extremely well-briefed operations officer.

3. RefPT - The referee's tool

Built specifically for the White Team (the exercise referees and assessors), RefPT was bound to tools querying indices containing Blue Team reports, scenario events, and the scoring criteria. Its purpose was to ensure uniform and fair scoring across all 40+ teams. The agent's system prompt was tuned to cross-reference submitted reports against known scenario events and scoring rubrics, helping assessors identify inconsistencies or gaps. When you've got assessors evaluating dozens of teams simultaneously, having an AI that can correlate reports against a structured scoring index is genuinely transformative for consistency.

Tines: AI-powered workflow automation

Tines was also a consumer of the on-range AI service. Each Blue Team had a dedicated Tines instance, with Tines action connectors provisioned in their Kibana space. Tines could leverage the Bedrock-backed AI capabilities for intelligent workflow automation, such as automated alert enrichment, AI-assisted triage decisions, natural-language summaries in notification workflows, and natural-language workflow creation. The Tines connector was configured per-team with credentials stored in Vault:

resource "elasticstack_kibana_action_connector" "tines_bt" {
  count = var.team_count

  name              = "BT-${local.team_numbers[count.index]}-Tines"
  connector_type_id = ".tines"
  space_id          = local.space_ids[count.index]

  config = jsonencode({
    url = "https://tines.dsoc.${local.team_numbers[count.index]}.dcm.ex/"
  })
}

Ensuring compliance: Guardrails and audit

Every AI interaction across all of these consumers was governed by strict AWS Bedrock Guardrails. We deployed guardrails with content filtering (hate, insults, sexual content, and violence at MEDIUM thresholds), PII protection (blocking email addresses, phone numbers, names, addresses, UK National Insurance numbers, credit card numbers, and IP addresses), topic-based filtering to prevent discussion of actual classified operations, and profanity filtering. Here's a snippet of the guardrail configuration from our Terraform:

resource "aws_bedrock_guardrail" "dcm5_elastic" {
  name        = "dcm5-prod-elastic-guardrail"
  description = "Guardrails for DCM5 Prod Elastic Kibana GenAI connectors"

  content_policy_config {
    filters_config {
      input_strength  = "MEDIUM"
      output_strength = "MEDIUM"
      type            = "HATE"
    }
    # ... additional content filters for INSULTS, SEXUAL, VIOLENCE
  }

  sensitive_information_policy_config {
    pii_entities_config {
      action = "BLOCK"
      type   = "UK_NATIONAL_INSURANCE_NUMBER"
    }
    pii_entities_config {
      action = "BLOCK"
      type   = "IP_ADDRESS"
    }
    # ... additional PII filters
  }

  topic_policy_config {
    topics_config {
      name       = "classified-information"
      definition = "Discussions about actual classified operations, current real-world military activities, or operational intelligence."
      type       = "DENY"
    }
  }
}

Each Blue Team space had its own IAM user for Bedrock access, and the genAiSettings:defaultAIConnectorOnly Kibana setting was enforced to prevent teams from configuring their own connectors. This meant every single API call could be traced back to a specific team via CloudWatch, and the NSOC had complete audit visibility. The CloudWatch log group /aws/bedrock/grantpt-prod/invocations captured every invocation and guardrail event.

The numbers for all AI consumers speak for themselves: 3 custom AI Agents, 2,797 conversations, and 785 million AI tokens consumed throughout the exercise.

In-game real-time monitoring

Within the exercise scenario, each team had access to RocketChat as their on-range messaging client. Every Blue Team got its own channel, the ability to direct message anyone in the exercise, and the freedom to spin up new channels as needed. Most critically for DCM tradition, this included the memes channel - the spiritual backbone of all inter-team ribbing and the creative morale-boosting humour that inevitably emerges when you put a few thousand cyber operators under pressure for a week.

All of this communication data represented a brilliant real-time window into range health, team sentiment, and the topics trending across the exercise. It felt too good to pass up, so we ingested the entire RocketChat conversation corpus into Elastic in real time and put it to work.

Sentiment analysis and named entity recognition

For named entity recognition, we deployed the dslim/bert-base-NER model from Hugging Face into a machine learning node on the NSOC deployment using the Elastic ELAND client. This was then wired into an Elasticsearch ingest pipeline that every RocketChat message passed through on ingestion. We took the extracted entities and surfaced the most common ones as dashboard themes, giving us a live view of the ebb and flow of conversation topics throughout the exercise.

We also analysed group activity, user statistics, and general communication patterns to build a picture of life patterns for each team - most active participants, message volume over time, and sentiment trends pivoted by individual users. All told, it gave us some genuinely interesting insight into what was happening on the range in near real time. When we switched Elastic Agent into Prevent mode, for instance, a word cloud on our dashboard immediately lit up with "Elastic" as the most discussed theme across all channels - Blue Teams discussing its effectiveness, Red Teams lamenting their lost beacons. Rather satisfying, that.

Meme analysis (yes, really)

Finally - and this one raised a few eyebrows - we pulled every meme submitted to the channels, vectorised the images, and ran nearest-neighbour evaluations to cluster similar memes and topics together. We also passed them through the zero-shot NER inference model to generate thematic descriptions of each meme's content. The logic was that these outputs might prove useful later for filtering, moderation, or other in-game interactions. Whether the meme analysis yielded operationally critical intelligence is debatable. Whether it was good fun is not.

Nipping problems in the bud

As much as we hoped everything would run smoothly during exercise week, things inevitably break, aren't fully understood, or need further customisation to suit how a particular team wants to use them. For this, we had our own subsection of the in-range helpdesk where Elastic and GenAI-specific requests could be raised by any team.

We manned this helpdesk for the entire duration of the exercise, providing guidance, documentation, issue debugging, and range-specific recommendations. That last point is worth expanding on. Sometimes, what a Blue Team was seeing in Elastic wasn't actually an Elastic problem at all, but rather Elastic faithfully surfacing something on the range that warranted further investigation (Red Teams can cause absolute mayhem, and the telemetry doesn't lie). Over the course of the exercise, we covered 125 individual support requests from teams specifically asking for help from us at Elastic.

Pre-emptive debugging with Tines

Beyond visiting teams via VTC or in person at EXCON, we also worked with Tines to try something a bit more proactive. We pulled the ticket body from incoming requests, attempted to categorise the problem, ran the categorisation against our corpus of previously resolved tickets, and had GenAI produce a summarised first-pass response aimed at solving the user's issue before triage brought it to our queue.

This is actually a pattern we borrowed from our own support organisation at Elastic, where we provide a similar capability using our extensive knowledge base of previously solved issues as a repository for supporting AI Agent context. The idea is straightforward: use past solutions to give a machine-generated, informed first stab at resolving a problem, and short-circuit the need for a support engineer to pick up every ticket manually. It didn't solve everything; some issues genuinely needed a human with range context, but it meaningfully reduced the queue pressure and got faster answers to the teams who needed them. This was such a success with our own specific tickets and queue that we actually extended the remit to the entire helpdesk in the latter part of the exercise, helping to reduce the load on the other groups in the Green team supporting the exercise.

Industry partnerships: Better together

One of the things we're most proud of is how our partnership ecosystem has grown year on year. DCM is not just an Elastic show; it's a genuine coalition of industry partners, each bringing something unique to the security platform.

Year 1 (DCM2) - Elastic joined as an industry partner, providing the security monitoring and endpoint detection platform.

Year 2 (DCM3) - We brought in Endace, providing 1:1 packet capture capability. Full packet capture alongside Elastic's network visibility gave teams the ability to conduct deep-dive forensics that log-based analysis alone can't provide.

Year 3 (DCM4) - Tines joined the family, bringing workflow automation to the table. Blue Teams could now build automated response playbooks, triage workflows, and notification chains, all integrated directly into their Elastic environment via the native Tines connector.

Year 4 (DCM26, formerly DCM5) - AWS came on board, providing Bedrock access for our AI agents and contributing funding towards the Elastic deployments. This was a significant milestone; having a hyperscaler directly invested in the exercise's success unlocked capabilities (such as compliant, UK-tenanted AI inference with full guardrails and audit logging) that simply wouldn't have been possible otherwise. Tines' integration this year was also enhanced by the addition of on-range access to LLMs. The DCM series also reached a milestone this year, transitioning from its origins as an Army Cyber Association initiative to an officially funded programme under Cyber and Specialist Operations Command.

To the teams at Endace, Tines, and AWS - sincere thanks. This exercise is better because of your contributions, and all Teams are better equipped because of the platform we've built together. We're already planning for DCM27. Cheers to the lot of you.

Culture, highlights, and the bits that make it worthwhile

The Challenge Coins

We had custom challenge coins minted for DCM26. If you know, you know, challenge coins are a long-standing military tradition, and having one made for the exercise felt like the right way to mark our fourth year of involvement.

The cocktail party

We were also grateful to be invited to the High Commission cocktail party hosted by the British High Commissioner to Singapore. There's something quite surreal about discussing Elasticsearch shard counts and Terraform state management whilst holding a gin and tonic at the ambassador's invitation. It was a brilliant evening, a genuine reminder that these exercises exist at the intersection of technology and diplomacy, and that the relationships built here extend well beyond the technical.

Wrapping up

The multi-tenanted architecture proved itself under sustained load; the native Elastic AI features (AI Assistant and Attack Discovery) gave teams capabilities that would have been science fiction a few years ago; and the custom AI agents exceeded our expectations for adoption. The partnership model continues to demonstrate that industry involvement in defence exercises creates outcomes that no single organisation could achieve alone.

Defence Cyber Marvel 2026 was a landmark iteration of an exercise that continues to grow in ambition, complexity, and impact. For Elastic, being trusted to provide the core defensive security platform for 40 Blue Teams from 29 nations, and this year, the AI capability as well, is something we don't take lightly. The exercise develops real skills for real people who will go on to defend real networks, and being a part of that mission is genuinely meaningful.

As the UK Government's press release put it, DCM demonstrates the practical value of real-life scenarios that reinforce international partnerships. We couldn't agree more.

We'll be back next year, and I suspect we'll have even more to talk about. In the meantime, we'll continue to improve the product so that support for environments such as Defence Cyber Marvel excels year over year.

See you on the range.

Follow the DCM26 story on social media:

Facebook | LinkedIn | Instagram

Further reading

Elastic Security & AI

• Elastic Security - The platform powering the Blue Team deployments
• AI Assistant for Security - Context-aware AI chat within Elastic Security
• Attack Discovery - LLM-powered alert correlation and threat narrative generation
• Agent Builder - Framework for building custom AI agents with Elasticsearch

Infrastructure & Tooling

• Elastic Stack Terraform Provider - Infrastructure as Code for the Elastic Stack
• Elastic Fleet Guide - Centrally managing Elastic Agents at scale
• Catapult by Clarified Security - Ansible-based cyber range provisioning

Exercise Context

• UK Government DCM26 Press Release - Official overview of the exercise

DaC as a methodology applies software development practices to the creation, management, and deployment of security detection rules. By treating detection rules as code, it enables version control, automated testing, and deployment processes, enhancing collaboration, consistency, and agility in response to threats. DaC streamlines the detection rule lifecycle, ensuring high-quality detections through peer reviews and automated tests. This methodology also supports compliance with change management requirements and fosters a mature security posture.

That's why we’re excited to share the latest updates to Elastic's detection-rules, our open repository for writing, testing, and managing security detection rules in Elastic, that also allows you to create your own Detections as Code (DaC) framework. Continue reading for highlighted implementation examples using extended functionality, and the announcement of Elastic's free Detections as Code Workshop.

Elastic Security DaC: The journey from alpha to general availability

With the functionality now provided in detection-rules repository, users can manage all their detection rules as code, review rule tunings, automatically test and validate rules, and automate rules deployment across their environments.

Pre-2024: Elastic’s internal use of DaC

Elastic threat research and detection engineering team created and used the detection-rules repository to develop, test, manage and release prebuilt rules, following DaC principles - reviewing rules as a team, automating their tests and release. The repository also has an interactive CLI to create rules, so engineers could start working on the rules right there.

As the security community's interest in as-code principles grew, and the available Elastic Security APIs already allowed users to implement their custom Detections as code solutions, Elastic decided to extend the detection-rules repository functionality to enable our users to benefit from our tooling and aid them in creating their DaC processes.

Here are the key milestones of Elastic’s user-focused DaC development from alpha to general availability.

May 2024: Alpha release of new "roll your own” features

Our detection-rules repository is adjusted for customer use, allowing for managing custom rules, adapting the test suite for user needs, and allowing for management of actions and exceptions alongside the rules.

Key additions:

• Custom rules directory support
• Select which test to run based on your requirements
• Exceptions and Actions support

We also published an extensive guidance for Detections as Code with examples of implementation with Elastic Security using detection-rules repository.

August 2024: "Roll your own” features now beta

The functionality is extended to allow import and export of custom rules between Elastic Security and repository, more configuration options and versioning functionality extended to custom rules.

New features added:

• Bulk import/export of custom rules (based on Elastic Security APIs)
• Fully configurable unit test, validation, and schemas
• Version lock for custom rules

March - August 2025: are generally available and supported

Using DaC with Elastic Security 8.18 and up:

• Supports prebuilt rules management. You can export all prebuilt rules from Elastic Security and store them alongside your custom rules.
• Support for rules filtering for export added.

Adjacent to DaC efforts, we also released new Terraform resources (V0.12.0 and V0.13.0) in October-December 2025, allowing Terraform users to manage detection rules and exceptions.

With this foundation spelled out, let's explore the powerful features that are available to streamline your detection engineering process.

Detection-rules DaC functionality highlights

There are a few worthwhile additions since our last DaC publication, which we’ll expand on below.

Additional filters

The filter functionality available when exporting rules from Kibana has been extended to allow you to precisely define which rules to sync in DaC. Here are the new flags:

| Flag | Description | | :---: | ----- | | -cro | Filters the export to only include rules created by the user (not Elastic prebuilt rules). | | -eq | Applies a query filter to the rules being exported. |

Let’s take an example of when you wish to organize rules by data source, and want to export the AWS rules to a specific folder. In this case, let’s use filtering on tags for data sources and export all rules with the Data Source AWS tag:

python -m detection_rules kibana export-rules -d dac_test/rules #add rules to the dac_test/rules folder
-sv #strip the version fields from all rules
-cro #export only custom rules
-eq "alert.attributes.tags: "Data Source: AWS"" # export only rules with "Data Source: AWS" tag

See Kibana documentation for query string filtering for the underlying API call used here and the list all detection rules API call for example available fields to construct the query filter.

Custom folder structure

In the detection-rules repo, we use a folder structure based on platform, integration, and MITRE ATT&CK information. This helps us with our organization and rule development. This is by no means the only method of organization. You may want to organize your rules by customer, date, or source as examples. This will vary greatly depending on your use case.

Whether you use this export process or manual organization, once you have your rules in a location or folder structure that you like, you can now keep this local structure even when re-exporting rules. It is important to note that the new rules need to be placed in their desired location manually. The local rule-loading mechanism detects where the rules are placed in order to know where to put them. If the rule is not there, it will then use the specified output directory to place the new rule(s). To use the local rule loading for updating existing rules use the --load-rule-loading / -lr flag for the kibana export-rules and import-rules-to-repo commands. These flags enable you to make use of the local folders specified in your config.yaml.

Let’s look at example with the rules organised in folders the following way:

rule_dirs:
- rules
my_test_rule.toml
- another_rules_dir
high_number_of_process_and_or_service_terminations.toml

We’ll specify the following in the config.yaml file:

rule_dirs:
- rules
- another_rules_dir

With the new -lr option, rule updates from Kibana will now use these additional paths instead of exporting directly to the specified directory.

Running python -m detection_rules kibana --space test_local export-rules -d dac_test/rules/ -sv -ac -e -lr,will export rules from test_local space, my_test_rule.toml will be written to dac_test/rules/ as it was already on disk there and high_number_of_process_and_or_service_terminations.toml will be written to dac_test/another_rules_dir/.

This can be particularly useful if you have the same rules in different sub-folder configurations for different customers. For example, let’s say you have your rules broken down by platform and integration similar to Elastic’s prebuilt rule folder structure. For your customers, SOCs, or threat-hunting teams, having the rules organized underneath these platform/integration folders may be the most useful mechanism for them to manage the rules. However, your information security team or primary detection engineering team may want to manage the rules by initiative or rule author instead so that all the rules a particular individual or team is responsible for are organized in one place. Now with the local rule-loading flags, you can simply have two configuration files and the duplicated rules in each structure. When you are exporting updates for the rules, you would then use the environment variable to select the appropriate configuration file and export the rule updates. These updates will then be applied to the rules in place, maintaining the directory structure.

Miscellaneous local loading updates

In addition to the above, we have added two smaller new features designed to help users who are adding local information in the detection rules TOML files and schema. These are as follows:

1. Local date support from the local files where the local date will be maintained from the original file
2. Upgrades to the auto gen feature to inherit known types from existing schema.

The local date component can be useful when one wants more manual control over the date field in the file. Without using the override, the date will be based on when the Kibana rule contents were exported. Using the --local-creation-date flag, the date will not be updated when the file contents are re-exported.

The automatic schema generation has been updated to inherit the types from other indices/integrations if they are present. This provides a potentially more accurate schema, as well as reducing the need for manual updates after the fact. For example, you have a rule that uses the index “new-integration*” with the following fields:

• host.os.type.new_field
• dll.Ext.relative_file_creation_time
• process.name.okta.thread

Instead of each of these fields being added to the schema with a default type, their types are inherited from existing schemas. In this case, the types for dll.Ext.relative_file_creation_time and process.name.okta.thread are inherited.

{
  "new-integration*": {
    "dll.Ext.relative_file_creation_time": "double",
    "host.os.type.new_field": "keyword",
    "process.name.okta.thread": "keyword"
  }
}

To see how to use this with your custom data types, see the Custom schemas usage section within the Implementation examples part of this blog.

Expanding on usage examples

Below you will find more examples of DaC implementations, these are not focused on new functionality additions, but go deeper on the topics we see discussed in the community.

It’s worth noting that Detections as Code features are provided as components that can be used to build a custom implementation for your chosen process and architecture. When implementing DaC in your production environment, treat it as an engineering process and follow the best practices.

DaC implementation with Gitlab

When we look at implementations of DaC typically this revolves around using some form of CI/CD product to automatically perform rule management based on a given trigger. These triggers vary considerably based on the desired setup, specifically the authoritative source of rules and the desired state of your version control system (VCS). For a much more in-depth exploration of some of these considerations, see our DaC Reference Material. Below is a simple example using Gitlab as VCS provider and using its in-built CI/CD via Gitlab Actions.

stages:                # Define the pipeline stages
  - sync               # Add a 'sync' stage

sync-to-production:    # Define a job named 'sync-to-production'
  stage: sync          # Assign this job to the 'sync' stage
  image: python:3.12   # Use the Python 3.12 Docker image
  variables:
    CUSTOM_RULES_DIR: $CUSTOM_RULES_DIR    # Set custom rules env var
  script:                                  # List of commands to run 
    - python -m pip install --upgrade pip  # Upgrade pip
    - pip cache purge                      # Clear pip cache
    - pip install .[dev]                   # Install package w/ dev deps
    - |  # Multi-line command to import rules                                        
      FLAGS="-d ${CUSTOM_RULES_DIR}/rules/ --overwrite -e -ac"
      python -m detection_rules kibana --space production import-rules $FLAGS
  environment:
    name: production   # Specify deployment environment as 'production'
  only:
    refs:
      - main           # Run this job only on the 'main' branch
    changes:
      - '**/*.toml'    # Run this job only if .toml files have changed

This is very similar to other inbuilt CI/CD from other Git-based VCS like Gitlab and Gitea. The main difference being in the syntax determining the triggering event. The DaC commands such as kibana import-rules would be the same regardless of VCS. In this example, we are syncing rules from our fork of the detection-rules repo to our Kibana Production Space. This is based on a number of prior decisions being made, for instance requiring unit tests to pass before merging rule updates and that rules on main being ready for prod. For a Github-based walkthrough of these considerations for this particular approach, please take a look at our demo video.

Custom Unit Testing tips and examples

When considering DaC as a capability to add to your detection toolkit, setting up the CI/CD and base infrastructure should be considered as the first step in an ongoing process to improve the quality and usefulness of your rules. One of the key purposes in having “as code” tooling is adding the ability to further customize tooling to your needs and environment.

One example of this is unit testing for rules. Beyond base functionality testing, some other key existing unit tests enforce Elastic-specific considerations around rule performance and optimization, as well as organization of metadata and tagging. This helps detection engineers and threat researchers remain consistent in their rule development. Building on this example, one may want to consider adding custom unit tests based on your specific needs.

To illustrate this, take a Security Operations Center (SOC) environment where there are a number of analysts responsible for various different domains and tasks. When an alert is raised in the SIEM, it may not be immediately obvious who should handle remediation, or what team(s) need to be informed of the incident. Tagging the rules with a team tag: e.g. Team: Windows Servers similarly to how Elastic uses tags for data sources, can provide the SOC with a point of contact directly in the alert for who can help with remediation.

In our DaC environment, we can quickly create a new testing module to enforce this on all of the custom rules (or pre-built too). For this test, we are going to enforce having a Team: <some name> tag on all production rules that are not authored by Elastic. In the detection-rules repo, our testing is handled through the Python test suite called pytest and as such unit tests are organized into python modules (files) and subsequent classes and functions in these files under the tests/ folder. To add tests simply either add classes or functions to the existing files or create a new one. In general, we recommend creating new test files so that you can receive updates to the existing tests from Elastic without having to merge the differences.

We will start by creating a new python file called test_custom_rules.py in the tests/ directory with the following contents:

# test_custom_rules.py

"""Unit Tests for Custom Rules."""

from .base import BaseRuleTest


class TestCustomRules(BaseRuleTest):
    """Test custom rules for given criteria."""

    def test_custom_rule_team_tag(self):
        """Unit test that all custom rules have a Team: <team_name> tag."""
        tag_format = "Team: <team_name>"
        for rule in self.all_rules:
            if "Elastic" not in rule.contents.data.author:
                tags = rule.contents.data.tags
                if tags:
                    self.assertTrue(
                        any(tag.startswith("Team: ") for tag in tags),
                        f"Custom rule {rule.contents.data.rule_id} does not have a {tag_format} tag",
                    )
                else:
                    raise AssertionError(
                        f"Custom rule {rule.contents.data.rule_id} does not have any tags, include a {tag_format} tag"
                    )

Now each non-Elastic rule will be required to have a tag in the specified pattern for a team responsible for remediation. E.g. Team: Team A.

Custom schemas usage

Elastic’s ability to bring your own data types also extends to our DaC capabilities. For example, let’s take a look at some custom schemas for network protocols. Diverse data you have in your stack can of course be queried by your rules, and we will also want to leverage the applicable validation and testing for any custom rules on these data types too. This is where Custom schemas come in handy.

When we are validating queries, the query is parsed into the respective fields and the types of these fields are compared against what is provided in a given schema (e.g. ECS schema, the AWS Integration for AWS data, etc.). For custom data types, this follows the same validation path, with the ability to pull from locally defined custom schemas. These schema files can be built by hand as one or more json files; however, if you have some sample data already in your stack, you can take advantage of this and use it as validation and generate your schemas automatically.

Assuming you already have a custom rules folder configured (if not see instructions), you can turn on automatic schema generation by adding auto_gen_schema_file: <path_to_your_json_file> to your config file. This will generate a schema file in the specified location that will be used to add entries for each field and index combination. The file will be updated during any command where rule contents are validated against a schema, including import-rules-to-repo, kibana export-rules, view-rule, and others. This will also automatically add it to your stack-schema-map.yaml file when using a custom rules directory and config.

With this power comes an increased responsibility on rule reviewers as any field used in the query is immediately assumed to be valid and added to the schema. One way to mitigate risk is to utilize a development space that has access to the data. In the PR, one can then link to a successful execution of the query with stack level validation on its data types. Once this is approved, one can remove the auto_gen_schema_file addition to the config and you now have a known valid schema based on your custom data. This provides a baseline for other rule authors to build upon as needed and maintains the type checking validation.

Learn more about DaC and try it yourself

You can experience Elastic Security's Detections as Code (DaC) functionality firsthand with our interactive Instruqt training. This training provides a straightforward way to explore core DaC features in a pre-configured test environment, eliminating the need for manual setup. Give it a try!

If you are implementing DaC, share your experience, ask your questions and help others on the community slack DaC channel.

Trial Elastic Security

To experience the full benefits of what Elastic has to offer for detection engineers, start your Elastic Security free trial. Visit elastic.co/security to learn more.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Why We Did It

Our primary motivation was to move from general assumptions to data-backed insights about:

• How quickly different teams and environments patch vulnerabilities
• Whether patching performance meets internal service level objectives (SLOs)
• Where bottlenecks or delays commonly occur
• What other factors can affect patching performance

Why Survival Analysis? A Better Alternative to Mean Time to Remediate

Mean Time to Remediate (MTTR) is commonly used to track how quickly vulnerabilities are patched, but both the mean and median suffer from significant limitations (we provide an example later in this article). The mean is highly sensitive to outliers[^1] and assumes the remediation times are evenly balanced around the average remediation time, which is rarely the case in practice. The median is less sensitive to extremes but discards information about the shape of the distribution and says nothing about the long tail of slow-to-patch vulnerabilities. Neither accounts for unresolved cases, i.e. vulnerabilities that remain open beyond the observation window, which are often excluded entirely. In practice, the vulnerabilities that remain open the longest are precisely the ones we should be most concerned about.

Survival analysis addresses these limitations. Originating in medical and actuarial contexts, it models time-to-event data while explicitly incorporating censored observations, meaning in our context vulnerabilities that remain open. (For more details on its application to vulnerability management we strongly recommend “The Metrics Manifesto”). Instead of collapsing remediation behavior into a single number, survival analysis estimates the probability that a vulnerability remains unpatched over time (e.g. 90% of vulnerabilities are remediated within 30 days). This allows for more meaningful assessments, such as the proportion of vulnerabilities patched within SLO (for example within 30, 90, or 180 days).

Survival analysis provides us with a survival function that estimates the probability a vulnerability remains unpatched over time.

::: This method offers a better view of remediation performance, allowing us to assess not just how long vulnerabilities persist, but also how remediation behavior differs across systems, teams, or severity levels. It’s particularly well-suited to security data, which is often incomplete, skewed, and resistant to assumptions of normality. :::

Context

Although we have applied survival analysis across different environments, teams and organizations, in this blog we focus on the results for the Elastic Cloud production environment.

Vulnerability age calculation

There are different methods to calculate vulnerability age.

For our internal metrics like vulnerability adherence SLO, we define vulnerability age as the difference between when a vulnerability was last found and when it was first detected (usually a few days after publication). This approach aims to penalize vulnerabilities that are reintroduced from an outdated base image. In the past, our base images were not updated frequently enough for our satisfaction. If a new instance is created, vulnerabilities can have a significant age (e.g., 100 days) from day one of discovery.

For this analysis, we find it more relevant to calculate the age based on the number of days between the last found date and the first found date. In this case, age represents the number of days the system was effectively exposed.

“Patch everything” strategy

In our Cloud environment, we maintain a policy to patch everything. This is because we almost exclusively use the same base image across all instances. Since Elastic Cloud operates fully on containers, there are no specific application packages (e.g., Elasticsearch) installed directly on our systems. Our fleet remains homogeneous as a result.

Data Pipeline

Ingesting and mapping data into the Elastic Stack can be cumbersome. Luckily, we have many security integrations that handle those natively, Qualys VMDR being one of them.

This integration has 3 main interests over custom ingestion methods (e.g. scripts, beats, …):

• It natively enriches vulnerability data from the Qualys Knowledge Base which add CVE IDs, threat intel information, … without needing to configure enrich pipelines.
• Qualys data is already mapped to the Elastic Common Schema which is a standardized way of representing data, whether it’s coming from one source or another: for example, CVEs are always stored in field vulnerability.id, independent of the source.
• A transform with the latest vulnerability is already set up. This index can be queried to get the latest vulnerabilities status.

Qualys agent integration configuration

For survival analysis, we need to ingest both active and patched vulnerabilities. To analyze a specific period, we need to set the number of days in field max_days_since_detection_updated. In our environment, we ingest Qualys data daily, so there’s no need to ingest a long history of fixed data, as we’ve already done that.

The Qualys VMDR elastic agent integration has been configured with the following:

| Property | Value | Comment | | :---- | :---- | :---- | | (Settings section) Username | | | | (Settings section) Password | | Since there are no API keys available in Qualys, we can only authenticate with Basic Authentication. Make sure SSO is disabled on this account | | URL | https://qualysapi.qg2.apps.qualys.com (for US2) | https://www.qualys.com/platform-identification/ | | Interval | 4h | Adjust it based on the number of ingested events. | | Input parameters | show_asset_id=1& include_vuln_type=confirmed&show_results=1&max_days_since_detection_updated=3&status=New,Active,Re-Opened,Fixed&filter_superseded_qids=1&use_tags=1&tag_set_by=name&tag_include_selector=all&tag_exclude_selector=any&tag_set_include=status:running&tag_set_exclude=status:terminated,status:stopped,status:stale&show_tags=1&show_cloud_tags=1 | show_asset_id=1: retrieve asset id show_results=1: details about what is the current installed package and which version should be installed max_days_since_detection_updated=3: filter out any vulnerabilities that haven’t been updated over the last 3 days (e.g. patched older than 3 days) status=New,Active,Re-Opened,Fixed: all vulnerability status are ingested filter_superseded_qids=1: ignore superseded ‘vulnerabilities Tags: filter by tags show_tags=1: retrieve Qualys tags show_cloud_tags=1: retrieve Cloud tags |

Once data is fully ingested, it can be reviewed either in Kibana Discover (logs-* data view -> data_stream.dataset : "qualys_vmdr.asset_host_detection" ), either in the Kibana Security App (Findings -> Vulnerabilities).

Loading data into Python with the elasticsearch client

Since the survival analysis calculation will be done in Python, we need to extract data from elastic into a python dataframe. There are several ways to achieve this, and in this article we’ll focus on two of them.

With ES|QL

The easiest and most convenient way is to leverage ES|QL with the arrow format. It’ll automatically populate the python dataframe (rows and columns). We recommend reading the blog post From ES|QL to native Pandas dataframes in Python to get more details.

from elasticsearch import Elasticsearch
import pandas as pd

client = Elasticsearch(
    "https://[host].elastic-cloud.com",
    api_key="...",
)

response = client.esql.query(
    query="""
   FROM logs-qualys_vmdr.asset_host_detection-default
    | WHERE elastic.owner.team == "platform-security" AND elastic.environment == "production"
    | WHERE qualys_vmdr.asset_host_detection.vulnerability.is_ignored == FALSE
    | EVAL vulnerability_age = DATE_DIFF("day", qualys_vmdr.asset_host_detection.vulnerability.first_found_datetime, qualys_vmdr.asset_host_detection.vulnerability.last_found_datetime)
    | STATS 
        mean=AVG(vulnerability_age), 
        median=MEDIAN(vulnerability_age)
    """,
    format="arrow",
)
df = response.to_pandas(types_mapper=pd.ArrowDtype)
print(df)

Today, we have a limitation with ESQL: we can’t paginate through results. Therefore we are limited to 10K output documents (100K if server configuration is modified). Progress can be followed through this enhancement request.

With DSL

In the elasticsearch python client, there is a native feature to extract all the data from a query with transparent pagination. The challenging part is to create the DSL query. We recommend creating the query in Discover and then click on Inspect, and then Request tab to get the DSL query.

query = {
    "track_total_hits": True,
    "query": {
        "bool": {
            "filter": [
                {
                    "match": {
                        "elastic.owner.team": "awesome-sre-team"
                    }
                },
                {
                    "match": {
                        "elastic.environment": "production"
                    }
                },
                {
                    "match": {
"qualys_vmdr.asset_host_detection.vulnerability.is_ignored": False
                    }
                }
            ]
        }
    },
    "fields": [
        "@timestamp",
        "qualys_vmdr.asset_host_detection.vulnerability.unique_vuln_id",
        "qualys_vmdr.asset_host_detection.vulnerability.first_found_datetime",
        "qualys_vmdr.asset_host_detection.vulnerability.last_found_datetime",
        "elastic.vulnerability.age",
        "qualys_vmdr.asset_host_detection.vulnerability.status",
        "vulnerability.severity",
        "qualys_vmdr.asset_host_detection.vulnerability.is_ignored"
    ],
    "_source": False
}

results = list(scan(
        client=es,
        query=query,
        scroll='30m',
        index=source_index,
        size=10000,
        raise_on_error=True,
        preserve_order=False,
        clear_scroll=True
    ))

Survival Analysis

You can refer to the code to understand or reproduce it on your dataset.

What We Learned

Leaning in on the research from the Cyentia Institute we looked at a few different ways to measure how long it takes to remediate vulnerabilities using means, medians, and survival curves. Each method gives a different lens through which we can understand time-to-patch data, and the comparison is important because depending on which method we use, we would draw very different conclusions about how well vulnerabilities are being addressed.

The first method focuses only on vulnerabilities that have already been closed. It calculates the median and mean time it took to patch them. This is intuitive and simple, but it leaves out a potentially large and important portion of the data (the vulnerabilities that are still open). As a result, it tends to underestimate the true time it takes to remediate, especially if some vulnerabilities stay open much longer than others.

The second method tries to include both closed and open vulnerabilities by using the time they’ve been open so far. There are many options to approximate a time-to-patch for the open vulnerabilities, but for simplicity here we assumed they were (will be?) patched at the time of reporting, which we know isn’t true. But it does offer a way to factor in their existence.

The third method uses survival analysis. Specifically, we used the Kaplan-Meier estimator to model the likelihood that a vulnerability is still open at any given time. This method handles the open vulnerabilities properly: instead of pretending they’re patched, it treats them as “censored” data. The survival curve it produces drops over time, showing the proportion of vulnerabilities still open as days or weeks pass.

How Long Do Vulnerabilities Last?

In the current 6-month snapshot[^2], the closed-only time-to-patch has a median ~33 days and a mean ~35 days. On the surface that looks reasonable, but the Kaplan-Meier curve shows what those numbers hide: at 33 days, ~54% are still open; at 35 days, ~46% are still open. So even around the “typical” one-month mark, about half of issues remain unresolved.

We also computed observed-so-far statistics (treating open vulnerabilities as if they were patched at the end of the measurement window). In this window they happen to be almost the same (median ~33 days, mean ~35 days) because the ages of today’s open items cluster near one month. That coincidence can make averages look reassuring, but it’s incidental and unstable: if we shift the snapshot to just before the monthly patch push and these same statistics drop sharply (we’ve seen an observed median of ~19 days and observed a mean of ~15 days) without any change in the underlying process.

The survival curve avoids that trap, because it answers the question of “% still open after 30/60/90 days”, and offers visibility into the long tail that stays open well past a month.

Patch Everything Everywhere The Same Way?

Stratified survival analysis takes the idea of survival curves one step further. Instead of looking at all vulnerabilities together in one big pool, it separates them into groups (or “strata”) based on some meaningful characteristic. In our analysis, we have stratified vulnerabilities by severity, asset criticality, environment, cloud provider, team/division/organization. Each group gets its own survival curve, and here in the example graph we compare how quickly different vulnerability severities are remediated over time.

The benefit of this approach is that it exposes differences that would otherwise be hidden in the aggregate. If we only looked at the overall survival curve, we can only make conclusions about the remediation performance across the board. But stratification reveals if different teams, environments or severity issues are addressed faster than the rest, and in our case that the patch everything strategy is indeed consistent. This level of detail is important for making targeted improvements, helping us understand not just how long remediation takes in general, but if and where real bottlenecks exist.

How Fast Do Teams Act?

While the survival curve emphasizes how long vulnerabilities remain open, we can flip the perspective by using the cumulative distribution function (CDF) instead. The CDF focuses on how quickly vulnerabilities are patched, showing the proportion of vulnerabilities that have been remediated by a given point in time.

Our choice of plotting the CDF provides a clear picture of remediation speed, however it’s important to note that this version includes only vulnerabilities that were patched within the observed time window. Unlike the survival curve which we compute over a rolling 6-month cohort to capture full lifecycles, the CDF is computed month-over-month on items closed in that month[^3].

As such, it tells us how quickly teams remediate vulnerabilities once they do so, and it doesn’t reflect how long unresolved vulnerabilities remain open. For example, we see that 83.2% of the vulnerabilities closed in the current month were resolved within 30 days of the first detection. This highlights patching velocity for recent, successful patches but does not account for longer-standing vulnerabilities that remain open and are likely to have longer time-to-patch durations. Therefore, we use the CDF for understanding short-term response behavior, whereas the full lifecycle dynamics are given by a combination of CDF alongside survival analysis: the CDF describes how fast teams act once they patch, whereas the survival curve shows how long vulnerabilities truly last.

Difference Between Survival Analysis and Mean/Median

Wait, we said that survival analysis is better to analyze time to patch to avoid the impact of outliers. But in this example, mean/median and survival analysis provide similar results. What is the added value? The reason is simple: we don’t have outliers in our production environments since our patching process is fully automated and effective.

To demonstrate the impact on heterogeneous data, we’ll use an outdated example from a non-production environment that lacks automated patching.

ESQL query:

FROM qualys_vmdr.vulnerability_6months
  | WHERE elastic.environment == "my-outdated-non-production-environment"
  | WHERE qualys_vmdr.asset_host_detection.vulnerability.is_ignored == FALSE
  | EVAL vulnerability_age = DATE_DIFF("day", qualys_vmdr.asset_host_detection.vulnerability.first_found_datetime, qualys_vmdr.asset_host_detection.vulnerability.last_found_datetime)
  | STATS
      count=COUNT(*),
      count_closed_only=COUNT(*) WHERE qualys_vmdr.asset_host_detection.vulnerability.status == "Fixed",
      mean_observed_so_far=MEDIAN(vulnerability_age),
      mean_closed_only=MEDIAN(vulnerability_age) WHERE qualys_vmdr.asset_host_detection.vulnerability.status == "Fixed",
      median_observed_so_far=MEDIAN(vulnerability_age),
      median_closed_only=MEDIAN(vulnerability_age) WHERE qualys_vmdr.asset_host_detection.vulnerability.status == "Fixed"

| | Observed so far | Closed only | | :---- | :---- | :---- | | Count | 833 | 322 | | Mean | 178.7 (days) | 163.8 (days) | | Median | 61 (days) | 5 (days) | | Median survival | 527 (days) | N/A |

In this example, using mean and median yield very different results. Choosing a single representative metric can be challenging and potentially misleading. The survival analysis graph accurately represents our effectiveness in addressing vulnerabilities within this environment.

Final Thoughts

The benefits of using survival analysis come not only from more accurate measurement but also from the insights into the dynamics of patching behaviour, showing where bottlenecks occur, factors that affect patching velocity and whether it aligns with our SLO. From a technical integration perspective, the use of survival analysis as part of our operational workflows and reporting can be achieved with minimal additional changes to our current Elastic Stack setup: survival analysis can run on the same cadence as our patching cycle with the results being pushed back into Kibana for visualization. The definitive advantage is to pair our existing operational metrics with survival analysis for both long-term trends and short-term performance tracking.

Looking forward, we’re experimenting with additional new metrics like Arrival Rate, Burndown Rate, and Escape Rate that give us a way to move toward a more dynamic understanding of how vulnerabilities are really handled.

Arrival Rate is the measure of how quickly new vulnerabilities are entering the environment. Knowing that fifty new CVEs show up each month, for example, tells us what to expect in the workload before we even start measuring patches. So the arrival rate is a metric that does not necessarily inform about the backlog, but more about the pressure applied to the system.

Burndown Rate (trend) shows the other half of the equation: how quickly vulnerabilities are being remediated relative to how fast they arrive.

Escape Rate adds yet another dimension by focusing on vulnerabilities that slip past the points where they should have been contained. In our context, an escape is about CVEs that miss patching windows or exceed SLO thresholds. An elevated escape rate doesn’t just show that vulnerabilities exist but it also shows that the process designed to control them is failing, whether because patching cycles are too slow, automation processes are lacking, or compensating controls are not working as intended.

Together, the metrics create a better picture: arrival rate tells us how much new risk is being introduced; burndown trends show whether we are keeping pace with that pressure or being overwhelmed by it; escape rates expose where vulnerabilities persist despite planned controls.

[1]:An outlier in statistics is a data point that is very far from the central tendency (or far from the rest of the values in a dataset). For example, if most vulnerabilities are patched within 30 days, but one takes 600 days, that 600-day case is an outlier. Outliers can pull averages upward or downward in ways that don’t reflect the “typical” experience. In the patching context, these are the especially slow-to-patch vulnerabilities that sit open far longer than the norm. They may represent rare but important situations, like systems that can’t be easily updated, or patches that require extensive testing.

[2]: Note: The current 6-month dataset includes both all vulnerabilities that remain open at the end of the observation period (independent of how long ago they have been open /first seen) and all vulnerabilities that were closed during the 6-month window. Despite this mixed cohort approach, survival curves from prior observation windows show consistent trends, particularly in the early part of the curve. The shape and slope over the first 30–60 days have proven remarkably stable across snapshots, suggesting that metrics like median time-to-patch and early-stage remediation behavior are not artifacts of the short observation window. While long-term estimates (e.g. 90th percentile) remain incomplete in shorter snapshots, the conclusions drawn from these cohorts still reflect persistent and reliable patching dynamics.

[3]:We kept the CDF on a monthly cadence for operational reporting (throughput and SLO adherence for work completed during the current month), while the Kaplan-Meier uses a 6-month window to properly handle censoring and expose tail risk across the broader cohort.

Why DaC?

From validation and automation to enhancing cross-vendor content, there are several reasons previously discussed to use a DaC approach for rule management. Our team of detection engineers have been using the detection rules repo for testing and validation of our rules for some time. We now can provide the same testing and validation that we perform in a more accessible way. We aim to empower our users by adding straightforward CLI commands within our detection-rules repo, to help manage rules across the full rule lifecycle between version control systems (VCS) and Kibana. This allows users to move, unit test, and validate their rules in a single command easily using CI/CD pipelines.

Improving Process Maturity

Security organizations are facing the same bottomline, which is that we can’t rely on static out-of-the-box signatures. At its core, DaC is a methodology that applies software development practices to the creation and management of security detection rules, enabling automation, version control, testing, and collaboration in the development & deployment of security detections. Unit testing, peer review, and CI/CD enable software developers to be confident in their processes. These help catch errors and inefficiencies before they impact their customers. The same should be true in detection engineering. Fitting with this declaration here are some examples of some of the new features we are supporting. See our DaC Reference Guide for complete documentation.

Bulk Import and Export of Custom Rules

Custom rules can now be moved in bulk to and from Kibana using the kibana import-rules and kibana export-rules commands. Additionally, one can move them in bulk to and from TOML format to ndjson using the import-rules-to-repo and export-rules-from-repo commands. In addition to rules, these commands support moving exceptions and exception lists using the appropriate flag. The ndjson approach's benefit is that it allows engineers to manage and share a collection of rules in a single file (exported by the CLI or from Kibana), which is helpful when access is not permitted to the other Elastic environment. When moving rules using either of these methods, the rules pass through schema validation unless otherwise specified to ensure that the rules contain the appropriate data fields. For more information on these commands, please see the CLI.md file in detection rules.

Configurable Unit Tests, Validation, and Schemas

With this new feature, we've now included the ability to configure the behavior of unit tests and schema validation using configuration files. In these files, you can now set specific tests to be bypassed, specify only specific tests to run, and likewise with schema validation against specific rules. You can run this validation and unit tests at any time by running make test. Furthermore, you can now bring your schema (JSON file) to our validation process. You can also specify which schemas to use against which target versions of your Stack. For example, if you have custom schemas that only apply to rules in 8.14 while you have a different schema that should be used for 8.10, this can now be managed via a configuration file. For more information, please see our example configuration file or use our custom-rules setup-config command from the detection rules repo to generate an example for you.

Custom Version Control

We now are providing the ability to manage custom rules using the same version lock logic that Elastic’s internal team uses to manage our rules for release. This is done through a version lock file that checks the hash of the rule contents and determines whether or not they have changed. Additionally, we are providing a configuration option to disable this version lock file to allow users to use an alternative means of version control such as using a git repo directly. For more information please see the version control section of our documentation. Note that you can still rely on Kibana’s versioning fields.

Having these systems in place provides auditable evidence for maintaining security rules. Adopting some or all of these best practices can dramatically improve quality in maintaining and developing security rules.

Broader Adoption of Automation

While quality is critical, security teams and organizations face growing rule sets to respond to an ever-expanding threat landscape. As such, it is just as crucial to reduce the strain on security analysts by providing rapid deployment and execution. For our repo, we have a single-stop shop where you can set your configuration, focus on rule development, and let the automation handle the rest.

Lowering the Barrier to Entry

To start, simply clone or fork our detection rules repo, run custom-rules setup-config to generate an initial config, and import your rules. From here, you now have unit tests and validation ready for use. If you are using GitLab, you can quickly create CI/CD to push the latest rules to Kibana and run these tests. Here is an example of what that could look like:

High Flexibility

While we use GitHub CI/CD for managing our release actions, by no means are we prescribing that this is the only way to manage detection rules. Our CLI commands have no dependencies outside of their python requirements. Perhaps you have already started implementing some DaC practices, and you may be looking to take advantage of the Python libraries we provide. Whatever the case may be, we want to encourage you to try adopting DaC principles in your workflows and we would like to provide flexible tooling to accomplish these goals.

To illustrate an example, let’s say we have an organization that is already managing their own rules with a VCS and has built automation to move rules back and forth from deployment environments. However, they would like to augment these movements with testing based on telemetry which they are collecting and storing in a database. Our DaC features already provide custom unit testing classes that can run per rule. Realizing this goal may be as simple as forking the detection rules repo and writing a single unit test. The figure below shows an example of what this could look like.

This new unit test could utilize our unit test classes and rule loading to provide scaffolding to load rules from a file or Kibana instance. Next, one could create different integration tests against each rule ID to see if they pass the organization's desired results (e.g. does the rule identify the correct behaviors). If they do, the CI/CD tooling can proceed as originally planned. If they fail, one can use DaC tooling to move those rules to a “needs tuning” folder and/or upload those rules to a “Tuning” Kibana space. In this way, one could use a hybrid of our tooling and one's own tooling to keep an up to date Kibana space (or VCS controlled folder) of what rules require updates. As updates are made and issues addressed, they could also be continually synchronized across spaces, leading to a more cohesive environment.

This is just one idea of how one can take advantage of our new DaC features in your environment. In practice, there are a vast number of different ways they can be utilized.

In Practice

Now, let’s take a look at how we can tie these new features together into a cohesive DaC strategy. As a reminder, this is not prescriptive. Rather, this should be thought of as an optional, introductory strategy that can be built on to achieve your DaC goals.

Establishing a DaC Baseline

In detection engineering, we would like collaboration to be a default rather than an exception. Detection Rules is a public repo precisely with this precept in mind. Now, it can become a basis for the community and teammates to not only collaborate with us, but also with each other. Let’s use the chart below as an example for what this could look like.

Reading from left to right, we have initial planning and prioritization and the subsequent threat research that drives the detection engineering. This process will look quite different for each user so we are not going to spend much time describing it here. However, the outcome will largely be similar, the creation of new detection rules. These could be in various forms like Sigma rules (more in a later blog), Elastic TOML rule files, or creating the rules directly in Kibana. Regardless of format, once created these rules need to be staged. This would either occur in Kibana, your VCS, or both. From a DaC perspective, the goal is to sync the rules such that the process/automation are aware of these new additions. Furthermore, this provides the opportunity for peer review of these additions — the first stage of collaboration.

This will likely happen in your version control system; for instance, in GitHub one could use a PR with required approvals before merging back into a main branch that acts as the authoritative source of reviewed rules. The next step is for testing and validation, this step could additionally occur before peer review and this is largely up to the desired implementation.

In addition to any other internal release processes, by adhering to this workflow, we can reduce the risk of malformed rules and errant mistakes from reaching both our customers and the community. Additionally, having the evidence artifacts, passing unit tests, schema validation, etc., inspires confidence and provides control for each user to choose what risks they are willing to accept.

Once deployed and distributed, rule performance can be monitored from Kibana. Updates to these rules can be made either directly from Kibana or through the VCS. This will largely be dependent on the implementation specifics, but in either case, these can be treated very similarly to new rules and pass through the same peer review, testing, and validation processes.

As shown in the figure above, this can provide a unified method for handling rule updates whether from the community, customers, or from internal feedback. Since the rules ultimately exist as version-controlled files, there is a dedicated format source of truth to merge and test against.

In addition to the process quality improvements, having authoritative known states can empower additional automation. As an example, different customers may require different testing or perhaps different data sources. Instead of having to parse the rules manually, we provide a unified configuration experience where users can simply bring their own config and schemas and be confident that their specific requirements are met. All of this can be managed automatically via CI/CD. With a fully automated DaC setup, one can take advantage of this system entirely from VCS and Kibana without needing to write additional code. Let’s take a look at an example of what this could look like.

Example

For this example, we are going to be acting as an organization that has 2 Kibana spaces they want to manage via DaC. The first is a development space that rule authors will be using to write detection rules (so let’s assume there are some preexisting rules already available). There will also be some developers that are writing detection rules directly in TOML file formats and adding them to our VCS, so we will need to manage synchronization of these. Additionally, this organization wants to enforce unit testing and schema validation with the option for peer review on rules that will be deployed to a production space in the same Kibana instance. Finally, the organization wants all of this to occur in an automated manner with no requirement to either clone detection rules locally or write rules outside of a GUI.

In order to accomplish this we will need to make use of a few of the new DaC features in detection rules and write some simple CI/CD workflows. In this example we are going to be using GitHub. Additionally, you can find a video walkthrough of this example here. As a note, if you wish to follow along you will need to fork the detection rules repo and create an initial configuration using our custom-rules setup-config command. Also for general step by step instructions on how to use the DAC features, see this quickstart guide, which has several example commands.

Development Space Rule Synchronization

First we are going to synchronize from Kibana -> GitHub (VCS). To do this we will be using the kibana import-rules and kibana export-rules detection rules commands. Additionally, in order to keep the rule versions synchronized we will be using the locked versions file as we are wanting both our VCS and Kibana to be able to overwrite each other with the latest versions. This is not required for this setup, either Kibana or GitHub (VCS) could be used authoritatively instead of the locked versions file. But we will be using it for convenience.

The first step is for us to make a manual dispatch trigger that will pull the latest rules from Kibana upon request. In our setup this could be done automatically; however, we want to give rule authors control for when they want to move their rules to the VCS as the development space in Kibana is actively used for development and the presence of a new rule does not necessarily mean the rule is ready for VCS. The manual dispatch section could look like the following example:

With this trigger in place, we now can write 4 additional jobs that will trigger on this workflow dispatch.

1. Pull the rules from the desired Kibana space.
2. Update the version lock file.
3. Create a PR request for review to merge into the main branch in GitHub.
4. Set the correct target for the PR.

These jobs could look like this also from the same example:

Now, once we run this workflow we should expect to see a PR open with the new rules from the Kibana Dev space. We also need to synchronize rules from GitHub (VCS) to Kibana. For this we will need to create a triggers on pull request:

Next, we just need to create a job that uses the kibana import-rules command to push the rule files from the given PR to Kibana. See the second example for the complete workflow file.

With these two workflows complete we now have synchronization of rules between GitHub and the Kibana Dev space.

Production Space Deployment

With the Dev space synchronized, now we need to handle the prod space. As a reminder, for this we need to enforce unit testing, schema validation, available peer review for PRs to main, and on merge to main auto push to the prod space. To accomplish this we will need two workflow files. The first will run unit tests on all pull requests and pushes to versioned branches. The second will push the latest rules merged to main to the prod space in Kibana.

The first workflow file is very simple. It has an on push and pull_request trigger and has the core job of running the test command shown below. See this example for the full workflow.

With this test command we are performing unit tests and schema validation with the parameters specified in our config files on all of our custom rules. Now we just need the workflow to push the latest rules to the prod space. The core of this workflow is the kibana import-rulescommand again just using the prod space as the destination. However, there are a number of additional options provided to this workflow that are not necessary but nice to have in this example, such as options to overwrite and update exceptions/exception lists as well as rules. The core job is shown below. Please see this example for the full workflow file.

And there we have it, with those 4 workflow files we have a synchronized development space with rules passing through unit testing and schema validation. We have the option for peer review through the use of pull requests, which can be made as requirements in GitHub before allowing for merges to main. On merge to main in GitHub we also have an automated push to the Kibana prod space, establishing our baseline of rules that have passed our organizations requirements and are ready for use. All of this was accomplished without writing additional Python code, just by using our new DaC features in GitHub workflows.

Conclusion

Now that we’ve reached this milestone, you may be wondering what’s next? We’re planning to spend the next few cycles continuing to test edge cases and incorporating feedback from the community as part of our business-as-usual sprints. We also have a backlog of features request considerations so if you want to voice your opinion, checkout the issues titled [FR][DAC] Consideration: or open a similar new issue if it’s not already recorded. This will help us to prioritize the most important features for the community.

We’re always interested in hearing use cases and workflows like these, so as always, reach out to us via GitHub issues, chat with us in our security-rules-dac slack channel, and ask questions in our Discuss forums!

When prioritizing detection engineering efforts, it's essential to understand the most prevalent tactics, techniques, and procedures (TTPs) observed in the wild. This knowledge helps defenders make informed decisions about the most effective strategies to implement - especially where to focus engineering efforts and finite resources.

To highlight these prevalent TTPs, we analyzed over 100,000 Windows malware samples extracted over several months from one of our dynamic malware analysis tools, Detonate. To generate this data and alerts, we leveraged Elastic Defend behavior (mapped to MITRE ATT&CK) and memory threat detection rules. It should be noted that this dataset is not exhaustive, it may not represent the entire spectrum of malware behavior, and specifically does not include long-term or interactive activity.

Below an ES|QL query to summarize our dataset by file type:

Tactics

Beginning with tactics, we aggregated the alerts generated by this corpus of malware samples and organized them according to the counts of process.entity_id and alerts. As depicted in the image below, the most frequent tactics included defense evasion, privilege escalation, execution, and persistence. Certain tactics commonly linked with post-exploitation activities, such as lateral movement, provided an anticipated lower prevalence because these actions are commonly manually driven by the threat actor after the initial implant is established vs. being automated by the malware in our dataset.

In the following sections, we will delve into each tactic and the techniques and sub-techniques of each that exerted the most influence.

Defense Evasion

Defense Evasion involves methods employed by adversaries to avoid detection by security teams or capabilities. The foremost tactic detected was defense evasion, triggering 189 distinct detection rules (nearly 40% of our current Windows rules). The primary techniques noted are associated with code injection, defense tampering, masquerading, and system binary proxy execution.

When we pivot by sub-techniques, it becomes evident that certain advanced techniques such as DLL side-loading and Parent PID Spoofing have become increasingly popular, even among non-targeted malwares. Both are frequently linked with code injection and masquerading.

Furthermore, system binary proxies Rundll32 and Regsvr32 remain highly abused, with a notable rise in the utilization of malicious MSI installers for malware delivery. The practice of masquerading as legitimate system binaries, whether through renaming or process hollowing, remains prevalent as well, serving as a means to evade user suspicion.

Tampering with Windows Defender stands out as the most frequently observed defense evasion tactic, emphasizing the importance for defenders to acknowledge that adversaries will attempt to obscure their activities.

Process Injection is prevalent across various malware families, whether they target legitimate system binaries remotely to blend in or employ self-injection (sometimes paired with DLL side-loading through a trusted binary). Furthermore, there is a noticeable uptick in the use of NTDLL unhooking to bypass security solutions reliant on user-mode APIs monitoring (Elastic Defend is not impacted).

From our shellcode alerts we can clearly see that self-injection is more prevalent than remote:

Almost 50 unique vendors’ binaries abused for DLL side-loading, of which Microsoft is the top choice:

Defense evasion comprises various techniques and sub-techniques necessitating comprehensive coverage due to their frequent occurrence. For instance, apart from memory threat protection, half of our rules are specifically tailored to address this tactic.

Privilege Escalation

This tactic consists of techniques that adversaries use to gain greater permissions on a system or network. The most commonly used techniques relate to access token manipulation, execution through privileged system services, and bypassing User Account Control.

The most frequently observed sub-technique involved impersonation as the Trusted Installer service, which aligns closely with defense evasion and often precedes attempts to manipulate system-protected resources.

Concerning User Account Control bypass, the primary method we observed was elevation by mimicking trusted directories, which is also related to DLL side-loading. Additionally, other methods like elevation via extended startupinfo (elevated parent PID spoofing) are increasingly prevalent among commodity malware.

As evident from the list below, there's a notable rise in the use of vulnerable drivers (BYOVD) to manipulate protected objects and acquire kernel mode execution privileges.

Below, you'll find a list of the most commonly exploited drivers triggered by our YARA rules:

Execution

Execution encompasses methods that lead to running adversary-controlled code on a local or remote system. These techniques are frequently combined with methods from other tactics to accomplish broader objectives, such as network reconnaissance or data theft.

The most common techniques observed here involved Windows command and scripting languages, with the proxying of execution via the Windows Management Instrumentation (WMI) interface closely trailing behind.

Powershell remains a preferred scripting language for malware execution chains, followed by Javascript and VBscript. Multi-stage malware delivery routinely involves a combination of two or more scripting languages.

Here is a list of the most frequently triggered endpoint behavior detections for this tactic:

Windows' default scripting languages remain the top preference for malware execution. However, there has been a slight uptick in the shift towards using other third-party scripting interpreters like Python, AutoIt, Java and Lua.

Persistence

It's common for malware to install itself on an infected host. No surprises here: the most frequently observed persistence methods include scheduled tasks, the run key and startup folder, and Windows services (which typically require administrator privileges).

The top three persistence sub-techniques depicted in the list below are also commonly encountered in regular software installations. Therefore, it's necessary to dissect them into multiple detections with additional suspicious signals to reduce false positives and enhance precision.

Initial Access

Considering the dataset's composition, initial access was associated with primarily macro-enabled documents and Windows shortcut objects. Although a significant portion of the detonated samples also involved other formats, such as ISO/VHD containers with MSI installers extensively utilized for delivery, their genuine malicious behavior typically manifests in areas such as defense evasion and persistence.

The most frequently abused Microsoft-signed binaries originating from malicious Microsoft Office documents align closely with execution and defense evasion tactics, command and scripting interpreters, and system binary proxy execution.

Here is a list of the most frequently triggered detections for initial access, regarding phishing attachments:

Credential Access

Credential access in malware is frequently linked to information stealers. The most targeted credentials are typically associated with Windows Credential Manager and browser password stores. Domain and system-protected credentials require elevated privileges and are more likely a feature of a subsequent stage.

Below a breakdown of the endpoint behavior detections that triggered the most on credentials access:

The majority of credentials access behaviors resemble typical file access events. Therefore, it's essential to correlate and enrich them with additional signals to reduce false positives and enhance comprehension.

Conclusion

Even though this small dataset of about 100,000 malware samples represents only a fraction of the possible malware in the wild right now, we can still derive important insights from it about the most common TTPs using our behavioral detections. Those insights help us make decisions about detection engineering priorities, and defenders should make that part of their strategies.

This post will be updated with each new article in the series, published monthly:

• Topic: Defense Evasion
• Topic: Credential Access

In April, we published an updated version of the Global Threat Report Spring Edition which included new insights online and interactive.

Threat intelligence resources like the 2022 Elastic Global Threat Report are critical to helping my team evaluate our organization’s visibility, capabilities, and expertise in identifying and preventing cybersecurity threats. It helps us answer questions such as:

• How is our environment impacted by the current and emerging threats identified in this report?
• Does this new information change our risk profile and impact our risk analysis?
• What adjustments do we need to make to our controls?
• Are we lacking visibility in any areas?
• Do we have the right detections in place?
• How might these insights affect my team’s workflows?

Elastic’s threat report provides a real-world roadmap to help my team make the connections necessary to strengthen our security posture. It influences our overall program roadmaps, helping us prioritize where we focus our resources, including adjusting our defenses, testing incident response plans, and identifying updates for our security operations center (SOC). And perhaps most importantly, the report underscores our belief that providing open, transparent, and accessible security for all organizations is key to defending ourselves against cybersecurity threats.

Check your cloud security, and then check it again

Threat reports often reinforce many of the existing trends and phenomena we see within security, but they can also reveal some unexpected insights. While the cloud enables organizations to operate faster and at scale, it also creates security gaps that leave room for potential attacks as threat actors continue shifting their focus to the cloud.

The Elastic Global Threat Report revealed that nearly 40% of all malware infections are on Linux endpoints, further emphasizing the need for better cloud security. With nine out of the top ten public clouds running on Linux, this statistic is an important reminder to organizations not to rely solely on their cloud provider’s standard configurations for security.

The findings further revealed that approximately 57% of cloud security events were attributed to AWS, followed by 22% for Google Cloud and 21% for Azure, and that 1 out of every 3 (33%) cloud alerts was related to credential access across all cloud service providers.

While the data points to an increased need for organizations to properly secure their cloud environments, it also reinforces our belief that cloud security posture management (CSPM) needs to evolve similarly to endpoint security.

Initially, endpoint security relied on simple antivirus, which was only as good as its antivirus signatures. To avoid increasingly sophisticated malware and threats, endpoint security evolved by employing more advanced technologies like next-gen antivirus with machine learning and artificial intelligence. CSPM is currently facing a similar situation. Right now, we are closer to the bottom of the cloud security learning curve than the top, and our technologies and strategies must continue to evolve to manage new and emerging threats.

The Elastic Global Threat Report demonstrates that native tools and traditional security tactics are ineffective when implemented in cloud environments and offers recommendations for how organizations can adapt to the evolving threat landscape.

Get the basics right first

Security leaders and teams should leverage insights from this report to inform their priorities and adjust their workflows accordingly.

The findings clearly show why focusing on and improving basic security hygiene is so crucial to improved security outcomes. Too often, an organization’s environment is compromised by something as simple as a weak password or failure to update default configurations. Prioritizing security fundamentals — identity and access management, patching, threat modeling, password awareness, and multi-factor authentication — is a simple yet effective way for security teams to prevent and protect against potential threats.

Develop security in the open

Organizations should consider adopting an open approach to security. For example, Elastic’s threat report links to our recent publication of protection artifacts, which transparently shares endpoint behavioral logic that we develop at Elastic to identify adversary tradecraft and make it freely available to our community.

The report also highlights how Elastic Security’s prebuilt detection rules map to the MITRE ATT&CK matrix for each cloud service provider. As an adopter of the MITRE framework since its inception, Elastic understands the importance of mapping detection rules to an industry standard. For my team, this helps us have deeper insights into the breadth and depth of our security posture.

Providing open detection rules, open artifacts, and open code enables organizations to focus on addressing gaps in their security technology stack and developing risk profiles for new and emerging threats. Without openness and transparency in security, organizations are putting themselves at greater risk of tomorrow’s cybersecurity threats.

Download the 2022 Elastic Global Threat Report.

Our hope is that threat researchers and the security industry as a whole will use this report to prepare for the next set of threats and campaigns. At Elastic, we are ensuring that our customers using the Elastic Security solution are best protected from these types of threats, including endpoint and cloud capabilities for automated protection.

This year, our report included six key forecasts and recommendations for strategists and practitioners to stay better informed of potential directions that threat actors may focus on in 2023 and beyond. Below, we summarize the first three of our forecasts. Further details on these and our other recommendations are available in our full, downloadable report for 2022:

Adversaries will continue to abuse built-in binary proxies to evade security instrumentation. 

The use of proven adversarial tactics remains a key area of focus for observed threat groups, and this year remains no different. Hostile groups leverage legitimate, native system binaries to load malicious software — evading many detection strategies used by modern enterprises.

With this continued focus, Elastic Security has enhanced our deep visibility and pre-built protections, including numerous rules and signatures, alongside ML models to detect these threats faster and more effectively.

LNK and ISO payloads will replace more conventional script and document payloads.

Adversarial behavior focuses on finding easier, more efficient pathways for attack — and this year, it is no different. System defaults have forced threat groups to pivot their strategies to leverage LNK and ISO payloads over familiar scripts and documents we have observed in the past.

LNK and ISO files are often used to smuggle malicious software into enterprises because most security technologies don't inspect them. Elastic Security has focused on building instrumentation into our products and platform, allowing us to determine the exact mechanisms used to better build a defense against these malicious acts.

Valid IAM accounts will continue to be a target for adversaries.

The early stages of many attacks focus on credential theft in all forms; however, IAM and administrative credentials often remain the area of focus for many adversarial groups looking to evade detection and avoid exploitation of services. 

Understanding standard account actions and user behaviors exhibited in environments is critical to defending them, and ensuring we have a comprehensive library of detections alongside integration capabilities within the stack has provided a strong foundation in detecting threats earlier.

This is just a small introduction to the findings found in the report. Far greater detail, recommendations, and source data are available in the 2022 Elastic Global Threat Report.

Those looking to learn more about the threats we observed and the mechanisms adversarial groups leveraged over the last year can read far more detailed information in our full report — alongside many recommendations and findings we have leveraged to help shape the strategy used within the Elastic Security solution, and future feature roadmap.

Feel free to check out the full 2022 Elastic Global Threat Report here.

In 8.3, our Elastic Stack Machine Learning team introduced a way to import third party Natural Language Processing (NLP) models into Elastic. As security researchers, we HAD to try it out on a security dataset. So we decided to build a model to identify malicious command lines by fine-tuning a pre-existing model available on the Hugging Face model hub.

Upon finding that the fine-tuned model was performing (surprisingly!) well, we wanted to see if it could replace or be combined with our previous tree-based model for detecting Living off the Land (LotL) attacks. But first, we had to make sure that the throughput and latency of this new model were reasonable enough for real-time inference. This resulted in a series of experiments, the results of which we will detail in this blog.

In this blog, we will briefly talk about how we fine-tuned a transformer model meant for a masked language modeling (MLM) task, to make it suitable for a classification task. We will also look at how to import custom models into Elastic. Finally, we’ll dive into all the experiments we did around using the fine-tuned model for real-time inference.

NLP for command line classification

Before you start building NLP models, it is important to understand whether an NLP model is even suitable for the task at hand. In our case, we wanted to classify command lines as being malicious or benign. Command lines are a set of commands provided by a user via the computer terminal. An example command line is as follows:

**move test.txt C:\**

The above command moves the file test.txt to the root of the **C:** directory.

Arguments in command lines are related in the way that the co-occurrence of certain values can be indicative of malicious activity. NLP models are worth exploring here since these models are designed to understand and interpret relationships in natural (human) language, and since command lines often use some natural language.

Fine-tuning a Hugging Face model

Hugging Face is a data science platform that provides tools for machine learning (ML) enthusiasts to build, train, and deploy ML models using open source code and technologies. Its model hub has a wealth of models, trained for a variety of NLP tasks. You can either use these pre-trained models as-is to make predictions on your data, or fine-tune the models on datasets specific to your NLP tasks.

The first step in fine-tuning is to instantiate a model with the model configuration and pre-trained weights of a specific model. Random weights are assigned to any task-specific layers that might not be present in the base model. Once initialized, the model can be trained to learn the weights of the task-specific layers, thus fine-tuning it for your task. Hugging Face has a method called from_pretrained that allows you to instantiate a model from a pre-trained model configuration.

For our command line classification model, we created a RoBERTa model instance with encoder weights copied from the roberta-base model, and a randomly initialized sequence classification head on top of the encoder:

model = RobertaForSequenceClassification.from_pretrained('roberta-base', num_labels=2)

Hugging Face comes equipped with a Tokenizers library consisting of some of today's most used tokenizers. For our model, we used the RobertaTokenizer which uses Byte Pair Encoding (BPE) to create tokens. This tokenization scheme is well-suited for data belonging to a different domain (command lines) from that of the tokenization corpus (English text). A code snippet of how we tokenized our dataset using RobertaTokenizer can be found here. We then used Hugging Face's Trainer API to train the model, a code snippet of which can be found here.

ML models do not understand raw text. Before using text data as inputs to a model, it needs to be converted into numbers. Tokenizers group large pieces of text into smaller semantically useful units, such as (but not limited to) words, characters, or subwords — called token —, which can, in turn, be converted into numbers using different encoding techniques.

• Check out this video (2:57 onwards) to review additional pre-processing steps that might be needed after tokenization based on your dataset.
• A complete tutorial on how to fine-tune pre-trained Hugging Face models can be found here.

Importing custom models into Elastic

Once you have a trained model that you are happy with, it's time to import it into Elastic. This is done using Eland, a Python client and toolkit for machine learning in Elasticsearch. A code snippet of how we imported our model into Elastic using Eland can be found here.
You can verify that the model has been imported successfully by navigating to Model Management \> Trained Models via the Machine Learning UI in Kibana:

Using the Transformer model for inference — a series of experiments

We ran a series of experiments to evaluate whether or not our Transformer model could be used for real-time inference. For the experiments, we used a dataset consisting of ~66k command lines.

Our first inference run with our fine-tuned RoBERTa model took ~4 hours on the test dataset. At the outset, this is much slower than the tree-based model that we were trying to beat at ~3 minutes for the entire dataset. It was clear that we needed to improve the throughput and latency of the PyTorch model to make it suitable for real-time inference, so we performed several experiments:

Using multiple nodes and threads

The latency numbers above were observed when the models were running on a single thread on a single node. If you have multiple Machine Learning (ML) nodes associated with your Elastic deployment, you can run inference on multiple nodes, and also on multiple threads on each node. This can significantly improve the throughput and latency of your models.

You can change these parameters while starting the trained model deployment via the API:

**POST \_ml/trained\_models/\\<model\_id\\>/deployment/\_start?number\_of\_allocations=2&threa ds\_per\_allocation=4**

number_of_allocations allows you to set the total number of allocations of a model across machine learning nodes and can be used to tune model throughput. threads_per_allocation allows you to set the number of threads used by each model allocation during inference and can be used to tune model latency. Refer to the API documentation for best practices around setting these parameters.

In our case, we set the number_of_allocations to 2 , as our cluster had two ML nodes and threads_per_allocation to 4 , as each node had four allocated processors.

Running inference using these settings resulted in a 2.7x speedup on the original inference time.

Dynamic quantization

Quantizing is one of the most effective ways of improving model compute cost, while also reducing model size. The idea here is to use a reduced precision integer representation for the weights and/or activations. While there are a number of ways to trade off model accuracy for increased throughput during model development, dynamic quantization helps achieve a similar trade-off after the fact, thus saving on time and resources spent on iterating over the model training.

Eland provides a way to dynamically quantize your model before importing it into Elastic. To do this, simply pass in quantize=True as an argument while creating the TransformerModel object (refer to the code snippet for importing models) as follows:

**# Load the custom model**
**tm = TransformerModel("model", "text\_classification", quantize=True)**

In the case of our command line classification model, we observed the model size drop from 499 MB to 242 MB upon dynamic quantization. Running inference on our test dataset using this model resulted in a 1.6x speedup on the original inference time, for a slight drop in model sensitivity (exact numbers in the following section) .

Knowledge Distillation

Knowledge Distillation is a way to achieve model compression by transferring knowledge from a large (teacher) model to a smaller (student) one while maintaining validity. At a high level, this is done by using the outputs from the teacher model at every layer, to backpropagate error through the student model. This way, the student model learns to replicate the behavior of the teacher model. Model compression is achieved by reducing the number of parameters, which is directly related to the latency of the model.

To study the effect of knowledge distillation on the performance of our model, we fine-tuned a distilroberta-base model (following the same procedure described in the fine-tuning section) for our command line classification task and imported it into Elastic. distilroberta-base has 82 million parameters, compared to its teacher model, roberta-base , which has 125 million parameters. The model size of the fine-tuned DistilRoBERTa model turned out to be 329 MB, down from 499 MB for the RoBERTa model.

Upon running inference with this model, we observed a 1.5x speedup on the original inference time and slightly better model sensitivity (exact numbers in the following section) than the fine-tuned roberta-base model.

Dynamic quantization and knowledge distillation

We observed that dynamic quantization and model distillation both resulted in significant speedups on the original inference time. So, our final experiment involved running inference with a quantized version of the fine-tuned DistilRoBERTa model.

We found that this resulted in a 2.6x speedup on the original inference time, and slightly better model sensitivity (exact numbers in the following section). We also observed the model size drop from 329 MB to 199 MB after quantization.

Bringing it all together

Based on our experiments, dynamic quantization and model distillation resulted in significant inference speedups. Combining these improvements with distributed and parallel computing, we were further able to reduce the total inference time on our test set from four hours to 35 minutes. However, even our fastest transformer model was still several magnitudes slower than the tree-based model, despite using significantly more CPU resources.

The Machine Learning team here at Elastic is introducing an inference caching mechanism in version 8.4 of the Elastic Stack, to save time spent on performing inference on repeat samples. These are a common occurrence in real-world environments, especially when it comes to Security. With this optimization in place, we are optimistic that we will be able to use transformer models alongside tree-based models in the future.

A comparison of the sensitivity (true positive rate) and specificity (true negative rate) of our tree-based and transformer models shows that an ensemble of the two could potentially result in a more performant model:

| | | | | | | ----------------------- | --------------- | ----------------------- | --------------- | ----------------------- | | Model | Sensitivity (%) | False Negative Rate (%) | Specificity (%) | False Positive Rate (%) | | Tree-based | 99.53 | 0.47 | 99.99 | 0.01 | | RoBERTa | 99.57 | 0.43 | 97.76 | 2.24 | | RoBERTa quantized | 99.56 | 0.44 | 97.64 | 2.36 | | DistilRoBERTa | 99.68 | 0.32 | 98.66 | 1.34 | | DistilRoBERTa quantized | 99.69 | 0.31 | 98.71 | 1.29 |

As seen above, the tree-based model is better suited for classifying benign data while the transformer model does better on malicious samples, so a weighted average or voting ensemble could work well to reduce the total error by averaging the predictions from both the models.

What's next

We plan to cover our findings from inference caching and model ensembling in a follow-up blog. Stay tuned!

In the meanwhile, we’d love to hear about models you're building for inference in Elastic. If you'd like to share what you're doing or run into any issues during the process, please reach out to us on our community Slack channeland discussion forums. Happy experimenting!