Most vendors cannot answer it cleanly. And the answer matters more than almost anything else on the RFP.

What open actually means

The industry has gotten comfortable treating “open” as a checkbox. Support an open schema, publish an API, sponsor a standard, done. But a standard tells you how data is shaped. It tells you nothing about whether you can actually get it. Open data means three things, and you need all three:

It is yours, at no additional cost. You generated this telemetry. You paid to collect it, and you paid to store it. If your vendor charges you a second time to access it, that is not a data feature. That is a toll booth on your own driveway.

It is all of your data. Not the alerts. Not a normalized summary. Not the tables the vendor decided are ready. The full-fidelity record because you cannot predict today which field matters in next year’s investigation.

It is real time. This one used to be a nice-to-have. Not anymore. CrowdStrike’s own 2026 Global Threat Report clocked the average eCrime breakout time at 29 minutes, the fastest at 27 seconds, and one intrusion where exfiltration started four minutes after initial access. Attackers are moving faster too, using AI to shorten the gap between breaking in and doing real damage. If your telemetry arrives in batches, minutes apart, the attack can be over before your data shows up. You would not accept a smoke detector that checks for fire twice an hour.

When a vendor fails any of these three, they are not protecting your data. They are building an artificial moat with it. Ingestion is always frictionless. Egress is licensed, delayed, or degraded. That asymmetry is not an accident of engineering. It is the business model, and the industry has a name for it: lock-in.

What the documentation actually says

Don’t take our word for it. Every claim in this table links to the vendor’s own documentation. Read it yourself, and hold us to the same bar.

Vendor

How you get your data out

Cost to access your data

Freshness

Fidelity

Openness

CrowdStrike

Falcon Data Replicator: batched file export to object storage

Licensed add-on

Bulk batches, not a live stream; deleted after 7 days for CrowdStrike managed-buckets

Raw telemetry is only available via FDR; the event stream API sends detections, not telemetry

Restricted

Palo Alto Networks

XSIAM Event Forwarding: batch files to a Palo Alto-managed bucket

Two paid Event Forwarding add-ons: GB and Endpoint

Batch; up to 2 hours to appear; kept 14 days

Endpoint and log data split across the two add-ons

Restricted

Microsoft

Defender streaming API; Sentinel data export

Streaming: pay Azure infra; Sentinel export billed per GB

Real-time stream (via Azure Event Hubs)

Generally-available fields only; Azure destinations only

Partially open

Google

Bulk export to your storage, or a continuous BigQuery feed

Export capped; BigQuery billed by query

Live feed 5-10 min behind (top tier only); raw export is point-in-time

Live feed is normalized only; no continuous raw path

Limited

Splunk 

Search/export REST API

Included

On-demand query via export data API; data available near-real-time

Full events returned as structured JSON

Open

Elastic

REST APIs including Search, Point in time and ES|QL

No export license or per-query export fee; Elastic Cloud applies ordinary cloud data-transfer metering, not an egress toll

On-demand query via API; data available to query as soon as it is indexed

Full raw and parsed events, returned as structured JSON

Open

This table reflects each vendor's public documentation as of September 2026.

A note on the ratings, because we want them to be defensible, not convenient. Open means all three tests pass: no added cost, full fidelity, and no batch window between the moment data arrives and the moment you can use it. Data you can query the moment it lands clears that bar; a scheduled batch measured in minutes or hours does not. Partially open means the vendor genuinely tries but with real constraints; credit to Microsoft for a true streaming path, even if it covers only generally-available fields and only lands in Azure. Limited means the data comes back, but late or in a form only the vendor can read. Restricted means access to your own telemetry is a paid product, a delayed batch, or both.

And since Elastic is on this list too, as an endpoint agent and a SIEM, hold us to the same bar. Our rating is about live access: APIs that return JSON, with no license between you and your own telemetry. On Elastic Cloud you pay ordinary data-transfer rates like any cloud service. What you never pay is a license to reach data that was already yours.

If any vendor believes we have mischaracterized their documentation, I genuinely want to hear it, and we will correct it.

So what does this actually cost you?

Here is what that comparison means when it really counts, in the middle of an incident. Detection you cannot act on in time is not detection. When your telemetry arrives in scheduled batches, a window opens between the alert and the evidence. Sometimes minutes, sometimes longer, and in that window the attack is live while your data is not yet in front of you. You paid to collect that telemetry. You paid to store it. And at the one moment it matters, it is still in transit. Or worse, it is not there at all because exporting it requires a separate license. That’s the difference between stopping an intrusion and reading about it afterward.

You do not create that gap during an incident. You inherit it at purchase. You already run a strong endpoint agent. It works, and you trust it. Now that same vendor offers to be your SIEM as well. One console, one relationship, one invoice. There is nothing wrong with one vendor doing both. We do both at Elastic. The question is what it costs you to change your mind. So look closely at what it takes to move that endpoint telemetry somewhere other than their own platform. Every vendor makes it easy to get data in. Getting it back out is something you buy, then wait for.

It is rarely just one handoff. Most security environments are heterogeneous by design, with each layer chosen because it is good at its own job. Modern attacks move across all of them, from a stolen identity to a cloud workload to an endpoint, and you only see the full chain if the data from each can meet in one place, quickly and without a toll. When a vendor makes its telemetry expensive or slow to share, it is not just charging you. It’s fragmenting the picture, and a fragmented picture is exactly where intrusions hide. The all-in-one pitch offers to solve that. But the fragmentation was manufactured: vendors made their data hard to move, then sold you the one console where it comes back together. So before convenience makes the decision for you, ask three questions, and make the vendor answer them in writing:

• Can I get all of my telemetry out, continuously, without buying a separate license to do it?

• How long, exactly, from the moment an event happens to the moment it is usable in a system I chose?

• On the day I add a different analytics engine, a data lake, or a new AI model, what breaks?

If the honest answers are “extra cost,” “in batches,” and “quite a lot,” then you are not buying a SIEM. You are renting access to your own data. The point is not that these layers must stay separate. Plenty of teams consolidate for good reasons, and we sell both layers ourselves. The point is that the choice should stay yours: you can add the analytics platform you want, or leave the one you have, without paying a toll or waiting on a batch to get your own data. The only reason to accept less is that a vendor made leaving hard enough that staying felt like a decision. It was not a decision. It was the absence of one.

The strategic question

Your SIEM is not a tool you bought. It is where isolated alerts become an attack story, and where you go to find out what actually happened. The vendor holding it is a strategic dependency, whether you planned it that way or not.

So evaluate them like one. Put data portability on the RFP, not in the demo, and score it on two things: what it costs to move your telemetry somewhere else, and how much delay it adds.

Because you cannot build a real-time defense on a delayed copy of your own telemetry, and you should not have to buy your data back to try.

It is your data. Any vendor who makes that complicated has told you what kind of partner they intend to be.

Detection rule change history with one-click rollback

Detection rules change constantly. Analysts add exceptions, engineers tune them, detection logic shifts to adapt to new threats. Until now, that history was gone the moment it happened. If a reliable rule stopped firing, there was no built-in way to see what changed, who changed it, or when, which is a debugging problem and a compliance problem in one.

In 9.5, Detection Rules History Management ships as GA. A History section on the rule details page shows a complete, chronological log of every saved rule state: who made the change, when, and the revision number. From there, you can preview any historical revision, compare it to the previous version, and restore it with a single click.

The log is immutable and append-only, and it captures changes made through the UI or the API. Compliance teams get a defensible, timestamped audit trail for ISO 27001, SOC 2, and DORA standards without any manual export or configuration. Detection engineers get a real undo button: no custom scripts, no digging through audit logs.

See it in action. Kseniia Ignatovych demonstrates the rule history log, version comparison, and one-click rollback in the short video below.

SOC case management: templates and case analytics

Cases are where investigations land, but the data inside them has rarely been reliable enough to learn from. Custom fields were limited in type and count. The same fields appeared on every case regardless of what the analyst was investigating. Building dashboards required manual index configuration that most teams never completed, so case data stayed useful in the moment and hard to aggregate at scale.

In 9.5, we rebuilt the template system to fix how data goes in and made cases queryable out of the box for everything downstream.

Investigation-specific case templates with custom fields

Admins can now define templates for specific investigation types. A "Compromised Account" case collects different information than a "Service Outage" case. Admins build templates using a YAML editor with an Actions menu helper and a live preview panel. Analysts pick the right template for their investigation, see only the fields that apply, and fill in what's actually relevant.

The previous cap of 10 templates was a ceiling for enterprise SOCs managing phishing, malware, insider threat, compliance audits, and more. That limit is gone, along with the cap on custom fields. Seven new field types are also now available, including:

• Checkboxes
• Radio buttons
• A user picker
• A date/time picker

Any field can be marked required before case closure, so regulated teams can enforce that fields like "Root Cause" or "Closing Reason" get filled in before a case closes. A Field Library lets admins define reusable fields once and apply them across templates.

Queryable case analytics on every deployment

Cases as Data, Elastic Security's case analytics feature, exposes case activity in dedicated analytics indices so teams can build dashboards tracking case volume, closure rates, time to close, and assignment load, rather than relying on the case UI alone.

We shipped Cases as Data as a tech preview in 9.2, but it required manual configuration, wasn't available on Serverless, and had a complex index structure that wasn't ready for broad adoption. In 9.5, it's GA, on by default, and available across every deployment type including Serverless.

The architecture is simpler, too:

| | Before 9.5 | 9.5 GA | | :---- | :---- | :---- | | Index architecture | 12 indices per space | 3 global indices | | Configuration | Manual setup required | Auto-provisioned | | Serverless | Not available | Available | | Data views | Manual creation | Pre-built Case Analytics view per space |

SOC managers, IR leads, and SRE teams can build case reporting on any deployment without manual index configuration.

Get started with detection rule history and case analytics

Detection engineers have been working without rule change history. SOC managers have been working without case data they can report on. Both change in 9.5.

Reliable change history and queryable case data are not the loudest features in a release. They are the groundwork the next wave of SOC automation depends on.

See the documentation for Detection rule change history, Case Templates, and Case Analytics to get started. Try the new capabilities on your deployment, or start a free trial. Connect with us on Elastic's community Slack to share feedback or tell us what you are building and how we can help.

Not the machine learning models. Not the anomaly detection algorithms. Not the risk scoring engine. The entities: the foundations to teach your system about the data and protect it.

Get the entities wrong, and everything downstream is contaminated, including AI. Your baselines are fiction. Your risk scores are noise. Your analysts are chasing ghosts. And the worst part? Most user and entity behavior analytics (UEBA) implementations get the entities wrong from day one. This blog explains why the entities you don't create matter and how a confidence-tiered model helps.

A note on terminology before we go further: in this piece we’ll distinguish between the real-world thing — a person, a host, a service — and the entity record Elastic Security creates to represent it. The argument that follows is about which records are worth creating, not about which things exist. We’ll use “entity record” when the distinction matters.

One username, hundreds of identities

Consider the simplest possible approach to creating a user entity record: Take a user.name field from an event log and call it an entity. A username like deploy appears in your telemetry, so you create a deploy entity record and start building a behavioral baseline.

The problem is immediate and severe. That deploy username might exist on 200 servers. It might be used by 12 engineers and a continuous integration and continuous deployment (CI/CD) pipeline. The behavioral baseline you're building is a smoothie blended from hundreds of different machines, used by different people, for completely different purposes. The system is treating a string match as an identity, barely one step above random.

When this entity record inevitably generates elevated risk scores, analysts investigate, only to discover that the "anomaly" was just a different engineer using the same shared account in a slightly different way. Multiply this across every common username in a large environment and you've built a system that generates investigative busywork at industrial scale.

The opposite mistake: An empty dashboard

Some security vendors recognize the problem and swing to the other extreme. They decide (correctly, in principle) that only identity-provider-backed entity records are trustworthy enough for behavioral analytics. If you can't tie an entity record to an authoritative account in a directory like Okta, Entra ID, or Active Directory, don't create one at all.

The engineering reasoning is defensible. The product experience is catastrophic.

A customer rolls out endpoint agents across their fleet, opens the security analytics dashboard, and sees nothing. Zero entities. The feature looks broken. The SOC analyst has no idea why no users are showing up, and worse, has no visibility into the activity happening on those endpoints right now. A purist entity data model has produced a blind spot. 

The missing middle: Host-scoped identity

Most vendors building UEBA have historically picked between two unsatisfying defaults, bare usernames that blend everyone into noise, or IdP-only entities that leave most deployments with nothing. But without explicit governance over which signals come from which source, the noise still leaks through.What's missing is a middle layer: entities derived from endpoint telemetry that are tightly scoped enough to be meaningful but carefully governed enough to avoid the noise problem.

The instinct might be to simply pair a username with a host and call it an entity record. But without guardrails, this just moves the noise problem down one level. deploy on prod-web-03 is still five engineers' blended activity. root on a shared bastion host is still everyone and no one. You've reduced the blast radius from "all servers" to "one server," but the behavioral baseline is still a fiction if the underlying account is shared, automated, or observed only through a failed brute-force attempt that never actually succeeded.

The real question isn't whether to create local host-scoped entity records. It's which host-scoped entity records are worth creating and with what governance over how they participate in risk scoring and identity resolution downstream.

The Elastic approach: Two kinds of entities, governed differently

One answer is to recognize that not all record sources are equal and to build that distinction into the architecture itself, governing how each record is created, enriched, scored, and resolved.

This is the approach Elastic Security takes. Instead of treating all entity records as interchangeable, the system draws a clear line between identity-provider-backed entities and endpoint-observed local host entities and governs each category differently under the hood.

Identity-provider-backed entities are created only by authoritative identity systems: Okta, Entra ID, Google Workspace, Active Directory, and other identity-provider namespaces across identity and access management (IAM), cloud platforms, software as a service (SaaS), and privileged access management (PAM) systems. These entity records represent verified accounts in systems that own and manage those accounts. They get the full analytical treatment, that is, rich behavioral baselines, cross-platform enrichment from 120+ security integrations, and full participation in person-level risk scoring.

Endpoint-observed entities are created from endpoint telemetry: Your endpoint detection and response (EDR) agent observes jdoe active on a specific host and creates a host-scoped entity record tied to that local machine. These entity records are real and useful, but the system knows they carry less identity certainty, and it governs them accordingly. 

Analysts don't need to think about any of this machinery. What they see is an entity store that's populated from day one with more accurate user entity records. The governance happens in the architecture so it doesn't have to happen in the analyst's head.

(An endpoint-observed entity for sarah.chen active on her corporate MacBook. The entity name (sarah.chen\@CORP-MAC-SC-2024) uses the human-readable hostname for readability. The entity ID (user:sarah.chen\@b92f1e3a-7d4c-4a8b-9f2e-1c3d5e7f9012\@local) uses the machine's hardware UUID instead of its hostname — this is intentional. Hostnames change when a device is renamed or reimaged; the hardware UUID is permanent. The @local suffix identifies this as an endpoint-observed entity: it represents sarah.chen's activity on this specific machine, not sarah.chen as a verified identity across your organization.)

From fragmented accounts to a Unified User Group. 

Even when you get entity record creation right, you’re still left with a fragmentation problem no amount of per-entity discipline can solve alone.

Consider John Doe in a large enterprise. He has an Okta account for SaaS access, an Entra ID account tied to his corporate laptop, and an Active Directory account for on-prem systems. Each is a legitimate, authoritative entity record by every standard described above, and yet they’re three separate records for the same human being, each with its own risk history, each generating signals in isolation.

When John’s Entra account shows a lateral movement indicator the same day his Okta account flags a suspicious login from an anomalous location, those signals may never connect. They exist as separate entities, investigated in isolation, by analysts who have no automated way to know they belong to the same person. The cross-surface campaign that entity analytics is supposed to catch hides in plain sight between identity providers.

Elastic Security solves this through automatic entity resolution: consolidating a user’s fragmented digital footprint across Okta, Entra ID, Active Directory, and more into a single unified identity. John Doe becomes a first-class citizen in the entity store: one primary record, all associated accounts grouped together, one aggregated risk score that reflects everything happening across every identity surface he touches. Resolution runs continuously as new integrations come online, without manual curation.

In the image above, we've created one entity record per user account and grouped them together, so when an analyst reviews the record, all related identities appear in one place.

What this changes for analysts

For the security operations center (SOC) analyst working a 2 a.m. alerted by their Elastic agentic workflow, these two architectural decisions (confidence-tiered entity governance and unified identity resolution) change the investigation fundamentally.

Instead of opening four separate entity cards for the same person, they open one. Cross-provider risk signals are aggregated into a single score, and attack narratives that span identity systems are visible as narratives, not as disconnected data points buried in separate views.

Compare this to investigating a bare deploy entity record with a risk score of 70 that’s actually an artifact of 12 people’s blended activity across 200 servers. One investigation leads somewhere. The other erodes trust in the entire capability, which is the real cost of noisy entity records. Not just the false positive itself, but the analyst who learns to ignore entity risk scores entirely.

The entities records you don't create

Perhaps the most underappreciated aspect of entity analytics design is restraint. Every entity record you create has a cost: Compute for baselining, storage for history, analyst attention when it generates alerts, and potential for noise propagation into the broader analytical model.

The discipline to not create an entity record (to require a minimum evidence threshold) is what separates an entity analytics system that gets more useful over time from one that slowly drowns its operators in noise.

In practice, this means maintaining a configurable exclusion list for common service and shared accounts: root, jenkins, deploy, postgres, and others like them. These accounts exist on hundreds of machines, are used by automated processes and multiple humans interchangeably, and would produce baselines that mean nothing. Elastic Security ships a default list covering the most common offenders. Today the list operates at the username level — root is excluded uniformly regardless of which host it appears on — and is fixed. Future iterations will make it configurable, letting teams add their own environment-specific service accounts and, eventually, specify compound patterns that combine username and host. That would allow excluding root globally on shared infrastructure while still creating a host-scoped entity when that account appears on a personal workstation where the activity is attributable to a specific person. The architecture is already built for it: because endpoint-observed entities are keyed as {user.name}@{host}, compound rules are a coherent extension, not a redesign.

The higher-fidelity approach we've described directly mitigates the broader noise problem. When entity records are created from any username string in a log, your ML models end up baselining service accounts, typo'd logins, and shared kiosk accounts as if they were people — a 2 a.m. backup job looks anomalous against a human baseline, and a one-event typo'd login never accumulates enough data to baseline at all. When entity records are anchored to authoritative identity sources and resolved across their various login forms, the model learns patterns for actual humans doing actual work. Anomalies become meaningful because the baseline is meaningful.

The best entity analytics isn't the one that creates the most entities. It's the one that creates the right entities, governs them by what it actually knows about their source and scope, and builds its analytical investment proportionally. Everything else (risk scoring, behavioral baselines, entity resolution, anomaly detection, and AI skills) is downstream of that foundational decision. Get the entity records right, and the analytics follow. Get them wrong, and no amount of machine learning or AI can save you.

Entity analytics is available in Elastic Security. Learn more about advanced entity analytics and how the entity store governs user entities.

Your riskiest entities are already known; your SIEM just doesn't know that

Security teams aren't starting from a blank slate. You already know certain people, hosts, and services deserve elevated scrutiny: the privileged admin whose access was never revoked after a role change, the engineer on a performance improvement plan, the acquired company's infrastructure not yet fully onboarded, the contractors brought in for a sensitive new business initiative.

The gap has never been awareness. The gap is that your security information and event management (SIEM) has no way to express that organizational knowledge as a first-class risk signal. Behavioral detection fires on anomalies. Threat intel fires on known bad indicators. But neither captures the context your security and HR teams carry in their heads, and that context is exactly what insider threat programs are built on.

| 83% of organizations experienced at least one insider attack in the past year | $17.4M average annual cost of insider threat incidents in 2025 | 246 days average time to identify and contain a credentials-based breach | | :---- | :---- | :---- |

Mature user and entity behavior analytics (UEBA) platforms allow some form of static lists or risk multipliers, but they're often rigid, buried in configuration, and disconnected from the analyst workflow. Security teams deserve something better: a purpose-built, first-class feature that lets them codify what they already know about their environment and to have that knowledge dynamically influence every risk calculation in the platform.

Introducing Entity Analytics Watchlists

Watchlists are a new capability in Elastic Security's Entity analytics suite, arriving in the v9.4 release. They let security teams create named, described, rule-driven, or manually curated lists of users, hosts, services, or other entities and to attach configurable risk score weightings to every entity on each list.

Think of Entity Analytics Watchlists as the bridge between your organization's institutional knowledge and your security information and event management’s (SIEM's) risk engine. You already know who deserves a second look. Now your platform knows too.

The term watchlist is a familiar concept in the industry, but Entity Analytics Watchlists go further. This isn't just a way to bookmark entities; it's a structured mechanism for injecting custom correlation factors directly into the risk scoring pipeline. Every entity that appears on a watchlist carries its membership as a weighted signal, compounded with alert activity, asset criticality, and behavioral anomalies to produce a single, prioritized risk score.

Take a concrete example: John Doe is a departing employee. He’s added to a "Departing Employees" Watchlist configured with an elevated risk weighting. He also owns a server on the "Critical Infrastructure" Watchlist. When John triggers an alert, for example, an unusual volume of file downloads, his risk score now compounds all three signals: the alert, the asset criticality, and both list memberships. The platform surfaces him far higher in the risk queue than it would for the same alert on an average employee. The analyst sees exactly why.

The lists your security program already maintains

Entity Analytics Watchlists are most powerful when they reflect the real-world risk categories your security, HR, and operations teams already track informally. Here are the most common starting points:

| 🚪 Departing employees On notice, performance improvement plans (PIPs), or offboarding: elevated exfiltration risk, regardless of whether an alert has fired. | 🔑 Privileged access users Admins and service account holders whose actions carry outsized blast radius. | 👑 Crown jewel hosts Critical infrastructure, IP repositories, and financial systems demanding tighter scrutiny. | | :---- | :---- | :---- | | 🤝 Mergers and acquisitions / acquisition cohorts Newly onboarded entities where trust has not yet been fully established. | 🚀 High-risk business initiatives Teams in sensitive new ventures, requiring extra monitoring during critical phases. | 🛡️ Known-safe allow lists Dampen scores for verified low-risk entities, keeping analyst focus where it matters. |

Custom correlation, finally, without the engineering overhead

Historically, bringing organizational context into a SIEM risk model has required significant custom engineering: lookup tables, enrichment pipelines, detection rule overrides, and constant maintenance as personnel and asset inventories change. For most security teams, that overhead means it simply doesn't get done. We remove that barrier entirely.
An insider threat analyst can create a "Departing Employees" list in minutes, add a risk weighting, and immediately see that context reflected in the entity risk queue. No Elasticsearch Query Language (ES|QL) required. No pipeline configuration. No ticket to the detection engineering team. The organizational knowledge that was previously locked in spreadsheets, HR systems, or informal team awareness is now a first-class signal in the platform.

This is the ability to build risk correlations factors that simply haven't been possible anywhere else in the market and to do it without requiring detection engineering expertise.

For more mature teams, our custom watchlists also integrate cleanly with automated population, meaning that lists can be kept automatically current as conditions change or through APIs. An HR integration that marks an employee as departing can trigger list membership automatically; when they're fully offboarded, they're removed. The signal stays fresh without manual upkeep.

Coming in Elastic Security v9.4

Entity Analytics Watchlists ship as a major roadmap item in the upcoming Elastic Security v9.4 release. They’re available to customers running Elastic Security with Entity analytics enabled.

If you're already using entity risk scoring and asset criticality, Entity Analytics Watchlists are the natural next step, layering your organization's operational context on top of the platform's behavioral and alert-based signals to produce the most accurate, prioritized risk picture possible.

We've heard from security teams across industries that this capability is one of the most anticipated additions to the UEBA toolkit. We can't wait to see what lists you build.

Frequently Asked Questions

Q: How do I add organizational context to Elastic Security's risk scoring? A: In Elastic Security v9.4, you can inject organizational context into your risk scoring by utilizing Entity Analytics Watchlists, which allow you to ingest custom lists of high-value entities such as users, hosts, or services and assign them specific risk weightings. These watchlists function as dynamic correlation factors; when an entity on a watchlist appears in an alert, the system automatically compounds its risk score based on your pre-configured weights. This ensures that threats involving your most critical assets are prioritized instantly, transforming raw security data into an outcome-driven investigation queue that reflects your company's unique threat landscape.

Q: How do I monitor high-risk employees in a SIEM without custom detection rules? A: Elastic Security Watchlists let you add entities like departing employees or privileged admins to a named list with an elevated risk weighting, with no ES|QL or pipeline configuration required. Their list membership is factored into risk scoring automatically alongside any alert activity.

Q: How do insider threat programs integrate with SIEM risk scoring? A: Elastic Security's Entity Analytics Watchlists let insider threat and security operations teams codify existing risk knowledge — departing employees, privileged access holders, acquisition cohorts — and have that context automatically influence entity risk scores without requiring detection engineering involvement.

Entity Analytics is available in Elastic Security. Learn more about Entity Analytics Watchlists and how the entity store governs user entities.

In simple terms, an Agentic SOC is a security operations center that has deployed AI Agents and corresponding AI Agent Skills to perform SOC-related workflows such as detection engineering, alert triage, incident investigation, escalation, response, and threat hunting. When these workflows are performed by AI agents, they’re often called “Agentic workflows.” These AI Agents and Skills may run natively in a security operations platform like SIEM, XDR, or security analytics, or they may be layered on top of legacy SIEM as an “AI SOC Agent” or “AI SOC analyst”, or they may even be run from an AI Coding Tool.

Regardless of how they are implemented, the shift to the Agentic SOC is not about AI replacing human analysts; it's about transforming how the SOC functions. To keep pace with rapidly evolving attackers, defenders must leverage AI and autonomous agents to respond as quickly as possible. At its core, an Agentic SOC is defined by how a security operations center uses AI and agents to protect against adversaries.

Let’s simplify a successful security operations center to three fundamental pillars, all of which the Agentic SOC significantly enhances:

1. Observe: The foundation of all security is centralized data—aggregating logs and events into one location, which is the core strength of a SIEM solution.
2. Detect: This involves deploying core protections like endpoint-based security (XDR, such as Elastic Defend) and security solution-focused detections (cloud, identity data). This technology drives the generation of high-quality alerts. Elastic, for example, ships over 1,700 pre-built rules for its SIEM by default, not including its XDR solution's endpoint rule library.
3. Act: This is the critical final stage of triaging, investigating, and acting on the generated alerts.

Agentic SOC in Action

Imagine this real-life scenario unfolding in your Security Operations Center using the Elastic security platform. It begins not with a siren, but with a simple, direct Slack notification. Building on our recent blog on Attack Discovery, Workflows, and Agent Builder, let's further examine how Elastic Security can help you respond to an active attack.

1. The Initial Alert and Immediate Action
   Your security analyst receives an urgent notification in their team channel. This message isn't just a heads-up; it points directly to an observed, active attack. Crucially, the Elastic Agentic SOC has already taken decisive, pre-emptive action: a vulnerable host has been isolated from the network to contain the threat and limit potential damage. This was all powered by Elastic Workflows and Elastic Agent Builder processing realtime alert and attack data from Elastic.
   
2. The Centralized Case
   The analyst's next step is a click away, moving from Slack directly to the centralized Case within Elastic that was created by the workflow. Elastic Case Management enables the SOC to coordinate the response and provides a single pane of glass into all aggregated critical information:

• Attack Summary: A high-level overview detailing what has occurred using Attack Discovery.

• Attached Alerts: The specific security alerts that triggered the initial observation.

• Observables: A list of suspicious artifacts (IP addresses, file hashes, domains, etc.) collected from the event.

• Attached Events: Non-alert events that, while not an alert themselves, provide critical context and are of further interest to the investigation.

  

1. Supporting the Investigation
   To support the immediate findings, detailed Investigations are attached directly to the Case. These searches allow the analyst to visually and contextually step through the sequence of events leading up to, during, and immediately following the attack.
   The Elastic Case also provides instant context by highlighting Similar cases. By cross-referencing observables, the system identifies previous incidents involving the same entities or artifacts, providing a deeper understanding of the threat actor's history and potential motives.
2. The Path to Resolution
   The agents don’t just catalog the past; it dictates the future. A clear set of Next steps and actions are outlined, with specific team members assigned for review and execution.

The analyst then steps through a methodical process reviewing the automated analysis:

1. Reviewing Findings: Scrutinizing all aggregated data, alerts, and investigations.
2. Evidence Collection: Collecting any additional forensic evidence needed for a complete analysis.
3. Remediation: Executing manual or automated actions, such as deleting malicious files or killing persistent processes on the isolated host with Elastic Defend.
4. Final Release: Eventually, the host is safely released back to the network, but not before additional, targeted rules or policies are automatically applied to prevent a recurrence based on the lessons learned from this incident.
   In the Agentic SOC, the analyst moves seamlessly from a high-level alert to a comprehensive investigation to full remediation—all within a unified, intelligent workflow powered by Elastic.

Elastic Security and Core SIEM Workflows

Before exploring advanced agentic workflows, it's essential to recognize that Elastic Security already provides a comprehensive suite of core capabilities crucial for modern security operations. This foundation begins with the ingestion of security-relevant data, which is automatically normalized to a common schema, ensuring consistency and ease of analysis. The platform offers Extended Detection and Response (XDR) capabilities via Elastic Defend, a robust detection engine built directly into the Elastic Stack, and sophisticated alert workflows that include built-in correlations to reduce noise and surface true threats.

Elastic Security further differentiates itself by tightly integrating key operational functions. This includes entity-based threat hunting, machine learning for anomaly detection and behavior analysis, and comprehensive case management for tracking incidents. Finally, the platform provides end-to-end response and forensic capabilities, enabling security teams to move swiftly from initial alert to investigation and remediation, all within a unified, scalable platform.

Empowering Analysts with Agentic Capabilities

AI-Powered Alert Triage and Prioritization

The Elastic Security Solution integrates AI capabilities via Agent Builder to augment and make SOC operations truly agentic. This is where efficiency improvements are most keenly felt:

• Conversational Triage: A built-in agent is readily available to Tier 1/2 analysts, allowing them to use conversational commands to query and prioritize open alerts (e.g., "What priority alerts should I review from the last 30 days?"). This is the first entry point for using AI to augment SOC operations.
• LLM Agnostic Platform: A key differentiating feature of Elastic's Agent Builder is that it is LLM agnostic, allowing organizations to pick their preferred model, even locally running models for privacy or regulatory reasons.
• Attack Discovery: This premier feature moves beyond basic triage. It uses LLM configurations to create higher-order attack detections, taking hundreds of open alerts and prioritizing them into a small, manageable subset of known attacks or incidents. This dramatically reduces the impact of alert fatigue.

Enriched Investigations

Once an attack or incident is found, the agent helps start the investigation:

• Summarization and Enrichment: The agent can be used to summarize the attack, identify important artifacts, and conduct automated third-party enrichments (like checking VirusTotal). This tailored experience provides a full assessment, including an attack chain, threat intelligence information, related cases, entity risk scoring, and a full investigation guide.
• Case Management: The agent can be instructed to take immediate action, such as generating a security case and notifying the team in Slack, all through simple conversational commands that execute pre-configured workflows.

Automated Response and Threat Hunting

The true power of the Agentic SOC is realized through action and automation that goes beyond simple conversation:

• Workflows and SOAR-like Automation: Agents can reference and execute Workflows, Elastic's SOAR-like automation tool. These workflows allow analysts to take immediate, complex actions. For example, a command like "Please create a case for this attack, and notify my team in Slack" triggers multiple, pre-defined steps. Further critical response actions, such as isolating a host, can be executed with a single workflow action while the investigation continues.

• AI-Assisted Threat Hunting: AI assists threat hunters by leveraging Entity Analytics and pre-built skills. The agent can be asked to find high-risk hosts and users to begin hunting, and then automatically generate specific ESQL queries (e.g., "Please tell me the most uncommon processes executed for each host") to uncover unusual or malicious activity.

  

The Mandate of Automation

For maximum effectiveness, all these steps,from alert triage and enrichment to case creation and host isolation,can be configured to run automatically as an Agentic Alert Triage workflow. This allows the system to solve problems as soon as they are discovered, setting up the human analyst in the loop with a consolidated case and all the necessary findings in a single pane of glass.

This approach delivers substantial efficiency improvements, making speed the single most important factor in a modern, Agentic SOC.

Elastic’s Agentic Security Operations Platform

Whether you use our UI, our agents, or your own, Elastic Security provides a strong open foundation for modern security operations. best-in-class data architecture, search, workflows, analytics, detection engineering content, and automation.

Getting started

Before you get started: AI coding agents operate with real credentials, real shell access, and often the full permissions of the user running them. When those agents are pointed at security workflows, the stakes are higher: you're handing an automated system access to detection logic, response actions, and sensitive telemetry. Every organization's risk profile is different. Before enabling AI-driven security workflows, evaluate what data the agent can access, what actions it can take, and what happens if it behaves unexpectedly

Don't have an Elasticsearch cluster yet? Start an Elastic Cloud free trial. It takes about a minute to get a fully configured environment.

At Elastic, the Infosec team is "Customer Zero”. We use the newest version of Elastic products extensively to secure our organization, which gives us unique insights into how to solve real-world security challenges. One of the ways we've improved Security Operations Center (SOC) efficiency is by creating a seamless, automated workflow that allows our analysts to open a detection tuning request directly from Kibana Cases with a single click.

In any SOC, the feedback loop between security analysts and detection engineers is crucial for maintaining a healthy and effective security posture. Analysts on the front lines are the first to see how detection rules perform in the real world. They know which alerts are valuable, which are noisy, and which could be improved with a bit of tuning. Alert fatigue from noisy alerts increases the risk of missing a true positive alert. Quickly tuning false positives is critical to responding to true positives. Capturing this alert feedback efficiently can be a challenge – manual processes, like sending emails, opening tickets, or direct messages can be inconsistent, time consuming, and hard to track.

With Elastic Security, an analyst can attach alerts to a new or existing case in Kibana, conduct their investigation, and with some customization and automation they can initiate a tuning request with a single click directly from Kibana Cases. This article will walk you through how we built this automation, and how you can implement a similar system to close the feedback loop and optimize your detection and response program.

Custom Fields in Kibana Cases

Custom fields are a key component of this automation within the Kibana Cases. Using these custom fields, we can capture the necessary information directly from the tool that the analysts are already using. These custom fields will appear on all new and existing cases, providing a clear and consistent way for analysts to flag a detection for review.

Note: The ability to add custom fields to cases was introduced in version 8.15. For more details, refer to the official Cases documentation.

Every Kibana Case is a document stored in a dedicated Elasticsearch index: .kibana_alerting_cases. This means all your case data is available for querying, aggregation, and automation, just like any other data source in Elastic. Each case document contains a wealth of information, but a few fields are particularly useful for metrics and automation. The cases.status field tracks whether a case is open, in-progress, or closed, while cases.created_at and cases.updated_at provide timestamps crucial for calculating metrics like Mean Time to Resolution (MTTR). Fields like cases.severity and cases.owner allow you to slice and dice your metrics to see how the team is performing. Most importantly for this blog, the cases.custom_fields object contains an array of the custom fields you've configured. Runtime fields can be used to parse the array of custom fields, allowing you to build queries, dashboards, visualizations, and detection rules that trigger workflows.

Beyond tuning requests, custom fields are incredibly versatile for tracking metrics and enriching cases. For example, we have a "Complex Case" custom field to flag cases that take more than an hour to resolve, helping us identify rules that may need better investigation guides or automation to help reduce the investigation time. We also use custom fields like "Detection rule valid" and "True Positive Alert" to gather granular feedback on rule performance and fidelity, allowing us to build powerful dashboards in Kibana to visualize the operational effectiveness of our SOC.

If you have not already created a data view for the Cases information you will need to do that if you want to use runtime fields and data visualizations with your cases.

Navigate to Index Patterns: In Kibana, go to Stack Management > Data Views and click ‘create new data view’.

Configure the Data view to map the .kibana_alerting_cases system index. You will need to click the Allow hidden and system indices button to allow this. For the timestamp field I recommend using the cases.updated_at field so the cases are displayed by the most recent activity.

Creating Custom fields

There are two types of custom fields; Text fields for free-form input, or Toggle fields for simple yes/no feedback. For our Tuning Request automation, we use one of each. The text field is an optional field used to capture any additional feedback from the analyst, and the toggle field is used to trigger the automation.

In Kibana, go to Security > Cases, then click on Settings in the top right. In the settings page you will see a Custom Fields section where you can add the new fields you want. The fields are displayed in the cases UI in alphabetical order so we prefix our fields with numbers to keep them in the order we want.

You can create the new custom fields, the Labels added in the UI are only for the analysts and are not stored in the cases index. These can be any value you want.

Add Custom Fields: We need two fields for this workflow.

• Field 1: Tuning Required Toggle

• This will be the button analysts click to initiate a tuning request.

  • Label: Open tuning request?
  • Type: Toggle
  • Default Value: Off

• Field 2: Tuning Request Details

  • This field allows the analyst to provide specific details about what needs to be changed, such as adding an exception, lowering the severity, or adjusting the query logic.
  • Name: Tuning request detail
  • Type: Text

• Default Value: Off

Using Runtime fields to map the custom fields

A challenge when working with custom fields in Kibana Cases is that the cases.custom_fields field is mapped as an array of objects, where each object represents a custom field with its name and value. This structure makes it difficult to query for specific custom fields directly in KQL. For example, you can't simply use a query like cases.custom_fields.open_tuning_request : "true". To overcome this, we can use runtime fields to parse and query the custom fields.

Runtime fields are fields that are evaluated at query time. They allow you to create new fields on the fly without having to reindex your data. We can define runtime fields on the .kibana_alerting_cases index to use a painless script to parse the cases.custom_fields array and extract the values we need into new, easily queryable fields.

For this workflow, we'll create two runtime fields that will map to the custom fields created above:
* TuningRequired: A boolean field that will be true if the "Open tuning request" toggle is on.
* TuningDetail: A text field that will contain the analyst's comments from the "Tuning request detail" field.

Before we can create the runtime fields, we first need to identify the unique ID (key) that Kibana assigns to each custom field. Currently, there isn't a straightforward way to view this ID in the UI. To find it, we used the following workaround:

1. Create the Fields. If you are using other custom fields you should create the custom fields one at a time to make it easier to identify the new field keys. If you only have the two fields mentioned above you can tell them apart using the type value which can be either text or toggle.
2. Create a new case. After adding the field, we created a test case in Kibana and added some data to the description field and toggled the tuning required field to true with all other custom fields set to false or blank.
3. Inspect the case document. We then navigated to Discover and queried the .kibana_alerting_cases index to find the document for the new case. By inspecting the cases.customFields array in the document's source, we could find the key associated with our new custom field. Save the values of the key fields to be used in the runtime scripts.

The cases.customFields data is formatted like this:

  [
    {
      "key": "4537b921-3ca4-4ff0-aa39-02dd6a3177bd",
      "type": "text",
      "value": "This alert is too noisy"
    },
    {
      "key": "cdf28896-c793-43d2-9384-99562e23a646",
      "type": "toggle",
      "value": true
    }
  ]

Creating the Runtime Fields

You can add runtime fields through the Kibana UI or by using the Elasticsearch API in the Dev Tools console. If you have not already created a data view for the Cases information you will need to do that first.

While viewing the new Kibana Cases Data view click the ‘Add Field’ button to open the flyout menu to create a new runtime field.

Enter the name of the field, in this example we are configuring TuningRequired as a new Boolean field type. Click the ‘Set Value’ toggle to configure this as a new Runtime field configured via a painless script. Update this painless script to replace TUNING_REQUIRED_FIELD_KEY_UUID with the key value from the Tuning Required custom field and paste it into the value field and save the new runtime field.

...
    if (params._source.containsKey('cases') &&
    params._source.cases != null &&
    params._source.cases.containsKey('customFields') &&
    params._source.cases.customFields != null) 
{
  for (def cf : params._source.cases.customFields) {
    if (cf != null &&
        cf.containsKey('key') &&
        cf.key != null &&
        cf.key.contains('TUNING_REQUIRED_FIELD_KEY_UUID') &&
        cf.containsKey('value') &&
        cf.value != null) {
      emit(cf.value);
      break;
    }
  }
}

Repeat this process for the TuningDetail field, remember to use the key value from the text field in this field’s painless script. If you have any additional custom fields in your cases that you want to use for dashboards or metrics you can map those as well with this same process.

If you control your cluster settings and data views ‘as code’ you can also add runtime fields to an index mapping using the Update mapping API from the Kibana Dev Tools console.

Automating the tuning request creation

We can trigger this automation in two ways: through a custom detection rule (that will create a new alert and send it to a connector when a case is updated with a tuning request) or via a scheduled external automation that queries the API.

This automation can be created using any automation platform such as Tines, Github Actions, or custom scripting. This is the logic we use for our automation:

Step 1: Find any cases recently tagged as TuningRequired

You can use this elasticsearch query to find any cases that have been updated within the last hour where the TuningRequired field has been set to true. This query uses the cases.updated_at field as the time range. The runtime field mappings must be included in the API request to query the custom fields.

This query will return all of the case documents from the .kibana_alerting_cases index that have been updated in the last hour and the TuningRequired field has been set to true

POST /.kibana_alerting_cases/_search  
{  
  "query": {  
    "bool": {  
      "must": [],  
      "filter": [  
        {  
          "bool": {  
            "should": [  
              {  
                "match": {  
                  "TuningRequired": true  
                }  
              }  
            ],  
            "minimum_should_match": 1  
          }  
        },  
        {  
          "range": {  
            "cases.updated_at": {  
              "format": "strict_date_optional_time",  
              "gte": "now-1h",  
              "lte": "now"  
            }  
          }  
        }  
      ],  
      "should": [],  
      "must_not": []  
    }  
  },  
 "runtime_mappings": {  
   "TuningDetail": {  
     "type": "keyword",  
     "script": {  
       "source": "if (\nparams._source.containsKey('cases') &&\nparams._source.cases != null &&\nparams._source.cases.containsKey('customFields') &&\nparams._source.cases.customFields != null\n) {\nfor (def cf : params._source.cases.customFields) {\nif (\ncf != null &&\ncf.containsKey('key') &&\ncf.key != null &&\ncf.key.contains('6cadc70a-7d68-4531-9861-7d5bc24c4c1c') &&\ncf.containsKey('value') &&\ncf.value != null\n) {\nemit(cf.value);\nbreak;\n}\n}\n}"  
     }  
   },  
   "TuningRequired": {  
     "type": "boolean",  
     "script": {  
       "source": "if (\nparams._source.containsKey('cases') &&\nparams._source.cases != null &&\nparams._source.cases.containsKey('customFields') &&\nparams._source.cases.customFields != null\n) {\nfor (def cf : params._source.cases.customFields) {\nif (\ncf != null &&\ncf.containsKey('key') &&\ncf.key != null &&\ncf.key.contains('496e71f2-2bce-47a2-93a8-00db0de2d1b4') &&\ncf.containsKey('value') &&\ncf.value != null\n) {\nemit(cf.value);\nbreak;\n}\n}\n}"  
     }  
   }  
 },  
  "fields": [  
    "TuningDetail",  
    "TuningRequired"  
  ]  
}

Any time a field is changed or a comment is made in a case it will update the updated_at field to the current time. Because any update or comment added to a case will update this timestamp, it is possible to have a single case returned multiple times by this automation if it is run regularly while the case is being updated. Any automation processes leveraged for this should have a deduplication process to prevent processing the same case multiple times in this scenario.

Step 2: Parsing each case

Loop through each of the cases returned by the previous query to process them one at a time. Each document returned will contain the fields array with the values from the custom fields, as well as other useful fields. Parse each of the following fields and store them for future use:

• The _id field will have a format like cases:{{case_ID}}. The case ID is used for future API requests in the automation to add comments to the case or retrieve all alerts attached to the case.
• cases.title is the title of the case
• cases.assignees is who the case is assigned to
• cases.updated_by is the last person to update the case, this is often the person submitting the tuning request and can be useful for knowing who to contact for more information.
• cases.tags can be useful if you are using tags to sort or identify your cases.

Step 3: Retrieving the alerts attached to the case

For each case you will want to know which alerts are attached to the case so you know which alerts need to be tuned. This can be done using the cases API with the _id field for the case.

/api/cases/{caseId}/alerts

This query will return an array of all alert id values that are attached to the case. Using this ID value you can query the .siem-signals* elasticsearch index to find the full information about each alert attached to the case that needs tuning.

POST /.siem-signals-*/_search  
{  
 "size": 1,  
 "query": {  
   "bool": {  
     "must": [],  
     "filter": [  
       {  
         "bool": {  
           "should": [  
             {  
               "match": {  
                 "_id": "{{alert_id}}"  
               }  
             }  
           ],  
           "minimum_should_match": 1  
         }  
       },  
       {  
         "range": {  
           "@timestamp": {  
             "format": "strict_date_optional_time",  
             "gte": "now-30d",  
             "lte": "now"  
           }  
         }  
       }  
     ],  
     "should": [],  
     "must_not": []  
   }  
 }  
}

From the results of this query you can extract information about the alert such as the name and creation date, along with any other information that could help for tuning such as the user.name or process.name fields. Because a case can have many alerts attached to it you will want to deduplicate the alerts by the signal.rule.name value.

Step 4: Opening a tuning request.

This step is dependent on the ticketing system you use in your environment. Our team uses github issues to track tuning requests and slack for notifications, but this could also be done with any ticketing or project management system that supports automation.

This is the logic flow we use for our automation using both Github and Slack to track tuning requests:

• Using the name of the alert we search for any existing open tuning requests.
• If an existing tuning request exists we update that request with the details from the case and the new request
• If no existing request exists we open a new tuning request issue and attach the information
• We then send a slack notification to the Detection engineering team’s slack channel containing a link to the tuning request, a link to the case, and details about the request and alert.
• We then use the Cases API to add a comment to the original case with a link to the tuning request issue
• Optional AI Agent: We are starting to experiment with the use of AI Agents to analyze the alert and case information and then provide even better context with the tuning request, potentially even recommending the changes to make to the detection rules.

The final result from this automation is that our SOC Analysts can create a detailed detection tuning request ticket with a single click from their case. We have seen a dramatic increase in the reduction of false positives and the overall efficiency of our detection rules because of this automation.

Conclusion

By using Kibana Cases with custom fields and integrating with automation platforms, you can optimize many of your manual processes. This automated workflow reduces the manual overhead associated with collecting analyst feedback, ensuring that valuable analyst insights are quickly translated into actionable improvements in detection rules. The result is a more efficient, accurate, and resilient SOC that can adapt rapidly to emerging threats and reduce alert fatigue.

Ready to optimize your SOC's efficiency and improve your detection posture? Explore Elastic Security and start building your own automated tuning request workflows today!

In today’s complex cybersecurity landscape, one of the most overlooked but critical elements in proactive threat detection is monitoring for TOR (The Onion Router) exit node activity. TOR enables anonymous communication, and while it serves legitimate privacy interests, it also provides cover for cybercriminals, malware campaigns, and data exfiltration.

What Are TOR Exit Nodes?

TOR exit nodes are the final relay points in the TOR network where encrypted traffic exits to the open internet. If a user browses the web anonymously via TOR, the website or service they access will see the IP address of the exit node, not the user's actual IP address.

In other words, any network traffic originating from a TOR exit node is untraceable to its source without cooperation from the TOR network, which is unlikely by design.

Why Should You Care?

While not all TOR activity is malicious, a substantial amount of malicious traffic uses TOR to mask its origin. Here’s why it matters:

1. Anonymized Reconnaissance: Attackers often perform scans and probes from TOR exit nodes. If someone is mapping your infrastructure using TOR, they may be preparing for a breach attempt while remaining anonymous.

2. Command and Control (C2) Channels: Many malware families use TOR for C2 communications, making it hard to trace the infected endpoint back to its controller.

3. Data Exfiltration: TOR is a common channel for exfiltrating sensitive data out of an organization. If sensitive files are being uploaded to external endpoints via TOR, you may already be compromised.

4. Compliance Risks: Some industries (e.g., healthcare, finance) require strict data handling and access controls. Allowing or ignoring TOR-originated traffic could violate these policies or industry regulations.

You should look for any interactions between TOR exit nodes and:

• host.ip
• server.ip
• destination.ip
• source.ip
• client.ip

This can occur in logs from firewalls, DNS, proxies, endpoint agents, cloud access logs, and more.

How to Monitor for TOR Exit Nodes

In order to collect, monitor, alert, and report on TOR Exit Node activity, we must first create a few components, namely, we will create an index template and an ingest pipeline. We will then hit the TOR API endpoint every 1 hour to request the most recent detailed information.

If you would like to learn more about options for monitoring TOR, you may read about them here. If you would like to know more about the TOR Project in general, you may read about it here.

Ingest Pipeline

First, let’s create an Ingest Pipeline that will accomplish the last bit of parsing our data before it is written to an index. In DevTools, simply apply the following: there are descriptions for each processor; should you want to know more about what each does and its associated condition, if present.

Here is what your screen may look like:

You may find the ingest pipeline on GitHub.

Index Template

Next, we need to create our index template to ensure our fields are correctly mapped.

Still in DevTools, submit the following request just as you completed with the ingest pipeline. You may find the index template via this link on GitHub.

Notice the priority of the index template; we set this to a much higher number so that this template will take precedence over the default logs-*-* template. While you will notice in the following steps that we set the ingest pipeline in our configuration for data collection, we may also apply it here as a safeguard to ensure data is written through this pipeline.

Elastic-Agent Policy

With these two items loaded, we may now navigate to Fleet and select the “agent policy” we want to install our integration to.

On the policy you wish to install the TOR collection to, simply click “Add integration”.

Select “Custom” from the left-hand category list, then click “Custom API”.

Click the blue “Add Custom API” button on your top right.

You may title your Integration anything you like; however, I will be using “TOR Node Activity” in this example.

Fill in the following fields:

Dataset name:
ti_tor.node_activity

Ingest Pipeline:
logs-ti_tor.node_activity

Request URL:
https://onionoo.torproject.org/details?fields=exit_addresses,nickname,fingerprint,running,as_name,verified_host_names,unverified_host_names,or_addresses,last_seen,last_changed_address_or_port,first_seen,hibernating,last_restarted,bandwidth_rate,bandwidth_burst,observed_bandwidth,flags,version,version_status,advertised_bandwidth,platform,recommended_version,contact

Request Interval:
60m

Request HTTP Method:
GET

Response Split:
target: body.relays

You will then need to click to expand the “> Advanced options” and scroll down a bit more.

You may find the necessary processor snippet to copy at GitHub here.

You may now click the “Save and continue” button and in a few minutes you will have TOR node activity available in your logs-* index!

Filebeat Installation Option

If you are not using Elastic-Agent and wish to ingest via Filebeat, that’s cool too! Instead of using the steps above, simply leverage the following “filebeat.inputs:” which will use the exact same ingest pipeline and index template as above! Simply copy and paste the input section into your filebeat.yml file, you will still need to add an output section.

Reviewing your data

Now that you've completed the configuration of the ingest pipeline and the agent integration, you can see the TOR nodes in the Discover view. From here, you can create rules, visualizations, dashboards, etc., to help keep tabs on how TOR is being used on your network.

What can you do next?

The beautiful thing about the naming convention for this index, is that it will automatically function with your Threat Intel IP Address Indicator Match rule available in the Elastic SIEM.

However, you may want to make your own rule using some of the wealth of information that is provided with this integration; particularly depending on the type of node observed environment. Since there was a considerable amount of geo-based data enriched with this index, now would be an excellent time to check out some of the map features within Kibana.

The release of the detection engine in Elastic Security brought automated threat detection to the Elastic Stack. Since the initial launch of the detection engine, the Elastic Security Intelligence & Analytics team has added 50+ additional rules, increasing the visibility of attacker techniques on Linux, macOS, and Windows operating systems. As we continue to expand coverage, you’ll see increased breadth in detections, covering new domains such as cloud services and user behavior.

Over the past few releases, we used an internal repository to manage rules for the detection engine. We’ve iteratively improved our testing procedures by adding automated tests for new contributions that validate Kibana Query Language (KQL) syntax, schema usage, and other metadata. Our rule development has matured, so we can move fast without breaking things.

By opening up our elastic/detection-rules GitHub repository, Elastic Security will develop rules in the open alongside the community, and we’re welcoming your community-driven detections. This is an opportunity for all of us to share our collective knowledge, learn from each other, and make an impact by working together.

What’s in this new repository?

In the elastic/detection-rules GitHub repository, you can find rules written for Elastic Security, with coverage for many MITRE ATT&CK® techniques. Our current rule logic is primarily written in KQL, and by leveraging the Elastic Common Schema (ECS), we only need to write rules once. By using the defined fields and categories in ECS, rules automatically work with Beats logs and other data sources that map properly to ECS.

Within the rules/ folder, rules are stored in TOML files and are grouped by platform. We tried to keep it simple with a flat hierarchy so that it’s easier to find and add new rules. If you’re looking for Windows-only rules, navigate to rules/windows. If you’re still struggling to find a rule or want to search across rules, you can use our CLI by running the command python -m detection_rules rule-search, which will show the files that have matching metadata.

Every rule contains several fields of metadata in addition to the query itself. This captures information like the title, description, noise level, ATT&CK mappings, tags, and the scheduling interval. We have a few additional fields to aid analysts performing triage, describing known false positives or helpful steps for an investigation. For more information on the metadata that pertains to rules, see the Kibana rule creation guide or our summary of rule metadata in the contribution guide.

Preview of the file behind the “MsBuild Making Network Connections” rule

How do these rules get to my detection engine?

If you’re using our Elastic Cloud managed service or the default distribution of the Elastic Stack software that includes the full set of free features, you’ll get the latest rules the first time you navigate to the detection engine. When you upgrade, the detection engine recognizes that rules were added or changed and prompts you to decide whether you want those rules upgraded. Follow the steps after upgrading and you’ll get the latest copy of the rules.

The same rule — “MsBuild Making Network Connections” — loaded in the detection engine

Who will use this repository?

This repository is where the Elastic Security Intelligence & Analytics team will develop rules, create issues, manage pull requests, and target releases. By making the repo public, we’re inviting all external contributors into this workflow. This will give contributors visibility into our development process and a clear path for rules to be released with the detection engine.

When you’re ready to contribute, please sign Elastic's contributor license agreement (CLA). This is standard for all Elastic GitHub repositories, and it means that we freely can distribute your code to Elastic users.

How do we approach threat detection?

In general, we tend to prefer detections that focus on adversary behaviors. This usually means that we focus on ATT&CK techniques. This might mean more research and effort is needed to figure out how a technique works before creating a rule. But by taking this approach, we do a better job detecting and stopping the attacks of today and tomorrow, instead of just the attacks of yesterday.

Taking a behavioral approach also means we need various types of rules. Some might detect atomic events, others may require aggregating multiple events or looking for deviances above a threshold. With Event Query Language (EQL), we will be able to write rules that look for sequences of behavior that span multiple events.

Of course, we understand that sometimes a technique can be hard for all users to detect behaviorally. In that case, by all means, feel free to add a rule that is more signature-like in nature and written towards a specific behavior or tool.

For a longer discussion on what makes a mature detection, read about the philosophy of the detection rules repository.

Why do we need a new repository?

If you’ve shared public rules before, you’re probably aware of other well-known GitHub repositories, like the Cyber Analytics Repository (CAR) by MITRE, Sigma, or even the EQL Analytics Library based on Elastic’s EQL. You might be wondering: Why do we need another repository? Why not use the ones that already exist?

Unsurprisingly, the best answer to this question starts with another question: Why do the other repositories exist? Both CAR and Sigma are purposefully agnostic of language or platform, and sometimes data source. On the other hand, the EQL Analytics Library was written with a specific language in mind.

With our new detection rules repository, we’re trying to serve a slightly different purpose. Our goal is to give users of Elastic Security the best possible detections that work across various data sources. We use ECS as the great equalizer of schemas, making it possible to write a rule once that applies to multiple data sources.

Since the Elastic Stack supports multiple languages, our rules should reflect that. Query languages are typically developed to solve different types of problems, and we shouldn’t constrain rule developers to a single language if another does the job better. We currently have KQL and Lucene rules, along with rules that use machine learning anomaly detection jobs in the repository. We’re working hard to bring EQL to the Elastic Stack and into our repository.

We can also ensure best practices are being followed for optimal Elasticsearch performance. For example, searching for process.path:*\\cmd.exe requires performing a wildcard check, which is more costly than a simple keyword check. Instead of searches containing leading wildcards, we can recommend using process.name:cmd.exe, which will result in better performance and the most accurate results. On a similar note, ECS also contains the field process.args, which is a parsed version of process.command_line. We recommend using the parsed field instead, because it gives us better performance and means that we are much less prone to trivial whitespace or quotation-based evasions. Win-win.

Can I add rules from another repository?

Within your own environment, you’re welcome to add rules to your own detection engine as long as your Kibana role has the right permissions. If you want to add rules to the elastic/detection-rules repository, the answer is an unsurprising: It depends…. As long as a rule can be sublicensed under the Elastic License, this is fair game. Most of the time, the requirements are fairly straightforward — retain the original authors in the rule.author array, and update the NOTICE.txt file accordingly to give attribution to the original authors. We don’t want to take credit for someone else’s work, so please help us be thorough!

For more information on how we approach licensing in the repository, check the Licensing section of the README.

How do I contribute?

Eager to share your rule logic? Hop on over to elastic/detection-rules on GitHub. We have detailed instructions there for navigating the repository, forking and cloning, and creating a rule. We include a command line tool for bulk editing the files and to make creating new rules easier. When you’re ready to add a new rule to the repository, run python -m detection_rules create-rule, and you’ll be prompted for the required metadata. We recommend using the CLI when possible, because it reduces copy-and-paste errors that happen when reusing contents from a TOML file from another rule or template.

When your rule is in a good state, you can run the command python -m detection_rules test to locally perform unit tests, which validate syntax, schema usage, etc. Then, create the pull request and someone on the Intelligence & Analytics team will review the contribution. If we request any changes, we’ll work with you to make the recommended changes.

If you have a good idea for a rule, but want to collaborate with us on the idea or get feedback, feel free to create a New Rule issue. We look forward to helping and brainstorming with you!

For more information, check out the contribution guide.

What’s next?

Welcome to our new rules repository and workflow! Your contributions are encouraged, and we look forward to seeing your name in the rule.author field. The detection rules repository will continue to evolve along with the rest of Elastic Security, and we’re excited for what’s next.

If you want to track the development of EQL in the stack, subscribe to this GitHub issue. Or take a peek at the ongoing documentation to watch what the Elasticsearch team is up to.

If you have any feedback or questions writing rules or navigating our new detection rules GitHub repository, please create an issue in GitHub, reach out on the discussion forum with the detection-rules tag, or find us in the #detection-rules channel of the Elastic Slack Community.