Why detection engineering needs AI-native tooling

The threat landscape has changed. Attackers are increasingly using AI to automate and scale their operations: generating phishing campaigns at volume, accelerating vulnerability research and exploitation, and launching credential attacks that would have required significant manual effort just a few years ago. The result is a faster, higher-volume threat environment where the window between a new attack pattern emerging and it hitting your environment is narrowing.

Detection engineering teams are on the other side of that equation. The expectation is that coverage keeps pace with the threat, but the tooling available to write, test, and deploy rules hasn’t historically matched the speed at which new attack patterns appear. Writing an effective detection rule from scratch requires deep familiarity with the query language, the field schema, and the aggregation logic needed to express the behavior you are trying to catch, before you even begin thinking about the threat itself. For most security teams, that friction means a growing backlog and gaps in coverage that attackers can exploit.

Arming detection engineers with native, AI-powered tooling isn’t just about convenience; it’s also about keeping pace with an adversary that’s already using AI to move faster. Elastic Security is now adding AI rule creation, powered by the Elastic Agent Builder. Unlike external AI tooling or stand-alone code generation workflows, this capability is built into the detection engineering experience: The rule is created and validated, with results preview generated entirely within your platform, against your own data, without leaving Elastic Security. Analysts can now describe what they want to detect in natural language and receive a complete, ready-to-review ES|QL rule in return, without leaving the rule creation workflow. This capability is available at the Enterprise license tier.

Support for ES|QL rule creation is available now. Additional rule types are on the roadmap, so keep an eye on upcoming releases as these capabilities expand.

Detections without the heavy lifting

ES|QL, Elastic's pipeline query language, is very helpful for behavioral and aggregation-based detections. Its pipe-based syntax makes it natural to express the kind of "filter, count, group by, threshold" logic that underlies most modern detections: How many failed logins came from this IP? Which accounts were targeted? Does this count exceed the expected baseline?

That same expressiveness is also what makes ES|QL harder to write by hand than a simple field-match query. You need to think in terms of pipelines: Filter first with WHERE, aggregate with STATS...BY , and then filter again on the computed values. It requires knowing the right Elastic Common Schema (ECS) field names, the correct function syntax, and how the pipeline stages interact. This is exactly the kind of structured, pattern-based logic that AI can translate reliably from a plain English description.

With the new detection engineering skills and the knowledge of Elastic documentation, ECS field definitions, and local data access, the Elastic AI Agent uses detection engineering best practices to come up with the rule, and moreover, the generated rule query is validated before it’s returned: What you see in the editor will run.

Walkthrough: Detecting Okta credential stuffing and account takeover

A credential stuffing attack that succeeds in breaching an account doesn’t stop at the login. The full attack chain (multifactor authentication [MFA] bypass, session establishment, privilege escalation, and policy modification) leaves a distinct footprint across Okta system logs if you know what to correlate. This is exactly the kind of multistage behavioral pattern that ES|QL handles well: Collect all the relevant event types, classify each one, aggregate by the shared identity attributes, and then apply threshold logic that requires the full sequence to be present before alerting.

Writing that query manually means knowing the Okta-specific event action names, knowing how to use EVAL with CASE to create per-event type flags, and how to then aggregate those flags with SUM to count each stage independently. It’s a realistic but nontrivial query, exactly the kind that benefits most from AI generation.

Imagine your team has an Okta integration and logs are coming in. Threat intelligence has flagged an active campaign targeting Okta tenants: automated credential stuffing followed by MFA fatigue and post-compromise privilege changes. You need detection coverage today. 

Note: We’re skipping the step of checking whether prebuilt detection rules exist or are already enabled, for the simplicity of the scenario here.

Opening the AI Agent rule creation flow

From the Elastic Security sidebar, navigate to Detection rules and click Create a rule -> AI rule creation. 

Describing the detection in plain language

No special syntax is required. Describe the full attack chain the way you would explain it to a colleague, including the data source and the specific event sequence you want to match:

Analyst prompt:

In Okta, detect when the same user and source IP shows: three or more failed logins due to bad credentials, at least one MFA failure, then a successful login, and then either a privilege grant or a policy update. That full sequence together is a credential stuffing attack that succeeded.

The AI Agent processes this against its knowledge base, including Okta integration field mappings and ECS conventions, and executes multiple steps that we can follow and review:

And then it returns a complete ES|QL rule that covers the full attack sequence described.

Reviewing and adjusting the generated rule logic

There’s a lot happening in this rule’s query, and it’s worth understanding each stage, because the structure itself tells the story of the attack chain.

The query is concise by design: It uses ES|QL's inline WHERE filtering inside COUNT() to compute each stage of the attack chain in a single STATS pass, without needing a separate EVAL block. Here’s what each part does:

FROM logs-okta* scopes the query to all Okta log indices, using a wildcard that picks up logs-okta.system-* and any other Okta data streams in the environment.

The STATS block is the core of the detection. It aggregates all Okta activity and computes four counters per unique combination of user.name and source.ip, one for each stage of the attack chain. failed_logins counts user.session.start events with outcome: failure (the password spray attempts). mfa_failurescounts failed MFA challenges, indicating the attacker encountered a second factor and attempted to push through it.

successful_logins counts user.session.start events with outcome: success; a value of one or more means the attacker got in. post_compromise_events counts any of six actions that indicate the attacker is acting on their objective after login: adding the account to a group, granting application access, escalating privileges, modifying a policy lifecycle, updating a policy rule, or changing the account profile. This is a broad net that covers the full range of post-compromise behavior seen in Okta account takeover incidents.

The WHERE clause after the aggregation requires all four conditions to be true simultaneously before a row becomes an alert. This is what makes the rule high-fidelity. A user who forgot their password and eventually logged in won’t match because they’ll have no post-compromise events. An attacker who got through but took no further action won’t match either. All four stages must be present.

The KEEP statement trims the output to the six fields that matter for triage (the targeted account, the source IP, and the count for each stage), giving the responding analyst everything they need to start an investigation without querying the raw logs first.

Along with the query, the AI Agent generates the following rule metadata: rule name, description, severity and risk score recommendations, MITRE ATT&CK technique and tactic mapping (T1110.004 Credential Stuffing, T1078 Valid Accounts, ), execution schedule, and tags. Where other rules exist, the AI Agent also reuses relevant tags from those rules, so new custom rules stay consistent with your existing detection library from the start. The data source is selected from indexes available in the system, or data ingestion is suggested. The rule fields are editable with an AI Agent before or after filling the resulting rule information in the rule creation form.

Tip: You can also ask the AI Agent to explain an existing rule query, suggest threshold adjustments based on a description of your environment, or help troubleshoot unexpected results.

Let’s keep working on the rule to adjust a few things. We want to ensure the rule detects credential stuffing and not other failure reasons, like expired passwords or locked accounts. We want to ensure the attack sequence is preserved.

Using AI Agent, we’ll ask it to fix these few things. It comes back with the adjusted query and summarizes what it did.

We now apply the changes to the rule form from the AI Agent chat:

Previewing and enabling the rule

Before enabling, use the Preview rule results panel to run the query against recent data in your environment. Any existing matches will surface immediately, running against your actual Okta log data in your Elastic deployment (no sample data, no sandbox, no external validation step required), useful both for validating that the query logic is correct and for checking whether an attack may already be in progress in your Okta tenant.

In this example, we’ve added sample logs to get a single alert generated:

Now, satisfied with the results, we’ll enable the rule. It will begin executing on its configured schedule and generate alerts for any user and source IP combination where the full attack sequence is observed within the query window.

If we execute the rule manually for the past week to find any past attacks and check resulting alerts, we see the same alert we’ve gotten in the Rule Preview.

Note: AI-generated rules should be reviewed before deployment in production environments. The AI Agent may not have full awareness of your specific data schema, log source quirks, or environment-specific baseline behavior. Use the rule preview to validate against your actual data before enabling. 

Impact on detection engineering workflows

The walk-through above, from opening the rule creation form to having a validated, multistage, MITRE-mapped ES|QL rule covering the full Okta account takeover chain, takes a few minutes. Writing the same query manually would require knowing the Okta-specific event action names, the correct okta.outcome.reason field and its enumerated values, how to structure EVAL with CASE to produce per-stage flags, how to aggregate those flags with SUM rather than COUNT, and how to express a compound post-compromise condition using OR across two aggregated fields. For an analyst onboarding a new data source under time pressure, that’s a significant amount of context to hold simultaneously.

The AI Agent doesn’t replace detection expertise. The analyst still makes every meaningful decision: which event types constitute the attack chain, what thresholds make sense for their environment, and whether the preview results look correct. What changes is the time it takes to get from having threat knowledge to having a working rule. Engineers who understand the attack and can describe it iterate and get a production-quality query back quicker, rather than spending time on implementation mechanics.

This matters most at the moments when speed is most critical: when a new campaign is active, when a data source has just been onboarded, or when an existing rule needs rapid refinement because the threat has evolved. AI-powered attackers aren’t waiting for your rule backlog to clear. Detection engineering tooling shouldn’t require it either.

What's next

AI rule creation for ES|QL is the first step in a broader expansion of AI Agent-driven detection engineering in Elastic Security. ES|QL was the natural starting point given its aggregation-first pipeline structure, which maps cleanly to the behavioral descriptions analysts naturally provide. Support for additional rule types and additional quality of life rule creation steps and beyond is on the roadmap. Keep an eye on the Elastic Security Labs blog and release notes for updates as new capabilities become available.

For a broader look at how AI Agents are reshaping the detection engineering role, from threat modeling and telemetry tuning through to rule authoring and maintenance at scale, see Supercharge Your SOC: Detection Engineering in the Era of AI Agents on Elastic Security Labs. For a comprehensive overview of the full detection engineering toolset available in Elastic Security today, including prebuilt rules, alert suppression, MITRE ATT&CK coverage, and Detections as Code, see Know your tools: The full range of Elastic Security's detection engineering capabilities.

Try the new AI rule creation capability on your deployment, or start a free trial. Connect with us on Elastic's community Slack to share feedback or tell us what detection use cases you’re building and how we can help.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third-party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third-party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

This article highlights how Elastic's new Terraform resources for security detection rules and exceptions expand practitioners' capabilities for detection-as-code deployment. Below you will find examples of defining and deploying your detection artifacts in Elastic Security with Terraform. We will also show how you can use Elastic's AI Agent to help quickly create the Terraform configuration for your custom rules. Finally, it also provides guidance on when to use the Elastic Stack Terraform provider versus tools from the detection-rules repository.

Managing Elastic with Terraform

Terraform is a tool created by HashiCorp (now IBM) to manage infrastructure in the cloud, or in self-managed environments, as code. With a simple stroke of HCL (HashiCorp configuration language), users can define the desired state of their cloud provider infrastructure, application, configuration, and in Elastic’s case, cluster settings, configuration, indices or streams, and now also detection rules and alerts, as fully configurable, traceable, and reviewable code in your favorite source management tool.

The Elastic Stack Terraform provider helps search, observability, and security professionals, as well as DevOps and SREs, configure their Elastic clusters with the right indices and mappings for their search use cases, SLOs or Fleet policies for their observability use case, and now, detection rules, alerts, and exceptions for their security use case. It can easily configure those, and many more objects and settings in the Elastic Stack.

Security Detection rules - now as code with Terraform

With V0.12.0 and V0.13.0 of the Elastic Stack Terraform provider, users can now manage their Detection rules and rule exceptions using Terraform. This is especially useful for users who have already been managing their Elastic deployments with Terraform and want to extend it to Detection rules.

Using the Elastic Stack Terraform Provider to deploy Rules and Exceptions

Let's look at an example of using the Elastic Stack Terraform Provider to deploy an Elastic Security Rule. In this example, we want to detect Windows Service Accounts that are performing an interactive logon on a host.

Service accounts typically have elevated privileges and rarely-rotated passwords, making them high-value targets for attackers. Since these accounts should only perform automated service logons, an interactive logon can indicate credential theft or misuse.

The first thing we need to think of is what telemetry we need to see which logons are happening on our host. Logon events are logged by the Windows Local Security Authority Subsystem Service (LSASS) whenever a logon session is successfully created on the machine. We can pick this up via an Elastic Agent with the Windows Integration installed.

The data will be written by the Elastic Agent into the system.security Data Stream, we can match it with this index pattern: logs-system.security-*. We also know that Logon events generate event code 4624 and that, in our example, the service account name starts with svc or ends with $. In addition, an interactive login will have a logon type of interactive.

So, we can match these events with an ES|QL rule like:

FROM logs-system.security-\*  
| WHERE event.code \== "4624" AND (user.name LIKE "svc\_\*" OR user.name LIKE "svc-\*"  
     OR user.name LIKE "\*\_svc" OR user.name LIKE "\*$")  
     AND winlog.logon.type IN ("Interactive", "RemoteInteractive",  
         "CachedInteractive", "CachedRemoteInteractive")  

There may be situations where we don't want this rule to run, for example, if there is a legacy application that we want to permit interactive logons from. So, we can create an Exception Item, like: user.name IS svc\_sqlbackup.

Now that we know what we want the Rule and its Exceptions to look like, we can use the Terraform provider's elasticstack_kibana_security_detection_rule, elasticstack_kibana_security_exception_list, and elasticstack_kibana_security_exception_item resources to define them in code.

Turning ES|QL rules into Terraform's configuration syntax, HCL, is a great use case for Elastic's AI Agent.
Elastic AI Agent capabilities help accelerate security operations across a wide range of tasks - from alerts triage and incident response to helping with detection lifecycle tasks.

Simply open AI Agent, and ask it to create Terraform configurations based on your query and exceptions.

You should end up with something like this:

Here's a closer look at the code.

There are a few elements to call out specifically:

• type: The type of exception list. For example: detection, endpoint, or endpoint_trusted_apps
• namespace_type: Determines whether the exception list is available in all Kibana spaces or just the single space in which it was created.

resource "elasticstack_kibana_security_exception_list" "svc_account_interactive_login" {
  list_id        = "svc-account-interactive-login-exceptions"
  name           = "Service Account Interactive Login Exceptions"
  description    = "Documented exceptions for service accounts that legitimately require interactive logon"
  type           = "detection"
  namespace_type = "single"
  tags           = ["service-accounts","windows","authentication"]
}  

This creates a new exception list.

Of note, the entries array contains the conditions under which the exception applies.

resource "elasticstack_kibana_security_exception_item" "svc_sqlbackup" {
  list_id        = elasticstack_kibana_security_exception_list.svc_account_interactive_login.list_id
  item_id        = "svc-sqlbackup-exception"
  name           = "svc_sqlbackup - Legacy SQL Backup Agent"
  description    = "Approved exception: Legacy SQL backup agent requires interactive logon per vendor documentation."
  type           = "simple"
  namespace_type = "single"
  tags           = ["sql","backup","approved"]
entries = [
    {
      field    = "user.name"
      type     = "match"
      operator = "included"
      value    = "svc_sqlbackup"
    }
  ]
} 

This adds our exception: that we don't want the rule to run if the username is svc\_sqlbackup.

Of note, the elements from enabled to the technique array are examples of the other properties that can be set on a rule.

resource "elasticstack_kibana_security_detection_rule" "svc_account_interactive_login" {
  name        = "Service Account Interactive Login"
  description = <<-EOT
    Detects interactive logins by service accounts. Service accounts should authenticate
    via service (Type 5) or batch (Type 4) logon types, not interactively. Interactive
    logins by service accounts may indicate credential theft or misuse.

    This rule identifies service accounts by common naming conventions (svc_*, svc-*,
    *_svc) and managed service accounts (*$).
  EOT

  type     = "esql"
  language = "esql"
  query    = <<-EOT
    FROM logs-system.security-* metadata _id, _version, _index
    | WHERE event.code == "4624"
      AND (user.name LIKE "svc_*" OR user.name LIKE "svc-*" OR user.name LIKE "*_svc" OR user.name LIKE "*$")
      AND winlog.logon.type IN ("Interactive", "RemoteInteractive", "CachedInteractive", "CachedRemoteInteractive")
    | KEEP @timestamp, host.name, user.name, user.domain, winlog.logon.type, source.ip, _id, _version, _index
  EOT

  enabled    = true 
  severity   = "high"
  risk_score = 73

  from     = "now-6m"
  to       = "now"
  interval = "5m"

  author  = ["Security Team"]
  license = "Elastic License v2"
  tags    = [
    "Domain: Endpoint",
    "OS: Windows",
    "Use Case: Identity and Access Audit",
    "Tactic: Initial Access",
    "Data Source: Windows Security Event Log"
  ]

  false_positives = [
    "Service accounts with documented exceptions that require interactive logon",
    "Break-glass procedures during incident response",
    "Initial service account configuration or troubleshooting"
  ]

  references = [
    "https://learn.microsoft.com/en-us/entra/architecture/service-accounts-on-premises",
    "https://blog.quest.com/10-microsoft-service-account-best-practices/",
    "https://attack.mitre.org/techniques/T1078/002/"
  ]

  threat = [
    {
      framework = "MITRE ATT&CK"
      tactic = {
        id        = "TA0001"
        name      = "Initial Access"
        reference = "https://attack.mitre.org/tactics/TA0001/"
      }
      technique = [
        {
          id        = "T1078"
          name      = "Valid Accounts"
          reference = "https://attack.mitre.org/techniques/T1078/"
          subtechnique = [
            {
              id        = "T1078.002"
              name      = "Domain Accounts"
              reference = "https://attack.mitre.org/techniques/T1078/002/"
            }
          ]
        }
      ]
    }
  ]

  exceptions_list = [
    {
      id             = elasticstack_kibana_security_exception_list.svc_account_interactive_login.id
      list_id        = elasticstack_kibana_security_exception_list.svc_account_interactive_login.list_id
      namespace_type = elasticstack_kibana_security_exception_list.svc_account_interactive_login.namespace_type
      type           = elasticstack_kibana_security_exception_list.svc_account_interactive_login.type
    }
  ]
}

Finally, we define the rule, including the ES|QL query we provided earlier and MITRE ATT&CK classification.

You can add these resource definitions into one configuration file (perhaps security-rules.tf), add it to your configured Elastic Stack terraform directory, and then run the `terraform apply` command and parameter to deploy the Rule.

terraform apply --auto-approve

Since terraform apply runs a plan before making changes, it will automatically detect if anyone has edited a rule directly in Kibana and show you exactly what drifted: no manual exports or diffs needed.

After Terraform has made the changes, we can see the Rule in Kibana:

We can also see the Exception List:

This way, you can define your detections in Terraform and benefit from automatic deployment along with other objects you manage with Terraform.

Terraform workspaces for multi-space Elastic deployments

Terraform uses a concept called “workspaces” allowing you to reuse the same infrastructure code for multiple deployments, for example, a dev, testing, and production environment. This concept is useful for managing rules across multiple deployments and/or Kibana spaces.

Managing detections with Terraform and Detections as code

Elastic also has Detections as Code functionality available via our open detection-rules repository.

The two tools have complementary strengths and are aligned with different user profiles and workflow stages for implementing Detections as Code.

Detection as Code features in detection-rules

• Best fit user profile: Detection engineers
• Intended workflow phase: Rule authoring and validation

With dual-sync between your GitHub repo and Kibana, linting, schema validation, and unit-testing, detection-rules functionality is well-suited to experienced Detection Engineers comfortable with Git-based version control.

Elastic Stack Terraform Provider

• Best fit user profile: DevOps engineers / Platform teams
• Intended workflow phase: Deployment and operations

For users already using Terraform to manage their Elastic clusters, the Terraform Provider is a great fit, bringing consistency to all "x-as-code" operations and familiar state management and parameterization.

The key differences and optimal use cases for each tool are detailed in the comparison table below:

In short, Detection Engineers are better served by the specialized creation and testing tools provided in the detection-rules repository, while DevOps/Platform Teams should use the Terraform provider to manage detection rules as part of their broader infrastructure-as-code strategy for deployment and operations.

Try it out

To experience the full benefits of what Elastic has to offer for detection engineers, upgrade to 9.3 or start your Elastic Security free trial. Visit elastic.co/security to learn more and get started.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

DaC as a methodology applies software development practices to the creation, management, and deployment of security detection rules. By treating detection rules as code, it enables version control, automated testing, and deployment processes, enhancing collaboration, consistency, and agility in response to threats. DaC streamlines the detection rule lifecycle, ensuring high-quality detections through peer reviews and automated tests. This methodology also supports compliance with change management requirements and fosters a mature security posture.

That's why we’re excited to share the latest updates to Elastic's detection-rules, our open repository for writing, testing, and managing security detection rules in Elastic, that also allows you to create your own Detections as Code (DaC) framework. Continue reading for highlighted implementation examples using extended functionality, and the announcement of Elastic's free Detections as Code Workshop.

Elastic Security DaC: The journey from alpha to general availability

With the functionality now provided in detection-rules repository, users can manage all their detection rules as code, review rule tunings, automatically test and validate rules, and automate rules deployment across their environments.

Pre-2024: Elastic’s internal use of DaC

Elastic threat research and detection engineering team created and used the detection-rules repository to develop, test, manage and release prebuilt rules, following DaC principles - reviewing rules as a team, automating their tests and release. The repository also has an interactive CLI to create rules, so engineers could start working on the rules right there.

As the security community's interest in as-code principles grew, and the available Elastic Security APIs already allowed users to implement their custom Detections as code solutions, Elastic decided to extend the detection-rules repository functionality to enable our users to benefit from our tooling and aid them in creating their DaC processes.

Here are the key milestones of Elastic’s user-focused DaC development from alpha to general availability.

May 2024: Alpha release of new "roll your own” features

Our detection-rules repository is adjusted for customer use, allowing for managing custom rules, adapting the test suite for user needs, and allowing for management of actions and exceptions alongside the rules.

Key additions:

• Custom rules directory support
• Select which test to run based on your requirements
• Exceptions and Actions support

We also published an extensive guidance for Detections as Code with examples of implementation with Elastic Security using detection-rules repository.

August 2024: "Roll your own” features now beta

The functionality is extended to allow import and export of custom rules between Elastic Security and repository, more configuration options and versioning functionality extended to custom rules.

New features added:

• Bulk import/export of custom rules (based on Elastic Security APIs)
• Fully configurable unit test, validation, and schemas
• Version lock for custom rules

March - August 2025: are generally available and supported

Using DaC with Elastic Security 8.18 and up:

• Supports prebuilt rules management. You can export all prebuilt rules from Elastic Security and store them alongside your custom rules.
• Support for rules filtering for export added.

Adjacent to DaC efforts, we also released new Terraform resources (V0.12.0 and V0.13.0) in October-December 2025, allowing Terraform users to manage detection rules and exceptions.

With this foundation spelled out, let's explore the powerful features that are available to streamline your detection engineering process.

Detection-rules DaC functionality highlights

There are a few worthwhile additions since our last DaC publication, which we’ll expand on below.

Additional filters

The filter functionality available when exporting rules from Kibana has been extended to allow you to precisely define which rules to sync in DaC. Here are the new flags:

Let’s take an example of when you wish to organize rules by data source, and want to export the AWS rules to a specific folder. In this case, let’s use filtering on tags for data sources and export all rules with the Data Source AWS tag:

python -m detection_rules kibana export-rules -d dac_test/rules #add rules to the dac_test/rules folder
-sv #strip the version fields from all rules
-cro #export only custom rules
-eq "alert.attributes.tags: "Data Source: AWS"" # export only rules with "Data Source: AWS" tag

See Kibana documentation for query string filtering for the underlying API call used here and the list all detection rules API call for example available fields to construct the query filter.

Custom folder structure

In the detection-rules repo, we use a folder structure based on platform, integration, and MITRE ATT&CK information. This helps us with our organization and rule development. This is by no means the only method of organization. You may want to organize your rules by customer, date, or source as examples. This will vary greatly depending on your use case.

Whether you use this export process or manual organization, once you have your rules in a location or folder structure that you like, you can now keep this local structure even when re-exporting rules. It is important to note that the new rules need to be placed in their desired location manually. The local rule-loading mechanism detects where the rules are placed in order to know where to put them. If the rule is not there, it will then use the specified output directory to place the new rule(s). To use the local rule loading for updating existing rules use the --load-rule-loading / -lr flag for the kibana export-rules and import-rules-to-repo commands. These flags enable you to make use of the local folders specified in your config.yaml.

Let’s look at example with the rules organised in folders the following way:

rule_dirs:
- rules
my_test_rule.toml
- another_rules_dir
high_number_of_process_and_or_service_terminations.toml

We’ll specify the following in the config.yaml file:

rule_dirs:
- rules
- another_rules_dir

With the new -lr option, rule updates from Kibana will now use these additional paths instead of exporting directly to the specified directory.

Running python -m detection_rules kibana --space test_local export-rules -d dac_test/rules/ -sv -ac -e -lr,will export rules from test_local space, my_test_rule.toml will be written to dac_test/rules/ as it was already on disk there and high_number_of_process_and_or_service_terminations.toml will be written to dac_test/another_rules_dir/.

This can be particularly useful if you have the same rules in different sub-folder configurations for different customers. For example, let’s say you have your rules broken down by platform and integration similar to Elastic’s prebuilt rule folder structure. For your customers, SOCs, or threat-hunting teams, having the rules organized underneath these platform/integration folders may be the most useful mechanism for them to manage the rules. However, your information security team or primary detection engineering team may want to manage the rules by initiative or rule author instead so that all the rules a particular individual or team is responsible for are organized in one place. Now with the local rule-loading flags, you can simply have two configuration files and the duplicated rules in each structure. When you are exporting updates for the rules, you would then use the environment variable to select the appropriate configuration file and export the rule updates. These updates will then be applied to the rules in place, maintaining the directory structure.

Miscellaneous local loading updates

In addition to the above, we have added two smaller new features designed to help users who are adding local information in the detection rules TOML files and schema. These are as follows:

1. Local date support from the local files where the local date will be maintained from the original file
2. Upgrades to the auto gen feature to inherit known types from existing schema.

The local date component can be useful when one wants more manual control over the date field in the file. Without using the override, the date will be based on when the Kibana rule contents were exported. Using the --local-creation-date flag, the date will not be updated when the file contents are re-exported.

The automatic schema generation has been updated to inherit the types from other indices/integrations if they are present. This provides a potentially more accurate schema, as well as reducing the need for manual updates after the fact. For example, you have a rule that uses the index “new-integration*” with the following fields:

• host.os.type.new_field
• dll.Ext.relative_file_creation_time
• process.name.okta.thread

Instead of each of these fields being added to the schema with a default type, their types are inherited from existing schemas. In this case, the types for dll.Ext.relative_file_creation_time and process.name.okta.thread are inherited.

{
  "new-integration*": {
    "dll.Ext.relative_file_creation_time": "double",
    "host.os.type.new_field": "keyword",
    "process.name.okta.thread": "keyword"
  }
}

To see how to use this with your custom data types, see the Custom schemas usage section within the Implementation examples part of this blog.

Expanding on usage examples

Below you will find more examples of DaC implementations, these are not focused on new functionality additions, but go deeper on the topics we see discussed in the community.

It’s worth noting that Detections as Code features are provided as components that can be used to build a custom implementation for your chosen process and architecture. When implementing DaC in your production environment, treat it as an engineering process and follow the best practices.

DaC implementation with Gitlab

When we look at implementations of DaC typically this revolves around using some form of CI/CD product to automatically perform rule management based on a given trigger. These triggers vary considerably based on the desired setup, specifically the authoritative source of rules and the desired state of your version control system (VCS). For a much more in-depth exploration of some of these considerations, see our DaC Reference Material. Below is a simple example using Gitlab as VCS provider and using its in-built CI/CD via Gitlab Actions.

stages:                # Define the pipeline stages
  - sync               # Add a 'sync' stage

sync-to-production:    # Define a job named 'sync-to-production'
  stage: sync          # Assign this job to the 'sync' stage
  image: python:3.12   # Use the Python 3.12 Docker image
  variables:
    CUSTOM_RULES_DIR: $CUSTOM_RULES_DIR    # Set custom rules env var
  script:                                  # List of commands to run 
    - python -m pip install --upgrade pip  # Upgrade pip
    - pip cache purge                      # Clear pip cache
    - pip install .[dev]                   # Install package w/ dev deps
    - |  # Multi-line command to import rules                                        
      FLAGS="-d ${CUSTOM_RULES_DIR}/rules/ --overwrite -e -ac"
      python -m detection_rules kibana --space production import-rules $FLAGS
  environment:
    name: production   # Specify deployment environment as 'production'
  only:
    refs:
      - main           # Run this job only on the 'main' branch
    changes:
      - '**/*.toml'    # Run this job only if .toml files have changed

This is very similar to other inbuilt CI/CD from other Git-based VCS like Gitlab and Gitea. The main difference being in the syntax determining the triggering event. The DaC commands such as kibana import-rules would be the same regardless of VCS. In this example, we are syncing rules from our fork of the detection-rules repo to our Kibana Production Space. This is based on a number of prior decisions being made, for instance requiring unit tests to pass before merging rule updates and that rules on main being ready for prod. For a Github-based walkthrough of these considerations for this particular approach, please take a look at our demo video.

Custom Unit Testing tips and examples

When considering DaC as a capability to add to your detection toolkit, setting up the CI/CD and base infrastructure should be considered as the first step in an ongoing process to improve the quality and usefulness of your rules. One of the key purposes in having “as code” tooling is adding the ability to further customize tooling to your needs and environment.

One example of this is unit testing for rules. Beyond base functionality testing, some other key existing unit tests enforce Elastic-specific considerations around rule performance and optimization, as well as organization of metadata and tagging. This helps detection engineers and threat researchers remain consistent in their rule development. Building on this example, one may want to consider adding custom unit tests based on your specific needs.

To illustrate this, take a Security Operations Center (SOC) environment where there are a number of analysts responsible for various different domains and tasks. When an alert is raised in the SIEM, it may not be immediately obvious who should handle remediation, or what team(s) need to be informed of the incident. Tagging the rules with a team tag: e.g. Team: Windows Servers similarly to how Elastic uses tags for data sources, can provide the SOC with a point of contact directly in the alert for who can help with remediation.

In our DaC environment, we can quickly create a new testing module to enforce this on all of the custom rules (or pre-built too). For this test, we are going to enforce having a Team: <some name> tag on all production rules that are not authored by Elastic. In the detection-rules repo, our testing is handled through the Python test suite called pytest and as such unit tests are organized into python modules (files) and subsequent classes and functions in these files under the tests/ folder. To add tests simply either add classes or functions to the existing files or create a new one. In general, we recommend creating new test files so that you can receive updates to the existing tests from Elastic without having to merge the differences.

We will start by creating a new python file called test_custom_rules.py in the tests/ directory with the following contents:

# test_custom_rules.py

"""Unit Tests for Custom Rules."""

from .base import BaseRuleTest


class TestCustomRules(BaseRuleTest):
    """Test custom rules for given criteria."""

    def test_custom_rule_team_tag(self):
        """Unit test that all custom rules have a Team: <team_name> tag."""
        tag_format = "Team: <team_name>"
        for rule in self.all_rules:
            if "Elastic" not in rule.contents.data.author:
                tags = rule.contents.data.tags
                if tags:
                    self.assertTrue(
                        any(tag.startswith("Team: ") for tag in tags),
                        f"Custom rule {rule.contents.data.rule_id} does not have a {tag_format} tag",
                    )
                else:
                    raise AssertionError(
                        f"Custom rule {rule.contents.data.rule_id} does not have any tags, include a {tag_format} tag"
                    )

Now each non-Elastic rule will be required to have a tag in the specified pattern for a team responsible for remediation. E.g. Team: Team A.

Custom schemas usage

Elastic’s ability to bring your own data types also extends to our DaC capabilities. For example, let’s take a look at some custom schemas for network protocols. Diverse data you have in your stack can of course be queried by your rules, and we will also want to leverage the applicable validation and testing for any custom rules on these data types too. This is where Custom schemas come in handy.

When we are validating queries, the query is parsed into the respective fields and the types of these fields are compared against what is provided in a given schema (e.g. ECS schema, the AWS Integration for AWS data, etc.). For custom data types, this follows the same validation path, with the ability to pull from locally defined custom schemas. These schema files can be built by hand as one or more json files; however, if you have some sample data already in your stack, you can take advantage of this and use it as validation and generate your schemas automatically.

Assuming you already have a custom rules folder configured (if not see instructions), you can turn on automatic schema generation by adding auto_gen_schema_file: <path_to_your_json_file> to your config file. This will generate a schema file in the specified location that will be used to add entries for each field and index combination. The file will be updated during any command where rule contents are validated against a schema, including import-rules-to-repo, kibana export-rules, view-rule, and others. This will also automatically add it to your stack-schema-map.yaml file when using a custom rules directory and config.

With this power comes an increased responsibility on rule reviewers as any field used in the query is immediately assumed to be valid and added to the schema. One way to mitigate risk is to utilize a development space that has access to the data. In the PR, one can then link to a successful execution of the query with stack level validation on its data types. Once this is approved, one can remove the auto_gen_schema_file addition to the config and you now have a known valid schema based on your custom data. This provides a baseline for other rule authors to build upon as needed and maintains the type checking validation.

Learn more about DaC and try it yourself

You can experience Elastic Security's Detections as Code (DaC) functionality firsthand with our interactive Instruqt training. This training provides a straightforward way to explore core DaC features in a pre-configured test environment, eliminating the need for manual setup. Give it a try!

If you are implementing DaC, share your experience, ask your questions and help others on the community slack DaC channel.

Trial Elastic Security

To experience the full benefits of what Elastic has to offer for detection engineers, start your Elastic Security free trial. Visit elastic.co/security to learn more.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.