Rolling your own Detections as Code with Elastic Security

By

Mika Ayenson,

Kseniia Ignatovych,

Justin Ibarra

Magnifying_glass.jpg

From its beginning, the Elastic detection-rules repo not only contained Elastic’s prebuilt detection rules, but also additional tooling for detection rule management — like a suite of tests, CLI commands, and automation scripts used by the Elastic Threat Research and Detection Engineering (TRaDE) team.

Elastic’s TRaDE team has been following Detections as Code (DaC) practices for years, supporting our internal development and release processes for Elastic prebuilt detection and endpoint rules. Customers historically have predominantly used the detection-rules CLI, which provided a Kibana API wrapper and CLI commands to facilitate rule management with the Elastic SIEM, but it required users to adopt Elastic’s internal processes and rigorous restrictions.

With DaC becoming more mainstream and continuing with our commitment to openness, we are working on making it easier for users to kick-start their own DaC process using Elastic’s detection-rules repo for their rules management. This DaC expansion will build upon prior detection rules features to provide an end-to-end experience for detection engineers.

This methodology enhances collaboration among security teams, streamlines updates, and facilitates a more agile response to evolving threats. Fundamentally, we are committed to improving detection engineering maturity for our customers by exposing more internal capabilities for them to embed within their processes.

Why Detections as Code?

If you’ve heard of DevOps concepts like Infrastructure as Code (IaC), then  Detections as Code should feel familiar. Where IaC focuses on managing and provisioning infrastructure via code, DaC applies similar principles to manage security detection rules as code. Detections as Code looks to adopt coding best practices in detection management, using peer review processes and tools and automated CI/CD pipelines. 

The benefits of DaC include high quality of detections, flexibility and scale of detections deployment, and compliance with change management requirements.

1 DAC LOCK

DaC adoption is driven by several factors:

  • Drive toward security team maturity: Implementing DaC encourages the development of mature, repeatable processes within security teams. It supports the maintenance of high-quality detections through systematic peer reviews and rigorous testing.

  • Ever-growing rule sets: Maintaining detection rules without DaC becomes untenable as the number of rules increases.

  • Expanding threat landscape: The breadth of coverage needed to protect against emerging threats necessitates a scalable approach suitable for diverse skill sets.

  • Broader adoption of automation: The wider movement toward automation in technology environments, exemplified by the adoption of IaC principles, calls for enhanced automation capabilities across all tools used by security teams. DaC aligns with this trend by integrating rule management into automated workflows, ensuring consistency, efficiency, and the ability to swiftly respond to new threats.

  • Compliance and governance requirements: Many organizations are adopting DaC to meet certification compliance requirements for peer review, change control, and disaster recovery of SIEM detections. Implementing DaC ensures that security rules are developed, reviewed, and maintained following strict governance standards, providing auditable evidence of compliance.

DaC approach and current development focus

Since DaC is really an approach, and our users have different infrastructure and process needs, there are multiple ways of architecting and implementing DaC with Elastic Security in your organization. But the key components remain the same, and users need: 

  • The Elastic Security solution 

  • A version control system of choice

  • A repository containing detections

  • Tooling to connect all the components, automate key processes, and provide a user interface

Our initial focus is on providing these components for customers to speed up DaC development, and in the first phase, we focused on detection-rules repo as the major building block.

Let’s go alpha!

We engaged with multiple users, understanding their needs and priorities concerning DaC. Based on the feedback we’ve received, the following enhancements were made: 

  • Enhancement 1: In the first stage, we focused on making the detection-rules repo easier to use for your custom rules management. We aimed to minimize merge conflicts for users when using a forked detection-rules repository for custom rules management. We now provide a config that specifies a custom rules directory for users to load using our built-in rules loader schema validators.

  • Enhancement 2: We made it possible to configure which of the Elastic-provided unit tests should run on your custom rules and which should be skipped. We understand that some customers would like to use some of our unit tests and follow our best practices, while other tests are designed specifically for Elastic rule management. You can now specify in a config file the tests you want to run.

  • Enhancement 3: We recognized the need to manage additional rule settings like exceptions and actions alongside the detections, so we added support to accommodate flexibly managing those in the repository.

Hierarchy and lexicon of concepts

One of the major outcomes of this initiative is the documentation of our thoughts and considerations. Even though we have refactored the detection-rules repo to be more flexible for external users, we recognize that DaC is not just a tool that can be installed. It is a fundamental shift in how security rules are managed. 

We’ve created a hierarchy and lexicon of concepts and documented them for you to consider before implementing any specific DaC workflows.

2 DaC high-level components
DaC high-level components

Within the reference documentation and slides, you can read more about the core components of DaC: 

  1. Maintaining rules within a Version Control System (VCS)

  2. Syncing rules from VCS to their respective platform

  3. Managing rules within the platform

  4. Syncing rules from the platform to VCS

Additionally, we’ve documented different models to govern the core components of DaC using either:

  1. VCS as authoritative

  2. Platform as authoritative

  3. Dual sync between VCS and the platform

Depending on the desired approach, we provide varying layers of detail, which becomes more abstract the further away from the Elastic Security solution.

What else are we providing?

Recently at BsidesOK24, we discussed Elastic’s approach to DaC and how you can take advantage of the detection-rules repo in our alpha branch. While we could break this up into a mini-series on how to implement DaC, these recommendations would be purely speculative as to how implementation would best fit your environment. Therefore we’re sharing the core principles of rolling your own detections as code.

3 BSides OK 2024
BSides OK 2024

Slides: These slides illustrate the high-level components, workflows, and delineations. It also touches on the benefits of using the detection-rules repo and a quickstart end-to-end reference example. 

Reference documentation: This reference document is packed with considerations, pros, and cons to developing, deploying, and managing detection rules, especially within Elastic Security environments. It is particularly beneficial for:

  • Security analysts who wish to leverage automation for more efficient rule management and to respond more rapidly to emerging threats

  • Detection engineers looking for methodologies to streamline detection logic development, testing, and deployment

  • Security team leads seeking to implement best practices for rule version control, collaboration, and quality assurance within their teams

  • DevOps engineers involved in integrating security practices into CI/CD pipelines, aiming for a more cohesive and automated approach to security within their workflows

  • IT security architects exploring ways to incorporate as-code principles into security operations to enhance agility, repeatability, and reliability

For folks who would like a more pointed direction, we do provide a bias to the detection-rules approach. Note that the alpha detection-rules branch, the content within these slides, and this reference guide are subject to change. Once we finally migrate the changes to the `main` branch, we will update the content accordingly.

Summary and next steps

In short, in the first stage we focused on making the detection-rules repo easier to use for your custom rules management. We aim to minimize merge conflicts for users when using a forked detection-rules repository for custom rules management. We’ve also started a DaC living guide with pro/con considerations to accompany the repo changes so you can hit the ground running hard. Lastly, we’ve started sharing our approach at BsidesOK with specific commands and use-cases that may apply to your custom setup.

Now that we’re in our alpha stage, we ask interested users to test it out on our detection-rules/DAC-feature branch and provide feedback!  

For more information, feel free to reach out to us on our Community Slack at #security-rules-dac or more directly on our detection-rules issue tracker.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial