Provide your SOC team with a unified, pre-built SIEM detection rule experience.

With the release of <a href="https://www.elastic.co/blog/elastic-security-7-6-0-released">Elastic Security 7.6</a>, we've announced our creation of&nbsp;a modern detection engine that provides SOC teams with a unified SIEM rule experience through&nbsp;Elastic SIEM detections.&nbsp;The&nbsp;detection engine draws from a purpose-built set of Elasticsearch analytics engines&nbsp;and runs on a new distributed execution platform in Kibana.&nbsp;In this post, we provide a brief overview of the flow of detections in Elastic SIEM and discuss the new UI and backend features that help these detections work seamlessly for our users.Before we jump into detections, a quick note: if you are ready to try the SIEM app, check out our <a href="https://www.elastic.co/blog/elastic-siem-for-small-business-and-home-1-getting-started">SIEM for small businesses and at home&nbsp;blog series</a>. The series&nbsp;covers getting set up on the cloud with our free <a href="https://www.elastic.co/products/elasticsearch/service">trial of Elasticsearch Service</a>, using Beats to securely collect and stream data from your systems to the SIEM, and more. (It’s far easier than you would think!) We also offer a <a href="https://www.elastic.co/guide/en/siem/guide/7.6/install-siem.html">getting started guide&nbsp;for hybrid deployments</a>. <h2>UI workflow for signal management
</h2>The bread and butter of Elastic SIEM detections&nbsp;are signals, which are Elasticsearch documents created when a signal detection rule's conditions are satisfied. In the simplest case, one signal document is created for each event matching the query defined in the rule. The signal document contains a copy of the fields from the matching document, and is kept in a separate signals index. The original events are not modified when a signal is created.Signals are surfaced in the SIEM app. When a practitioner first sees a new signal, it is in an open state. After analysis and determining the next steps, the practitioner changes it to a closed state. All of these changes can be managed in the Detections view in the SIEM app.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt828482255973f9c4/5e669becc39d9b5d36c3d124/siem_detections_histogram_sorted_risk_score.png"> 
 The signal&nbsp;count histogram shows open signals and enables quick comparisons across key attributes:<ul><li>Score, severity, type, name, or MITRE ATT&CK™ tactic name&nbsp;</li><li>Source or destination IP address</li><li>Event action or <a href="https://www.elastic.co/guide/en/ecs/1.4/ecs-allowed-values-event-category.html">category</a></li><li><a href="https://www.elastic.co/guide/en/ecs/1.4/ecs-allowed-values-event-category.html"></a>Host or user name</li></ul><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltebdcb7c7316b8bb7/5e669d474e03445d41facf35/siem_detections_signal_histogram.gif"> Investigating signals in the Timeline is the next step: <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4e14d9a0b0452596/5e669de00b333d44908d0e40/siem_detections_signals_timeline_view_in_timeline_with_id.png"> 
If you did not specify a timeline template when&nbsp;<a href="https://www.elastic.co/guide/en/siem/guide/7.6/rules-ui-create.html#create-rule-ui">creating a rule</a>, the Timeline is populated with a signal document. If you did specify a timeline template, the Timeline will be populated with what the user saved, speeding up investigations for certain types of rules. <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4ddf4e3f1426a1a2/5e669dad6d9a795c6ecf10d4/siem_detections_signals_timeline_view_in_timeline_with_template.png"> Practitioners can view alerts from external alerting systems, such as&nbsp;<a href="https://www.elastic.co/blog/introducing-elastic-endpoint-security">Elastic Endpoint Security</a>, Suricata, or Zeek, in the dedicated `External alerts` tab. Many organizations also implement rules generating signals for high-value external alerts so that they can benefit from the enhanced investigative workflow for signals.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt761ddf4865c94d43/5e669f706a12c038f19ec035/siem_detections_external_alerts_tab.png"> Once a signal or set of signals has been investigated to the analyst’s satisfaction, they can close the signals individually or in bulk. Signals can also be reopened, if necessary. We’re working on ways to automate the closing of signals in future releases. <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltda92d9e0aceff2b7/5e669fc018b328555145dd83/siem_detections_signals_close_signal_hover.png"> <h2 style="background-color: rgb(255, 255, 255);"> </h2><h2 style="background-color: rgb(255, 255, 255);">UI workflow for creation of rules</h2>For signals to start showing up, detections need rules to run! Creating a rule for&nbsp;SIEM detections is simple and straightforward. It comes down to three basic steps:1) Generate the query to be used each time the rule runs. This query can be Lucene syntax, <a href="https://www.elastic.co/guide/en/kibana/7.6/kuery-query.html">KQL</a>, a saved search, or the query can be imported from a saved timeline (with many more options for rule queries currently under development for future release):<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc1a892690f4dc394/5e66a04363a80b5c6fc5a1bf/siem_detections_create_rule_custom_query.png"> 2) Add some information describing the rule (title, description, etc.): <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt950d4ac86c74a8c6/5e66a09b6d9a795c6ecf10da/siem_detections_create_about_rule.png"> 3) Schedule the interval the rule should run at and any&nbsp;<a href="https://www.elastic.co/guide/en/siem/guide/7.6/rules-ui-create.html#create-rule-ui">additional look-back time</a>&nbsp;for sanity checks. We generally recommend some amount of look-back time to allow for delays that may occur in a given user's ingestion pipeline. We also recommend some look-back time as the rules are not guaranteed to run exactly at their scheduled interval and therefore might be delayed between runs. An overloaded task manager worker queue or insufficient <a href="https://www.elastic.co/blog/advanced-tuning-finding-and-fixing-slow-elasticsearch-queries">computing resources</a>&nbsp;can cause these delays. <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7175ab0a1e156ec1/5e66a0cc329c6c3829894d56/siem_detections_create_rule_schedule.png"> These three things are the basic components that make up a detections rule. We also provide settings to classify this rule according to MITRE ATT&CK tactics and techniques, as well as&nbsp;links to additional references. <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltadabd5e1e1fac099/5e66a11f63a80b5c6fc5a1c5/siem_detections_create_advanced_settings_blank_timeline.png"> Users can also perform actions on existing rules individually or in bulk, such as duplicate (for customizations), deactivate, export, and delete rules. We also have a guide for more information on&nbsp;<a href="https://www.elastic.co/guide/en/siem/guide/7.6/rules-ui-create.html">general rule management</a>. <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd9fc55f1c8850ce7/5e66a1540b333d44908d0e52/siem_detections_rules_deactivate_selected.png"> <h2 style="background-color: rgb(255, 255, 255);"> </h2><h2 style="background-color: rgb(255, 255, 255);">Prebuilt rules</h2>Rules can be difficult to develop and are time intensive to test. Because of this, detections started with 92 prebuilt rules developed by the Intelligence & Analytics team at Elastic Security&nbsp;and&nbsp;have&nbsp;<a href="https://www.elastic.co/blog/securing-our-own-endpoints-with-elastic-security">been used extensively at Elastic in a production environment</a>. New&nbsp;rules that respond to the latest critical threats are&nbsp;<a href="https://www.elastic.co/blog/playing-defense-against-gamaredon-group">continuously being developed</a>. Getting them loaded and ready to run is as easy as a click of a button! You can read more about using and tuning the pre-built rules&nbsp;<a href="https://www.elastic.co/guide/en/siem/guide/7.6/prebuilt-rules.html">here</a>.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte20be84367f8f502/5e66a17e18b328555145dd8b/siem_detections_signal_rules_load_prebuilt_rules_screen.png"> <h2 style="background-color: rgb(255, 255, 255);"> </h2><h2 style="background-color: rgb(255, 255, 255);">Detections implementation details</h2>Shortly after&nbsp;<a href="https://www.elastic.co/blog/alerting-in-the-elastic-stack">Alerting in the Elastic Stack</a>&nbsp;made its way into Kibana to provide support for alerts as first-class entities, Elastic SIEM utilized alerting for the foundation of detections. Behind the UI, detections uses an&nbsp;<a href="https://www.elastic.co/guide/en/siem/guide/7.6/rule-api-overview.html">API</a>&nbsp;layered on top of the&nbsp;<a href="https://github.com/elastic/kibana/blob/v7.6.0/x-pack/legacy/plugins/alerting/README.md">Alerting API</a>. The SIEM detections API brings convenience, workflows (such as opening and closing signals), the domain specifics of security (such as MITRE ATT&CK identification), and&nbsp;<a href="https://www.elastic.co/guide/en/kibana/7.6/kuery-query.html">KQL Support</a>.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfc2a6219ce8d8cd5/5e66a1a418b328555145dd91/siem_detections_rules_define_rule_saved_query.png"> 
Rules run behind the scenes by creating an API Key&nbsp;and then utilizing that API Key to make requests on behalf of the user using&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.6/search-request-body.html#request-body-search-search-after">search after</a>&nbsp;to find matching events and&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.6/docs-bulk.html">bulk create</a>&nbsp;to copy the information from the event into a signal document in the signal index. A signal is made up of the rule details and the details of the original event document matched by the rule. <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfc78b7cafe4517d9/5e66a1bf0b6249392652ce1d/siem_detections_api_key_screen.png"> 
If more than 100 matching documents are found in a single rule execution, only the last 100 matches —&nbsp;by descending `@timestamp` sort order —&nbsp;are copied to the signals index. The signals index is automatically created per Kibana space the first time you visit the signal detection rules page. The index name format is `.siem-signals-<space name="">`. For the default space or, if spaces are not enabled, the signals index name will be `.siem-signals-default`. Each signals index created for each space has an index&nbsp;<a href="https://github.com/elastic/kibana/blob/v7.6.0/x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/index/signals_policy.json">lifecycle management</a>&nbsp;setting of 50 GB or 30 days before it rolls over.&nbsp; Signals'&nbsp;indices&nbsp;are retained indefinitely.</space>The mapping of the SIEM signals index is a combination of&nbsp;<a href="https://github.com/elastic/kibana/blob/v7.6.0/x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/index/ecs_mapping.json">Elastic Common Schema (ECS)</a>&nbsp;and a custom mapping of our definition of what a&nbsp;<a href="https://github.com/elastic/kibana/blob/v7.6.0/x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/index/signals_mapping.json">signal is</a>. When a matching document is detected from the rule query, it will copy the fields over from the source indices&nbsp;and the resulting signal fields will be searchable if the fields in the source document are compliant with&nbsp;<a href="https://www.elastic.co/blog/introducing-the-elastic-common-schema">ECS</a>. If the fields from the source indices&nbsp;are not part of ECS, they will still be stored in the signal’s `_source` and viewable within Timeline and other parts of the application. However, they will not be searchable. <h2 style="background-color: rgb(255, 255, 255);">Scalability</h2>The detections UI is built atop the newly developed&nbsp;<a href="https://github.com/elastic/kibana/blob/v7.6.0/x-pack/legacy/plugins/alerting/README.md">Kibana Alerting framework</a>&nbsp;and the Kibana task manager. These two provide horizontal and vertical scaling capabilities, allowing flexibility that best suits whatever hardware is available at the time. Kibana&nbsp;<a href="https://github.com/elastic/kibana/blob/v7.6.0/x-pack/plugins/task_manager/server/README.md">task manager workers</a>&nbsp;can be increased in number to take advantage of vertical scaling, or can be replicated across separate Kibana instances and scaled horizontally.When multiple Kibana instances are running, the task managers will coordinate across the wire to balance the tasks across the instances. By updating the number of max_workers inside of the kibana.yml file from it’s default of 10, you can vertically scale up or down to appropriately allocate resources more efficiently per Kibana node. <h2 style="background-color: rgb(255, 255, 255);">Signal deduplication</h2>When a rule is running, it generates signals based on the events that it finds that match the rule’s query. Sometimes duplicate signals can be created either by overlapping queries in separate rules or by a rule running twice in a row and catching the same signal due to a long additional look-back time. To prevent a duplicate signal from appearing in the signals table, we identify signals based on the index the source event is from, the source event’s document id, the source event’s version number, and the running rule’s id. By hashing on&nbsp;<a href="https://github.com/elastic/kibana/blob/v7.6.0/x-pack/legacy/plugins/siem/server/lib/detection_engine/signals/utils.ts#L11-L19">these properties</a>, we ensure only unique signals are added to the signals index.<h2 style="background-color: rgb(255, 255, 255);">Errors</h2>Sometimes errors will appear due to a syntax error in a rule’s query or some other issue during a rule’s execution period. We bubble these up in the errors tab on the rule details page. We plan to expand the visibility of rule execution information and general rule monitoring in the future.&nbsp;<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltffc432e29392e581/5e66a1de0b6249392652ce23/siem_detections_error_string.png"> And here we can see the failure history, which displays the last five errors that occurred during rule execution: <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt92ce96e19860407b/5e66a1f6c39d9b5d36c3d13e/siem_detections_failure_history_tab.png"> 
<h2 style="background-color: rgb(255, 255, 255);"> </h2><h2 style="background-color: rgb(255, 255, 255);">SIEM detections of tomorrow</h2>The most exciting part of working on and releasing this Elastic SIEM detections beta is the early and continuous community feedback on the&nbsp;<a href="https://discuss.elastic.co/c/siem">Elastic SIEM discussion forum</a>&nbsp;and our&nbsp;<a href="https://github.com/elastic/kibana/issues?utf8=%E2%9C%93&q=is%3Aopen+is%3Aissue+label%3ATeam%3ASIEM+detection">open feature tracking list</a>.&nbsp;We have big plans for making&nbsp;detections even more powerful. Expanding rule queries to include aggregations, machine learning jobs, and EQL are just a few of them. If you think of something that is a great security use case, or want to ask a question or two about what is happening, please join us!

blog-gears-steel-banner.jpg

blog-gears-steel-thumb.jpg

Elastic SIEM detections

Devin Hurley

Articles By

Register to Attend

Sign In to Attend

Register to Watch

You'll also receive an email with related content

Watch Now

Learn More

Contact Info

See All Posts

What to explore next...

Load more news

Load more press releases

Load More

View more learning opportunities

Translate Content

Articles by Devin Hurley

Elastic SIEM detections

Products & Solutions

Company

Resources