Elastic Security has opened its detection rules repository to the world. We will develop rules in the open alongside the community, and we’re welcoming your community-driven detections. This is an opportunity to share collective security knowledge.

<p>At Elastic, we believe in the <a href="https://www.elastic.co/about/why-open-source">power of open source</a> and understand the importance of community. By putting the community first, we ensure that we create the best possible product for our users. With Elastic Security, <a href="https://www.elastic.co/security">two of our core objectives</a> are to <em>stop threats at scale</em> and <em>arm every analyst</em>. Today, we’re opening up a new GitHub repository, <a href="https://github.com/elastic/detection-rules">elastic/detection-rules</a>, to work alongside the security community, stopping threats at a greater scale.
</p><p>The release of the <a href="https://www.elastic.co/blog/elastic-siem-detections">detection engine</a> in Elastic Security brought automated threat detection to the Elastic Stack. Since the initial launch of the detection engine, the Elastic Security Intelligence & Analytics team has added 50+ additional rules, increasing the visibility of attacker techniques on Linux, macOS, and Windows operating systems. As we continue to expand coverage, you’ll see increased breadth in detections, covering new domains such as cloud services and user behavior.
</p><p>Over the past few releases, we used an internal repository to manage rules for the detection engine. We’ve iteratively improved our testing procedures by adding automated tests for new contributions that validate Kibana Query Language (KQL) syntax, schema usage, and other metadata. Our rule development has matured, so we can move fast <em>without</em> breaking things.
</p><p>By opening up our <a href="https://github.com/elastic/detection-rules">elastic/detection-rules</a> GitHub repository, Elastic Security will develop rules in the open alongside the community, and we’re welcoming your community-driven detections. This is an opportunity for all of us to share our collective knowledge, learn from each other, and make an impact by working together.
</p> <strong><h2>What’s in this new repository?</h2></strong><p>In the <a href="https://github.com/elastic/detection-rules">elastic/detection-rules</a> GitHub repository, you can find rules written for Elastic Security, with coverage for many <a href="https://attack.mitre.org/">MITRE ATT&CK</a>® techniques. Our current rule logic is primarily written in <a href="https://www.elastic.co/guide/en/kibana/master/kuery-query.html">KQL</a>, and by leveraging the <a href="https://www.elastic.co/guide/en/ecs/current/index.html">Elastic Common Schema (ECS)</a>, we only need to write rules once. By using the defined fields and categories in ECS, rules automatically work with Beats logs and other data sources that map properly to ECS.
</p><p>Within the <a href="https://github.com/elastic/detection-rules/tree/main/rules">rules/</a> folder, rules are stored in TOML files and are grouped by platform. We tried to keep it simple with a flat hierarchy so that it’s easier to find and add new rules. If you’re looking for Windows-only rules, navigate to <a href="https://github.com/elastic/detection-rules/tree/main/rules/windows">rules/windows</a>. If you’re still struggling to find a rule or want to search across rules, you can use our CLI by running the command <code>python -m detection_rules rule-search</code>, which will show the files that have matching metadata.
</p><p>Every rule contains several fields of metadata in addition to the query itself. This captures information like the title, description, noise level, ATT&CK mappings, tags, and the scheduling interval. We have a few additional fields to aid analysts performing triage, describing known false positives or helpful steps for an investigation. For more information on the metadata that pertains to rules, see the <a href="https://www.elastic.co/guide/en/siem/guide/current/rules-ui-create.html#create-rule-ui">Kibana rule creation guide</a> or our <a href="https://github.com/elastic/detection-rules/blob/main/CONTRIBUTING.md#rule-metadata">summary of rule metadata</a> in the contribution guide.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt683eae32569f7543/5efb61e38304ac0dbc4a5a0e/detection-rules-repo-blog-msbuild.png" data-sys-asset-uid="blt683eae32569f7543" alt="detection-rules-repo-blog-msbuild.png">
</p><figcaption>Preview of the file behind the “MsBuild Making Network Connections” rule
</figcaption><p> <strong></strong>
</p><h2><strong>How do these rules get to my detection engine?</strong></h2><p>If you’re using our <a href="https://www.elastic.co/cloud/">Elastic Cloud managed service</a> or the default distribution of the Elastic Stack software that includes the <a href="https://www.elastic.co/subscriptions">full set of free features</a>, you’ll get the latest rules the first time you navigate to the detection engine. When you upgrade, the detection engine recognizes that rules were added or changed and <a href="https://www.elastic.co/guide/en/siem/guide/current/rules-ui-create.html#load-prebuilt-rules">prompts</a> you to decide whether you want those rules upgraded. Follow the steps after upgrading and you’ll get the latest copy of the rules.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1ca938b9d35957ee/5efb6250f715ab0f6bc3dea1/detection-rules-repo-blog-msbuild-network-connections.png" data-sys-asset-uid="blt1ca938b9d35957ee" alt="detection-rules-repo-blog-msbuild-network-connections.png">
</p><figcaption>The same rule — “MsBuild Making Network Connections” — loaded in the detection engine
</figcaption><p> <strong></strong>
</p><h2><strong>Who will use this repository?</strong></h2><p>This repository is where the Elastic Security Intelligence & Analytics team will develop rules, create issues, manage pull requests, and target releases. By making the repo public, we’re inviting all external contributors into this workflow. This will give contributors visibility into our development process and a clear path for rules to be released with the detection engine.
</p><p>When you’re ready to contribute, please sign Elastic's <a href="https://www.elastic.co/contributor-agreement">contributor license agreement (CLA)</a>. This is standard for all Elastic GitHub repositories, and it means that we freely can distribute your code to Elastic users.
</p> <strong><h2>How do we approach threat detection?</h2></strong><p>In general, we tend to prefer detections that focus on adversary behaviors. This usually means that we focus on ATT&CK techniques. This might mean more research and effort is needed to figure out how a technique works before creating a rule. But by taking this approach, we do a better job detecting and stopping the attacks of today and tomorrow, instead of just the attacks of yesterday.
</p><p>Taking a behavioral approach also means we need various types of rules. Some might detect atomic events, others may require aggregating multiple events or looking for deviances above a threshold. With <a href="https://eql.readthedocs.io">Event Query Language (EQL)</a>, we will be able to write rules that look for sequences of behavior that span multiple events.
</p><p>Of course, we understand that sometimes a technique can be hard for all users to detect behaviorally. In that case, by all means, feel free to add a rule that is more signature-like in nature and written towards a specific behavior or tool.
</p><p>For a longer discussion on what makes a mature detection, read about the philosophy <a href="https://github.com/elastic/detection-rules/tree/main/PHILOSOPHY.md">of the detection rules repository</a>.
</p> <strong><h2>Why do we need a new repository?</h2></strong><p>If you’ve shared public rules before, you’re probably aware of other well-known GitHub repositories, like the <a href="https://github.com/mitre-attack/car">Cyber Analytics Repository (CAR)</a> by MITRE, <a href="https://github.com/Neo23x0/sigma">Sigma</a>, or even the <a href="https://eqllib.readthedocs.io">EQL Analytics Library</a> based on Elastic’s EQL. You might be wondering: <em>Why do we need another repository? Why not use the ones that already exist?</em>
</p><p>Unsurprisingly, the best answer to this question starts with another question: <em>Why do the other repositories exist?</em> Both CAR and Sigma are purposefully agnostic of language or platform, and sometimes data source. On the other hand, the EQL Analytics Library was written with a specific language in mind.
</p><p>With our new detection rules repository, we’re trying to serve a slightly different purpose. Our goal is to give users of Elastic Security the best possible detections that work across various data sources. We use ECS as the great equalizer of schemas, making it possible to write a rule once that applies to multiple data sources.
</p><p>Since the Elastic Stack supports multiple languages, our rules should reflect that. Query languages are typically developed to solve different types of problems, and we shouldn’t constrain rule developers to a single language if another does the job better. We currently have <a href="https://www.elastic.co/guide/en/kibana/master/kuery-query.html">KQL</a> and <a href="https://www.elastic.co/guide/en/kibana/current/lucene-query.html">Lucene</a> rules, along with rules that use <a href="https://www.elastic.co/guide/en/machine-learning/6.8/ml-jobs.html">machine learning anomaly detection</a> jobs in the repository. We’re <a href="https://github.com/elastic/elasticsearch/issues/49581">working hard</a> to bring <a href="https://eql.readthedocs.io">EQL</a> to the Elastic Stack and into our repository.
</p><p>We can also ensure best practices are being followed for optimal Elasticsearch performance. For example, searching for <code>process.path:*\\cmd.exe</code> requires performing a wildcard check, which is more costly than a simple keyword check. Instead of searches containing leading wildcards, we can recommend using <code>process.name:cmd.exe</code>, which will result in better performance and the most accurate results. On a similar note, ECS also contains the field <code>process.args</code>, which is a parsed version of <code>process.command_line</code>. We recommend using the parsed field instead, because it gives us better performance and means that we are much less prone to <a href="https://github.com/elastic/detection-rules/blob/main/PHILOSOPHY.md#does-a-rule-have-trivial-evasions">trivial whitespace or quotation-based evasions</a>. Win-win.
</p> <strong><h2>Can I add rules from another repository?</h2></strong><p>Within your own environment, you’re welcome to add rules to your own detection engine as long as your Kibana role has the right permissions. If you want to add rules to the <a href="https://github.com/elastic/detection-rules">elastic/detection-rules</a> repository, the answer is an unsurprising: <a href="https://www.elastic.co/blog/it-depends"><em>It depends...</em></a><em>. </em>As long as a rule can be sublicensed under the Elastic License, this is fair game. Most of the time, the requirements are fairly straightforward — retain the original authors in the <code>rule.author</code> array, and update the <code>NOTICE.txt</code> file accordingly to give attribution to the original authors. We don’t want to take credit for someone else’s work, so please help us be thorough!
</p><p>For more information on how we approach licensing in the repository, check the <a href="https://github.com/elastic/detection-rules#licensing">Licensing</a> section of the README.
</p> <strong><h2>How do I contribute?</h2></strong><p>Eager to share your rule logic? Hop on over to <a href="https://github.com/elastic/detection-rules">elastic/detection-rules</a> on GitHub. We have detailed instructions there for navigating the repository, forking and cloning, and creating a rule. We include a command line tool for bulk editing the files and to make creating new rules easier. When you’re ready to add a new rule to the repository, run <code>python -m detection_rules create-rule</code>, and you’ll be prompted for the required metadata. We recommend using the CLI when possible, because it reduces copy-and-paste errors that happen when reusing contents from a TOML file from another rule or template.
</p><p>When your rule is in a good state, you can run the command <code>python -m detection_rules test</code> to locally perform unit tests, which validate syntax, schema usage, etc. Then, create the pull request and someone on the Intelligence & Analytics team will review the contribution. If we request any changes, we’ll work with you to make the recommended changes.&nbsp;
</p><p>If you have a good idea for a rule, but want to collaborate with us on the idea or get feedback, feel free to create a <a href="https://github.com/elastic/detection-rules/issues/new/choose">New&nbsp;Rule</a> issue. We look forward to helping and brainstorming with you!
</p><p>For more information, check out the <a href="https://github.com/elastic/detection-rules/tree/main/CONTRIBUTING.md">contribution</a> guide.
</p> <strong><h2>What’s next?</h2></strong><p>Welcome to our new rules repository and workflow! Your contributions are encouraged, and we look forward to seeing your name in the <code>rule.author</code> field. The detection rules repository will continue to evolve along with the rest of Elastic Security, and we’re excited for what’s next.
</p><p>If you want to track the development of EQL in the stack, subscribe to this <a href="https://github.com/elastic/elasticsearch/issues/49581">GitHub issue</a>. Or take a peek at the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/eql.html">ongoing documentation</a> to watch what the Elasticsearch team is up to.
</p><p>If you have any feedback or questions writing rules or navigating our new <a href="https://github.com/elastic/detection-rules">detection rules</a> GitHub repository, please <a href="https://github.com/elastic/detection-rules/issues/new/choose">create an issue</a> in GitHub, reach out on the <a href="https://discuss.elastic.co/c/security">discussion forum</a> with the <em>detection-rules</em> tag, or find us in the <em>#detection-rules</em> channel of the <a href="http://ela.st/slack">Elastic Slack Community</a>.
</p>

blog-banner-gears-steel.jpg

blog-thumb-gears-steel.jpg

Elastic Security opens public detection rules repo

We’re excited to announce the new release of more EQL analytics and tooling, including an interactive shell, to make EQL even more usable and powerful.

<p>It has been an exciting summer in the security community for the Event Query Language (EQL) as we delivered presentations at <a href="https://circlecitycon.com/talks/the_hunter_games_how_to_find_the_adversary_with_event_query_language/">Circle City Con</a> and <a href="https://www.bsidessatx.com/presentations-2019.html">Bsides San Antonio</a>. These talks showcased creative ways to hunt for adversaries in your environment with EQL.&nbsp; If you couldn't make either of these events, we are sorry we missed you, but good news: EQL is making its way to <a href="https://www.blackhat.com/us-19/briefings/schedule/index.html#fantastic-red-team-attacks-and-how-to-find-them-16540">Blackhat USA</a> in a joint talk with <a href="https://redcanary.com/">Red Canary</a> called "Fantastic Red-Team Attacks and How to Find Them". We hope to see you there at the presentation or at Endgame’s booth (#1253) to talk more about behavior based detection with EQL.&nbsp; In advance of Blackhat, we’re excited to announce today’s release of more EQL analytics and tooling to make EQL even more usable and powerful.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1007e618f5f9e6af/5dfd0289c848085fa46e5d1c/eql-shell-summer-update-blog.png" data-sys-asset-uid="blt1007e618f5f9e6af" alt="eql-shell-summer-update-blog.png" style="display: block; margin: auto;">
</p><p>Before we review the exciting updates to EQL, let's pause for a quick recap. EQL was built by Endgame to express relationships between events. The language is data source and platform agnostic, can answer stateful questions, and enables the hunter by including data stacking pipes to sift through data. If you have structured data, you can start asking questions now.&nbsp; For more background, please see the following posts: <a href="https://www.endgame.com/blog/technical-blog/introducing-event-query-language">Introducing EQL,</a> <a href="https://www.endgame.com/blog/technical-blog/eql-for-the-masses">EQL For the Masses,</a> and <a href="https://www.endgame.com/blog/technical-blog/getting-started-eql">Getting Started with EQL.</a>
</p><p>With today’s <a href="https://github.com/endgameinc/eql/blob/master/CHANGELOG.md#version-07">update to 0.7,</a> we added an interactive shell so you can better explore your own data. It will dynamically learn your schema, and includes tab-completion, syntax highlighting, table output, helpful error messages, and the ability to export results from a search to a CSV file.
</p><p>In addition to the improved EQL experience, we also added over 60 new analytics to our <a href="https://eqllib.readthedocs.io/">Analytics Library</a>.&nbsp; The goal of these new analytics is to help you enrich your data and gain context by matching high-noise MITRE ATT&CK™ techniques against your security events.&nbsp; Finally, we’ve updated our contribution guides, making it easier to contribute to the EQL community.&nbsp;
</p><h2>Interactive Shell</h2><p>We’re happy to make a powerful <a href="https://eql.readthedocs.io/en/latest/cli.html">shell</a> available to improve your experience with EQL&nbsp; It is included with EQL, so after you install EQL 0.7 (pip install eql), you can use the shell to quickly explore your own data set. It has syntax highlighting and supports queries that span multiple lines. You can define a schema or teach it to learn your schema in order to validate your queries.&nbsp; We provided a powerful table viewer to allow for readability and flexibility of query results. With the shell, you can tab complete through EQL keywords, EQL functions, and even your own data fields.&nbsp;
</p><p>Let's show some examples using the interactive shell by running the command eql. First, here’s a quick video.
</p><p><a href="https://asciinema.org/a/dQHwBytDOpemwRQI6gUc95fng"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9af24d46e7b8ee7d/5dfd02ae5f3f935f8592fb68/endgame-eql-shell-demo-blog.png" data-sys-asset-uid="blt9af24d46e7b8ee7d" alt="endgame-eql-shell-demo-blog.png" style="display: block; margin: auto;"></a>
</p><figcaption>A quick video showing the EQL Shell. Showing features ranging from importing data, tab completion, type/field/syntax checking, table outputs, and more!</figcaption><p>Let’s highlight a couple features. For instance, the tab complete is very useful for quickly building your EQL query or knowing the available fields from your data.<br>
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1bb622a1ce59ee72/5dfd02d2631b8140b1f03fe0/eql-shell-tab-complete-blog.png" data-sys-asset-uid="blt1bb622a1ce59ee72" alt="eql-shell-tab-complete-blog.png" style="display: block; margin: auto;">
</p><figcaption>Tab completion of EQL functions and data fields within EQL Shell</figcaption><p>Additionally, the table is a powerful utility to quickly customize your EQL match results. During the alert triage process, it can be helpful to quickly make pivot tables and narrow your investigation.<br>
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt84236b7ac8906870/5dfd02e094b1d7403bbd35c5/eql-shell-table-results-blog.png" data-sys-asset-uid="blt84236b7ac8906870" alt="eql-shell-table-results-blog.png" style="display: block; margin: auto;">
</p><figcaption>EQL table showing query results.</figcaption><p>Check out the documentation for more information and ways to use the EQL shell.&nbsp; We hope you enjoy it as much as we do!
</p><h2>Improved Query Validation</h2><p>In the Endgame product, we recently gave our users the ability to <a href="https://www.endgame.com/news/press-releases/endgame-introduces-reflex">create their own realtime detections with EQL</a>. As part of this work, we updated EQL to include a system for detecting query errors (this functionality is available in the EQL shell as well). If you try to create a query with an unknown function, event type or field, EQL will tell you exactly where you went wrong, with a helpful message indicating the root problem. We also added a type system to the internals of EQL, and now you'll get more concise and useful errors if you pass the wrong type to a function, or try to compare across types.<br>
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt6086a2992d6d971e/5dfd02fbf5916f4381bc7ede/eql-shell-error-highlight-blog.png" data-sys-asset-uid="blt6086a2992d6d971e" alt="eql-shell-error-highlight-blog.png" style="display: block; margin: auto;">
</p><figcaption>Schema and type validation in core EQL. The first example showing an error for the field, cmdline, as it did not exist in our data and the second showing a type error, as the length function expected an array/string.</figcaption><h2>Enriching with EQL</h2><p>Security isn’t always about alerting. Contextualizing an alert to further comprehend what’s happening is a great way to help security analysts go from reacting to yet another alert to understanding a potential incident. We can provide better context by enriching logs with ATT&CK™ tactics and techniques.&nbsp; This isn't the same thing as positively identifying something as malicious, but serves to enhance raw data with critical information for performing triage or finding trends in lower confidence detections&nbsp;
</p><p>Often attackers use native-os tools for their own purposes, tradecraft often known as living off the land.&nbsp; It can become difficult for defenders to separate malicious intent from benign usage of an administrative tool, but regardless of the intent, execution of these tools is noteworthy.&nbsp; By enriching this data when it is seen, we may notice some seemingly benign actions representing more nefarious intent when viewed collectively. Or, defenders may use labels to prioritize and rank event collections that include certain enrichments. Either way, the crux is labeling security data, and we provide a solution with EQL.
</p><p>We are releasing over 60 analytics to our the <a href="https://eqllib.readthedocs.io/">EQL Analytics Library</a>.&nbsp; These are categorized as enrichments so you can start labeling your data or even start hunting for suspicious behaviors in your environment. Additionally, they are all mapped to <a href="https://attack.mitre.org/">MITRE ATT&CK</a>™!&nbsp; These queries are better enrichment labels than alerting detections.&nbsp; A majority have high potential for noise and could represent legitimate activity, but if you’ve been struggling to enhance your coverage of ATT&CK™ without success, this kind of enrichment can help you get a great deal closer.&nbsp; Going farther, if you understand normal baseline activity in your environment, then you could selectively choose which EQL queries could act as detections for you and use them accordingly.&nbsp;&nbsp;<br>
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt62cd270be5b1f277/5dfd032399f6f64041a29f05/eql-shell-eqllib-enrichment-blog.png" data-sys-asset-uid="blt62cd270be5b1f277" alt="eql-shell-eqllib-enrichment-blog.png" style="display: block; margin: auto;">
</p><figcaption>An example of using eqllib to survey a dataset and return analytic matches. We could use this as a means to prioritize our triage by count.</figcaption><p>Endgame provides enrichments to our users.&nbsp; Attack timelines which come back with alerts are enriched with additional ATT&CKTM-based context displayed to the user.&nbsp; This helps our users understand their alerts and take appropriate actions.<br>
</p><p><a href="https://pages.endgame.com/rs/627-YBU-612/images/endgame-enrichment.png"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4275db8f344c2b4e/5dfd033794b1d7403bbd35cb/eql-shell-endgame-enrichment-blog.png" data-sys-asset-uid="blt4275db8f344c2b4e" alt="eql-shell-endgame-enrichment-blog.png" style="display: block; margin: auto;"></a>
</p><figcaption>Enrichment in the Endgame platform (click image for larger view)</figcaption><h2>How to Contribute to EQL</h2><p>And finally, we want to collaborate with you, so we updated our contribution guides (<a href="https://github.com/endgameinc/eql/blob/master/CONTRIBUTING.md">EQL</a> and the <a href="https://github.com/endgameinc/eqllib/blob/master/CONTRIBUTING.md">Analytics Library</a>). Our intention is to provide a clear path for you to contribute to the <a href="https://eqllib.readthedocs.io/en/latest/index.html">EQL Analytics Library</a> or the <a href="https://eql.readthedocs.io/en/latest/index.html">language directly</a>. We recommend everyone following basic git development workflows by creating branches and submitting pull requests, but we added plenty of useful help docs to assist. We also created templates to make the process as painless as possible.
</p><p>We pledge to be timely in our review of all submitted work. And of course, always feel free to reach out and communicate with the EQL team via chat on <a href="https://gitter.im/eventquerylang/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge">Gitter</a>, on Twitter (<a href="https://twitter.com/eventquerylang?lang=en">@eventquerylang</a>) or with Github issues.
</p><p>We hope to work with you soon!
</p>

blog-banner-generic-blue.png

blog-thumb-generic-blue.png

EQL’s highway to shell

We created the Event Query Language (EQL) for hunting and real-time detection, with a simple syntax that helps practitioners express complex queries.

<p>Adversarial activity is no longer described purely in terms of static Indicators of Compromise (IOCs). Focusing solely on IOCs leads to detections which are brittle and ineffective at discovering unknown attacks, because adversaries modify toolkits to easily evade indicator-based detections. Instead, practitioners need durable detections based on malicious behaviors. MITRE’s <a href="https://attack.mitre.org/wiki/Main_Page">ATT&CK</a>&nbsp;framework helps practitioners focus their defensive tradecraft on these malicious behaviors. By organizing adversary tradecraft and behaviors into a matrix of tactics and techniques, ATT&CKTM is ideal to progress detection beyond IOCs and toward behavior-based detection.
</p><p>With a comprehensive and robust model of adversarial behavior in place, the next step is to build an architecture for event collection that supports hunting and real-time detection, along with a language that promotes usability and performance. We created the Event Query Language (EQL) for hunting and real-time detection, with a simple syntax that helps practitioners express complex queries without a high barrier to entry. I’ll discuss the motivation behind EQL, how it fits into the overall Endgame architecture, and provide several examples of EQL in action to demonstrate its power to drive hunting and real-time detection of adversarial behavior.
</p><h2>Simplifying Complex Queries</h2><p>Many database and search platforms are cumbersome and unintuitive, with complex syntax and a high barrier to entry. Detecting suspicious behavior requires analysis of multiple data sources and seamless data collection, aggregation, and analysis capabilities. Searching and exploring data should be intuitive and iterative, with the flexibility to fine-tune questions and hone in on suspicious behavior.
</p><p>Within the Endgame platform, we create functionality to overcome these data analysis and detection challenges. Our goal is to empower users without overwhelming them. That’s why we initially created <a href="https://www.endgame.com/blog/technical-blog/artemis-intelligent-assistant-cyber-defense">Artemis</a>, which enables analysts to search using natural English, with intuitive phrases like “Find the process wmic.exe running on active Windows endpoints.” However, analysts also often want to search for things that are difficult to describe clearly and concisely in English. We knew that addressing this next challenge would require a new query language that supports our unique architecture and equips users to find suspicious activity. Our solution, EQL, balances usability while significantly expanding capabilities for both hunting and detection. It can be used to answer complex questions without burdening users with the inner workings of joins, transactions, aggregations, or state management that accompany many database solutions and analysis frameworks. EQL has proven effective, and we’re excited to introduce it to the community to drive real-time detection.
</p><h1>Endgame’s Eventing Architecture</h1><p>Several architectural decisions in the Endgame platform played a pivotal role in the design and development of EQL. In the age of big data, Endgame takes an efficient and distributed approach to event collection, enrichment, storage, and analysis. Most solutions require data to be forwarded to the cloud to perform analysis, generate alerts, and take action. This introduces a delay for detection, increases network bandwidth utilization, and most importantly, implies that disconnected endpoints are less protected.
</p><p>With the Endgame platform, live monitoring, collection, and analysis happen where the action happens: on the endpoint. This endpoint-focused architecture allows for <a href="https://www.endgame.com/news/press-releases/endgame-26-achieves-unprecedented-stopping-power-995-prevention-efficacy-85">rapid search at scale</a> while minimizing bandwidth, cloud storage, and time spent waiting for results. It also enables process and endpoint enrichment, unique forms of stateful analysis—like tracking process lineage—and autonomous decisions without any need for platform or cloud connectivity. When an endpoint detects suspicious behavior, it makes a prevention or detection decision and alerts the platform of suspicious activity with the corresponding events. This decision happens without requiring a round trip to the Endgame management platform or the cloud, assuring that disconnected and connected endpoints are equally protected from suspicious behavior.
</p><p>These architectural decisions drove the logical next step: structuring a language that optimizes these capabilities. As we developed EQL, we aimed to create a language that is accomodating to users, optimized for our architecture, and shareable for defenders.
</p><h2>Designing a Language</h2><p>We wanted to ensure EQL supported sophisticated questions within a familiar syntax to limit the learning curve and maximize functionality. EQL provides abstractions that allow a user to perform stateful queries, identify sequences of events, track process ancestry, join across multiple data sources, and perform stacking. In designing EQL, we focused first on exposing the underlying data schema. Every collected event consists of an event type and set of properties. For example, a process event has fields such as the process identifier (PID), name, time, command line, parent information, and a subtype to differentiate creation and termination events. At the most basic level, an event query matches an <em>event type</em> to a <em>condition</em> based on some Boolean logic to compare the fields. The <code>where</code> keyword is used to tie these two together in a query. Conditionals are combined with Boolean operators <code>and</code>, <code>or</code>, and <code>not</code>; comparisons <code>&lt;</code>, <code>&lt;=</code>, <code>==</code>, <code>!=</code>, <code>&gt;=</code>, <code>&gt;</code>, and <code>in</code>; and function calls. Numbers and strings are expressed easily, and wildcards (<code>*</code>) are supported. All of this leads to a simple syntax that should feel similar to Python.
</p><p>When searching for a single event, it is easy to express many English questions with EQL in a readable, streamlined syntax. For example, the question:
</p><p><em>What unique outgoing IPv4 network destinations were reached by svchost.exe to port 1337 with the IP blocks 192.168.0.0/16 or 172.16.0.0/16?</em>
</p><p>Is expressed as an EQL:
</p><pre class="prettyprint">network where  
  event_subtype_full == "ipv4_connection_attempt_event" and  
  process_name == "svchost.exe" and  
  destination_port == 1337 and 
  (destination_address == "192.168.*" or destination_address == "172.16.*") 
| unique destination_address destination_port
</pre><p>EQL supports searches for multiple related events that are chained together with a sequence of event queries as well as post-processing similar to unix-pipes |. Defenders can assemble these components to define simple or sophisticated behaviors without needing to understand the lower-level mechanics. These building blocks are assembled to build powerful hunting analytics and real-time detections in the Endgame platform.
</p><h2>Applying EQL&nbsp;</h2><p>In the Endgame platform, when valid EQL is entered, the query is compiled and sent to the endpoints which then execute the query and return results. This happens quickly and in parallel, allowing users to immediately see results. Let’s take a look at how EQL is expressed in the following scenarios.
</p><h3>IOC SEARCH</h3><p>IOC searching isn’t hunting, but it is an important piece of many organizations’ daily security operations. Endgame users can express a simple IOC search in EQL:
</p><pre class="prettyprint">process where sha256=="551d62be381a429bb594c263fc01e8cc9f80bda97ac3787244ef16e3b0c05589"
</pre><p>Many Endgame users choose to use Artemis’ natural language capabilities for this sort of search by simply asking, "Search processes for 551d62be381a429bb594c263fc01e8cc9f80bda97ac3787244ef16e3b0c05589".
</p><h3>TIME BOUND SEARCHES</h3><p>There are many times during incident response in which it is useful to know everything that happened at a specific time on an endpoint. Using a special event type called <code>any</code>, which matches against all events, EQL can match every event within a five-minute window.
</p><p><em>What events occurred between 12 PM UTC and 12:05 PM UTC on April 1, 2018?</em>
</p><pre class="prettyprint">any where timestamp_utc &gt;= "2018-04-01 12:00:0Z" and timestamp_utc &lt;= "2018-04-01 12:05:0Z"
</pre><h3>STACKING ON THE ENDPOINT</h3><p>We can filter and process our data as it is searched, without having to pull all the raw results back for stacking, establishing situational awareness, and identifying outlier activity. The language provides data pipes, which are similar to unix pipes, but instead of manipulating lines of input, a data pipe receives a stream of events, performs processing, and outputs another stream of events. Supported pipes enable you to:
</p><ul>
	<li>Count the number of times something has occurred</li>
</ul><pre class="prettyprint">| count &lt;expr&gt;,&lt;expr&gt;,...
</pre><ul>
	<li>Output events that meet a Boolean condition</li>
</ul><pre class="prettyprint">| filter &lt;condition&gt;
</pre><ul>
	<li>Output the first N events</li>
</ul><pre class="prettyprint">| head &lt;number&gt;
</pre><ul>
	<li>Output events in ascending order</li>
</ul><pre class="prettyprint">| sort &lt;expr&gt;, &lt;expr&gt;, …
</pre><ul>
	<li>Output the last N events</li>
</ul><pre class="prettyprint">| tail &lt;number&gt;
</pre><ul>
	<li>Remove duplicates that share properties</li>
</ul><pre class="prettyprint">| unique &lt;expr&gt;,&lt;expr&gt;,...
</pre><p><em>Which users ran multiple distinct enumeration commands?</em> <br>
</p><pre class="prettyprint">join by user_name
    [process where process_name=="whoami.exe"]
    [process where process_name=="hostname.exe"]
    [process where process_name=="tasklist.exe"]
    [process where process_name=="ipconfig.exe"]
    [process where process_name=="net.exe"]
| unique user_name
</pre><p><em>Which network destinations first occurred after May 1st?</em>
</p><pre class="prettyprint">network where event_subtype_full=="ipv4_connection_attempt_event"
| unique destination_address, destination_port
| filter timestamp_utc &gt;= "2018-05-01"
</pre><h3>SEQUENCE OF EVENTS</h3><p>Many behaviors aren’t atomic and span multiple events. To define an ordered series of events, most query languages require elaborate joins or transactions, but EQL provides a sequence construct. Every item in the sequence is described with an event query between square brackets <code>[&lt;event query&gt;]</code>. Sequences can optionally be constrained to a timespan with the syntax <code>with maxspan=&lt;duration&gt;</code>, or expire with the syntax <code>until [&lt;event query&gt;]</code>, or match values with the <code>by</code> keyword.
</p><p><em>What files were created by non-system users, first ran as a non-system process, and later ran as a system-level process within an hour?</em>
</p><pre class="prettyprint">sequence with maxspan=1h
  [file where event_subtype_full=="file_create_event" and user_name!="SYSTEM"] by 
file_path
  [process where user_name!="SYSTEM"] by process_path
  [process where user_name=="SYSTEM"] by process_path
</pre><h3>PROCESS ANCESTRY</h3><p>The language can even express relationships in a process ancestry. This could be used to look for anomalies that may have normal parent-child relationships, but are chained together in a suspicious way. To check if a process has a certain ancestor, the syntax <code>descendant of [&lt;ancestor query&gt;]</code> is used.
</p><p><em>Did any descendant processes of Word ever create or modify any executable files in system32?</em>
</p><pre class="prettyprint">file where file_path=="C:\\Windows\\System32\\*" and file_name=="*.exe" and descendant of [process where process_name=="WINWORD.exe"]
</pre><p>Process ancestry relationships also support nesting and Boolean logic, facilitating rigorous queries. This helps hone in on specific activity and filter out noise.
</p><p><em>Did net.exe run from a PowerShell instance that made network activity and wasn’t a descendant of NoisyService.exe?</em>
</p><pre class="prettyprint">process where process_name=="net.exe" and 
  descendant of [network where process_name=="powershell.exe" and
      not descendant of [process where process_name == "NoisyService.exe"]]
</pre><h3>FUNCTION CALLS</h3><p>Functions can extend EQL capabilities by defining and exposing new functions without having to change any syntax. The <code>length()</code> function is useful for finding suspicious and rare PowerShell command lines, for example:
</p><p><em>What are the unique long Powershell command lines with suspicious arguments?</em>
</p><pre class="prettyprint">process where process_name in ("powershell.exe", "pwsh.exe") and length(command_line) &gt; 400 and (command_line=="*enc*" or command_line=="*IO.MemoryStream*" or command_line == "*iex*" or command_line=="* -e* *bypass*")
| unique command_line
</pre><h3>SORTING WITH THRESHOLDS</h3><p>EQL can also be used for performing outlier analysis. The pipes <code>sort</code> and <code>tail</code> filter data on the endpoint so only outliers are returned. With a search constructed this way, there is a well-defined upper bound on how many results are returned by an endpoint. That means there is less bandwidth used, no need for number crunching after the fact, and less time sifting through results. In other words, you don’t have to obtain and store data you don’t need.
</p><p><em>What top five outgoing network connections transmitted more than 100MB?</em>
</p><pre class="prettyprint">network where total_out_bytes &gt; 100000000
| sort total_out_bytes
| tail 5
</pre><h3>MACRO EXPANSION</h3><p>There is often logic that is shared between various queries. Multiple detections can utilize macros for code reuse and consistency. For instance, a macro could exist that identifies if a file is associated with system or network enumeration:
</p><pre class="prettyprint">    macro ENUM_COMMAND(name)
        name in ("whoami.exe", "hostname.exe", "ipconfig.exe", "net.exe", ...)
</pre><p>  
Once defined, macros are used and expanded with the function call syntax. One useful query for hunting may be to find enumeration commands that were spawned from a command shell that is traced back to an unsigned process:
</p><pre class="prettyprint">process where ENUM_COMMAND(process_name) and parent_process_name=="cmd.exe"
    and descendant of [process where signature_status=="noSignature"]
</pre><h2>Real-Time Detection With EQL</h2><p>Since historical searches and real-time analytics are both described with EQL, it’s easy to check new protections for noise before deployment. This is crucial because alert fatigue is one of the most common problems faced by the defender today. When creating a new analytic, Endgame researchers use a refinement process to filter out false positives.
</p><p>When detecting malicious behavior and attacker techniques, the first step is often detonating malware or a malicious script and collecting endpoint data. Once data is collected, events are explored with EQL and a new query is written. To establish a reasonable degree of confidence, we then evaluate the query against many sources of data, including Endgame’s internal network, partner data, and custom environments that are intentionally noisy. After passing these checks, a new tradecraft analytic expressed in EQL is enriched with metadata and converted to a format which Endgame’s detection engine understands. Finally, this machine representation of the query is loaded into the sensor, where new events are analyzed in real-time and an alert is generated immediately when a match is detected.
</p><h2>Writing Behavioral Malware Detections</h2><p>EQL is not limited by the underlying data. In fact, we use EQL in our malware detonation and analysis sandbox, &lt;a href="https://www.endgame.com/blog/executive-blog/endgame-arbiter-solving-now-what-problem"&gt;Arbiter&lt;/a&gt;® , which has a different underlying data schema. Expressing behavioral detections of malware with EQL in Arbiter® is painless, and our custom analysis engine performs orders of magnitude faster than other approaches we evaluated, allowing us to rapidly perform dynamic malware analysis and detect new behaviors.
</p><h3>PROCESS INJECTION DETECTION IN ARBITER</h3><p>A traditional <a href="https://attack.mitre.org/wiki/Technique/T1055">remote shellcode injection</a> technique uses several well documented Windows APIs to open a handle to a remote process, allocate memory, write to the newly allocated memory, and start a thread. The code generally looks something like:
</p><pre class="prettyprint">hVictim = OpenProcess(PROCESS_ALL_ACCESS, 0, victimPid);
lpDestAddress = VirtualAllocEx(hVictim, NULL, numBytes, MEM_COMMIT|MEM_RESERVE, PAGE_EXECUTE_READWRITE);
WriteProcessMemory(hVictim, lpDestAddress, lpSourceAddress, numBytes, NULL);
CreateRemoteThread(hVictim, NULL, 0, lpStart, NULL, NULL, NULL);
CloseHandle(hVictim);
</pre><p>  
To build an analytic using API events, the process handle needs to be correctly tracked. The handle 
	<code>hVictim</code> is first returned by <code>OpenProcess</code> and then used as an argument in the calls to <code>VirtualAllocEx</code>, <code>WriteProcessMemory</code>, and <code>CreateRemoteThread</code>. However, if and when <code>CloseHandle</code> is called, the handle is invalidated and all state for that handle needs to be thrown away, because it may be reused. It may sound complicated, but a stateful detection for Arbiter® is easy to create with EQL.
</p><pre class="prettyprint">sequence
  [api_ret  where function_name=="OpenProcess"] by return_value
  [api_call where function_name=="VirtualAllocEx"] by arguments.hProcess
  [api_call where function_name=="WriteProcessMemory"] by arguments.hProcess
  [api_call where function_name=="CreateRemoteThread"] by arguments.hProcess
until
  [api_call where function_name=="CloseHandle"] by hObject
</pre><p>One sequence is not enough to detect all forms of process injection. There are <a href="https://www.endgame.com/blog/technical-blog/ten-process-injection-techniques-technical-survey-common-and-trending-process">many methods</a> to gain arbitrary code execution in another process, each with different API calls, and each requiring another detection. Consequently, as the attacker’s playbook continues to evolve, defenders need to react quickly and find new ways to detect the latest techniques while simultaneously promoting layered detections.
</p><h2>Unifying Hunt and Detection</h2><p>With the Event Query Language, we can describe events that correspond to adversary techniques without dealing with the mechanics of traditional databases, rule engines, or an awkward data schema. Search, hunt, and detect are unified within the Endgame platform by EQL, where exploring events is made easy without sacrificing power and flexibility. Ultimately, EQL helps Endgame and Endgame customers quickly find suspicious activity and improve detections of attacker techniques defined in MITRE’s ATT&CKTM.
</p><p>Advancing our collective understanding and adoption of security tools is of utmost importance to combat today’s threats. At Endgame, sharing information about capabilities we’ve built and the underlying architecture which motivated our decisions is one way for us to do that. We look forward to discussions within the security community going forward about EQL and its value in driving advanced hunt and detection in a way that is performant, robust, and most importantly, empowering to defenders.
</p>

blog-banner-generic-black.png

blog-thumb-generic-black.png

Introducing Event Query Language

EQL is a language to express relationships between events and has the power to normalize your data regardless of data source and not constrained by platform.

Getting started with EQL

Event Query Language is an extensible, powerful language built in-house at Endgame to express relationships between security-relevant events.

<p>EQL, or the Event Query Language, is an elegant, powerful, and extensible language built in-house at Endgame to express relationships between security-relevant events. We designed it from the ground up to be generic, apply to multiple use cases, and avoid reliance on any particular architecture.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt55d6fabdc2152c65/5dfce2cebddecf4317a15059/eql-masses-logo-blog.png" data-sys-asset-uid="blt55d6fabdc2152c65" alt="eql-masses-logo-blog.png" style="display: block; margin: auto;"></p><p>EQL is part of the core technology that drives our endpoint security product. It powers high confidence detections that run on customer endpoints, it is used to perform basic searching, and it gives hunt teams the tooling necessary to sift through massive amounts of data across their endpoint data and detect intrusions. But, there’s nothing inherently tied to Endgame with the language itself.
</p><p>Up until now, the only way to use EQL was via the Endgame product. That changed today with the <a href="https://www.endgame.com/news/press-releases/endgame-announces-public-availability-eql">public release of EQL</a>. This release includes the <a href="https://github.com/endgameinc/eql">core EQL language</a>, a schema mapping to Sysmon, and a <a href="https://github.com/endgameinc/eqllib">set of analytics</a> initially focused on <a href="https://eqllib.readthedocs.io/en/latest/atomicblue.html">Atomic Blue</a>.
</p><h2>Why did we build and open source this?
</h2><p>It’s becoming a cliché, but it’s true: looking for signatures of known malware or infrastructure IOCs is helpful, but it’s not enough. Security practitioners must assume that adversaries have breached defenses and are conducting operations inside their networks. Because of this, security teams must not only worry about prevention, but also about how they can detect activity post-compromise.
</p><p>The security community has rallied around the MITRE ATT&CK framework as an important knowledge base of post-compromise adversary activity. We’ve been thrilled to see rapid growth in the number of security teams and researchers working to understand the detection opportunities and challenges in the post-compromise space. However, we note (as do many others) that the tools available to our community for universal expression of post-compromise analytics have some serious limitations which we’ll describe further below.
</p><p>Arguably the most significant challenge has been the coupling between everyone’s unique data sources and their analytics built on top of that data and corresponding schema(s). This has made it difficult to share actionable analytics between teams. After reading our introductory EQL <a href="https://www.endgame.com/blog/technical-blog/introducing-event-query-language">blogpost</a>, taking our product for a spin, or watching Endgame researchers <a href="https://www.youtube.com/watch?v=a3hIIzJrH14">talk about EQL</a>, a number of researchers began talking to us about how EQL could fill this gap. This matches internal observations we’ve had for over a year - that if we release it to the community, EQL will improve the community’s collective ability to express detection logic and share amongst teams.
</p><p>While EQL has been linked to the Endgame product, there are no inherent ties or dependencies. We of course natively support a mapping to our rich endpoint-centric dataset, robust query support, and automated detections via EQL in our powerful <a href="https://www.endgame.com/news/press-releases/endgame-expands-flexible-architecture-cloud-premises-and-hybrid-options">architecture</a>. However, mappings are possible to any security dataset, and we believe now is the time to allow people outside our customer base to use EQL.
</p><p>With this release, we are providing the <a href="https://github.com/endgameinc/eql">core language</a>, a sysmon integration, and a python-based EQL engine, which includes CLI functionality to run EQL queries over JSON. We’re also providing example analytics which are described later in this post. With this toolkit, users can immediately begin prototyping analytics in EQL.
</p><h2>EQL advantages
</h2><p>We are not here to criticize other technologies and projects which have sought to achieve goals like common schemas, querying structure, and other things similar to what we’ve done with EQL. EQL has some significant advantages which we hope cause people in the security community to take a closer look, even if other technologies are being used or considered. These include:
</p><ul>
	<li>Supporting the necessary logic to express relationships between security-relevant events. EQL can compare values and fields with exact or wildcard matching and supports basic AND, OR, NOT boolean search operations. Fields can be compared to strings, integers, decimal values, and against other fields. Most importantly, matching is enabled across a series of events including different types of events (e.g. process, file, and registry) rather than matching on only a single event.</li>
	<li>Minimal learning curve to write analytics. EQL looks like many other query languages. It is intended to search across structured data in an intuitive manner which is highly conducive to quickly writing behavioral analytics. This also leads to excellent readability of each analytic. It also supports traditional IOC searching, but EQL makes it easy to accurately describe activity and behavior beyond simple IOCs.</li>
	<li>No dependence on particular data sources or schema. Other technologies are tied closely to a given data source, and those writing analytics need to focus heavily on that particular data source and schema to write an analytic instead of just focusing on the logic they’re trying to express. EQL’s method of abstracting data sources via extensible schema mappings is powerful and allows for easy use without any need to pre-normalize data.</li>
	<li>Built to hunt. EQL includes strong native post-processing capabilities such as sorting, filtering, and stacking, which allow a user to easily filter out noise. Its schema translation capability also makes it straightforward to extend across multiple data sources without a need for data normalization. Data normalization into the universal schema is supported for users who want to eliminate the need for query-time normalization.</li>
</ul><h2>EQLLib and Atomic Blue
</h2><p>We’re providing robust <a href="https://eql.readthedocs.io/">documentation</a> on the many interesting EQL primitives and operators which are part of this release. We wanted to go further than just unleashing EQL on people in the form of a tool and <a href="https://eql.readthedocs.io/">documentation</a> alone. To that end, we have provided a rich set of analytics called <a href="https://eqllib.readthedocs.io/">EQLLib</a> to help people become familiar with the language and try things out.
</p><p>We’re huge fans of our partner Red Canary’s Atomic Red Team project. It is the most well-known and expansive test framework of adversary techniques out there. Adversary simulation projects are generally great in that they allow teams interested in detection aligned to the MITRE ATT&CK matrix to easily generate artifacts on systems.
</p><p>Atomic Red does a great job generating artifacts for the majority of techniques described by ATT&CK, but there’s no expansive mapping of Atomic Red into data source-agnostic language that a defender can action. <a href="https://eqllib.readthedocs.io/en/latest/atomicblue.html">Atomic Blue</a> starts to fill this gap. Atomic Blue is a curated set of EQL logic which describes how to find endpoint artifacts associated with execution of a significant number of techniques covered by Atomic Red Team.
</p><p>You can find Atomic Blue within the <a href="https://github.com/endgameinc/eqllib">EQLLib repository</a>. This initial analytics repository is significant, but we’ve only scratched the surface of what’s possible, and we look to the community to help us expand even further.
</p><h2>What’s next for this project?
</h2><p>This initial release is the beginning of the journey, not the final destination. Next week, you’ll see an expanded “how-to” guide along with some data to make it even easier to use EQL. To expand the applicability of EQL, we plan to release support for additional data sources and technologies in the coming months. You’ll also see us releasing additional analytics on a regular basis as the technology expands and evolves.
</p><p>We’re looking forward to feedback and contributions from others. We strongly believe that we’re collectively lacking a good way to describe detection logic universally across datasets, and EQL is a great way to address this and other limitations. Please try it out and let us know what you think.
</p>

EQL for the masses

Screen_Shot_2020-06-30_at_10.44.12_AM.jpg

Senior Security Research Engineer

Ross Wolf

Articles By

Register to Attend

Sign In to Attend

Register to Watch

You'll also receive an email with related content

Watch Now

Learn More

Contact Info

See All Posts

What to explore next...

Load more news

Load more press releases

Load More

View more learning opportunities

More

Date

Location

Agenda

Hosted by

Related workshops

View next

Translate Content

Articles by Ross Wolf

Elastic Security opens public detection rules repo

EQL’s highway to shell

Introducing Event Query Language

Getting started with EQL

EQL for the masses

Products & Solutions

Company

Resources

Featured topics

News

Articles by Ross Wolf

Elastic Security opens public detection rules repo

EQL’s highway to shell

Introducing Event Query Language

Getting started with EQL

EQL for the masses