Cyber threat intelligence is contextual information on emerging threats that can enhance the protective posture of contested environments. Learn how Elastic Security can help organizations establish robust cyber threat intelligence capabilities.

As a powerful search engine, Elasticsearch provides various ways to collect and enrich data with threat intel feeds, while the Elastic Security detection engine helps security analysts to detect alerts with threat indicator matching. In this blog post, we’ll provide an introduction to threat intelligence and demonstrate how Elastic Security can help organizations establish robust cyber threat intelligence (CTI) capabilities.
CTI is contextual information obtained through research and analysis of emerging threats and the application of defensive countermeasures to enhance the protective posture of contested environments. It helps organizations to better understand past, current, and future threats.&nbsp;
Threat feeds that can be obtained from various open source and commercial platforms become actionable “threat intelligence” when raw feeds are processed, parsed, and interpreted by security platforms to detect and understand the statistical breakdowns of the source and targets of attack activities.&nbsp;
 <h2>Threat intelligence in security operation centres (SOCs)</h2>Threat information is typically delivered via data feeds that can be of different types, such as file hashes, FQDN, IP addresses, URL reputation data, CVE, etc. — generally referred to as indicators of compromise (IOC) — and such intelligence has an important role in the SOC.&nbsp;
IOCs are the main deliverable for such tactical threat intelligence feeds and these are particularly useful for updating signature-based defence systems to defend against known attack types. IOCs also prove useful in proactive measures such as threat hunting.
The SOC team will typically be responsible for processing threat data and using it to add additional context to internal sources of data that are malicious. This is particularly valuable to SOC analysts that are responsible for analysing big data sets across SIEM platforms. Threat intelligence integration with SIEM detection capabilities helps with the prioritisation of alerts.
Threat intelligence integration with SIEM detection capabilities helps detect threats that might currently prevail inside the network through indicator lookups. Detections with known indicators increase the detection maturity in day-to-day security operations.
 <h2>Planning and collection of threat feeds</h2>Threat feeds from various sources can be of different types and in different forms. Methods to process and collect these feeds vary depending on the source platform but are mostly supported with APIs and standard file formats such as STIX, JSON, CSV, etc. Elastic, with its rich API-driven capabilities, provides a comprehensive data collection module for threat feeds from various sources.
Filebeat <a href="/guide/en/security/current/threat-intel-filebeat-module-v8.x-indicator-match.html">Threat Intel module</a> supports different threat feed streams, including open source feeds, and has rich flexibility in enabling and disabling feeds, filtering the types of feed, time intervals to pull the feeds and, importantly, it automatically deduplicates feeds from the same source. <a href="https://www.elastic.co/blog/ingesting-threat-data-with-threat-intel-filebeat-module">This blog post</a> covers the detailed steps for ingesting threat data with the Filebeat Threat Intel module.

<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte123059b3c083800/6115414acf33c613b2c05778/blog-threatintel-elasticsecruity-1.png" data-sys-asset-uid="blte123059b3c083800" alt="blog-threatintel-elasticsecruity-1.png">
Figure 1: Prebuilt threat intel dashboard - Alienvault OTX threat feed
<h2>Analytics with threat intelligence</h2>The data collected by threat feeds provide previously identified indicators of a potential compromise and can assist in improving the effectiveness of the security devices that can leverage this information to detect or even block these known threats.
Dynamic or static indicators (threat data) that are streamed into an Elasticsearch index can be used with multiple use cases such as:&nbsp;
<ul>
	<li aria-level="1">Detecting malicious events and traffic through indicator match rules</li>
	<li aria-level="1">Enriching incoming traffic to add additional context&nbsp;</li>
</ul> <h3>Indicator match detection rules</h3>The Detection Rules GitHub repository is the home for rules created by the community and the Elastic Intelligence and Analytics team. Elastic provides out-of-the-box <a href="https://www.elastic.co/guide/en/security/current/prebuilt-rules.html">prebuilt detection rules</a> addressing different threat use cases and this library of rules is continuously growing. Additionally, Elastic also provides the user with the ability to define and create their own detection rules to match their environmental use case requirements.
<a href="https://www.elastic.co/guide/en/security/current/rules-ui-create.html#create-indicator-rule">Indicator match rules</a> are one of the detection rule types used to generate alerts when network or host events are observed that match the data provided by threat feeds.&nbsp; The indicator match rule compares the field values of the specified event and the indicator field values in the threat intel index. When the field values are identical, an alert is generated in the detection engine.
Elastic detection provides various out-of-the-box detection rules, including threat intel detection. <a href="/guide/en/security/current/threat-intel-filebeat-module-v8.x-indicator-match.html">Threat Intel detection rule</a> is triggered when indicators from the threat intel Filebeat module have a match against local file or network observations.
Additionally, Elastic detection also provides the ability to <a href="https://www.elastic.co/guide/en/security/current/rules-ui-create.html">create your own detection rules</a> across various rule types.
As an example, let’s create an indicator match rule to identify a Network connection to a Malicious IP. With this indicator match rule, we can detect and generate an alert for outgoing network traffic to an IP address from a threat feed. When this alert is generated, this may indicate that the host generating the connection may be compromised or engaged in other suspicious user activity that could warrant further investigation.

<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5170b9cd16512d48/6115418b25695f14a34d6a8b/blog-threatintel-elasticsecurity-2.png" data-sys-asset-uid="blt5170b9cd16512d48" alt="blog-threatintel-elasticsecurity-2.png" "="">
Figure 2: Indicator match detection rule creation 
 
Index patterns:&nbsp; Elasticsearch indices where you would like this rule to run
[Multiple index patterns supported]
Custom query: Relevant data sets to be queried in the provided index pattern.
[In the above example: Query to look for any outbound network connection
event.category: network and network.direction : ("outbound" or "egress" ) ]
Indicator index pattern: Index pattern of the threat intel feeds
Indicator Index query: Relevant data sets to be queried from the threat intel index
[In the above example: Query to look for IPV4 indicator type
event.module : "threatintel" and threatintel.indicator.type : "ipv4-addr" ]
Indicator mapping: Fields to lookup between the data index and the threat intel index. In our example, we’re matching the destination IP value from the network data with the indicator IP address from the threat intel feed.
This simple example indicator match rule can provide a straightforward way to identify network activity to a potentially compromised destination.
Triaging threat intel alerts:&nbsp;
When the indicator lookups trigger an alert, Elastic improves the analyst experience with a flyout that provides a summary of the alert that includes the threat intelligence context.&nbsp;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt412b1182ebb74266/611541bee74bc3148654c236/blog-threatintel-elasticsecurity-3.png" data-sys-asset-uid="blt412b1182ebb74266" alt="blog-threatintel-elasticsecurity-3.png">
Figure 3: Alert rendering from indicator match rule
	

<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4544fb721891e6db/611541cc60bb0f13c2eac892/blog-threatintel-elasticsecurity-4.png" data-sys-asset-uid="blt4544fb721891e6db" alt="blog-threatintel-elasticsecurity-4.png">
Figure 4: Alert summary with threat intel information
 <h3>Threat intel enrichment</h3>In general, data enrichment refers to the process of appending and enhancing collected data with relevant context obtained from additional sources. Adding context from threat information during the time of data collection has a unique advantage of quickly narrowing down bad events across a large data set. For example, the collected threat feed data about domains can be used to enrich the network logs to see if there are any known threat activities associated with that particular domain connection.&nbsp;
The Elasticsearch <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-enriching-data.html">enrich processor</a> provides an efficient way to add additional context to the incoming data during ingestion. Enrich processors work with an <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/enrich-setup.html#enrich-setup">enrich policy</a>. The purpose of an enrich policy is to take the data in an index and make it available to the enrich processor in an efficient format, so that the enrich processor can enrich incoming documents very quickly.&nbsp;
Let us take the domain enrichment example;&nbsp;
<ol>
	<li aria-level="1">The first step would be to create a enrich policy, which can easily be done using the Dev Tools in Kibana or by using the Elasticsearch API</li>
</ol><pre class="prettyprint">PUT /_enrich/policy/ti-domain-policy 
{ 
 "match": { 
 "indices": "threatintel-*", 
 "match_field": "threatintel.indicator.domain", 
 "enrich_fields": [ 
 "threatintel.indicator.type", 
 "fileset.name", 
 "threatintel.indicator.domain", 
 "malicious-event", 
 "tags" 
 ] 
 } 
}
</pre><ol start="2">
	<li aria-level="1">The second step is to execute the enrich policy. Behind the scenes, the execute policy API creates a system dot index that is fully managed by Elasticsearch</li>
</ol><pre class="prettyprint">POST /_enrich/policy/ti-domain-policy/_execute
</pre>Note : With continuous updates of threat feeds, Enrich policy can be executed once in a day using the above execution API to update the enrich index with new feeds
<ol start="3">
	<li aria-level="1">Now that that enrich policy has been executed, a pipeline with enrich processors can be defined:</li>
</ol><pre class="prettyprint">PUT /_ingest/pipeline/threatintel-enrichment 
{ 
 "processors": [ 
 { 
 "enrich": { 
 "description": "Enrich dns domain with threat intel indicators", 
 "policy_name": "ti-domain-policy", 
 "field": "dns.question.name", 
 "target_field": "enrichment", 
 "max_matches": "1", 
 "ignore_missing": true 
 } 
 } 
 ] 
}
</pre>The defined pipeline creates an enrich processor that utilises the domain enrich policy which will match the <code>dns.question.name</code> field value from the incoming document to the domain indicator (<code>threatintel.indicator.domain</code>) field value in the threatintel index. Once the lookup matches, the incoming document is added with relevant threat information associated with the domain indicator.
By using the defined pipeline in the data ingestion, respective data fields are looked up with threat intel feeds and enriched with additional threat intel information when matched.

<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltad7e8aaf3633d178/611541e4e74bc3148654c23a/blog-threatintel-elasticsecurity-5a.png" data-sys-asset-uid="bltad7e8aaf3633d178" alt="blog-threatintel-elasticsecurity-5a.png">
Figure 5a: Enriched network data

<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7c8c68a0b0136899/611541f91e409f13bb3b8c0c/blog-threatintel-elasticsecurity-5b.png" data-sys-asset-uid="blt7c8c68a0b0136899" alt="blog-threatintel-elasticsecurity-5b.png">
Figure 5b: Enriched network data
These enriched malicious events can be a good metric to evaluate the blacklisting across security devices and this assists security analysts in narrowing down the time to event resolution.

<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd97187b6c112f040/6115420f906dcc139b6f2632/blog-threatintel-elasticsecurity-6.png" data-sys-asset-uid="bltd97187b6c112f040" alt="blog-threatintel-elasticsecurity-6.png">
Figure 6: Threat intel enrichment metrics in Kibana canvas
 <h2>Wrapping up</h2>Cyber threat intelligence (CTI) is considered one of the essential capabilities in an organization’s security program. When used properly, CTI can enable better-informed security and business decisions, and ultimately allow organizations to take decisive action to protect their users, data, and reputation against adversaries.
It’s easy to get started with Elastic Security — we offer <a href="https://www.elastic.co/training/elastic-security-quick-start">Quick Start training</a> to set you up for success. Start your <a href="https://cloud.elastic.co/registration?elektra=en-security-page">free 14-day trial</a> (no credit card required) or <a href="https://www.elastic.co/downloads/">download our products</a>, free, for your on-prem deployment

blog-banner-security-radar-anomaly-monitor.png

blog-thumb-security-radar-anomaly-monitor.png

Establish robust threat intelligence with Elastic Security

In this blog, we cover security teams' top 5 SIEM considerations of 2021, and how Elastic Security satisfies each.

<a href="https://www.elastic.co/what-is/siem" target="_self">Security information and event management (SIEM) systems</a> are centralized logging platforms that enable security teams to analyze event data in real time for early detection of targeted cyber attacks and data breaches. A SIEM is used as a tool to collect, store, investigate, and report on log data for threat detection, incident response, forensics, and regulatory compliance.SIEM has continuously evolved since its early days of adoption. SIEM software today needs to support big data and provide credible risk assessments and forensics capabilities to piece together events after an incident has occurred. Security analysts have long needed to adapt to changing threats, environments, and perimeters. The ability to integrate with new technologies and increase flexibility in order to detect rapid threat changes has given rise to new expectations and demands on modern SIEM software.&nbsp;In addition to SIEM, <a href="https://www.elastic.co/security" target="_self">Elastic Security</a> is used for leading security use cases such as endpoint security, threat hunting, and cloud monitoring. Here are today’s top five considerations that security teams expect out of their SIEM, and how Elastic Security for SIEM can satisfy each:<h2>1. Stronger cloud integrations and monitoring capabilities</h2>All major cloud services now provide in-depth analysis on the complete cloud infrastructure and application data. Most organizations require such deep-level analytical logs. After all, the more information teams have, the better they are able to protect their company infrastructure and data. Integration with in-depth cloud data is one of the key metrics to be strengthened with a centralized log management process.&nbsp;Elastic <a href="https://www.elastic.co/integrations" target="_self">integrates with a wide range of data sources</a> from major cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud applications through data modules. This includes, but is not limited to, security-relevant cloud datasets such as metrics, traffic from virtual networks, cloud storage, application logs, audit trails, streaming services, containerized environments, and serverless functions. These modules are also built in with sample dashboards for rich visualizations and quick analysis of data.<h2>2. Data protection</h2>Most security teams will use SIEM as their front line detection and analyzing tool. With such large sets of data at hand, data protections — the process of safeguarding important information from corruption, compromise, or loss — play a major role.Authentication, data segregation, encryption, and authorization are top security practices to safeguard data from breach. The importance of data protection increases as the amount of data created and stored continues to grow at unprecedented rates. Key pieces of information that are commonly stored by businesses need to be protected. This is to prevent that data from being misused by third parties for fraud, such as phishing scams and identity theft.<h3>Data protection in the Elastic Stack</h3>Elastic offers free security features to ensure your data stored in Elasticsearch indices is secure and protected from unauthorized users and unintentional modification.&nbsp;<h3>Authentication</h3>This is the one of the critical steps to protect data flowing through Elasticsearch, Kibana, Beats, and Logstash .With the Elastic Stack, you can integrate with a number of industry-standard identity management systems. <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-up-authentication.html" target="_self">User authentication</a> provides built-in realms such as native, LDAP, active_directory, PKI, file, SAML, and OIDC.Use options like certificates, Kerberos, and SAML — or build a custom realm that supports your home-grown identity management system.<h3>Authorization</h3><a href="https://www.elastic.co/guide/en/kibana/current/spaces-api.html" target="_self">Kibana Spaces</a>: Segregate Kibana features and access to those features into multiple spaces within a single Kibana instance.Role-based access control (RBAC): Control which users and roles can access each space, including the specific Kibana features and apps within it.<h3>Encryption</h3>Protect sensitive data, credit card numbers, email addresses, and accounts across your clusters and clients. With SSL/TLS <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/encrypting-communications.html" target="_self">encryption</a>, you can secure node-to-node, HTTP, and transport client traffic across your Elastic Stack. Encrypt data at rest with filesystem encryption on the actual host running Elasticsearch.&nbsp;&nbsp;<h2>3. Integration with security ecosystems</h2>Security analytics have long needed to rapidly adapt to changing threats. Threat detection, investigation, and response are more involved than ever. SIEMs that have basic workflow automation have managed to maintain efficiency up to now, but as companies experience growth, additional capabilities are required.&nbsp;Modern security analytics is much more than just SIEM. Effective analytics include close integration with a security ecosystem within the organization. Analysts need to be able to create custom analytics, integrate with threat intelligence platforms, correlate across different data types and sets, and orchestrate with third-party platforms using APIs.<a href="https://www.elastic.co/guide/en/security/current/security-apis.html" target="_self">Elastic's rich API-driven capabilities</a> augment and orchestrate security data and detections by integrating security detections with notifications and third-party systems via webhooks.<a href="https://www.elastic.co/security" target="_self">Elastic's unified protection</a> combines log analysis (SIEM) with endpoint security, threat hunting, cloud monitoring, and other leading security use cases to equip analysts to successfully prevent, detect, and respond to threats at speed and scale.<h2>4. Advanced analytics</h2>The increasing complexity of attacks is driving the need for advanced analytics beyond the log aggregation of traditional SIEM solutions. Advanced analytical features enhance the threat hunting capabilities within security operations and reduce the mean time to detect sophisticated attacks.&nbsp;Elastic Security fulfills advanced analytics needs with layered analysis such as:<h3>Anomaly detection</h3>Anomaly detection is an important tool for detecting rare events that may have great significance but are hard to find. Elastic’s <a href="https://www.elastic.co/what-is/elasticsearch-machine-learning" target="_self">machine learning</a> automates the analysis of time series data by creating accurate baselines of normal behavior in the data and identifying anomalous patterns. It can automatically understand the data pattern over the cycle and will be able to detect any deviation/anomalies from normal activity.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte5ff49c84aeb46ad/603e58485aedc043351b4b10/1-anomaly-detection-blog-top-siem-considerations.png" data-sys-asset-uid="blte5ff49c84aeb46ad" alt="1-anomaly-detection-blog-top-siem-considerations.png"/> <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta2acce7302cbf696/603e5857f9638443346d3cee/2-critical-anomaly-blog-top-siem-considerations.png" data-sys-asset-uid="blta2acce7302cbf696" alt="2-critical-anomaly-blog-top-siem-considerations.png"/><h3>Detection engine rule types</h3>Elastic Security provides different ways to address different threat detection use cases.&nbsp;<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt53ae31d3e9e1c738/603e586908636f3d7749a1b3/3-create-new-rule-blog-top-siem-considerations.png" data-sys-asset-uid="blt53ae31d3e9e1c738" alt="3-create-new-rule-blog-top-siem-considerations.png"/><h3>Custom query</h3>Security operations look for particular conditional logic in order to detect suspicious events.&nbsp;This query-based rule searches the defined indices and creates an alert when a document matches the rule’s query.<h3>Threshold rules</h3>Certain conditional logic might need to be expanded with the number of times the event has occurred. A lower number of occurrences on certain events might be normal, but the same event with a higher occurrence is suspicious. Threshold rules search&nbsp;the defined indices and create&nbsp;a detection&nbsp;alert when the&nbsp;field’s value meets the threshold number during a single execution.<h3>Machine learning</h3>Machine learning rules are used for creating alerts when an anomaly is triggered by an Elastic machine learning job.<h3>Event correlation</h3>Correlation rules are used for detecting potential threats through relationships between multiple events. Elastic’s Event Query Language (EQL) is a query language for event-based time series data that allows you to express relationships between events. EQL provides searches and detections for multiple related events that are chained together with a sequence of event queries.<h3>Detection with third party data (threat intels)</h3>As a powerful search engine, Elasticsearch provides various ways to enrich data with threat intel feeds, while the Elastic Security detection engine helps users detect alerts with threat indicator matching. Dynamic or static indicators (threat intels) can be streamed into a dedicated index and create an alert using detection rules when the indicator of compromise (IOC) matches with relevant data fields.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt6adc76e6548ea8e5/603e5885fa12da5a61656179/4-indicator-match-blog-top-siem-considerations.png" data-sys-asset-uid="blt6adc76e6548ea8e5" alt="4-indicator-match-blog-top-siem-considerations.png"/><h3>Prebuilt detections&nbsp;</h3>Elastic provides a continuously growing library of out-of-the-box detection <a href="https://www.elastic.co/guide/en/security/current/prebuilt-rules.html" target="_self">rules</a> to address different threat use cases. Elastic detections further provide users the ability to define and create their own detection rules to match environmental use cases.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9c27e4162e92218c/603e58912d310e5a62e169ac/5-direct-outbound-connection-blog-top-siem-considerations.png" data-sys-asset-uid="blt9c27e4162e92218c" alt="5-direct-outbound-connection-blog-top-siem-considerations.png"/><h3>Detection mechanism based on a standardized cybersecurity framework</h3>Over recent years, the MITRE ATT&amp;CK® framework has become the de facto threat detection framework, based on models of how attackers operate. <a href="https://www.elastic.co/campaigns/mitre-attack" target="_self">Elastic adopts the MITRE ATT&amp;CK framework</a> across built-in detection use cases. Such standardized cybersecurity frameworks help analysts better understand and track gaps in their environment to detect attacks before they progress.Example of a visualization created on Top SIEM alerts mapped with MITRE techniques:<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt41b5949dd5008074/603e58a3e122b53af550f90e/6-mitre-techniques-blog-top-siem-considerations.png" data-sys-asset-uid="blt41b5949dd5008074" alt="6-mitre-techniques-blog-top-siem-considerations.png"/> <h2>Flexible delivery models</h2>Analyst-driven security solutions need to adopt different and flexible delivery models, including on-premises, cloud-based, and hybrid deployments. Such flexible deployment and log management ability provides adaptability to infrastructure needs.&nbsp;From platform deployment to data integration, Elastic provides flexibility in the way the logs and log platforms are managed:<ul><li aria-level="1"><a href="https://www.elastic.co/cloud/" target="_self">Elastic Cloud</a> and enterprise-grade, service-oriented architecture with <a href="https://www.elastic.co/ece" target="_self">Elastic Cloud Enterprise (ECE)</a> lets you scale each service separately, with different reliability and performance requirements.</li><li aria-level="1"><a href="https://www.elastic.co/elastic-cloud-kubernetes" target="_self">Elastic Cloud on Kubernetes (ECK)</a>, built on the Kubernetes Operator pattern, extends Kubernetes orchestration capabilities to support the setup and management of Elasticsearch and Kibana on Kubernetes.</li><li aria-level="1">And, of course, native on-prem <a href="https://www.elastic.co/guide/en/cloud/current/ec-configure.html" target="_self">deployment</a> and subscription to Elasticsearch Service from your preferred <a href="https://www.elastic.co/guide/en/cloud/current/ec-marketplaces.html" target="_self">marketplace</a> brings Elastic to your preferred deployment method.&nbsp;</li></ul><h2>Wrapping up</h2>These top five considerations provide us with strong insight into how SIEM is evolving beyond being merely a <a href="https://www.elastic.co/observability/log-monitoring" target="_self">log monitoring tool</a>. There will always be continuous changes in both the security threat landscape and the way enterprise systems address challenges, and we at Elastic will continue to be there to help security teams take those changes head on.&nbsp;It’s easy to get started with Elastic Security — we offer <a href="https://www.elastic.co/training/elastic-security-quick-start" target="_self">Quick Start training</a> to set you up for success. Start your <a href="https://cloud.elastic.co/registration?elektra=en-security-page" target="_self">free 14-day trial</a> (no credit card required) or <a href="https://www.elastic.co/downloads/" target="_self">download our products</a>, free, for your on-prem deployment.Check out the <a href="https://www.elastic.co/what-is/siem" target="_self">SIEM Overview Guide</a> to learn more.

blog-banner-full-spectrum-security.jpg

blog-thumb-full-spectrum-security.jpg

Top 5 SIEM considerations of 2021 and how Elastic Security solves them

Security information and event management (SIEM) systems are centralized log platforms to analyze event data in real time for early detection of targeted cyber attacks and data breaches. A SIEM is used as a tool to collect, store, investigate, and report on log data for threat detection, incident response, forensics, and regulatory compliance.Log collection is the lifeblood of SIEMs and other security analytics technologies. Application data, network traffic, and endpoint data tells us interesting information about our infrastructure. The more data sources that are available, the more that can be accomplished with security analytics. With all the available log sources, long gone are the days of speaking in terms of gigabytes of data — companies today are now regularly working with terabytes and petabytes.Scaling and native distributed architecture are key when handling an explosion in data. These traits also happen to be two of the core values of Elasticsearch. In this blog post, we’ll explore how to understand data sources relevant to security analytics and how to bring them into the Elastic Stack.As a note to the reader, the details shown here are intended only as a baseline guide to begin data collection for security analytics.<h2>Data collection and enrichment</h2>Log collection depends on various use cases and infrastructure. Knowledge of your network is important to decide on log collection scope and volume. It’s best to begin by understanding your IT network infrastructure. In any use case, you’ll need some sort of ingest mechanism.With the Elastic Stack, <a href="/products/beats">Beats</a> and <a href="/guide/en/logstash/7.4/input-plugins.html">Logstash</a> comprise the ingest layer to perform data collection and enrichment.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3226eb7dd70071af/5da629478854f368b9b0b7ba/knowsecurity-blog-1.png" data-sys-asset-uid="blt3226eb7dd70071af" alt="knowsecurity-blog-1.png" width="186" height="217" style="width: 186;height: 217;"/>&nbsp;<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt908e2167af481162/5da6297341cfbe67ebcd7fee/knowsecurity-blog-4.png" data-sys-asset-uid="blt908e2167af481162" alt="knowsecurity-blog-4.png" width="184" height="216" style="width: 184;height: 216;" rel="width: 184px; height: 216px;"/><h3>Beats</h3>Beats are lightweight data shippers that address various data collection requirements such as file-based logs, metrics, network packets, cloud infrastructure, and Windows and Linux events. Beat modules are regularly developed to simplify data collection and provide pre-built data enrichment, dashboards, and machine learning jobs.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd7209b5658440a7a/5da629a741cfbe67ebcd7ff4/knowsecurity-blog-9.png" data-sys-asset-uid="bltd7209b5658440a7a" alt="Elastic Beats for shipping security data" width="538" height="230" style="width: 538;height: 230;display: block;margin: auto;"/>We’ll discuss the various Beats and what they can be used for, but for now, know that these modules can be centrally defined and managed using <a href="/blog/introducing-beats-central-management-in-the-elastic-stack">Beats central management</a>. You can also quickly deploy configuration changes to all Beats running across your enterprise through this management platform.Beats can ship data directly to an <a href="/guide/en/elasticsearch/reference/7.4/ingest.html">ingest node</a> in Elasticsearch. In some cases, Logstash is used separately or alongside beats as an extract, transform, load (ETL) tool for log collection and enrichment.<h3>Logstash</h3>Logstash is an open source data collection engine with real-time pipelining capabilities. It can dynamically unify data from disparate sources and normalize the data. Any type of event can be enriched and <a href="/guide/en/logstash/7.4/transformation.html">transformed</a>. Many native codecs are supported with Logstash inputs, further simplifying the ingestion process.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt689b9a07c1e97d8e/5da629c91a18ec672d9bc36f/knowsecurity-blog-2.png" data-sys-asset-uid="blt689b9a07c1e97d8e" alt="Logstash for ingest and enrichment" width="467" height="298" style="width: 467;height: 298;display: block;margin: auto;"/>Multiple Logstash configurations can also be managed using <a href="/guide/en/logstash/7.4/logstash-centralized-pipeline-management.html">centralized pipeline management</a>, which centralizes the creation and management of Logstash configuration pipelines.Log enrichment is the process of adding additional information and analytical values to the logs collected. A few examples:<ul><li><a href="/guide/en/logstash/7.4/plugins-filters-geoip.html">Geo-location</a> belonging to an IP address</li><li>Threat intel lookup [IOC - indicators of compromise]</li><li>Tracking asset ownership</li><li>Tagging asset severity</li><li>Tagging with vulnerability status reports</li><li>Correlating email address with users</li></ul>Such <a href="/guide/en/logstash/7.4/lookup-enrichment.html">enrichments</a> can be performed during data collection using Logstash plugins or Beats processors.Beats or Logstash? The choice is not always one over the other, as they may be used together or in parallel for different parts of the data processing and use cases.<h2>Types of data sources</h2>Your environment has numerous data sources and organizations are sometimes tempted to deem everything a top priority — but where to begin? Here are some common data sources to consider collecting to enable security analytics:<h3>Network infrastructure</h3>The following technologies provide information about activity in your environment, with collection typically performed via native log collection or sniffing through a TAP or SPAN port.<ul><li>Routers</li><li>Switches</li><li>Domain controllers</li><li>DNS traffic</li><li>Wireless access points</li><li>Application servers</li><li>Databases</li><li>Intranet applications</li></ul><a href="/products/beats/packetbeat">Packetbeat</a> captures insights from network packets — parsing application-level protocols on the fly — and processes messages into transactions. Packetbeat supports flows, DNS, and other protocols.<a href="/products/beats/filebeat">Filebeat</a> monitors and collects the log files or locations that you specify. Filebeat <a href="/guide/en/beats/filebeat/7.4/filebeat-modules.html">modules</a> simplify the collection, parsing, and visualization of common log formats such as <a href="/guide/en/beats/filebeat/7.4/filebeat-module-cisco.html">CISCO</a>, <a href="/guide/en/beats/filebeat/7.4/filebeat-module-panw.html">Palo Alto</a> Networks, <a href="/guide/en/beats/filebeat/7.4/filebeat-module-suricata.html">Suricata IDS</a>, Ubiquiti IPTables, and <a href="/guide/en/beats/filebeat/7.4/filebeat-module-zeek.html">Zeek (Bro)</a>.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd76642b06eac1ce3/5da629f17ad68468b438421c/knowsecurity-blog-5.2.jpg" data-sys-asset-uid="bltd76642b06eac1ce3" alt="Filebeat dashboard" style="display: block;margin: auto;"/><h3>Security devices</h3>Logs from your security control (applications used for securing your infrastructure) are valuable for the analysis of events for <a href="https://www.elastic.co/security/automated-threat-protection" target="_self">automated threat detection</a> and proactive hunting:<ul><li>IDS/IPS (Intrusion detection/prevention system)</li><li>Firewalls</li><li>Endpoint security (antivirus, anti-malware)</li><li>Data loss prevention (DLP)</li><li>VPN Concentrators</li><li>Web filters</li><li>Honeypots</li></ul>Most of these technologies transmit data in syslog, which can be effectively collected, enriched, and transformed with Logstash or Filebeat.<h3>Host events and server logs</h3>There are hundreds of thousands of events that can trigger within operating systems and their associated applications can generate tens of thousands of unique events. Understanding the log events that are most vital is integral for effective monitoring and prioritization.<h4>Windows monitoring</h4>Windows log events provide significant insight on OS- and application-level security details within the host. Windows tracks specific events in its log files with an Event ID for each type of event that can help identify concerning cyber activity.<a href="/products/beats/winlogbeat">Winlogbeat</a> is installed as a Windows service and ships Windows event logs with pre-built monitoring dashboards and machine learning jobs. Winlogbeat can capture event data from any event logs running on your system — security, application, hardware, Powershell, and Sysmon events.<a href="https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon">Sysmon</a> is an open source Windows system service and device driver developed by Microsoft that provides detailed information about process creations, network connections, and changes to file creation time. Augmenting Sysmon with Windows security events helps advance threat hunting use cases.The following tables list Windows and Sysmon events that are often associated with malicious activity.<table><tbody><tr><td>Windows event ID</td><td>Potential criticality</td><td>Event summary</td></tr><tr><td>4618</td><td>High</td><td>A monitored security event pattern has occurred</td></tr><tr><td>4649</td><td>High</td><td>A replay attack was detected — may be a harmless false positive due to misconfiguration errors</td></tr><tr><td>4719</td><td>High</td><td>System audit policy was changed</td></tr><tr><td>4765</td><td>High</td><td>SID History was added to an account</td></tr><tr><td>4766</td><td>High</td><td>An attempt to add SID History to an account failed</td></tr><tr><td>4794</td><td>High</td><td>An attempt was made to set the Directory Services Restore Mode</td></tr><tr><td>4897</td><td>High</td><td>Role separation enabled</td></tr><tr><td>4964</td><td>High</td><td>Special groups have been assigned to a new logon</td></tr><tr><td>5124</td><td>High</td><td>A security setting was updated on the OCSP Responder Service</td></tr><tr><td>1102</td><td>Medium to High</td><td>The audit log was cleared</td></tr></tbody></table>Microsoft has a complete list of <a href="https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/plan/appendix-l--events-to-monitor">Event IDs</a> to monitor that can be enabled based on the need and requirement.Sysmon events<table><tbody><tr><td>Event class</td><td>Event IDs</td><td>Importance</td></tr><tr><td>Process Create</td><td>1</td><td>Detect initial infection and malware child processes and provide ability to investigate hashes captured by the Sysmon module with VirusTotal.com</td></tr><tr><td>Process Terminate</td><td>2</td><td>Useful for forensic investigations. May be correlated with process creation events</td></tr><tr><td>Drive Load</td><td>6</td><td>Detect device drivers loading</td></tr><tr><td>Image Load</td><td>7</td><td>Detect DLL injection, unsigned DLL loading</td></tr><tr><td>File Creation Time Changed</td><td>2</td><td>Detect anti-forensic activity (timestamp changed to cover tracks)</td></tr><tr><td>Network Connection</td><td>3</td><td>Identify network activity, connection to malware C&amp;C servers, connection to ransomware server to download encryption keys</td></tr><tr><td>CreateRemoteThread</td><td>8</td><td>Detect code injections used by malware;Credential theft tools also use this technique to inject code into the LSASS process</td></tr><tr><td>RawAccessRead</td><td>9</td><td>Detect dropping off SAM or NTDS.DIT from compromised hosts</td></tr></tbody></table><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt877305aa163de0fa/5da62a3a0e045e6904a28e98/knowsecurity-blog-6.2.jpg" data-sys-asset-uid="blt877305aa163de0fa" alt="Windows monitoring dashboard" style="display: block;margin: auto;"/><h4>Linux monitoring</h4><a href="/products/beats/auditbeat">Auditbeat</a> is a lightweight data shipper to collect and centralize audit events from the Linux Audit Framework. Audibeat does the same thing as auditd itself.Auditbeat file integrity monitoring (FIM) and System module can run on any OS (Linux, macOS, Windows) and are used to collect and monitor packages, processes, logins, sockets, users, and groups.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte67c9f80539eb755/5da62a528854f368b9b0b7c2/knowsecurity-blog-7.2.jpg" data-sys-asset-uid="blte67c9f80539eb755" alt="Auditbeat dashboard" style="display: block;margin: auto;"/>Linux systems can be ingested into the Elastic Stack in support of security use cases. The following table lists several security-relevant Linux log files to collect with Filebeat or Auditbeat in support of security monitoring.<table><tbody><tr><td>Log path</td><td>Details</td><td>Use case</td></tr><tr><td>/var/log/auth.log (Debian)/var/log/secure (Centos)</td><td>Authentication-related events</td><td>Brute-force attacks and other vulnerabilities related to user authorization mechanism</td></tr><tr><td>/var/log/boot.log</td><td>Booting-related information</td><td>Issues related to improper shutdown, unplanned reboots or booting failures</td></tr><tr><td>/var/log/dmesg</td><td>Kernel ring buffer messages</td><td>Monitoring server functionality</td></tr><tr><td>/var/log/faillog</td><td>Information on failed login attempts</td><td>Security breaches involving username/password hacking and brute-force attacks</td></tr><tr><td>/var/log/cron</td><td>Cron job information</td><td>To monitor errors or new cron schedule status</td></tr><tr><td>/var/log/mail.log</td><td>Logs related to mail server</td><td>Trace headers of email and monitoring possible spamming attempts</td></tr><tr><td>var/log/httpd/</td><td>Logs recorded by the Apache server</td><td>Log to monitor for apache web server and httpd errors</td></tr><tr><td>/var/log/mysql.log (debian)/var/log/mysqld.log (centos)</td><td>MySQL log files</td><td>Monitor events related to mysql service</td></tr></tbody></table><h3>Cloud infrastructure</h3>Collecting data from cloud servers and services enables many important security use cases. Security-relevant cloud datasets include metrics, traffic from virtual networks, cloud storage, application logs, and audit trails. Together, this data provides a detailed view of your cloud infrastructure.<a href="/products/beats/functionbeat">Functionbeat</a> is a data shipper that runs on serverless architecture. Deploy it as a function in your serverless framework (e.g., AWS Lambda) to collect data from your cloud services.<h2>The Elastic Stack as a central platform</h2>Put as much security-relevant data in the Elastic Stack as possible — it can handle it. The more data your security practitioners have, the more security use cases they can address.Elasticsearch can ingest almost any kind of data, including structured and unstructured text, flow data, metrics, geo points, and more. This enables organizations to ingest whatever data they need, regardless of the type or origin. And with Elastic SIEM, real-time access to log data allows you to filter and locate that one “needle in a haystack” event that could reveal the cause of a security breach.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdc224b959d843664/5da62a6a3c3a9367e61d4661/knowsecurity-blog-8.2.jpg" data-sys-asset-uid="bltdc224b959d843664" alt="The Elastic Stack as a security platform" style="display: block;margin: auto;"/>It would be great if every data source recorded information, but unfortunately not all do. <a href="/blog/introducing-the-elastic-common-schema">Elastic Common Schema (ECS)</a> helps users address this issue by providing a set of fields to which diverse data sources can be applied, simplifying analysis. Beats supports ECS right out of the box and many of its modules include pre-built, ECS-enabled monitoring dashboards and machine learning jobs.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8d92e4987fe2a4eb/5da62a89eaa22f65f14fba4a/knowsecurity-blog-3.2.jpg" data-sys-asset-uid="blt8d92e4987fe2a4eb" alt="Packetbeat dashboard" style="display: block;margin: auto;"/>The <a href="/blog/introducing-elastic-siem">Elastic SIEM</a> app in Kibana, which provides an interactive workspace for security teams to triage events and perform initial investigations, is also enabled by ECS. Further, many new out-of-the-box <a href="/blog/introducing-machine-learning-for-the-elastic-stack">machine learning</a> rules (including those integrated with the SIEM app) apply ECS.<h2>Wrapping up</h2>Log collection from various data sources depends on the requirement of the organization. As a general practice, collection usually starts with logs from security controls that can provide an overall view of security posture and monitoring within the enterprise. From here, gradually increase your data collection with network infrastructure logs and host system events.If you want to give it a try yourself, spin up a <a href="/products/elasticsearch/service">free, 14-day trial of Elasticsearch Service</a> and start ingesting. Or explore and experience the Elastic Stack features in our live <a href="https://demo.elastic.co/app/siem">Demo environment</a> with no configuration necessary.

unlocked_large_2.jpg

unlocked-small.jpg

Understanding and unlocking security data sources

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

headshot-haran-kumar-300x300.jpg

Senior Security Specialist

Haran Kumar

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Haran Kumar

Senior Security Specialist, Elastic

Establish robust threat intelligence with Elastic Security

Top 5 SIEM considerations of 2021 and how Elastic Security solves them

Understanding and unlocking security data sources