In this blog, we cover security teams' top 5 SIEM considerations of 2021, and how Elastic Security satisfies each.

Security information and event management (SIEM) systems are centralized logging platforms that enable security teams to analyze event data in real time for early detection of targeted cyber attacks and data breaches. A SIEM is used as a tool to collect, store, investigate, and report on log data for threat detection, incident response, forensics, and regulatory compliance.
SIEM has continuously evolved since its early days of adoption. SIEM software today needs to support big data and provide credible risk assessments and forensics capabilities to piece together events after an incident has occurred. Security analysts have long needed to adapt to changing threats, environments, and perimeters. The ability to integrate with new technologies and increase flexibility in order to detect rapid threat changes has given rise to new expectations and demands on modern SIEM software.&nbsp;
In addition to SIEM, <a href="https://www.elastic.co/security">Elastic Security</a> is used for leading security use cases such as endpoint security, threat hunting, and cloud monitoring. Here are today’s top five considerations that security teams expect out of their SIEM, and how Elastic Security for SIEM can satisfy each:
<h2>1. Stronger cloud integrations and monitoring capabilities</h2>All major cloud services now provide in-depth analysis on the complete cloud infrastructure and application data. Most organizations require such deep-level analytical logs. After all, the more information teams have, the better they are able to protect their company infrastructure and data. Integration with in-depth cloud data is one of the key metrics to be strengthened with a centralized log management process.&nbsp;
Elastic <a href="https://www.elastic.co/integrations">integrates with a wide range of data sources</a> from major cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud applications through data modules. This includes, but is not limited to, security-relevant cloud datasets such as metrics, traffic from virtual networks, cloud storage, application logs, audit trails, streaming services, containerized environments, and serverless functions. These modules are also built in with sample dashboards for rich visualizations and quick analysis of data.
<h2>2. Data protection</h2>Most security teams will use SIEM as their front line detection and analyzing tool. With such large sets of data at hand, data protections — the process of safeguarding important information from corruption, compromise, or loss — play a major role.
Authentication, data segregation, encryption, and authorization are top security practices to safeguard data from breach. The importance of data protection increases as the amount of data created and stored continues to grow at unprecedented rates. Key pieces of information that are commonly stored by businesses need to be protected. This is to prevent that data from being misused by third parties for fraud, such as phishing scams and identity theft.
 <h3>Data protection in the Elastic Stack</h3>Elastic offers free security features to ensure your data stored in Elasticsearch indices is secure and protected from unauthorized users and unintentional modification.&nbsp;
 <h3>Authentication</h3>This is the one of the critical steps to protect data flowing through Elasticsearch, Kibana, Beats, and Logstash .
With the Elastic Stack, you can integrate with a number of industry-standard identity management systems. <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-up-authentication.html">User authentication</a> provides built-in realms such as native, LDAP, active_directory, PKI, file, SAML, and OIDC.
Use options like certificates, Kerberos, and SAML — or build a custom realm that supports your home-grown identity management system.
 <h3>Authorization</h3><a href="https://www.elastic.co/guide/en/kibana/current/spaces-api.html">Kibana Spaces</a>: Segregate Kibana features and access to those features into multiple spaces within a single Kibana instance.
Role-based access control (RBAC): Control which users and roles can access each space, including the specific Kibana features and apps within it.
 <h3>Encryption</h3>Protect sensitive data, credit card numbers, email addresses, and accounts across your clusters and clients. With SSL/TLS <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/encrypting-communications.html">encryption</a>, you can secure node-to-node, HTTP, and transport client traffic across your Elastic Stack. Encrypt data at rest with filesystem encryption on the actual host running Elasticsearch.&nbsp;&nbsp;
<h2>3. Integration with security ecosystems</h2>Security analytics have long needed to rapidly adapt to changing threats. Threat detection, investigation, and response are more involved than ever. SIEMs that have basic workflow automation have managed to maintain efficiency up to now, but as companies experience growth, additional capabilities are required.&nbsp;
Modern security analytics is much more than just SIEM. Effective analytics include close integration with a security ecosystem within the organization. Analysts need to be able to create custom analytics, integrate with threat intelligence platforms, correlate across different data types and sets, and orchestrate with third-party platforms using APIs.
<a href="https://www.elastic.co/guide/en/security/current/security-apis.html">Elastic's rich API-driven capabilities</a> augment and orchestrate security data and detections by integrating security detections with notifications and third-party systems via webhooks.
<a href="https://www.elastic.co/security">Elastic's unified protection</a> combines log analysis (SIEM) with endpoint security, threat hunting, cloud monitoring, and other leading security use cases to equip analysts to successfully prevent, detect, and respond to threats at speed and scale.
<h2>4. Advanced analytics</h2>The increasing complexity of attacks is driving the need for advanced analytics beyond the log aggregation of traditional SIEM solutions. Advanced analytical features enhance the threat hunting capabilities within security operations and reduce the mean time to detect sophisticated attacks.&nbsp;
Elastic Security fulfills advanced analytics needs with layered analysis such as:
 <h3>Anomaly detection</h3>Anomaly detection is an important tool for detecting rare events that may have great significance but are hard to find. Elastic’s <a href="https://www.elastic.co/what-is/elasticsearch-machine-learning">machine learning</a> automates the analysis of time series data by creating accurate baselines of normal behavior in the data and identifying anomalous patterns. It can automatically understand the data pattern over the cycle and will be able to detect any deviation/anomalies from normal activity.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte5ff49c84aeb46ad/603e58485aedc043351b4b10/1-anomaly-detection-blog-top-siem-considerations.png" data-sys-asset-uid="blte5ff49c84aeb46ad" alt="1-anomaly-detection-blog-top-siem-considerations.png"> <h3><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta2acce7302cbf696/603e5857f9638443346d3cee/2-critical-anomaly-blog-top-siem-considerations.png" data-sys-asset-uid="blta2acce7302cbf696" alt="2-critical-anomaly-blog-top-siem-considerations.png"> </h3><h3>Detection engine rule types</h3> 
Elastic Security provides different ways to address different threat detection use cases.&nbsp;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt53ae31d3e9e1c738/603e586908636f3d7749a1b3/3-create-new-rule-blog-top-siem-considerations.png" data-sys-asset-uid="blt53ae31d3e9e1c738" alt="3-create-new-rule-blog-top-siem-considerations.png"><h3>Custom query</h3>Security operations look for particular conditional logic in order to detect suspicious events.&nbsp;
This query-based rule searches the defined indices and creates an alert when a document matches the rule’s query.
<h3>Threshold rules</h3>Certain conditional logic might need to be expanded with the number of times the event has occurred. A lower number of occurrences on certain events might be normal, but the same event with a higher occurrence is suspicious. Threshold rules search&nbsp;the defined indices and create&nbsp;a detection&nbsp;alert when the&nbsp;field’s value meets the threshold number during a single execution.<h3>Machine learning</h3>Machine learning rules are used for creating alerts when an anomaly is triggered by an Elastic machine learning job.
<h3>Event correlation</h3>Correlation rules are used for detecting potential threats through relationships between multiple events. Elastic’s Event Query Language (EQL) is a query language for event-based time series data that allows you to express relationships between events. EQL provides searches and detections for multiple related events that are chained together with a sequence of event queries.
 <h3>Detection with third party data (threat intels)</h3>As a powerful search engine, Elasticsearch provides various ways to enrich data with threat intel feeds, while the Elastic Security detection engine helps users detect alerts with threat indicator matching. Dynamic or static indicators (threat intels) can be streamed into a dedicated index and create an alert using detection rules when the indicator of compromise (IOC) matches with relevant data fields.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt6adc76e6548ea8e5/603e5885fa12da5a61656179/4-indicator-match-blog-top-siem-considerations.png" data-sys-asset-uid="blt6adc76e6548ea8e5" alt="4-indicator-match-blog-top-siem-considerations.png"><h3>Prebuilt detections&nbsp;</h3>Elastic provides a continuously growing library of out-of-the-box detection <a href="https://www.elastic.co/guide/en/security/current/prebuilt-rules.html">rules</a> to address different threat use cases. Elastic detections further provide users the ability to define and create their own detection rules to match environmental use cases.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9c27e4162e92218c/603e58912d310e5a62e169ac/5-direct-outbound-connection-blog-top-siem-considerations.png" data-sys-asset-uid="blt9c27e4162e92218c" alt="5-direct-outbound-connection-blog-top-siem-considerations.png"><h3>Detection mechanism based on a standardized cybersecurity framework</h3>Over recent years, the MITRE ATT&CK® framework has become the de facto threat detection framework, based on models of how attackers operate. <a href="https://www.elastic.co/campaigns/mitre-attack">Elastic adopts the MITRE ATT&CK framework</a> across built-in detection use cases. Such standardized cybersecurity frameworks help analysts better understand and track gaps in their environment to detect attacks before they progress.
Example of a visualization created on Top SIEM alerts mapped with MITRE techniques:<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt41b5949dd5008074/603e58a3e122b53af550f90e/6-mitre-techniques-blog-top-siem-considerations.png" data-sys-asset-uid="blt41b5949dd5008074" alt="6-mitre-techniques-blog-top-siem-considerations.png"> <h2>Flexible delivery models</h2>Analyst-driven security solutions need to adopt different and flexible delivery models, including on-premises, cloud-based, and hybrid deployments. Such flexible deployment and log management ability provides adaptability to infrastructure needs.&nbsp;
From platform deployment to data integration, Elastic provides flexibility in the way the logs and log platforms are managed:
<ul>
	<li aria-level="1"><a href="https://www.elastic.co/cloud/">Elastic Cloud</a> and enterprise-grade, service-oriented architecture with <a href="https://www.elastic.co/ece">Elastic Cloud Enterprise (ECE)</a> lets you scale each service separately, with different reliability and performance requirements.</li>
	<li aria-level="1"><a href="https://www.elastic.co/elastic-cloud-kubernetes">Elastic Cloud on Kubernetes (ECK)</a>, built on the Kubernetes Operator pattern, extends Kubernetes orchestration capabilities to support the setup and management of Elasticsearch and Kibana on Kubernetes.</li>
	<li aria-level="1">And, of course, native on-prem <a href="https://www.elastic.co/guide/en/cloud/current/ec-configure.html">deployment</a> and subscription to Elasticsearch Service from your preferred <a href="https://www.elastic.co/guide/en/cloud/current/ec-marketplaces.html">marketplace</a> brings Elastic to your preferred deployment method.&nbsp;</li>
</ul> <h2>Wrapping up</h2>These top five considerations provide us with strong insight into how SIEM is evolving beyond being merely a log monitoring tool. There will always be continuous changes in both the security threat landscape and the way enterprise systems address challenges, and we at Elastic will continue to be there to help security teams take those changes head on.&nbsp;
It’s easy to get started with Elastic Security — we offer <a href="https://www.elastic.co/training/elastic-security-quick-start">Quick Start training</a> to set you up for success. Start your <a href="https://cloud.elastic.co/registration?elektra=en-security-page">free 14-day trial</a> (no credit card required) or <a href="https://www.elastic.co/downloads/">download our products</a>, free, for your on-prem deployment.

blog-banner-full-spectrum-security.jpg

blog-thumb-full-spectrum-security.jpg

Top 5 SIEM considerations of 2021 and how Elastic Security solves them

Haran Kumar shares his passion for open source, and the inspiration he gained through the Elasticsearch community before joining Elastic.

Want to know what life at Elastic is like? You've come to the right place. Hear from our Elasticians first hand as we highlight their stories and photos from around the world. In this edition, Haran Kumar shares his passion for open source, and the inspiration and professional development he gained through the Elasticsearch community before being recruited to join Elastic.
Learning never ends. I apply this concept in both my professional and personal life, and when it comes to information security, I’m constantly striving to learn and improve my knowledge. In fact, it’s this drive to learn and grow that led me from being a part of the open source community to joining Elastic.
<h2>Experience before Elastic</h2>When I joined my first employer right after university as a campus recruit, I was informed of the various IT operations taking place there. I was fascinated when security was explained to me — the various tools like network intrusion detection, virus protection, and the way security professionals hunt for malware and attackers. What I found really exciting was how security tools and projects would connect me with real world use cases.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1369acbf562dd127/5db07d52df78486c826dce22/Haran_white_shirt.jpg" data-sys-asset-uid="blt1369acbf562dd127" alt="Haran Kumar" style="display: block; margin: auto;">
<figcaption>Haran Kumar.</figcaption>After that introduction, I volunteered for all the security domain trainings offered during my probationary period, and from there leapt into my security career. I was fortunate to work with great people at the company, and play around with various security tools from a wide array of vendors.
As I worked with these tools, I began to realize that because the software I was using was closed;I was offered fewer learning opportunities and a narrower scope of work. I saw a lot of possibilities beyond using closed software and I wanted to explore those options to improve my abilities as a security professional.
I began to develop my knowledge of open source solutions outside of my day-to-day tasks with self-paced, online courses and by taking certification exams. I also started to make connections outside of work by talking to people online, going to meetUps, and attending conferences focused on security.
Exploring resources outside of work was my first chance to play around with open source software. I’d always had the ambition of creating something on my own, and as I explored open source, I saw the possibility of making this ambition a reality. <a href="https://www.elastic.co/elasticsearch">Elasticsearch</a> was a starting point for this. I started exploring and following projects built on top of Elasticsearch, and the adoption of it within the security community fascinated and motivated me to try some of my own projects using Elasticsearch and other open source tools.
<h2>Community collaboration</h2>When I started exploring Elasticsearch on security use cases, I found there was <a href="https://www.elastic.co/community/">a large community ready to help</a>. I was intrigued that a single search product was used in so many different ways. For example, some people used Elasticsearch as a data lake for log collection. Some folks used Elasticsearch as a centralised threat hunting platform by integrating it with various open source tools. There were also people creating network security monitoring platforms with Elasticsearch as its heart. I also found that various commercial security products used Elasticsearch under the hood because of its speed and scaling capabilities.The possibilities seemed endless.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4a4d94d5977d978a/5db1b634e9effa6ba5296867/Haran_community_event.JPG" data-sys-asset-uid="blt4a4d94d5977d978a" alt="" style="display: block; margin: auto;">
<figcaption>Haran at a community event.</figcaption>I began working on projects during weekends and after office hours. I was fascinated with the Elastic community and the people who contributed to Elasticsearch through the open source collaboration, and with how people adapted Elasticsearch differently. Seeing how they worked, and how they plugged into the Elastic Stack helped me learn how to to do introduce various open source integrations for security projects at my job, and also enabled me to share that knowledge with my teammates. When I didn’t know an answer, <a href="https://discuss.elastic.co/">the Elastic discussion forum</a> became my go-to-place for my queries.
After working on my own projects for a while, I <a href="https://www.linkedin.com/pulse/lets-darp-2-elastalert-haran-kumar">shared a blog post</a> about SIEM rules and alerting with Elasticsearch, and created a <a href="https://github.com/HKcyberstark/TI_Mod">Threat Intel Project</a> using open source intel feeds with Elastic Stack at its heart. Today it’s in the development phase.
<h2>Joining Elastic</h2>This great experience with the Elastic community not only gave me the knowledge and self-confidence I needed while developing my skills, but it was also the reason I was ready to take the call when I was approached about a job opening with Elastic.
 
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt70961870f9528519/5db07d5c823d426762826125/Elastic_meetup.jpg" data-sys-asset-uid="blt70961870f9528519" alt="" style="display: block; margin: auto;">
<figcaption>Haran at the APAC Solutions Architects team dinner.</figcaption>Interviewing at Elastic was one of the toughest and organised interviews I’d had so far in my career. Luckily, during one of my interview rounds, I was scheduled to talk with an Elastician I admired, who I had also met during one of the Elastic meetups. It was great to hear that he and other Elasticians were familiar with my open source projects, and it put me at ease. It was great to see how closely Elastic collaborates with the community. Whenever the next Elastic interview would be scheduled, I was excited and nervous. Luckily, at every step, there were familiar people that I followed in community and security field. The idea that I would work with such great minds gave me so much confidence. My experience in the community gave me the knowledge and experience necessary to get this role!
<h2>Life @ Elastic</h2>It is always a big decision, often with mixed feelings, to change organisations, roles, and locations. Luckily, the transition to Elastic was easy. The people are awesome, talented, humble, and passionate about things they do. Beyond the people, I am inspired by the vision of the company for the technology and our business model.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfcd5f120e4e8eba0/5db07debdf78486c826dce28/Haran_presentation.jpg" data-sys-asset-uid="bltfcd5f120e4e8eba0" alt="" style="display: block; margin: auto;">
<figcaption>Haran giving a presentation.</figcaption> 
It feels great to have moved from being an Elastic fan as a community member to becoming part of the team. Here at Elastic, I don’t feel like an employee working just for the end of the month paycheck — I see myself growing as a professional and as an individual as I do this important work.
And the learning doesn’t end. I am encouraged by my mentor here at Elastic to continue my open source projects, because being able to scratch that itch of contributing back to the community works for the greater good.
The culture and <a href="https://www.elastic.co/about/our-source-code">the Source Code</a> of the company puts my comfort as an employee first and makes me feel at home. With the distributed nature of work, it is the trust that my team and Elastic has of their employees that drives me to work harder, while also working in a more efficient way.
Connecting with people across various sectors, helping customers to address their security needs, and being part of their architectural needs makes my career even more fulfilling as a security enthusiast. I am proud to call myself a fan and, now, an Elastician.
All the learning, inspiration, and the help of so many people have had an incredible impact on me and my development. I just want to say thank you! I believe we are in this together, and I know there is so much more we will still do!
Interested in finding other Elastic fans like you? Join <a href="https://www.elastic.co/community/">the Elastic community</a> and meet us on <a href="https://www.elastic.co/elasticon/">the Elastic{ON} Tour</a>. We’re also hiring! Check out <a href="https://www.elastic.co/about/teams">our teams</a> and <a href="https://www.elastic.co/about/careers/">find your fit</a>! Want to read more about life at Elastic? Read more <a href="https://www.elastic.co/blog/category/culture">on our blog</a>.

Elastic_meetup.jpg

Haran_white_shirt_small.jpg

Life@ Elastic | Haran Kumar: From community member to Elastician

In this blog post, we’ll explore how to understand data sources relevant to security analytics and how to bring them into the Elastic Stack.

Security information and event management (SIEM) systems are centralized log platforms to analyze event data in real time for early detection of targeted cyber attacks and data breaches. A SIEM is used as a tool to collect, store, investigate, and report on log data for threat detection, incident response, forensics, and regulatory compliance.
Log collection is the lifeblood of SIEMs and other security analytics technologies. Application data, network traffic, and endpoint data tells us interesting information about our infrastructure. The more data sources that are available, the more that can be accomplished with security analytics. With all the available log sources, long gone are the days of speaking in terms of gigabytes of data — companies today are now regularly working with terabytes and petabytes.
Scaling and native distributed architecture are key when handling an explosion in data. These traits also happen to be two of the core values of Elasticsearch. In this blog post, we’ll explore how to understand data sources relevant to security analytics and how to bring them into the Elastic Stack.
As a note to the reader, the details shown here are intended only as a baseline guide to begin data collection for security analytics.
 <h2>Data collection and enrichment</h2>Log collection depends on various use cases and infrastructure. Knowledge of your network is important to decide on log collection scope and volume. It’s best to begin by understanding your IT network infrastructure. In any use case, you’ll need some sort of ingest mechanism.
With the Elastic Stack, <a href="/products/beats">Beats</a> and <a href="/guide/en/logstash/7.4/input-plugins.html">Logstash</a> comprise the ingest layer to perform data collection and enrichment.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3226eb7dd70071af/5da629478854f368b9b0b7ba/knowsecurity-blog-1.png" data-sys-asset-uid="blt3226eb7dd70071af" alt="knowsecurity-blog-1.png" width="186" height="217" style="width: 186px; height: 217px;">&nbsp;<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt908e2167af481162/5da6297341cfbe67ebcd7fee/knowsecurity-blog-4.png" data-sys-asset-uid="blt908e2167af481162" alt="knowsecurity-blog-4.png" width="184" height="216" style="width: 184px; height: 216px;" rel="width: 184px; height: 216px;">
<h3>Beats</h3>Beats are lightweight data shippers that address various data collection requirements such as file-based logs, metrics, network packets, cloud infrastructure, and Windows and Linux events. Beat modules are regularly developed to simplify data collection and provide pre-built data enrichment, dashboards, and machine learning jobs.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd7209b5658440a7a/5da629a741cfbe67ebcd7ff4/knowsecurity-blog-9.png" data-sys-asset-uid="bltd7209b5658440a7a" alt="Elastic Beats for shipping security data" width="538" height="230" style="width: 538px; height: 230px; display: block; margin: auto;">
We’ll discuss the various Beats and what they can be used for, but for now, know that these modules can be centrally defined and managed using <a href="/blog/introducing-beats-central-management-in-the-elastic-stack">Beats central management</a>. You can also quickly deploy configuration changes to all Beats running across your enterprise through this management platform.
Beats can ship data directly to an <a href="/guide/en/elasticsearch/reference/7.4/ingest.html">ingest node</a> in Elasticsearch. In some cases, Logstash is used separately or alongside beats as an extract, transform, load (ETL) tool for log collection and enrichment.
 <h3>Logstash</h3>Logstash is an open source data collection engine with real-time pipelining capabilities. It can dynamically unify data from disparate sources and normalize the data. Any type of event can be enriched and <a href="/guide/en/logstash/7.4/transformation.html">transformed</a>. Many native codecs are supported with Logstash inputs, further simplifying the ingestion process.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt689b9a07c1e97d8e/5da629c91a18ec672d9bc36f/knowsecurity-blog-2.png" data-sys-asset-uid="blt689b9a07c1e97d8e" alt="Logstash for ingest and enrichment" width="467" height="298" style="width: 467px; height: 298px; display: block; margin: auto;">
Multiple Logstash configurations can also be managed using <a href="/guide/en/logstash/7.4/logstash-centralized-pipeline-management.html">centralized pipeline management</a>, which centralizes the creation and management of Logstash configuration pipelines.
Log enrichment is the process of adding additional information and analytical values to the logs collected. A few examples:
<ul>
	<li><a href="/guide/en/logstash/7.4/plugins-filters-geoip.html">Geo-location</a> belonging to an IP address</li>
	<li>Threat intel lookup [IOC - indicators of compromise]</li>
	<li>Tracking asset ownership</li>
	<li>Tagging asset severity</li>
	<li>Tagging with vulnerability status reports</li>
	<li>Correlating email address with users</li>
</ul>Such <a href="/guide/en/logstash/7.4/lookup-enrichment.html">enrichments</a> can be performed during data collection using Logstash plugins or Beats processors.
Beats or Logstash? The choice is not always one over the other, as they may be used together or in parallel for different parts of the data processing and use cases.
 <h2>Types of data sources</h2>Your environment has numerous data sources and organizations are sometimes tempted to deem everything a top priority — but where to begin? Here are some common data sources to consider collecting to enable security analytics:
 <h3>Network infrastructure</h3>The following technologies provide information about activity in your environment, with collection typically performed via native log collection or sniffing through a TAP or SPAN port.
<ul>
	<li>Routers</li>
	<li>Switches</li>
	<li>Domain controllers</li>
	<li>DNS traffic</li>
	<li>Wireless access points</li>
	<li>Application servers</li>
	<li>Databases</li>
	<li>Intranet applications</li>
</ul><a href="/products/beats/packetbeat">Packetbeat</a> captures insights from network packets — parsing application-level protocols on the fly — and processes messages into transactions. Packetbeat supports flows, DNS, and other protocols.
<a href="/products/beats/filebeat">Filebeat</a> monitors and collects the log files or locations that you specify. Filebeat <a href="/guide/en/beats/filebeat/7.4/filebeat-modules.html">modules</a> simplify the collection, parsing, and visualization of common log formats such as <a href="/guide/en/beats/filebeat/7.4/filebeat-module-cisco.html">CISCO</a>, <a href="/guide/en/beats/filebeat/7.4/filebeat-module-panw.html">Palo Alto</a> Networks, <a href="/guide/en/beats/filebeat/7.4/filebeat-module-suricata.html">Suricata IDS</a>, Ubiquiti IPTables, and <a href="/guide/en/beats/filebeat/7.4/filebeat-module-zeek.html">Zeek (Bro)</a>.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd76642b06eac1ce3/5da629f17ad68468b438421c/knowsecurity-blog-5.2.jpg" data-sys-asset-uid="bltd76642b06eac1ce3" alt="Filebeat dashboard" "="" style="display: block; margin: auto;">
<h3>Security devices</h3>Logs from your security control (applications used for securing your infrastructure) are valuable for the analysis of events for automated threat detection and proactive hunting:
<ul>
	<li>IDS/IPS (Intrusion detection/prevention system)</li>
	<li>Firewalls</li>
	<li>Endpoint security (antivirus, anti-malware)</li>
	<li>Data loss prevention (DLP)</li>
	<li>VPN Concentrators</li>
	<li>Web filters</li>
	<li>Honeypots</li>
</ul>Most of these technologies transmit data in syslog, which can be effectively collected, enriched, and transformed with Logstash or Filebeat.
 <h3>Host events and server logs</h3>There are hundreds of thousands of events that can trigger within operating systems and their associated applications can generate tens of thousands of unique events. Understanding the log events that are most vital is integral for effective monitoring and prioritization.
<h4>Windows monitoring</h4>Windows log events provide significant insight on OS- and application-level security details within the host. Windows tracks specific events in its log files with an Event ID for each type of event that can help identify concerning cyber activity.
<a href="/products/beats/winlogbeat">Winlogbeat</a> is installed as a Windows service and ships Windows event logs with pre-built monitoring dashboards and machine learning jobs. Winlogbeat can capture event data from any event logs running on your system — security, application, hardware, Powershell, and Sysmon events.
<a href="https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon">Sysmon</a> is an open source Windows system service and device driver developed by Microsoft that provides detailed information about process creations, network connections, and changes to file creation time. Augmenting Sysmon with Windows security events helps advance threat hunting use cases.
The following tables list Windows and Sysmon events that are often associated with malicious activity.
<table>
<tbody>
<tr>
	<td>
		Windows event ID
		
	</td>
	<td>
		Potential criticality
		
	</td>
	<td>
		Event summary
		
	</td>
</tr>
<tr>
	<td>
		4618
		
	</td>
	<td>
		High
		
	</td>
	<td>
		A monitored security event pattern has occurred
		
	</td>
</tr>
<tr>
	<td>
		4649
		
	</td>
	<td>
		High
		
	</td>
	<td>
		A replay attack was detected — may be a harmless false positive due to misconfiguration errors
		
	</td>
</tr>
<tr>
	<td>
		4719
		
	</td>
	<td>
		High
		
	</td>
	<td>
		System audit policy was changed
		
	</td>
</tr>
<tr>
	<td>
		4765
		
	</td>
	<td>
		High
		
	</td>
	<td>
		SID History was added to an account
		
	</td>
</tr>
<tr>
	<td>
		4766
		
	</td>
	<td>
		High
		
	</td>
	<td>
		An attempt to add SID History to an account failed
		
	</td>
</tr>
<tr>
	<td>
		4794
		
	</td>
	<td>
		High
		
	</td>
	<td>
		An attempt was made to set the Directory Services Restore Mode
		
	</td>
</tr>
<tr>
	<td>
		4897
		
	</td>
	<td>
		High
		
	</td>
	<td>
		Role separation enabled
		
	</td>
</tr>
<tr>
	<td>
		4964
		
	</td>
	<td>
		High
		
	</td>
	<td>
		Special groups have been assigned to a new logon
		
	</td>
</tr>
<tr>
	<td>
		5124
		
	</td>
	<td>
		High
		
	</td>
	<td>
		A security setting was updated on the OCSP Responder Service
		
	</td>
</tr>
<tr>
	<td>
		1102
		
	</td>
	<td>
		Medium to High
		
	</td>
	<td>
		The audit log was cleared
		
	</td>
</tr>
</tbody>
</table>Microsoft has a complete list of <a href="https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/plan/appendix-l--events-to-monitor">Event IDs</a> to monitor that can be enabled based on the need and requirement.
Sysmon events
<table>
<tbody>
<tr>
	<td>
		Event class
		
	</td>
	<td>
		Event IDs
		
	</td>
	<td>
		Importance
		
	</td>
</tr>
<tr>
	<td>
		Process Create
		
	</td>
	<td>
		1
		
	</td>
	<td>
		Detect initial infection and malware child processes and provide ability to investigate hashes captured by the Sysmon module with VirusTotal.com
		
	</td>
</tr>
<tr>
	<td>
		Process Terminate
		
	</td>
	<td>
		2
		
	</td>
	<td>
		Useful for forensic investigations. May be correlated with process creation events
		
	</td>
</tr>
<tr>
	<td>
		Drive Load
		
	</td>
	<td>
		6
		
	</td>
	<td>
		Detect device drivers loading
		
	</td>
</tr>
<tr>
	<td>
		Image Load
		
	</td>
	<td>
		7
		
	</td>
	<td>
		Detect DLL injection, unsigned DLL loading
		
	</td>
</tr>
<tr>
	<td>
		File Creation Time Changed
		
	</td>
	<td>
		2
		
	</td>
	<td>
		Detect anti-forensic activity (timestamp changed to cover tracks)
		
	</td>
</tr>
<tr>
	<td>
		Network Connection
		
	</td>
	<td>
		3
		
	</td>
	<td>
		Identify network activity, connection to malware C&C servers, connection to ransomware server to download encryption keys
		
	</td>
</tr>
<tr>
	<td>
		CreateRemoteThread
		
	</td>
	<td>
		8
		
	</td>
	<td>
		Detect code injections used by malware;
		
		Credential theft tools also use this technique to inject code into the LSASS process
		
	</td>
</tr>
<tr>
	<td>
		RawAccessRead
		
	</td>
	<td>
		9
		
	</td>
	<td>
		Detect dropping off SAM or NTDS.DIT from compromised hosts
		
	</td>
</tr>
</tbody>
</table>
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt877305aa163de0fa/5da62a3a0e045e6904a28e98/knowsecurity-blog-6.2.jpg" data-sys-asset-uid="blt877305aa163de0fa" alt="Windows monitoring dashboard" style="display: block; margin: auto;">
<h4>Linux monitoring</h4><a href="/products/beats/auditbeat">Auditbeat</a> is a lightweight data shipper to collect and centralize audit events from the Linux Audit Framework. Audibeat does the same thing as auditd itself.
Auditbeat file integrity monitoring (FIM) and System module can run on any OS (Linux, macOS, Windows) and are used to collect and monitor packages, processes, logins, sockets, users, and groups.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte67c9f80539eb755/5da62a528854f368b9b0b7c2/knowsecurity-blog-7.2.jpg" data-sys-asset-uid="blte67c9f80539eb755" alt="Auditbeat dashboard" style="display: block; margin: auto;">
Linux systems can be ingested into the Elastic Stack in support of security use cases. The following table lists several security-relevant Linux log files to collect with Filebeat or Auditbeat in support of security monitoring.
<table>
<tbody>
<tr>
	<td>
		Log path
		
	</td>
	<td>
		Details
		
	</td>
	<td>
		Use case
		
	</td>
</tr>
<tr>
	<td>
		/var/log/auth.log (Debian)
		
		/var/log/secure (Centos)
		
	</td>
	<td>
		Authentication-related events
		
	</td>
	<td>
		Brute-force attacks and other vulnerabilities related to user authorization mechanism
		
	</td>
</tr>
<tr>
	<td>
		/var/log/boot.log
		
	</td>
	<td>
		Booting-related information
		
	</td>
	<td>
		Issues related to improper shutdown, unplanned reboots or booting failures
		
	</td>
</tr>
<tr>
	<td>
		/var/log/dmesg
		
	</td>
	<td>
		Kernel ring buffer messages
		
	</td>
	<td>
		Monitoring server functionality
		
	</td>
</tr>
<tr>
	<td>
		/var/log/faillog
		
	</td>
	<td>
		Information on failed login attempts
		
	</td>
	<td>
		Security breaches involving username/password hacking and brute-force attacks
		
	</td>
</tr>
<tr>
	<td>
		/var/log/cron
		
	</td>
	<td>
		Cron job information
		
	</td>
	<td>
		To monitor errors or new cron schedule status
		
	</td>
</tr>
<tr>
	<td>
		/var/log/mail.log
		
	</td>
	<td>
		Logs related to mail server
		
	</td>
	<td>
		Trace headers of email and monitoring possible spamming attempts
		
	</td>
</tr>
<tr>
	<td>
		var/log/httpd/
		
	</td>
	<td>
		Logs recorded by the Apache server
		
	</td>
	<td>
		Log to monitor for apache web server and httpd errors
		
	</td>
</tr>
<tr>
	<td>
		/var/log/mysql.log (debian)
		
		/var/log/mysqld.log (centos)
		
	</td>
	<td>
		MySQL log files
		
	</td>
	<td>
		Monitor events related to mysql service
		
	</td>
</tr>
</tbody>
</table> <h3>Cloud infrastructure</h3>Collecting data from cloud servers and services enables many important security use cases. Security-relevant cloud datasets include metrics, traffic from virtual networks, cloud storage, application logs, and audit trails. Together, this data provides a detailed view of your cloud infrastructure.
<a href="/products/beats/functionbeat">Functionbeat</a> is a data shipper that runs on serverless architecture. Deploy it as a function in your serverless framework (e.g., AWS Lambda) to collect data from your cloud services.
 <h2>The Elastic Stack as a central platform</h2>Put as much security-relevant data in the Elastic Stack as possible — it can handle it. The more data your security practitioners have, the more security use cases they can address.
Elasticsearch can ingest almost any kind of data, including structured and unstructured text, flow data, metrics, geo points, and more. This enables organizations to ingest whatever data they need, regardless of the type or origin. And with Elastic SIEM, real-time access to log data allows you to filter and locate that one “needle in a haystack” event that could reveal the cause of a security breach.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdc224b959d843664/5da62a6a3c3a9367e61d4661/knowsecurity-blog-8.2.jpg" data-sys-asset-uid="bltdc224b959d843664" alt="The Elastic Stack as a security platform" style="display: block; margin: auto;">
It would be great if every data source recorded information, but unfortunately not all do. <a href="/blog/introducing-the-elastic-common-schema">Elastic Common Schema (ECS)</a> helps users address this issue by providing a set of fields to which diverse data sources can be applied, simplifying analysis. Beats supports ECS right out of the box and many of its modules include pre-built, ECS-enabled monitoring dashboards and machine learning jobs.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8d92e4987fe2a4eb/5da62a89eaa22f65f14fba4a/knowsecurity-blog-3.2.jpg" data-sys-asset-uid="blt8d92e4987fe2a4eb" alt="Packetbeat dashboard" style="display: block; margin: auto;">
The <a href="/blog/introducing-elastic-siem">Elastic SIEM</a> app in Kibana, which provides an interactive workspace for security teams to triage events and perform initial investigations, is also enabled by ECS. Further, many new out-of-the-box <a href="/blog/introducing-machine-learning-for-the-elastic-stack">machine learning</a> rules (including those integrated with the SIEM app) apply ECS.
<h2>Wrapping up</h2>Log collection from various data sources depends on the requirement of the organization. As a general practice, collection usually starts with logs from security controls that can provide an overall view of security posture and monitoring within the enterprise. From here, gradually increase your data collection with network infrastructure logs and host system events.
If you want to give it a try yourself, spin up a <a href="/products/elasticsearch/service">free, 14-day trial of Elasticsearch Service</a> and start ingesting. Or explore and experience the Elastic Stack features in our live <a href="https://demo.elastic.co/app/siem">Demo environment</a> with no configuration necessary.

Articles by Haran Kumar

Top 5 SIEM considerations of 2021 and how Elastic Security solves them

Life@ Elastic | Haran Kumar: From community member to Elastician

Understanding and unlocking security data sources

Products & Solutions

Company

Resources

Featured topics

News

Articles by Haran Kumar

Top 5 SIEM considerations of 2021 and how Elastic Security solves them

Life@ Elastic | Haran Kumar: From community member to Elastician

Understanding and unlocking security data sources