This blog is a guide to configuring Elastic’s Security App to work with Cross Cluster Search

<p>The Elastic Infosec Detections and Analytics team is responsible for building, tuning, and maintaining the security detections used to protect all Elastic systems. Within Elastic we call ourselves Customer Zero and <a href="https://www.elastic.co/blog/elastic-on-elastic-how-infosec-deploys-infrastructure-and-stays-up-to-date-with-eck">we strive to always use the newest versions</a> of our products.&nbsp;
</p>
<p>In the previous blog posts we gave an overview of our architecture and what data we send to our clusters. In this blog post we will provide instructions on how we use Cross Cluster Search (CCS) with the Security and Machine Learning (ML) applications
</p>
<h2>Configuring CCS<br></h2>
<p>Getting the Security app to work with CCS only requires a few minor changes. The first step is to configure the Security app so that it knows to look for the CCS index patterns instead of local index patterns. To do this open the Kibana ‘Advanced Settings’ within the Stack Management menu, then located within the Security Solution section update the Elasticsearch indices. These should be set to match the CCS index patterns such as *:auditbeat-*, *:filebeat-*, *:logs-*, and any other index patterns you want to add to your Security app.&nbsp;
</p>
<p><img src="https://lh4.googleusercontent.com/kxD1eOYM7-2VRGKIc0PAkZgUV3cvNg6bSSgcYv-i5HQox3VLEgQXNDSxbS9sgqNfDT_AhbvWuZcEfQA_OJxi68lEsQZIJY7w15EwoM4e2uFdnyAoBnGS3O3-cudSiz1pU4H-CPvy=s0" width="602" height="249">
</p>
<p>Now when you return to the Security app you should see the Overview, Hosts, and Network pages displaying events from all of your remote clusters.
</p>
<h2>Configuring Built In Detection Rules to use CCS</h2>
<p>Creating brand new custom Detection rules does not require any additional steps, the CCS index patterns will be available to select when you create a new detection rule. If you want to use the 500+ built in Detection rules with CCS it requires a little extra work. To do this you will need to import, duplicate, and bulk edit all of the rules to change the index patterns to use Cross Cluster Search.&nbsp;
</p>
<p>The first step is to load all of the pre-built Elastic rules into your Security App. Within the detections tab of the security app navigate to the ‘Manage Detections’ page and click the ‘Load Elastic prebuilt rules’ button.
</p>
<h2 style="line-height:1.38;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:18pt 0pt 6pt 0pt;"><img src="https://lh5.googleusercontent.com/UolDnHQhugxnjt3-eXvkWouN6ttWFaJsA0G_PYBCAqcL5qXKHu8-sEjvXWtR4LysIiuiAwoGqwdJzWkXzhyaKNhNfLo77bqS9J3TpjmoDkJO5dKYTjKD3io9KZ0jt4Sa5vmowWOl=s0" width="266" height="65"></h2>
<p>The Prebuilt rules can not be directly modified but they can be duplicated and then the duplicates can be modified. To do this you will need to select all of the rules and use the ‘Bulk Actions’ menu to duplicate them all. Before starting this process I recommend changing the ‘refresh settings’ to disable automatically refreshing the table. If you don’t then the table may refresh automatically while you are trying to duplicate the rules causing you to deselect some rules.
</p>
<p><img src="https://lh4.googleusercontent.com/b6A0AsuACM2hHZ1nDmCklF09TdWKmfoQoUPC-35sKP7mDNN2I3Gnl4O-n8R5nQNL2ldEeLSNuyLpyV6GZFpXDW9P3S5vc3n79sxV9Lsu1cuNRsJPViO_mcHy4rz_W2cY9sZP2v31=s0" width="216" height="77">
</p>
<p>To Select all rules on clusters older than 7.14 you will need to change the ‘Rows per page’ to the maximum amount. <br>
</p>
<h2 style="line-height:1.38;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:18pt 0pt 6pt 0pt;" rel="line-height:1.38;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:18pt 0pt 6pt 0pt;"><img src="https://lh3.googleusercontent.com/THYRjUpwmz2qiAWpp780ZtW_SrQ_3rO3bnPlj67rC4uhbGCzDzzUNJ9aewX29WsZUqa5LTihcGefyLbus8L47PuXWHoezF9Ctqg7yZBcQToWSq6SP4CpLc3bGTZa20jvp2LsdLnw=s0" width="192" height="464"></h2>
<p>When all of the rules are displayed select all rules and then in the bulk actions menu select ‘Duplicate Selected’. Duplicating all of the rules may take a couple minutes.
</p>
<p rel="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><img src="https://lh5.googleusercontent.com/E1ZILEFSNwUHOWUaRBlvRGBVe1WCCcfhIoU7szJgWDp0DnIy5w1-u00JL9xJPq25H6cP8ISFftr9rh_-AtdC09zrNVK46ThtvGygYrJUVhHMMUXKAaxjOJ_jeilOZnMPiL4rOgGT=s0" width="348" height="406">
</p>
<p>When this is complete you will have two copies of every rule. Using the filter on the right side of the rule table select only the ‘Custom rules’ that you just created<br><br>
</p>
<p><img src="https://lh4.googleusercontent.com/_P7bkklIKPRBtuwuQzFwvDEZ-zE6bNH5rxCQEsIcDWA-nDyj9sqiRYXie_ThCvsHsk8PBHFe1xvLK7afHpGnqEkfLWh-YCYZhZbQ9C585TL8OsoJzcfp6Cct4-8FY35-50TjllbJ=s0" width="297" height="57">
</p>
<p>With only the Custom Rules displayed ‘select all’ again and select ‘Export Selected’ from the ‘bulk actions’ menu to download all of the rules as an ndjson file.&nbsp;
</p>
<p>Within this ndjson file you will need to find and replace all instances of the normal index patterns with the CCS index pattern. For example, to use the remote auditbeat index pattern you will need to find "auditbeat-*" and replace it with "*:auditbeat-*".&nbsp; Repeat this for the other index patterns until all of the index patterns have been changed to the new CCS pattern.<br><br>
</p>
<p><img src="https://lh5.googleusercontent.com/SjH1dydxRimjXQP5NGTeRdaQnrCWgZUCcO-g4O5qZCUhKLcKKzbv_LqIgtv55gU2TbjnJg13wta5hM26e7UTI4FoJZOI3XCtiXgfqIcN49tTPFvnpRmrHQPkq3jOrIDZFamCOAw4=s0" width="375" height="76">
</p>
<p>I also recommend replacing the word ‘[Duplicate]’ that was appended to every rule name with the current stack version. This will help you manage your rules over time and track when each rule was installed or last updated.
</p>
<p>After all of the changes have been saved to the custom rules ndjson file you can import the modified rules over the existing rules. Click the ‘Import rule’ button to open the rule import interface
</p>
<p><img src="https://lh6.googleusercontent.com/LlSpXbKOK9bvQRIJcp27-vmCAm2NcJJoQFZwY8JiuQ3KqMQghZt6D2ldD10dfExemIhIsMUDtlkEN-X8cWWh94PmxIN6CnjYiDWSWe7ercmrZT6sTjuTw1OwOdokPiEAlvLjWWVK=s0" width="592" height="166">
</p>
<p>Drag and drop the modified rule file to the pop up window and select the button to ‘Automatically overwrite saved objects with the same rule ID’. If you forget to do this it will create new rules instead of updating the existing rules and you will probably want to clean up all of the unnecessary rules.
</p>
<p><img src="https://lh4.googleusercontent.com/EIcsjUGxFJPwMCo_GYTsIl9N-yAmUGysZyP0NSW7VBc644oDRArFLtyjsuXmiM4IOPy7EDBLc08IPRaJsy9K7St3fzdZE6Ak_hklXDzTQX7oPleLyWlg2E3P_cjv3iF9A5cEl9Ak=s0" width="602" height="405">
</p>
<p>After the rules have all been re-imported you can activate all of the rules that are applicable to your environment.<br><br>
</p>
<h2>Note on Elastic Query Language (EQL)</h2>
<p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/eql.html">EQL</a> support for CCS is available as of Elastic release 7.14, but it requires that all of your remote clusters have also been upgraded to 7.14 or newer. Many of the built in Elastic detection rules use EQL so if you are using CCS with clusters older than 7.14 you will need to disable those rules.<br><br>
</p>
<h2>Machine Learning with CCS</h2>
<p>To use the Machine Learning with CCS you will need to update each of the datafeeds to use the CCS index patterns. When creating your own custom Machine Learning jobs this is easy to do, you simply select the CCS index pattern when creating the new job. To use the built in Security Machine Learning jobs requires modifying the datafeeds of each job to use the CCS index patterns. Because you cannot change a datafeed’s index pattern via the UI, doing this requires access to the Kibana Devtools or API access. My method for doing this is to load the built-in Machine Learning jobs, stop all of the jobs, and then use the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-apis.html">Machine Learning API</a> to <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-datafeed.html">Get the datafeed</a>, make the change to the datafeed’s Index pattern, and then <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-update-datafeed.html">update the datafeed</a> with the changes. After this is complete all of the ML jobs and datafeeds can be restarted. If you used the built in Security ML jobs you can now enable the built in Detection rules that use Machine Learning.<br>
</p>
<h2>Conclusion</h2>
<p>In this post we showed you how to configure your Security Application and Machine Learning jobs to work with Cross Cluster Search. Keep an eye out for future blog posts from the Elastic Infosec team on how we use Elastic to protect Elastic.
</p>

blog-banner-security-pattern-color.png

blog-thumb-security-pattern-color.png

Elastic on Elastic: Configuring the Security app to use Cross Cluster Search

In this series of blogs we will provide an overview of our architecture, what data we send to our clusters, how and why we use Cross Cluster Search with the Security and Machine Learning (ML) applications, and how we tune, manage and notify.

<p>The Elastic Infosec Detections and Analytics team is responsible for building, tuning, and maintaining the security detections used to protect all Elastic systems. Within Elastic we call ourselves Customer Zero and <a href="https://www.elastic.co/blog/elastic-on-elastic-how-infosec-deploys-infrastructure-and-stays-up-to-date-with-eck">we strive to always use the newest versions</a> of our products.&nbsp; </p>  <p>In this series of blog posts we will provide an overview of our architecture, what data we send to our clusters, how and why we use Cross Cluster Search with the Security and Machine Learning (ML) applications, and how we tune, manage and notify analysts for those alerts. </p>  <p>In the <a href="https://www.elastic.co/blog/elastic-on-elastic-deep-dive-into-our-siem-architecture">previous blog post</a> we provided an overview of our internal Elastic infrastructure that we use in Infosec and how we use Cross Cluster Search to connect multiple clusters into a single interface for Security Analysts. In this blog post we will go into more detail about the specific types of data we collect. </p>   <strong><h2>Types of data searchable in our SIEM</h2></strong><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt75786479af524e1d/6132317ecc95cf70fa047d1c/blog-elastic-siem-1.png" data-sys-asset-uid="blt75786479af524e1d" alt="blog-elastic-siem-1.png"></p><strong></strong> <strong><h2>Auditbeat Cluster</h2></strong> <p>We use <a href="https://www.elastic.co/guide/en/beats/auditbeat/7.13/auditbeat-overview.html">Auditbeat</a> to monitor activity on all of our Linux servers and containers. We use the <a href="https://www.elastic.co/guide/en/beats/auditbeat/7.13/auditbeat-module-auditd.html">auditd</a> and <a href="https://www.elastic.co/guide/en/beats/auditbeat/7.13/auditbeat-module-system.html">System Module</a> to collect process execution, logins, network connections, system information, and the <a href="https://www.elastic.co/guide/en/beats/auditbeat/7.13/auditbeat-module-file_integrity.html">File Integrity Module</a> for monitoring of critical files. We use several <a href="https://www.elastic.co/guide/en/beats/auditbeat/7.13/filtering-and-enhancing-data.html">processors</a> in the auditbeat config to filter and enhance the data as we as it's collected.&nbsp; </p> <ul> <li aria-level="1">We use <a href="https://www.elastic.co/guide/en/beats/auditbeat/7.13/drop-event.html">drop_event</a> processors to filter out known good events from being logged. This lets us keep our costs down when centrally logging events from cloud systems. The drop_event process lets you write complex filters to remove your noisiest events before they ever logged.&nbsp;</li> <li aria-level="1"><a href="https://www.elastic.co/guide/en/beats/auditbeat/7.13/add-process-metadata.html">add_process_metadata</a> is used to add the parent process information to process events. Many of the Security app detections use parent-child process relationships to identify strange behavior so having this processor is critical.</li> <li aria-level="1"><a href="https://www.elastic.co/guide/en/beats/auditbeat/7.13/add-cloud-metadata.html">add_cloud_metadata</a> adds the cloud information to each event making it easier to identify which cloud provider, region, and account a system belongs to.</li> <li aria-level="1"><a href="https://www.elastic.co/guide/en/beats/auditbeat/7.13/add-docker-metadata.html">add_docker_metadata</a> to add container ID, name, and labels to the event.</li> <li aria-level="1"><a href="https://www.elastic.co/guide/en/beats/auditbeat/7.13/include-fields.html">include_fields</a> lets you add custom fields to a config. We use it to add custom ‘config.version’ and ‘team’ fields to the events so we can easily identify the owner of the system and any systems using an older version of the auditbeat config.</li> </ul> <strong><h3>Filebeat Cluster</h3></strong> <p>Our <a href="https://www.elastic.co/guide/en/beats/filebeat/7.13/index.html">Filebeat</a> cluster is where we use Filebeat to collect logs from the many third party systems we use at Elastic. Some of the third party services have built in Filebeat modules that make it very easy to to configure and collect the events in ECS formatting. The following built in Filebeat modules are being used: </p> <ul> <li aria-level="1"><a href="https://www.elastic.co/guide/en/beats/filebeat/7.13/filebeat-module-okta.html">Okta Module</a> collects events from the Okta API and it is used for many of our detections and investigations. We use Okta for Single Sign On to many applications in Elastic so having these logs is critical to identifying who accessed what.</li> <li aria-level="1"><a href="https://www.elastic.co/guide/en/beats/filebeat/7.13/filebeat-module-o365.html">Office 365 Module</a> is useful for security to monitor access to documents, and Azure AD authentication.</li> <li aria-level="1"><a href="https://www.elastic.co/guide/en/beats/filebeat/7.13/filebeat-module-nginx.html">Nginx module</a> collects access logs from the proxies in front of our critical web services. This can be very useful for tracking access or alerting on strange activity and web attacks.</li> <li aria-level="1"><a href="https://www.elastic.co/guide/en/beats/filebeat/7.13/filebeat-module-google_workspace.html">Google Workspace</a> module collects events from Gmail, Google Drive, and other Google services</li> <li aria-level="1"><a href="https://www.elastic.co/guide/en/beats/filebeat/7.13/filebeat-module-gcp.html">Google Cloud Module</a> to build alerts and investigate activity in GCP</li> <li aria-level="1"><a href="https://www.elastic.co/guide/en/beats/filebeat/7.13/filebeat-module-aws.html">AWS Module</a> to build alerts and investigate activity in AWS</li> <li aria-level="1"><a href="https://www.elastic.co/guide/en/beats/filebeat/7.13/filebeat-module-azure.html">Azure Module</a> to build alerts and investigate activity in Azure</li> </ul>  <p>Sometimes a built-in Filebeat module doesn’t exist so the SecEng team will have to build custom scripts and configurations for Filebeat to collect the information we need. The following logs are being collected in this way: </p> <ul> <li aria-level="1">Qualys - a script retrieves the results from our Qualys scans from the service and then converts the results from XML to json where it is then ingested by Filebeat. In addition to visualizing and alerting for Vulnerability Management, the Qualys data can be used to enrich other events. For example, if we see an alert for an exploit, and according to Qualys the host is vulnerable to that exploit we can elevate the priority of the investigation.</li> <li aria-level="1">Github - The Github events API is used to collect events about activity in all Elastic Owned repositories.&nbsp;</li> <li aria-level="1">HackerOne - Submitted reports from our bug bounty program. </li> <li aria-level="1">Shodan - The Shodan API is used to gather information for our Threat Intel team tracking exposure of Elastic clusters.</li> <li aria-level="1">RecordedFuture - Threat Intel feeds ingested for enriching events.</li> <li aria-level="1">Jamf - Inventory information about our Fleet of Mac systems. This makes it easier to find the registered owner of the system.&nbsp;</li> <li aria-level="1">LastPass - Activity from our corporate LastPass subscription.</li> </ul> <strong><h3>Monitoring Cluster</h3></strong> <p>A <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.13/monitor-elasticsearch-cluster.html">Monitoring Cluster</a> cluster is used to collect monitoring information from all of the other clusters. This information is used to audit activity on the other clusters and our Endgame SMPs as well as to monitor the performance on those clusters. Metricbeat, Filebeat, and Auditbeat logs from all of the other clusters are stored on this cluster.&nbsp; </p> <strong><h3>Endgame Cluster</h3></strong> <p>We use Endgame to <a href="https://www.elastic.co/blog/securing-our-own-endpoints-with-elastic-security">secure all of the workstations</a> used by Elasticians around the globe. In addition to being amazing at preventing attacks, Endgame can easily be configured to stream events to an Elastic cluster where we can use the machine learning and detection engine capabilities as well. Streaming the events to our SIEM lets us see the entire picture of activity on Elastic systems with the workstation events in the same dashboards and visualizations as the Okta SSO, Google Workspace, and other events.&nbsp; </p> <strong><h3>Fleet Cluster</h3></strong> <p>Our fleet cluster is where we manage the Elastic Agent and its new integrations such as Endpoint Security and OSquery. With Fleet you can deploy Elastic Agent to systems to collect observability data and have the ability to deploy and remove <a href="https://www.elastic.co/guide/en/fleet/current/fleet-overview.html#configuring-integrations">integrations</a> to the Elastic Agents to collect additional data as needed. </p>  <p>Because we are using Endgame for endpoint protection on our workstations we are not yet using the Endpoint Security fleet integration at scale. As the Endpoint Security integration reaches feature parity with Endgame we will be migrating our systems off of Endgame and onto Endpoint Security. We cannot deploy Endgame and Endpoint Security to the same systems at the same time because they are not compatible with each other. All of the other Elastic Agent integrations can be used with Endgame. </p>The primary Fleet integration we use at this time is the <a href="https://www.elastic.co/about/press/elastic-announces-osquery-management-integration-for-unified-data-analysis-to-address-cyber-threats">OSQuery Manager</a> integration. The OSQuery Manager lets us schedule and run live OSQuery queries to actively gather information from our fleet of systems. <a href="https://osquery.io/">OSQuery</a> is an open source project that enables analysts to directly query their systems to gather information such as running processes, installed applications, disk encryption status, named pipes, installed Chrome extensions, and over 250 other types of queries. For a more in depth dive on how we use OSquery Manager at Elastic we presented this Webcast with the SANS institute:  <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;" id="docs-internal-guid-3cfe59ce-7fff-7eb1-5ee4-7b5d98521e53"><a href="https://www.sans.org/webcasts/operationalize-osquery-at-scale-with-elastic/">https://www.sans.org/webcasts/operationalize-osque...</a>  </span><strong><h2>Malware Sandbox</h2></strong> <p>This cluster is another Fleet server but this one is used only for deploying Elastic Agent to analyst VMs to instrument them as a <a href="https://www.elastic.co/blog/how-to-build-a-malware-analysis-sandbox-with-elastic-security">Malware Analysis Sandbox</a>. Rather than having each analyst maintain their own cluster for this the SecEng team created a single managed cluster that all of us can use to manage log collection from our Sandbox VMs. The ability to manage Fleet and to add and remove agents and policies dynamically requires Super User privileges on the cluster so this activity needs to be a separate cluster from the production clusters. With CCS All of the analysts can see the logs from this cluster which means we can have one Analyst detonate and analyze the malware on their sandbox but everyone has access to the events that were created. These events can then be added to a case and we can use the indicators of compromise from the sandbox to quickly search through our live data for any evidence of compromise. </p>  <strong><h2>Customer Zero continued</h2></strong> <p>In this post we walked through the types of searchable data in a single Elastic SIEM interface and how we use that data for Security Detection, Incident Response, Threat Hunting, Threat Intelligence, Compliance Auditing, and Vulnerability Management. </p>  <p>Be sure to check back for our Third part of this series which will show you how we configure the Security app and Detection Rules to work with Cross Cluster Search.&nbsp; </p>

Elastic on Elastic Series: Data collected to the Infosec SIEM

Take a deep dive into Elastic Infosec team’s architecture, the many sources of data collected for security uses, how and why cross-cluster search is used, and how to configure Elastic Security and machine learning to work with cross-cluster search.

<h2><strong>Elastic on Elastic&nbsp;</strong></h2><p>The Elastic Infosec Detections and Analytics team is responsible for building, tuning, and maintaining the security detections used to protect all Elastic systems. Within Elastic we call ourselves Customer Zero and <a href="https://www.elastic.co/blog/elastic-on-elastic-how-infosec-deploys-infrastructure-and-stays-up-to-date-with-eck">we strive to always use the newest versions</a> of our products.&nbsp;
</p><p>In this series of blog posts we will provide an overview of our architecture, what data we send to our clusters, how and why we use Cross Cluster Search (CCS) with the Security and Machine Learning (ML) applications, and how we tune, manage and notify analysts for those alerts.
</p> <strong><h2>Overview of the Elastic Infosec environment</h2></strong><p>Many cybersecurity products and security controls are built to support a traditional “Corporate Network” with workstations and servers on a local network, secured by a firewall. When an alert fires and you need to conduct forensics on a host, you physically go to the system to dump memory and image the hard drive. At Elastic we are not a ‘traditional’ company by design, so the Infosec team has to be innovative and flexible to secure Elastic without trying to force the company into a mold ‘in the name of security’. Luckily, the <a href="https://www.elastic.co/elastic-stack/">Elastic Stack</a> is the perfect solution for this.
</p><p>Elastic is a worldwide company that is <a href="https://www.elastic.co/about/distributed">distributed by design</a> with customers and employees in over 80 countries around the globe. Most of our employees are 100% remote, conducting their daily work from an Elastic owned laptop connected to the internet, via untrusted networks around the world. As an open source company, our source code is on GitHub and our Continuous Integration and software build systems are cloud based and <a href="https://elasticsearch-ci.elastic.co/">publicly</a> visible to contributors. Successfully monitoring and conducting forensics in this modern, distributed enterprise network requires collecting the right information and having it all available to analysts in a single location.
</p><p>In addition to our workstations and enterprise systems, we monitor and protect Elastic Cloud which provides Elastic SaaS services currently running over 37,000 Elastic clusters in more than 40 cloud regions hosted in AWS, GCP, Azure, and IBM. Elastic Cloud is a separate elevated privilege environment managed by the Cloud team, but the Elastic Infosec team is responsible for creating, monitoring, and responding to alerts there as well.&nbsp;
</p> <strong><h2>Zooming in on Elastic Infosec infrastructure&nbsp;</h2></strong><p>The Infosec’s Security Engineering (SecEng) team <a href="https://www.elastic.co/blog/elastic-on-elastic-how-infosec-deploys-infrastructure-and-stays-up-to-date-with-eck">builds and maintains</a> our collection of clusters using <a href="https://www.elastic.co/downloads/elastic-cloud-kubernetes">Elastic Cloud on Kubernetes</a> and Helm to deploy infrastructure and remain up-to-date. SecEng manages seven different clusters; Six ‘data’ clusters with different roles, and a seventh ‘search head’ cluster as our SIEM that uses CCS to query and alert on the data in all of the other clusters. We use the most recent stack versions and release candidates with real-world data, so&nbsp;stack developers will regularly use our clusters to find and fix problems and to improve the Elastic stack for our customers.&nbsp;
</p><p>A key to being able to successfully monitor and protect any organization is to collect the right data and to have it searchable in a single location. There is no single data source that has everything you need for Infosec. A common challenge many orgs face is logging and storing the right types of data, but storing each type of data in its own separate location. Sometimes the data has to stay separate due to different sensitivity or privacy levels, often the data sources are owned and maintained by different teams, and sometimes it is due to the costs of transferring data to a single location. With the power of CCS we are able to keep the data stored in separate clusters and still search and alert on that data from a single Kibana interface giving analysts the complete picture.
</p><p>With all of the information visible in a single location the Analysts can build their Detection, Threat Intel, and Incident Response environments within the Elastic Stack. When investigating an alert we can use Timelines in the Security app to investigate an alert by querying for a value across every index. In addition to the Security App timelines, we also use multiple custom ‘Investigation’ dashboards to quickly display the information we need from all of the various indices.&nbsp;
</p> <strong><h2>Critical alert scenario</h2></strong><p>Imagine a scenario where we receive a critical alert that an employee’s Okta account may be compromised. We can also see that there were some low risk behavioral alerts for that user’s workstation in the last month that were initially dismissed as false alarms but now require a second look. We can use the User Investigation dashboard to search with that user’s email address and quickly review all of their Okta authentication activity, any apps accessed using Okta Single Sign On, any Google Drive files that they accessed or changed, GitHub activity during this time, and any o365 files that were accessed during the time when their account may have been compromised. A dashboard panel displays the JAMF system inventory about the work laptop issued to that user as well as the user’s contact information.
</p><p>Using the system information we can pivot to the Host Investigation dashboard to see all activity on that system and if it has been compromised as well or just the Okta account. Seeing that there was no logged in user at the time of the alerts lets us know that the system may be compromised and needs to be isolated. The Okta events contain the IP address that was used to log in, we can also search for this IP to see if it has been seen in any of our other data sets such as the cloud services at that time.&nbsp;
</p><p>Investigations like this can be extremely time consuming if you do not have all of the events available in a single location. Without centralized collection of your events you will need to log into each of the hosted services administration consoles to review the logs within their environment.
</p><strong><h2>Cross Cluster Search for the win</h2></strong><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt32ab2081bae3add6/61278fc2227d807ff1d9d77d/blog-elastic-siem-1.png" data-sys-asset-uid="blt32ab2081bae3add6" alt="blog-elastic-siem-1.png">
</p><p>The initial reason we decided to use&nbsp;CCS was that we already had existing clusters containing the data, but we wanted to have all of our data searchable from a single cluster to simplify the management of watchers, dashboards, Machine Learning jobs, and detection engine rules. Instead of migrating all of the data into a single cluster we decided to use CCS. Before CCS, we had to maintain all of those objects and configurations in multiple clusters, often duplicating work across clusters.&nbsp;
</p><p>When investigating an alert we previously had to open multiple browser tabs to view the various data sources about the event. Now, when we investigate an alert we can create a case and timeline in the Security App that contains every event related to the case. For example, if we see an alert for strange activity on a server in the auditbeat data and the event contains an IP address we can query for that IP within a <a href="https://www.elastic.co/guide/en/security/7.13/timelines-ui.html">Timeline</a> on all indices and quickly see if that IP was seen on any of our other services such as the nginx proxy data, Okta, GitHub, Cloud provider logs, Endgame workstation events and more. Thanks to <a href="https://www.elastic.co/guide/en/ecs/current/ecs-reference.html">ECS</a> the fields have a standard format across all of the index patterns so a query for source.ip will return documents from any index with a match.
</p><p>In addition to these initial reasons, we have found many more advantages to this setup. Below are some of the many advantages we’ve found to using Cross Cluster Search for security.
</p><ul>
	<li aria-level="1">Storing different types of data in separate clusters means we don’t have a single point of failure for all of our data. When upgrading a cluster, all of the other clusters are available to use.</li>
	<li aria-level="1">We can grant developers access to test and troubleshoot on a specific data cluster without granting them access to all of the data used by Infosec.</li>
	<li aria-level="1">Data from Observability and APM clusters is often very useful to security analysts during an investigation. Rather than duplicating all of the data, we can add the remote clusters to our CCS configuration.</li>
	<li aria-level="1">Analysts can be given elevated privileges to build and manage alerts and indices on the SIEM cluster with read-only access to the original data on the remote cluster. This gives analysts full capabilities in Kibana while protecting the integrity of the original data for audit and regulatory purposes.</li>
	<li aria-level="1">Upgrading the primary ‘search-head’ Elastic cluster to a new version will grant you access to all of the new Kibana features, even if the other clusters have not been upgraded yet. This takes a lot of stress out of upgrading your production SIEM early and often because it lets you test out the newest features without risking an outage to the clusters that contain the actual data.</li>
	<li aria-level="1">As of 7.13, Fleet users must have superuser privileges on the cluster. We can keep the number of Fleet administrators very small on the fleet cluster while granting all other users read-only privileges to the events collected by the Fleet cluster.</li>
</ul> <strong><h2>Customer Zero continued</h2></strong><p>In this post we walked through an overview of how Elastic’s Infosec team uses the many capabilities of the Elastic stack to gather data from many different data sources in order to provide Analysts with incredible visibility into security related events.</p><p>Be sure to check back for our <a href="https://www.elastic.co/blog/elastic-on-elastic-data-collected-to-the-infosec-siem">second part of this series</a> which will dive into the types of searchable data in the Elastic SIEM and how you can help your teams simplify and streamline their efforts.
</p>

Elastic on Elastic Series: Deep dive into our SIEM architecture

Elastic on Elastic: Deep dive into our SIEM architecture

In this blog post, we will demonstrate how the Elastic Infosec team uses the Elastic Stack with Elastic Endpoint Security to build a fully instrumented malware analysis sandbox using free software.

AUTHOR

Articles by Aaron Jewitt

Elastic on Elastic: Configuring the Security app to use Cross Cluster Search

Elastic on Elastic Series: Data collected to the Infosec SIEM

Elastic on Elastic: Deep dive into our SIEM architecture

How to build a malware analysis sandbox with Elastic Security

Sign up for Elastic Cloud free trial

Products & Solutions

Company

We're hiring

Resources

Solutions

Elastic (ELK) Stack

New

Featured topics

News

AUTHOR

Articles by Aaron Jewitt

Elastic on Elastic: Configuring the Security app to use Cross Cluster Search

Elastic on Elastic Series: Data collected to the Infosec SIEM

Elastic on Elastic: Deep dive into our SIEM architecture

How to build a malware analysis sandbox with Elastic Security

Sign up for Elastic Cloud free trial

We're hiring