Utilising the Elastic Stack with ArcSight SIEM and Kafka

<table style="background: #FFFFD2;">
<tbody>
<tr>
	<td> Get your ArcSight security data into Elasticsearch and visualized in Kibana in literally minutes with the Logstash ArcSight module. <a href="https://www.elastic.co/arcsight">Learn more</a>.&nbsp;
	</td>
</tr>
</tbody>
</table>
Editor's Note: Be sure to check out the other posts in this 6-part blog series.&nbsp;<a href="https://www.elastic.co/blog/integrating-elasticsearch-with-arcsight-siem-part-1">Part 1</a>&nbsp;kicks off the series with getting started content.&nbsp;<a href="https://www.elastic.co/blog/integrating-elasticsearch-with-arcsight-siem-part-2">Part 2</a>&nbsp;continues the story with how to proactively monitor security data in Elasticsearch using X-Pack.&nbsp;This is part 3 that&nbsp;walks you through how to scale the architecture.&nbsp;<a href="https://www.elastic.co/blog/integrating-elasticsearch-with-arcsight-siem-part-4">Part 4</a>&nbsp;and&nbsp;<a href="https://www.elastic.co/blog/integrating-elasticsearch-with-arcsight-siem-part-5">Part 5</a>&nbsp;provide examples of setting up alerts for common security threats using the alerting features in X-Pack.&nbsp;<a href="https://www.elastic.co/blog/integrating-elasticsearch-with-arcsight-siem-part-6">Part&nbsp;6</a>&nbsp;extends the alerting&nbsp;story&nbsp;with automated anomaly detection using machine learning. 
In our previous blog in this series (<a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-1">part 1</a>, <a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-2">part 2</a>) we demonstrated a simple architecture for integrating ArcSight with Elasticsearch.
In this blog post, &nbsp;we will continue with the same theme by showing you how to scale the architecture.
If you need to have high indexing throughput and a long retention policy, you can set up a <a href="/blog/hot-warm-architecture-in-elasticsearch-5-x">hot-warm architecture for Elasticsearch</a>. For the ingestion side, you can use a message queue between ArcSight and Elasticsearch.&nbsp;This provides architectural isolation between the producer (Arcsight Smart Connector) and the consumer, thus allowing Logstash to be scaled independently. To process more data, you simply have to add more Logstash instances for a topic.
Kafka also helps to buffer the incoming data in case it exceeds the Elasticsearch cluster’s ability to ingest the data during a peak periods or spikes. Whilst Logstash persistent queues ensure end to end delivery, data loss is still possible due to the loss of a Logstash instance. Through data replication, Kafka protects against this scenario.
How to integrate ArcSight with Elasticsearch using Kafka as a messaging queue:
The following architecture becomes possible with the ArcSight Smart Connector, with HP recently adding Kafka as a supported destination. &nbsp;This architecture also requires the user to add a Kafka destination to the ArcSight Smart Connector, with Logstash nodes pulling data from the appropriate Kafka topic before indexing into Elasticsearch.
Logstash can read directly from Kafka using the <a href="/guide/en/logstash/current/plugins-inputs-kafka.html">Kafka input plugin</a>, which integrates natively using the <a href="https://kafka.apache.org/082/javadoc/overview-summary.html">Java APIs</a>.
You can find more details about using Kafka with the Elastic Stack in the blog series “Just Enough Kafka for the Elastic Stack” &nbsp;<a href="/blog/just-enough-kafka-for-the-elasticsearch-part1">Part 1</a> and <a href="/blog/just-enough-kafka-for-the-elasticsearch-part2">part 2</a>.
<img src="https://api.contentstack.io/v2/uploads/58890ecf08bb58b93e7b4ff6/download?uid=blt44d4da67526552a3" data-sys-asset-uid="blt44d4da67526552a3" alt="arcsight-3.png">
 
<h2>Setup the Elastic Stack </h2><ol>
	<li><a href="https://www.elastic.co/downloads/elasticsearch" target="_blank">Download and install Elasticsearch</a>. In the Elasticsearch install directory, install X-Pack and start Elasticsearch: <code>bin/elasticsearch-plugin install x-pack</code> <code>bin/elasticsearch</code></li>
	<li><a href="https://www.elastic.co/downloads/kibana" target="_blank">Download and install Kibana.</a> In the Kibana install directory, install X-Pack and start Kibana: <code>bin/kibana-plugin install x-pack</code> <code>bin/kibana</code></li>
	<li><a href="https://www.elastic.co/downloads/logstash" target="_blank">Download and install Logstash</a>. In the Logstash install directory, install X-Pack and update the Logstash Kafka input plugin: <code>bin/logstash-plugin install x-pack</code> <code>bin/logstash-plugin install logstash-input-kafka</code></li>
	<li>Start the Logstash ArcSight module by running the following command in the Logstash install directory: <code>bin/logstash --modules arcsight --setup -M "arcsight.var.inputs=kafka" -M "arcsight.var.input.kafka.bootstrap_servers=192.168.0.13:9092" -M "arcsight.var.elasticsearch.username=elastic" -M "arcsight.var.elasticsearch.password=changeme" -M "arcsight.var.kibana.username=elastic" -M "arcsight.var.kibana.password=changeme" -M "arcsight.var.input.kafka.topics=cef"</code></li>
</ol><ul>
	<li>The `--modules arcsight` option spins up an ArcSight CEF-aware Logstash pipeline for ingestion. The `--setup` option creates an “arcsight-*” index pattern in Elasticsearch and imports Kibana dashboards and visualizations. On subsequent module runs or when scaling out the Logstash deployment, the `--setup` option should be omitted to avoid overwriting the existing Kibana dashboards. For a full list of ArcSight module settings, see the documentation on <a href="https://www.elastic.co/guide/en/logstash/5.6/arcsight-module.html#arcsight-module-config" target="_blank">“ArcSight Module Configuration Options”. </a></li>
</ul><h2>Set up the Elastic Stack and Kafka with Docker</h2>In this example, we use Docker to simplify the installation and configuration of the Elastic and Kafka components.
<ol>
	<li>Ensure that:</li>
	<ul class="list-black">
		<li>You have <a href="https://docs.docker.com/engine/installation/">Docker Engine</a> installed.</li>
		<li>Your host meets the <a href="/guide/en/elasticsearch/reference/5.1/docker.html#docker-cli-run-prod-mode">prerequisites</a>.</li>
		<li>If you are on Linux, <a href="https://github.com/docker/compose/releases/latest">docker-compose</a> need to be installed.</li>
	</ul>
	<li>Download an example from <a href="https://github.com/elastic/examples/tree/master/Security%20Analytics/cef_with_kafka">here</a> to help you installing the full Elastic Stack with X-Pack plugin and also install Kafka. </li>
	<li>Set the environment variable &nbsp;KAFKA_ADVERTISED_HOST_NAME in the docker-compose.yml file.</li>
	<li>Then issue in the command line:</li>
</ol><pre class="pretty-print">$ docker-compose up.
</pre><h2>
Add Kafka destination to ArcSight Connectors </h2><ol>
	<li>Run the command &lt;installdir&gt;\current\bin\arcsight agentsetup</li>
	<li>Select ‘Yes’ to start the ‘wizard mode’</li>
	<li>Select ‘I want to add/remove/modify ArcSight Manager destinations’</li>
	<li>Select ‘add new destination’</li>
	<li>Select ‘Event Broker (CEF Kafka)’</li>
	<li>Add the information of the Kafka server and port you used in the docker-compose for the environments variable &nbsp;KAFKA_ADVERTISED_HOST_NAME and KAFKA_ADVERTISED_PORT</li>
	<li>Add the Topic name cef </li>
</ol><h2>Visualize and explore your data</h2><ol>
	<li>Point your web browser at http://localhost:5601/ to open Kibana. You should be prompted to log in to Kibana. To log in, you can use the built-in ‘elastic’ user and the password ‘changeme’. NOTE: These are the same credentials used in the arcsight module installation. When you change them ensure you update your logstash configuration.</li>
	<li>Open the “[ArcSight] Network Overview Dashboard”</li>
	<li>For more information on exploring security data with the ArcSight module, see the documentation on <a href="https://www.elastic.co/guide/en/logstash/5.6/arcsight-module.html#arcsight-module-config" target="_blank">“Exploring Your Security Data”.</a></li>
</ol><img src="https://lh4.googleusercontent.com/c0zdKKlzGhp5YL1OlnDhJj3c4eXz9bANII8MP5RBij3zgC_Y7Ufs-UE-RhBSS2g0u5JXyqGPLZldGklck9ZBpPdICgQpPgMWw4MCvKOwQsBVftNocaYCdxWsE0q_P0LWl0fbn45X" style="width: 898px; height: 1522px;" width="898" height="1522"> In our next post we will talk about &nbsp;X-Pack alerting features to detect and alert on more complex patterns within the same dataset.
Interesting in learning more? Check out other posts in the Elasticsearch + ArcSight series.
<ul>
	<li><a href="https://www.elastic.co/blog/integrating-elasticsearch-with-arcsight-siem-part-1">Part 1 - How to send Common Event Format data from ArcSight to Elasticsearch</a></li>
	<li><a href="https://www.elastic.co/blog/integrating-elasticsearch-with-arcsight-siem-part-2">Part 2 - How to proactively monitor security data in Elasticsearch with X-Pack alerting features</a></li>
	<li><a href="https://www.elastic.co/blog/integrating-elasticsearch-with-arcsight-siem-part-3">Part 3 - Scaling your ArcSight and Elasticsearch&nbsp;architecture</a></li>
	<li><a href="https://www.elastic.co/blog/integrating-elasticsearch-with-arcsight-siem-part-4">Part 4&nbsp;-&nbsp;Detecting Successful SSH Brute Force Attacks</a></li>
	<li><a href="https://www.elastic.co/blog/integrating-elasticsearch-with-arcsight-siem-part-5">Part 5&nbsp;-&nbsp;Detecting Unusual Process using Rules Based Alerts</a></li>
	<li><a href="https://www.elastic.co/blog/integrating-elasticsearch-with-arcsight-siem-part-6">Part 6&nbsp;-&nbsp;Using Machine Learning to Detect Rare (unusual) Processes on a Server</a></li>
</ul>

globe-magnifying-glass-man-banner.jpg

globe-magnifying-glass-man-thumb.jpg

Integrating the Elastic Stack with ArcSight SIEM - Part 3

A recommendation for using Elasticsearch 5.x for larger time-data analytics: indices & a tiered architecture with 3 different types of nodes, called “Hot-Warm”.

<html><head></head><body>When using elasticsearch for larger time data analytics use cases, we recommend using time-based indices and a tiered architecture with 3 different types of nodes (Master, Hot-Node and Warm-Node), which we refer to as the &quot;Hot-Warm&quot; architecture. Each node has their own characteristics, which are described below.
<h3>Master nodes</h3>We recommend running 3 dedicated master nodes per cluster to provide the greatest resilience. When employing these, you should also have the discovery.zen.minimum_master_nodes setting at 2, which prevents the possibility of a &quot;split-brain&quot; scenario. Utilizing dedicated master nodes, responsible only for handling cluster management and state enhances overall stability. Because they do not contain data nor participate in search and indexing operations they do not experience the same demands on the JVM that may occur during heavy indexing or long, expensive searches. And therefore are not as likely to be affected by long garbage collection pauses. For this reason they can be provisioned with CPU, RAM and Disk configurations much lower than those that would be required for a data node.
<h3>Hot nodes</h3>This specialized data node performs all indexing within the cluster. They also hold the most recent indices since these generally tend to be queried most frequently. As indexing is a CPU and IO intensive operation, these servers need to be powerful and backed by attached SSD storage. We recommend running a minimum of 3 Hot nodes for high availability. Depending on the amount of recent data you wish to collect and query though, you may well need to increase this number to achieve your performance goals.
<h3>Warm nodes</h3>This type of data nodes are designed to handle a large amount of read-only indices that are not as likely to be queried frequently. As these indices are read-only, warm nodes tend to utilize large attached disks (usually spinning disks) instead of SSDs. As with hot nodes, we recommend a minimum of 3 Warm nodes for high availability. And as before, with the caveat that larger amounts of data may require additional nodes to meet performance requirements. Also note that CPU and memory configurations will often need to mirror those of your hot nodes. This can only be determined by testing with queries similar to what you would experience in a production situation.
The Elasticsearch cluster needs to know which servers contain the hot nodes and which server contain the warm nodes. This can be achieved by assigning arbitrary <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.1/allocation-awareness.html#forced-awareness">attributes</a> to each server.
For instance, you could tag the node with <code>node.attr.box_type: hot</code> in elasticsearch.yml, or you could start a node using <code>./bin/elasticsearch -Enode.attr.box_type=hot</code>
The nodes on the warm zone are &quot;tagged&quot; with <code>node.attr.box_type: warm</code> in elasticsearch.yml or you could start a node using <code>./bin/elasticsearch -Enode.attr.box_type=warm</code>
The box_type attribute is completely arbitrary and you could name it whatever you like. These arbitrary values will be used to tell Elasticsearch where to allocate an index.
We can ensure today&apos;s index is on the hot nodes that utilize the SSD&apos;s by creating it with the following settings:
<pre class="prettyprint">PUT /logs_2016-12-26
{
 &quot;settings&quot;: {
 &quot;index.routing.allocation.require.box_type&quot;: &quot;hot&quot;
 }
}
</pre>After few days if the index no longer needs to be on the most performant hardware, we can move it to the nodes tagged as warm by updating its index settings:
<pre class="prettyprint">PUT /logs_2016-12-26/_settings 
{ 
 &quot;settings&quot;: { 
 &quot;index.routing.allocation.require.box_type&quot;: &quot;warm&quot;
 } 
}
</pre>Now how can we achieve that using logstash or beats:
If the index template is being managed at the logstash or beats level the index templates should be updated to include allocation filtering. The &quot;index.routing.allocation.require.box_type&quot; : &quot;hot&quot; setting will cause any new indices to be created on the hot nodes.
Example:
<pre class="prettyprint">{
 &quot;template&quot; : &quot;indexname-*&quot;,
 &quot;version&quot; : 50001,
 &quot;settings&quot; : {
 &quot;index.routing.allocation.require.box_type&quot;: &quot;hot&quot;
 ...
</pre>Another strategy is to add a generic template for any index in the cluster, &quot;template&quot;: &quot;*&quot; that creates new indices in hot nodes.
Example:
<pre class="prettyprint">{
 &quot;template&quot; : &quot;*&quot;,
 &quot;version&quot; : 50001,
 &quot;settings&quot; : {
 &quot;index.routing.allocation.require.box_type&quot;: &quot;hot&quot;
 ...
</pre>When you have determined an index is not undergoing writes and is not being searched frequently it can be migrated from the hot nodes to the warm nodes. This can be accomplished by simply updating its index settings: &quot;index.routing.allocation.require.box_type&quot; : &quot;warm&quot;. 
	Elasticsearch will automatically migrate the indices over to the warm nodes.
Finally we can also enable better compression on all the warm data nodes by setting index.codec: best_compression in the elasticsearch.yml. 
	When data is moved to warm nodes we can call the _forcemerge API in order to merge segments: not only will it save memory, disk-space and file handles by having fewer segments, but it will also have the side-effect of rewriting the index using this new best_compression codec.
It would be a bad idea to force merge the index while it was still allocated to the strong boxes, as the optimization process will swamp the I/O on those nodes and impact the indexing speed of today&apos;s logs. But the medium boxes aren&apos;t doing much, so it is safe to force merge them.
Now that we have seen how to manually change the shard allocation of an index, let&apos;s look at how to use one of our tools called <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/current/installation.html">Curator</a> to automate the process.
In the example below we are using curator 4.2 to move the indexes from hot nodes to warm nodes after 3 days:
<pre class="prettyprint">actions:
 1:
 action: allocation
 description: &quot;Apply shard allocation filtering rules to the specified indices&quot;
 options:
 key: box_type
 value: warm
 allocation_type: require
 wait_for_completion: true
 timeout_override:
 continue_if_exception: false
 disable_action: false
 filters:
 - filtertype: pattern
 kind: prefix
 value: logstash-
 - filtertype: age
 source: name
 direction: older
 timestring: &apos;%Y.%m.%d&apos;
 unit: days
 unit_count: 3
</pre>Finally we can use curator to force merge the index. Ensure you wait long enough for the reallocation to finish before running optimize. You can do that by setting wait_for_completion in action 1 or change the unit_count to select indices older than 4 days in the action 2, so they&apos;ve had a chance to fully migrate before a force merge.
<pre class="prettyprint"> 2:
 action: forcemerge
 description: &quot;Perform a forceMerge on selected indices to &apos;max_num_segments&apos; per shard&quot;
 options:
 max_num_segments: 1
 delay:
 timeout_override: 21600 
 continue_if_exception: false
 disable_action: false
 filters:
 - filtertype: pattern
 kind: prefix
 value: logstash-
 - filtertype: age
 source: name
 direction: older
 timestring: &apos;%Y.%m.%d&apos;
 unit: days
 unit_count: 3
</pre>Note timeout_override should be increased the default is 21600 seconds, but it may go faster or slower, depending on your setup
Since Elasticsearch 5.0 we can also use Rollover and shrink api to reduce the number of shards , which is a simpler, more efficient way of managing time-based indices. You can find more details about it in this <a href="/blog/managing-time-based-indices-efficiently">blog</a>.
Like this topic? Dive deeper at <a href="https://www.elastic.co/elasticon/conf/2017/sf">Elastic{ON}, our user conference,</a> and discuss it with our engineering team face to face.&#xA0;We hope to see you there. </body></html>

sunset-tree-cropped.jpg

sunset-tree-720x420.jpg

“Hot-Warm” Architecture in Elasticsearch 5.x

In this blog series we will provide an overview of how to extend and complement the capabilities of your existing SIEM to create an effective security analyt...

<html><head></head><body>Editor&apos;s Note: Like what you read? There&apos;s more where that came from. <a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-2">Part 2</a> continues the story with how to proactively monitor security data in Elasticsearch using X-Pack. <a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-3">Part 3</a> walks you through how to scale the architecture. <a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-4">Part 4</a> and <a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-5">Part 5</a> provide examples of setting up alerts for common security threats using the alerting features in X-Pack. <a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-6">Part 6</a> extends the alerting story with automated anomaly detection using machine learning.

In this blog series we will provide an overview of how to extend and complement the capabilities of your existing SIEM to create an effective security analytics solution for your organization. For the purposes of example, we will demonstrate the use of an X-Pack enabled Elastic Stack with one of the SIEM solutions...ArcSight.

The following demonstrates an example of Elasticsearch with the ArcSight SIEM. The existing ArcSight Smart Connector can be used to send data to Elasticsearch, with multiple possible approaches to configuration. The simplest solution is to add a CEF syslog destination to the ArcSight Smart Connector allowing it to send data to the Logstash. In Logstash we will use&#xA0; Logstash ArcSight module to setup this integration. The module includes a Logstash configuration for ingesting and enriching CEF-formatted data from ArcSight Smart Connectors, while bundling a set of Kibana dashboards to view events from common sources. 

<img src="https://api.contentstack.io/v2/uploads/59b8db123ef8e08c0d90b26d/download?uid=blt7eeec6c86c1f6691" data-sys-asset-uid="blt7eeec6c86c1f6691" alt="ArcSight Smart Connectors">


<h2>Setup the Elastic Stack</h2>

<ol>
	<li><a href="https://www.elastic.co/downloads/elasticsearch">Download and install Elasticsearch</a>. In the Elasticsearch install directory, install X-Pack and start Elasticsearch: <pre class="prettyprint">bin/elasticsearch-plugin install x-pack</pre> <pre class="prettyprint">bin/elasticsearch</pre></li>
	<li>
	<a href="https://www.elastic.co/downloads/kibana">Download and install Kibana</a>. In the Kibana install directory, install X-Pack and start Kibana: <pre class="prettyprint">bin/kibana-plugin install x-pack</pre> <pre class="prettyprint">bin/kibana</pre></li>
	<li><a href="https://www.elastic.co/downloads/logstash">Download and install Logstash</a>. In the Logstash install directory, install X-Pack and <pre class="prettyprint">bin/logstash-plugin install x-pack</pre></li>
	<li>Start the Logstash ArcSight module by running the following command in the Logstash install directory: 
<pre class="prettyprint">bin/logstash --modules arcsight --setup -M 
&quot;arcsight.var.inputs=smartconnector&quot; -M
&quot;arcsight.var.input.smartconnector.port=5000&quot; -M
&quot;arcsight.var.elasticsearch.username=elastic&quot; -M
&quot;arcsight.var.elasticsearch.password=changeme&quot; -M
&quot;arcsight.var.kibana.username=elastic&quot; -M
&quot;arcsight.var.kibana.password=changeme&quot;
</pre> The `--modules arcsight` option spins up an ArcSight CEF-aware Logstash pipeline for ingestion. The `--setup` option creates an &quot;arcsight-*&quot; index pattern in Elasticsearch and imports Kibana dashboards and visualizations. On subsequent module runs or when scaling out the Logstash deployment, the `--setup` option should be omitted to avoid overwriting the existing Kibana dashboards. For a full list of ArcSight module settings, see the documentation on &quot;<a href="https://www.elastic.co/guide/en/logstash/5.6/arcsight-module.html#arcsight-module-config">ArcSight Module Configuration Options</a>&quot;.</li>
</ol>

<h2>Setup the Elastic Stack with Docker</h2>
For a quick setup you can download an example <a href="https://github.com/elastic/examples/blob/master/Security%20Analytics/cef_with_logstash/docker-compose.yml">docker-compose.yml </a>definition to help you to install all the elastic stack with x-plugin ( step 1 to 5 ), then issue:

<pre class="prettyprint">$ docker-compose up</pre>

But First, ensure that:


<ol>
	<li>You have<a href="https://docs.docker.com/engine/installation/"> Docker Engine</a> installed.</li>
	<li>
	Your host meets the <a href="/guide/en/elasticsearch/reference/5.1/docker.html#docker-cli-run-prod-mode">prerequisites</a>.</li>
	<li>If you are on Linux, that<a href="https://github.com/docker/compose/releases/latest"> docker-compose</a> is installed.</li>
</ol>
<h2>Configure ArcSight Smart Connectors</h2>





	<ol>
		<li>Configure ArcSight connectors to send data to Logstash</li>
		<li>Run the command ..&lt;installdir&gt;\current\bin\arcsight agentsetup</li>
		<li>Choose yes to start the &#x2018;wizardmode&#x2019;</li>
		<li>Choose &#x2018;I want to add/remove/modify ArcSight Manager destinations&#x2019;</li>
		<li>Choose &#x2018;add new destination&#x2019;</li>
		<li>Choose &#x2018;CEF syslog&#x2019;</li>
		<li>Add the information of the logstash host and port 5000 you prepared and choose the TCP protocol.</li>
	</ol>


<h2>Explore your data with Kibana </h2>
<ol>
	<li>Point your web browser at http://localhost:5601/ to open Kibana. You should be prompted to log in to Kibana. To log in, you can use the built-in &#x2018;elastic&#x2019; user and the password &#x2018;changeme&#x2019;. NOTE: These are the same credentials used in the arcsight module installation. When you change them ensure you update your logstash configuration.</li>
	<li>Open the &quot;[ArcSight] Network Overview Dashboard&quot;</li>
</ol>
<img src="https://api.contentstack.io/v2/uploads/59b8dcda75d9f7760dfc1887/download?uid=bltb837402809a9b164" data-sys-asset-uid="bltb837402809a9b164" alt="[ArcSight] Network Overview Dashboard">

<img src="https://api.contentstack.io/v2/uploads/59b8dc6e75d9f7760dfc187f/download?uid=blt65b98cf98daaefd9" data-sys-asset-uid="blt65b98cf98daaefd9" alt="[ArcSight] Network Overview Dashboard">


For more information on exploring security data with the ArcSight module, see the documentation on &quot;<a href="https://www.elastic.co/guide/en/logstash/5.6/arcsight-module.html#arcsight-module-config">Exploring Your Security Data</a>&quot;.

In this blog series, we will also cover using ArcSight with Kafka, and X-Pack Alerting to notify when security events occur.
	

If you want to check out more about security analytics presentations, we suggest<a href="/elasticon/conf/2016/sf/tapping-out-security-threats-at-fireeye"> </a><a href="/elasticon/conf/2016/sf/all-quiet-digital-front-security-analytics-usaa">Security Analytics @ USAA</a>, <a href="/elasticon/conf/2016/sf/tapping-out-security-threats-at-fireeye">Tapping Out Security</a> with FireEye,<a href="/elasticon/conf/2016/sf/hunting-the-hackers-how-cisco-talos-is-leveling-up-security"> Hunting the Hackers</a> by Cisco&apos;s Talos,<a href="/elasticon/tour/2015/los-angeles/keeping-your-data-from-getting-swiped-right-away-security-analytics-at-tinder"> Tinder: Keeping Your Data From Getting Swiped Right Away</a> , and<a href="/elasticon/tour/2015/washington-dc/cyber-security-log-analytics-at-decision-lab"> Cyber Security Log Analytics</a> with Decision Lab.
	

Interesting in learning more? Check out other posts in the Elasticsearch + ArcSight series.
	

<ul>
	<li><a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-1">Part 1 - How to send Common Event Format data from ArcSight to Elasticsearch</a></li>
	<li><a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-2">Part 2 - How to proactively monitor security data in Elasticsearch with X-Pack alerting features</a></li>
	<li><a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-3">Part 3 - Scaling your ArcSight and Elasticsearch&#xA0;architecture</a></li>
	<li><a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-4">Part 4&#xA0;-&#xA0;Detecting Successful SSH Brute Force Attacks</a></li>
	<li><a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-5">Part 5&#xA0;-&#xA0;Detecting Unusual Process using Rules Based Alerts</a></li>
	<li><a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-6">Part 6&#xA0;-&#xA0;Using Machine Learning to Detect Rare (unusual) Processes on a Server</a></li>
	
</ul></body></html>

blog-siem-fb.jpg

blog-siem-thumb.jpg

Integrating the Elastic Stack with ArcSight SIEM - Part 1

Learn about Docker networking and how to use it with an Elasticsearch cluster.

<html><head></head><body>In this&#xA0;blog we&#x2019;ll talk about network considerations when using Docker with an Elasticsearch cluster.&#xA0;Note: In this blog we will reference the Elasticsearch image found on the Docker Hub. The development and production of this Docker image is not affiliated with Elastic.You can create your own image by following our recommendation in the blog <a href="https://www.elastic.co/blog/how-to-make-a-dockerfile-for-elasticsearch">How to make a Dockerfile for Elasticsearch</a>.
There are different ways to setup networking in Docker and by default three&#xA0;network types are presented. We can list all of them using the following command:
<pre class="prettyprint prettyprinted">$ docker network ls
NETWORK ID NAME DRIVER
d610d782daa0 bridge bridge 
16a982d835f8 none null 
7d80e0e91caf host host
</pre><h3>None Network</h3>It completely disables networking, which is not useful when running an Elasticsearch cluster.
<h3>Host Network </h3>If you use <code>--net=host</code> then the container will use the host network and this can be dangerous. It will allow you to change the host network from within the container and if you have an application running as root and it has a vulnerability, there is risk of unsolicited remote control of the host network via the the Docker container. In general we recommend against using it for security reasons, but it can be useful when you need to get the best network performance because it&#xA0;is as fast as normal host networking.
<h3>Bridge Network</h3>The bridge network is the default network in Docker.
We can check the details of the default&#xA0;bridge network using the following command:
<pre class="prettyprint prettyprinted">$ docker network inspect bridge
[
 {
 &quot;Name&quot;: &quot;bridge&quot;,
 &quot;Id&quot;: &quot;b38c312777a0f3890034c9b396669842947b80c9051d10a283c9d43937910578&quot;,
 &quot;Scope&quot;: &quot;local&quot;,
 &quot;Driver&quot;: &quot;bridge&quot;,
 &quot;IPAM&quot;: {
 &quot;Driver&quot;: &quot;default&quot;,
 &quot;Options&quot;: null,
 &quot;Config&quot;: [
 {
 &quot;Subnet&quot;: &quot;172.17.0.0/16&quot;
 }
 ]
 },
 &quot;Containers&quot;: {},
 &quot;Options&quot;: {
 &quot;com.docker.network.bridge.default_bridge&quot;: &quot;true&quot;,
 &quot;com.docker.network.bridge.enable_icc&quot;: &quot;true&quot;,
 &quot;com.docker.network.bridge.enable_ip_masquerade&quot;: &quot;true&quot;,
 &quot;com.docker.network.bridge.host_binding_ipv4&quot;: &quot;0.0.0.0&quot;,
 &quot;com.docker.network.bridge.name&quot;: &quot;docker0&quot;,
 &quot;com.docker.network.driver.mtu&quot;: &quot;1500&quot;
 }
 }
]
</pre>The&#xA0;bridge network name here is <code>docker0</code>. When you add a container,&#xA0;each of them will have&#xA0;its own virtual Ethernet interface connected to the docker bridge&#xA0;<code>docker0</code> and it will have an IP address allocated to the&#xA0;virtual interface.
This bridge will automatically forward packets between any other network interfaces that are attached to it and also allow containers to communicate with the host machine as well as with the containers on the same host.
By default,&#xA0;Docker containers can make connections to the outside world,&#xA0;they connect via the <code>docker0</code> interface but the outside world cannot connect to containers.
External connectivity is provided by IP forwarding and <code>iptables</code> rules. You can achieve that using the port mapping.
When running Elasticsearch, you will need to ensure it&#xA0;publishes to an IP address that is reachable from outside the container; this can be configured via the setting <code>network.publish_host</code>.
For the discovery between the nodes you have to configure&#xA0;Zen Discovery via the settings <code>discovery.zen.ping.unicast.hosts</code> and <code>discovery.zen.minimum_master_nodes</code>.
Example:
<pre class="prettyprint prettyprinted">docker run -d -p 9200:9200 -p 9300:9300 elasticsearch:2 \
 elasticsearch \
 -Des.discovery.zen.ping.unicast.hosts=192.168.99.100,192.168.99.101 \
 -Des.discovery.zen.minimum_master_nodes=2 \
 -Des.network.publish_host=192.168.99.100
</pre><pre class="prettyprint prettyprinted">docker run -d -p 9200:9200 -p 9300:9300 elasticsearch:2 \
 elasticsearch \
 -Des.discovery.zen.ping.unicast.hosts=192.168.99.100,192.168.99.101 \
 -Des.discovery.zen.minimum_master_nodes=2 \
 -Des.network.publish_host=192.168.99.101
</pre>By default, a Docker container is configured to&#xA0;use IPv4 only. It is possible to configure IPv4/IPv6 by starting the Docker daemon with the <code>--ipv6</code> flag.
When creating a container it will get a link-local IP address. You can also assign a globally routable IPv6 addresses to your containers.
Using routable IPv6 addresses allows you to realize communication between containers on different hosts. The following article provides a lot more information on this subject:&#xA0;<a href="https://docs.docker.com/v1.5/articles/networking/#ipv6">Advanced networking</a>.<h3>Overlay Network</h3>In recent versions of Docker they introduced a new type of network called overlay network which Docker recommends for multi-host networking.
To use overlay networking you will need to setup a key-value store so that nodes can be discovered and added to the cluster; Docker currently supports only Consul, etcd, and ZooKeeper.
</body></html>

docker-construction.jpg

docker-networking-image.jpg

Docker Networking

A recommendation for using Elasticsearch for larger time-data analytics: indices & a tiered architecture with 3 different types of nodes, called “Hot-Warm”.

<html><head></head><body>The current blog applies to Elasticsearch versions 1.x and 2.x. If you are looking for &#x201C;Hot-Warm&#x201D; Architecture in Elasticsearch 5.x click <a href="https://www.elastic.co/blog/hot-warm-architecture-in-elasticsearch-5-x">here.</a> When using Elasticsearch for larger time-data analytics use cases, we recommend using time-based indices and a tiered architecture with 3 different types of nodes (Master, Hot-Node and Warm-Node), which we refer to as the &quot;Hot-Warm&quot; architecture. Each node has their own characteristics, which are described below.<h3>Master nodes</h3>We recommend running 3 dedicated master nodes per cluster having dedicated master nodes which run in their own JVM increases stability and resilience as they are not affected by garbage collection that can 
affect other types of nodes. These nodes do not handle requests and do not hold any data, and therefore only require less resources (such as CPU, RAM and Disk)<h3>Hot data nodes</h3>Hot data nodes are designed to perform all indexing within the cluster and will also hold the most recent daily indices that generally tend to be queried most frequently. As indexing is a very IO intensive operation, these servers need to be powerful and backed by attached SSD storage.<h3>Warm data nodes</h3>Warm data nodes are designed to handle a large amount of read-only indices that are not queried frequently.&#xA0;As these indices are read-only, warm nodes tend to utilise very large attached spinning disks instead of SSDs.Elasticsearch needs to know which servers contain the hot nodes and which server contain the warm nodes. This can be achieved by assigning arbitrary tags to each server.For instance, you could tag the node with <code>node.box_type: hot in 
elasticsearch.yml</code>, or you could start a node using <code>./bin/elasticsearch 
--node.box_type hot</code>.And the nodes on the warm zone are &quot;tagged&quot; with <code>node.box_type: warm in elasticsearch.yml</code> or you could start a node using <code>./bin/elasticsearch --node.box_type warm</code>The box_type parameter is completely arbitrary and you could name it whatever you like. These arbitrary values will be used to tell Elasticsearch where to allocate an index.We can ensure that today&#x2019;s index is on our SSD boxes by creating it with the following settings:<pre class="prettyprint">PUT /logs_2015-08-31
{
 &quot;settings&quot;: {
 &quot;index.routing.allocation.require.box_type&quot; : &quot;hot&quot;
 }
}</pre>After few days if the index no longer needs to be on our strongest boxes, we can move it to the nodes tagged as warm by updating its index settings:<pre class="prettyprint">PUT /logs_2014-08-31/_settings
{
 &quot;index.routing.allocation.require.box_type&quot; : &quot;warm&quot;
}</pre>Now how can we achieve that using Logstash :We need update the index templates to include allocation filtering &quot;index.routing.allocation.require.box_type&quot; : &quot;hot&quot; so that any new 
indices are created on the nodes that are tagged with hot.<h3>Example:</h3><pre class="prettyprint">{
 &quot;template&quot;: &quot;logstash-*&quot;,
 &quot;settings&quot;: {
 &quot;index.refresh_interval&quot;: &quot;5s&quot;,
 &quot;index.routing.allocation.require.box_type&quot;: &quot;hot&quot;
 },
 &quot;mappings&quot;: {
 &quot;_default_&quot;: {
 &quot;_all&quot;: {
 &quot;enabled&quot;: true,
 &quot;omit_norms&quot;: true
 },
 &quot;dynamic_templates&quot;: [
 {
 &quot;message_field&quot;: {
 &quot;match&quot;: &quot;message&quot;,
 &quot;match_mapping_type&quot;: &quot;string&quot;,
 &quot;mapping&quot;: {
 &quot;type&quot;: &quot;string&quot;,
 &quot;index&quot;: &quot;analyzed&quot;,
 &quot;omit_norms&quot;: true
 }
 }
 },
 {
 &quot;string_fields&quot;: {
 &quot;match&quot;: &quot;*&quot;,
 &quot;match_mapping_type&quot;: &quot;string&quot;,
 &quot;mapping&quot;: {
 &quot;type&quot;: &quot;string&quot;,
 &quot;index&quot;: &quot;analyzed&quot;,
 &quot;omit_norms&quot;: true,
 &quot;fields&quot;: {
 &quot;raw&quot;: {
 &quot;type&quot;: &quot;string&quot;,
 &quot;index&quot;: &quot;not_analyzed&quot;,
 &quot;ignore_above&quot;: 256
 }
 }
 }
 }
 }
 ],
 &quot;properties&quot;: {
 &quot;@version&quot;: {
 &quot;type&quot;: &quot;string&quot;,
 &quot;index&quot;: &quot;not_analyzed&quot;
 },
 &quot;geoip&quot;: {
 &quot;type&quot;: &quot;object&quot;,
 &quot;dynamic&quot;: true,
 &quot;properties&quot;: {
 &quot;location&quot;: {
 &quot;type&quot;: &quot;geo_point&quot;
 }
 }
 }
 }
 }
 }
}</pre>When indexes are no longer being written to and not being searched 
frequently they no longer need to be on the hot nodes. We can move them 
to the nodes tagged as warm by updating their index settings: 
&quot;index.routing.allocation.require.box_type&quot; : &quot;warm&quot;.Elasticsearch will automatically migrate the indices over to the warm nodes.&#xA0;Finally we recommend running an optimize on the migrated indices.&#xA0;It would be a bad idea to optimize the index while it was still allocated to the strong boxes, as the optimization process could swamp the I/O on those nodes and impact the indexing of today&#x2019;s logs. But the 
medium boxes aren&#x2019;t doing very much, so we are safe to optimize.Now that we have seen how to change the allocation of an index, let&apos;s look at how to use Curator to automate the process.In the example below we are using Curator 3.0 to move the indexes from hot nodes to warm nodes after 3 days.<pre class="prettyprint">/usr/bin/curator --logfile /var/log/curator.log --loglevel INFO --logformat default --master-only --host localhost --port 9200 allocation --rule box_type=warm indices --time-unit days --older-than 3 --timestring &apos;%Y.%m.%d&apos;</pre>Finally we can use Curator to optimize the index, make sure you wait enough for the reallocation to finish before running optimize.<pre class="prettyprint">/usr/bin/curator --host localhost --port 9200 optimize indices --older-than 3 --time-unit days --timestring &apos;%Y.%m.%d&apos;</pre></body></html>

Articles by Samir Bennacer

Integrating the Elastic Stack with ArcSight SIEM - Part 3

“Hot-Warm” Architecture in Elasticsearch 5.x

Integrating the Elastic Stack with ArcSight SIEM - Part 1

Docker Networking

“Hot-Warm” architecture

プロダクト

Elasticについて

Resources

SaaS

制御

Articles by Samir Bennacer

Integrating the Elastic Stack with ArcSight SIEM - Part 3

“Hot-Warm” Architecture in Elasticsearch 5.x

Integrating the Elastic Stack with ArcSight SIEM - Part 1

Docker Networking

“Hot-Warm” architecture